In my view it's okay to tell them that at any time, and just as acceptable for them to throw it back in our faces. What's worrisome is when action is taken to silence and suppress speech considered problematic, because that is a precedent that eventually can be used against anyone.

So yes. The world will only grow louder.