"freedom of association" and "freedom of speech" are governmental concepts, used to limit the behaviors of governments.

They are not some core, universal rights that every individual must respect when interacting with other individual.

The accused in this case absolutely still has the citizen rights of association and speech. He can gather with people and he can publish his thoughts. The fact that a bunch of individuals have decided they don't want to gather with him is in no way a reduction of his rights.

To be clear, I agree with you -- that was the point I was making.

There's no right to being accepted, and no right to make people approve of your actions.

It's not actually a problem in society that needs fixing if people decide not to associate with someone on the basis of their behaviour.

I agree with you too.

It's not a problem you can, or should, solve legally.