That's the whole point: fraud, perjury, and several other harmful things are subcategories of speech.