I guess it's not end-to-end, it's decrypted on the server.

Presumably if you want to send an encrypted message from one literal endpoint to another, you'd use some other technology. I'm prepared to bet there are enough people doing just that, too.