But easily revised to "Do you want to come with us?"

Not that I particularly think the rules are inviolable. I see them as guidelines: if you're using the passive voice then make sure that it won't be better in active. If you're ending with a preposition, then perhaps there's a clearer way to phrase it.

Which is extremely informal, as the pronoum "me" or "us" is omitted from the end, but the context of usage would imply its presence.