There’s been some success training models on top of differential privacy.
I imagine that with live requests it would be quite challenging but not impossible, assuming you could somehow sanitize all sorts of private data that people throw at these prompts.
This might be the cost of privacy, and it might be worth paying, unless cloud models reach an inflection point that make local models archaic.