If I don't sue you, yeah. But if you're not sure that I won't have a problem with it, you should better ask me in advance.
In your example, this sounds silly, but it's just not categorically different from examples where it doesn't, like say someone who stalks me asks you for my phone number, or a scam caller does.
