Almost nobody is buying exclusively-zoned property for racist reasons. But the availability of those properties under those terms is an intended consequence of racist decisions, and so it is fair to re-evaluate whether it's reasonable for those terms to be enforced by the state's monopoly on violence. I say that in most exclusively-zoned areas in or near urban cores, the clear answer to that question is "no": most of what's single-family zoned today should be upzoned to fourplexes. That argument is easy to make on the merits: there's a real (and rippling) cost to maintaining single-family zones near cities, and the benefits accrue to a fortunate few.
I say this as someone who owns an extremely and multivalently single-family property, the only property I own, and is working actively and primarily on upzoning that specific area. I have neighbors that would say what you're saying: you bought into wherever you live based on the promise that the state would commit arms to preventing any of your neighbors from admitting too many new residents. My response, when we succeed: "we are altering the deal."
I say this as someone who owns an extremely and multivalently single-family property, the only property I own, and is working actively and primarily on upzoning that specific area. I have neighbors that would say what you're saying: you bought into wherever you live based on the promise that the state would commit arms to preventing any of your neighbors from admitting too many new residents. My response, when we succeed: "we are altering the deal."