My take: I think that whiteness is not a lack of culture. It is the dominant culture, which makes it feel like an absence of culture to most within it. Like a massive white (pun intended) wall, which makes other cultures clearly visible in front of it.

I would also guess that most people saying that are centrists/moderates cosplaying a different political ideology, making it even harder to see any distinct features or a sense of community and belonging.