The relatively poor could be generating substantial economic value for the the rich and providing useful services for the folks in the middle or they could be costing substantial resources to poorly manage on the street, wandering into emergency rooms as a train wreck and causing massive bills both in terms of law enforcement, social services, and medicine that they will never pay back based on whether we think affordable housing and safety nets matter.

A functioning society is in everyone's interest. Only the really rich can afford to trivially live somewhere else and profit enough from austerity to net out to positive even while wrecking else's backyard which is why its so strange to see those in the middle jumping on that train.