I know. I’m not from NYC anyway. But it was the grandparent that started talking about the five Burroughs.
I think your causality is backwards. Why has downtown and midtown always been the two main population centers, and not one continuous city? Because that’s where the exposed bedrock is, with some swampy midlands in between in the West Village. Geology.