To elaborate; downtown and midtown are the two transport nexuses in New York City where most if not all suburban labor markets are easy to access, and employers want to have their pick of employees.
NYC is an archipelago separated by large bodies of water, and it is hard to avoid going through Manhattan to go between the western suburbs, the northern suburbs/Bronx, and the eastern suburbs/Brooklyn/Queens. What few links exist are always horribly congested.
The streetcars were equivalent to today’s buses, and were about as fast. Brooklyn and Queens is still only about 4m of the 21m total population of the metro area, so it’s still a small share. Even Nassau to Brooklyn or Queens takes overly long and/or is too expensive.
The West Village is actually not very well connected transport wise, and historically the waterfronts were docks-oriented. Downtown is the historic core where pre-bridges, all the ferries went, and where many bridges still exist. Midtown is the location of the rail terminals, so a suburbanite can walk to their job without having to pay a second fare. (IIRC Grand Central is where it is because at some point the city banned surface-level steam trains below.)
Downtown is where all the ferry landings were, and where a lot of the bridges were built after.
Midtown is where the rail terminals stop, so suburbanites can quickly walk to their jobs instead of paying a second fare for subway/bus/taxi.
There is no correlation between bedrock depth and height in Manhattan, as explained in the article of the grandparent comment. In fact, most of Downtown is landfill.
NYC is an archipelago separated by large bodies of water, and it is hard to avoid going through Manhattan to go between the western suburbs, the northern suburbs/Bronx, and the eastern suburbs/Brooklyn/Queens. What few links exist are always horribly congested.