You see apartment-finding startups all the time here in NYC. The problem to finding a good apartment isn't that there isn't a good interface, it's that a lot of the listings are given exclusively to brokers, and even if you find the apartment yourself, your still paying the broker fee.
I'm just one example, but I sublet from an individual owner in a co-op (in the Village). For giving the unit to a broker (for free, since the broker usually collects on the renter, not the landlord), he didn't have to fly up here from Miami and show the unit, figure out how to run a credit check, schedule my interview with the co-op, have me fill out a pile of forms, etc. Instead, he simply received a copy of the contract, signed and returned.
Point being: NYC has a ~1% vacancy rate, so there's little incentive for things to change on the sell side. Anything that requires a landlord to change their behavior is a tough sell.
A slightly higher rent is washed away in 1 month of not having the place rented out, it's definitely not worth any risk of not being able to rent a place out for a higher rent, so you charge what the market takes right now in a rush.
It's New York, they can charge basically whatever they want and somebody will still rent it. If they can't afford the rent, they'll turn it into an Air BnB.
This... really isn't true. Very expensive markets are still markets - you are, for example, very unlikely to find a taker for a studio 5th-floor walkup on the Lower East Side for $5k.
NYC rents are pretty nutty, but it is still a market, and pricing pressure is still very aggressive in this market. The volume that exists in the market, and the similarities and substitutability of apartments (one walkup is largely like another) all serve to apply more pricing pressure.
I'm just one example, but I sublet from an individual owner in a co-op (in the Village). For giving the unit to a broker (for free, since the broker usually collects on the renter, not the landlord), he didn't have to fly up here from Miami and show the unit, figure out how to run a credit check, schedule my interview with the co-op, have me fill out a pile of forms, etc. Instead, he simply received a copy of the contract, signed and returned.
Point being: NYC has a ~1% vacancy rate, so there's little incentive for things to change on the sell side. Anything that requires a landlord to change their behavior is a tough sell.