It is a good question but the answer may be unsatisfactory: it depends. I think that the popularity of scrum is due to its catch-all nature and the way it sounds reasonable when you explain it to someone, especially a non-developer.
Here is an answer that (I believe) works well: Hire a team lead that is willing to shield a small dev team (less than about 7 people) from the politics above. The devs still talk to users and other people in the company but they do not necessarily have to be accountable to them. The team lead understands the company’s budget cycles, has a vision for the product being developed and, importantly, has the time to sit down with each developer on the team to look at what they are making. Not a code review but a regular show-and-tell kind of arrangement. The fine line in this approach is to make sure it doesn’t degrade to micromanagement and ego poking.
Sometimes a lot can be understood from the team and it's current process in terms of where it did, or didn't come from.
Having a team lead that is technical as a product manager can be very helpful as you are outlining. Being able to speak the language of and maintain the respect of both is so valuable in terms of "getting it".
Clearing the way for devs to learn and do with customers and each other is the other key thing I think about a lot.
Startups likely have less politics (hopefully) but the longer they operate as startups politics likely increase, or hides itself better.
When I see startups leaping to hire VP engineering, etc, I can't help but think of my own experiences where having the founders at those seats translating what is being learned from customers directly into the product was so critical.
For existing or larger organizations, I think what you're saying is very true.
Here is an answer that (I believe) works well: Hire a team lead that is willing to shield a small dev team (less than about 7 people) from the politics above. The devs still talk to users and other people in the company but they do not necessarily have to be accountable to them. The team lead understands the company’s budget cycles, has a vision for the product being developed and, importantly, has the time to sit down with each developer on the team to look at what they are making. Not a code review but a regular show-and-tell kind of arrangement. The fine line in this approach is to make sure it doesn’t degrade to micromanagement and ego poking.