Gravity 'helps' too much, by making you go faster as you get lower, and then you slingshot right back up to your original height.

That's why a lopsided orbit is stable. It wouldn't be if down was easier than up.