There have been incredibly clever attacks based on tricking intervening routers into routing the traffic to the target gateway, but more prosaicly my next hop ISP is itself a threat I worry about.