It could be the case where buffer management on routers along the way makes a difference, transient congestion conditions, buffer sizes, fq vs fifo vs something more complicated, etc.

Things work a lot faster if all of your small packets make it through, but if any of those packets are dropped you incur delays for retransmits, tcp slow start, etc. Basically extra round-trip delays that could be a lot longer than a local nagle delay, depending on what you are sending.