XPath 1.0 is a pain to write queries for. XPath 2.0 adds features that make it easier to write queries. XPath 3.1 adds support for maps, arrays, and JSON.
And the default Python XPath support is severely limited, not even a full 1.0 implementation. You can't use the Python XPath support to do things like `element[contains(@attribute, 'value')]` so you need to include an external library to implement XPath.
The problem of XPath 3.1 is that it is very complex [1] - this is a long page. Compared to the 1.0 spec, it is just too complex in my view. For the open source project I'm working on, I never felt that the "old" XPath version is painful.
I think simplicity is better. That's why Json is used nowadays, and XML is not.