Yes, there is always an issue upsampling since you have to make up new values. For downsampling by an integral factor, you have honest samples at those times, but any other rate will also need new samples to be constructed. So for these, in either direction, there will be the need to make things up.

No matter the process, whether naive fourier transform based, or any possible algorithm that is smarter and tries more things, it cannot make up the true values in all cases, since that data is simply not in the signal.

Ancient methods treated sound as a sum of band limited pure sinewaves, which it is not, and you can claim you reconstructed up to that model, but it's pretty bad, and fails even theoretically for the reasons I posted elsewhere.

So you then start doing all sorts of signal processing, but none can actually reproduce all sounds, and most (all?) methods make tradeoffs - what error do you accept for what gains?

And if you optimize a system for one rate, and don't do it to cover all other sample rates you want to handle well, then you will make some rates worse and some better.

Well designed signal processing has to be careful to handle all the ranges it cares about.

And no one who knows or has worked in these areas would simply drop in a new system that is untested for major uses for such a widespread tool as ffmpeg. Precisely because they understand the nuances in these things.

Opus also does everything in “48KHz”. But apparently doing some upsampling will implode the whole signal… more so than the compression itself… which indeed is actually processing and storing data in the frequency domain (MDCT).