there are tons of articles here in HN that have shown that SIM swamp (at least in US) is much easier then trying to brute force (or using quantum computer) to break TOTP encryption. One of main reason for SMS is also meta data collection of your number.