From what I understand the 3G/4G uses a different mechanism for authenticating the mobile and the tower (AKA). As such, having the encryption key, will not make it possible for a bad actor to connect to the tower and listen on the conversations, because the data encryption key is different from the shared key used for mutual authentication.

Now, while this is true, I believe that a bad actor can still listen to the radio transmissions passively and decrypt those. But that is a lot harder than just plugging in to the tower and listening.