How can I disable auto routing calls to external bluetooth speaker?

Here is a design problem I've been facing for years now. I use a BlueTooth speaker for playing music. However I never ever want to listen to incoming/outgoing calls over that speaker. It's music. And that's it.

However when connected with an external BlueTooth speaker, the iphone *always* connects to it first, when a call comes in or when I try to make a call.

So many times I'm just talking and hearing nothing, only to faintly hear someone speaking in the distance... because the call was auto routed to the speaker and that speaker is in a completely different room!

And here comes the fun part. Even after you discovered this error on time and when you explicitly tell the iphone to route the conversation to your phone, not the speaker, try walking out of the speaker's BlueTooth range and then back into it. And voila, iphone detects speaker and again your call is routed to it. While in the midst of a conversation, the other person suddenly is audible in the other room again.

What is the design reasoning behind this? It's maddening to be honest, especially when you are in a sensitive phone call and want no one to listen into the call.

I've looked in the settings everywhere and there seems to be no option at all to always route incoming and outgoing calls to your phone. If I want it to be connecting to an external speaker, I will explicitly tell it to. Not the other way around.

If anyone knows a solution, please let me know!