Airplay streaming output

I frequently stream from my iPhone using Airplay to a HomePod Mini here on my desk. From time to time, I wish to switch the output from the Desk HomePod to my Kitchen HomePod. However, when I tap on the other device, it doesn't switch to that device, it adds the device and continues streaming through both. In the process, it creates a two-device virtual speaker that can be tricky to undo.

My workaround is to open the Airplay options, switch to output on the iPhone itself, and then tap on the second HomePod. Is there a way to just switch the output in one tap instead of having to do this workaround?