HomePod Mini "Control Other Speakers & TVs" with Apple Music, loading forever

I've had my HomePod for a couple years, and one of the main reasons I got it was to offload music streaming + airplay from draining my phone battery using the "Control Other Speakers & TVs" feature of iOS

The feature has never worked properly though. When I switch to control the device and then select a song in Apple Music, Now Playing just displays "Loading..." and it's a coin toss whether it stays like that forever or starts playing after a couple minutes of waiting.

This doesn't seem to be a particularly common problem on the internet, though that seems to be because most people mistake AirPlaying to HomePod for "Controlling" it, when they're two separate features

Has anyone experienced this issue and has a fix?