Can't AirPlay music to HomePods linked to Apple TV anymore

Hello,

I have a HomePod and HomePod 2. Each are linked to an Apple TV 4K (1st Gen) in different rooms as a default audio source.

I don't always watch TV, so I like to AirPlay ambient music to both HomePods using either my iPhone 13 Pro Max or iPad Pro 12.9" M1. I use either the Apple Music app, or the ambient music options in the control center.

All devices are fully updated to the latest OS and verified to be connected to the same WiFi network, as of the date of this post.

Lately, I've been having issues AirPlaying music to either HomePod when they are still linked to an Apple TV. When I attempt to cast, I get a vague error message: "AirPlay: Unable to connect to (HomePod)" with an "OK" button.

A workaround (for now) is to manually unlink them from the Apple TV, but then it requires extra steps to put them back as an audio source whenever I decide to watch TV, which is annoying. This was not the case before.

Unplugging the Apple TVs from power still leaves the HomePod in a state where it "thinks" it's still linked to an Apple TV, as it shares the name with that Apple TV in the AirPlay source list, and I still cannot AirPlay music to them with the same error.

Is there any way to fix this?