Apple Music on Homepod

I have two 1st gen Homepods paired as stereo. I have an AppleTV 4K (3rd Gen) that is connected to eARC on my TV. In the Settings on Apple TV I default audio set to the homepod speakers, and Audio Return Channel set to Play Television Audio. It all works great, except when I turn off the TV, and then ask Siri to play any music on the homepods, Siri says "OK, (music) now playing..." and nothing plays. Then after about 1 min. Siri says "Sorry, there's a problem with Apple Music." The only way I can get it to play music is to go back into the AppleTV and change the default audio back to TV speakers, which I don't want to do all the time. Any ideas anyone?