iPhone keeps automatically playing music to Apple TV.

I just hooked up my Apple TV yesterday and it works great. But this morning I was watching something on it and using the remote that came with it and it all worked well. But then I tested playing music on my iPhone and it overtook the thing being played on my TV and when AirPlay comes up on my phone it doesn't even give me an option to play the music from my iPhone. Sometimes it does but it doesn't even let me click on that option. Automatically AirPlay to TVs is on Never under AirPlay & Handoff. Handoff is off as well. Before when I had just a HomePod speaker (which is now being used with the AppleTV) it would always automatically play on my phone and then I would have to make it play to the speaker. Now I cannot get any music to play at all on my phone. Even if I turned the AppleTV off it just plays through the HomePod without me selecting it.