Control Center Apple TV remote on iPhone

Why does the control center Apple TV remote keep having issues? Every time I open it, the controls are very spotty and often pressing the main control just flickers the Apple TV back and forth vs displaying expected behavior.

I can do the reset option, but sometimes I have to do it several times a day.

I’ve checked that both the Apple TV and the iPhone are on the same network.