Hey there Ststevenson007,
Thanks for reaching out to the Apple Support Communities.
Since you're able to temporarily resolve the issue by restarting the remote, have you checked to ensure that the Apple TV is running the latest tvOS version?
If it's not, since updating can often bring the latest features, security updates, and bug fixes, we would recommend updating the tvOS version and test the issue further. For help with updating the software on the Apple TV, check out this link here: Update your Apple TV - Apple Support
Update your Apple TV manually
1. Go to Settings > System > Software Updates and select Update Software.
2. If there's an update, select Download and Install.
3. Wait for your Apple TV to download the update. Keep your Apple TV connected and plugged in to power until the update is complete.
After the update downloads, your Apple TV will restart, prepare the update, then install it. When the update is complete, your Apple TV will automatically restart again.
Keep us updated on the results, and we'll continue from there.
Take care.