2 Apple TVs in the same room

I have 2 Apple TVs in the same room. I have 2 separate remotes dedicated for each Apple TV but the problem is both remotes respond to both Apple TVs at the same time. What can I do to have 1 remote for 1 Apple TV