I have the same problem with my iPhone (15 Pro) the remote app works intermittently
That is not the same as OP’s issue with tapping vs. swiping.
Approx 3M from the Apple TV
As the software remote uses the home network, the distance to the router is relevant (and any obstructions in between), not the distance between iPhone and Apple TV box.
Restarting the iPhone has helped
Is that a permanent fix, or does this have to be repeated?
My wife has the same phone and her remote app is usually more reliable.
Possibly you all have the regular seating per person, as many households do, where localized interference may play a role? Interference may be in the room or house, but it could also be from a neighboring network at the same Wi-Fi frequency or close to it. As little a few feet could make a difference, depending on source of interference. Does the issue switch when you change seating places? Does the issue switch when you use each other’s phone (for the remote)?