Handoff keeps launching software I'm not using and interfering with Universal Control

I have two macs, an iMac Pro and a Mac Studio. I use the Studio for video editing, and the iMac for everything else. I have them connected using Universal Control so I can use one mouse and keyboard for both.

What's frustrating is that Handoff apparently keeps launching a browser helper that is interfering with using the mouse, as it takes focus when it launches. It was happening with my main browser, Vivaldi, which I have installed on both computers, so I removed it from the Studio, and now it's doing it with Safari, which isn't running on either computer?! The mouse and keyboard are connected to the iMac, so I assume that's why this is happening on the Studio, but how do I get it to stop? Or at least not take focus, because it's frustrating to be typing, lose focus, and have to refocus on Premiere.

As far as I can tell, I need Handoff to make Universal Control work and there are no granular settings. Do I have to uninstall everything else to prevent this? Or just all the browsers that are on both systems?

What do I do?!