Universal control constantly disconnects every 2-5 minutes

Hello, I've recently bought Mac mini M4 and wanted to connect with my work MacBook M2Pro via universal control.

It is working, but I get constant disconnects from universal control, and of after disconnect my keyboard and trackback on Mac mini doesn't work for about one minute.

Devices are like 5 cm away connected to the same WiFi ( Mac mini additionally connected via ethernet cable but to the same network)

Is there any fix for that ? it's very annoying