Magic mouse switching between 2 macs doesn't work

Hi,

I've two Macbooks an M1 Max and an M1 Pro, I've paired my magic mouse 2 with both.

I can connect to either of the Macs and continue using next time when mouse connects automatically.

As soon as I switch from one Mac to other problem starts arising. Magic mouse doesn't connect automatically and even after connecting it shows blue (connected) under bluetooth devices but doesn't actually connect (and doesn't show the mouse image that shows up everytime on successful connect). I'm required to unpair the mouse and pair it back which is really annoying specially when the Macs are used on a monitor and always closed.

Same process has to be repeated with an unpair/pair back cycle whenever moving between these Macs.

It's really hard to accept considering this is a native ecosystem with latest available Macs and magic mouse. Weirdly Logitech's MX Master performs, same goes with it's keyboard.

Any solutions?