Virtual Display lag in Vision Pro

I'm trying use the virtual display with my mac mini m4 but everything from lag to video play back to mouse movement is laggy.

I have seen messing with the resolution is a fix for some but it is not affecting the lag at all. I am using the logitec mx series mouse and keyboard. But as I said even video play back and stuff is laggy so it's not even just relegated to the peripherals.

I switched to my MacBook Pro Intel 2019 and the lag from the mouse is way less but the lag from video play back is still there.

I'm about a foot away from my router.