macOS Sequoia LAN is totally unstable!

I've been having disconnection issues on SSH, VNC and SMB, since I installed macOS Sequoia, to the point that I couldn't keep an SSH connection up for over 5 minutes! In the beginning I thought it was just SSH, so I was trying different settings, different MTU which made it a bit more stable, I even downloaded a version of the SSH client that uses OpenSSL instead of LibreSSL that the macOS is using, but I was getting disconnections with corrupted packets of a huge random 32bit number invalid length. I was using a Realtek ethernet adapter, so I tried eleven different ones without success. I even dug out my old Apple Thunderbolt 2 to ethernet adapter, combined with a Thunderbolt 2 to 3 adapter, which showed a better behaviour on SSH. Using WireShark to capture the connections when they were failing didn't help, since the invalid packet length number wasn't anywhere to be found in the packet itself, as it was most likely on the SSH application layer, and thus encrypted.

However when I was trying to do a tar backup of 4TB SSD on my Synology NAS, the NAS kept disconnecting anytime from 10min to 120min randomly, completely destroying my backup and having to start it over. VNC connections on my Raspberry Pi weren't any different than the SSH.

I thought it was planned obsolescence for my 2019 Intel Core i9 MacBook Pro, but I've heard from others having M2 MacBooks having similar issues.

Is there any suggested solution?