Time Machine backup to remote Mac (over fast network connection with VPN tunnel) not working

I have a fast network connection (gigabit up and down), and over that I'm using a VPN to my workplace.

There is a Mac server there, and I can access the volumes there with no problems. Also, Screen Sharing works, ssh works, everything works. And it's all fast.

But what doesn't work is Time Machine backups. The destination drive is APFS, and exported with the "Use for Time Machine" option enabled.

After mounting the drive on my local Mac I can use it as a backup destination, and initially things appear to start working, and if I login to the server I can see the temporary folder appearing and starting to be filled.

But after a couple of hours Time Machine gives up, with the error "The backup disk image could not be created.".

Any suggestions on what could be going wrong, and/or how I can debug this? I have full admin access both to my local Mac and to the server...