Recommendation for Mac-friendly server hardware with reliable external access

This sounds more like a network issue to me than a storage technology. The fact that SMB (over VPN) and AFP both exhibit poor performance remotely indicates that your office network connection may be the issue. Do you know what it's rated for?

More to the point, if it is network related, replacing the Synology may not make any difference, since any alternative storage solution is likely to suffer from the same issues.

That's not to say you shouldn't upgrade the Synology if you need to (e.g. for storage space reasons), just be aware that it's not automatically going to improve the network performance.

From your post it sounds like AFP performance in the office is OK, but tanks when accessed remotely. That also points to a network issue rather than protocol (AFP vs. SMB) or the Synology itself.

I'm also somewhat concerned by the fact you seem to be able to access the Synology from outside the network without a VPN connection. That sounds like a problem to me.

If you're not familiar with network troubleshooting, start by accessing https://speedtest.net/ from the office to see what you get. Also time uploading and downloading a reasonably-sized file both in the office and remotely.

If you can, try from multiple remote locations, to make sure it's not, say, your home internet connection that's getting in the way.