The first part of the reply is asking, "are you sure you really need to do this?" First, as you've already discovered, Apple no longer includes mail services on macOS as Server.app is long dead. Apple used Dovecot/Postfix in Server.app. But realize you must also have a fixed IP address (or use a dynamic DNS service which could have issues if updates are slow), and if you are in the US you may run into some residential ISPs blocking mail hosting ports (like SMTP port 25) to prevent residential customers from running businesses out of their homes (or being spam bots). And, as you mentioned, you should also consider power redundancy and runtime, backing up to an external location, and all the security concerns that come with self-hosting. You will need SSL certificate, you likely should implement some form of MFA if supported by your mail server, and you should be on constant watch for attacks and breaches. This is on top of the full time job managing and training your spam engine. Oh, and you will need to allow port forwards through your firewall. Again, if you are on a residential service, you may not have a configurable device that will allow the traffic to pass.
Ok, if you want a full time job to manage your own mail server, you have options. You can do what Apple did and install Dovecot and Postfix. But your setup will be command line. If you are not comfortable with that, you likely should look at Kerio Mail Server. That will at the least give you an intuitive web UI to configure everything. Disable
As for your storage needs, you can enable FileVault on the server. But, if the device reboots or is rebooted, you must be physically present to unlock the drive in order to boot. This can be a problem if you lose power and you are not home. When power is restored, the unit will power but halt at the decrypt screen. Regarding backups, you can simply encrypt your external media. But you are still putting all eggs in one basket. If the server and the backup are in the same physical location and something happens to that location (fire, flood, lightning, alien invasion, zombie apocalypse, human stupidity), then you run the risk of losing both your production system (live data) and your backup media.
On the surface, I would discourage you from pursuing this path. It is a ton of work and requires a lot of stars to align just right. However, if you are intent, look at Kerio as I feel it will be the easiest to deploy and maintain.
