Mounting an external Time Machine disk takes more than three minutes and locks up apps

It’s how Time Machine and APFS work on big spinning disks.

Time Machine’s APFS format is highly fragmented by design, and every time you mount the drive macOS has to walk through a huge chain of metadata to verify integrity. That’s fast on SSDs, painfully slow on 16TB HDDs. Three minutes isn’t unusual with that capacity and fragmentation level. There’s no magic fix, but using smaller Time Machine volumes, moving to SSDs, or splitting backups across multiple drives will cut that lag.

Suggestion:

You have a quantity of three 16TB HDDs. IMHO, time to consider a NAS - Network Array Storage.

The easiest option would be to just buy a QNAP or Synology NAS with enough room to grow. So a 4-5 bay NAS is perfect. All commercial NAS systems support emulating a Time Capsule which broadcasts it's presence to the subnet via Bonjour/Zeroconf/mDNS while using SMB / CIFS protocol (Windows sharing) and APFS based Time Machine based disk image. If you use a NAS with ZFS or BTFS you gain extra protection. The internals of the sparse disk image layers files onto the underlying ZFS / BTFS filesystem which is near bullet-proof, preventing data corruption. Ever since I put my Time Capsule on ZFS I have far fewer issues.

There's also 45 Drives Homelab systems. A 4-Bay / 8-Bay / 15-Bay models. These are typically a Supermicro motherboard with an Intel Xeon or AMD Ryzen or EPYC CPU and comes with software for RAID & Containers, etc. You can also just install TrueNAS Scale on the same hardwre. Or Ubuntu Server and LXD if that is what you want. You could also install Proxmox. Or setup a Kubernetes server.

You connect the NAS to Ethernet and it becomes a storage server for all your devices. If you wish to reach these resources from outside your LAN - Local Area Network such as when away from home. Consider Tailscale, it's a modern mesh zero-trust VPN solution. The idea being an extra network layer on top of your base network. Servers, devices running Tailscale client can join your Tailnet. Tailscale offers ACL - Access Control Lists so you can define what machines and systems can communicate with each other. It even works with cloud services. You install Tailscale in your cloud system and tunnel a secure encrypted connection to it. I use Tailscale so I don't need to pay to access systems via cloud. i.e. IoT's, home automation, etc. Tailscale is also handy to interconnect multiple remote office locations. Or from your home to say your parents home and your brothers home and everyone can connect and share that NAS and whatever you've got running on it. LIke a Jellyfin server, etc.

RAID / NAS is NOT BACKUP! Yes it's redundant and yes it will save your bacon if an HDD bites the dust. But it is still very possible to lose an entire disk array. It's not bullet-proof. You still need to either replicate the data to another NAS and perhaps one offsite as a form of backup. Or perhaps backing up to Backblaze, etc. For me, I only backup truly important data. Losing everything would be awful but not the end of the world if I have that critical data backed up. Use a cloud drive, backup service, a second NAS sync'd to the 1st NAS then move the 2nd NAS offsite and it will stay in sync if you've got a VPN tunnel such as Tailscale or one of its competitors. I recommend Tailscale because it's drop dead simple to stand up and get it working and they have great documentation.

johnrellis wrote:

This smells like an implementation defect of APFS or the way Time Machine uses APFS on spinning-disk drives. (Time Machine requires APFS on new drives.)

I found a handful of others who have experienced the same symptoms, with guesses but no confirmed solutions posted:

First, excellent post providing the relevant info and steps taken. Well done.

It’s not so much a defect as it is a fundamental (and intentional) incompatibility between APFS and TM. AFPS is designed for SSDs, not HDDs. The latter are not very tolerant of disk fragmentation.

TM is designed for maximizing the number of backups in the most storage-conservative manner. When a file is updated, the next backup deletes only the changed blocks from the prior version and stores only the new blocks, and the entire dataset is linked by pointers. In other words, TM gains efficiency by maximizing fragmentation.

The first thing macOS does when a TM backup is connected is verify the integrity of the fragmented-by-design backup database…fast on an SSD, slow on an HDD and enormously compounded with a very large backup.

In my opinion, this fundamental incompatibility means there is no ‘solution’ per se.

A possible workaround would be smaller TM backup disks or/and SSDs. Personally, my primary TM backup is to a 2 x 10 TB RAID1 NAS, but the TM volume for my 2 TB internal SSD is limited to 4 TB. I also use a pair of 4 TB SSDs as supplemental backups (swapping them offsite every other week).

I think just the sheer size of the TM backup drive is a contributing factor...16TB. Add on to it the things @neuroanatomist mentioned can extend the time.

Keep in mind that the fastest transfer rates I've seen for a single Hard Drive (Seagate Ironwolf NAS) is during sequential reads & writes with a transfer rate of about 200-250MB/s. Random reads & writes will be slower. If your drives don't have multiple read/write heads per platter, then the transfers may at best be half of those numbers. Of course Hard Drives speeds diminish as the data being access nears the center of the platter.

johnrellis wrote:

But my main issue is that many apps lock up with the spinning wheel during the several minutes Macos is mounting the Time Machine drives. That's what I meant by an "implementation defect" -- apps shouldn't lock up for minutes at a time during a mount operation of a Time Machine volume. 

Unfortunately that tends to occur whenever the Finder or an external device ties up the system. I've seen it occur with every OS out there, although macOS may be a bit worse in that regard. If the Finder gets stuck, then any app which is needing to access storage can become stuck until the Finder's first job is done.

You can provide Apple with product feedback here:

Feedback - macOS - Apple

Or perhaps you should consider using some other backup software which doesn't need to scan the backup destination. After all you are dealing with two huge TM backup drives. I don't think TM was designed to handle that large of a backup.