I don't have an M4 Mac but my dual boot M1 MacBookAir comes up in a few seconds when the internal drive is set as the "default" boot drive and maybe 10-15 seconds when the external drive is the "default" boot drive. Also, with more drives attached, even non-system, it will take a bit longer as the boot loader in recovery waits to make sure all mounted drives have been checked for an OS.
What really hasn't been explained very clearly is that there are two recovery partitions on dual boot systems, one on each boot drive when external drives are used, one in each container if separate partitions are used on an internal, and one in each volume group if a single container/partition is used. So, if one boot system is say Sonoma and the other is Sequoia, if your default boot system is Sonoma, when booting into recovery, you will boot to the Sonoma recovery system and visa versa.
The default boot system is determined when you select the boot system in System Settings->Startup Disk. If you boot into recovery and select the boot system there, the default boot is not changed unless you enter recovery itself and select the startup disk from the Startup Disk menu.