I think the HDD part of your Fusion Drive has problems. The bottom line is that HDD issues are the most common problem with Macs. Over the years I had 2 iMacs and 1 Mini with failing Fusion Drives/HDDs. The first symptom of the failure tends to be abnormal time to boot (1-10 minutes or more) or random crashes, before they might become completely unbootable, unreadable or unmountable.
You can try out DriveDX for a thorough diag and decide on the next steps.
It is desirable to separate the SSD from the Fusion Drive, and reinstall the OS on the SSD part (You seem to be a light user judging from the disk usage). Or reinstall and boot from an external SSD that connects to TB3/USB-C of you iMac, if you need a larger OS disk.
Make sure to make full backups using Time Machine and/or CCC to an external disk, that will be helpful for migrating to a new OS SSD later.
Also, make a USB MacOS installer available (and test it to work) with your current system, with Terminal commands.
When reinstall the OS, separate the Fusion drive into 2 disks (SSD + HDD) in Terminal. This will minimize the performance impact from HDD. And use Disk Utility to Erase and set up the internal SSD for OS install.
Finally, you can still use the HDD. First erase it as a new HFS+ (not APFS as this is not HDD friendly) disk, and you can hold non-essential data, such installers, videos, music, books, that can be redownloaded from the Internet or other providers. You can also use it as an extra Time Machine disk for monthly or quarterly backups, and iPhone backups as a cold storage in the iMac.
One more thing, consider increasing the RAM to at least 24G, and reduce the RAM usage by disabling unnecessary apps and utilities for auto startup (Settings-Users-Startup Programs). A larger RAM reduce unnecessary swap drive writes, and thereby speedy the system and increase the lifespan of the SSD.