That is certainly too long to put a picture into an email! But what do you mean by "it has to load the entire library from scratch?" Where is it being loaded from? Most people have their Photos Library in their Pictures folder, and image manipulations are nearly instantaneous. With a 1 TB drive and 20GB Library, it seems that you would have room to keep your's there. If so, something weird is going on. On the other hand, there are other systems that you might mean by "load the entire Library."
Lots of people use Apple's iCloud.com to sync their Photos Libraries with their iPhones, Macs, and iPads. This is mostly transparent and fast. To save storage space, Photos can be asked to keep only smaller copies of the pictures on the internal drive, and ti gets full sized images from iCloud as needed taking, usually, a second or two. Is this what you're seeing? If you're syncing with iCloud you have iCloud turned on in your Photos Preferences. If this is the cause of the delays, you can still use iCloud, but use Photos>Preferences>iCloud to tell Photos to "Download Originals to this Mac." Then there will be no need to load the library.
Photos Libraries should never be located on "cloud drives" like One Drive and such. All kinds of things go wrong.
Some people keep their Photos Library on an external drive. Photos is quite picky about the external drive-- it must be formatted in either APFS format or Mac OS Extended (Journaled) format. And the drive can not have been used for Time Machine since it was formatted. Many drives, including all Flash drives, must be re-formatted before they can be successfully used for Photos. And flash drives are usually very slow.
If this is helpful, let us know. And if you need more information, tell us about your preferences especially the following:
In Photos>Preferences>General,
Is "Use as System Photo Library" grayed out (indicating it is) or solid (indicating this isn't your system library)?
Is "Copy Item to the Photos Library" checked? (This is the default, indicating a "Managed Library")
In Photos>Preferences>iCloud
Is "iCloud Photos" checked?
If it is, is "Optimize Mac Storage" check?