How big is your library?
How many videos do you have?
Where is the library located? If on an EHD how is it formatted and connected to your Mac?
Do you have iCloud Photos enabled?
Remember Photos is a free app and they certainly don't haver access to the NSA's for CIA's artificial intelligent algorithms for face/object indemnification.
Personally I've not had problems with it on a 35,000+ library. I don't run it all the time as it's not my primary, System Photo Library, but it does get the faces identified when I leave it open for an afternoon.
If you don't have iCloud Library enabled try opening the library with the Option+Command keys held down so as to repair the library. See if that helps.
Next, boot into Safe Mode according to How to use safe mode on your Mac and test to see if the problem persists. Reboot normally and test again.
NOTE 1: Safe Mode boot can take up to 3 - 5 minutes as it's doing the following;
• Verifies your startup disk and attempts to repair directory issues, if needed
• Loads only required kernel extensions (prevents 3rd party kernel/extensions from loading)
• Prevents Startup Items and Login Items from opening automatically
• Disables user-installed fonts
• Deletes font caches, kernel cache, and other system cache files
NOTE 2: if you have a wireless keyboard with rechargeable batteries connect it with its charging cable before booting into Safe Mode. This makes it act as a wired keyboard as will insure a successful boot into Safe Mode.
