Unable to select iCloud checkbox in photo's settings

A Mac can have several Photos Libraries, but only the System Library can connect to iCloud. So I'm thinking that you have another Library somewhere that is designated the System Library. Is that Library connected to iCloud? If you go to iCloud.com and check Photos, what pictures are there?

If iCloud.com Photos is empty, then all you have to do is to designate this Library as the System Library in Photos' Settings>General

Here you can see that "Use as System Photo Library" is grayed out. That's because this Library is already designated as the System Library.

Once you have the System Library set, you should be able to turn on iCloud Photos. If you choose Optimize Storage, then you won't be able to make local backups of your pictures. On my Mac, I don't use Optimize.

If there are pictures already on Photos at iCloud.com, then pictures from this Library will be added to those, and the iCloud.com pictures will be added to this Library. Make sure you have enough room on the Mac and at iCloud for all those pictures. Photos will try not to include duplicates when combining pictures.

Let us know how this goes…