The operation couldn't be completed. Operation not permitted. Safari Downloads

I have a 2019 iMac running a dual HD setup with an SSD as the primary drive with OS installed and a HDD as the secondary. It is not a hybrid drive setup. Running Mac OX 12.6.

I have my default downloads folder for Safari set to a Downloads folder on the HDD. It works most of the time, but every once in a while I'll get an error: The operation couldn't be completed. Operation not permitted. From there on nothing will download from Safari to the folder until I go in and change the default download folder to somewhere else. Doesn't matter where. Then downloads will work again. From there I can switch it back the the same folder where I was getting the error before and downloads continue to work again. No problems downloading for a couple days until the error occurs again then I have to repeat the process over again.

It's just annoying. Trying to minimize writing to the SSD for silly downloads. Seems like there might be a permission issue? Anyone know how to permanently resolve this?

Thanks,

Nick