Stop iCloud Drive downloading to Mac

Hey,

How can I stop iCloud Drive downloading the same files to my Mac?

I have 3 videos of speeches from my cousin's wedding. They're all around 15GB each. I stored them in iCloud Drive to save space on my MacBook Pro and iPhone.

On my Mac, I keep right clicking on them and pressing "Remove Download" - as soon as I do this, it starts downloading each time!

I press the i icon next to iCloud Drive on Finder - it shows the Downloads. I click the small X and it will not stop the download.

I'm getting seriously fed up of it - it downloads all 3 videos, I then have around 5GB left on my MacBook, I then click "Remove Download" - it keeps them removed for a few seconds, and I'm back up to around 50GB storage. Then it starts downloading them AGAIN!! The fan on my MacBook goes crazy each time, and it slows the internet up and every other app. It's making the Mac unusable. It's either slow due to downloading, or slow due to no storage space.

What can I do to stop this?