iCloud photos on iPhones

My husband and I accidentally clicked on the 'iCloud Photos' switch so all of our photos are merged onto each others phones. I am backing up our phones to the cloud regularly and have our pics downloaded on my Mac. When I try to turn off iCloud photos, it won't let me 'download photos & videos' (which I already did). But when I choose 'remove from iPhone', I get a message that 6000+ pics will be removed from the phone. How do I get our own photos back to our devices and stop the pics from going on each other's phone?