Photos not removing person from Photo

So photos asked me to review some images if they were the same as the person I was on. Missed that one person was not the same. Instead of giving a merge warning it combined two different people into one group, mixing thousands of photos. Now I'm trying to go through and fix it and have run into a snag.

I selected a group of photos under the first person and chose "[name] is not in this picture" from the menu. It removed the person from many of the photos and removed those photos from the group. In other photos it removed the persons name, but left the photo with unknown person in the group. It didn't unlink the wrong person from the photo.

So even though there's 4 people in the photo, it shows 5 people linked to it even though 1 person is labeled in the actual photo. It's not letting me remove/unlink that one person.

I tried switching the photos to 'faces' to see what it was highlighting, but it highlights a random part of the photo with no face in it. And there's no face circle for me to remove in these areas either.

Any ideas on how to fix this so I can remove the one person from the group?