Trying to remove a user from Home gets me “Could not modify user’s access” error

I’m trying to remove a Family Member from my Home App (iOS 16.2 on iPhone 14 pro max). Once I tap remove person and confirm, I get error “Could not modify user’s access”. It does not remove the person.

When I go to my Family Members iPad (iPad 9th gen on iPadOS 16.2) and open the Home App, it sees an empty default home and not my home I’ve set up in HomeKit.