Preventing HomePod being discoverable to family devices

Hi, I bought a HomePod mini a while ago and ever since my family has been really annoyed by my HomePod being discoverable on their devices, by discoverable, I mean they have the option to play audio via airplay (though when trying to play audio it fails) or when in calls, pressing on the speaker button opens a drop-down menu including my HomePod.

things I've tried:

• created my own home in the home app and added my HomePod to it, making sure I’m the only user with that home.
• change the home settings to “Only People Sharing This Home”
• factory resetting the HomePod
• change my whole family’s iPhone settings to:

Tap on Settings > General > AirPlay & Handoff and toggled it all off and removed automatically transferring airplay to TVs

Would love for some help because this is an actual deal breaker for me, seems a bit weird that a expensive smart home speaker is missing basic functionality like this.

thanks for the help!

Roe