Remove HomePod as speaker option when making a phone call

We have 2 HomePods installed in our home. For some bizarre reason my father, who is has "joined" our home, now has one of them appearing as an option of a sound source instead of just the speaker phone. No one else has this on their phone just him. And I cannot for the life of me find how to disable it. Can anyone help?