Does having a HomePod override all other "hey Siri" requests? Is there a way to get HomePod to only respond to my voice?

Love the HomePod. But HATE that it responds to my wife's voice when she says "Hey Siri" into her phone. Or if a friend is over, same thing. Anyway to make it recognize, or respond to only my voice? It seems that "Hey Siri" on HomePod overrides everyone else's "hey Siri." If 5 people with iPhones are in the are and sey "hey Siri" it won't respond on their phone. Even if their wifi is off and they aren't signed into "home?" Am I right about that?