“Hey Siri” through AirPods 3 responds through iPhone speaker

Hello!

I bought a set of AirPod 3 and after some difficulty with getting them to respond to “Hey Siri”, I’ve got it working. However, Siri is responding through my iPhone (13 Pro Max) speaker which is not ideal for when I’m out in public places such as the gym.

I was just wondering if this is normal or if it’s potentially a problem with the AirPods (which is annoying since they’re new). If it’s the latter, is there a way to fix this?

TIA