iOS 18.4.1 Hey siri is not working, not even if I push side button

After some time my iPhone 16 Pro is running, Siri gets stuck and does not react neither to "Hey Siri" nor the side button. I have to restart the iPhone and then it's working again.

I tried to turn off Hey Siri and turn it on again in Settings, but it didn't help. I have Apple Intelligence turned on since iOS 18.4. Before that I never had an issue like this.