Siri is too loud at night, even after changing voice volume.

Siri lowers its volume when I ask “Hey Siri, will you please speak softer?” But if I ask another request after lowering the volume of its voice, Siri reverts back to the high volume that I wanted to avoid.

The music I’m playing at night is its lowest volume, but Siri is just too loud.

The problem doesn’t appear to be the type of request I make since what I say does get Siri to respond quietly, but only to that request. How can I keep it quiet during the night, and not just momentarily?