Stop Apple Music from opening when pressing the stem buttons on AirPods Pro

Has anyone found a way to stop Apple Music from opening whenever you accidentally press the stem buttons on the AirPods? When you press on the left or right stem of the AirPods the Music app opens up. This also happens when you first put in the AirPods and accidentally press the stem while you are fitting the pods in your ears.

There are a few online discussions about this issue but no answers have been given so far. If I open the AirPods Pro settings, under the actions for "Press and hold" there is no option for "Music". The only options are "Noise Control" or "Siri". So where is the setting that says Apple Music should open when you press any of the stem buttons?

There is no such setting in the Music app settings either.

EDIT: This is on macOS Ventura. I posted this under iPhone for some reason, sorry. Perhaps someone can move the discussion under a better topic.