Apple Watch and AirPods

Scenario: doing my morning power walk wearing my Apple Watch series 4, my 2nd gen AirPods and my iPhone 10 is in my backpack. All are using the latest iOS/watchOS. I’m using the exercise app and it automatically plays my ‘Walking Mix’ when I start my walk. The playlist is loaded on my watch. So far, so good.

The problem begins if/when I remove either one of the AirPods to talk to someone. When I remove the AirPod, the music stops (as it should), however, when I reinsert the AirPod, the music does not and will not restart. The watch shows that it’s still connected to the AirPods. Even if I manually touch play on the playlist, it will not resume on my AirPods. Sometimes it hands the music off to my iPhone (as it should not). The only way I can get the situation resolved is to reboot the watch..which I don’t want to do during a workout.

Any ideas?