Apple TV audio scramble issue with two sets of airpods

I have an issue when connecting two sets of AirPods to an Apple TV 4K.

It works for about 2 minutes then the Audio starts sounding scrambled on both sets of AirPods until one set is disconnected and reconnected.

This used to work OK. All firmware on AirPods and tv is up to date.