AirPods Pro 2 can connect to iPhone but not Macbook

To start, every device is running the latest software as of 7.September 2023.

I connected my AirPods to my iPhone and it works flawlessly. When I try to connect the AirPods to my Macbook, it manages to connect when I manually , but when I try to change sound output is just shows the loading icon. After about 4-5 seconds, it stops loading and the AirPods disconnect itself.

What's really strange, is that if I choose "forget device" on the AirPods, and first connect them to my Macbook. They will then work perfect on the Mac, but they won't connect to my iPhone. I have tried every troubleshooting tips but nothing works. Did I get some random faulty AirPods Pro?