Homepod mini 3-sec delay with Mac mini m4

I was having a 3-second delay when using Homepod. It was resolved when I set "Allow Speaker & TV Access" -> "Everyone" in the settings. Now Homepod works perfectly with iPhone 15, but with Mac mini, there is still a 3-second delay. Is there a way to solve this?