How do I use HomeKit and Philips Hue with multiple wifi access points (2 routers)?

I have two Wi-Fi access points (1 main router - with internet and 1 second router). My Philips Hue Bridge is plugged into one of the access points (into second router). When my HomePod mini is connected via Wi-Fi to the same access point (the second router) as the Hue Bridge, HomeKit controls the lights as expected. However, if I move the HomePod mini to a different part of the house and my HomePod switches to the second wifi access point (the main router), even if is the same network, HomeKit (HomePod mini) is unable to communicate with the Hue Bridge.

I reset the Hue Bridge, HomePod mini, HomeKit and both routers and recreated the linkage but it still won't work as described above.

Note: Same network with one DHCP server. Same SSID for all Wi-Fi (2.4 GHz Main Router, 2.4 and 5.0 GHz Second Router)

I appreciate any help, suggestions, etc.