iOS forces LIFX bulb setup with homekit

I have created a separate WLAN network in my home for IoT devices. I am currently re-setting up all my IoT devices. The normal sequence of events is "open the app, decide what device you're setting up, switch to WIFI in your iPhone and select the device's own WIFI network and connect to it, switch back to the app, insert the new network/password manually, finish setup".

However, these bulbs are designed with HomeKit in mind, and for added security, iOS is not letting me connect to the bulb's WIFI - instead it is forcing me to "setup with HomeKit" and there is no way for me to switch out of the HomeKit setup -- the Homekit app seems to sense the switch to another app and forces a disconnect. So I can setup these bulbs the way they were meant to be setup.

How do I opt-out of the HomeKit "forced" setup?

I already checked if I could "Connect to other network", but the bulb's network seems to be reserved.

I don't have another "older" iDevice I could use that doesn't force me into this.