How to unlock the user guest mode without a password?

Try the Administrator password. It certainly looks like this screen was preceded by some operation requiring a password.

Change Guest User settings on Mac - Apple Support

It looks like your Mac is using a firmware password: Set a firmware password on your Mac - Apple Support.

The same article mentions:

If you can't remember your firmware password, schedule an in-person service appointment with an Apple Store or Apple Authorized Service Provider. Bring your Mac and its original receipt or invoice as proof of purchase.