macOS Accessories setting greyed out by SmartCard

Hello,

I have an issue with one of my Privacy and Security settings. I'm running the latest MacOS Tahoe, but that issue persist even before that.

Under the settings for "Accessories", the toggle is set to Always allow and it is greyed out due to "this setting is disabled by SmartCard".

I tried everything that I found on the internet to disable any smartcard restrictions (disabling via Terminal and etc) and I also do not have and MDM configs or profiles installed, nevertheless this issue still persist...

Any recommendations how to remove those smartcard restrictions and change my accessories setting to a more private and secured one?

Thanks for your help!