Mac Mini M4 and USB-A keyboard problems

Every time I restart my Mac Mini M4, the USB-A Apple keyboard won't connect. I'm using a USB-A to USB-C adapter. I have to diddle with trying different ports repeatedly before it finally connects. I've tried several different adapters with no change. Once I get it to working, it doesn't seem to matter which port I plug into.

I'm afraid that one of these days it won't connect at all, and then I'll be screwed.