Another issue can be what sort of devices you have connected to
you're MacMini. Sometimes they can cause interference, especially
at boot time with Bluetooth. Also, operating via WiFi can also rarely
be an issue. Also, do you have any other Bluetooth devices assigned
to your Mac?
I don't have the keypad version, but do have the new Magic Keyboard
(purchased recently) and is working without any issues as you describe
with my iMac.
If possible, use the Lightning charging cable and connect it directly
to your Mini and see if the issue persists.
Also, one other thing, how far away is the keyboard from the Mini
and is there any sort of obstructions between?