Latency with Logic Pro X on NEW MAC MINI - never had an issue with my super old and slow iMac

I have been recording music on my iMac for years, it was a super slow computer but once I got Logic running it would work well. I never had any issues with latency. Finally I got over the slow start up of my iMac and bought a Mac Mini and I have bad latency when recording, regardless of what I do with the settings.

What can I do to fix this as it is devastating!?

It is the Apple M4 chip, 16GB memory, running Sequoia 15.2. I am using a Scalett audio interface.

Please help

[Re-Titled by Moderator]