Mac Studio very slow to access drives

I purchased an M1 Mac Studio at the end of December. I have a few hard drives attached including a 16Tb Thunderbolt RAID, a 4Tb TimeMachine backup using USB-A and a 3Tb portable drive using USB-A. The problem is that EVERY time I access anything on the drives it takes upwards of 20-30 seconds for what looks like Spotlight creating the directory list. This is incredibly frustrating.

At first I thought it just needed time for Spotlight to catalog everything and then it would be fast but it has never got faster. On some folders with lots of files I can go and make a cup of coffee before it finishes and actually displays the files.

While the computer itself is fast, the overall process of using it is way slower than either my 2020 iMac or 2022 MacBook Pro, both of which are also using Ventura 13.4.1.

It doesn't seem to be the operating system itself, since both of those non-Apple Silicon machines instantly show folder contents from the same drives that are connected to the Mac Studio, accessing them over the network. I'm completely lost as to what is going on but it negates all the benefits of using an M1 chip, since my overall productivity is actually way slower now than it is with my older Macs. It doesn't matter if Premiere and Photoshop are blazing fast if every time I need to open a file it takes a minute or two or five just to display the folder contents. I'm losing thousands of dollars in wasted time because of these delays.

Since I transferred my system files I have to assume that this has something to do with the M1 chip but I have no idea what is causing it or how I might be able to resolve it. Should I wipe it and restart from scratch? It would mean two days of downtime to re-install 150+ apps so I don't want to do it if there's no point though. Is there a simple setting that I'm missing somewhere?