How do I speed up moving 1.3 million files into smaller folders on my Mac?

Question

Level 1

4 points

How do I speed up moving 1.3 million files into smaller folders on my Mac?

I'm a photographer and have been shooting digital since 1999. I had an external HDD fail, 24TB disk with approx 16TB used so sent it to a specialist for data recovery. Data now mostly all recovered but the recovery process was not able to recover my old folder structure and only creates folders for each file type. I now have 1,356,684 jpg files in a single folder and it's taken three days continuous running just for finder to open the folder.

It's so slow and for any practical purpose completely unusable so I'm attempting to move the files to new folders, on the same disk, with approx 10,000 to 12,000 images per folder. This, in itself is a very slow process, with approx. 2,000 files taking approx 20 minutes to move. Image size ranges between 40MB and 30kb but the speed to move them is broadly similar. In view options, everything is unchecked, no other apps running, no other HDDs connected, wired network and NAS both unplugged.

I do have a second copy of all the files and another identical 24TB HDD on the way (SanDisk G-Drive, USB-C) but if I can't cross reference the second copy with the recovered copy, I don't know what, if anything, I've lost.

Does anyone have any ideas how I might be able to speed up the process?

TIA

[Re-Titled by Moderator]

Original Title: Very large folder issue

Mac Studio, macOS 15.5

Posted on Aug 8, 2025 11:33 AM

Reply

Answer 1

Niel

Community+ 2025

Level 10

739,739 points

Aug 8, 2025 11:55 AM in response to GsyPhoto

Choose Utilities from the Finder's Go menu, open the Script Editor, and run the following:

set thePath to POSIX path of (choose folder)

repeat with thisStart in {"N"} --change this so that the list contains enough starts of names to divide the contents

do shell script "mkdir '" & thePath & "/" & thisStart & "'"

try

do shell script "mv " & thePath & thisStart & "* " & thePath & thisStart

end try

end repeat

(260982)

Reply

Answer 2

GsyPhoto Author

Level 1

4 points

Aug 8, 2025 04:12 PM in response to Niel

Thank you Niel although I don't really understand your suggestion. I've never used the script editor and I don't know how to. Could you make it a bit easier to understand for a complete doofus i.e. me?

Reply

Answer 3

Niel

Community+ 2025

Level 10

739,739 points

Aug 9, 2025 07:14 AM in response to GsyPhoto

It subdivides the folder by name. For example, to separate all the files which start with A, B, or C, use:

repeat with thisStart in {"A", "B", "C"}

None of the rest of it is supposed to be changed by the runner.

(260991)

Reply

Answer 4

GsyPhoto Author

Level 1

4 points

Aug 9, 2025 10:03 AM in response to Niel

Thanks for the update. The images all use the same naming scheme e.g. jpg_1920x1080_0000001.jpg meaning it's difficult to separate them so is there a command that would move a set number of files instead? Say 5000 or 10,000 at a time?

Reply

Answer 5

Niel

Community+ 2025

Level 10

739,739 points

Aug 9, 2025 10:18 AM in response to GsyPhoto

Use:

set thePath to POSIX path of (choose folder)

set theFiles to (do shell script "ls " & thePath)

set fileCount to count paragraphs of theFiles

repeat with folderNum from 1 to (1 + (fileCount div 5000))

do shell script "mkdir " & thePath & folderNum

end repeat

repeat with thisFile in paragraphs of theFiles

do shell script "mv '" & thePath & thisFile & "' " & thePath & (1 + (fileCount div 5000))

end repeat

(260995)

Reply

Answer 6

Aug 9, 2025 10:21 AM in response to GsyPhoto

Supposing that you do get these 1,356,684 jpg files divided among 200 or more folders.

How much time is it going to take to go through all of these files by hand to figure out what is in them and to give them meaningful names? If it takes just one minute per file, you would be looking at nearly four years of 16-hour work days.

If you're talking about automated cross-referencing, you're gong to need an efficient way to do that. It will have to be something more clever than

    for (all of the 1,356,684 files on the recovery disk)
        for (all of the approximately 1,356,684 files on the undamaged disk)
            compare the two files (and use the comparison to make notes on which,
            if any, of the files on the recovery disk are ones that are not saved
            in a more organized fashion on the undamaged disk)

That type of naive comparison algorithm would have O( N^2 ) running time, and if you literally did N^2 comparisons, you'd be looking at something like 1.8 TRILLION automated file-to-file comparisons. You might be waiting a little bit for that to finish …

Reply

Answer 7

GsyPhoto Author

Level 1

4 points

Aug 10, 2025 01:24 AM in response to Servant of Cats

Yeah, would be a serious black hole of time if I had to go through all of them but I don't - there are a lot of files on there that I don't need. As well as the originals, there's thumbnails, web sized copies plus a whole load of duplicates that the recovery process seems to have created. Those can all go straight away.

Plus, I used to shoot a lot of events, for instance, I did a kids football event every year for about 10 years. It was over a weekend with around 20 - 30 teams taking part so there are probably around 100,000 images just from that. I've also been shooting a dance school for over 20 years and they do 8 shows a year so again, loads of images. However, I know I have two further backups of all of those so once the images are all in the new folders, I'm going to ingest them into a new Lightroom catalogue, one folder at a time and just delete the ones I don't need. I would estimate that I'll cull over 90% and be left with 100k or so that will need more detailed cross referencing. That will still take some time but hopefully all done this year.

Reply

Answer 8

GsyPhoto Author

Level 1

4 points

Aug 15, 2025 05:48 AM in response to GsyPhoto

SOLUTION:

I wasn't really getting on with the Script Editor so I researched apps that do the same job as Finder and found a couple. I tested Forklift and Commander One. Both work but Commander One is the clear winner. It took a while to index the folder (I left it overnight) and now I can select batches of files, 5,000 to 15,000 at a time, quickly and move them to a new sub folder. The move process is especially fast with 10,000 files being relocated in just a few minutes. I haven't timed it precisely but I'd say I can move 100,000 files in about 20 minutes which is way better than any other option I tried. By the end of today, I should have all the files in new sub folders and I can begin the sort & cull with Lightroom.

Thanks everyone for your help and input.

Reply