Continued corespotlightd process CPU overload issues

Was trying to solve this issue and happened to notice the setting below. Help Apple Improve Search in the Spotlight options.

I don't have any recollection of letting Apple improve search! Disabled. And found spotlightcored dropped to effectively zero CPU!

No idea if this will remain the case. But seems worth a go if it is selected on your machine.

I was also convinced that this was related to having a large Pages file open. But then, after archiving last month's 35MB+ file and starting a new, empty one, I had the same issues that would not go away, even if I quit Pages.app and waited. It was making my Mini virtually unusable for writing (my main task) so I switched to my iPad, rebooted the Mini and started running Clean my Mac routines on it. No significant change. corespotlightd had shot to the top of the usage list within a minute or so of reboot and stayed there. By this time, I had two terminal widows open alongside Activity Monitor and I was watching what was going on carefully ('-s 5' to slow things down and '-o state' to list stuck processes at the top). At various times I could have up to 20 stuck processes. One of them was launchd which led me to some further research and the idea of disconnecting my Time Machine Drive. Within a few minutes corespotlightd had vanished from the CPU usage list. Reopening the previously opened Pages file was fine until I swapped to another programme and then corespotlightd returned. I saved and quit Pages and it went away again. I'm including all this info in case anyone else is having the same issues as I am. Maybe try disconnecting the time machine drive? (It seems a bit nuts, I know Spotlight isn't supposed to index that drive.)

Mitch Stone wrote:

Interesting. Checking these folders on my M2, I find that the size of the CoreSpotlight folder is 37GB, but no file within it is even a megabyte in size. The SpotlightKnowlegeEvents folder clocks in at 463 MB. It contains far more subfolders, so it is difficult to figure out where this data is hiding. Has your performance been improved by deleting the contents of these folders?

And another FWIW: Since the last time I deleted the Spotlight plist about a week ago, I have had no corespotlightd process issues. So I do think this is worth trying.

As noted, this issue does not appear to be as severe on Apple Silicon systems as it is on Intel systems. My M2 Max MBP has about what yours has in the CoreSpotlight folder: about 42 GB. But many of the files in there are tens of megabytes, some over 100 MB (but I edit large Pages documents, and others have pointed out that this can exacerbate the problem).

So far I've deleted the contents of the referenced folders on the two Intel systems I own—a 2020 27-Inch iMac and an iMac Pro—and saw immediate performance gains, especially with anything having to do do with search: Spotlight searches, smart folders in Mail, etc. Corespotlightd also seems to have calmed down significantly, generally using less than 15% of available CPU time (yesterday I saw it go as high as 1,400% on an 8-core system). Prior to deleting this metadata, it could take upwards of five minutes just to log out of my account; now it's just however long it takes to quit all running apps).

I haven't tried deleting the spotlight .plists yet (there are appear to be five plist files with "spotlight" in the filename, at least on Intel systems), but I do see that Spotlight is still writing large amounts of metadata to the user library folder, 8.4 GB in just the last two hours. If that trend continues, I'l try deleting the .plist files as well.

CaptainJoy wrote:

I'm wondering why I was affected and others not? One unusual thing about me is that I was upgrading from a 2014 Mac Mini. Maybe the jump from Monterey (OS 12) to Sequoia (OS 15) when migrating my old stuff over via Time Machine had something to do with it?

How heavy a user of Pages are you? What I can say for a virtual certainty is that editing large (>10 MB) Pages documents massively increases the amount of metadata saved to the folders inside ~/library/metadata/ that are concerned with Spotlight indexing. If I don't have any Pages documents open on a Mac, metadata in these folders might grow by a couple of hundred megabytes over a span of 24 hours (which is still a lot, but it's not insane). With a large Pages file open, I might see an additional gigabyte of new metadata over a period of less than an hour.

My guess as to what's happening here is that, when you edit a Pages document, it's saved automatically to storage (and if iCloud sync is turned on, it's also saved to the cloud). But instead of the various Spotlight indexing processes just indexing the new content in the file, they reindex the entire file, and save the additional data alongside of existing metadata rather than replacing it. Consequently, a single 10 MB file might result in tens of gigabytes of metadata being saved if you spend a lot of time editing that one Pages file.

As evidence in support of this hypothesis, I've opened up several of the larger (tens to hundreds of megabytes) of the .journal files that are saved in these metadata folders in TextEdit. A simple search shows that the larger ones contain tens of thousands of references to the specific Pages file I happen to be editing.

That seems pretty close to a QED in terms of what's happening.

CaptainJoy wrote:

Now that my Time Machine backups are working again, I will probably try the above the next time the corespotlightd process gets too big for its breeches. After less than a day, it's up to 21.41 GB. I wish I noted how big it was before I cleaned it out. Going forward, I request people let us know how big their CoreSpotlight folders are before they delete them.

I've been keeping notes on the four Macs I own, all of which have this issue to a greater or lesser extent. The two Intel systems (a 27-inch iMac and an iMac Pro) each had about half a terabyte of Spotlight metadata in the two folders in ~/library/metadata/ that are associated with Spotlight. I deleted the contents of those folders as noted above, and by the next day those folders had grown to about 11 GB on both of them. Since then, depending on how much I've been using Pages on those systems, they have grown by between four and twenty-five gigabytes a day. The iMac Pro is already up to 103 GB (after eight days since I deleted the contents of those two folders).

On the two Apple Silicon systems, an M1 Ultra Mac Studio and an M2 Max MBP, metadata has grown much more slowly, largely I think because I tend not to use Pages as much on those two systems. In the four months since I updated to Sequoia, those systems have accumulated about 22 and 37 GB respectively, and so I have never deleted the contents of their metadata folders (largely because metadata at that size does not seem to impose much of a load on the system).

Since the metadata folders have exceeded 100 GB on my iMac Pro (in barely over a week), I'm going to delete the contents of those folders again. As I was typing this, I saw corespotlightd CPU usage spike dramatically, and at one point the entire system halted for about fifteen seconds.

That can't be good.

AshkaTheMoltenFury wrote:

So in short: give the cleanup of the ~/Library/Caches folder a try. It might help and solve this high CPU usage of corespotlightd. Hope this helps anyone.

tl;dr — cleanup of the ~/Library/Caches folder did not work; trashing the contents of ~/Library/Metadata/CoreSpotlight and ~/Library/Metadata/CoreSpotlight/SpotlightKnowledgeEvents did work.

Folder/File Sizes Before "Fix"

• /System/Volumes/Data/.Spotlight-V100 at zero bytes
• ~/Library/Metadata/CoreSpotlight at 66.66 GB
• ~/Library/Metadata/CoreSpotlight/SpotlightKnowledgeEvents at 9.59 GB
• ~/Library/Caches at 1.9 GB

I do not believe “Optimize Storage” is turned on

Disk Writing:

• kernel_task had written 7.32 TB
• mds_stores had written 954.75 GB
• launchd had written 535.32 GB (https://www.technewstoday.com/mds-stores-on-mac-high-cpu-usage/ recommends disabling Spotlight—which is throwing the baby out with the bathwater in my opinion)
• backupd had written 82.42 GB
• corespotlightd had written 51.61 GB

"Fix" Attempts

Feb 11 6:18 PM — trashed ~/Library/Caches

• corespotlightd remained around ≥100 % CPU for 10 minutes
• No indication this did anything to improve my situation

Feb 11 6:28 PM — trashed the contents of ~/Library/Metadata/CoreSpotlight and ~/Library/Metadata/CoreSpotlight/SpotlightKnowledgeEvents which immediately resulted in:

• corespotlightd down to <25%
• Disk writing no longer happening constantly
• Every indication this has "fixed" my problems.

This was the 2nd time I've had to "fix" my sluggish, cursor freezing, beach-ball generating 2024 M4 Mac Mini running Sequoia 15.3. I put "fix" in quotes because this is only a temporary solution. The last time I had to implement this "fix" was 3 February, so it seems to last about a week for me. I was no longer keeping Pages documents open unless actively using them ; I think I'll go back to leaving my planner Pages document open like I used to and see how much it cuts down the time before my next "fix".

PS AshkaTheMoltenFury is an hilarious handle.

fronesis47 wrote:

2. I can now report back on my own experiment: I've got a script that runs every 2 or so days and automatically deletes the corespotlight folder. I've now been running for more than a week and (knock on wood) everything is fine. I never notice any issues deleting the folder, and by deleting it every couple of days it usually stays under 2 gigs in size (though I've seen it as high as 5 gigs). In my experience, the problems don't start until the folder gets north of 25 Gbs.

What's odd about these CoreSpotlight folders is how little effect on Spotlight functionality deleting them has. It does seem like for the first ten minutes or so Spotlight will not return results (and stuff like Smart Folders in Mail don't work properly). But after ten or fifteen minutes Spotlight results are fine.

So it seems like macOS is expending significant system resources in terms of both storage space and CPU time, to produce metadata that is of very limited utility.

After intense guru meditation, I believe this is related to version history of the pages document. I didn't know Pages was doing version history, but I'm not surprised, and am grateful, except for this behavior we are all lamenting. Pages->file->revert to->show all versions produces a mighty number of versions, each with small incremental changes (which is desirable behavior, mind). I suspect that some bug is triggering some weird loop therein that spotlight can't contend with. Next stop on the troubleshooting train: exclude ~/Library/Metadata from spotlight. Why would we need spotlight to index that anyway for routine operation?

I've had a similar issue on my M1 Mac Mini, Sequoia 15.4.1 with freeze-ups every couple of minutes lasting 5-10 seconds. The Disk appeared to freeze for this period, unresponsive mouse (but the keyboard would work) followed by a massive surge in write outs and high activity on Kernel_Task and Corespotlightd.

I think I've fixed the issue thanks to several comments on this and other threads pointing towards problems with Pages.

I have recently produces a large document containing HEIC photos transferred to my Downloads folder which I have deleted and then emptied the Trash and the problem has completely gone away.

Thanks to everyone who has helped me and I hope my addition helps someone else.

revpilot wrote:

Thank you Eric. I have gone to that and there is no folder listed as metadata. ** I looking in the wrong place? If I **, I would appreciate you telling me how to find this. I have gone to Settings, clicked Spotlight, Privacy and told it to rebuild, and have not have major success with that. I did this when an Apple Support person told me to this. Any help is greatly appreciated.

The metadata folder I'm referring to is in the user library folder. To get to your user library folder, go to the "Go" menu in the Finder. If you don't see the Library in the "Go" menu, hold down the Option key on the keyboard. Scroll down to the Metadata folder inside the Library folder. Inside, among a few other folders, you will find a folder titled "CoreSpotlight."

Apple recommends that you delete the contents of the CoreSpotlight folder, but not the folder itself. That's how I've always done it. Other users have deleted the folder itself, apparently without ill effects. Either way, macOS will recreate the folder (if you delete that) or its contents (if you just delete the contents). You might need to wait a while for Spotlight searches to return usable results after deleting, so I usually delete the metadata when I know I won't be using the computer for a while, either right before I leave work or before I retire for the evening.

Note that this is not a one-time fix. I typically delete this metadata when it gets above 40 GB, which depending on who knows what other factors can take anywhere from weeks, or months, to less than twelve hours. In my

experience having a large Pages file open reduces the time before I have to delete the metadata again.

I deleted the large folders only in ~/Library/Metadata/CoreSpotlight/. I sorted the folders by size and any folder that has grown in GBs from a previous check gets deleted. My problem happens soon after an update on my M4 iMac. Another indication of the problem is that my GB of used space (System Settings/General/Storage) jumps from what I know to be the used space for me of around 60 GB of space used. I keep an eye on the used space on a regular basis and keep pictures to a minimum.

Update: Slow upload data speeds appear to be causing lags/spinning wheels while working in my larger Pages documents on iCloud. I often work at remote locations using a VPN. I have no control over the upload speeds on these remote Wifi connections (community college Wifi, coffee shops, etc). So, I moved my large Pages files off iCloud drive to my computer drive. Since moving the files, I have not encountered any issues high CPU usage, lags, or spinning wheels while working on those documents. Emboldened, I turned on Apple Intelligence and so far, everything is working perfectly. No processes are hogging the CPU and there are no delays/spinning wheels while working within any app, including Pages.