Continued corespotlightd process CPU overload issues

What I meant was that the problems I experienced stopped. My problems were the beach ball spinning and programs momentarily stopping, then starting. That started me looking for the cause. My goal is always to have the computers run smoothly. If I keep using them and these two problem happen again, the culprit is always large folders in ~/Library/Metadata/CoreSpotlight/. After I delete the specific folders that are ~ 2 GB or larger, the beach ball spinning and momentary program stopping does not happen.

The folders so far continue to balloon. Another indicator of the issue if the amount of disk space used. This increases with the above problems and the CoreSpotlight folders increasing in size. This disk space issue is also solved by deleting the folders. The space used always drops after deleting the folders - no surprise.

I spent hours reloading the OS and speaking with several levels of Apple support up to their highest level. One tech from the highest level suggested deleting the largest folders in ~/Library/Metadata/CoreSpotlight/ and it was the only suggestion that worked.

My M1 MBA had this problem with the Sequoia upgrade but it isn't nearly as bad with Tahoe. I haven't had to delete folders after doing it once or twice. My M4 iMac folders still ballon.

I hope this answers your question.

fronesis47 wrote:

I have just now again deleted the entire corespotlight folder in (in the library folder) and the preferences file you mentioned, but I think I see the corespotlight folder already growing again. We'll see.

This has been my experience too. An hour or so after deleting the contents of these folders, I'll see a few GB of metadata accumulating already. And if I have a Pages file open, that growth will continue. Just a week after deleting all of this metadata (on a computer that had a particular Pages file open and being edited more or less continuously), the CoreSpotlight folders had already grown to over 100 GB.

In my experience, having a Pages file open on a system, at least a large one, but even if you're not actually editing it, seems to accelerate the accumulation of metadata. On one of my systems I had a ~14 MB Pages file open, and metadata seemed to accumulate at 10-20 GB per day. Closing that one file when I wasn't actively editing it greatly slowed down the accumulation of metadata.

As noted elsewhere, my hypothesis is that when you have a Pages file open (especially if you're editing it), the various Spotlight processes don't just reindex the changes; they re-index the entire file, and not by replacing existing metadata but by adding to it. In my case, the Pages files in question are synced via iCloud and are being edited on multiple Macs. It seems that if you close the file, then edit it on another computer, and then later re-open it, Spotlight reindexes the entire file, but only that one time (until you edit it some more). If you leave it open while you're editing it on another system, the same thing happens: Spotlight seems to reindex the entire file with every edit. This can add tens of gigabytes a day of metadata on a file that might only be a few megabytes in size.

Until Apple releases a fix for this issue (which may or may not ever happen), I think the best way to avoid rapid accumulations of metadata is to close Pages files when you're not actively editing them. Even having them open on another system and editing them there (if they're synced via iCloud) will still lead to vast quantities of metadata creation, as much at 20 GB a day.

Just to provide an update and add more context to the discussion:

I'm no longer experiencing any hiccups, mini-freezes, or beachballing. While I’m currently writing my thesis in Pages, my issue might not have been directly related to the app after all.

A few key points:

• I'm working entirely with iCloud files (Pages), and while corespotlightd still spikes occasionally, it does so as expected and without causing any slowdowns.
• kernel_task and corespotlightd are no longer consuming excessive CPU resources.

The turning point seemed to be when I deleted my 50GB ~/Library/Caches folder. After restarting, the system automatically rebuilt the Cache folder, but it has remained steady at around 600MB ever since.

Additionally, I reset my Spotlight index, first via System Settings and then using Terminal with sudo commands to wipe and rebuild it. After indexing completed (which took around an hour or so), my ~/Library/Metadata/Corespotlight folder settled at around 30GB. Despite its size, my system is now running as smoothly and responsively as I expect it to be.

I personally can't work without spotlight anymore and try to index most of the files on my system to have an extremely fast access via CMD+Space. What I'm trying to say is: although my Spotlight folder currently seems to be this big I'm not seeing any performance issues on my end.

Things too keep in mind about my situation:

• I'm not working on super big Pages files with file sizes over dozens of MBs
• I'm not using storage optimization via "Store in iCloud" in the storage settings
• I have updated from 15.2 to 15.3 with obviously very big Cache and Indexes (~/Library/Caches & ~/Library/Metadata/Corespotlight basically never emptied since I own this M1 MacBook from 2021 onwards)

CaptainJoy wrote:

tl;dr — trash your ~/Library/Metadata/CoreSpotlight folder

This has consistently been my experience. I start running into issues with Spotlight Search when this folder exceeds about 40 GB. The system runs into serious performance issues on my computers (which have 32+ GB RAM) when it gets above about 100 GB.

I'm a heavy Pages user, with 10+ MB Pages files open all the time, which seems to accelerate the growth of this folder. So once a week or so I just trash the entire folder.

It's kind of like mowing the lawn.

In my case the issue was related to Docker Desktop that I installed on my Intel-based MacBook Pro running Sequoia 15.7.1. I found more than 40GB of data in my ~/Library/Containers/com.docker.docker/Data folder. After uninstalling Docker App and clearing docker Data folder the issue with corespotlightd using a lot of CPU went away within few seconds.

I am not certain that I installed and used Docker Desktop correctly, therefore this is not a judgement on Docker Desktop (I find Docker very useful). Also, I uninstalled Docker Desktop and cleared the Data folder at the same time, so I am not sure if the issue was not related to Docker Daemon activity. However, inspecting ~/Library folders may help in troubleshooting the problem discussed in this thread.

Mitch,

I think you could be onto something here. My brand-new, maxed-out Mac Mini was working perfectly, then after a kernel panic, reboot, and relaunch of all my open apps, corespotlightd started dominating the CPU and causing cursor and data input freezes. Pages.app has an iCloud-resident, password-protected file of text and images that I add to daily and that relaunched as a part of the system reboot. That file is currently 29.5 MB.

After reading your post I closed the large Pages file and after a few minutes corespotlightd dropped off the top of the CPU list in Activity Monitor. I did not delete any SpotlightResources.plist files. When I reopened the large file, corespotlightd again started climbing to the top and intermittent cursor freezes reoccured.

I quit Pages.app and corespotlightd disappeared as did the freezes. Now I've relaunched the app and reopened the large file,. All is well, no cursor freezes and corespotlightd is at 0% of CPU.

Looking at Console.app, I see this in one of the diagnostic reports (date and time concurrent with the freezing issue).

Command:          corespotlightd
Path:             /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Support/corespotlightd
Codesigning ID:   com.apple.corespotlightd
Resource Coalition: "com.apple.corespotlightd"(849)
On Behalf Of:     445 samples Pages [971] (445 samples originated by Pages [971])
Architecture:     arm64e
Parent:           launchd [1]
PID:              4063

I don't know if any of that is relevant. I speculate (wildly) that spotlight might have been trying to index an open, yet password-protected large Pages file and that was causing the system-wide issues.

The file has been open for 30 minutes and there are no issues and corespotlightd is not showing up on the CPU list.

I hope this helps.

Mitch, I can reliably reproduce the bug by opening a large (34 MB) Pages file, adding some text then letting the application sleep (TOP command in Terminal confirmed). After several minutes, corespotlightd takes over the CPU resulting in stuttered data input and spaces swaps, and cursor freezes. After quitting Pages the issue resolves after a few minutes. If you're going to open a bug report, I'm happy to submit configuration data and log files.

I'm not sure how many other people with this issue have seen degraded Spotlight results (including in Mail) as a result of corespotlightd's misbehavior, but I managed to at least temporarily resolve some of these issues by, on the advise (or at least consent) of Apple support, deleting the contents of the two folders, CoreSpotlight and SpotlightKnowledgeEvents, from ~/library/metadata/.

Note that I deleted the contents of these two folders, not the folders themselves. Also note that on Apple Silicon systems the SpotlightKnowledgeEvents folder is inside the CoreSpotlight folder. On Intel systems, it's at the root of ~/library metadata.

However, deleting the contents of these folders (on my system those contents comprised over half a terabyte of data) did not permanently resolve the issue. In barely twelve hours Spotlight added 22 GB of new metadata to these two folders. But I think until Apple resolves this issue (I doubt it will be in 15.3), simply deleting the contents of these folders when they get over a couple of hundred GB will definitely improve system performance, especially search.

Also note that in my experience these issues are less serious on Apple Silicon Macs. On my M2 Max MBP and my M1 Ultra Mac Studio, these folders are large but not enormous; 40 GB on the first system and 18GB on the second one.

My 2024 M4 Mac Mini was unusable: spinning beachballs, screen locking up for a few seconds. I tried several things—force quitting the corespotlightd process, Rebuild the Spotlight index on your Mac, turning off Apple Intelligence, deselecting all categories from Spotlight search—none of this worked.

I can confirm, that as instructed by ericmurphysf above, when I deleted the metadata out of the ~/library/metadata/Corespotlight and ~/library/metadata/SpotlightKnowledgeEvents (while leaving the folders themselves intact), I was rewarded with near-immediate improvement. Hopefully this holds up.

I'm wondering why I was affected and others not? One unusual thing about me is that I was upgrading from a 2014 Mac Mini. Maybe the jump from Monterey (OS 12) to Sequoia (OS 15) when migrating my old stuff over via Time Machine had something to do with it?

sugarskyline wrote:

The person I spoke to wouldn't add my case to their file unless I tried booting in Safe Mode to see if the issue was still present, so be prepared for that, or potentially anything else disruptive for them to give your case validity. If you start with a chat that moves to the phone then also have your case number ready because the person on the phone won't have access to your chat log otherwise. The person I spoke to was fantastic so I wouldn't worry about dealing with typical poor customer service like you would from other companies. The call happened 5 minutes after the scheduled time and the chat representative showed up almost immediately.

One of the first things I tried, after this issue escalated so severely on my iMac Pro that Time Machine ceased working entirely, was to reinstall macOS (since I know from past experience Apple Support will frequently recommend doing so and will be reluctant to proceed further until you've done so). Unsurprisingly that had no effect on the issue, but Apple Support at least were willing to progress once I've told them I already tried that. If anyone does get such a suggestion I'd recommend you refer the advisor to this very thread, where numerous posters have stated that doing so has no effect on the problem.

Also, given the complexity of this issue, I would recommend that as soon as you get a support advisor on the phone, you request the issue be immediately be escalated to a senior advisor. No offense to Apple support advisors (who are excellent by comparison to the rest of the industry), but this is not the kind of issue a level I tech, who is mainly helping people install printers or replacing application icons that have been inadvertently dragged out of the Dock, are likely to be able to advise on.

Given what posters on this thread have determined so far, it's almost impossible to believe restarting in Safe Mode would have an effect. Doing so generally just disables non-macOS processes, but since the culprits are almost certainly one or more of the various Spotlight-related processes which are core to macOS and would still run under Safe Mode, doing so would largely be a time-wasting exercise.

Also, creating a new user account and using that might eventually be able to reproduce the problem, but since it seems like the metadata folders need to grow to at least 50 GB or more in size to be problematic, one would likely need to create a new user account, and then create/open/edit one or more Pages files, and wait a week or more (although some people report metadata growth rates that might bring that down to just a few days) to try to reproduce the problem.

Honestly, at this point I think Apple's engineers need to focus on how Spotlight indexes Pages files, and possibly other types of data files. At least in my experience (which admittedly does not seem to be universal), the core of the problem appears to be that Spotlight indexing processes will repeatedly reindex entire Pages files over and over again, appending those results to existing metadata rather than overwriting that data.

As evidence of that, I will again point to my experience over this last weekend. I had deleted the metadata from my 27-inch iMac on January 28, at which point it was 597 GB. From the 28th until February 7th it grew from zero to about 62 GB, with no ill effects. But then after leaving a single 145 kB Pages file open (but not being edited) from Friday evening until Sunday evening on a system that was completely idle, the metadata folders ballooned to over 120 GB, nearly doubling in 48 hours.

By contrast, with no Pages files open, this system would add about two gigabytes of metadata over a weekend.

Update: I agree the problem is primarily with Sequoia as I didn't have this at all until that update.

Since realizing the connection with Pages documents and the solution of emptying the ~/Library/Metadata/Corespotlight folder, I have only had to empty the folder twice. (about every 3 weeks of heavy use).

Key Points:

1. Turn OFF the Pages App when you are not using it. My Corespotlight folder WILL reduce in size when Pages is closed.
   1. Duplicate large/old Pages files in the finder and work with them. I've broken some documents into parts.
2. Empty the ~/Library/Metadata/Corespotlight folder when you notice slow down (on Activity Monitor or other).
3. Hope that Apple fixes the problem.

Thank you to those who gave safe solutions..

I can confirm that this works for me (MacBook Pro M3 14", Nov. 2023 + Sequoia 15.5)

1) Work on local Pages documents (temporary folder in Downloads for example).

2) Copy from time to time to iCloud folder such as Documents.

After a couple of days the size of ~/Library/Metadata/CoreSpotlight folder went down from 50 Gb to 0.7 Gb (!!!)

I had similar problems and posted a while ago about sluggish programs and overload issues with an M4 iMac. Per instructions from Apple 'elevated' support, I deleted the entire large folders in ~/Library/Metadata/CoreSpotlight/. I found that they rebuilt later but were smaller. It was not necessary to go into the folders and delete their contents.