Continued corespotlightd process CPU overload issues

Turning off Apple Intelligence on my M2 MBA macOS 15.2 stopped corespotlightd from hogging the CPU, dropping the process from in excess of 250% of CPU to 0.0 % within 8 hours. However, when I went back into my Pages doc (currently 1.2 MB, stored on iCloud), I continued to experience the spinning rainbow wheel, although less frequently and for less time. I've also experienced hesitation in my other apps (including while typing this post!), but no spinning wheel. I've talked with a member of the Senior Support Team at Apple and have arranged for them to collect data for Engineering to evaluate. That happens this coming Tuesday (my schedule doesn't fit with theirs until then). As much as we all don't like it, it appears it's a problem we'll have to live with for a bit until Engineering can figure out what's happening

I've had similar problems with all four of my Macs (a Mac Studio with an M1 Ultra, a MacBook Pro with an M2 Max, an iMac Pro, and a 17-inch 2020 iMac with an 8-core Core i7). On all four systems, I noted sometime around when 15.1.1 came out that corespotlightd would frequently top the list of processes, using anything from 100% CPU all the way to 750%(!) of CPU (on the iMac), causing the fans to spin up on the Intel systems to annoying levels. Updating to 15.2 did not resolve the issue for me, and in fact may have worsened it.

I did notice that corespotlightd calmed down quite a bit simply by closing a large Pages doc (200+ MB), and even more reliably by simply closing Pages on systems where I wasn't actively editing documents (at least, documents that are synced to iCloud). Corespotlightd will still occasionally ramp up to 100+% of CPU, but it won't stay there indefinitely, and most of the time it will be under 50%.

But another more serious issue: I found that after I installed 15.1.1, I could no longer back up the Intel systems via Time Machine. If I tried, one or both situations would arise: (i) a Time Machine backup would be "preparing" for hours or even days; and (ii) attempting such backups would frequently lead to repeated kernel-panics. The only resolution for this latter issue I have found is to disable Time Machine backups entirely. Even starting a backup on a freshly-erased backup drive with no existing Time Machine backups would still lead to the same behavior vis à vis interminable "preparation" of backups and repeated kernel panics.

One thing occurred to me during all of this trouble-shooting: I believe that Time Machine relies on Spotlight to identify files which have been modified since the last backup. I think it may be there is some bug in corespotlightd that aside from consuming vast system resources, also leads to complete (albeit short-lived) system freezes where they don't cause a kernel panic, and also interferes with the proper operation of Time Machine.

Just for, I don't know, comic relief maybe: I came home today to find that corespotlighd was using 1,400% of available CPU time on my 8-core iMac Pro.

How the system avoided crashing completely is a bit of a mystery.

Mitch Stone wrote:

Interesting. Checking these folders on my M2, I find that the size of the CoreSpotlight folder is 37GB, but no file within it is even a megabyte in size. The SpotlightKnowlegeEvents folder clocks in at 463 MB. It contains far more subfolders, so it is difficult to figure out where this data is hiding. Has your performance been improved by deleting the contents of these folders?

And another FWIW: Since the last time I deleted the Spotlight plist about a week ago, I have had no corespotlightd process issues. So I do think this is worth trying.

As noted, this issue does not appear to be as severe on Apple Silicon systems as it is on Intel systems. My M2 Max MBP has about what yours has in the CoreSpotlight folder: about 42 GB. But many of the files in there are tens of megabytes, some over 100 MB (but I edit large Pages documents, and others have pointed out that this can exacerbate the problem).

So far I've deleted the contents of the referenced folders on the two Intel systems I own—a 2020 27-Inch iMac and an iMac Pro—and saw immediate performance gains, especially with anything having to do do with search: Spotlight searches, smart folders in Mail, etc. Corespotlightd also seems to have calmed down significantly, generally using less than 15% of available CPU time (yesterday I saw it go as high as 1,400% on an 8-core system). Prior to deleting this metadata, it could take upwards of five minutes just to log out of my account; now it's just however long it takes to quit all running apps).

I haven't tried deleting the spotlight .plists yet (there are appear to be five plist files with "spotlight" in the filename, at least on Intel systems), but I do see that Spotlight is still writing large amounts of metadata to the user library folder, 8.4 GB in just the last two hours. If that trend continues, I'l try deleting the .plist files as well.

ericmurphysf wrote:

So far I've deleted the contents of the referenced folders on the two Intel systems I own—a 2020 27-Inch iMac and an iMac Pro—and saw immediate performance gains, especially with anything having to do do with search: Spotlight searches, smart folders in Mail, etc. Corespotlightd also seems to have calmed down significantly, generally using less than 15% of available CPU time (yesterday I saw it go as high as 1,400% on an 8-core system). Prior to deleting this metadata, it could take upwards of five minutes just to log out of my account; now it's just however long it takes to quit all running apps).

Progress report: since deleting all this metadata, not only has the system seemingly stopped writing more of it (or at least slowed way, way down in adding to it), but corespotlightd has been at barely above 0% in terms of CPU utilization. I'm not sure this is a magic bullet in resolving this issue, but so far it seems to have resolved a plethora of issues I've had with my Intel Macs.

CaptainJoy wrote:

I'm wondering why I was affected and others not? One unusual thing about me is that I was upgrading from a 2014 Mac Mini. Maybe the jump from Monterey (OS 12) to Sequoia (OS 15) when migrating my old stuff over via Time Machine had something to do with it?

How heavy a user of Pages are you? What I can say for a virtual certainty is that editing large (>10 MB) Pages documents massively increases the amount of metadata saved to the folders inside ~/library/metadata/ that are concerned with Spotlight indexing. If I don't have any Pages documents open on a Mac, metadata in these folders might grow by a couple of hundred megabytes over a span of 24 hours (which is still a lot, but it's not insane). With a large Pages file open, I might see an additional gigabyte of new metadata over a period of less than an hour.

My guess as to what's happening here is that, when you edit a Pages document, it's saved automatically to storage (and if iCloud sync is turned on, it's also saved to the cloud). But instead of the various Spotlight indexing processes just indexing the new content in the file, they reindex the entire file, and save the additional data alongside of existing metadata rather than replacing it. Consequently, a single 10 MB file might result in tens of gigabytes of metadata being saved if you spend a lot of time editing that one Pages file.

As evidence in support of this hypothesis, I've opened up several of the larger (tens to hundreds of megabytes) of the .journal files that are saved in these metadata folders in TextEdit. A simple search shows that the larger ones contain tens of thousands of references to the specific Pages file I happen to be editing.

That seems pretty close to a QED in terms of what's happening.

CaptainJoy wrote:

Now that my Time Machine backups are working again, I will probably try the above the next time the corespotlightd process gets too big for its breeches. After less than a day, it's up to 21.41 GB. I wish I noted how big it was before I cleaned it out. Going forward, I request people let us know how big their CoreSpotlight folders are before they delete them.

I've been keeping notes on the four Macs I own, all of which have this issue to a greater or lesser extent. The two Intel systems (a 27-inch iMac and an iMac Pro) each had about half a terabyte of Spotlight metadata in the two folders in ~/library/metadata/ that are associated with Spotlight. I deleted the contents of those folders as noted above, and by the next day those folders had grown to about 11 GB on both of them. Since then, depending on how much I've been using Pages on those systems, they have grown by between four and twenty-five gigabytes a day. The iMac Pro is already up to 103 GB (after eight days since I deleted the contents of those two folders).

On the two Apple Silicon systems, an M1 Ultra Mac Studio and an M2 Max MBP, metadata has grown much more slowly, largely I think because I tend not to use Pages as much on those two systems. In the four months since I updated to Sequoia, those systems have accumulated about 22 and 37 GB respectively, and so I have never deleted the contents of their metadata folders (largely because metadata at that size does not seem to impose much of a load on the system).

Since the metadata folders have exceeded 100 GB on my iMac Pro (in barely over a week), I'm going to delete the contents of those folders again. As I was typing this, I saw corespotlightd CPU usage spike dramatically, and at one point the entire system halted for about fifteen seconds.

That can't be good.

fronesis47 wrote:

But before I do that, I just have to say that Apple's "sort by rank" default option is a nightmare: it just makes the thread seem like meaningless nonsense, even though so many here are trying hard to solve the problem together.

With respect to this comment: yes, the "sort by rank" sorting isn't just useless; it's actively detrimental to actually following any kind of conversation. And to make matters worse, it's the default.

You can, however, sort by more useful orders, such as oldest first, newest first, etc. But the only place you can do that is on the first page of the thread, where the option is placed inconspicuously below the OP. It took me years of using Apple's community fora to even find this option.

Mitch Stone said:
Curiously, deleting the Spotlight plist file does tame this behavior for better part of a day.
Turning off the "help Apple" selection in the Spotlight settings as suggested by another user 
did as well. But this is not a sticky fix. The problem always returns, at least for me.

Mitch Stone, you've been advocating for deleting the Spotlight plist file. The reason I opted for deleting the contents of the CoreSpotlight and SpotlightKnowledgeEvents folders was because you said on Dec. 30 that deleting the Spotlight plist file was a fix that would only last about a day. Am I misunderstanding something?

yes - mac studio here, OS15.3.. i did follow this.. as well as turned off spotlight in terminal as instructed and computer is running lag free again. I hope for OS update soon and will turn spotlight back on... thank you.

[Edited by Moderator]

As an experiment, I left Pages running over the weekend on my work computer which was otherwise idle all weekend. The only Pages file open was a single document that's all of 145 kB. Nevertheless, the Spotlight metadata folders on this system have exploded since Friday evening, going from 63 GB to 123 GB. Otherwise the system seems to be running normally, with the CPUs at about 85% idle and corespotlightd nowhere to be seen in Activity Monitor (it's using about 3% CPU).

Nevertheless, having learned my lesson, the first thing I'll be doing when I get in to work tomorrow is simply deleting the metadata folders. I've done this three times before with no ill effects, it's completely resolved the Spotlight and Time Machine issues I've been having, and until Apple resolves this issues (assuming they ever do), it's a quick, easy, and relatively painless solution. Having to delete a couple of folders every few weeks doesn't seem that onerous, even if I have to do it indefinitely.

PolyRod wrote:

From my perspective, Pages seems to fail when, after making an edit which might itself be trivial, I scroll. Pages disappears, then I get the windows asking if I want the report to go to Apple (and I always do allow that), and if I want to let Pages restart.

Time after time, Pages only loses the tiniest bit of editing. (Hats off to Apple on that score.) But it takes an annoying time to be able to get back to editing the document again.

I'm assuming you've tried the fix of deleting the Spotlight folders from your ~/library/metadata/ folder? If so, has the problem with Pages persisted even after doing so?