Continued corespotlightd process CPU overload issues

Mitch Stone wrote:

Earlier in this discussion it was established that iCloud is not the culprit. Files that will trigger the problem will do so whether they are stored locally or in iCloud. I meant corespotlightd because this the process I see as being hyperactive when the CPU is overloaded. Either way I have had this large Pages file open for over an hour now and the file has not grown at all. Unfortunately all of the theories we've come up with so far are incomplete or flawed. They only seem to work for some users some of the time.

I realize iCloud is not the cause per se (because the problem has been reproduced with Pages files that are not on iCloud Drive). I'm curious about exploring the possibility that the problem is worse without the optimize function on. It seems to me that with my two machines, there's just much slower growth of the folder on the machine with optimize on.

I'll also add: I'm not sure your case completely disproves the general thesis. 60Gb is a really large metadata file. It's already at the point that corespotlightd is taking over the CPU. So by this point it doesn't really matter that the folder has stopped (or significantly) slowed its growth; it's already a problem. And I guess I have to assume that the folder got that big for the same reasons as everyone else. If you could delete your folder, and then open a pages file and have the folder not grow – that would be a totally different data point.

Now that I've been deleting my metadata folder (as an experiment) I have absolutely zero issues with corespotlightd. If the corespotlight folder is smaller than 10Gb I find I never have any slowdowns at all.

I have another question: do people with Apple Silicon systems ever see the size of their Spotlight metadata folders decline? I've got two Intel systems and two AS systems, and while I never have seen these folders' sizes shrink (other than when I manually delete them) on Intel systems, I often see them get smaller on their own on AS systems, especially on my M2 Max MBP.

I'm not sure this is actually happening (I suspect it's just the Finder having difficulty giving accurate sizes to folders with a lot of subfolders, which is more the case with AS systems). But I have noted that AS systems do not seem to see as rapid growth of Spotlight metadata as my Intel systems do. I've repeatedly deleted metadata on Intel systems only to see it grow back to an absurd extent. But I've never deleted the metadata out of either AS system, and so far neither one has ever gotten any bigger than 47 MB, and currently neither system has more than 25 GB worth of Spotlight metadata.

ericmurphysf wrote:

I have another question: do people with Apple Silicon systems ever see the size of their Spotlight metadata folders decline? I've got two Intel systems and two AS systems, and while I never have seen these folders' sizes shrink (other than when I manually delete them) on Intel systems, I often see them get smaller on their own on AS systems, especially on my M2 Max MBP.

I can keep an eye out and report back! 🙂

Thank you for this clear summary. I have developed this problem. I AM using large Pages files. The problem developed after one of the fairly recent system updates. I'm on an M2 Air 8 mb memory running 15.3. I am a heavy but ordinary user who is NOT comfortable deleting random files or using Terminal. Can you give us some simple instructions until APPLE fixes this problem. As the computer is quite Unusable at this point. It also has started crashing with the same error (can't recall - but something about devices not loading upon reboot).

For those who can't find your Library file go to Finder, hold the option key while clicking GO - your Library folder will appear. Still concerned about just deleting the Corespotlightd file - but at least I've found it. Thanks for the discussion - I hope Apple does something about it. perhaps modify Pages somehow.

Bets wrote:

Still concerned about just deleting the Corespotlightd file - but at least I've found it. Thanks for the discussion - I hope Apple does something about it. perhaps modify Pages somehow.

I haven't yet had occasion to delete the CoreSpotlight folder on an M-series system yet, but I've deleted all the Spotlight-related folders' contents (while leaving the folders themselves in place) on Intel systems, and have seen no ill effects.

If you want to be super-careful on an M-series system I would recommend you open the ~/library/metadata/CoreSpotlight folder on an Apple Silicon system, and delete all the contents except for the SpotlightKnowledgeEvents folder. Then, when everything else has been deleted, open the SpotlightKnowledgeEvents folder and delete its contents.

This is probably a belt-and-suspenders approach; other users have simply deleted the entire Corespotlight folder without ill effects.

Mitch Stone wrote:

I don't think we know how large this folder is supposed to be, and I can attest to the fact that no process is taking over the CPU with the folder containing this much data.

I definitely don't know, but I think it's safe to assume that this folder (or these folders) definitely should not be more than a hundred GB. On my Intel Macs I didn't really start running into issues until they grew well in excess of that size.

But I think it's safe to assume that 500 GB is well beyond any rational size for metadata, on a system that otherwise only has about a terabyte of total storage in use. As I'd mentioned either here or on another thread specifically devoted to the metadata folders, at their maximum size these folders accounted for the majority of the data in my user folder.

But all that being said, one odd phenomenon I noticed was that when I cleared out these folders on my Intel Macs, doing so also seemed to resolve on my M-series Macs, even though I didn't remove any data from either of those two machines. I can only assume either (i) a happy coincidence; or (ii) that iCloud sync is somehow implicated, because even though my Mac Studio never had more than about 25 GB of metadata, I was having severe issues with 10- to 30-second long freezes, a kernel-panic or two, Time Machine issues, and overall poor performance that near-miraculously resolved after I'd deleted the metadata on the two Intel Macs I own.

I'm prepared to accept (i) might be true, but it's a pretty big coincidence if (ii) is not the case.

ericmurphysf wrote:

I definitely don't know, but I think it's safe to assume that this folder (or these folders) definitely should not be more than a hundred GB.

Agreed. All of MacOS only takes up around 23Gb of space. This metadata folder simply shouldn't grow to more than a few Gbs.

I also think it's relevant that there seems to be, on my system, NO DOWNSIDE to deleting the metadata folder. Spotlight works just as well, maybe better than before. (And note that I am NOT reindexing, toggling the indexing status, or doing anything at all with spotlight – just deleting this folder.)

Mitch wrote:

My issues with this process hogging the CPU seem ridiculously random.

But have you ever had the corespotlightd CPU problem when the metadata folder was fewer than 10Gbs in size?

I can attest that I have not. Corespotlightd processes only become a problem for me when the metadata folder grows to around 30Gb or so. Before that, it's fine.

Regarding the two large folders (NSFileProtectionCompleteUntilFirstUserAuthentication & Priority) under the ~Library/Metadata/CoreSpotlight folder; I don't have "Advanced Data Protection" turned on, so how is it that the NSFile... folders exist - is this part of Apple Intelligence? The NSFile... folders do not exist on my Intel iMac, only on the M2 laptop.

Has anybody removed (or renamed) these large directories, turned AI off and restarted - I just wondered if they get recreated.

Also, has Apple Support shed any light as yet?

Alongside the spotlight issues, has anyone else been having problems with Pages itself falling over?

I have a few quite complex documents and have been finding two things have increased in parallel: all the beachballs/spotlight issues AND Pages falling over and reporting to Apple.

From my perspective, Pages seems to fail when, after making an edit which might itself be trivial, I scroll. Pages disappears, then I get the windows asking if I want the report to go to Apple (and I always do allow that), and if I want to let Pages restart.

Time after time, Pages only loses the tiniest bit of editing. (Hats off to Apple on that score.) But it takes an annoying time to be able to get back to editing the document again.

I'm adding in to this thread because my perception is that both issues have worsened which makes me wonder if they are intertwined?

Given the vast number of automatic reports from me alone, I'd have hoped Apple would notice and fix the Pages issue. And, if it is related, maybe fix the spotlight issue by so doing. Or vice versa.

I also want to appreciate the efforts put in to get the spotlight issue resolved.

Has anyone else noticed that upon restarting the Mac, without ever opening Pages, the disk reads and writes are barely anything at all, but after opening and closing Pages, it will continue to read and write varying low amounts of data at very random and frequent intervals? And seemingly in a very unpredictable fashion compared to if Pages was open? My Mac has been restarted for hours and the disk had so little activity going on. The graph was quite flat. Now after opening Pages and closing it entirely after just a couple of seconds, it won't stop writing very low amounts and reading random amounts of data. I've noticed this for days but today really seals it for me that I'm not imagining it. Once the bug is triggered it cannot truly be eliminated unless you restart.

If there's anything we can say about different users' reports of their experiences with this bug, it's that it manifests very differently on different users' computers. But the one common thread appears to be the problem is most severe when you've got a Pages file open (and it doesn't seem to matter how large that file is or even if it's being edited).

That said, I have observed oddities like today, where I've had a 15 MB Pages file open for three hours that I've been actively editing, and have seen metadata increase from about 11 GB (after I'd deleted all Spotlight metadata last night) to 13 GB, but over the weekend had a single 145 kB Pages file open with no edits being made to it, which apparently took Spotlight metadata from 63 GB to over 120 GB.

I suspect that the larger these metadata files are, the faster they grow in absolute terms. It's almost like compound interest. 10 GB of metadata might increase by 20% a day, but 60 GB might increase by more like 50% or more a day.

That's what worries me about this problem. With the manifestation of it being so diverse and unpredictable despite the commonalities, it could make diagnosing and solving the issue much more complicated for Apple compared to standard bugs. Ironically, when Apple Support was asking me to provide a screenshot, that was the very first time since I bought this Mac that Pages behaved. I had to wait several minutes to get it to trigger again. Very bizarre.

Is it accurate to say though that everyone here was having a good time before Sequoia 15.2?

sugarskyline wrote:

Is it accurate to say though that everyone here was having a good time before Sequoia 15.2?

YES. I never had this problem prior to Sequoia, and I'm 99% sure it didn't show up on 15.0.x or on 15.1.x.