Continued corespotlightd process CPU overload issues

sugarskyline wrote:

Is it accurate to say though that everyone here was having a good time before Sequoia 15.2?

No. I first started running into problems in mid-December, when I think was still running 15.1, and looking at the metadata files written before that time I can see that they were starting to grow not long after I installed Sequoia. Moreover, online research suggests users were running into this issue as early as macOS Ventura, although it appears to have gotten much more prevalent after Sequoia was released.

A couple of other observations that are specifically about Spotlight's effects on storage space and battery life on M-series Apple laptops:

First: it seems that while Spotlight processes definitely pile up quite a bit of metadata on M-series Macs, including laptops, it's not as severe as on Intel Macs. One of the reasons it seems to be less severe is that while on Intel Macs I have never seen the CoreSpotlight folders referenced on this thread ever decline in size unless I manually remove data, I'd say that I see day-to-day declines on M-series Macs about 40% of the time. Sometimes the declines are as much as 50% from the previous day.

Second: even though Spotlight metadata does not accumulate on Apple Silicon Macs the way it does on Intel Macs, Spotlight in combination with Pages seems to have a dramatic effect on battery life while the system is ostensibly "sleeping."

If I have a Pages document open on my M2 Max MBP overnight, that will drain the battery by between thirty and forty percent. If I quit Pages, I won't typically see any drain overnight, i.e., the battery will still be at 100% if I charge it after my last use of the day. There doesn't seem to be a way to prevent M-series Macs from doing all kinds of stuff while they're ostensibly asleep with the lid closed ("Power Nap" doesn't seem to be a function you can disable), but in any case Spotlight appears to continue to index the filesystem even while the laptop is asleep if a Pages document is open on it.

This is a serious problem, Apple. I have a new, basic MBA M3 and it is LAGGING terribly, to the point of being extremely annoying. I have turned off AI, turned off Spotlight on all the folders, put user files into the 'secure' (not indexed). I turned off Spotlight in the terminal (sudo mdutil -i off), force-killed it repeatedly using the Activity Monitor, yet corespotlightd repeatedly restarts. A zombie Spotlight that keeps stealing my CPU is very aggravating, not what we expect or paid for.

fronesis47 wrote:

2. I can now report back on my own experiment: I've got a script that runs every 2 or so days and automatically deletes the corespotlight folder. I've now been running for more than a week and (knock on wood) everything is fine. I never notice any issues deleting the folder, and by deleting it every couple of days it usually stays under 2 gigs in size (though I've seen it as high as 5 gigs). In my experience, the problems don't start until the folder gets north of 25 Gbs.

What's odd about these CoreSpotlight folders is how little effect on Spotlight functionality deleting them has. It does seem like for the first ten minutes or so Spotlight will not return results (and stuff like Smart Folders in Mail don't work properly). But after ten or fifteen minutes Spotlight results are fine.

So it seems like macOS is expending significant system resources in terms of both storage space and CPU time, to produce metadata that is of very limited utility.

I have iMac 27" intel. I delete System Data related to Library>Metadata>"CoreSpotlight" and "Spotlight KnowledgeEvents" EVERY NIGHT (and delete from Trash, too), or I hit the 50GB threshold that affects pace of storage accumulation and performance significantly. Thanks to this Group for this bandage to keep operational.

My tech knowledge is maxed out at starting my lawnmower. Amazed that I found this group. Apple Support let me down badly. At year end, I began to get pop-ups about "Your system has run out of application memory." Apple support baffled. Ultimately they had me delete and refill MacOS and all data, which is still causing me grief. All for nothing. Sigh...

After intense guru meditation, I believe this is related to version history of the pages document. I didn't know Pages was doing version history, but I'm not surprised, and am grateful, except for this behavior we are all lamenting. Pages->file->revert to->show all versions produces a mighty number of versions, each with small incremental changes (which is desirable behavior, mind). I suspect that some bug is triggering some weird loop therein that spotlight can't contend with. Next stop on the troubleshooting train: exclude ~/Library/Metadata from spotlight. Why would we need spotlight to index that anyway for routine operation?

Could you please define large in "large Pages document"?

I had trouble with Pages files 3-6 MB. Since then, I deleted the metadata several time, narrowed Spotlight's scope, eventually to zero, turned off AI, and even switched to Word for a while. Obviously rebooted. Finally the CPU settled down. Now, I have turned Spotlight back on, turned on AI, and gone back to using Pages with the same 3+ MB files, so far no CPU problems. Magic!? (The System data is ridiculously large at 100 GB, but that is not a problem compared with the lag from an over-busy CPU.)

Update - yeah - the mouse update changed functioning only when Pages is not open. Still having lag issues as well as CPU and CoreSpotlight Metadata problems.

For me Pages is a vital tool that cannot be replaced. Dumping the corespotlight metadata one a week or so will have to do until they can do the fix.

REALLY HOPING they are not planning to ditch Pages… it is the most usable program outside of InDesign for my purposes - and InDesign is WAY too expensive!

Mitch Stone wrote:

Just to be clear, everything we suggest here can be nothing more than a Band-Aid. Only Apple can provide the fix. We are told Apple engineering is monitoring this discussion, so any real-world experiences we can document, and anything that works even as a temporary solution, not only helps us manage the problem in the short term, but perhaps will also point Apple towards the permanent solution we all want.

All this said, based on the now 12(!) pages of discussion since I started this thread, I have become convinced that the problem is Spotlight trying to index documents with a large number of edits. This is exactly how it manifested for me, with an 80k word Pages document being edited by two people with Track Changes turned on. Between us, this resulted in probably more than a thousand edits. Towards the end of the editing, I was seeing beach balling every time I opened this document for more than a few minutes at a time, and had one kernel panic.

Once this editing process was completed, I Finder copied the document. I can now open and make additional edits to the copy without incident. If I watch Activity Monitor (I leave it open in Stage Manager with corespotlightd selected), I will see some spikes in the process, but they are not nearly as high or as prolonged as before, and I also don't see any beach balls. To me, this proves the theory pretty conclusively.

Just to add to Mitch's experiences: I am an inveterate journal-keeper, writing in Pages on a daily basis. It needn't be emphasized that a daily journal grows over time, and by the last couple of months of the year, my Pages journal file is typically over a thousand pages, 750k words or more, which with dozens of embedded graphics will result in a file size between 200–300 MB. Moreover, after nearly a year, we're talking literally tens of thousands of edits. When I was first researching this issue at the beginning of this year, I discovered that my Spotlight metadata had increased on two Intel systems to over 500 GB each. On both machines, Time Machine had essentially ground to a halt, Spotlight search results were useless, both systems suffered multiple kernel panics, and corespotlightd would sometimes pin all CPU cores on both systems, using as much as 1400% of CPU time spread across all sixteen cores (eight of which are virtual hyperthreading cores).

For me, deleting Corespotlight metadata resolved all of these issues. Not permanently; I have to weed out the metadata folders every week to ten days. But so long as I keep that metadata below ~50 GB (on relatively high-performance systems with lots of storage space), my computer life remains pretty peaceful. So long as I remember to quit Pages when I'm not actually using it.

But that said, I was editing my journal file last night, and with that one file open in Pages, which is about 18.5 MB, I watched Spotlight metadata grow by literally a megabyte per second or more, for as long as the file was open. In a single day that would add thousands if not tens of thousands of MB of metadata. But as soon as I quit Pages, metadata growth stopped in its tracks, and was the same value this morning before I left for work, twelve hours later.

YMMV, obviously, but for me these kinds of results could not be more dispositive of the problem here: Spotlight coupled with Pages files with many edits.

I just experienced this issue and my ~/Library/Metadata/Corespotlight was 50GB when I first looked, and within an hour it was 60GB in size! I just deleted that, and my caches folder and it appears to have fixed the issue. Thanks for the wonderful folk in here who identified a way to fix it for now.

Daniel_145 wrote:

What would you recommend, please? Reboots don't help, killing the spotlightd process in Activity Monitor helps only for a few minutes. Closing the Pages app helps but unfortunately I really need the document, it's the only reason why I didn't buy a pen and paper instead

Delete the contents of the ~/library/metadata/coresporlight/ folder. This has worked for multiple users including myself.

It's a temporary fix, in that eventually that folder will grow to the size you're seeing now. But I just see it as maintenance. Every couple of weeks I delete that data.