Etrecheck says 'Failing hard drive - SMART status failing'

khlota wrote:

Thank you:) My husband is a Sr Network Engineer and believes he can change the hard drive for me. I’ve thought about getting an external drive for backups as well. Do you think that with Time Machine is better than an offsite service? I’ve actually never used Time Machine :)

Time Machine works, both locally to directly-attached storage, and to a NAS, and various NAS options can then back up offsite. There are local apps that can back up local data remotely, though I’m less certain about the ease of use and particularly re-installing and recovering from those backups. Recovering files and documents sure, but not so certain about restoring everything and then booting. Time Machine can be used for restoring files and documents, and also for restoring macOS and also migrating to a re-installed or a new Mac.

Time Machine is free, works, deeply integrated with restores and replacements, and supports mixes of multiple local and remote backup targets.

As for NAS hardware choices, there are various previous discussions around here, and Ubiquiti UNAS options (your spouse might like that choice and its integration with other Ubiquiti networking gear, too) and Synology are common choices, and FreeNAS and UGreen and other choices exist. (Synology is reportedly again allowing third-party HDDs with DSM 7.3.) Whatever NAS you pick must support Time Machine server and SMB.

Getting at the internal iMac hard drive for the repair is the hassle. There are tools available, and at least one third-party teardown and third-party service manual document. Search for fix it Mac Intel 21.5" Retina 4K Display 2019 or such. Yeah, yours is a 27”, and not a 21.5”. Search for that 21.5” anyway.

For the internaI hardware swap, I would install an SSD and not an HDD (why-so-slow link below), and would not configure the new SSD into a Fusion.

But if somebody here does choose to install another HDD and not an SSD: How to fix a split Fusion Drive - Apple Support (downside: if the SSD is what has failed here, that’s AFAIK not repairable this side of a mainboard replacement. Which is probably too much of an time-and-materials investment for a 2019 iMac.)

Follow caveats around properly handing electronic devices, including hazardous voltages and anti-static practices, etc.

Far and away the easiest replacement approach with the iMac is to install an external SSD, and migrate to and boot and run from that pending Mac replacement. (Article below.) When the iMac is eventually retired, the external SSD can be re-used. This in addition to whatever external HDD or NAS is used for backups.

• Why is my hard disk drive iMac so slow? - Apple Community
• Use an external SSD as your startup disk … - Apple Community

khlota wrote:

My husband is a Sr Network Engineer and believes he can change the hard drive for me. I’ve thought about getting an external drive for backups as well.

MrHoffman wrote:

Getting at the internal iMac hard drive for the repair is the hassle. There are tools available, and at least one third-party teardown and third-party service manual document. Search for fix it Mac Intel 21.5" Retina 4K Display 2019 or such.

👍

OWC also has a model-specific video so that OP's husband can learn what he would be getting into. So as not to run afoul of posting direct URLs to a "DIY" video that is likely to be removed, begin with:

1. OWC Install Videos, then choose
2. 27-inch iMac with Retina 5K (2019), and then
3. Hard Drive

Personally I would not replace the HDD with anything other than an OWC SSD — their Mercury Extreme Pro 6G

in particular. I have been using them for a ridiculous amount of time, and have had zero failures to date. Contrast that with cheap models described on this site, some of which don't last a year.

The thermal sensor, bracket, and tools are required. Tools are generic though.

You cut off the information for the SSD. I don't know if EtreCheck distinguishes between HD & SSD when reporting on drive errors....many apps do not.

It would not surprise me that the HD portion of the Fusion Drive is failing, but it would be nice to see the information regarding the SSD to be sure.

It will be much less expensive to use an external USB3 SSD for your boot drive. Performance may be a bit slower than that provided by the Fusion Drive, but it will still get about 400-500MB/s transfer rates or if you use a Thunderbolt3 SSD it may be faster than the Fusion Drive (up to 4GB/s, but depends on the SSD used & whether you are using both Thunderbolt ports). When you do finally retire this iMac, then you can take the external SSD & use it with your new Mac so the investment is not wasted.

Please run DriveDx (free trial period) and post the complete text report for both internal drives here using the "Additional Text" icon which looks like a piece of paper on the forum editing toolbar.

I'm especially interested in seeing the information for the SSD to better understand the 102% Used value being reported, but would also like to see the health report for the internal HD as well.

etresoft wrote:

HWTech wrote:

You cut off the information for the SSD. I don't know if EtreCheck distinguishes between HD & SSD when reporting on drive errors....many apps do not.

It does. This is a worst-case scenario report cut-off.

If the hard drive had the SMART failure, then the line right after "Internal SATA 6 Gigabit Serial ATA" would have been, "S.M.A.R.T. Status: Failed".

The first line that's cut off is probably "Internal PCI 8.0 GT/s x4 Serial ATA". The next line after that must be "S.M.A.R.T. Status: Failed".

Yeah, very unfortunate. I know that the SMART information has not been printed for the HDs with the other internal HD details like it does with the internal SSD (I realize macOS doesn't report it the same way).

I recall that you had a separate independent line for the SMART Status Verified/Failed somewhat apart from the drive reporting...I just didn't recall if that still exists since you added the health info line after the SSD info line....I think it still exists and seems confirmed by you reference later in the post.

I found an old Fusion Drive report and hacked it up to verify how it would appear with both the SSD and HDD.

Doesn't surprise me you would go into that much detail & testing for an app you developed from what I've seen you do when replying to posts here...mix of curiosity & being detail oriented. Lots of respect for you.

Also, there are two levels of SMART status checking going on. This is the older logic where it just checks the overall SMART status as reported by diskutil. The next line after "S.M.A.R.T. Status: Failed" would have been the SMART details line that I just recently added. This information only shows up for SSDs, which is the source of the failure in this unusual case.

Yes, I thought you still had the "diskutil" SMART status listed, but wasn't sure if that "SMART Status: Failed" would be a generic HD reference or if it would distinguish between HD & SSD (I thought the former) especially when the SSD information was cut off in the screenshot.

SMART failures are quite rare. How ironic that we finally get to see one in real life, with the new SMART details information, only the report gets cut off right at that exact point.

You will see them more often with an SSD than a HD. With a HD, the system usually becomes so slow that the drive will be replaced or the computer recycled long before the HD's SMART Status even nears a failing state. I've personally only ever seen two cases in 20 years where macOS showed a "SMART Status: Failed" for a HD, although I've replaced hundreds if not thousands of worn out & failing HDs that macOS reported the SMART Status as "Verified" indicating Ok.

While rare in an SSD, I have seen it a few times, but usually only when the SSD has PBs of writes. I'm actually still using a third party SSD at home which has a SMART Failure condition (not for excessive writes, it had a bad block it didn't take out of service so it maxed out the number of allowed errors....I reset the SSD to factory defaults which stopped it accumulating errors....it has enough other health attributes I can monitor to keep an eye on whether anything changes & of course good backups).

I know I mentioned to you about the "Media & Data Integrity Errors" in another post some time ago. I believe you mentioned there were two different items being reported by macOS & you didn't know which one to use. I think that would be a very useful tidbit of information to include since I have seen that occur more often than SSD SMART Status Failed. I can help you with figuring out which value to use if you want to create a thread in the Lounge. I actually have a laptop (Intel 2017 model) with those "Media & Data Integrity Errors" so I can run the command to retrieve the data to determine which value you need to pull. I just don't recall the specific format of the command to retrieve that data.

etresoft wrote:

I don't know what it means hardware-wise when an SSD hits that 102% usage.

It is my understanding that value is the percentage of expected drive life used. In human terms it means it doesn't owe you anything.

OP could probably run DriveDX which is likely to provide an overwhelming amount of additional data, but as a practical matter I don't use such things because there's nothing anyone can do about it. OP's initial screenshot makes it obvious enough that startup drive is already operating in a state of failure. Whether one chooses to characterize that as an advanced state of failure or just an ordinary moderately soon to be imminent critical state of failure or whatever has no practical value.

There is another reason I don't use DriveDX, which is that it gave a hard disk drive that was obviously worn out to the point of near-uselessness a clean bill of health. The disk was toast so I didn't care what it reported.

Having said that it's a stupendously well-written app that can provide an outrageous amount of information... for whatever that's worth.

Disk drive failure is still best diagnosed anecdotally. EtreCheck is as good as anything. If it says it's failing, it probably is. The only question is how bad, and the answer is "who cares".

HWTech wrote:

You cut off the information for the SSD. I don't know if EtreCheck distinguishes between HD & SSD when reporting on drive errors....many apps do not.

It does. This is a worst-case scenario report cut-off.

If the hard drive had the SMART failure, then the line right after "Internal SATA 6 Gigabit Serial ATA" would have been, "S.M.A.R.T. Status: Failed".

The first line that's cut off is probably "Internal PCI 8.0 GT/s x4 Serial ATA". The next line after that must be "S.M.A.R.T. Status: Failed".

I found an old Fusion Drive report and hacked it up to verify how it would appear with both the SSD and HDD.

Also, there are two levels of SMART status checking going on. This is the older logic where it just checks the overall SMART status as reported by diskutil. The next line after "S.M.A.R.T. Status: Failed" would have been the SMART details line that I just recently added. This information only shows up for SSDs, which is the source of the failure in this unusual case.

SMART failures are quite rare. How ironic that we finally get to see one in real life, with the new SMART details information, only the report gets cut off right at that exact point.

HWTech wrote:

Doesn't surprise me you would go into that much detail & testing for an app you developed from what I've seen you do when replying to posts here...mix of curiosity & being detail oriented. Lots of respect for you.

It's not that much work. An EtreCheck report is just a gzipped XML file. You can drag one right onto BBEdit and hack it up. Then run EtreCheck on the file to see what it says. Something that already has a node, like /etrecheck/disk/drive/smartstatus can simply be changed from "Verified" to "Failed". If you wanted to create new nodes, you'd have to know what EtreCheck's expecting. But one might be able to figure that out from the XSL inside the app bundle.

But I actually have no idea what it would say if it has failed. I know that I've only ever seen "Verified" and "Not supported". So EtreCheck will print out any value that's different. But I've never seen this in the wild.

You will see them more often with an SSD than a HD. With a HD, the system usually becomes so slow that the drive will be replaced or the computer recycled long before the HD's SMART Status even nears a failing state. I've personally only ever seen two cases in 20 years where macOS showed a "SMART Status: Failed" for a HD, although I've replaced hundreds if not thousands of worn out & failing HDs that macOS reported the SMART Status as "Verified" indicating Ok.

I've seen it once when I tried to install the OS on a drive I knew wasn't healthy. But it refused to even install, citing a SMART failure.

I know I mentioned to you about the "Media & Data Integrity Errors" in another post some time ago. I believe you mentioned there were two different items being reported by macOS & you didn't know which one to use. I think that would be a very useful tidbit of information to include since I have seen that occur more often than SSD SMART Status Failed. I can help you with figuring out which value to use if you want to create a thread in the Lounge. I actually have a laptop (Intel 2017 model) with those "Media & Data Integrity Errors" so I can run the command to retrieve the data to determine which value you need to pull. I just don't recall the specific format of the command to retrieve that data.

Just run the following command on any part of the disk:

diskutil info -plist /dev/disk0

Replacing "disk0" with the disk in question.

It will spit out a plist file (XML theoretically) with this confidence-inspiring entry: "SMARTDeviceSpecificKeysMayVaryNotGuaranteed".

This is what it should look like:

	<key>SMARTDeviceSpecificKeysMayVaryNotGuaranteed</key>
	<dict>
		<key>AVAILABLE_SPARE</key>
		<integer>100</integer>
		<key>AVAILABLE_SPARE_THRESHOLD</key>
		<integer>99</integer>
		<key>CONTROLLER_BUSY_TIME_0</key>
		<integer>0</integer>
		<key>CONTROLLER_BUSY_TIME_1</key>
		<integer>0</integer>
		<key>DATA_UNITS_READ_0</key>
		<integer>221061926</integer>
		<key>DATA_UNITS_READ_1</key>
		<integer>0</integer>
		<key>DATA_UNITS_WRITTEN_0</key>
		<integer>110775950</integer>
		<key>DATA_UNITS_WRITTEN_1</key>
		<integer>0</integer>
		<key>HOST_READ_COMMANDS_0</key>
		<integer>6129559319</integer>
		<key>HOST_READ_COMMANDS_1</key>
		<integer>0</integer>
		<key>HOST_WRITE_COMMANDS_0</key>
		<integer>1459523568</integer>
		<key>HOST_WRITE_COMMANDS_1</key>
		<integer>0</integer>
		<key>MEDIA_ERRORS_0</key>
		<integer>0</integer>
		<key>MEDIA_ERRORS_1</key>
		<integer>0</integer>
		<key>NUM_ERROR_INFO_LOG_ENTRIES_0</key>
		<integer>0</integer>
		<key>NUM_ERROR_INFO_LOG_ENTRIES_1</key>
		<integer>0</integer>
		<key>PERCENTAGE_USED</key>
		<integer>2</integer>
		<key>POWER_CYCLES_0</key>
		<integer>476</integer>
		<key>POWER_CYCLES_1</key>
		<integer>0</integer>
		<key>POWER_ON_HOURS_0</key>
		<integer>1403</integer>
		<key>POWER_ON_HOURS_1</key>
		<integer>0</integer>
		<key>TEMPERATURE</key>
		<integer>297</integer>
		<key>UNSAFE_SHUTDOWNS_0</key>
		<integer>10</integer>
		<key>UNSAFE_SHUTDOWNS_1</key>
		<integer>0</integer>
	</dict>

Some values have a _0 and a _1. The _1 values always seem to be zero.

etresoft wrote:

HWTech wrote:

I know I mentioned to you about the "Media & Data Integrity Errors" in another post some time ago. I believe you mentioned there were two different items being reported by macOS & you didn't know which one to use. I think that would be a very useful tidbit of information to include since I have seen that occur more often than SSD SMART Status Failed. I can help you with figuring out which value to use if you want to create a thread in the Lounge. I actually have a laptop (Intel 2017 model) with those "Media & Data Integrity Errors" so I can run the command to retrieve the data to determine which value you need to pull. I just don't recall the specific format of the command to retrieve that data.

Just run the following command on any part of the disk:

diskutil info -plist /dev/disk0

Replacing "disk0" with the disk in question.

It will spit out a plist file (XML theoretically) with this confidence-inspiring entry: "SMARTDeviceSpecificKeysMayVaryNotGuaranteed".

Ok, the following two values match up with the output of the "smartctl" open source utility:

diskutil info -plist /dev/disk0 smartctl -A /dev/disk0

MEDIA_ERRORS_0 "Media and Data Integrity Errors"

NUM_ERROR_INFO_LOG_ENTRIES_0 "Error Information Log Entries"

The "Error Information Log Entries" is a count of extended error log entries the drive has stored which can include more details about the errors. Any non-zero entry indicates something wrong is going on with the SSD. Whether or not those extended log details can be accessed varies between various Apple SSDs (I've only tried accessing them using Linux using the "nvme" utility...only rarely have been able to view them....the "diskutil" command did not provide those extended log entries). Apparently the extended log entries list only include details on the 64 most recent errors.

FYI, the Apple SSD from a MBPro 2017 I have here has over 100K values for both of the above keys, but SMART Status is still shown as "Verified". I imagine only the "Available Spare" & "Percentage Used" will trigger a SMART failure status (perhaps the "Data Units Written", "host_write_commands") when they reach a critical value.

etresoft wrote:

I've seen it once when I tried to install the OS on a drive I knew wasn't healthy. But it refused to even install, citing a SMART failure.

It does not surprise me especially with the recent versions of macOS since Apple includes system firmware updates as part of an OS upgrade. Apple doesn't want to run the firmware updater from a flaky drive.....and the installer seems to insist on using the internal drive to stage the system firmware update even when installing to an external drive.

Based on the tiny peek you posted, I would start by doing the test under "Before You Begin..." in this Apple support article:

How to fix a split Fusion Drive - Apple Support

Do not proceed, report is that the shows your drive is "split

"Fusion" consisted of a small SSD and a larger mechanical hard drive linked by software to act as onefaster drive. Potentially very useful, in practice we see a lot of issues with Fusion iMacs reported here.

Previously, a properly configured Fusion drive looked like the one in your partial report, but recently I have seen cases where the Fusion drive was "split" but did not show in the EtreCheck report.

As for EtreCheck, it is the best tool we have for diagnosing issues in this setting where we can neither see nor remotely access your computer, just not from an EtreCheck "snippet." Please post the entire report. What seems insignificant to a new Etrecheck user can hold answers for those of us who have reviewed thousands of those reports.  Etrecheck scrubs any personal info if you follow the posting steps in the posting article den.thed posted.

Get backups now today yesterday — ahead of all other steps, and before all diagnostics — if you don’t have current backups. Preserve (as much of) your data (as is still recoverable ) first and foremost.

khlota wrote:

Sorry, I didn't think the full report was necessary but I'm happy to share ;)

Thanks!

Apple doesn't like those personal sites, so the moderators might delete it. For those following at home, here is the interesting data:

disk1 - APPLE SSD SM0128L 121.33 GB (Solid State - TRIM: Yes)
Internal PCI-Express 8.0 GT/s x4 NVM Express
S.M.A.R.T. Status: Failing
  S.M.A.R.T. Details: 102% used, 168.76 TB written, 100% health, 124 unsafe shutdowns
  disk1s1 - EFI [EFI] 315 MB
  disk1s2 [APFS Fusion Drive] 120.88 GB
  disk2 [APFS Virtual drive] 2.12 TB (Shared by 6 volumes)
  disk2s1 (APFS) [APFS Container] (11.26 GB used)
  disk2s1s1 - Macintosh HD (APFS) [APFS Snapshot] (11.26 GB used)
  disk2s2 - Macintosh HD - Data (APFS) [APFS Virtual drive] (1.98 TB used)
  disk2s3 - Preboot (APFS) [APFS Preboot] (2.40 GB used)
  disk2s4 - Recovery (APFS) [Recovery] (1.33 GB used)
  disk2s5 - VM (APFS) [APFS VM] (2.15 GB used)
  disk2s6 - Update (APFS) (2 MB used)

102% used. Impressive. That would explain it.

I think the other relevant part is this:

Mounted Volumes:
disk2s1s1 - Macintosh HD [APFS Snapshot]
Filesystem: APFS
Mount point: /
Fusion drive
Read-only: Yes
Used: 11.26 GB
Shared values
Size: 2.12 TB
Free: 117.77 GB
Available: 120.52 GB

I think you've just beaten that computer to death.

However, aside from the scary "failing" notice, your report doesn't look bad. It's a little bit slow, but within the range of what I would expect for that computer.

I don't know what it means hardware-wise when an SSD hits that 102% usage. I know it can't be good. Does that mean it's at risk of imminent failure? HWTech would be a better person to answer that. My guess is that it's completely run out of extra allocation blocks and you're going to start getting corrupt data and/or crashes, which might not be recoverable.

Nobody ever did wrong by having too many backups. I don't have a lot of confidence in those online backups or NAS volumes. You've got a full 2 TB of data there. I recommend getting an external 4 TB and doing a Time Machine backup too. It's good to know you're going to need a backup beforehand.

khlota wrote:

My husband is a Sr Network Engineer and believes he can change the hard drive for me.

He may have the technical ability, but he might not be able to get the parts. Apple doesn't use off-the-shelf PC parts anymore.

And this is an unusual failure. At one time, there was a little cottage industry of parts and tools to swap out the mechanical portion of a Fusion Drive, because that's the component that's far more likely to fail. But I don't know about the SSD. And on the last generation iMac, the SSD might very well just be a chip on the logic board.

I’ve thought about getting an external drive for backups as well. Do you think that with Time Machine is better than an offsite service? I’ve actually never used Time Machine :)

I'm just not familiar with offsite backups, and I know how much of a hassle network backups can be. And you have a full 2 TB of data. That's going to take a long time even with a NAS. But trying to restore that much data over the internet is going to be a real challenge.

One critical fact can be determined from the limited information you posted — the lack of a Time Machine backup.

The symptoms you describe and the limited information indicate the startup disk has been operating in a state of failure. If you do not already have a backup, its data might as well be gone already.

HWTech wrote:

diskutil info -plist /dev/disk0 smartctl -A /dev/disk0

MEDIA_ERRORS_0 "Media and Data Integrity Errors"

NUM_ERROR_INFO_LOG_ENTRIES_0 "Error Information Log Entries"

The "Error Information Log Entries" is a count of extended error log entries the drive has stored which can include more details about the errors. Any non-zero entry indicates something wrong is going on with the SSD.

Thanks for the tip!

But what about MEDIA_ERRORS_1 and NUM_ERROR_INFO_LOG_ENTRIES_1?

All those _1 entries are a bit of a mystery.