Solved HDD repair?

byrnejb

byrnejb

I have pulled a WDC WD30EFRX-68AX9N0 2Tb drive from a zfs pool and replaced it with a spare drive. The pulled drive was throwing read errors localized to LBA 843538014, 843538015, and 843538848. Before I dispose of this drive I would like to know if there is any reasonable way of 'sparing' these LBAs and repairing the drive for further use. Is there?
 
there may be a way to "remap" failing sectors/blocks. Not sure, it's been a really long time but maybe google "wdc remap failing sectors" may give a clue.
Of course, you could remap these but then have more fail.
I know with SSDs a recommendation is to use "10% less" meaning if it's a 1TB drive partition to use 0.9*1TB because the firmware on the SSD can use that 10% to remap failing stuff.
 
mer said:
I know with SSDs a recommendation is to use "10% less" meaning if it's a 1TB drive partition to use 0.9*1TB because the firmware on the SSD can use that 10% to remap failing stuff.
Click to expand...
I'm not certain about this but isn't that more of an early-SSD-days thing? I was under the impression that these days SSDs already ship with a higher capacity than advertised and handle spares that way.
At least SMART data of my SSDs appears to indicate that it carries/holds/uses spares without me having actually provided any room for those myself (using 100% of the advertised capacity).
Hopefully somebody more knowledgeable than me in this topic can enlighten us.
 
  • Like
Reactions: mer
jbo you could be correct, I really don't know because I haven't done a lot of analysis. Even so, I've simply found it easier to just partition a little less than full capacity.
 
byrnejb said:
if there is any reasonable way of 'sparing' these LBAs and repairing the drive for further use.
Click to expand...
Repearing, as in being able to reuse those blocks with no read errors? No. Modern hard disks already have a 'spare' bit of space to remap those 'bad' blocks (check 197/198 of SMART). The firmware of the drive takes care of this. Up to the point when this 'spare' bit of space is all used up. Then bad blocks are just there. Nothing you, as consumer, could do to fix those.

Back in 'ye olde days' drives didn't have this. The trick was to create a fake file, usually called 'bad.blocks', and link all the bad blocks to this file. As long as you never try to actually read that file, or overwrite it, you could use the rest of the disk. You ran a regular scan program that would automatically attach any new bad blocks to this 'bad.blocks' file.

Could you still use this drive? Probably. Maybe. Maybe not. I wouldn't use it to save anything worthwhile on it. Bad blocks often multiply. That said,
Code: 
May 25 22:27:33 molly smartd[1526]: Device: /dev/da3 [SAT], 192 Currently unreadable (pending) sectors
May 25 22:27:33 molly smartd[1526]: Device: /dev/da3 [SAT], 192 Offline uncorrectable sectors
This drive has been giving me these 192 'offline uncorrectable' sectors for the past year or so, they haven't increased for some time. I've been too lazy to order a new drive and actually replace it. The rest of the drive still works fine. But it could suddenly have a lot more of those 'uncorrectable' sectors. Or just die on me completely.
 
byrnejb said:
I have pulled a WDC WD30EFRX-68AX9N0 2Tb drive from a zfs pool and replaced it with a spare drive. The pulled drive was throwing read errors localized to LBA 843538014, 843538015, and 843538848. Before I dispose of this drive I would like to know if there is any reasonable way of 'sparing' these LBAs and repairing the drive for further use. Is there?
Click to expand...
If you also meant in terms of technical repair: not doable at home. There are professional data recovery companies around, and they've got high standard clean rooms for those repairs for very good reason.
 
What cracauer@ said. If the block can't be read by the firmware, it can't be remapped. When you scribble to every block with dd, the firmware gets a chance to remap the bad blocks (because the firmware now knows that the previous contents may be discarded).

However, the TrueNAS Hard Drive Troubleshooting Guide says that if any of the SMART IDs 5 (Reallocated_Sector_Ct), 197 (Current_Pending_Sector), and 198 (Offline_Uncorrectable) are greater than zero then the drive has failed for RMA purposes. So claim your warranty if you can.

WD are now selling direct. 2TB CMR red drives are US$80 with free shipping (but I'm not sure if they ship to Canada).
 
cracauer@ said:
Simply writing with dd all over the drive should do the trick. Unreadable sectors are remapped when written to.
Click to expand...
Yes. BUT: If the disk is sick enough that a handful sectors already had unrecoverable failures, it is likely that more things will go wrong. Personally, I would not use that disk for any data I need. Nor on a system where intermittent crashes and weird behavior would annoy me. As a "known unreliable scratch disk", or a user who has lots of patience? Sure, go have fun with it.
 
I replaced the failing drive with a new spare successfully and it was re-silvered. Everything in the pool is now reported as clean. I also wrote over the failing drive with pv < /dev/zero > /dev/da1. When I rerun a long test with smartctl the drive reports no errors:
Code: 
[root@vhost03 ~ (master)]# /usr/local/sbin/smartctl -a /dev/da1 
smartctl 7.3 2022-02-28 r5338 [FreeBSD 13.1-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68AX9N0
Serial Number:    WD-WCC1T0371529
LU WWN Device Id: 5 0014ee 208200201
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database 7.3/5319
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon May 29 17:22:08 2023 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:         (39600) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 398) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x70bd)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1511
  3 Spin_Up_Time            0x0027   179   179   021    Pre-fail  Always       -       6008
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       62
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   049   049   000    Old_age   Always       -       37816
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       60
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       51
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       10
194 Temperature_Celsius     0x0022   123   110   000    Old_age   Always       -       27
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     37815         -
# 2  Extended offline    Completed: read failure       90%     37634         843538014
# 3  Extended offline    Completed: read failure       90%     37466         843538015
# 4  Extended offline    Completed: read failure       90%     37299         843538848
# 5  Extended offline    Completed: read failure       90%     37275         843538014
# 6  Extended offline    Completed: read failure       90%     37131         843538015
# 7  Extended offline    Completed: read failure       90%     36963         843538014
# 8  Extended offline    Completed: read failure       90%     36795         843538015
# 9  Extended offline    Completed: read failure       90%     36532         843538014
#10  Extended offline    Completed: read failure       90%     36364         843538015
#11  Extended offline    Completed: read failure       90%     36196         843538015
#12  Extended offline    Completed: read failure       90%     36028         843538848
#13  Extended offline    Completed: read failure       90%     35860         843538014
#14  Extended offline    Completed: read failure       90%     35692         843538015
#15  Extended offline    Completed: read failure       90%     35620         843538848
#16  Extended offline    Completed: read failure       90%     35524         843538848
#17  Extended offline    Completed: read failure       90%     35356         843538848
#18  Extended offline    Completed: read failure       90%     35189         843538015
#19  Extended offline    Completed: read failure       90%     35021         843538015
#20  Extended offline    Completed: read failure       90%     34901         843538848
#21  Extended offline    Completed: read failure       90%     34853         843538848
20 of 20 failed self-tests are outdated by newer successful extended offline self-test # 1

I have no current plans for this drive and it is labeled as failing. But it seems that a rewrite to the entire disc did fix the read problems.
 
Why is "reallocated sector count" still zero?
Sometimes I can't make head or tails (fails?) out of what vendors do with their SMART data.
 
I agree on SATA1 and SATA2 drives I understood. But by SATA3 you think they would be getting it right.
You think its on purpose? They don't want you to know...
Maybe the reallocated sector won't show up until the next drive health report.
 
cracauer@ said:
Why is "reallocated sector count" still zero?
Sometimes I can't make head or tails (fails?) out of what vendors do with their SMART data.
Click to expand...
The firmware can re-allocate a sector that:
  1. can be read, but has to be corrected with ECC; or
  2. that can't be written successfully.
because, in both cases, it has a copy of the "correct" data in a buffer.

But, once written successfully, the firmware must not re-allocate a sector that can't be read. It has no "correct" data to re-allocate. This preserves the "loud" corruption, which is the best available option. Any other approach would cause silent corruption. It's also why (repeatedly) writing fresh data on each sector of disk with dd(1) can bring it back to life (rule 2 is very likely to be brought into play).

So this disk has some sectors that were written successfully, and subsequently went completely unreadable.

My experience of the old 3TB CMR WD Reds [WD30EFRX] is that they run slow, run cool, are quite reliable, and play well with SMART.
 
I actually suspect that this SMART data is for the wrong drive: A drive that has had permanent read errors at certain LBAs should be showing non-zero statistics in SMART.

cracauer@ said:
Why is "reallocated sector count" still zero?
Click to expand...
Even worse: The reallocated event count also has dropped to zero. That one is supposed to count how many times a reallocate was performed, not how many sectors are currently reallocated. It should never decrease. In contrast, the pending sector count may decrease when pending sectors (those currently unreadable) stop being unreadable, by virtue of being overwritten. In theory, the reallocated sector count (parameter 5) is also not supposed to decrease.
Edit: See below, if these were read errors and were overwritten before being reallocated, the count should be zero.

Sometimes I can't make head or tails (fails?) out of what vendors do with their SMART data.
Click to expand...
That problem is TERRIBLE with the implementation in SATA; while the standard defines the rough protocol, the exact meaning of each parameter (and their numbering) is left up to the implementor. That's because that standard is ancient, I think from the 90s. I think Seagate follows the standard somewhat reasonably: not because they're a great company or well-engineered drives, but because they helped write the standard (together with Compaq and IBM); the other disk drives companies that were involved back then have all ceased to exist (Quantum, Imprimis, Maxtor, Conner, ...). I don't know how much of the corporate memory has been lost in the IBM -> Hitachi -> WD transitions (my old IBM friends are now nearly all retired).

The situation in SCSI is somewhat better, since there the meaning of the fields is defined in the SPC standard; I think the SCSI committee learned from the mistakes of the ones who went before it. The thing the SCSI standard is missing is how PFA (predictive failure analysis) results are to be transmitted, so there you once again have to be vendor- and model-specific. For real-world usage in production systems, you pretty much need to contact the manufacturer, get the SMART (or SCSI) manual for each model you intend to use, and implement model-specific stuff.
 
ralphbsz said:
I actually suspect that this SMART data is for the wrong drive: A drive that has had permanent read errors at certain LBAs should be showing non-zero statistics in SMART.
Click to expand...
Attribute 1 is reported differently by different vendors, but this WD disk is showing a clear problem:
Code: 
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1511
ralphbsz said:
Even worse: The reallocated event count also has dropped to zero. That one is supposed to count how many times a reallocate was performed, not how many sectors are currently reallocated. It should never decrease.
Click to expand...
The reallocated sector count is reported as:
Code: 
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
These are the exact same numbers I would expect to see on a brand new WD drive. I can't see any evidence that the reallocated event count has decreased.

I'm happy to be corrected, but I still interpret this drive as having unreadable sectors that can't be reallocated because they can't be read.
 
Drives don't report a sector that could not previously be read as reallocated if writing to it worked successfully. An example is I had a drive in a case where as it got hotter due to additional hardware+inadequate cooling it would end up with unreadable sectors as the data became corrupted. I couldn't read that data again from the impacted sectors but a write to the sectors would be successful and therefore they did not get reallocated. If a write failed on a sector then that would have caused the reallocation process to happen and increment the counter. Pending sectors can decrease once problematic sectors are dealt with whether its from succeeding or reallocating. Watching for '...error rate' values for sudden unusual rates of change can indicate a problem when the drive is otherwise under the same load. Having a nonzero value and a changing value does not necessarily mean anything is considered wrong with the drive and it is not a count of sectors with issues. When udma_crc_error_count increments, you are counting communication errors that can be the drive, but can also be things like bad/dirty connections, communicatoin got too noisy, power instability, etc. Sometimes SMART data of the other values are externally influenced as the drive can also be externally influenced. Watching for general performance anomalies on a drive can indicate failures are near while SMART gives a perfect thumbs up for a drive. Extended offline won't report issues that it doesn't detect with a drive, and a drive that is running slow but successfully with a dependency on its error correction can still pass albeit much slower than expected. SSDs + SMART are an even stranger mess and NVMe hasn't been very friendly about having useful measurements in my experience (which is limited compared to SATA drives).
SSDs that I have worked with have had their own reserve of sectors to use for remappping failed sectors (think we should be saying 'blocks' at this point). The need to leave unallocated space on a drive is so that it can have a quantity of usable blocks for wear leveling. An operating system that provides TRIM instructions to mark previously used blocks as no longer in use just gave any decent drive firmware information about additional blocks to have as a pool for wear leveling and/or future writes; you may be able to get away with not allocating unused space if you don't fill your drive with data.
 
You must log in or register to reply here.
Back
Top