Do I have to change the disk?

hannibal80 · Nov 8, 2011

About two weeks ago I added smartctl to my FreeBSD box.

This is my configuration:
HP Proliant Microserver
AMD Neo N36 dual core @ 1.3MHz
2x4GB DDR2 ECC Memory modules
3x1TB mixed disks in ZFS RaidZ
1x2TB disk in ZFS simple pool
1x250GB UFS system disk

The system is a file/media server with Serviio serving my Samsy TV and SMB to share files over the home network.

From the really first days I obtained some error reports from the smartd daemon.

Then I switched on the mail service and I started to collect statistics on it:
02 Nov

Code:

Device: /dev/ad10, 16 Offline uncorrectable sectors
Device: /dev/ad10, 17 Currently unreadable (pending) sectors

03 Nov

Code:

Device: /dev/ad10, 17 Offline uncorrectable sectors
Device: /dev/ad10, 17 Currently unreadable (pending) sectors

04 Nov

Code:

system off

05 Nov

Code:

system off

06 Nov

Code:

Device: /dev/ad10, 13 Offline uncorrectable sectors
Nothing about pending sectors

07 Nov

Code:

Device: /dev/ad10, 28 Offline uncorrectable sectors
Device: /dev/ad10, 29 Currently unreadable (pending) sectors

ZFS does not report any error.

The ad10 disk is one of the raidz pool.

Do I have to (quickly!) get a new disk?

Thanks a lot, bye.

Beastie · Nov 8, 2011

Disks don't always report SMART attributes correctly. So it could be a false positive. Or the disk may die within weeks, maybe days. Some day the partitions may become inaccessible and the next time you boot the system, the BIOS doesn't even see the disk...

You better be on the safe side and replace the disk. But whatever you do, make sure you have backups!!!

respite · Nov 8, 2011

Use [cmd=]smartctl -t long /dev/ad10[/cmd] to produce a full current report, take note of the time, and view after completion: [cmd=]smartctl -a /dev/ad10[/cmd]

The usual signs of a failing disk are the numeric values Raw_Read_Error_Rate, Reallocated_Sector_Ct, or Seek_Error_Rate rising at a fast rate.

hannibal80 · Nov 9, 2011

This was the ouput:

Code:

localhost# smartctl -a /dev/ad10
smartctl 5.40 2010-10-16 r3189 [FreeBSD 8.2-RELEASE amd64] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi Deskstar 7K1000.B
Device Model:     Hitachi HDT721010SLA360
Serial Number:    STF605MH1TKMMW
Firmware Version: ST6OA31B
User Capacity:    1,000,204,886,016 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Wed Nov  9 08:19:34 2011 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84)	Offline data collection activity
					was suspended by an interrupting command from host.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      ( 121)	The previous self-test completed having
					the read element of the test failed.
Total time to complete Offline 
data collection: 		 (14090) seconds.
Offline data collection
capabilities: 			 (0x5b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 235) minutes.
SCT capabilities: 	       (0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       1
  2 Throughput_Performance  0x0005   131   131   054    Pre-fail  Offline      -       116
  3 Spin_Up_Time            0x0007   119   119   024    Pre-fail  Always       -       480 (Average 482)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       332
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       100
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   125   125   020    Pre-fail  Offline      -       33
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       3156
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       332
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       349
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       349
194 Temperature_Celsius     0x0002   176   176   000    Old_age   Always       -       34 (Min/Max 20/47)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       149
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       38
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
ATA Error Count: 3
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 3 occurred at disk power-on lifetime: 3052 hours (127 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 88 48 2b 34 ec  Error: UNC 136 sectors at LBA = 0x0c342b48 = 204745544

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 00 d0 2a 34 ec 00      06:41:38.800  READ DMA
  c8 00 00 d0 2c 34 ec 00      06:41:38.800  READ DMA
  c6 00 10 00 00 00 e0 00      06:41:38.800  SET MULTIPLE MODE
  ef 02 00 00 00 00 e0 00      06:41:38.800  SET FEATURES [Enable write cache]
  ef aa 00 00 00 00 e0 00      06:41:38.800  SET FEATURES [Enable read look-ahead]

Error 2 occurred at disk power-on lifetime: 3052 hours (127 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 d0 2b 34 ec  Error: UNC at LBA = 0x0c342bd0 = 204745680

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 00 d0 2b 34 ec 00      06:41:24.100  READ DMA
  c6 00 10 00 00 00 e0 00      06:41:24.100  SET MULTIPLE MODE
  ef 02 00 00 00 00 e0 00      06:41:24.100  SET FEATURES [Enable write cache]
  ef aa 00 00 00 00 e0 00      06:41:24.100  SET FEATURES [Enable read look-ahead]
  ef 03 45 00 00 00 e0 00      06:41:24.100  SET FEATURES [Set transfer mode]

Error 1 occurred at disk power-on lifetime: 3052 hours (127 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 fe d2 29 34 ec  Error: UNC 254 sectors at LBA = 0x0c3429d2 = 204745170

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 00 d0 29 34 ec 00      06:41:09.500  READ DMA
  c8 00 00 d0 28 34 ec 00      06:41:09.500  READ DMA
  c8 00 00 d0 27 34 ec 00      06:41:09.500  READ DMA
  c8 00 00 d0 26 34 ec 00      06:41:09.400  READ DMA
  c8 00 00 d0 25 34 ec 00      06:41:09.400  READ DMA

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%      3147         204742092
# 2  Short offline       Completed without error       00%      3080         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

So I had:

Code:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       1
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       100
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0

What do you think about it?

Thank you!

jb_fvwm2 · Nov 9, 2011

The reallocated sector count indicates that the disk should be replaced, how soon is up to you but I'd do it immediately.

kpa · Nov 9, 2011

Code:

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%      3147         204742092

That is a sure sign of imminent failure, get a new one fast.

olav · Nov 9, 2011

Yes, its absolutely time to get a new disk. You probably have warranty anyway

hannibal80 · Nov 10, 2011

Ok guys, thanks a lot.

It could not have happened at a worse time, with disks at stellar prices... :\

xibo · Nov 10, 2011

Mine's better then that

Code:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   093   093   016    Pre-fail  Always       -       1835014
  2 Throughput_Performance  0x0005   133   133   054    Pre-fail  Offline      -       90
  3 Spin_Up_Time            0x0007   133   133   024    Pre-fail  Always       -       437 (Average 427)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       159
  5 Reallocated_Sector_Ct   0x0033   098   098   005    Pre-fail  Always       -       195
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0

When replacing the disk these days, take a workstation/enterprise/server/whatever-they-call it disk, as price difference is low due to the exploded price levels, and their reliability is 10-100 times(depending on vendor) better.