Hi, over the last couple of days my FreeBSD 10.1 box is starting to give some interesting errors.
A sample of these are -
Originally I suspected a disk going down, as the parity errors are all on ada0, but unless I'm misunderstanding the output, the command timeout errors are more widespread.
I ran a scrub which found and corrected lots of errors on ada0, which again points to the disk, but I when I look at smartctl(8) for /dev/ada0 I can't really see any issues.
I've got another long test running now, but I would appreciate any ideas as to whether it's the disk at issue, or the controller, or whether I should be looking at things like cabling (although the case hasn't been opened in a couple of years)
Thanks in advance
A sample of these are -
Code:
Jul 9 13:38:39 HAYLESNAS kernel: (aprobe2:ahcich4:0:0:0): CAM status: Command timeout
Jul 9 13:39:09 HAYLESNAS kernel: (aprobe0:ahcich3:0:0:0): CAM status: Command timeout
Jul 9 13:39:09 HAYLESNAS kernel: ahcich4: (aprobe1:ahcich2:0:0:0): CAM status: Command timeout
Jul 9 13:39:09 HAYLESNAS kernel: (aprobe2:ahcich4:0:0:0): CAM status: Command timeout
Jul 9 13:39:40 HAYLESNAS kernel: ahcich2: (aprobe0:ahcich3:0:0:0): CAM status: Command timeout
Jul 9 13:39:40 HAYLESNAS kernel: ahcich4: (aprobe1:ahcich2:0:0:0): CAM status: Command timeout
Jul 9 13:39:40 HAYLESNAS kernel: (aprobe2:ahcich4:0:0:0): CAM status: Command timeout
Jul 9 13:40:10 HAYLESNAS kernel: ahcich3: (aprobe1:ahcich2:0:0:0): CAM status: Command timeout
Jul 9 13:40:10 HAYLESNAS kernel: ahcich4: (aprobe0:ahcich3:0:0:0): CAM status: Command timeout
Jul 9 13:40:10 HAYLESNAS kernel: (aprobe2:ahcich4:0:0:0): CAM status: Command timeout
Jul 9 13:54:55 HAYLESNAS kernel: (ada0:ahcich0:0:0:0): CAM status: Uncorrectable parity/CRC error
Jul 9 13:54:55 HAYLESNAS kernel: (ada0:ahcich0:0:0:0): CAM status: Uncorrectable parity/CRC error
Jul 9 14:09:04 HAYLESNAS kernel: (ada0:ahcich0:0:0:0): CAM status: Uncorrectable parity/CRC error
Jul 9 14:09:04 HAYLESNAS kernel: (ada0:ahcich0:0:0:0): CAM status: Uncorrectable parity/CRC error
Jul 9 14:14:56 HAYLESNAS kernel: (ada0:ahcich0:0:0:0): CAM status: Uncorrectable parity/CRC error
Jul 9 14:14:56 HAYLESNAS kernel: (ada0:ahcich0:0:0:0): CAM status: Uncorrectable parity/CRC error
Jul 9 14:23:27 HAYLESNAS kernel: (ada0:ahcich0:0:0:0): CAM status: Uncorrectable parity/CRC error
Jul 9 14:23:27 HAYLESNAS kernel: (ada0:ahcich0:0:0:0): CAM status: Uncorrectable parity/CRC error
Jul 9 15:48:22 HAYLESNAS kernel: (ada0:ahcich0:0:0:0): CAM status: Uncorrectable parity/CRC error
Jul 9 15:48:22 HAYLESNAS kernel: (ada0:ahcich0:0:0:0): CAM status: Uncorrectable parity/CRC error
Jul 9 16:41:19 HAYLESNAS kernel: (ada0:ahcich0:0:0:0): CAM status: Uncorrectable parity/CRC error
I ran a scrub which found and corrected lots of errors on ada0, which again points to the disk, but I when I look at smartctl(8) for /dev/ada0 I can't really see any issues.
HAYLESNAS# smartctl -a /dev/ada0
Code:
smartctl 6.2 2013-07-26 r3841 [FreeBSD 10.1-RELEASE-p10 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda Green (AF)
Device Model: ST2000DL003-9VT166
Serial Number: 5YD14P19
LU WWN Device Id: 5 000c50 02f1bfc53
Firmware Version: CC32
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 5900 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Jul 9 22:41:50 2015 NZST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 249) Self-test routine in progress...
90% of test remaining.
Total time to complete Offline
data collection: ( 623) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 341) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x30b7) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 115 099 006 Pre-fail Always - 89899776
3 Spin_Up_Time 0x0003 094 092 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 53
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 077 060 030 Pre-fail Always - 52029794
9 Power_On_Hours 0x0032 063 063 000 Old_age Always - 32844
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 52
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 099 099 000 Old_age Always - 1
188 Command_Timeout 0x0032 100 001 000 Old_age Always - 1043
189 High_Fly_Writes 0x003a 099 099 000 Old_age Always - 1
190 Airflow_Temperature_Cel 0x0022 049 045 045 Old_age Always In_the_past 51 (Min/Max 49/53)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 47
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 53
194 Temperature_Celsius 0x0022 051 055 000 Old_age Always - 51 (0 17 0 0 0)
195 Hardware_ECC_Recovered 0x001a 028 003 000 Old_age Always - 89899776
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 139 000 Old_age Always - 2116
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 125937031086155
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 1665586290
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 4252201402
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Self-test routine in progress 90% 32844 -
# 2 Short offline Completed without error 00% 32843 -
# 3 Short offline Completed without error 00% 32821 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Thanks in advance