3ware array degraded

dvl@ · Jan 8, 2013

Hello,

I have this on FreeBSD 8.2-stable with a 3Ware 7000 series controller.

Code:

$ sudo tw_cli
Password:
//ngaio> /c0 show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    DEGRADED       -       -       -       149.05    W      -      

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     149.05 GB   312581808     WD-WCAL97852740     
p1     DEGRADED         u0     149.05 GB   312581808     WD-WCAL97759170     
p2     NOT-PRESENT      -      -           -             -
p3     NOT-PRESENT      -      -           -             -

//ngaio> /c0/u0 show

Unit     UnitType  Status         %RCmpl  %V/I/M  Port  Stripe  Size(GB)
------------------------------------------------------------------------
u0       RAID-1    DEGRADED       -       -       -     -       149.05    
u0-0     DISK      OK             -       -       p0    -       149.05    
u0-1     DISK      DEGRADED       -       -       p1    -       149.05

What's the proper way to rebuild this?

Amongst other things, I've tried:

Code:

//ngaio> maint rebuild c0 u0 p1
Sending rebuild start request to /c0/u0 on 1 disk(s) [1] ... Failed.
(0x0B:0x0035): Replacement drive configuration is invalid for rebuild operation

//ngaio>

Confused...

pboehmer · Jan 8, 2013

I have 3ware 8000 series controllers and the tw_cli port installed, but have never had to actually use it *knock* *knock*. Looking at the documentation, unless you have a 9000 series controller, it looks like you are severely limited on what you can do from the CLI. You could try issuing:

Code:

focus /c0/p1
remove

and physically remove the drive. See if the drive in P1 is no longer present. Put in the replacement drive and then issue:

Code:

focus /c0
rescan

See if the new drive is detected. If you can see the drive and "show status" shows rebuilding, then congrats. If you cannot see the drive, then you are probably going to have to reboot the system for the controller to see the new drive. If you can see the new drive and the state is still degraded, try:

Code:

focus /c0
set rebuild=3

Where 3 is balanced IO/rebuild. See man page for more info. If "show status" still shows no signs of rebuilding, you will probably have to reboot the box and rebuild through the controller's BIOS.

pboehmer · Jan 8, 2013

As a side note, the man page does not mention the command "maint", but it looks like a valid command in tw_cli. Now I'm just as confused as you are

I did a Google search and came up with this link:
http://idolinux.blogspot.com/2008/08/replace-bad-disk-on-3ware-raid.html

dvl@ · Jan 8, 2013

Thanks. I'll try some of those tonight. I have tried a rescan before, but the system froze. Ugh.

I may just put in a new hdd. This system is at least 6 years old and I don't know how long those 160GB PATA drives have been in there.

FYI, here's the smartctl output for the 'dead' hdd:

Code:

 sudo /usr/local/sbin/smartctl -a -i -d 3ware,1 /dev/twe0 /dev/twe0
Password:
Sorry, try again.
Password:
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.2-STABLE i386] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar SE
Device Model:     WDC WD1600JB-00GVC0
Serial Number:    WD-WCAL97759170
Firmware Version: 08.02D08
User Capacity:    160,041,885,696 bytes [160 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   6
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Tue Jan  8 18:51:41 2013 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                ( 4623) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        No General Purpose Logging support.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (  62) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x003f) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0007   163   160   021    Pre-fail  Always       -       4375
  4 Start_Stop_Count        0x0032   100   100   040    Old_age   Always       -       78
  5 Reallocated_Sector_Ct   0x0033   198   198   140    Pre-fail  Always       -       21
  7 Seek_Error_Rate         0x000b   200   200   051    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   031   031   000    Old_age   Always       -       50638
 10 Spin_Retry_Count        0x0013   100   253   051    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x0013   100   253   051    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       78
194 Temperature_Celsius     0x0022   030   030   000    Old_age   Always       -       120
196 Reallocated_Event_Count 0x0032   196   196   000    Old_age   Always       -       4
197 Current_Pending_Sector  0x0012   200   200   000    Old_age   Always       -       1
198 Offline_Uncorrectable   0x0012   200   200   000    Old_age   Always       -       2
199 UDMA_CRC_Error_Count    0x000a   200   253   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0009   200   200   051    Pre-fail  Offline      -       1

SMART Error Log Version: 1
ATA Error Count: 4
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 4 occurred at disk power-on lifetime: 50621 hours (2109 days + 5 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  01 51 04 00 00 00 e0  Error: AMNF 4 sectors at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 04 00 00 00 00 00      01:14:07.050  READ DMA EXT
  25 00 04 00 00 00 00 00      01:14:07.050  READ DMA EXT
  ef 03 45 00 00 00 00 00      01:14:07.050  SET FEATURES [Set transfer mode]
  e7 00 00 00 00 00 00 00      01:14:07.050  FLUSH CACHE
  b0 d8 00 00 4f c2 00 00      01:14:07.050  SMART ENABLE OPERATIONS

Error 3 occurred at disk power-on lifetime: 50621 hours (2109 days + 5 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  01 51 04 00 00 00 e0  Error: AMNF 4 sectors at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 04 00 00 00 00 00      01:14:04.650  READ DMA EXT
  ef 03 45 00 00 00 00 00      01:14:04.650  SET FEATURES [Set transfer mode]
  e7 00 00 00 00 00 00 00      01:14:04.650  FLUSH CACHE
  b0 d8 00 00 4f c2 00 00      01:14:04.650  SMART ENABLE OPERATIONS
  ec 00 01 00 00 00 00 00      01:14:04.650  IDENTIFY DEVICE

Error 2 occurred at disk power-on lifetime: 50621 hours (2109 days + 5 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  01 51 04 00 00 00 e0  Error: AMNF 4 sectors at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 04 00 00 00 00 00      01:12:42.800  READ DMA EXT
  25 00 04 00 00 00 00 00      01:12:42.800  READ DMA EXT
  ef 03 45 00 00 00 00 00      01:12:42.800  SET FEATURES [Set transfer mode]
  e7 00 00 00 00 00 00 00      01:12:42.800  FLUSH CACHE
  b0 d8 00 00 4f c2 00 00      01:12:42.800  SMART ENABLE OPERATIONS

Error 1 occurred at disk power-on lifetime: 50621 hours (2109 days + 5 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  01 51 04 00 00 00 e0  Error: AMNF 4 sectors at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 04 00 00 00 00 00      01:12:40.400  READ DMA EXT
  ef 03 45 00 00 00 00 00      01:12:40.400  SET FEATURES [Set transfer mode]
  e7 00 00 00 00 00 00 00      01:12:40.400  FLUSH CACHE
  b0 d8 00 00 4f c2 00 00      01:12:40.400  SMART ENABLE OPERATIONS
  ec 00 01 00 00 00 00 00      01:12:40.400  IDENTIFY DEVICE

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     50365         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

jb_fvwm2 · Jan 8, 2013

Reallocated Sector Count ... last number greater than zero. Once or twice I've used the freezer trick to get the last data off of it, no raid involved though.

phoenix · Jan 8, 2013

Can you install the 3dm2 port and access the web GUI of the controller? I've always found the web GUI to be much easier/nicer to work with than tw_cli.

Granted, I've only used 1 7000-series controller back in the day, so it may not be supported. All our current 3Ware controllers are 9550, 9650, or 9750 variants.

User23 · Jan 9, 2013

dvl@ said:
Thanks. I'll try some of those tonight. I have tried a rescan before, but the system froze. Ugh.

I had the same problems with this old PATA type 3ware controllers. You need to shutdown the system to switch the drive safely.

Shutdown, remove the old degraded drive and put in a "new" one. After reboot enter the controller bios and rebuild the degraded unit0 with the new drive on p1. After pressing F8 to save this setting, reboot and after some minutes uptime /var/log/messages should show something like kernel: twe0: INFO: rebuild unit0 started

Or you use the the tw_cli or 3dm2 webinterface to remove the drive, shut down, replace with a new drive, and use tw_cli or 3dm2 to start the rebuild process. But the controller wont care about removing by software or just physically removing the drive while the system is off.

dvl@ · Jan 9, 2013

Freddie: the 3dm2 port causes the system to reboot.

I found that out a few weeks ago.

User23: I plan to add a new PATA drive soon, perhaps tonight.

User23 · Jan 10, 2013

dvl@ said:
Freddie: the 3dm2 port causes the system to reboot. I found that out a few weeks ago.

Maybe the 3dm2 did a rescan. It sounds cruel. But you have a backup anyway, right?

dvl@ · Jan 10, 2013

I have plenty of backups...

In fact, that is why I have not yet replaced the dud HDD. Backups were running. I will try replacing the HDD tonight.

dvl@ · Jan 11, 2013

I pulled the OLD HDD, and put in the new HDD. But, I failed. The replacement unit was 80GB. It needed to be at least 160GB. The box is still running; degraded.

dvl@ · Jan 11, 2013

FYI, when I opened the case, I found I had stuck labels to the end of each HDD. On the label was the model and serial number.

It made replacing the bad HDD straight forward.