ZFS zpool replace problem

M

Momo

Hi,

I have to maintain a backup server using FreeBSD 11.1.
I don't have all the informations because the guy who installed the server
has left for two month and won't be there before next year.

The problem : the server is connected to a DELL MD disk array of 60 disks.

The zpool
Code: 
  pool: md04-tank
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
        repaired.
  scan: scrub repaired 96K in 158h2m with 0 errors on Mon Jul  8 06:48:04 2019
config:

        NAME                     STATE     READ WRITE CKSUM
        md04-tank                DEGRADED     0     0     0
          raidz2-0               ONLINE       0     0     0
            da8                  ONLINE       0     0     0
            da0                  ONLINE       0     0     0
            da5                  ONLINE       0     0     0
            da1                  ONLINE       0     0     0
            da9                  ONLINE       0     0     0
            da10                 ONLINE       0     0     0
            da2                  ONLINE       0     0     0
            da3                  ONLINE       0     0     0
            da11                 ONLINE       0     0     0
            da4                  ONLINE       0     0     0
          raidz2-1               ONLINE       0     0     0
            da7                  ONLINE       0     0     0
            da6                  ONLINE       0     0     0
            da12                 ONLINE       0     0     0
            da15                 ONLINE       0     0     0
            da13                 ONLINE       0     0     0
            da14                 ONLINE       0     0     0
            da23                 ONLINE       0     0     0
            da19                 ONLINE       0     0     0
            da22                 ONLINE       0     0     0
            da17                 ONLINE       0     0     0
          raidz2-2               DEGRADED     0     0     0
            da18                 ONLINE       0     0     0
            da20                 ONLINE       0     0     0
            da21                 ONLINE       0     0     0
            da16                 ONLINE       0     0     0
            da25                 ONLINE       0     0     0
            da34                 ONLINE       0     0     0
            da32                 ONLINE       0     0     0
            da27                 ONLINE       0     0     0
            da24                 ONLINE       0     0     0
            9189834293223397025  OFFLINE      0    26     1  was /dev/da35
          raidz2-3               ONLINE       0     0     0
            da28                 ONLINE       0     0     0
            da26                 ONLINE       0     0     0
            da29                 ONLINE       0     0     0
            da30                 ONLINE       0     0     0
            da31                 ONLINE       0     0     0
            da33                 ONLINE       0     0     0
            da36                 ONLINE       0     0     0
            da45                 ONLINE       0     0     0
            da46                 ONLINE       0     0     0
            da44                 ONLINE       0     0     0
          raidz2-4               ONLINE       0     0     0
            da38                 ONLINE       0     0     0
            da37                 ONLINE       0     0     0
            da40                 ONLINE       0     0     0
            da43                 ONLINE       0     0     0
            da39                 ONLINE       0     0     0
            da47                 ONLINE       0     0     0
            da42                 ONLINE       0     0     0
            da41                 ONLINE       0     0     0
            da51                 ONLINE       0     0     0
            da48                 ONLINE       0     0     0
          raidz2-5               DEGRADED     0     0     0
            da49                 ONLINE       0     0     0
            da50                 ONLINE       0     0     0
            da59                 ONLINE       0     0     0
            da52                 FAULTED      0   180     0  too many errors
            da55                 ONLINE       0     0     0
            da53                 ONLINE       0     0     0
            6680052414426837873  OFFLINE      1     1     0  was /dev/da58
            da56                 ONLINE       0     0     0
            da54                 ONLINE       0     0     0
            da57                 ONLINE       0     0     0

errors: No known data errors

3 disks show errors : da35, da52 and da58.
da35 and da58 were removed by the system. da52 is still alive but with lots of errors.
I contacted support and they sent 3 disks.

Is it safe to replace the disks without stopping the server ?
I usually do it but on hardware RAID not software.
So I replaced the two failed disks, thinking I just had to do a zpool replace.
But it doesn't work.
I have read a lot of docs and everywhere I saw command like this
zpool replace tank old-disk new-disk

Here is the answer :
Code: 
zpool replace md04-tank 9189834293223397025 da35
cannot replace 9189834293223397025 with da35: new device must be a single disk

I really don't understand the error message.

What would be a normal procedure for replacing disks ?
Did I have to shutdown the server ?
Where is/are the mistake(s) ?

Please help.
 
Momo said:
So I replaced the two failed disks, thinking I just had to do a zpool replace.
But it doesn't work.
I have read a lot of docs and everywhere I saw command like this
zpool replace tank old-disk new-disk

Here is the answer :
Code: 
zpool replace md04-tank 9189834293223397025 da35
cannot replace 9189834293223397025 with da35: new device must be a single disk
Click to expand...
Just hypothesis - it says da35 should be a single disk. Do you have a partition table on that?
 
No partition table.
But the other disks in the zpool don't have one.
I also have a lot of SCSI errors each time I try to access one of the physically changed disks.
Code: 
Nov 17 12:12:53 md04-zfs kernel: (da58:mps0:0:73:0): CAM status: SCSI Status Error
Nov 17 12:12:53 md04-zfs kernel: (da58:mps0:0:73:0): SCSI status: Check Condition
Nov 17 12:12:53 md04-zfs kernel: (da58:mps0:0:73:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
Nov 17 12:12:53 md04-zfs kernel: (da58:mps0:0:73:0): Error 22, Unretryable error
Nov 17 12:12:53 md04-zfs kernel: (da58:mps0:0:73:0): READ(16). CDB: 88 00 00 00 00 03 a3 81 28 00 00 00 01 00 00 00
Think, this is the cause of my problems. Something does not talk to the disk : enclosure / OS ...
Maybe a reboot ?
 
Are they in a hot-swap cabinet? Try re-inserting the drive, it may not have been properly seated. If you had the same kind of errors with the disk you replaced there may be something wrong with the connector, perhaps it's a little bent or broken. You may also want to check the cabling. The problems could also be caused by a faulty port extender. What does smartctl(8) tell you about the individual drives?
 
Yes the disks are in a Dell MD3060. This is an enclosure for 60 disks.
I'll try tomorrow to remove/reinsert the two disks.

smartctl -a for both disks returns good health :

Code: 
smartctl -a /dev/da35
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.1-RELEASE-p1 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               SEAGATE
Product:              ST8000NM014A
Revision:             CSL7
Compliance:           SPC-5
User Capacity:        8,001,563,222,016 bytes [8.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Formatted with type 2 protection
8 bytes of protection information per logical block
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000c500d2a32677
Serial number:        WKD2WLH2
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Tue Nov 17 17:11:10 2020 CET
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Disabled or Not Supported

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Grown defects during certification <not available>
Total blocks reassigned during format <not available>
Total new blocks reassigned <not available>
Power on minutes since format <not available>
Current Drive Temperature:     45 C
Drive Trip Temperature:        60 C

Manufactured in week 27 of year 2020
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  58
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  56
Elements in grown defect list: 0

Vendor (Seagate Cache) information
  Blocks sent to initiator = 13344
  Blocks received from initiator = 18528
  Blocks read from cache and sent to initiator = 11749
  Number of read and write commands whose size <= segment size = 77
  Number of read and write commands whose size > segment size = 0

Vendor (Seagate/Hitachi) factory information
  number of hours powered up = 25.77
  number of minutes until next internal SMART test = 20

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0        0         0         0          0          0.007           0
write:         0        0         0         0          0          0.086           0

Non-medium error count:        0

No Self-tests have been logged
 
Momo said:
Here is the answer :
Code: 
zpool replace md04-tank 9189834293223397025 da35
cannot replace 9189834293223397025 with da35: new device must be a single disk
Click to expand...
I would say: At the time you ran that command, da35 was not a disk drive. Perhaps it didn't even exist. However, at a later time it did exist, because you showed results from "smartctl ... /dev/da35". Suggestion: Rerun that command, after making sure /dev/da35 exists, and is a device (not some other thing, like a file or softlink).

Momo said:
I also have a lot of SCSI errors each time I try to access one of the physically changed disks.
Code: 
Nov 17 12:12:53 md04-zfs kernel: (da58:mps0:0:73:0): CAM status: SCSI Status Error
Nov 17 12:12:53 md04-zfs kernel: (da58:mps0:0:73:0): SCSI status: Check Condition
Nov 17 12:12:53 md04-zfs kernel: (da58:mps0:0:73:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
Nov 17 12:12:53 md04-zfs kernel: (da58:mps0:0:73:0): Error 22, Unretryable error
Nov 17 12:12:53 md04-zfs kernel: (da58:mps0:0:73:0): READ(16). CDB: 88 00 00 00 00 03 a3 81 28 00 00 00 01 00 00 00
Think, this is the cause of my problems. Something does not talk to the disk : enclosure / OS ...
Click to expand...
The SCSI command listed there is perfectly legal, it is a READ(16) command. Go to wikipedia, decode the CDB by hand, and the ASC/ASCQ and corresponding error message "invalid command operation code" makes no sense; any modern disk can execute a READ(16) command. Something in the stack underneath the OS SCSI layer is confused. Maybe it tried delivering that command to a SCSI device that's not a disk (like the enclosure controller)?

My answer would be: If these error messages go away by themselves, great, the confusion cleared itself out. Otherwise, I would reset the HBA (the mps0) device. And unfortunately, the easiest way to do that is a full reboot (as much as I hate suggesting reboots as a cure for problems, they're easy).
 
After rebooting the server, the disk were ready and the errors disappeared...
So the reboot was the solution.
I've been then able to do zpool replace with no problem.
Thanks, to all.
 
You must log in or register to reply here.
Back
Top