ZFS device name wiped

circus78

Active Member

Thanks: 6
Messages: 197

#1
Hi,
I use 11.2-RELEASE and I am reading the Handbook's chapter about ZFS.
In particular, I am reading the paragraph 19.3.8 ("Self-Healing").
In such example, command dd is used to simulate some data corruption on one of two disks.

This is my scenario:

Code:
root@bsdscratch:~ # zpool import
   pool: mypool
     id: 4003180980568308002
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

    mypool      ONLINE
      mirror-0  ONLINE
        ada1    ONLINE
        ada2    ONLINE
Now I use dd:

Code:
root@bsdscratch:~ # dd if=/dev/random of=/dev/ada1 bs=1m count=200
After this, it seems ada1 has his name changed:

Code:
root@bsdscratch:~ # zpool status
  pool: mypool
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
    attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
    using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: none requested
config:

    NAME                                 STATE     READ WRITE CKSUM
    mypool                               ONLINE       0     0     0
      mirror-0                           ONLINE       0     0     0
        diskid/DISK-VB3b3a7569-72b00eda  ONLINE       0     0     6
        ada2                             ONLINE       0     0     0

errors: No known data errors
Even after scrub:

Code:
root@bsdscratch:~ # zpool scrub mypool
root@bsdscratch:~ # zpool status
  pool: mypool
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
    attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
    using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 9.50K in 0h0m with 0 errors on Sun Nov  4 13:17:23 2018
config:

    NAME                                 STATE     READ WRITE CKSUM
    mypool                               ONLINE       0     0     0
      mirror-0                           ONLINE       0     0     0
        diskid/DISK-VB3b3a7569-72b00eda  ONLINE       0     0    10
        ada2                             ONLINE       0     0     0

errors: No known data errors
It seems this side effect isn't considered in Handbook.
I am aware that when data is corrupted in real scenarios, it is unlikely that very first bytes in a disk are altered.
Anyway, I wondering if there is a way to rename diskid/DISK-VB3b3a7569-72b00eda to ada1 in this case.

Thank you!
 

ShelLuser

Son of Beastie

Thanks: 1,490
Messages: 3,262

#2
What does sysctl kern.disks show you? As far as I know this should be reset automatically once the device is properly repaired.

Did anything change after running # zpool clear? Optionally after a reboot.
 

Emrion

Member

Thanks: 6
Messages: 43

#3
I tried to reproduce your problem in a VM. But when it came to dd'ed ada1, I got "Operation not permitted". Which is a good new for distracted admins. I had to type before: zpool offline mydata ada1. mydata being the name of my pool.

Code:
# zpool offline mydata ada1
# dd if=/dev/random of=/dev/ada1 bs=1m count=200
# zpool online mydata ada1
# zpool status mydata
pool: mydata
State: ONLINE
Scan: resilvered 60K in 0h0m with 0 errors on Sun Nov  4 20:13:10 2018
config:

    NAME                                 STATE     READ WRITE CKSUM
    mydata                               ONLINE       0     0     0
      mirror-0                           ONLINE       0     0     0
        ada1                             ONLINE       0     0     0
        ada2                             ONLINE       0     0     0

errors: No known data errors
So, I don't know how you got such a trouble.
 
OP
OP
C

circus78

Active Member

Thanks: 6
Messages: 197

#4
What does sysctl kern.disks show you? As far as I know this should be reset automatically once the device is properly repaired.

Did anything change after running # zpool clear? Optionally after a reboot.
After reboot:

Code:
root@bsdscratch:~ # sysctl kern.disks
kern.disks: cd0 ada2 ada1 ada0
Code:
root@bsdscratch:~ # zpool status
  pool: mypool
 state: ONLINE
  scan: scrub repaired 9.50K in 0h0m with 0 errors on Sun Nov  4 13:17:23 2018
config:

    NAME                                 STATE     READ WRITE CKSUM
    mypool                               ONLINE       0     0     0
      mirror-0                           ONLINE       0     0     0
        diskid/DISK-VB3b3a7569-72b00eda  ONLINE       0     0     0
        ada2                             ONLINE       0     0     0

errors: No known data errors
Thank you
 
OP
OP
C

circus78

Active Member

Thanks: 6
Messages: 197

#5
I tried to reproduce your problem in a VM. But when it came to dd'ed ada1, I got "Operation not permitted". Which is a good new for distracted admins. I had to type before: zpool offline mydata ada1. mydata being the name of my pool.

Code:
# zpool offline mydata ada1
# dd if=/dev/random of=/dev/ada1 bs=1m count=200
# zpool online mydata ada1
# zpool status mydata
pool: mydata
State: ONLINE
Scan: resilvered 60K in 0h0m with 0 errors on Sun Nov  4 20:13:10 2018
config:

    NAME                                 STATE     READ WRITE CKSUM
    mydata                               ONLINE       0     0     0
      mirror-0                           ONLINE       0     0     0
        ada1                             ONLINE       0     0     0
        ada2                             ONLINE       0     0     0

errors: No known data errors
So, I don't know how you got such a trouble.
I didn't used offline command, just zfs export (as Handbook indicates).
Maybe is for that reason you got different behavior?
 

ShelLuser

Son of Beastie

Thanks: 1,490
Messages: 3,262

#6
I tried to reproduce your problem in a VM. But when it came to dd'ed ada1, I got "Operation not permitted".
Are you sure you were logged on as root and that this environment didn't have any other failsaves? Or maybe you ran a developer snapshot?

Which is a good new for distracted admins.
It's not, goes right against basic principles. Unix and therefor also FreeBSD doesn't know, nor care, about details such as files (or devices) being in use. Not to mention the hassle this would cause if you had legitimate reasons to overwrite said device.

I didn't used offline command, just zfs export (as Handbook indicates).
This requires more than a mere export, also see the default instructions. The device got trashed so you're looking at a required reset of said device, and a mere export won't do that. Try running # zpool clear as showcased. Or take the device off and then back online.

The sysctl command was merely meant to verify that FreeBSD itself still recognized the medium as-is.
 

Emrion

Member

Thanks: 6
Messages: 43

#7
Are you sure you were logged on as root and that this environment didn't have any other failsaves? Or maybe you ran a developer snapshot?
Pretty sure. It makes sense that the system does not allow to modify a running pool by this means.

I didn't used offline command, just zfs export (as Handbook indicates).
Maybe is for that reason you got different behavior?
Seems probable. I didn't read the Handbook concerning this very point but... Does it show how to corrupt a pool ? ;)

Franckly, neither zpool offline nor zpool export is a real scenario. That being said, I will try to reproduce what you got. As far as I know, there isn't such information (I mean "ada1") in the structure of zfs but I'm still studying it. You destroyed the first two vdel label, the boot block and some data after. Nothing irrecuperable.
 

ShelLuser

Son of Beastie

Thanks: 1,490
Messages: 3,262

#8
Pretty sure. It makes sense that the system does not allow to modify a running pool by this means.
That's just the thing: it doesn't know anything about that. This is no different from running dd on a mounted partition (UFS) which will also simply result in the filesystem getting overwritten.

The best thing dd has is a test for the filesystem type to encounter for the maximum filesize:
Code:
        case `df -T . | tail -n 1 | cut -wf 2` in                                
                "ufs")                                                           
                        atf_skip "UFS's maximum file size is too small";;                        "zfs") ;; # ZFS is fine                                          
                "tmpfs")                                                                                 atf_skip "tmpfs can't create arbitrarily large spare file               *) atf_skip "Unknown file system";;
From /usr/src/bin/dd/tests/dd2_test.sh, which will end up in /usr/tests. But other than that there's no mention of filesystem types, not to mention that the source doesn't even contain the error message you shared, so if that did got triggered then it wasn't dd doing it.

Franckly, neither zpool offline nor zpool export is a real scenario.
You usually use zpool offline to actually take a device offline before you can remove (or replace) it from a pool. This prevents the pool from actually writing to that device, which can be quite useful. Whatever gives you the idea that this isn't a realistic thing to use?
 

Emrion

Member

Thanks: 6
Messages: 43

#9
It doesn't know? See that, then:

export_pool.png


By the way, I have no regular user on this test VM, just root...

Now, circus78, concerning your issue, again I can't reproduce what you experienced.

In fact, if I understand correctely, zpool export mypool move the target pool in a state it can't no longer be used until you type zpool import mypool. It's not mounted nor displayed by zpool status. So I don't understand how you get the situation you describe. I think you typed more commands than you show us.

As for a real disk failure, and I lived several of this kind: no more disk at all. You can hear the heads that scratch inside. Then, if you don't have redondancy, you loose the system and/or the data.

I'm trying to simulate a "disk disappearence" and I get weird problems with zfs mirror. I will continue to investigate them and I will post here my questions.
 

Bobi B.

Active Member

Thanks: 91
Messages: 214

#10
I've experienced this few times myself. I believe this is due to same device appearing with multiple names on FreeBSD (see glabel(8)). Would you try to export the pool and import it using -d option of zpool(8): zpool import -d /dev tank? I'm not sure I've finally decided this for myself, but lately I tend to use device IDs whose include drives' serial numbers. Hence, in my case, zpool import -d /dev/diskid tank.
 

Emrion

Member

Thanks: 6
Messages: 43

#11
After some investigations, I found that "Operation not permitted" doesn't come from dd but (probably) from the geom(4) framework.
Anyway, even in root, if I believe my tests, you cannot dd'ed a disk if a partition belonging to it is mounted (but you can do what you want in case the disk is not in use).

Unless kern.geom.debugflag is set to 0x10...

(From geom(4))
0x10 (allow foot shooting)
Allow writing to Rank 1 providers. This would, for example,
allow the super-user to overwrite the MBR on the root disk or
write random sectors elsewhere to a mounted disk. The implica-
tions are obvious.
So, you can see that the system (kernel) is able to protect itself, even against some errors (bullshit...) from the root user.
 
OP
OP
C

circus78

Active Member

Thanks: 6
Messages: 197

#12
I've experienced this few times myself. I believe this is due to same device appearing with multiple names on FreeBSD (see glabel(8)). Would you try to export the pool and import it using -d option of zpool(8): zpool import -d /dev tank? I'm not sure I've finally decided this for myself, but lately I tend to use device IDs whose include drives' serial numbers. Hence, in my case, zpool import -d /dev/diskid tank.
Hi, it seems I got same behavior.
I tried to use dd on second hard disk (ada2 in my case).
Again, I was perfectly able to do this, no "Operation not permitted" error at all.

First scenario:

Code:
root@bsdscratch:~ # zpool status
  pool: mypool
 state: ONLINE
  scan: scrub repaired 9.50K in 0h0m with 0 errors on Sun Nov  4 13:17:23 2018
config:

    NAME                                 STATE     READ WRITE CKSUM
    mypool                               ONLINE       0     0     0
      mirror-0                           ONLINE       0     0     0
        diskid/DISK-VB3b3a7569-72b00eda  ONLINE       0     0     0
        ada2                             ONLINE       0     0     0

errors: No known data errors
Then I used zfs export command:

Code:
root@bsdscratch:~ # zpool export mypool
root@bsdscratch:~ # mount
/dev/gpt/bsdrootfs on / (ufs, local, soft-updates)
devfs on /dev (devfs, local, multilabel)
root@bsdscratch:~ # zpool status
no pools available
Then I used dd in order to do nasty things on ada2 :)

Code:
root@bsdscratch:~ # dd if=/dev/random of=/dev/ada2 bs=1m count=200
200+0 records in
200+0 records out
209715200 bytes transferred in 211.316450 secs (992423 bytes/sec)
After zfs import, even second drive has device name changed:

Code:
root@bsdscratch:~ # zpool import mypool
root@bsdscratch:~ # zpool status
  pool: mypool
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
    attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
    using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 9.50K in 0h0m with 0 errors on Sun Nov  4 13:17:23 2018
config:

    NAME                                 STATE     READ WRITE CKSUM
    mypool                               ONLINE       0     0     0
      mirror-0                           ONLINE       0     0     0
        diskid/DISK-VB3b3a7569-72b00eda  ONLINE       0     0     0
        diskid/DISK-VB3520007a-8aeb2232  ONLINE       0     0     1

errors: No known data errors

The command ShelLuser suggested me seems ok:

Code:
root@bsdscratch:~ # sysctl kern.disks
kern.disks: cd0 ada2 ada1 ada0
I also tried your suggestion:

Code:
root@bsdscratch:~ # zpool import -d /dev/diskid mypool
root@bsdscratch:~ # zpool status
  pool: mypool
 state: ONLINE
  scan: scrub repaired 26.5M in 0h0m with 0 errors on Sat Nov 10 11:38:42 2018
config:

    NAME                                 STATE     READ WRITE CKSUM
    mypool                               ONLINE       0     0     0
      mirror-0                           ONLINE       0     0     0
        diskid/DISK-VB3b3a7569-72b00eda  ONLINE       0     0     0
        diskid/DISK-VB3520007a-8aeb2232  ONLINE       0     0     0

errors: No known data errors
Anyway, it is not a big problem.. just for curiosity! :)

Thank you very much
 
OP
OP
C

circus78

Active Member

Thanks: 6
Messages: 197

#13
After some investigations, I found that "Operation not permitted" doesn't come from dd but (probably) from the geom(4) framework.
Anyway, even in root, if I believe my tests, you cannot dd'ed a disk if a partition belonging to it is mounted (but you can do what you want in case the disk is not in use).

Unless kern.geom.debugflag is set to 0x10...

(From geom(4))

So, you can see that the system (kernel) is able to protect itself, even against some errors (bullshit...) from the root user.
Here my sysctl options:

Code:
root@bsdscratch:~ # sysctl -a |grep debugflag
kern.geom.debugflags: 0
vfs.zfs.debugflags: 0
debug.fdc.debugflags: 0
 

Emrion

Member

Thanks: 6
Messages: 43

#14
circus78 said:
tried to use dd on second hard disk (ada2 in my case).
Again, I was perfectly able to do this, no "Operation not permitted" error at all.
You can dd'ed a disk that belongs to a pool if you run zpool export mypool before. This is because it unmounts all datasets within the pool, see zpool(8).
zpool export [-f] pool ...

(...)

Before exporting the pool, all datasets within the pool are
unmounted.
A pool can not be exported if it has a shared spare that
is currently being used.
Try the same just after zpool import mypool and you will get "Operation not permitted".

Concerning your problem. Here, when I type zpool import mypool (after dd'ed ada1), I keep ada1 and ada2 names. I run zpool scrub mypool to repair and then, zpool clear mypool. No problem at all.

Maybe, it's because My FreeBSD test is a VM (under Windows/VirtualBox)?

I got strange behaviors in my own tests (but the scenario was to "unplug" one of the two disks). After a dozen of tries, I come to think this is a problem related to VirtualBox and its snapshots.
 
Top