ZFS ran freebsd-update with 2xinternal sata and 1xexternal usb drive now won't boot: ZFS: i/o error - all block copies unavailable

Hastur

Hastur

Hi I have desktop with 2 sata drives in zfs root, and added usb external drive to the zpool

things ran fine for a few days, and yes the usb pasues on boot as the external usb drive warms up but then boots fine

I ran freebsd update frm 12.1-p7 to p8 and when I rebooted

ZFS: i/o error - all block copies unavailable

I then rebooted few times same thing

I then chose option 5 which I think is the un upgraded kernel then hit 1 to boot and here I am....

How do I fix this? Searched around a bit and didn't see a solution.....



Code: 
root@mercantilism:~ # gpart show
=>       40  976773088  ada0  GPT  (466G)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048   33554432     2  freebsd-swap  (16G)
   33556480  943216640     3  freebsd-zfs  (450G)
  976773120          8        - free -  (4.0K)

=>       40  976773088  ada1  GPT  (466G)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048   33554432     2  freebsd-swap  (16G)
   33556480  943216640     3  freebsd-zfs  (450G)
  976773120          8        - free -  (4.0K)

=>         6  1220942634  da0  GPT  (4.5T)
           6  1220942634    1  freebsd-zfs  (4.5T)

root@mercantilism:~ # uname -a
FreeBSD mercantilism 12.1-RELEASE-p7 FreeBSD 12.1-RELEASE-p7 GENERIC  amd64
root@mercantilism:~ # zpool status -x
all pools are healthy
root@mercantilism:~ # zpool status
  pool: zroot
state: ONLINE
  scan: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        zroot          ONLINE       0     0     0
          ada0p3       ONLINE       0     0     0
          ada1p3       ONLINE       0     0     0
          gpt/seagate  ONLINE       0     0     0

errors: No known data errors

root@mercantilism:~ # zdb
zroot:
    version: 5000
    name: 'zroot'
    state: 0
    txg: 371885
    pool_guid: 18301116476300194985
    hostid: 1545946101
    hostname: 'mercantilism'
    com.delphix:has_per_vdev_zaps
    vdev_children: 3
    vdev_tree:
        type: 'root'
        id: 0
        guid: 18301116476300194985
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 14732638310234165604
            path: '/dev/ada0p3'
            whole_disk: 1
            metaslab_array: 78
            metaslab_shift: 32
            ashift: 12
            asize: 482922201088
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_leaf: 65
            com.delphix:vdev_zap_top: 66
        children[1]:
            type: 'disk'
            id: 1
            guid: 9983834860724692708
            path: '/dev/ada1p3'
            whole_disk: 1
            metaslab_array: 69
            metaslab_shift: 32
            ashift: 12
            asize: 482922201088
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_leaf: 67
            com.delphix:vdev_zap_top: 68
        children[2]:
            type: 'disk'
            id: 2
            guid: 10350705110928001847
            path: '/dev/gpt/seagate'
            whole_disk: 1
            metaslab_array: 347
            metaslab_shift: 35
            ashift: 12
            asize: 5000976138240
            is_log: 0
            create_txg: 352910
            com.delphix:vdev_zap_leaf: 345
            com.delphix:vdev_zap_top: 346
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
 
Why did you add a USB drive to a pool with SATA disks? That's a recipe for disaster. Combine this with a non-redundant striped set and you're bound to lose everything. Remember the whole set will be gone if any of the disks fail (for whatever reason).
 
SirDice said:
Why did you add a USB drive to a pool with SATA disks? That's a recipe for disaster. Combine this with a non-redundant striped set and you're bound to lose everything. Remember the whole set will be gone if any of the disks fail (for whatever reason).
Click to expand...

bittorrent and home net sufring box and learning box

not server class

Wonder why this happened tho?

It booted fine and still does in this backup kernel....

Is there something that needs to be done -p8 introduces?

forums.freebsd.org

Solved - ZFS: i/o error - all block copies unavailable

2020-03-04: The cause of this issue was eventually traced to a hardware failure on the system motherboard. FreeBSD-12.1p2 raidz2 on 4x8TB HDD (reds) root on zfs We did a hot restart of this host this morning and received the following on the console: ZFS: i/o error - all block copies...
forums.freebsd.org forums.freebsd.org

found this but not sure if it applies
 
Your USB disk might be a little slower to get detected. That means one disk of the set is missing, which causes the whole pool to fail.
 
SirDice said:
Your USB disk might be a little slower to get detected. That means one disk of the set is missing, which causes the whole pool to fail.
Click to expand...


Yes but with p7 is just waits for the usb timesout 3 times and then finds da0 and boots

why in p8 does this not happen?
 
SirDice said:
Your USB disk might be a little slower to get detected. That means one disk of the set is missing, which causes the whole pool to fail.
Click to expand...

Is there something I need to do to have the system wait for the usb drive?
It boots now in the p7 no problem...it just fails 4x then finds da0 and boots...
 
root@mercantilism:~ # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
gpart: /dev/da0p1: Operation not permitted
 
As SirDice pointed out already, your setup is like a rabbit dancing in front of a hungry lion shouting "catch me if you can". The lion will catch the rabbit eventually. You have no control whatsoever on which of the disks your data lands. Reinstall with a reasonable setup. If you have two identical disks, so put them into a mirror.
 
mjollnir said:
As SirDice pointed out already, your setup is like a rabbit dancing in front of a hungry lion shouting "catch me if you can". The lion will catch the rabbit eventually. You have no control whatsoever on which of the disks your data lands. Reinstall with a reasonable setup. If you have two identical disks, so put them into a mirror.
Click to expand...


Lets not focus on that as I like the setup.

The question is why does it work fine on 12.1-p7 and not 12.1-p8?

I honestly can't imagine why it would not work and what broke it on -p8.....
 
If you use striping, like you did, then all drives in the striped zpool have to be present for the zpool to work.

If you boot from ZFS, then the zpool that your root file system is on has to work in order to boot.

As several other posters already said, USB is slow and unreliable. Timeouts of USB initialization will screw you up, and those timeouts may change with the weather, the phase of the moon, and OS versions (which may answer your -p7 versus -p8 question). You have build a house of cards, and you are looking for the root cause that it collapses all the time and focusing on the change in version. You are ignoring the fact that you have built a house of cards. The "let's not focus" answer is exactly backwards: we need to focus on the fact that USB is slow and unreliable, because that's the problem.

My suggestion: Change your setup so the boot/root part of the system is on reliable disks (the internal ones). Suggestion: Partition your two SATA disks, creating a smallish partition on each (for example 100 gig). Use just those two small partitions for the root install. Then take the rest of the two SATA disks and the USB disk, and create a much larger pool out of that, which you put unimportant data (like media and torrents on). Now your system will always come up, and at least root can log in. If the USB disk is slow, then you can log in as root and fix things.
 
E.g. a suggestion for a partition layout for your two (identical?) internal disks:
Instead of a 450 GB freebsd-zfs, have a relatively small (e.g. 36-128 GB) partition in a mirrored ZPOOL for the OS. The rest you can put into freebsd-zfs partitions that make up two separate (unmirrored) ZPOOLs for your data, or you put them into one striped ZPOOL like you do now (seems you do not have high demands on storage reliabilty, that's ok). For the USB disk, usually MSDOS or NTFS is choosen because that's portable, but you're free to put a ZFS on it. BUT in it's own ZPOOL, please!
 
ralphbsz said:
If you use striping, like you did, then all drives in the striped zpool have to be present for the zpool to work.

If you boot from ZFS, then the zpool that your root file system is on has to work in order to boot.

As several other posters already said, USB is slow and unreliable. Timeouts of USB initialization will screw you up, and those timeouts may change with the weather, the phase of the moon, and OS versions (which may answer your -p7 versus -p8 question). You have build a house of cards, and you are looking for the root cause that it collapses all the time and focusing on the change in version. You are ignoring the fact that you have built a house of cards. The "let's not focus" answer is exactly backwards: we need to focus on the fact that USB is slow and unreliable, because that's the problem.

My suggestion: Change your setup so the boot/root part of the system is on reliable disks (the internal ones). Suggestion: Partition your two SATA disks, creating a smallish partition on each (for example 100 gig). Use just those two small partitions for the root install. Then take the rest of the two SATA disks and the USB disk, and create a much larger pool out of that, which you put unimportant data (like media and torrents on). Now your system will always come up, and at least root can log in. If the USB disk is slow, then you can log in as root and fix things.
Click to expand...

Your suggestiion are cool but I like my setup.
Why do you think it won't boot?
It boots fine in 12.1p7
If you don't know thats cool, but I am still curious why...
 
mjollnir said:
E.g. a suggestion for a partition layout for your two (identical?) internal disks:
Instead of a 450 GB freebsd-zfs, have a relatively small (e.g. 36-128 GB) partition in a mirrored ZPOOL for the OS. The rest you can put into freebsd-zfs partitions that make up two separate (unmirrored) ZPOOLs for your data, or you put them into one striped ZPOOL like you do now (seems you do not have high demands on storage reliabilty, that's ok). For the USB disk, usually MSDOS or NTFS is choosen because that's portable, but you're free to put a ZFS on it. BUT in it's own ZPOOL, please!
Click to expand...

Again I don't want to do that, everything boots fine under 12.1p7 why not on 12.1p8 I wonder?
 
Can you reproduce that? Did you make a snapshot (boot environment) before freebsd-update(8) which you can boot to -p7? Obviously there are changes between -p7 and -p8 ...
EDIT You asked for advice & answered the very same advice of 4 respected voices: but I like my setup. Did you reflect on what we told you? Most others would see a speedup on boot (not waiting for USB so long) as an enhancement...
 
Hastur said:
If you don't know thats cool, but I am still curious why...
Click to expand...
I don't know, and it seems nobody else has any idea either. But clearly, the setup with disks of very different IO characteristics and reliability in the same pool is a bad idea.

Suggestion: Since you seem to be willing to run with an unreliable and fragile system, why not just reinstall 12.1p7? I don't know how to do a downgrade (never done one, and it is not recommended), but since you're willing to run your system at the edge of a disaster, why not try?
 
Hastur said:
Lets not focus on that as I like the setup.
Click to expand...

you don't like it and you don't like the risk that this fragile setup brings, otherwise you would not have asked for advice here. I agree with all others: change your setup if you value your time and your data.
 
You must log in or register to reply here.
Back
Top