zfs mount error 6 after 11.0 to 11.1 upgrade

Alejandro

Alejandro

Hello everyone.
Today i decided to upgrade my 11.0 host to 11.1 (ports deprecated) from source. All what was added to kernel-conf compared to 11.0 version is options EARLY_AP_STARTUP.
After the merging ( mergemaster -iUF) host was rebooted and error is up
Code: 
Trying to mount root from zfs:zroot []...
Mounting from zfs:zroot failed with error 6

    vfs.root.mountfrom=zfs:zroot
I can boot with live-cd or memstick and import (-d /dev/gpt -R /mnt) any pool and work with content.
Can someone give advice of what to do next to fix this issue?
All what comes to my mind is reinstalling it (11.0) on other flash/ssd/hdd and move backed up configs from old to new OS... But it doesn't feel right.
System installed on flash-drive (USB3.0) and boot vars are
Code: 
zfs_load="YES"
kern.cam.boot_delay="10000"
kern.cam.scsi_delay="3000"
:
Code: 
# gpart show da1
=>      40  30343088  da1  GPT  (14G)
        40       472    1  freebsd-boot  (236K)
       512  25165824    2  freebsd-zfs  (12G)
  25166336   5176792       - free -  (2.5G)
Code: 
# zpool status zroot
  pool: zroot
 state: ONLINE
  scan: none requested
config:

    NAME          STATE     READ WRITE CKSUM
    zroot         ONLINE       0     0     0
      gpt/zroot0  ONLINE       0     0     0

errors: No known data errors
There are 2 other pool in system:
Code: 
# zpool status data ssd
  pool: data
 state: ONLINE
  scan: none requested
config:

    NAME           STATE     READ WRITE CKSUM
    data           ONLINE       0     0     0
      raidz3-0     ONLINE       0     0     0
        gpt/hdd0   ONLINE       0     0     0
        gpt/hdd1   ONLINE       0     0     0
        gpt/hdd2   ONLINE       0     0     0
        gpt/hdd3   ONLINE       0     0     0
        gpt/hdd4   ONLINE       0     0     0
        gpt/hdd5   ONLINE       0     0     0
        gpt/hdd6   ONLINE       0     0     0
        gpt/hdd7   ONLINE       0     0     0
        gpt/hdd8   ONLINE       0     0     0
        gpt/hdd9   ONLINE       0     0     0
        gpt/hdd10  ONLINE       0     0     0
        gpt/hdd11  ONLINE       0     0     0
        gpt/hdd12  ONLINE       0     0     0
        gpt/hdd13  ONLINE       0     0     0
    logs
      mirror-1     ONLINE       0     0     0
        gpt/zil0   ONLINE       0     0     0
        gpt/zil1   ONLINE       0     0     0
    cache
      gpt/l2arc0   ONLINE       0     0     0
      gpt/l2arc1   ONLINE       0     0     0
    spares
      gpt/hdd14    AVAIL   
      gpt/hdd15    AVAIL   

errors: No known data errors

  pool: ssd
 state: ONLINE
  scan: none requested
config:

    NAME           STATE     READ WRITE CKSUM
    ssd            ONLINE       0     0     0
      mirror-0     ONLINE       0     0     0
        gpt/zssd0  ONLINE       0     0     0
        gpt/zssd1  ONLINE       0     0     0

errors: No known data errors
And datasets are:
Code: 
NAME                                  USED  AVAIL  REFER  MOUNTPOINT
data                                 1.87T  33.7T   279K  none
data/home                            1.57T  33.7T   279K  /mnt/usr/home
data/home/user0                       114G  33.7T   114G  /mnt/usr/home/user0
data/logs                            2.27M  33.7T  2.27M  /mnt/var/log
data/pkgdb                           33.4M  33.7T  33.4M  /mnt/var/db/pkg
data/ports                           2.33G  33.7T  1.99G  /mnt/usr/ports
data/ports/distfiles                  345M  33.7T   345M  /mnt/usr/ports/distfiles
data/ports/packages                  5.42M  33.7T  5.42M  /mnt/usr/ports/packages
data/portsdb                          779K  33.7T   779K  /mnt/var/db/ports
data/portsnapdb                       473M  33.7T   473M  /mnt/var/db/portsnap
data/src                             2.51G  33.7T  2.51G  /mnt/usr/src
zroot                                6.91G  4.66G  1.05G  /mnt
zroot/swap                           4.13G  8.39G   407M  -
zroot/usr                            1.71G  4.66G  1.71G  /mnt/usr
zroot/var                            22.4M  4.66G  10.7M  /mnt/var
zroot/var/crash                        96K  4.66G    96K  /mnt/var/crash
zroot/var/db                         10.9M  4.66G  10.9M  /mnt/var/db
zroot/var/empty                        96K  4.66G    96K  /mnt/var/empty
zroot/var/mail                        108K  4.66G   108K  /mnt/var/mail
zroot/var/run                         140K  4.66G   140K  /mnt/var/run
ssd                                  2.33G   128G    96K  none
ssd/obj                              2.32G   128G  2.32G  /mnt/usr/obj
ssd/vartmp                            176K   128G   176K  /mnt/var/tmp
 
Some of the commands you mentioned make no sense to me. You say you can import 'any' pool, then list this: (-d /dev/gpt -R /mnt). Why would you specify -d? It's not required, ZFS should be able to this on its own, and I can't help wonder if this could be related to your issue.

First things first: setting up a ZFS pool with only one drive isn't the best of practices because you won't gain much advantages. It can't protect you from data rot, data isn't redundant and despite all that it will still eat into your resources (gobble up memory for example).

What happens if you boot your system, get into the boot menu, hit escape to drop to the boot prompt ("ok>") and then use: lsdev. Does it mention all your pools?

Also, your pastes are a bit chaotic, but from what I can tell you didn't specify the root in /boot/loader.conf. It should also contain this line:
Code: 
vfs.root.mountfrom="zfs:zroot"
Put that below zfs_load and you should be ok.
 
Thank you for your answer and patience.
ShelLuser said:
Why would you specify -d? It's not required, ZFS should be able to this on its own, and I can't help wonder if this could be related to your issue.
Click to expand...
My mistake, "-d" was in command to make sure it uses GPT labels. Checked and yes, it is not necessary to put this flag in, pools are imported without it (LiveCD environment).
ShelLuser said:
It can't protect you from data rot, data isn't redundant and despite all that it will still eat into your resources (gobble up memory for example).
Click to expand...
Yes, it was experiment. In near future it will be mirror of two ssd's.
ShelLuser said:
and then use: lsdev. Does it mention all your pools?
Click to expand...
Yes, all pools are at the end of listing
ShelLuser said:
you didn't specify the root in /boot/loader.conf. It should also contain this line:
Click to expand...
Not sure it should be there, but that didn't help.
 
Alejandro said:
Not sure it should be there, but that didn't help.
Click to expand...
I'm sure alright, and so should you be ;) Take a closer look at the error message you posted in the OP: it mentions this line for a very good reason.

Ok, step two... Boot using a rescue CD ("rescue environment") and import the system pool ('zroot', so the pool which contains your root filesystem which includes /boot and your kernel and all). For example: # zpool import -fR /mnt zroot. Now check this: zpool get bootfs zroot. Does that give you any results?

If not then that's most likely part 2 of your problem. Fixed with: # zpool set zroot bootfs zroot.
 
Well, in the mean time I verified a few things and I confirmed for myself that the default installer doesn't put the line which I mentioned earlier in /boot/loader.conf. Have to admit that this surprised me because it is considered good practice, but then again there are more things of which I personally think that installer doesn't do them very well.

Back to your problem. My next question: what kernel does the system use? Do you use a generic (default) kernel or did you set up a custom one? I'm assuming generic, but if it is custom then I can't help but wonder if that could be the problem. Then I'd recommend setting up a generic kernel to see how that goes.

I closely re-read the OP again and right now I'm wondering if this could be related to a classic issue which several PC's have: they provide USB support through the BIOS which enables the machine to boot from the USB connected media. But after that control is transferred to the OS (to keep it simple) and if the OS doesn't have the proper drivers then that could result in the OS not being able to access drive anymore.

The problem with that theory though is that it worked on 11.0, and when checking the 11.1 release notes I can't see any change which could explain this behavior.

Another question: when you get the "Error 6" error you're dropped to a command line I assume? What happens if you manually try to enter zfs:zroot? Same error no doubt but....

When you upgraded your system, did you also update the boot blocks? That's definitely something worth trying; sometimes there are changes introduced and then the boot code also needs updating. For the record: it shouldn't matter with a minor version upgrade, but....

So rescue media time again, as before; # zpool import -fR /mnt zroot.

And after that something in the likes of: gpart bootcode -i 1 -b /mnt/boot/pmbr -p /mnt/boot/gptzfsboot da1.

Optionally you could consider making a backup first: # gpart backup da1 > da1.backup.

Hope this can help.
 
ShelLuser said:
what kernel does the system use?
Click to expand...
It is a custom kernel. I'll sure try generic one (is it possible to build generic in rescue media environment for my existing installation?), but again - it worked on 11.0, i've mentioned in OP about only option added to 11.1 kernel...
ShelLuser said:
I'm wondering if this could be related to a classic issue which several PC's have: they provide USB support through the BIOS which enables the machine to boot from the USB connected media. But after that control is transferred to the OS (to keep it simple) and if the OS doesn't have the proper drivers then that could result in the OS not being able to access drive anymore.
Click to expand...
Do you mean "XHCI/EHCI Handoff"? If so, i've already tested with this option off/on in BIOS with no result - "error 6". (And it was not an issue in 11.0)
ShelLuser said:
you're dropped to a command line I assume? What happens if you manually try to enter zfs:zroot? Same error no doubt but....
Click to expand...
No doubt it's the same.
ShelLuser said:
When you upgraded your system, did you also update the boot blocks?
Click to expand...
Not in first place, but yes, i've already tried this command ( gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1) with no result.
 
Alejandro said:
It is a custom kernel. I'll sure try generic one
Click to expand...
That would definitely be my current recommendation, just to rule the whole thing out. No offense intended obviously, but right now (from my perspective) I'm tempted to point at the kernel as a possible cause for all this.

Alejandro said:
(is it possible to build generic in rescue media environment for my existing installation?),
Click to expand...
Absolutely!

Here's what I usually do... On the installation media (for example the disc1 CD ISO image) the different components (base system, kernel, doc, source code) are provided as packages. So what I usually do is extract /usr/freebsd-dist/kernel.txz into /boot/kernel.gen (gen as in 'generic'). Just use tar and you're home free.

Then the nifty part, add this to /boot/loaders.conf:
Code: 
kernels="kernel kernel.old kernel.gen"
I hope you're familiar with the FreeBSD boot menu? It allows you to select a kernel and normally you swap between the regular kernel and the backup copy (kernel.old). This adds a new option to that list: the generic kernel.

Alejandro said:
but again - it worked on 11.0, i've mentioned in OP about only option added to 11.1 kernel...
Click to expand...
Agreed. This is also the flaw in the theory I shared, I don't see how this would work in 11.0 and stopped working on 11.1. Unless of course there was something wrong which now manifested itself. It's possible but...

For now I hope GENERIC will shed some light on this.
 
You must log in or register to reply here.
Back
Top