10.1-RELEASE-p11 update broke zfs mountroot

mbeichorn · Jun 10, 2015

I just applied the 10.1-RELEASE-p11 update via:

 freebsd-update fetch

freebsd-update install

and sent my system to restart.

The system has not come back and is stuck at the following prompt
mountroot>

The previous line is:
Mounting from zfs:zroot/ROOT/default failed with error 2: unknown file system

Manual mouting returns the same result.

After performing a hard reset I noticed the following error:
ZFS: i/o error - all block copies unavailable

The system a zfs on root two drive mirror as configured by the zfs option in 10-RELEASE installer.
No ZIL, No L2ARC, No dedup, lz4 compression on.

does anyone know how to fix this?

junovitch@ · Jun 11, 2015

I tried and wasn't able to replicate the issue in a VM either with a single disk or 2 disk mirror. So it may be due to some configuration or the on disk state real world use had your system in. I would suggest seeing if you can boot from an ISO and attempt to import the pool from there. You may be able to do a little exploratory work and see what's going on from there.

rainer_d · Jun 11, 2015

I also updated a system with mountroot (physical, HP DL 380G8, booting from a HW RAID1) - without problems. I saw the initial postings right after hitting "reboot" (it did not help that EN15-07 was not on the website at this point and AFAIK, freebsd-update(8) does not allow to go to a specific patch-release). No problems so far.

adri · Jun 12, 2015

I have not had any problem upgrading to p11 using freebsd-update(8) either.
This is a physical server with 1 vdev mirror root pool.

ged · Jun 28, 2015

I am facing a very similar situation, only I upgraded from 8.x. The error is slightly different though.

Mounting from zfs:zroot/ROOT/newroot failed with error 6.

At this point, I have:
- Verified that the pools are OK
- Upgraded all pools and filesystems to the latest version
- Tried both a valid zpool.cache - and booting without it, as is currently supported
- Tried various alternatives for making the kernel actually mount the filesystem as root, including setting the mountpoint to / and "legacy".
- Set debugging on for loading the root.

The "funny" part is that the bootloader finds the kernel just fine, and the kernel starts and loads modules from the same file system.

Thus I suspect that there is some kind of regression in how the kernel locates the root file system when using zfs.

In my case, for historical reasons, the disk is MBR, and there is a freebsd partition which contains the freebsd-zfs pool. ada2s1a, or something similar. Unfortunately, I cannot read the vdev_geom debugging messages - they scroll past too fast. Is there any way of scrolling back/capturing them?

Any ideas?

Supermule · Jun 28, 2015

Where do you see p11 and how to check for that in release? Can't find any info on p9, p11 and p13 and the difference?

ged · Jun 28, 2015

OK - one correction - it's the most recent version as of now, I'm not quite sure it's p11. Hard to check with an unbootable server.

junovitch@ · Jun 29, 2015

Supermule said:
Where do you see p11 and how to check for that in release? Can't find any info on p9, p11 and p13 and the difference?

Check freebsd-version as uname -a just tells you the kernel version and won't show an update that didn't touch the kernel.

For details on what was fixed in each patch level:
Errata notices -- https://www.freebsd.org/security/notices.html
Security advisories -- https://www.freebsd.org/security/advisories.html

ged · Aug 24, 2015

Dead silent on the response part here - I still have a badly crippled system that won't mount root (although the partition is there, and the kernel itself was successfully loaded from that zfs partition).

Any ideas/fixes?