a word of warning regarding ZFS in 8.1

[Jago]

http://www.freebsd.org/cgi/query-pr.cgi?pr=148655

A lot of folks would probably call this issue "critical". I know I won't be updating my 8.0 system until this is fixed.

 

[UNIXgod]

As interesting as zfs on root looks. I can see that it is a big 'no no' for production. I've had no problems with ufs for boot. I plan to keep it that way for a long time to come.

 

[gkontos]

Apparently it is not even included in the errara.
Also this is being discussed here.

George

 

[kpa]

It's not broken for everyone, my system boots just fine with only one drive of zfs mirror attached.

My system details:

Intel D975xbx2 MB, 2x320GB SATA HDs set to AHCI mode GPT partitioned, 8 GB of RAM, FreeBSD 8.1-RELEASE amd64.

 

[gkontos]

It's not broken for everyone, my system boots just fine with only one drive of zfs mirror attached.

My system details:

Intel D975xbx2 MB, 2x320GB SATA HDs set to AHCI mode GPT partitioned, 8 GB of RAM, FreeBSD 8.1-RELEASE amd64.

Does it also boot if you remove the first drive ?

George

 

[kpa]

Yeah it's not working if I remove the first drive... even if I connect the second drive with cable of the first drive. The bootloader that's written to the 2nd disk is broken.

 

[gkontos]

Yeah it's not working if I remove the first drive... even if I connect the second drive with cable of the first drive. The bootloader that's written to the 2nd disk is broken.

Same issue here. This was tested with no issues on 8.0 release.

 

[kreyszig]

This is pretty bad! I hope this will be fixed ASAP

 

[mm@]

Patch for testing

Could you please test the patch located at:
http://people.freebsd.org/~mm/patches/zfs/head-zfsimpl.c.patch

It applies against 9-CURRENT, 8-STABLE and also 8.1-RELEASE.

It should fix the problem described in:
http://www.freebsd.org/cgi/query-pr.cgi?pr=148655

Instructions:

1. Have 9-CURRENT, 8-STABLE or 8.1-RELEASE sources in /usr/src

2. Download the patch

3. Run the following commands:

cd /usr/src
patch -p0 < /path_to/head-zfsimpl.c.patch
cd /usr/src/sys/boot/zfs
make clean
make depend
make
cd /usr/src/sys/boot/i386/zfsloader
make clean
make depend
make
make install
cd /usr/src/sys/boot/i386/gptzfsboot
make clean
make depend
make
make install
gpart bootcode -p /boot/gptzfsboot -i 1 ad4
gpart bootcode -p /boot/gptzfsboot -i 1 ad6

Repeat the last lines for all your disks in the ZFS mirror or raidz, assuming that the first partition is of type freebsd-boot and replace disk names "ad4", "ad6", etc. with your GEOM disk names

To view GEOM disk names, use:
gpart show

 

[gkontos]

fbsd# make
Warning: Object directory not changed from original /usr/src/sys/boot/i386/gptzfsboot
cc -Os  -fno-guess-branch-probability  -fomit-frame-pointer  -fno-unit-at-a-time  -mno-align-long-strings  -mrtd  -mno-mmx -mno-3dnow -mno-sse -
mno-sse2 -mno-sse3  -DGPT -DBOOT2  -DSIOPRT=0x3f8  -DSIOFMT=0x3  -DSIOSPD=9600  -I/usr/src/sys/boot/i386/gptzfsboot/../../common  -
I/usr/src/sys/boot/i386/gptzfsboot/../../zfs  -I/usr/src/sys/boot/i386/gptzfsboot/../../../cddl/boot/zfs  -
I/usr/src/sys/boot/i386/gptzfsboot/../btx/lib -I.  -I/usr/src/sys/boot/i386/gptzfsboot/../boot2  -Wall -Waggregate-return -Wbad-function-cast -
Wcast-align  -Wmissing-declarations -Wmissing-prototypes -Wnested-externs  -Wpointer-arith -Wshadow -Wstrict-prototypes -Wwrite-strings  -Winline 
--param max-inline-insns-single=100 -ffreestanding -mpreferred-stack-boundary=2  -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -m32 -
march=i386 -std=gnu99   -c /usr/src/sys/boot/i386/gptzfsboot/../gptboot/gptldr.S
ld -static -N --gc-sections -nostdlib -m elf_i386_fbsd -e start -Ttext 0x7c00 -o gptldr.out gptldr.o
objcopy -S -O binary gptldr.out gptldr.bin
make: don't know how to make /usr/src/sys/boot/i386/gptzfsboot/../btx/lib/crt0.o. Stop

Any ideas ?

 

[gkontos]

Went through the whole world + kernel rebuild with the patch and it appears that it is now working. Booting with first disk disconnected: (vm image always)

> zpool status
  pool: zroot
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
	the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scrub: none requested
config:

	NAME           STATE     READ WRITE CKSUM
	zroot          DEGRADED     0     0     0
	  mirror       DEGRADED     0     0     0
	    gpt/disk0  UNAVAIL     12   734     0  cannot open
	    gpt/disk1  ONLINE       0     0     0

errors: No known data errors

> uname -a
FreeBSD  8.1-RELEASE FreeBSD 8.1-RELEASE #0: Fri Aug  6 20:26:51 UTC 2010     gkontos@:/usr/obj/usr/src/sys/GENERIC  amd64

After rebooting to connect the first disk again the pool went online without resilvering so I had to do it manually:

# zpool status
  pool: zroot
 state: ONLINE
 scrub: resilver completed after 0h0m with 0 errors on Fri Aug  6 21:00:40 2010
config:

	NAME           STATE     READ WRITE CKSUM
	zroot          ONLINE       0     0     0
	  mirror       ONLINE       0     0     0
	    gpt/disk0  ONLINE       0     0     0  374K resilvered
	    gpt/disk1  ONLINE       0     0     0  212K resilvered

errors: No known data errors

George

 

[ScottJ97]

The patch fixed one problem for me but I hit another right afterward. I'm running under Sun VirtualBox.

BTX loader 1.00  BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive A: is disk0
BIOS drive C: is disk1
BIOS 639kB/1047488kB available memory

FreeBSD/i386 ZFS enabled bootstrap loader, Revision 1.1
(root@almeida.cse.buffalo.edu, Mon Jul 19 01:59:01 UTC 2010)
Consoles: internal video/keyboard
BIOS drive A: is disk0
BIOS drive C: is disk1
FATAL: int13_harddisk: function 42. Can't use 64bits lba

This time, again, it only happens when the first disk is disconnected. With
the first disk alone (second disk disconnected), it works as expected.

 

[ScottJ97]

I tried the patch on bare metal and got a different result, but still broken.

Like before, I can boot from either disk so long as both are connected, and I can boot from disk #1 with disk #2 unplugged. However booting from disk #2 with disk #1 unplugged gives this:

Can't work out which disk we are booting from
Guessed BIOS device 0xffffffff not found by probes, defaulting to disk0:

Can't load 'kernel'

Followed by the loader prompt.

 

[spork]

I'm seeing the same thing on a clean 8.1 install in virtualbox on a raidz. Have not tested a mirror install yet. For giggles I also tried setting up vb to use ata and scsi with the same result.

BTX loader 1.00  BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive A: is disk0
BIOS drive C: is disk1
BIOS 639kB/1047488kB available memory

FreeBSD/i386 ZFS enabled bootstrap loader, Revision 1.1
(root@almeida.cse.buffalo.edu, Mon Jul 19 01:59:01 UTC 2010)
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
FATAL: int13_harddisk: function 42. Can't use 64bits lba

I did apply the patch and then used gpart to replace the bootcode, same result.

I have a number of real (non virtual) production hosts running 8.1 after an 8.0 upgrade, but have not had any disk failures that would have presented an opportunity to "test" this behavior.

 

[kpa]

I tried the patch on bare metal and got a different result, but still broken.

Like before, I can boot from either disk so long as both are connected, and I can boot from disk #1 with disk #2 unplugged. However booting from disk #2 with disk #1 unplugged gives this:

Can't work out which disk we are booting from
Guessed BIOS device 0xffffffff not found by probes, defaulting to disk0:

Can't load 'kernel'

Followed by the loader prompt.

Same result here when I disconnect the first drive and try to boot.

 

[gkontos]

Were you able to install the patch with the instructions posted ?

I had to go through the whole buildworld && buildkernel procedure. Also I have tested this on a bare machine without any issues.

George

 

[mm@]

My instructions missed reinstalling zfsloader (+recompiling libzfsboot), I have edited my previous post: http://forums.freebsd.org/showpost.php?p=96194&postcount=9

 

[ScottJ97]

Thanks Martin, that did it. After properly rebuilding everything, both my VM install and my bare metal install are fully functional.

 

[kpa]

Further testing on my system reveals that I can boot with the 1st disc disconnected if I set up the drives in IDE mode in BIOS but not if I set them to AHCI. Must be some BIOS bug with my board (Intel D975xbx2).

 

[retheoff]

I am getting the same error gkontos did on the step to make zfsloader.

../btx/lib/crt0.o: No such file or directory[code]  

I tried going through the buildworld & kernel & install after applying the patch, system boots fine, but still cannot make zfsloader.  Thought I'd test the new kernel install, but booting still won't work without the first drive connected. (although that was expected)  

What is this [FILE]crt0.o[/FILE] anyway?  I looked in my system, and can only find a crt1.o file.  Couldn't I do a "make obj" on this? But I don't know where to do that?

Anyway, any help is appreciated.

 

[ScottJ97]

I am getting the same error gkontos did on the step to make zfsloader. "../btx/lib/crt0.o: No such file or directory". I tried going through the buildworld & kernel & install after applying the patch, system boots fine, but still cannot make zfsloader. Thought I'd test the new kernel install, but booting still won't work without the first drive connected. (although that was expected)

What is this crt0.o anyway? I looked in my system, and can only find a crt1.o file. Couldn't I do a "make obj" on this? But I don't know where to do that?

Anyway, any help is appreciated.

I got a handful of errors like this while rebuilding with this patch. In each case, I cd'd to the directory where it was looking for this file (../btx/lib in this case) and ran make for the missing file ('make crt0.o' in this case). Then go back and retry.

After 2-4 of these, it eventually finished building everything.

 

[retheoff]

Scott thank you!
Actually, I ran into the same error though, at first. I think the command to make clean under zfsloader removes the crt0.o file. I also didn't notice that the file is really in /usr/obj/usr/src....../btx/lib. Anyway, I went to ../btx/lib, ran [cmd=]make crt0.ocmd] and it doesn't not create the file right there, it's way under the /usr/obj dir.

After that, and running into a few other errors like you mentioned, I went to to the [cmd=]make clean depend ; make[/cmd] under zfsloader. Same issue. Started over yet again, make the crt0.o, but this time, just run make under zfsloader without a make clean, and all went fine after that.

Thanks for the help!! I finally got FreeBSD running the way I always wanted!

 

[ronnylov]

make: don't know how to make /usr/src/sys/boot/i386/zfsloader/../../ficl/libficl.a. Stop

I get this error. What to do?

 

[retheoff]

@ronnylov
That's the same thing I ran into, but that was easily fixed. Just change to that ficl directory and make it. Assuming you are currently in the zfsloader directory:

cd ../../ficl
make clean 
make depend
make

I ran into this one, and I think also saw one for "libi386", which I just ran the same procedure.

 

[ronnylov]

I tried the rebuilding world according to http://www.freebsd.org/doc/handbook/makeworld.html but mergemaster -p did not work in single user mode (it did not recognize the command) so I did not preceed with make installworld and mergemaster.

However after rebooting I could compile according to the steps in post #9 and ramake crt0.o according to post #22 by avoiding make clean in the zfsloader directory then there were no more errors.

I have not yet tested to remove one of the mirrored boot disks to see if the patch worked.
(I have a new installed FreeBSD 8.1 RELEASE amd64 system). I am happy I could boot at all after all these steps...