HOWTO: FreeBSD ZFS Madness

volatilevoid said:
Do I need to have a separate SLOG device for my ZIL when using a pure SSD pool? I've read contradictory information on that. Some say yes, some say no. :OOO

Do you expect that the required write bandwidth will exceed what the SSD can do? What is your use case like, lots of NFS writes?
 
Well the ZIL won't be used unless there are syncronous writes that need to be journaled for proper disaster recovery. You'll have to figure out if you have any applications that will use synchronous writes. NFS is one of them and it's usually the main reason to use a dedicated device for the ZIL. If you know that the ZIL isn't going to get many writes there's no point in using a dedicated device for it.
 
kpa said:
Well the ZIL won't be used unless there are syncronous writes that need to be journaled for proper disaster recovery. You'll have to figure out if you have any applications that will use synchronous writes. NFS is one of them and it's usually the main reason to use a dedicated device for the ZIL. If you know that the ZIL isn't going to get many writes there's no point in using a dedicated device for it.
Thank you for your explanation. I also read an article on the technical background which was very interesting. I don't think that I'll use many applications which depend on synchronous writes so I'll try my luck without a dedicated device for the ZIL.
 
Hi,

I'm trying to boot using BE with no success. I created a BE using beadm as described here, but I can't boot on BE system with FreeBSD boot loader. Let's suppose I had a problem after my reboot, so can't I boot on my last backup?

Thanks
 
markbsd2 said:
I'm trying to boot using BE with no success. I created a BE using beadm as described here, but I can't boot on BE system with FreeBSD boot loader. Let's suppose I had a problem after my reboot, so can't I boot on my last backup?

FreeBSD Boot Loader does not support MENU for chosing BE at Boot, but You can have such menu by installing PC-BSD or TrueOS from iXsystems (both are based on FreeBSD) because they use a modified GRUB2 as Boot Loader.
 
Hi @vermaden,

That's the point. I don't want to use PC-BSD or TrueOS. I'd like to use FreeBSD. I tried to install GRUB on FreeBSD 10+ZFZ, and it doesn't work. Here are my steps:

Code:
root@x:~ # sysctl kern.geom.debugflags=16
kern.geom.debugflags: 0 -> 16

root@x:~ # gpart show
=>      34  19844973  ada0  GPT  (9.5G)
        34      1024     1  freebsd-boot  (512K)
      1058   4194304     2  freebsd-swap  (2.0G)
   4195362  15649645     3  freebsd-zfs  (7.5G)

root@x:~ # gpart delete -i 1 ada0
ada0p1 deleted

root@x:~ # gpart add -t bios-boot ada0
ada0p1 added

root@x:~ # grub-install --force /dev/ada0
Installing for i386-pc platform.
Installation finished. No error reported.

root@x:~ # grub-mkconfig -o /boot/grub/grub.cfg /dev/ada0
Generating grub configuration file ...
done
Installing GRUB to gptid/de790ea3-02d8-11e4-b0db-080027b9b042
Installing for i386-pc platform.
grub-install: error: cannot open `/dev/gptid/de790ea3-02d8-11e4-b0db-080027b9b042': Operation not permitted.

Thanks.
 
Last edited by a moderator:
markbsd2 said:
That's the point. I don't want to use PC-BSD or TrueOS. I'd like to use FreeBSD. I tried to install grub on FreeBSD 10+ZFZ, and it doens't work. Here are my steps:

Code:
# gpart show
=>      34  19844973  ada0  GPT  (9.5G)
        34      1024     1  freebsd-boot  (512K)
      1058   4194304     2  freebsd-swap  (2.0G)
   4195362  15649645     3  freebsd-zfs  (7.5G)

From what I recall GRUB2 needs at least a 1M first partition. Also PC-BSD/TrueOS uses a modified version of GRUB2 which can be found in ports as sysutils/grub2-pcbsd. You may also install TrueOS to check how and what configuration for GRUB2 it generates. You may also check PC-BSD installer source for needed commands.
 
markbsd2 said:
Hi @vermaden,

That's the point. I don't want to use PC-BSD or TrueOS. I'd like to use FreeBSD. I tried to install GRUB on FreeBSD 10+ZFZ, and it doesn't work. Here are my steps:

Code:
root@x:~ # sysctl kern.geom.debugflags=16
kern.geom.debugflags: 0 -> 16

root@x:~ # gpart show
=>      34  19844973  ada0  GPT  (9.5G)
        34      1024     1  freebsd-boot  (512K)
      1058   4194304     2  freebsd-swap  (2.0G)
   4195362  15649645     3  freebsd-zfs  (7.5G)

root@x:~ # gpart delete -i 1 ada0
ada0p1 deleted

root@x:~ # gpart add -t bios-boot ada0
ada0p1 added

root@x:~ # grub-install --force /dev/ada0
Installing for i386-pc platform.
Installation finished. No error reported.

root@x:~ # grub-mkconfig -o /boot/grub/grub.cfg /dev/ada0
Generating grub configuration file ...
done
Installing GRUB to gptid/de790ea3-02d8-11e4-b0db-080027b9b042
Installing for i386-pc platform.
grub-install: error: cannot open `/dev/gptid/de790ea3-02d8-11e4-b0db-080027b9b042': Operation not permitted.

Thanks.
Hey, @markbsd2, DON'T rely on the grub-mkconfig command. It screws things up and so far doesn't work for some reasons!
And basically: what message do you have on reboot, after your grub2 installation "finished with no errors"?
Do you get a GRUB prompt at least (not grub-rescue!)?
 
Last edited by a moderator:
Anyway, you can read my experience with GRUB2 and some tips on it.
Interesting, you were able to create the bios-boot partition using -t bios-boot. I had to use the command #gpart add -t \!21686148-6449-6E6F-744E-656564454649 -s <size> -i <index> <geom>.

Another thing I had to do was to create a "custom" GRUB image including both zfs and part_gpt modules (not there by default): grub-install --modules="part_gpt zfs" /dev/ada0, without which all I could get was the grub rescue prompt, which is almost useless.

...Whether or not you need the above, you will definitely have to create your own /boot/grub/grub.cfg file by hand, as grub-mkconfig can't create a working one. The truth is, this UUID stuff grub-mkconfig is trying to figure out is not going to be of any use, because grub2 just can't figure out ZFS pool using these UUIDs correctly. But if you point it to the right partition, you can see your ZFS pool file structure... So you will generally need something like this:
Code:
menuentry 'FreeBSD-10.0 on ZFS' {
insmod zfs
insmod part_gpt
set root=(hd0,gpt2)
	kfreebsd /ROOT/working/@/boot/kernel/kernel
	kfreebsd_loadenv /ROOT/working/@/boot/device.hints
	kfreebsd_module /ROOT/working/@/boot/zfs/zpool.cache -type /boot/zfs/zpool.cache
	kfreebsd_module_elf /ROOT/working/@/boot/kernel/opensolaris.ko
	kfreebsd_module_elf /ROOT/working/@/boot/kernel/zfs.ko
	kfreebsd_module_elf /ROOT/working/@/boot/kernel/linux.ko
	kfreebsd_module_elf /ROOT/working/@/boot/modules/nvidia.ko
	set kFreeBSD.vfs.root.mountfrom=zfs:mypool/ROOT/working
	set kFreeBSD.vfs.root.mountfrom.options=rw
}
Make sure to use the right grub device (mine is (hd0,gpt2)) and the right PATH in your grub.cfg ;)

And yes, GRUB2 IS great and much better than the FreeBSD BTX loader. With GRUB2 you can do a lot more and tell it to do what you can't tell BTX to do. Not to mention that it boots quicker... This way you can even have several OSes (NOT Oracle Solaris, though!) in your zpool/ROOT and boot each one separately using GRUB, because you won't even need a bootfs porperty to be set for that to work... but since FreeBSD is the best one among them, I only do this for testing purposes :D
 
free-and-bsd said:
Interesting, you were able to create the bios-boot partition using -t bios-boot. I had to use the command
Code:
#gpart add -t \!21686148-6449-6E6F-744E-656564454649 -s <size> -i <index> <geom>

Its one of the options now, from man gpart:
Code:
     bios-boot        The system partition dedicated to second stage of the
                      boot loader program.  Usually it is used by the GRUB 2
                      loader for GPT partitioning schemes.  The scheme-spe‐
                      cific type is "!21686148-6449-6E6F-744E-656564454649".
 
That's a nice job by gpart developers :)!
How quickly they respond to the current needs, that's great. I must have missed this because of using the installation media version of gpart... or maybe because I didn't look for it ;)?
...Given this small improvement, FreeBSD is keeping its status of the most user-friendly among the OSs.
 
free-and-bsd said:
That's a nice job by gpart developers :)!
How quickly they respond to the current needs, that's great. I must have missed this because of using the installation media version of gpart... or maybe because I didn't look for it ;)?
...Given this small improvement, FreeBSD is keeping its status of the most user-friendly among the OSs.

They have added some more popular types, a lot more are defined in the FreeBSD's fdisk source tree [1].

Its albo PITA to convert 0x0C - DOS or Windows 95 with 32 bit FAT (LBA) - to !12 to use that as an argument for gpart command.

[1] Complete partition information in FreeBSD's fdisk source tree is that:
Code:
% grep -m 1 -A 90 part_types /usr/src/sbin/fdisk/fdisk.c
static const char *const part_types[256] = {
        [0x00] = "unused",
        [0x01] = "Primary DOS with 12 bit FAT",
        [0x02] = "XENIX / file system",
        [0x03] = "XENIX /usr file system",
        [0x04] = "Primary DOS with 16 bit FAT (< 32MB)",
        [0x05] = "Extended DOS",
        [0x06] = "Primary DOS, 16 bit FAT (>= 32MB)",
        [0x07] = "NTFS, OS/2 HPFS, QNX-2 (16 bit) or Advanced UNIX",
        [0x08] = "AIX file system or SplitDrive",
        [0x09] = "AIX boot partition or Coherent",
        [0x0A] = "OS/2 Boot Manager, OPUS or Coherent swap",
        [0x0B] = "DOS or Windows 95 with 32 bit FAT",
        [0x0C] = "DOS or Windows 95 with 32 bit FAT (LBA)",
        [0x0E] = "Primary 'big' DOS (>= 32MB, LBA)",
        [0x0F] = "Extended DOS (LBA)",
        [0x10] = "OPUS",
        [0x11] = "OS/2 BM: hidden DOS with 12-bit FAT",
        [0x12] = "Compaq diagnostics",
        [0x14] = "OS/2 BM: hidden DOS with 16-bit FAT (< 32MB)",
        [0x16] = "OS/2 BM: hidden DOS with 16-bit FAT (>= 32MB)",
        [0x17] = "OS/2 BM: hidden IFS (e.g. HPFS)",
        [0x18] = "AST Windows swapfile",
        [0x1b] = "ASUS Recovery partition (NTFS)",
        [0x24] = "NEC DOS",
        [0x3C] = "PartitionMagic recovery",
        [0x39] = "plan9",
        [0x40] = "VENIX 286",
        [0x41] = "Linux/MINIX (sharing disk with DRDOS)",
        [0x42] = "SFS or Linux swap (sharing disk with DRDOS)",
        [0x43] = "Linux native (sharing disk with DRDOS)",
        [0x4D] = "QNX 4.2 Primary",
        [0x4E] = "QNX 4.2 Secondary",
        [0x4F] = "QNX 4.2 Tertiary",
        [0x50] = "DM (disk manager)",
        [0x51] = "DM6 Aux1 (or Novell)",
        [0x52] = "CP/M or Microport SysV/AT",
        [0x53] = "DM6 Aux3",
        [0x54] = "DM6",
        [0x55] = "EZ-Drive (disk manager)",
        [0x56] = "Golden Bow (disk manager)",
        [0x5c] = "Priam Edisk (disk manager)", /* according to S. Widlake */
        [0x61] = "SpeedStor",
        [0x63] = "System V/386 (such as ISC UNIX), GNU HURD or Mach",
        [0x64] = "Novell Netware/286 2.xx",
        [0x65] = "Novell Netware/386 3.xx",
        [0x70] = "DiskSecure Multi-Boot",
        [0x75] = "PCIX",
        [0x77] = "QNX4.x",
        [0x78] = "QNX4.x 2nd part",
        [0x79] = "QNX4.x 3rd part",
        [0x80] = "Minix until 1.4a",
        [0x81] = "Minix since 1.4b, early Linux partition or Mitac disk manager",
        [0x82] = "Linux swap or Solaris x86",
        [0x83] = "Linux native",
        [0x84] = "OS/2 hidden C: drive",
        [0x85] = "Linux extended",
        [0x86] = "NTFS volume set??",
        [0x87] = "NTFS volume set??",
        [0x93] = "Amoeba file system",
        [0x94] = "Amoeba bad block table",
        [0x9F] = "BSD/OS",
        [0xA0] = "Suspend to Disk",
        [0xA5] = "FreeBSD/NetBSD/386BSD",
        [0xA6] = "OpenBSD",
        [0xA7] = "NeXTSTEP",
        [0xA9] = "NetBSD",
        [0xAC] = "IBM JFS",
        [0xAF] = "HFS+",
        [0xB7] = "BSDI BSD/386 file system",
        [0xB8] = "BSDI BSD/386 swap",
        [0xBE] = "Solaris x86 boot",
        [0xBF] = "Solaris x86 (new)",
        [0xC1] = "DRDOS/sec with 12-bit FAT",
        [0xC4] = "DRDOS/sec with 16-bit FAT (< 32MB)",
        [0xC6] = "DRDOS/sec with 16-bit FAT (>= 32MB)",
        [0xC7] = "Syrinx",
        [0xDB] = "CP/M, Concurrent CP/M, Concurrent DOS or CTOS",
        [0xDE] = "DELL Utilities - FAT filesystem",
        [0xE1] = "DOS access or SpeedStor with 12-bit FAT extended partition",
        [0xE3] = "DOS R/O or SpeedStor",
        [0xE4] = "SpeedStor with 16-bit FAT extended partition < 1024 cyl.",
        [0xEB] = "BeOS file system",
        [0xEE] = "EFI GPT",
        [0xEF] = "EFI System Partition",
        [0xF1] = "SpeedStor",
        [0xF2] = "DOS 3.3+ Secondary",
        [0xF4] = "SpeedStor large partition",
        [0xFE] = "SpeedStor >1024 cyl. or LANstep",
        [0xFF] = "Xenix bad blocks table",
};
 
Hi,

Just to update this topic. It's just worked :)

@free-and-bsd, I did what you suggested, ran grub-install --modules="part_gpt zfs" /dev/ada0.

After that i used your sample config with just one change. I replaced set root=(hd0,gpt2) with search --no-floppy -s -l zroot

I tried this one on FreeBSD 10.0-RELEASE. Now i'll update my server to STABLE and I'll see if it keep working.

Thanks for your help!
 
Last edited by a moderator:
as I understand it, it doesn't include zfs module by default. And without it grub2 fails to figure out the partition, on which all its files and modules (including zfs!)are situated, so it escapes to rescue prompt with no way out of it.

...And so, search --no-floppy -s -l zroot works?
Great!! For the device name independent config this is preferable. The point there, i understand, is to avoid using UUIDs with grub2 for a ZFS pool. Don't know if this is related or not, but in Linux, if you want to import a FreeBSD zfs pool, you need
Code:
zpool import -d /dev/disk/by-partuuid ...
without which it will report FAULTED zpool instead of working one. Perhaps, grub2 is more linux-minded in this regard, too. So the grub-mkconfig that comes with sysutils/grub2 is still rather linux-minded. To finalize it, I must add, that for Linux on ZFS the script works as poorly as it does for FreeBSD on ZFS... But since these things are nearly "experimental", one doesn't have to complain ;))
 
wblock@ said:
-STABLE is to FreeBSD what Service Pack 3 is to others.
Really? Then I must have done something wrong. When the new release 10 was out, something changed there with what then became the STABLE release. I updated to STABLE and ended up with kernel flooding the screen with some error message from one of the HDD controllers. It was even impossible to log in. Or may it have been 10-STABLE?
 
Maybe a problem with a particular driver. -STABLE is not the shaky alpha people associate with early releases of commercial operating systems. Most of the time, it's just a bug-fixed version of -RELEASE.
 
wblock@ said:
Maybe a problem with a particular driver. -STABLE is not the shaky alpha people associate with early releases of commercial operating systems. Most of the time, it's just a bug-fixed version of -RELEASE.
And if I have updated the RELEASE running freebsd-update (last time today) -- have I thus updated to STABLE?
 
Back
Top