mount zfs /etc on boot

M

mamalos

Hi guys,

I am stuck in a situation which I think is easy, but I cannot see the solution that must be in front of me. I installed my fbsd 8-stable system on a ufs filesystem, and then I created a zfs pool (on another partition). On the pool I created a few filesystems (/etc,/var/,/usr/,/usr/local and more). I copied my files from my initial /etc to my zfs-pool /etc (my pool is called msi_root), and on /etc/fstab I have:

Code: 
# Device		Mountpoint	FStype	Options		Dump	Pass#
/dev/ad4s2a		/		ufs	rw		1	1
msi_root/etc		/etc		zfs	rw		0	0
msi_root/home		/home		zfs	rw		0	0
msi_root/root		/root		zfs	rw		0	0
msi_root/usr		/usr		zfs	rw		0	0
msi_root/usr/local	/usr/local	zfs	rw		0	0
msi_root/usr/local/share	/usr/local/share	zfs	rw		0	0
msi_root/usr/obj	/usr/obj	zfs	rw		0	0
msi_root/usr/ports	/usr/ports	zfs	rw		0	0
msi_root/usr/share	/usr/share	zfs	rw		0	0
msi_root/usr/src	/usr/src	zfs	rw		0	0
msi_root/var		/var		zfs	rw		0	0
msi_root/var/db	/var/db		zfs	rw		0	0
msi_root/var/log	/var/log	zfs	rw		0	0
/dev/ad4s2b		none		swap	sw		0	0
/dev/ad4s1		/mnt/nt		ntfs	ro,noauto,-CUTF-8 0	0
/dev/acd0		/cdrom		cd9660	ro,noauto	0	0

My problem is that when my system boots, if first reads my initial /etc (which is later hidden by my zfs /etc), so things I write on msi_root/etc/rc.conf, for example, are not valid unless I write them on my initial /etc/rc.conf.

Which means that I have to shutdown to single user mode, unmount my zfs/etc filesystem and write my changes on my initial /etc/rc.conf in order for me to work.

I know I am missing something but I don't know what it is. What should I do so as to make my system "see" the zfs /etc on boot? (this is not my only problem. When I log onto single user mode, my system sees only my initial /etc, /usr, blabla filesystems. The kernel loaded is always the latest (I follow stable). The binaries and libraries located in /usr/ are those that I installed the first time. I assume that at some moment those two will become out of sync and I won't be able to operate my system).

Please, don't instruct me to install zfsboot. I know how to do this, I had my system running like this in the past, but I encountered a problem during a zfs pool upgrade that made me reinstall my system from scratch (it wouldn't update the mbr, no matter what). I prefer to be able to boot from a ufs system, and keep /etc/, /usr and other filesystems stored in zfs.

Thank you all for your time in advance.
 
...and a few more things I forgot to mention before:

Code: 
$ zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
msi_root                  76.2G  42.9G    21K  /msi_root
msi_root/etc              1.04M  42.9G  1.04M  /etc
msi_root/home             66.2G  42.9G  66.2G  /home
msi_root/root              414M  42.9G   414M  /root
msi_root/usr              9.40G  42.9G   457M  /usr
msi_root/usr/local        5.91G  42.9G  5.26G  /usr/local
msi_root/usr/local/share   666M  42.9G   666M  /usr/local/share
msi_root/usr/obj           989M  42.9G   989M  /usr/obj
msi_root/usr/ports        1.74G  42.9G  1.74G  /usr/ports
msi_root/usr/share        30.4M  42.9G  30.4M  /usr/share
msi_root/usr/src           310M  42.9G   310M  /usr/src
msi_root/var               211M  42.9G  96.7M  /var
msi_root/var/db            114M  42.9G   114M  /var/db
msi_root/var/log           401K  42.9G   401K  /var/log

Code: 
$ zpool status
  pool: msi_root
 state: ONLINE
 scrub: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	msi_root    ONLINE       0     0     0
	  ad4s2d    ONLINE       0     0     0

errors: No known data errors
Code: 
$ uname -a
FreeBSD mymachine 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #0: Tue Jan 11 17:46:17 EET 2011     root@mymachine:/usr/obj/usr/src/sys/GENERIC  
amd64
 
Hi,

FreeBSD has special options built in to allow booting from ZFS (which would include critical OS directories such as /etc). Ie "vfs.root.mountfrom="
I think the issue you have is the result of your config being a bit broken/wrong. I think you will need to either boot from ZFS or not, not have a hybrid setup as you are trying to do.
If you want to boot from ZFS there is a guide here:

http://wiki.freebsd.org/ZFSOnRoot

Cheers Andy.
 
Andy, thanx for your answer.

As I've already mentioned, I know how to boot from zfs (I had my system booting from zfs for a long time), and this is not my intention now. What I want is to have more than one filesystems on my system, and obviously have them "online" once the kernel is loaded. I don't wish to have them ready before the kernel is loaded, just after the kernel is loaded. Can't I achieve that? I used to do it with ufs filesystems in the past, and I don't recall having such problems. I am sure it can be done, and that I can't achieve it cos I am missing something obvious (and I don't mean booting via zfs).

Thanx again
 
What I want is to have more than one filesystems on my system, and obviously have them "online" once the kernel is loaded. I don't wish to have them ready before the kernel is loaded, just after the kernel is loaded. Can't I achieve that? I used to do it with ufs filesystems in the past, and I don't recall having such problems
Click to expand...

There is no problem with having as many seperate zfs/ufs filesystems as you want. All of which will be available immediately once the system has booted. However, it doesn't quite work with /etc.

Imagine you tried to make a seperate UFS partition for /etc. If you tried to mount that the same way as any other UFS partition, you would put it in /etc/fstab. You should instantly see the problem with that.

It appears to me that FreeBSD expects /etc to be part of the root filesystem, available as soon as root has been mounted by the boot process, and before mounting extra local filesystems. I think your only option is to leave /etc on your root UFS filesystem.

I know I am missing something but I don't know what it is. What should I do so as to make my system "see" the zfs /etc on boot? (this is not my only problem. When I log onto single user mode, my system sees only my initial /etc, /usr, blabla filesystems. The kernel loaded is always the latest (I follow stable). The binaries and libraries located in /usr/ are those that I installed the first time. I assume that at some moment those two will become out of sync and I won't be able to operate my system).
Click to expand...

Single user mode does not mount any other filesystems other than /. However, any updates you do to the kernel and core system utilities should end up in /boot,/sbin,/bin,etc on your UFS partition, so single mode should work fine and you should be able to mount your ZFS /usr to access the updated userland utilities if required.

It's for reasons like this that I tend to follow the same setup as Phoenix, and keep /, /etc, and most of /usr on the root filesystem. That way the system is always usable, even in single user mode with only / mounted. (I actually still have a 'mypool/usr' filesystem to keep the ZFS layout neat and tidy, but its 'canmount' property is no).
My root fs is usually read-only and I just keep a recent dump of it somewhere incase I need to restore from scratch.
 
Hey guys,

thanx for taking the time to drop your thoughts, you've been most helpful.

usdmatt, regarding your comment:

Imagine you tried to make a seperate UFS partition for /etc. If you tried to mount that the same way as any other UFS partition, you would put it in /etc/fstab. You should instantly see the problem with that
Click to expand...

I understood the nature of this restriction, but I had this unexplainable feeling :) that zfs would be able to surpass this "shortcoming" somehow through some of its many, "magic" options (that I haven't used until now).

You have both made your points clear, and I will follow your advice to keep my base system (except /var I think, which seems harmless to move elsewhere) on my root fs. What I didn't quite understand was how phoenix's setup is (readonly-ufs rootfs, and some recent copy of it on zfs?), cause I would like to have my "base system" (excluding some subfolders of /var (or all of it) and /etc) mounted ro and the rest of it rw. If there's a thread you could me address to, it would be very gentle of you.

Thank you once more for your thought-sharing and your time guys.
 
Yeah, there's no problem having /var separate. You probably already know that sysinstall makes /var a seperate filesystem by default if you use the auto option.

In regard to the phoenix comment, I wasn't very clear but I was just referring to having / and the important parts of /usr on UFS. I'm not sure what he does as far as backing up the root FS.

I personally boot off a read only USB stick and have a copy of that created with dump(8) in case I need to restore.

Just for interest, this is the layout of my current ZFS system. As I mentioned previously, the filesystem with a mountpoint of /usr isn't actually mounted, it just keeps the sub-filesystems tidy.


Code: 
storage                      1.58T  3.76T  29.9K  /storage
storage/documents             173G  3.76T   173G  /storage/documents
storage/media                1.41T  3.76T  47.1K  /storage/media
storage/media/movies         1.27T  3.76T  1.27T  /storage/media/movies
storage/media/music          45.4G  3.76T  45.2G  /storage/media/music
storage/media/photos         96.6G  3.76T  96.5G  /storage/media/photos
storage/tmp                  32.9K  3.76T  32.9K  /tmp
storage/usr                  1.41G  3.76T  26.9K  /usr
storage/usr/home             39.6K  3.76T  39.6K  /usr/home
storage/usr/local            1.07G  3.76T  1.07G  /usr/local
storage/usr/ports             343M  3.76T   297M  /usr/ports
storage/usr/ports/distfiles  45.5M  3.76T  45.5M  /usr/ports/distfiles
storage/usr/utils             350K  3.76T  34.4K  /usr/utils
storage/var                   201M  3.76T   142M  /var

I used to have /tmp as a memory disk but got fed up of it running out of space every time a program used it to untar something.
 
I don't have a read-only / filesystem, but it's not too far off what I use. Considering how rarely an installworld/freebsd-update is done, I could probably flip the read-only bit without any issues. Just haven't done so yet. :)

Basically, I put / and /usr onto a single UFS filesystem with SoftUpdates disabled, on top of gmirror. That's the entire base OS, so it makes recovery very easy as you have access to everything in single-user mode.

Then I create a ZFS pool and create the following filesystems:
  • /home
  • /var
  • /usr/local
  • /usr/ports
  • /usr/src
  • /usr/obj
  • whatever else is needed for that box
 
Thanx both for the advices, and sorry for the delayed answer!

Phoenix, the reason you don't have soft-updates enabled on your / and /usr is speed or information loss oriented? Cause, if I 've understood correctly (as far as soft-updates is concerned) having it enabled makes "certain" that after a system crash the filesystem will be consistent; through this process, some write-ops might get lost on the other hand.

I am asking you this, because on mail servers I am administering, I have soft-updates enabled (on the maildir folder), but I am not sure if after a power loss some data (user's mails) may be lost in the sake of filesystem consistency. If so, is the "zfs-way" considered more safe in data loss?

Thanx again and sorry for slightly diverting from the main issue (which is resolved), but the question came out naturally :). In case you don't wish to continue this thread, please use pm.
 
SoftUpdates will makes things faster as it reorders things in RAM before writing them out to disk, and it can optimise out some operations (for instance, create new file, rename file, delete file without writing data to it and the file operations will never hit the disk). However, if the system crashes, you can lose file data (file metadata and filesystem will be consistent, but file contents may be lost), and have to run through a (possibly long) fsck.

/ and /usr filesystems rarely change, with config files being the only ones written to on a regular basis; you really don't want those to get corrupted. SoftUpdates turned off slows things down, but makes the file contents safer. Still have to go through long fsck after crashes, though.

So it's a trade-off: speed vs file contents safety.

ZFS is better in that all write operations are atomic: either you have the old file data or the new file data, never anything in between (as in new file name but old contents, or empty files). But it's "slower" due to all the extra consistency checks.

So you need to decide what's most important: absolute raw write speed (UFS mounted async) or absolute data safety (use ZFS) with a whole spectrum of options in between (UFS w/SoftUpdates, UFS w/SoftUpdates Journalled, UFS+SUJ+GJournal, etc).
 
Ok phoenix, you've been most explanatory. I thought the thing with speed and soft-updates was kind-of the other way round, but I assume I remembered wrong. The thing that interests me (as far as my email users is concerned) is no data-loss. I couldn't realize from your sayings what you said about this issue regarding zfs. Let's say that an email reaches smtpd (through the network) and while the copy is made in the queue (active queue, still not delivered/forwarded) the server crashes due to power outage. What will happen to the email? Will the sender smptd process be informed that the client went down during the copy procedure or will the sender think that the mail was delivered just fine?

Thanx again!
 
You must log in or register to reply here.
Back
Top