USB memstick installation mini-HOWTO (discussion)

[SirDice]

[Mod: modding my own posts Split off from Thread usb-memstick-installation-mini-howto.87132 ]

USR is mounted read-only. Which can be changed whenever there's need to install additional pkg -- see below.

Package/port prefix is /usr/local/. Everything else in /usr is actually part of the base (/usr/bin,/usr/sbin, /usr/lib etc).

/dev/gpt/USR /usr ufs ro,noatime 0 0

If the filesystem is read-only the atime cannot be modified anyway (needs write access for that). So no need to explicitly state it.

 

[free-and-bsd]

Package/port prefix is /usr/local/. Everything else in /usr is actually part of the base (/usr/bin,/usr/sbin, /usr/lib etc).


If the filesystem is read-only the atime cannot be modified anyway (needs write access for that). So no need to explicitly state it.

Right. The idea was to mount as much as possible read-only. Though it seems the /usr part of base is not that big... but I didn't want to micro-manage it and used /usr for them all.

 

[SirDice]

The idea was to mount as much as possible read-only.

Then why isn't the root filesystem read-only?

but I didn't want to micro-manage it and used /usr for them all.

No need to micromanage, just mount that filesystem on /usr/local. The difference between base OS and 3rd party applications (ports/package) is very clear cut.

There's also very little need to mount /usr/local specifically read-only when everything else is read-write.

 

[Whattteva]

Right. The idea was to mount as much as possible read-only. Though it seems the /usr part of base is not that big... but I didn't want to micro-manage it and used /usr for them all.

This is linuxism. Reason why I'm using FreeBSD is to get away from stuff like this.

 

[SirDice]

Speaking of 3rd party applications, the pkg(8) database is stored in /var/db/pkg. As your /var is a memory filesystem every time you boot the system it will think there's nothing installed. Every time you want to install something it'll bootstrap pkg(7), and then install the package. Which it will promptly forget about when you reboot the system.

md    /var    mfs    rw,sync,noatime,-P,-F=/var.img    0    0

Sorry, not a memory filesytem (got thrown off by the md(4)). But there's no point in making this an image file backed filesystem. It's not going to reduce the writes to /var compared to having /var as a 'regular' filesystem. Writes to /var/ (/var/log/messages for example) will just be 'converted' into writes to the /var.img image file. It only adds complexity without any beneficial improvements.

 

[astyle]

I have successfully used the regular FreeBSD installer to install FreeBSD, complete with ZFS and ports and everything onto a bootable USB stick. The key is to see if the USB stick is visible to the installer. When you get a choice of the disk to install to (Your USB stick or the computer's internal SSD/HDD), just pick the USB stick.

I have successfully booted that installation on 2 different Windows machines - it does work fine.

One caveat: the USB stick should be seated rather firmly. Otherwise, hardware events that get logged to dmesg - that interferes with my shell, because the text pops up to the same TTY. My workaround to that was to note the IP address (from ifconfig ue0) and SSH into that from another machine. This is pretty useful for a few things I'm trying to do.

 

[free-and-bsd]

This is linuxism. Reason why I'm using FreeBSD is to get away from stuff like this.

Up to you For me stuff like THIS doesn't really matter much.... besides, I think you're mistaken. In OpenBSD suggested partitioning scheme also includes a separate partition for /usr... anyway.

 

[free-and-bsd]

Speaking of 3rd party applications, the pkg(8) database is stored in /var/db/pkg. As your /var is a memory filesystem every time you boot the system it will think there's nothing installed. Every time you want to install something it'll bootstrap pkg(7), and then install the package. Which it will promptly forget about when you reboot the system.

md    /var    mfs    rw,sync,noatime,-P,-F=/var.img    0    0

Sorry, not a memory filesytem (got thrown off by the md(4)). But there's no point in making this an image file backed filesystem. It's not going to reduce the writes to /var compared to having /var as a 'regular' filesystem. Writes to /var/ (/var/log/messages for example) will just be 'converted' into writes to the /var.img image file. It only adds complexity without any beneficial improvements.

Are you SURE? This is not the impression I got from man mdmfs. The idea there is a memory (temporary) file system that is POPULATED from the md image (not a blank ufs). But it never mentions the writes to mfs would then committed back to the md image...

EDIT: yes, this is true ((. I was mislead into thinking that FreeBSD mfs is like OpenBSD mfs. That one does NOT commit writes to the file from which it is populated.

 

[free-and-bsd]

Then why isn't the root filesystem read-only?

From my experience, this will make all the other mounts (/var etc) read-only. Would be glad to learn it is not so.

 

[Whattteva]

5. Do we need swap? Well, reasons in favor of it explain the need by the possible situation when system becomes unresponsive for some reason, so the dump then would be saved on swap… Do you need THAT for your USB flash based system? Given that simple reboot will bring the system back online, in this particular case swap would seem to be not so critical.

I wouldn't use any swap. It will kill the memstick fast. In fact, I think for any USB stick OS install to work, you really need to mount either root read-only or load the entire OS in RAM cause most memsticks just simply don't like being written to much without dying an early death.

 

[free-and-bsd]

I wouldn't use any swap. It will kill the memstick fast. In fact, I think for any USB stick OS install to work, you really need to mount either root read-only or load the entire OS in RAM cause most memsticks just simply don't like being written to much without dying an early death.

Absolutely. I even wonder if ram-based root filesystem is better still, than a disk-based root.

 

[SirDice]

I even wonder if ram-based root filesystem is better still, than a disk-based root.

It'll be slightly faster initially, until the filesystem cache catches up. Then most of it will be read from memory anyway. It's not going to make a difference in writes, as there will be very little to begin with, definitely no writes to a read-only filesystem.

Are you SURE? This is not the impression I got from man mdmfs.

Now you mentioned it, I'm definitely not sure. I think I got confused with mdconfig(8), which certainly would write to the image file.

 

[free-and-bsd]

Ok, good eye-openers here, SirDice You were right, this vnode-based mdmfs actually DOES write changes back to the file system. Too bad...
However! I've just tired various other options and finally came up with this modified /etc/fstab that seems to solve those problems:

/dev/gpt/Root    /    ufs    ro    1    1
/dev/gpt/USR    /usr    ufs    ro    0    0
#/dev/gpt/HOME    /home    ufs    rw,noatime    0    0
devfs    /dev    devfs    rw    0    0
md    /var    mfs    rw,-s60m    2    0
tmpfs    /tmp    tmpfs    rw,mode=1777,size=300M    0    0

What does that (seem to) achieve?
1. Read-only root and other disk-based fs.
2. Read-write mounted devfs(5) for which there is a special fstab entry. Thought that WAS necessary since our root is read-only and so is everything on it... not sure though. But this way it works (see below)
3. Read-write CLEAN /var which is populated on boot by system itself, so the necessary stuff is there, it seems. At least my Freeradius3 and isc-dhcp-server work fine with this configuration. I run a WiFi AP with EAP-TLS Radius-based auth.
So far, everything on the disk is mounted read-only, which was the original idea.
And when we need to install additional packages? Well I keep the original /var as /var.save, which in that case can be renamed to /var and mounted read-write, together with /usr and whatever else is needed. Once this done, bring things back to read-only.
The general idea is to make it look as much as possible like an embedded system.

Of course, I didn't try this with many apps, just my basic setup including (1) system itself, (2) isc-dhcpd, (3) Freeradius3 serving EAP-TLS for Wifi AP, (4) hostapd.
An OpenVPN server can run there as well. But for more complicated things like X --- I don't think that a USB memstick installation can be expected to bring us very far... my idea was to use it as OS for a home/small office router with some networking apps. Or a rescue disk... but for that I would rather use a USB with FreeBSD + SysRescCD (linux). Which is not complicated at all, but goes beyond the scope of this document.

 

[PMc]

From these, only root / needs be mounted rw. Trying to keep root size at minimal, I came by the figure of 2G. At that, there is no much writing to / partition anyway. But it has to be mounted rw, or else all the mounted stuff will be read-only.

I think it is possible to mount a r/w filesystem into a r/o filesystem. So it should be possible to mount / readonly.
I'm not sure that it really works, but it already happened to me accidentially in single-user when I brought in my filesystems and forgot to remount the root r/w.

Sorry, not a memory filesytem (got thrown off by the md(4)). But there's no point in making this an image file backed filesystem. It's not going to reduce the writes to /var compared to having /var as a 'regular' filesystem. Writes to /var/ (/var/log/messages for example) will just be 'converted' into writes to the /var.img image file. It only adds complexity without any beneficial improvements.

Then an option might be to copy the in-memory /var to the stick only occasionally. Maybe take a dump from it.
The risk is then still that things may run out of sync when not properly shutting down.

I tried such things long ago, and gave up on them because performance was a horror and things tended to crash or hang. Sticks have not gotten better in the meantime (rather the opposite) but at least our USB implementation has.

From my experience, this will make all the other mounts (/var etc) read-only. Would be glad to learn it is not so.

It is not so:

root@disp:~ # df /usr/src /usr/obj
Filesystem   1K-blocks    Used    Avail Capacity  Mounted on
build/sd/src  46694920  649772 46045148     1%    /usr/src
build/sd/obj  48890844 2845696 46045148     6%    /usr/obj
root@disp:~ # mkdir /usr/src/TMP
root@disp:~ # umount /usr/src
root@disp:~ # umount /usr/obj
root@disp:~ # mount -t zfs -o ro build/sd/src /mnt
root@disp:~ # rmdir /mnt/TMP
rmdir: /mnt/TMP: Read-only file system
root@disp:~ # mount -t zfs  build/sd/obj /mnt/TMP
root@disp:~ # mkdir /mnt/TMP/CRAP
root@disp:~ # rmdir /mnt/TMP/CRAP
root@disp:~ # umount /mnt/TMP
root@disp:~ # umount /mnt
root@disp:~ # zfs mount build/sd/obj
root@disp:~ # zfs mount build/sd/src
root@disp:~ # rmdir /usr/src/TMP

 

[Alain De Vos]

The "advantage" of var as memory disk is very unclear to me.

 

[free-and-bsd]

The "advantage" of var as memory disk is very unclear to me.

So you think writing logs to USB stick will just do fine?

 

[free-and-bsd]

Logs, pid files, sockets and I honestly don't know what other temporary stuff gets written to /var and where exactly.

 

[Alain De Vos]

Yeah i see,
In memory this is :

mdmfs -s 1G /var/log

But you can't use vnode-backed memory disk because then data is written to the USB file var.img.

 

[free-and-bsd]

I didn't originally understand that from man mdmfs. Was mislead into thinking it was like the OpenBSD mfs.
I now changed it to swap-based mfs. That seems to work (tested).
EDIT: pls read post #13 about the changes.