FreeBSD on Flash Media Setup

[Polybiusx]

I am trying to determine an ideal setup for a new system, and am not certain how to handle the root-on-zfs issue. For reference, I will be installing the latest FreeBSD-10, and keeping the OS up to date (current-stable).

Detailed Question, skip to end for TLDR version.
So I want this machine to provide several services; most critically I want it to function as a NAS. If this was all, I would obviously go with FreeNAS, but I would also like to have internet-facing accessibility through functions like OwnCloud or OpenVPN. I would also like standard services like DNS server, LDAP, some form of ACL (TBD later after tinkering), Ventrilo, and the list goes on and on. The machine would be performing this functions as well as that of a NAS, which will likely connect to other machines via Samba, CIFS, NFS, and so forth.

In order to plan out this system correctly, I currently have more than the minimum equipment necessary (sufficient ECC RAM, etc.) to run a root-on-ZFS setup, but I only have two, 4TB drives that I wish to run in a zpool mirror. I've read multiple conflicting reports that you should install the OS to flash media so as to prevent slicing drives so ZFS can hog them, and others that suggest that it's fine to slice. I am also concerned about the long-term integrity of the pool as FreeBSD goes through upgrades to new versions; if there is a problem, I'd like to troubleshoot it on the OS side without risking all of my sensitive data.

I would like to run the FreeBSD OS on flash media, but I don't know if this would impair the functions or behavior of the OS for the number of actions I want it to perform. I would likely also be setting up an action to take ZFS snapshots and saving them to a network-available drive so as to mitigate risk of data loss. Would FreeBSD lose substantial performance running on flash media? When saving snapshots across the network, will running the OS on flash media bottleneck the service? Along that same vein, if I wanted to have snapshots of my FreeBSD OS, could I still take those and back them up on other network-available drives?

Another concern I have is multi-user streaming. If a media file (example.mkv) is being accessed by local area users, as well as users tunneled through the internet (VPN, SSH, etc.), will the OS be able to keep up with the file demands if installed on flash media? To be clear this is a demand of <25 users streaming at once or utilizing the NAS through VPN, no more than that.

One last concern I have with putting the OS on flash media is data encryption. If I decide to encrypt the data, will the OS be able to perform its functions utilizing resources about the same as being installed directly on the HDDs, or will it severely cripple performance?

TLDR Version: Should I install FreeBSD onto flash media (thumb drive, SD card, etc.) or slice and partition? Which is better for a multifunction SOHO level server performing multiple actions?

 

[junovitch@]

Wow. Well I would say "it depends". I know that's not a very clear answer. Let's clear up the "I've read multiple conflicting reports that you should install the OS to flash media so as to prevent slicing drives so ZFS can hog them, and others that suggest that it's fine to slice." Basically, ZFS performs the best when you give access to the whole drive. Don't set up disk with UFS on the first slice/partition and ZFS on another. Use the whole drive for ZFS wherever possible.

Now if you are having 25 user stream movies from the flash drive, I highly doubt that would keep up unless it were an SSD. However, since you mentioned FreeNAS, in FreeNAS's case it's an very lightweight system that is on slow flash while the real work is done from spinning drives. If you based your installation off of that model, you can use a base install with minimal additions and opt to install all the services you mention as jails running from your zpool mirror. The trade off would probably be a noticeable hit with boot speeds and slower speeds installing and upgrading software on the host OS. Your data and services running from each jail would be limited by the speed of the drive in the pool. Depending on just how much you have on the host from from flash, most of it disk I/O would be on /var so you would probably want to use the NanoBSD approach used by FreeNAS where /var is a RAM disk and /tmp is sym-linked to /var/tmp. That way the stuff that gets written the most isn't limited by slow flash speed. The trade off is the extra maintenance work to keep the stuff that you really would need across reboots like your package database.

Ultimately, you're going to have to weigh if the extra work of setting up a nicely separated system is worth having the separation of a base OS separate from your data drives and if you're willing to go through the effort making sure the stuff that needs to be accessed quickly is either in RAM or on faster disk rather than slow thumb drive. For what it's worth, I agree with your concern of being able to deal with OS issues separately from important data. When I first tried out a ZFS install on my backup servers main drive I ran into issues and went back to UFS on an SSD and a 3 drive RAIDZ for all my important data. The difference here is that my SSD is fast enough to just run my OS and all my jails without worrying about using memory disks, running software from disk, dealing with the space constraints of a thumb drive.