Server layout: rootfs on USB flash drive or NVMe?

Mjölnir

Mjölnir

After 2 SSD crashes within 4 years with data loss, I decided I want a NAS box. Found a neat used one, only 3-4 years old, and since it's fairly decent hardware I'll also use it as a general-purpose server. For storage It has an internal USB 2 plug (inside the case), 2xNVMe and 2xSATA. For the setup of the base OS I have these options, and I'd like to hear your opinions which one is best:
  1. base OS / root filesystem on the internal USB flash drive.
    This is the genuine setup of the Linux-based OS, already replaced with the real thing. I would use a 3 partition setup like nanobsd(8) with UFS. I had that setup on a xterminal until it broke a while ago and liked it. You switch between two partitions on an update, these are read-only during normal operation, the 3rd partition is for the config. Very clean, easy to handle, if s/th goes wrong I can help myself.
  2. base OS on a zpool(8) on the USB stick
    I have no experience with ZFS on an USB flash ("thumb") drive. I'd have an extra ROOT/default/etc dataset that I can rw/ro as needed, and also mount most of the base OS ro during normal operation. /var could be a md(4) like above, saved to the stick daily or so, or to the data disks.
  3. base OS on the NVMe SSD with ZFS
    Maybe the most natural setup? I want to use the NMVe drives mainly for L2ARC cache, intent log (mirrored), maybe special (mirrored) vdevs and swap. Placing the OS here would take away some space, but having ZFS on a "real" disk device appeals me because I have all the bells and whistles of ZFS vs. "boring" UFS.
  4. OS on the SATA HDD, but I don't like that. I'd like to have the base OS separate from the data.
The 2nd option sounds strange to me... isn't it? Is there a "best practice" guide I can follow? Should I mirror swap or is that overkill? What would I loose if one SSD breaks? The swap breaks, the OS crashes, so what? IIUC I loose the fresh data of a few minutes, usually I wouldn't care. Or could that cause more serious damage?
 
So you already experienced data loss and want to use USB sticks?

Also: on such a small NAS you definitely won't need an L2ARC. And no, you also won't need a SLOG device.

Depending on the use case (read: how fast the data is supposed to be written/read), use 2 singe-vdev pools: one mirror of SATA drives, one mirror of NVMe drives. Use either one for storage that suits your use case the most.
For a bog standard home data grave, use SATA and install the OS on NVMe. If you want to run VMs on the NAS (and if it can actually handle that load - hint: most consumer NAS won't) and/or need to read/write data to/from that pool faster than SATA allows, use NVMe for the data pool. This of course only makes sense if the NAS is attached with at least a 10Gbps nic - for a simple 1Gbps NIC even SATA is able to easily saturate that link.

And just for the record: a NAS is no replacement for backups.
 
I would install a small FreeBSD on the nvme , create a NVME 32GB cache log and special device to use with ZFS on the NAS.
Use this freebsd install to setup ISCI connection to NAS drive where I would create a ZPOOL to use previous cache log special devices. Then as much as possible move directories from original install to this NAS.
[PS : ISCSI is a block device , i think it uses 4K pages, and behaves like any normal drive, so you can create partitions , format then , UFS ,ZFS , create ZPOOLS. When you add drives to the NAS they will become visible].


I worked previous with professional NAS , NETAPP (running own freebsd) , & fiber optic switch (also running own freebsd) , no bragging , my budget was 2.000.000 Euro . All Windows install where on this NAS , diskless , using ISCSI initiator.
This IIS webserver is still running on it, using the same NAS as storage device.
https://g-o.be/
 
sko said:
So you already experienced data loss and want to use USB sticks?
Click to expand...
That's only for the base OS. It would be backuped on a 2nd USB stick, so if it breaks eventually, I can get the system up again quickly.
sko said:
Also: on such a small NAS you definitely won't need an L2ARC. And no, you also won't need a SLOG device.
Click to expand...
Because I don't know in advance which data needs fast access and which doesn't? Or maybe the data labeled "fast please" grows beyond the size of the NVMe, so it needs to spill over into the slow SATA HDD. But I'll make a thick mark on my list on your point.
sko said:
[...] If you want to run VMs on the NAS (and if it can actually handle that load - hint: most consumer NAS won't)
Click to expand...
It has 16 GB RAM, 4xCPU@2 GHz - 2.9 GHz (no HTT, it's a N5095 IIRC). That's overkill for a small home NAS, and IMHO it should be able to run a few VM's easily.
sko said:
and/or need to read/write data to/from that pool faster than SATA allows, use NVMe for the data pool. This of course only makes sense if the NAS is attached with at least a 10Gbps nic - for a simple 1Gbps NIC even SATA is able to easily saturate that link.
Click to expand...
The box has 2x2.5 Gbps, I can't saturate that bandwidth in my network... The speed of data access is only important for the services and VM's running on the box. So if you say a cache and log will not make a noticable difference with such a box, I'll have a little bit more storage space.
 
How much room do you have for expansion? Are you really limited to a single USB port, the 2xNVME and 2xSATA?

My opinion, L2ARC/SLOG depends on the anticipated use patterns. I think SLOG is useful if you are doing lots of synchronous writes (database doing stores) L2ARC maybe if you are doing lots of reads of the same data (think data is movies, multiple people watching the same movie. Same data gets cached in ARC but because everyone is at a different place in the movie, data falls out of ARC but is in L2ARC so don't need to go back to device).

ZFS on a USB device is really not much difference (except for speed) than ZFS on any other device. Being able to duplicate a USB device would give a really quick recovery (warm swap almost).

I really don't have anything else but use patterns can/should drive a lot along with desired data safety vs storage space
 
USB thumb drives as root filesystems always make us think "Rodime SSD". Cheap thumb drives use garbage flash that can't retain data. This is why our root pool is an NVMe mirror.
 
You must log in or register to reply here.
Back
Top