Server layout: rootfs on USB flash drive or NVMe?

Mjölnir

Mjölnir

After 2 SSD crashes within 4 years with data loss, I decided I want a NAS box. Found a neat used semi-professional SOHO NAS box, only 3-4 years old, and since it's fairly decent hardware I'll also use it as a general-purpose server at home. For storage It has an internal USB 2 plug (inside the case), 2xNVMe 1.3 and 2xSATA III. For the setup of the base OS I have these options, and I'd like to hear your opinions which one is best:
  1. base OS / root on UFS filesystem on the internal USB flash drive.
    This is the genuine setup of the Linux-based OS, already replaced with the real thing. I would use a 3 partition setup like nanobsd(8). I had that setup on a xterminal until it broke a while ago and liked it. You switch between two partitions on an update, these are read-only during normal operation, the 3rd partition is for the configuration files. Very clean design, easy to handle, if s/th goes wrong I can help myself. Thanks to geom_uzip(4) the system can reside on a very small USB thumb drive. Rollback after a failed update is easy, just boot from the other partition. Changes to the configuration would have to be managed by an external tool, I like the good old cvs(1) for that (IIRC rcs(1) can not be used by root, else I would take that because I don't need the "C" in cvs).
  2. base OS on a zpool(8) on the internal USB stick
    I have no experience with ZFS on an USB flash ("thumb") drive. I'd have an extra ROOT/default/etc dataset that I can rw/ro as needed, and also mount most of the base OS read-only during normal operation. /var could be a md(4) or tmpfs(4) like above, saved to the stick or to the data disks regularly. ZFS compression would keep the size small, like in the 1st option. Creation of a new boot environment on every update makes rollback easy. Changes in the configuration would be tracked twice: 1. with automatic snapshots, either regular or triggered by a file monitor 2. with an external tool like above.
  3. base OS on the NVMe SSD with ZFS
    Maybe the most natural setup? I want to use the NMVe drives mainly for L2ARC cache, intent log (mirrored), maybe special (mirrored) vdevs and swap. Placing the OS here would take away some space, but having ZFS on a "real" disk device appeals me because I have all the bells and whistles of ZFS vs. "boring" UFS.
  4. OS on the SATA HDD, but I don't like that. I'd like to have the base OS separate from the data.
The 2nd option sounds strange to me... isn't it? Is there a "best practice" guide I can follow? Should I mirror swap or is that overkill? What would I loose if one SSD breaks? The swap breaks, the OS crashes, so what? IIUC I loose the fresh data of a few minutes, usually I wouldn't care. Or could that cause more serious damage?
 
Last edited:
So you already experienced data loss and want to use USB sticks?

Also: on such a small NAS you definitely won't need an L2ARC. And no, you also won't need a SLOG device.

Depending on the use case (read: how fast the data is supposed to be written/read), use 2 singe-vdev pools: one mirror of SATA drives, one mirror of NVMe drives. Use either one for storage that suits your use case the most.
For a bog standard home data grave, use SATA and install the OS on NVMe. If you want to run VMs on the NAS (and if it can actually handle that load - hint: most consumer NAS won't) and/or need to read/write data to/from that pool faster than SATA allows, use NVMe for the data pool. This of course only makes sense if the NAS is attached with at least a 10Gbps nic - for a simple 1Gbps NIC even SATA is able to easily saturate that link.

And just for the record: a NAS is no replacement for backups.
 
I would install a small FreeBSD on the nvme , create a NVME 32GB cache log and special device to use with ZFS on the NAS.
Use this freebsd install to setup ISCI connection to NAS drive where I would create a ZPOOL to use previous cache log special devices. Then as much as possible move directories from original install to this NAS.
[PS : ISCSI is a block device , i think it uses 4K pages, and behaves like any normal drive, so you can create partitions , format then , UFS ,ZFS , create ZPOOLS. When you add drives to the NAS they will become visible].


I worked previous with professional NAS , NETAPP (running own freebsd) , & fiber optic switch (also running own freebsd) , no bragging , my budget was 2.000.000 Euro . All Windows install where on this NAS , diskless , using ISCSI initiator.
This IIS webserver is still running on it, using the same NAS as storage device.
https://g-o.be/
 
sko said:
So you already experienced data loss and want to use USB sticks?
Click to expand...
That's only for the base OS. It would be backuped on a 2nd USB stick, so if it breaks eventually, I can get the system up again quickly.
sko said:
Also: on such a small NAS you definitely won't need an L2ARC. And no, you also won't need a SLOG device.
Click to expand...
Because I don't know in advance which data needs fast access and which doesn't? Or maybe the data labeled "fast please" grows beyond the size of the NVMe, so it needs to spill over into the slow SATA HDD. But I'll make a thick mark on my list on your point.
sko said:
[...] If you want to run VMs on the NAS (and if it can actually handle that load - hint: most consumer NAS won't)
Click to expand...
It has 16 GB RAM, 4xCPU@2 GHz - 2.9 GHz (no HTT, it's a N5095 IIRC). That's overkill for a small home NAS, and IMHO it should be able to run a few VM's easily.
sko said:
and/or need to read/write data to/from that pool faster than SATA allows, use NVMe for the data pool. This of course only makes sense if the NAS is attached with at least a 10Gbps nic - for a simple 1Gbps NIC even SATA is able to easily saturate that link.
Click to expand...
The box has 2x2.5 Gbps, I can't saturate that bandwidth in my network... The speed of data access is only important for the services and VM's running on the box. So if you say a cache and log will not make a noticable difference with such a box, I'll have a little bit more storage space.
 
How much room do you have for expansion? Are you really limited to a single USB port, the 2xNVME and 2xSATA?

My opinion, L2ARC/SLOG depends on the anticipated use patterns. I think SLOG is useful if you are doing lots of synchronous writes (database doing stores) L2ARC maybe if you are doing lots of reads of the same data (think data is movies, multiple people watching the same movie. Same data gets cached in ARC but because everyone is at a different place in the movie, data falls out of ARC but is in L2ARC so don't need to go back to device).

ZFS on a USB device is really not much difference (except for speed) than ZFS on any other device. Being able to duplicate a USB device would give a really quick recovery (warm swap almost).

I really don't have anything else but use patterns can/should drive a lot along with desired data safety vs storage space
 
USB thumb drives as root filesystems always make us think "Rodime SSD". Cheap thumb drives use garbage flash that can't retain data. This is why our root pool is an NVMe mirror.
 
1. I use FreeBSD UFS stick to load the kernel.
2. With it i mount as root filesystem a FREEBSD install UFS on SSD drive. Then my system becomes 10 faster ...
3. So rc.conf is read from SSD drive.
4. I mount /usr & /var from separate SSD drive ZFSPOOL.

Now i must find a backup strategy. If SSD dies ... GONE BYE BYE baby.
Think using "clone" to backup
/usr/local/etc/
/etc/
/home
All the rest i don't care.
 
Note , i found on amazon,
TERRAMASTER F8 SSD NAS Storage - 8Bay All SSD NAS Server N95 QuadCore CPU, 8GB DDR5 RAM, 10GbE Port, Palm-Sized Powerful Network Attached Storage (Diskless)

€586.49 (lacking budget :)
These things are interesting, SSD is 10X faster then spinning disk. & when you make ZFS mirror on SSD, no danger when one dies ... With this setup you can have 16TB data mirrored.
 
That looks like the bigger & newer version of what I have, a Terramaster F2-423. Like I already wrote this small box is oversized to serve me just as a NAS. IMHO semi-professional hardware AFAICT up to now. And I got it used for cheap money. Mine has DDR4 and 2.5 Gbps instead of 10Gbps. All my clients have 1 Gbps so I can use the box the next 15 years ;)
 
mer said:
How much room do you have for expansion? Are you really limited to a single USB port, the 2xNVME and 2xSATA?
Click to expand...
The board can take up to 2x16 = 32 GB DDR4 LPDIMM and in addition to the internal USB 2.0 it has 2xUSB 3.x on the outside. So I could plug in more external HDD. But AFAICS a simple mirror with 2x6 TB = 4 GB mirrored + (2 + 2) = 4 TB striped for "scratch" data that does not need redundancy will be perfectly enough for me for the next 5-10 years.

On the other topics, wether to have cache, log etc. I want to wait for more feedback since I have no experience with that. Maybe sko is right and it wouldn't make a noticable difference. I'm not in a hurry, the HDD will arrive in 2-3 days, and the 2nd NVMe IDK.
 
  • Like
Reactions: mer
Like i said urlier, you don't want to use any USB unless for boot or backup.
I never had any spinning disk die , or NVME die.
And yes i had an SSD die , completely dead , completely suddenly, and all data gone, if i would have used zpool mirror i would still have the data.
Note , mirror does not stripe. Mirror has reduncancy 100%.
To stripe just add partitions of DIFFERENT disks to the same zpool. Striping happens automatic.
[ PS : If you have 4 disks , you can combine striping with mirroring, then you have striping for performance , ie on different controller & mirroring for redundancy (duying disks)]
You could use a small 64GB freebsd-zfs partition , anywhere to add to the ZPOOL , as ARC cache device. The zfs system will use it or not use it as needed. It does not hurt. For best performance you could use an internal NVME partition. Terramaster F2-423 : 2 slots, so you need to make a choice. Mirroring(safety)(for SSD) or Striping(performance)(spinning disk no longer die) .
It's nice to have a btrfs partition if you want to use Linux , and a good editor like Clion,Idea,Rider
 
You can put in 2 2.5inch SSD drives upto 30GB each. But it has also has high speed USB ports. They can be used if you want to make backups eg a large cheap usb drive.
I find it an interesting product for its price.
When i win the lotto i'll buy one, put in 2 SSD's and zpool mirror.
 
Alain De Vos said:
Rule of thumb , only use USB connections/devices , to BOOT & MOUNT, or to BACKUP.
Click to expand...
Exactly! Aside from USB flash sticks being cheaply produced, and less reliable over the long term, they are generally sssssllllllloooooowwwww...I think that a quality SSD/NVMe is always the best rootfs/boot option, provided you tune it to not do excessive writes (I know, not totally necessary with modern devices, but I'm a creature of habit)

If you're burning out SATA/NMVe devices then you need to figure out why instead of looking for a work-around.
 
Allow me to put a small warning , my pc,
smartctl -a /dev/nvme0 | grep -i Used
Percentage Used: 37%
Yep my nvme drive with zpool special device is already 37% dead.
If small special device dies. Whole zpool on SSD gone ...
Will try to remove stuff & just keep essential.
 
Just found out my Windows drive has a wear level of 99%. & Windows not like to be moved ... Certainly not to a drive with other partitions...
My dear friend we all gone die one day. But not today. Will stop gaming
Later today,
cat windows.txt
wmic path SoftwareLicensingService get OA3xOriginalProductKey
powershell
(Get-WmiObject -query 'select * from SoftwareLicensingService').OA3xOriginalProductKey
 
Mjölnir said:
That's only for the base OS. It would be backuped on a 2nd USB stick, so if it breaks eventually, I can get the system up again quickly.
Click to expand...
they are a) too slow and b) *will* die quite often. just don't.
Mjölnir said:
Because I don't know in advance which data needs fast access and which doesn't? Or maybe the data labeled "fast please" grows beyond the size of the NVMe, so it needs to spill over into the slow SATA HDD. But I'll make a thick mark on my list on your point.
Click to expand...
L2ARC is basically only a very last resort if you are completely maxed out on RAM and absolutely need some data to stay in a slightly-faster-than-the-disks-cache. L2ARC also needs memory - so you *increase* memory pressure on the system.
For a home NAS you don't need L2ARC or SLOG. At most you might want to use flash drives as 'special' device when having a pool of spinning rust as those are abysmally slow for metadata and small files.

Mjölnir said:
It has 16 GB RAM, 4xCPU@2 GHz - 2.9 GHz (no HTT, it's a N5095 IIRC). That's overkill for a small home NAS, and IMHO it should be able to run a few VM's easily.
Click to expand...
I wouldn't bother. Some services in jails would be fine, but VMs would be rather laggy on such hardware.

Mjölnir said:
The box has 2x2.5 Gbps, I can't saturate that bandwidth in my network... The speed of data access is only important for the services and VM's running on the box. So if you say a cache and log will not make a noticable difference with such a box, I'll have a little bit more storage space.
Click to expand...
In this case I'd definitely would use NVMe for OS (+ jails for some other network services you might want to run) and stick to cheaper SATA SSDs for the storage (if one can call any type of flash storage 'cheap' nowadays... I just ordered the same bottom-end M.2 SATA disk for 300eur today for which we paid ~70eur only half a year ago...)
 
There is arc in memory and arc on disk.
For arc in memory you can specify min & max. I use for max half my ram. Min one quarter.
For arc in disk its a cache device. Adapative. Used when it can be used. I would say 2 to 4 times your ram.
But both are different in nature.
 
Alain De Vos said:
There is arc in memory and arc on disk.
For arc in memory you can specify min & max. I use for max half my ram. Min one quarter.
For arc in disk its a cache device. Adapative. Used when it can be used. I would say 2 to 4 times your ram.
But both are different in nature.
Click to expand...
has nothing to do with what I wrote...

L2ARC requires memory for its allocation tables; roughly 100MB per 1GB stored in L2ARC as a rule of thumb. So L2ARC is always the wrong tool against memory pressure on such low-end systems as you will worsen the problem.
 
So you limit it. The arc is dynamic. When the system asks , arc will say can i can set free. If it can it will. But sometimes arc & applications will fight for the same memory. Have 16GB memory. Never any problem. Maybe for Rapsberry it can be problematic.
 
If it were my system, I would get a pair of quality NVMe SSDs as large as I could afford. Consider heatsinks as the design stage. Mirror them using ZFS. But see the caveat bellow about copy-on-write file systems -- you may wish to use a GEOM mirror for the whole disk, and have multiple types of file systems.

I would install the ZFS root on this.

I would also install swap here, but keep watch and move it to a SATA mirror if the swap gets too intensive.

I would think carefully about your need for synchronous writes to any file systems on the (much slower) SATA disks. It's not common, but if you have the need put a ZFS SLOG on the SSDs (serving the slower SATA disks). A SLOG never needs to be larger than main memory.

I would always consider putting a special VDEV on the SSDs. This is essentially a cache for the file system metadata on the SATA disks. They are a challenge to size correctly. So put it adjacent to the swap space, and move the swap to SATA if more space is required. Plan for this when you partition the disks.

I would try place my important VMs on the SSDs. But beware, you don't want both the hypervisor and the VM client using copy-on-write file systems. And you don't want VMs swapping madly to any underlying SSD.

I would never build a NAS without 100% redundancy. I would mirror the SATA disks... and migrate storage between SSD and SATA as needs dictate.
 
cracauer@ said:
USB 2 is not fast enough for the root filesystem.

Are you sure the internal port is USB 2?
Click to expand...
99% sure. Internal 1 x USB 2, external 2 x USB 3 (dont know 3.0/.1/.2).
I thought what is needed often from the OS will be cached by the MFU cache in RAM more or less quickly, so the speed of the storage is generally of minor importance. If you say no, it does make a noticable difference then I'm going to install the OS on the NVMe drives.

Alain De Vos With all respect, s/t I think you're an AI bot... What do you think I dunno that a stripe has no redundancy??? C'mon, be serious, man! I'll have two partitions on each HDD: 2/3 for a mirror, 1/3 for a stripe. The result is that I have a total capacity of 2/3, 50/50 mirrored/striped. The stripe will hold "scratch" data, i.e. all that I can easily download from the internet again (e.g. /var/cache/pkg) or what is intermediate data (mostly short lived, e.g. *.obj files when building software). The mirror holds the valuable data. Isn't that a standard setup? It's just that I have to use partitions instead of whole disks because I have only 2 disks. Which also means that this box will not stress my electricity bill.
 
First there is no standard setup. That's an illusing. It all depends, your requirements, your hardware , your usage, your history. There are "good common practices".

With all respect, s/t I think you're an AI bot... What do you think I dunno that a stripe has no redundancy??? C'mon, be serious, man!
Sorry no bot, just flesh and bones dead in hunderd years
But until then , please give me output of ,
zpool list -v,
Then i come back.
 
You must log in or register to reply here.
Back
Top