ZFS Which method (or utility set) would be best for making EVERYDAY ZFS SNAPSHOT to internal USB 2.0 pen-drive?

[Sergei_Shablovsky]

Which method (or utility set) would be best for making EVERYDAY ZFS snapshots of server’s current FreeBSD system (whole system installed but NOT THE WHOLE DRIVE) to internal USB 2.0 pen-drive?

Need to NOTE: this USB 2.0 pen-drive are FORMATTED as bootable (by periodically downloading from online new FreeBSD RELEASE and making bootable by dd command, INTENDED FOR INITIAL INSTALL together with guided bsdinstall), but I also need to place DAILY SNAPSHOT at another volume ON THE SAME INTERNAL USB 2.0 pen-drive (INTENDED FOR RESCUE RESTORE, if something happens with internal main drives or whole main board malfunction or crash).

Another aspect that may affect choosing the toolset for this, are: the server are frequently on high-load conditions with both INTENSIVE network I/O and disk I/O.

Ansible native or 3rd-party roles/modules OR sh/bash script would be preferable.

Thank You all for detailed answering and explanation.

Have a nice sunny day!

 

[Sergei_Shablovsky]

I clearly understand that both therms “internal ISB 2.0 pen-drive” and “server are frequently in hi-load conditions with both intensive network I/O and disk I/O” are NOT living friendly with each other, but anyway need to finding best possible solution in this conditions.

Because most of all servers that are used in production in US/European’s DCs now still are on Intel chipsets which contains USB 2.0 only, not 3.0 or 3.1.
Moreover, latest motherboards from IBM, Dell, HP, Fujitsu frequently need to update BIOS/UEFI f/w, CPU microcode, etc. to achieve 3.0/3.1 speeds ON A BOOTING STEP. And even after that update was successful, sometimes speed on booting (and later in FreeBSD) still 2.0 for unknown reasons.

Because of this the “doing all as only USB 2.0 exist” strategy are best for now, in 2024.

Am I wrong with this?

 

[Mirror176]

Am I correct in assuming you partitioned the USB drive and dd'd the image into a partition on the drive which the BIOS/UEFI is supporting booting from instead of writing dd output to the drive directly?

The 2.0 of the drive just means its likely slower to read/write. Depending on the drive it may be fine or may be low durability when regular writes are happening. The slower writes may help slow down reads on the main pool which you said was under heavy I/O already. Whatever tool you use, you will want to be doing incremental transfers to minimize the I/O.

You need to decide if you want to store the snapshot as a file within a filesystem on the drive or receive it to a pool on the drive. Storing it just as a file opens up better compression ratios by piping it through non-zfs compression which uses CPU/RAM to lower writes and space used on the USB drive but you will not be able to work with its contents until it is restored to a ZFS pool. Comparably, ZFS compression can be used to have the data able to be read directly but take less space on the drive. % of space saved will be approximately equivalent to a drive that is that % faster unless bottlenecked by CPU on the compress or decompress step.

I've only managed zfs backups manually still. sysutils/zfsnap(2) are /bin/sh scripts to manage creation and deletion of snapshots but you are on your own to then transfer them. sysutils/zap is a sh script that does that with send/recv steps also being managed; not sure if it has options or requires modifications if trying to control compression. Deciding when to run the script due to load is likely left up to you.

Unless the system's use will not grow, you should be considering routes to reduce the heavy I/O if it is bottlenecking system operations negatively: more ram, faster disks, downtime to restore from backup if layout gets bad/slow performing, possibly add caching or other device types to the array for speed, review various ZFS paramaters and consider if turning up compression would have a positive impact instead of negative.

 

[Sergei_Shablovsky]

Mirror176
At the first let say THANK YOU fir detailed answering and efforts to solve the case!

Next, please take a time to look at this topic, - may be You find more usable tools than sysutils/zfsnap(2).
For example sanoid/syncoid…

 

[Sergei_Shablovsky]

Am I correct in assuming you partitioned the USB drive and dd'd the image into a partition on the drive which the BIOS/UEFI is supporting booting from instead of writing dd output to the drive directly?

Right now only cronejob exist that running sh script once a day to sound routine “check the new FreeBSD release -> if new release exist, download it in tmp ~> rewrite whole usb pen drive by dd from already downloaded image/archive”.

So, I need to extend procedure (rewrite sh/bash script/Ansible role and setup some toolset for ZFS snapshot writing) to:

- once a day in a certain time checking for new FreeBSD release and renew the boot partition on a usb pen-drive (not to erase whole usb pen-drive, because ZFS snapshot of a current system may exist already on a second partition);

- once a day in a certain time write a ZFS snapshot of a whole current FreeBSD system with as little as possible impact on a overall server’s productivity (eating CPU, mem, interrupts, system calls, bus bandwidth, etc. AS LITTLE AS POSSIBLE).
In this certain use case no matter how long (in reasonable time frame) all operation running: 10min, 30min, 1h, 2h.

MUCH MORE IMPORTANT:
- receive guaranteed successfully result;
- create minimal possible impact on system’s loading;

Of course, usable addition to this procedure would be prevent other task running that may interrupt this procedure (shutting down or rebooting by SysAdmin by Ansible role or manually, etc…) but this is another step.
I return for this a bit later.

The 2.0 of the drive just means its likely slower to read/write. Depending on the drive it may be fine or may be low durability when regular writes are happening. The slower writes may help slow down reads on the main pool which you said was under heavy I/O already.

Agreed. This is another one benefit from using USB 2.0-only pen-drive.

Whatever tool you use, you will want to be doing incremental transfers to minimize the I/O.

You need to decide if you want to store the snapshot as a file within a filesystem on the drive or receive it to a pool on the drive. Storing it just as a file opens up better compression ratios by piping it through non-zfs compression which uses CPU/RAM to lower writes and space used on the USB drive but you will not be able to work with its contents until it is restored to a ZFS pool. Comparably, ZFS compression can be used to have the data able to be read directly but take less space on the drive. % of space saved will be approximately equivalent to a drive that is that % faster unless bottlenecked by CPU on the compress or decompress step.

Thank You for explanation!

Of course from SysAdmin’s and hardware engineer’s point of view much better to have direct access to ZFS snapshot data w/o needs to restoring to ZFS pool before.
If I understand You correctly, also any compression in this case not using, so we also have a much less impact on server’s resources (even we already have AES-NI in CPU/chipset or Intel QAT in PCIe installed).

Need to point You on the importance of having ~20% of free space on whole USB pen-drive: when this ~20% of whole capacity are free, write operations speed not degrade.
This is true for must of all USB pen-drives, no matter which 2.0/3.0/3.1/3.2 standard used. Because of schematic principles of this type of drives.

So, the TOTAL USB DRIVE SIZE must be >= [BOOT PARTITION SIZE] + [PARTITION FOR ZFS SNAPSHOT] + [20% IF SUM OF ABOVE TWO].

For now, 64/128 Gb would be perfect choice in terms of distributive and system grow.

And may be no any compression needed at all. Am I wrong with this?

I've only managed zfs backups manually still. sysutils/zfsnap(2) are /bin/sh scripts to manage creation and deletion of snapshots but you are on your own to then transfer them. sysutils/zap is a sh script that does that with send/recv steps also being managed; not sure if it has options or requires modifications if trying to control compression. Deciding when to run the script due to load is likely left up to you.

24/7/365 the overall load are constantly.

Unless the system's use will not grow, you should be considering routes to reduce the heavy I/O if it is bottlenecking system operations negatively: more ram, faster disks, downtime to restore from backup if layout gets bad/slow performing, possibly add caching or other device types to the array for speed, review various ZFS paramaters and consider if turning up compression would have a positive impact instead of negative.

Thank You for detailed suggestions!

I try to make all measurements:
- on system without any loading at all;
- on a real production server under real loading, when all parameters that You means are optimized for production use.

And only after this “try to play” with each of this factors.

 

[Sergei_Shablovsky]

Mirror176

So, all I need to do (by script or Ansible role):

- instruct dd making bootable USB pen-drive with 2(two) ZFS partitions and write downloaded fresh FreeBSD image (with bsdinstall installing script) in first partition
OR
just write freshly downloaded FreeBSD image (with bsdinstall installing script) in first partition if 2nd partition contain ZFS snapshot;

AND

- use some toolset to making ZFS snapshot in 2nd ZFS partition on USB pen drive;

Am I right?

If Yes which would be sh/bash commands to doing this initial USB pen drive formatting on 2(two) ZFS partitions for exactly my use case?

 

[unitrunker]

Somewhat related to this ...

ZFS is now available on FreeBSD, Linux and Windows. Since ZFS also supports encryption, I am looking into replacing VeraCrypt with ZFS as a portable way to securely exchange data (via thumb drive or SD card) between multiple hosts.

 

[Mirror176]

Next, please take a time to look at this topic, - may be You find more usable tools than sysutils/zfsnap(2).
For example sanoid/syncoid…

sanoid is perl which was outside of your stated preference. Numerous tools exist though I don't have a good comparison of features. zfsnap or zxfer are both written for /bin/sh. A general search would likely be "zfs replication script"

Right now only cronejob exist that running sh script once a day to sound routine “check the new FreeBSD release -> if new release exist, download it in tmp ~> rewrite whole usb pen drive by dd from already downloaded image/archive”.

So, I need to extend procedure (rewrite sh/bash script/Ansible role and setup some toolset for ZFS snapshot writing) to:

- once a day in a certain time checking for new FreeBSD release and renew the boot partition on a usb pen-drive (not to erase whole usb pen-drive, because ZFS snapshot of a current system may exist already on a second partition);

Maybe test other boot options like easy2boot that can setup a USB drive to contain files that can be booted directly (iso, etc.). If that works then you need to make sure if the boot system can handle booting from a fragmented layout (unless you want to pass the image through a defragmenter or redo the partition that holds it each time to guarantee its not fragmented). Once past that step, now you can have another partition on the same drive that contains the ZFS pool.

Alternatively you could backup (optional) and recreate+rewrite the partition after that dd.

Additionally, you could choose 'if' you need to write the newest image before you have even tested it. Replacing a tested installer with an untested one isn't the most reliable choice; were you concerned about out of date ZFS code if updating pools? Have you considered manually creating/updating the partition instead of overwriting with an image? Even if not creating your own install media, I'd imagine you can mount a new image and your old partition to rsync things around fairly effectively and then just make sure steps are taken for a boot loader update on it too.

- once a day in a certain time write a ZFS snapshot of a whole current FreeBSD system with as little as possible impact on a overall server’s productivity (eating CPU, mem, interrupts, system calls, bus bandwidth, etc. AS LITTLE AS POSSIBLE).
In this certain use case no matter how long (in reasonable time frame) all operation running: 10min, 30min, 1h, 2h.

MUCH MORE IMPORTANT:
- receive guaranteed successfully result;
- create minimal possible impact on system’s loading;

The more often you transfer a snapshot incrementally, the less you will have to transfer in the next snapshot.

Of course, usable addition to this procedure would be prevent other task running that may interrupt this procedure (shutting down or rebooting by SysAdmin by Ansible role or manually, etc…) but this is another step.
I return for this a bit later.

ZFS can resume interrupted send/recv transfers; only the previous state is accessible until the transfer finishes properly. It does require specifying a resume token so either needs to be done manually or the tool that did the work needs to support that capability if it is desired. Look into `man zfs-recv` section on -s flag.

Of course from SysAdmin’s and hardware engineer’s point of view much better to have direct access to ZFS snapshot data w/o needs to restoring to ZFS pool before.
If I understand You correctly, also any compression in this case not using, so we also have a much less impact on server’s resources (even we already have AES-NI in CPU/chipset or Intel QAT in PCIe installed).

Direct access to files is always my preference. You need to consider things like making sure the backup doesn't mount over the original pool and likely want to set the backup as read only. Hopefully any good script doing things for you did this, but otherwise its easy with options+manual use of zfs send+recv.

Compression reduces I/O load on disk but costs CPU/RAM to do it. Your original post mentioned excessive network and disk load but not CPU or RAM being loaded; the mention is to trade use of CPU+RAM to move less data to+from disk as less data transferring to+from will be a faster transfer.

Testing is the best way to find out for sure but lz4 normally has so little impact on the system that its usually more detrimental to turn it off. zstd can vary depending on the setting. My old i7-3820 at 1/2 the cores disabled could handle writing at level 12 compression without much CPU bottleneck but heavy CPU load for compression and level 18 decompressing took maybe <20% CPU while the magnetic disk was still the bottleneck (sustained writes were severely CPU bottlenecked at 18). I suspect the really high compressors add some latency to access data even if decompression throughput is high but haven't ran formal tests.

A 15% saving from compression is a 15% saving in throughput for that data to transfer to/from the drive. If limited by IOPS, you are limited by requests; magnetic disks usually have seek time as the biggest factor. Compression still helps but its impact will seem smaller. Compressed data also takes less space in ARC so its like your ARC grew bigger, but at the cost of some CPU/RAM gets used to do the decompression.

Need to point You on the importance of having ~20% of free space on whole USB pen-drive: when this ~20% of whole capacity are free, write operations speed not degrade.
This is true for must of all USB pen-drives, no matter which 2.0/3.0/3.1/3.2 standard used. Because of schematic principles of this type of drives.

So, the TOTAL USB DRIVE SIZE must be >= [BOOT PARTITION SIZE] + [PARTITION FOR ZFS SNAPSHOT] + [20% IF SUM OF ABOVE TWO].

For now, 64/128 Gb would be perfect choice in terms of distributive and system grow.

And may be no any compression needed at all. Am I wrong with this?


24/7/365 the overall load are constantly.


Thank You for detailed suggestions!

I try to make all measurements:
- on system without any loading at all;
- on a real production server under real loading, when all parameters that You means are optimized for production use.

And only after this “try to play” with each of this factors.

I've only got experience with consumer USB thumb drives which don't support TRIM. For some of those devices it seemed like they needed to be erased and not just keep free space to keep performance up. If it just needs free space and doesn't support trim, don't partition part of the drive's space. If trim is supported, you could evaluate if ZFS's autotrim played nice enough or if you want to manually initiate a trim. ZFS can be set to keep some amount of space unused (refreservation, quota, etc.) but without a TRIM order telling the drive what spots are free it won't help like unpartitioned space.

Different USB drives perform differently on different systems, but if the drive you are using is rated for USB3, you can run USB2 tests if you very slowly plug it in. When you connect to a USB3 port, there is a slight delay in handshaking and 3.0 pins are further back in the connector and connect second. Plugging it in partially will establish a USB2 link if you don't finish plugging it in quick enough so very slowly or stop just after it starts to connect and you can now run tests at hopefully the comparably slow USB2 performance.

I don't know of anything to limit or prioritize ZFS disk I/O by process but things like sysutils/cpulimit send STOP+CONT signals to processes. With a slow enough setting (or manually scripting STOP/CONT signals) you could impact disk throughput. I haven't looked into network throttling since probably my use of Win98 but I think firewalls support connecting in with load balancing and would be easy enough to use.

For 24/7 operation you should already have redundancies in order like backups and if uptime is important then an entire extra system that runs a copy of the same storage at the same time. You may find times when you could use the backup or the additional machine to minimize the impact other activities see as this work is performed.

 

[Mirror176]

Somewhat related to this ...

ZFS is now available on FreeBSD, Linux and Windows. Since ZFS also supports encryption, I am looking into replacing VeraCrypt with ZFS as a portable way to securely exchange data (via thumb drive or SD card) between multiple hosts.

ZFS on Windows and ZFS encryption are still newer; make sure to have backups if stuff like https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6swwBZXgXwdCPKgp4SbPZwTexCg/htmlview is still anywhere near accurate. There are also differences which could lead to things like filenames being leaked/read which is not the norm for full disk encryption.

 

[unitrunker]

Mirror176 Great point.

See the Veracrypt bug list: https://github.com/veracrypt/VeraCrypt/issues

Choose your poison.

 

[Sergei_Shablovsky]

Somewhat related to this ...

ZFS is now available on FreeBSD, Linux and Windows. Since ZFS also supports encryption, I am looking into replacing VeraCrypt with ZFS as a portable way to securely exchange data (via thumb drive or SD card) between multiple hosts.

Which manner?
Size, bandwidth, time, etc…

May be just using drives with encryption on a disk-controller level is a fastest (encoding/decoding only 1 time on à hardware level) and easy (no need any additional software, OS support this natively, just copy/paste password or use Yubikey) way for You?

STILL THE BEST drives (in SAS/SATA 2-4Tb classe, ENTERPRISE GRADE with huge MTBF; 3Tb MODEL ARE THE BEST of all of them according DC reports and independent technical reviews):

Ultrastar 7K3000

2TB & 3TB

3.5-INCH ENTERPRISE HARD DRIVE



SATA models

HUA723030ALA640

HUA723030ALA641

HUA723020ALA640

HUA723020ALA641



Ultrastar 7K4000

2TB, 3TB & 4TB

3.5-INCH ENTERPRISE HARD DRIVE



HUS724040ALE640 // HUS724040ALA640

HUS724030ALE640 // HUS724030ALA640

HUS724020ALE640 // HUS724020ALA640
———
HGST Ultrastar 7K3000 HUA723030ALA640 (0F12456) 3TB 7200 RPM 64MB Cache SATA 6.0Gb/s 3.5" Enterprisel Hard Drive Bare Drive

Capacity: 3TB

Options: SATA ULTRA 512N



NewEgg

Item#: 9SIAD5GBR29627

https://www.newegg.com/hgst-ultrastar-7k3000-hua723030ala640-3tb/p/N82E16822145477



ATTENTION NOTE

—————————

Interface: SATA 6.0Gb/s (According to the Hitachi document, the drive is shipped either as a 6Gbps model or 3Gbps. The model numbers are identical, but the part numbers are different (0F12456 6Gb/s versus 0F12457 3Gb/s).



HUA723030ALA640 3TB SATA 3.5-inch 7200 64MB

0F12456 6Gb/s

0F12457 3Gb/s

 

[Sergei_Shablovsky]

Mirror176
Thank You for SO MUCH DETAILED answering! My respect and hat-off for this!!!

So, before I write reply could You be so please to point me on a professional-level resource where I may find info specified to “ZFS on removable media”?
(I more than sure, You know better way than just googling…

 

[unitrunker]

May be just using drives with encryption on a disk-controller level is a fastest (encoding/decoding only 1 time on à hardware level) and easy (no need any additional software, OS support this natively, just copy/paste password or use Yubikey) way for You?

Hi Sergei_Shablovsky! Your suggestion makes me think you misread my post.

I copy data to a USB thumb drive or SD card, carry the device to another machine, and copy files from the device onto the other host. The hosts are air-gapped or located behind firewalls where direct network transfer isn't practical. Thumb drives and SD cards are small and easily lost. The data on the thumb drive or SD card is stored there in encrypted form (it may not be when it reaches the destination host) so that, if it is lost, I need not worry about a stranger examining the contents. Currently I use Veracrypt but it seems ZFS can now serve this purpose (with added benefit of other ZFS features).

There's a weekly Zoom call run by Michael Dexter on OpenZFS (Wednesdays at 3pm CST). This topic came up in the Zoom call a few weeks ago. Sign up to the FreeBSD Discord and click on Meetings to get full details on the weekly Zoom calls.

To answer your question directed to Mirror176: run 'gpart' and 'zpool create' with the appropriate parameters to put ZFS on removable media and 'zpool import' to mount the result. That doesn't cover encryption as I do not yet run ZFS with encryption. Consider that an exercise for the reader.

 

[Mirror176]

So, before I write reply could You be so please to point me on a professional-level resource where I may find info specified to “ZFS on removable media”?
(I more than sure, You know better way than just googling…

I don't know of any official source. Just combine things I have both read and observed. USB is often slow enough to be the bottleneck and my personal experience has had lots of intermittent issues: cases with build defects on USB ports, 3rd party internal and external expansion hubs with design defects, different chipsets having their own quirks and incompatibilities. USB devices have their own quirks, incompatibilities, quality issues, and lack of features compared to more direct options. Add in 3rd party operating systems and the 'fun' continues to increase. I bought a Kingwin EZ-Clone USI-2535CLU3 (at least I think that's the model I had when testing, bought another for a project separately but haven't used it) which I found slower than some other USB3 adapters but much more reliable than the alternatives we sold when I worked at Fry's Electronics and was given a chance to test them for speed and look for obvious reliability issues; if that didn't work I probably would have downgraded to a USB2 adapter because 3.0 had so many bad adapters in my testing. That can vary depending on the motherboard + OS drivers you connect it to but Windows was my primary testing focus.

The closest to official for ZFS I have hard is it seems to not be recommended or said to seem to work good. When not recommending it they mention the concern of it disconnecting; depending on the device, it could be a data interruption or a complete power loss to the drive. Being internal, you should avoid most of those problems as long as its not getting opened to be bumped, has bad mechanical connection that vibrates loose, or drive is being overheated by itself or adjacent hardware.

My first attempts to ZFS format a USB stick was many years ago and ended up showing bugs between the cheap USB controller + features it reported that FreeBSD tried to use but weren't actually there. I'd get a few minutes of runtime before errors if I recall. I don't remember if FreeBSD changed default 'quirks' or something for drives to work around that. A quality external USB hard drive shouldn't have such issues.

If the drive offers onboard encryption then that is an option that may impact speed and system load but most drives I've personally seen with that 'feature' were using proprietary software to interact with its encryption. Pretty easy to run a `geli -s 4k /dev/gpt/disk-partition` between gpart and zpool create then point zpool create at the .eli of that partition though you could change sector size, other geli encryption options, and use a not gpt-label device reference. Won't know which is faster or less system load without testing but with aes-ni on my old i7-3820 it seems like its around 25% WCPU for its many threads to read about 200MB/s; I'm new to playing with geli so still testing.

ZFS COW means files won't be rewritten to the same place (good for SSD wear, but and quality flash media has its own wear leveling) but also means fragmentation can become an issue (shows much more on magnetic media, worse as pools fill up or contents are intermittently/nonsequentially replaced over and over. ZFS has some sectors that will get reused over and over but with a certain amount of activity before each rewrite; could be important if memory had poor wear leveling algorithm.

In any case, I'd avoid unnecessary rewrites happening over and over regularly to flash memory that isn't known for its high endurance (if you didn't pay for that and/or it isn't stated, then its endurance is not a selling point and should be considered relatively poor). Watching modification time/date, compare vs a saved hash, or read and compare the files before an overwrite can all help reduce that as long as steps like 'blindly re-dd the drive with a copy of the image again' are avoided and if not avoiding it, then keep notes of the last image that was written to see if the new one is actually 'new'.

 

[CeXP1917]

Right now only cronejob exist that running sh script once a day to sound routine “check the new FreeBSD release -> if new release exist, download it in tmp ~> rewrite whole usb pen drive by dd from already downloaded image/archive”.

So, I need to extend procedure (rewrite sh/bash script/Ansible role and setup some toolset for ZFS snapshot writing) to:

- once a day in a certain time checking for new FreeBSD release and renew the boot partition on a usb pen-drive (not to erase whole usb pen-drive, because ZFS snapshot of a current system may exist already on a second partition);

- once a day in a certain time write a ZFS snapshot of a whole current FreeBSD system with as little as possible impact on a overall server’s productivity (eating CPU, mem, interrupts, system calls, bus bandwidth, etc. AS LITTLE AS POSSIBLE).
In this certain use case no matter how long (in reasonable time frame) all operation running: 10min, 30min, 1h, 2h.

What about this cake: Ventoy on flash drive, installation media just downloading as usual file and freebsd-zfs partition for backup pool?

 

[Sergei_Shablovsky]

Just for clarification: as I see there are 2(two) main ways:

1.
Download FreeBSD official bootable image ~> make usb bootable by dd from that image (and copying custom bsdinstall script) ~> add another partition to usb drive for later ZFS backups;

2.
Download FreeBSD official bootable image ~> modify this bootable image by adding custom bsdinstall script AND adding extra one partition ~> making usb bootable by dd;

Which is better?