ZFS zfs send -R

i.am.the.problem · Nov 12, 2021

Assuming I have the following zfs file systems on a zpool

Code:

NAME                                    USED  AVAIL  REFER  MOUNTPOINT
zroot                                   3.41T   102G    88K  /zroot
zroot/data                              3.32T   102G    96K  /zroot/data
zroot/data/backups                      1.14T   102G    88K  /zroot/data/backups
zroot/data/backups/server01             18.8G   102G  18.8G  /zroot/data/backups/server01
zroot/data/backups/user1                200K   102G    88K  /zroot/data/backups/user1
zroot/data/backups/user1/workstation    112K   102G   112K  /zroot/data/backups/user1/workstation
zroot/data/backups/server02             1.12T   102G   847G  /zroot/data/backups/server01

If I create a snapshot at the root of the file systems as follows

Code:

zfs snapshot zroot@backup01

Then execute the following command

Code:

zfs send -R zroot@backup01 > /tmp/backup01

Will the /tmp/backup01 contain all the files systems?

When receiving the /tmp/backup01 file into another pool do I need to create all the file systems first or will the receive process do that for me?

Alain De Vos · Nov 12, 2021

-R, --replicate
Generate a replication stream package, which will replicate the
specified file system, and all descendent file systems, up to the
named snapshot.
No you don't need to create a receiving dataset before.
PS : when you do zfs receive give parameter -v , so you can see what happens.

i.am.the.problem · Nov 12, 2021

Alain De Vos said:
-R, --replicate
Generate a replication stream package, which will replicate the
specified file system, and all descendent file systems, up to the
named snapshot.
Yes, everything is replicated.
No you don't need to create a receiving dataset before.
PS : when you do zfs receive give parameter -v , so you can see what happens.

What happens if the receiving end has the file systems but no files in them?

Alain De Vos · Nov 12, 2021

On the receiving side everything will be created as needed.
It is more a problem if it already exist on the receiving side.

Eric A. Borisch · Nov 13, 2021

i.am.the.problem said:
If I create a snapshot at the root of the file systems as follows

Code:

zfs snapshot zroot@backup01

Then execute the following command

Code:

zfs send -R zroot@backup01 > /tmp/backup01

Will the /tmp/backup01 contain all the files systems?

You need to use zfs snapshot -r zroot@backup01. The -r tells zfs-snapshot(8) to take a snapshot of each filesystem (snapshots are a per-filesystem, not per-pool item), not just the top level one.

The zfs-send(8) with -R will fail without that step, complaining about missing @backup01 snapshots of the descendant file systems. (So at least you won't think you backed everything up when in reality you had not.)

Alain De Vos · Nov 13, 2021

Indeed I made a mistake, i.e. "RECURSIVE snapshot & replication of datasets".

Cath O'Deray · Nov 13, 2021

<https://openzfs.github.io/openzfs-docs/man/8/zfs-send.8.html#R>

If (for example) there is more than one boot environment, and if replication is not used, then the resulting dates of creation may be incorrect.

For example, from <https://pastebin.com/iKbjmYez> (2021-11-07):

Code:

% bectl list -c creation
BE                             Active Mountpoint Space Created
default                        -      -          789M  2021-08-26 16:33
n247565-b49ba74deeb-f          -      -          36.7G 2021-08-26 19:50
n248269-941650aae97-e          -      -          40.5M 2021-08-26 22:54
n247798-f39dd6a9784-a          -      -          22.9M 2021-08-26 22:54
n248269-941650aae97-b          -      -          672M  2021-08-26 22:54
n248269-941650aae97-f          -      -          5.03G 2021-08-26 22:54
n247798-f39dd6a9784-e          -      -          328M  2021-08-26 22:55
n248269-941650aae97-d          -      -          260M  2021-08-26 22:55
n247798-f39dd6a9784-j          -      -          5.46G 2021-08-26 22:55
14.0-CURRENT_2021-09-18_113006 -      -          10.2G 2021-09-18 11:30
n250242-eab5358b908-d          -      -          1.80G 2021-10-29 05:42
n250242-eab5358b908-e          -      -          258M  2021-11-01 06:23
n250242-eab5358b908-f          -      -          26.8M 2021-11-06 13:32
n250511-5f73b3338ee-a          -      -          18.6M 2021-11-06 23:03
n250511-5f73b3338ee-b          NR     /          71.1G 2021-11-07 00:56
%

– chronological order requested, however it's disorderly and misleading due to non-preservation of properties:

note the six-digit number that follows each n
the b49ba74deeb environment that was reportedly created 2021-08-26 was truly created two months earlier … and so on.

Luckily, most of my environments were named in a way that suits sort(8). The first few lines from <https://pastebin.com/vHdsXUfn>:

Code:

% bectl list | sort
14.0-CURRENT_2021-09-18_113006 -      -          10.2G 2021-09-18 11:30
BE                             Active Mountpoint Space Created
default                        -      -          789M  2021-08-26 16:33
n247565-b49ba74deeb-f          -      -          36.7G 2021-08-26 19:50
n247798-f39dd6a9784-a          -      -          22.9M 2021-08-26 22:54
n247798-f39dd6a9784-e          -      -          328M  2021-08-26 22:55
n247798-f39dd6a9784-j          -      -          5.46G 2021-08-26 22:55
n248269-941650aae97-b          -      -          672M  2021-08-26 22:54
n248269-941650aae97-d          -      -          260M  2021-08-26 22:55
n248269-941650aae97-e          -      -          40.5M 2021-08-26 22:54
n248269-941650aae97-f          -      -          5.03G 2021-08-26 22:54
n250242-eab5358b908-d          -      -          1.80G 2021-10-29 05:42
n250242-eab5358b908-e          -      -          258M  2021-11-01 06:23
n250242-eab5358b908-f          -      -          26.8M 2021-11-06 13:32
n250511-5f73b3338ee-a          -      -          18.6M 2021-11-06 23:03
n250511-5f73b3338ee-b          NR     /          71.1G 2021-11-07 00:56
% bectl list -s
…

<https://openzfs.github.io/openzfs-docs/man/8/zfs-snapshot.8.html>

Eric A. Borisch said:
You need to use zfs snapshot -r zroot@backup01. …

Will that command preserve dates of creation for boot environment purposes?

i.am.the.problem · Nov 14, 2021

I created the snapshot as follows:

Code:

root@:~ # zfs snapshot -r nsgroot/nsg@backup01
root@:~ # zfs list -t all
NAME                                                                     USED  AVAIL  REFER  MOUNTPOINT
nsgroot                                                                 90.1G   584G    88K  /nsgroot
nsgroot/ROOT                                                            10.4G   584G    88K  none
nsgroot/ROOT/default                                                    10.4G   584G  10.4G  /
nsgroot/nsg                                                             62.8G   584G    88K  /nsg
nsgroot/nsg@backup01                                                        0      -    88K  -
nsgroot/nsg/services_persistent                                         17.5G   584G    88K  /nsg/services_persistent
nsgroot/nsg/services_persistent@backup01                                    0      -    88K  -
---MANY MORE FILESYSYSTEMS AS CHILDREN OF nsgroot/nsg and  nsgroot/nsg/services_persistent

Then sent using the following command:

Code:

zfs send -R nsgroot/nsg@backup01' > nsg.zfs.snapshot

On another system I started with the following zfs pool:

Code:

root@:/backups # zfs list
NAME                   USED  AVAIL  REFER  MOUNTPOINT
nsgroot               27.3G   465G    88K  /nsgroot
nsgroot/ROOT          10.4G   465G    88K  none
nsgroot/ROOT/default  10.4G   465G  10.4G  /
nsgroot/tmp             88K   465G    88K  /tmp
nsgroot/usr            702M   465G    88K  /usr
nsgroot/usr/home        88K   465G    88K  /usr/home
nsgroot/usr/ports      701M   465G   701M  /usr/ports
nsgroot/usr/src         88K   465G    88K  /usr/src
nsgroot/var           16.2G   465G    88K  /var
nsgroot/var/audit       88K   465G    88K  /var/audit
nsgroot/var/crash       88K   465G    88K  /var/crash
nsgroot/var/log       16.2G   465G  16.2G  /var/log
nsgroot/var/mail        96K   465G    96K  /var/mail
nsgroot/var/tmp         88K   465G    88K  /var/tmp

I then tried the following command zfs receive command which gives the following result:

Code:

zfs receive -vn nsgroot < /backups/temp/nsg.zfs.snapshot     
cannot receive new filesystem stream: destination 'nsgroot' exists
must specify -F to overwrite it

So I added the -F flag which gives the following result:

Code:

zfs receive -vnF nsgroot < /backups/temp/nsg.zfs.snapshot
would receive full stream of nsgroot/nsg@backup01 into nsgroot@backup01
would receive full stream of nsgroot/nsg/services_persistent@backup01 into nsgroot/services_persistent@backup01
cannot open 'nsgroot/services_persistent': dataset does not exist
cannot receive new filesystem stream: unable to restore to destination

Not sure where to go from here..

i.am.the.problem · Nov 14, 2021

I created all the filesystems and executed the following command which seemed to work:

Code:

root@:/backups # zfs receive -vnFe nsgroot < /backups/temp/nsg.zfs.snapshot
would receive full stream of nsgroot/nsg@backup01 into nsgroot/nsg@backup01
would receive full stream of nsgroot/nsg/services_persistent@backup01 into nsgroot/nsg/services_persistent@backup01
---lots more output about other files systems---

Then I tried the command without the -n flag:

Code:

root@:/backups # zfs receive -vFe nsgroot < /backups/temp/nsg.zfs.snapshot
receiving full stream of nsgroot/nsg@backup01 into nsgroot/nsg@backup01
received 52.9KB stream in 1 seconds (52.9KB/sec)
receiving full stream of nsgroot/nsg/services_persistent@backup01 into nsgroot/nsg/services_persistent@backup01
received 52.9KB stream in 1 seconds (52.9KB/sec)
---lots of lines of output---
cannot mount 'nsgroot/nsg': mountpoint or dataset is busy
cannot mount 'nsgroot/nsg/services_persistent': mountpoint or dataset is busy
---etc---

I couldn't see the data until I rebooted.

Cath O'Deray · Nov 15, 2021

I don't entirely understand that last chunk, but did part of the routine make the relevant part of the received @backup01 your boot environment, and so on?