ZFS `df` and `zpool list` showing disk practically full

[mer]

Hm, I do have them on my 13.1-13.2 machines. Is this unasked creation a new feature? Does freebsd-update create them and leaves them around? Any way to disable their creation anywhere? Is it safe to remove them?

Those are new Boot Environments (BE) created by freebsd-update. It is probably safe to remove almost all of them but you need to do investigation first.

Step 1:
what is the output of the bectl list command? That will list all the boot environments on the machine, it "Active" state, the Space used and when it was created. The Active column is the important one. The current boot environment will have "NR" in the active column (Next boot, Running). You do not want to delete that one.

Step 2:
Look at the output of the bectl list command, then in chronological order oldest to newest do:
bectl destroy -o "whatever the name of the be is".
Do NOT bectl destroy the BE that has NR in the Active column.

Step 3:
If you do not want freebsd-update to create new BEs, edit the file /etc/freebsd-update.conf. The last line probably reads something like:

CreateBootEnv yes

If it is not commented out you can either comment it out or explicitly set it to no. I think at some point the default behavior of freebsd-update was flipped: if never used to automatically create a new BE, but then a decision was made to have it "always create a new BE" (always create is the current behavior).



You can find a lot of similar discussion in this thread, a good example of the bectl list output:

ZFS - Snapshot deletion dilemma - What to check before deletion

I am yet another member with the same old question. I do not have a recent backup so I need to ask the following: Following arethe outputs of #zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 457G 55.2M...

forums.freebsd.org

 

[rihad]

Thanks a lot! Shooot.... this is so unbelievably stupid of me) I removed all snapshots on all servers without waiting for replies here. This means the machine will not come up after next boot? I hope next freebsd-update will make sure the machine is bootable. I will disable this CreateBootEnv setting - it was absent enirely in 13.0 - just checked. Because its behaviour is unpredictable - it consumes tens of mb on some machines, and 250+GB on others.

 

[rihad]

This is the output:

$ bectl list
BE      Active Mountpoint Space Created
default NR     /          1.90G 2023-05-10 19:57

The machine was freebsd-updated to 13.2-RELEASE-p1 after that time.
Does it mean the good BE was lost by deleting the snapshot?

 

[rihad]

Doesn't snapshot mean something from the past? In which case this is not a reason to worry, if the unsnapshotted FS is bootable anyway?

I think this is the case, it least by looking in freebsd-update:

        # Create a boot environment if enabled
        if [ ${BOOTENV} = yes ]; then
                bectl check 2>/dev/null
                case $? in
                        0)
                                # Boot environment are supported
                                CREATEBE=yes
                                ;;
                        255)
                                # Boot environments are not supported
                                CREATEBE=no
                                ;;
                        *)
                                # If bectl returns an unexpected exit code, don't create a BE
                                CREATEBE=no
                                ;;
                esac
                if [ ${CREATEBE} = yes ]; then
                        echo -n "Creating snapshot of existing boot environment... "
                        VERSION=`freebsd-version -ku | sort -V | tail -n 1`
                        TIMESTAMP=`date +"%Y-%m-%d_%H%M%S"`
                        bectl create ${VERSION}_${TIMESTAMP}
                        if [ $? -eq 0 ]; then
                                echo "done.";
                        else
                                echo "failed."
                                exit 1
                        fi
                fi
        fi

 

[Emrion]

Yes, it's a rather new feature. It creates a Boot Environment at each update / upgrade. It's simpler to delete them with bectl (option -o) or sysutils/beadm.

You can change this behaviour with CreateBootEnv no in freebsd-update.conf(5).

Edit: sorry to be late, 4 posts for the time I write mine...

 

[mer]

It should boot as long as you don't delete the current BE you should be fine.
A snapshot is simply "how the dataset exists at the time the snapshot is taken", which is in the past.
So when freebsd-update creates the new BE, it is "the root dataset before we apply any updates". That means it's typically "last known good before we updated".

The machine was freebsd-updated to 13.2-RELEASE-p1 after that time.
Does it mean the good BE was lost by deleting the snapshot?

Nope. if you do the command:
freebsd-version -kru
in your currently booted BE it should indicate that it is on 13.2-RELEASE

freebsd-update behavior half the people like the autocreate, half don't like, the other two-thirds don't really care either way.
A lot of people will manually create a new BE, mount it and then doing chroot type of stuff will update into that. The default behavior of freebsd-update means they wind up with 2 new BEs. Having it created automatically means you can easily roll back if the update fails; but then the onus is on you to realize you need to manage the BEs better.
I'm sure it was in the Release Notes when it changed, but who reads them?

 

[chungy]

Use the bectl(8) program to manage those boot environments. freebsd-update(8) automatically creates them.

Edit: I wrote this without seeing there was a page 2 to the thread.

 

[elgrande]

# du -sh /bin/

du -h -d 1 seems easier.

 

[elgrande]

yha maybe zpool-scrub(8) helps?

 

[Erichans]

[...] I will disable this CreateBootEnv setting - it was absent enirely in 13.0 - just checked. Because its behaviour is unpredictable - it consumes tens of mb on some machines, and 250+GB on others.

Initially a snapshot will practically consume zero extra space. Snapshots form the basis of Boot Environments (BEs) and, from an implementation point of view, the creation of snapshots is almost free of charge because of the Copy-on-Write nature of ZFS; that does all the heavy lifting behind the scenes. Snapshots accumulate space when the "distance" between their sibling snapshots or the current state of the file system becomes bigger. That distance is mostly defined by the number of (ZFS) blocks that differ from one snapshot to the next.

For example, when you upgrade FreeBSD to the next minor version and have created two BEs during the upgrade process, chances are that the changes (in ZFS blocks that differ) are small. With the advancement of time these changes wil grow. Remember, for BEs only those changes are captured by the BEs that are relevant; only a special set of the file system (things that are important for the OS system parts of FreeBSD) is snapshotted for a BE (you can manipulate that set). Those two BEs created are relatively close to each other, their distance is relatively small and—most importantly—fixed. Deleting one will probably free very little space, deleting both of them will release a lot more blocks on disk, especially when the rest of the system files captured by a BE have changed with the advancement of time. When you upgrade to a next major version, deletion of the (auto) created BEs will free up more space obviously.

Perhaps have a closer look at Boot Environments:

1. Managing Boot Environments
2. Let’s Talk OpenZFS Snapshots
3. How ZFS snapshots really work and why they perform well (usually) - by Matt Ahrens

Also have a look at vermaden's message about ZFS Boot Environments. For a deeper ZFS dive I recommend the two ZFS books: FreeBSD Development: Books, Papers, Slides