ZFS `df` and `zpool list` showing disk practically full

[yha]

Hello,

I'm no ZFS expert and I never messed with my ZFS setup on this machine (aside from whatever the installer did when I installed FreeBSD), but today I was trying to zero-out some large text files with cat /dev/zero > MYFILES* (inside /home/yousef/files), and at one point I received some error messages saying the disk was full (before that I had mistakenly used cat /dev/null > MYFILES*`) . I had to hard reset the machine. After booting up again, df and zpool list show that my pool is full.

NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
root   314G   301G  12.5G        -         -    75%    96%  1.00x    ONLINE  -

I'm pretty sure I have plenty of free space as far my files are concerned (at least 150+ GB of free space from the available 300GB), but whatever I did is causing the system to report a full disk. Can anyone give me any pointers on how to troubleshoot this?

Thanks.

 

[Emrion]

You may have snapshots.

What is the output of: zfs list -t snap

 

[yha]

It says 'no datasets available'.

 

[Emrion]

Ok. No snapshot, then. What about just: zfs list

 

[Erichans]

For more detailed info: zfs list -o space

 

[yha]

This is the output from `zfs list`:

NAME   USED  AVAIL     REFER  MOUNTPOINT
root   301G  2.77G      301G  none

To prove my point, here's the output of du -sh for every directory under /:

legion5# ls -l
total 397
-rw-r--r--   2 root  wheel    1023 Apr  9  2021 .cshrc
-rw-r--r--   2 root  wheel     507 May 16  2022 .profile
-r--r--r--   1 root  wheel    6109 Apr  9  2021 COPYRIGHT
drwxr-xr-x   2 root  wheel      46 Dec  2 06:45 bin
drwxr-xr-x  15 root  wheel      68 Jan  6 15:43 boot
drwxr-xr-x   3 root  wheel       3 May 18  2022 compat
drwxrwxrwx   1 root  wheel    4096 Dec 25 21:05 data
dr-xr-xr-x  13 root  wheel     512 Jan  6 15:45 dev
-rw-------   1 root  wheel    4096 Jan  6 15:45 entropy
drwxr-xr-x  28 root  wheel     111 Dec  2 06:45 etc
lrwxr-xr-x   1 root  wheel       8 Apr 19  2022 home -> usr/home
drwxr-xr-x   5 root  wheel      67 Nov  6 15:54 lib
drwxr-xr-x   3 root  wheel       5 May 16  2022 libexec
drwxr-xr-x   2 root  wheel       2 Apr  9  2021 media
drwxr-xr-x   2 root  wheel       2 Apr  9  2021 mnt
drwxr-xr-x   2 root  wheel       2 Apr  9  2021 net
dr-xr-xr-x   1 root  wheel       0 Jan  6 16:50 proc
drwxr-xr-x   2 root  wheel     150 Dec  2 06:45 rescue
drwxr-x---   8 root  wheel      16 Jan  6 16:48 root
drwxr-xr-x   2 root  wheel     149 Dec  2 06:45 sbin
drwxrwxrwx   1 root  wheel  131072 Jan  1  1970 shared
lrwxr-xr-x   1 root  wheel      11 Apr  9  2021 sys -> usr/src/sys
drwxrwxrwt  19 root  wheel      40 Jan  6 16:06 tmp
drwxrwxrwx   1 root  wheel    8192 Dec 27 00:39 udata
drwxr-xr-x  16 root  wheel      16 Apr 19  2022 usr
drwxr-xr-x  25 root  wheel      25 Jan  6 15:45 var
legion5# du -sh /bin/
1.5M    /bin/
legion5# du -sh /boot/
266M    /boot/
legion5# du -sh /compat/
5.5K    /compat/
legion5# du -sh /dev/ 
5.0K    /dev/
legion5# du -sh /etc/
2.4M    /etc/
legion5# du -sh /home/
153G    /home/
legion5# du -sh /lib/
 15M    /lib/
legion5# du -sh /libexec
249K    /libexec
legion5# du -sh /net/ 
512B    /net/
legion5# du -sh /proc/
  0B    /proc/
legion5# du -sh /rescue/
 13M    /rescue/
legion5# du -sh /root/
210M    /root/
legion5# du -sh /sbin/
7.4M    /sbin/
legion5# du -sh /sys/
516M    /sys/
legion5# du -sh /tmp/
 26M    /tmp/
legion5# du -sh /usr/
169G    /usr/
legion5# du -sh /var/
 14G    /var/

Compared to the /home/ and /usr/ directories, everything else is negligible. /usr/ is 169G and /home/ (which is symlinked to /usr/) is 153G. I did not list /data/, /udata/, and /shared/ because these are vfat/exfat partitions that live on a different disk. So the math adds up to about 170-190G and I should have about 130-140G free.

legion5% zfs list -o space
NAME  AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
root  2.77G   301G        0B    301G             0B      42.8M

 

[mer]

How about output of:
bectl list

I'm assuming that in your first post in /home/yousef/files you have files name MYFILES1, MYFILES2, ... or something similar and that's what you were initially trying to zero out? Were you actually trying to truncate them or keep them the same size but fill with zeros? I'm trying to figure out if somehow the "*" got expanded in a weird manner by the shell and wound up working on the "." and ".." entries in the directory.

Have you tried doing rm -f /home/yousef/files/MYFILES* ? I'm assuming that at this point the contents of those files don't really matter anymore.

How about running scrub on the pool? As root/sudo/su "zpool scrub root" (It looks like "root" is your pool name)

 

[SirDice]

but today I was trying to zero-out some large text files with cat /dev/zero > MYFILES*

Why? This is a pretty useless exercise with a COW filesystem, you're never actually overwriting the old data. Besides that, the command you gave will just copy zeros to those files until the filesystem is full (as you already found out). If you want to 'empty' a file use truncate(1) and set its size to 0.

 

[gotnull]

Although this won't help you to use ZFS in any better way, I just want to say you can identify what and where your bigger files are by using ncdu (written in C) or gdu(written in Go), these tools can be handy.

 

[yha]

`bectl list` gives 'libbe_init("") failed.'

I was trying to keep the files the same size but fill them with zeros. And yes, I deleted the files afterwards.

I tried scrubbing the pool as you suggested but it still reports a full disk. Running `ncdu` in the root directory reports the same results I posted in my previous post (i.e. the bulk of the used space is in /usr/ at 169G).

 

[yha]

Why? This is a pretty useless exercise with a COW filesystem, you're never actually overwriting the old data. Besides that, the command you gave will just copy zeros to those files until the filesystem is full (as you already found out). If you want to 'empty' a file use truncate(1) and set its size to 0.

Yeah, I just realized that I meant to use `dd` which will only overwrite the length of the files...

 

[VladiBG]

whatever the installer did when I installed FreeBSD

which version of FreeBSD?
head -n 4 /var/log/bsdinstall_log

 

[Crivens]

Maybe this is very simple. Yes, the pool has close to 3GB free, but you may run into the user ./. root space reservation. Does your root user have the same problem when you create a small file? Don't fill the pool up completely as root user. Things will get ugly then.

 

[yha]

legion5% zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
root 314G 301G 12.5G - - 75% 96% 1.00x ONLINE -
legion5% dd if=/dev/zero of=testfile.txt bs=1g count=1
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.493959 secs (718722655 bytes/sec)
legion5% ls -lh testfile.txt
-rw-r--r-- 1 yousef yousef 1.0G Jan 9 16:35 testfile.txt
legion5% zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
root 314G 302G 11.5G - - 75% 96% 1.00x ONLINE -
legion5%
head -n 4 /var/log/bsdinstall_log
DEBUG: Running installation step: auto
DEBUG: dialog.subr: DEBUG_SELF_INITIALIZE=[]
DEBUG: UNAME_S=[FreeBSD] UNAME_P=[amd64] UNAME_R=[13.0-RELEASE]
DEBUG: common.subr: Successfully loaded.
legion5% freebsd-version
13.1-RELEASE-p5
legion5%

This is done from my home directory. There is still a little bit of space left, but I'm not sure where the remaining 100+G "disappeared" (even though I deleted the files in /usr/home/yousef/files/).

Edit:

I should add that although I do have a little bit of space left, I'm unable to do certain tasks like upgrading packages:


Number of packages to be removed: 1
Number of packages to be installed: 14
Number of packages to be upgraded: 304
Number of packages to be reinstalled: 7

The process will require 2 GiB more space.
2 GiB to be downloaded.

Proceed with this action? [y/N]: y
pkg: Not enough space in /var/cache/pkg, needed 2197 MiB available 1901 MiB
legion5#


Edit 2:

After deleting the 1G test file I created, I was able to proceed with the `pkg upgrade`. However, it still remains that I have very little space to play with.

 

[Erichans]

For a readable layout, instead of "CMD", please use "code"*:
[code]
legion5% zpool list <etc. ...> [/code]

---

I'm no ZFS expert and I never messed with my ZFS setup on this machine (aside from whatever the installer did when I installed FreeBSD), but [...]

This is the output from `zfs list`:

NAME   USED  AVAIL     REFER  MOUNTPOINT
root   301G  2.77G      301G  none

This output says there is only one ZFS dataset that also does not have a mountpoint. This does not seem like a usual layout of ZFS datasets created by the FreeBSD installer. Please give the output of zfs mount


___
* the columns will be aligned:

legion5% zpool list   
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
root   314G   301G  12.5G        -         -    75%    96%  1.00x    ONLINE  -
<snap>

 

[Crivens]

You may use sysutils/gdmap to zero in on where the space is wasted.

 

[yha]

legion5% zfs mount
root                            /
legion5%

gdmap shows 187G total used disk space in /.

Could this be a bug of some sort? There is about 120G of missing space. Everything I tried (ncdu, du, gdmap, etc) suggests I only used about 180G out of the 300G available. Only zfs thinks the disk is full (and the system will give me write errors and whatnot because it actually thinks that the whole 300G is used).

 

[covacat]

zfs list -r -t all
zfs get all

 

[yha]

Note that prior to issuing the commands below, I deleted some large iso files so I now have about 30G extra free space as compared to before.

legion5% zfs list -r -t all
NAME   USED  AVAIL     REFER  MOUNTPOINT
root   277G  27.6G      277G  none
legion5% zfs get all
NAME  PROPERTY              VALUE                  SOURCE
root  type                  filesystem             -
root  creation              Wed Apr 20  4:23 2022  -
root  used                  277G                   -
root  available             27.6G                  -
root  referenced            277G                   -
root  compressratio         1.00x                  -
root  mounted               yes                    -
root  quota                 none                   default
root  reservation           none                   default
root  recordsize            128K                   default
root  mountpoint            none                   local
root  sharenfs              off                    default
root  checksum              fletcher4              local
root  compression           off                    default
root  atime                 on                     default
root  devices               on                     default
root  exec                  on                     default
root  setuid                on                     default
root  readonly              off                    default
root  jailed                off                    default
root  snapdir               hidden                 default
root  aclmode               discard                default
root  aclinherit            restricted             default
root  createtxg             1                      -
root  canmount              on                     default
root  xattr                 on                     default
root  copies                1                      default
root  version               5                      -
root  utf8only              off                    -
root  normalization         none                   -
root  casesensitivity       sensitive              -
root  vscan                 off                    default
root  nbmand                off                    default
root  sharesmb              off                    default
root  refquota              none                   default
root  refreservation        none                   default
root  guid                  16813533776196609815   -
root  primarycache          all                    default
root  secondarycache        all                    default
root  usedbysnapshots       0B                     -
root  usedbydataset         277G                   -
root  usedbychildren        42.6M                  -
root  usedbyrefreservation  0B                     -
root  logbias               latency                default
root  objsetid              54                     -
root  dedup                 off                    default
root  mlslabel              none                   default
root  sync                  standard               default
root  dnodesize             legacy                 default
root  refcompressratio      1.00x                  -
root  written               277G                   -
root  logicalused           276G                   -
root  logicalreferenced     276G                   -
root  volmode               default                default
root  filesystem_limit      none                   default
root  snapshot_limit        none                   default
root  filesystem_count      none                   default
root  snapshot_count        none                   default
root  snapdev               hidden                 default
root  acltype               nfsv4                  default
root  context               none                   default
root  fscontext             none                   default
root  defcontext            none                   default
root  rootcontext           none                   default
root  relatime              off                    default
root  redundant_metadata    all                    default
root  overlay               on                     default
root  encryption            off                    default
root  keylocation           none                   default
root  keyformat             none                   default
root  pbkdf2iters           0                      default
root  special_small_blocks  0                      default
legion5%

Edit:

Here's a screenshot of gdmap after deleting the 30G or so of iso files:

 

[VladiBG]

This is not the default dataset which the bsdinstall of 13.0-RELEASE is creating. It create separate datasets for /home and /usr and doesn't symlink them under one dataset.
When you upgrading from 13.0 -> 13.1 freebsd-update is set to create a snapshot unless you modify freebsd-update.conf(5) (CreateBootEnv)
Can you show the history of the zpool it will list how was the datasets were created.
Also check if you unhide the .zfs/snapshot directory (snapdir) is there something in it.

root@bsdtest:~ # zfs list
NAME                 USED  AVAIL     REFER  MOUNTPOINT
zroot                613M   106G      153K  /zroot
zroot/ROOT           610M   106G      153K  none
zroot/ROOT/default   610M   106G      610M  /
zroot/tmp            153K   106G      153K  /tmp
zroot/usr            614K   106G      153K  /usr
zroot/usr/home       153K   106G      153K  /usr/home
zroot/usr/ports      153K   106G      153K  /usr/ports
zroot/usr/src        153K   106G      153K  /usr/src
zroot/var            997K   106G      153K  /var
zroot/var/audit      153K   106G      153K  /var/audit
zroot/var/crash      153K   106G      153K  /var/crash
zroot/var/log        230K   106G      230K  /var/log
zroot/var/mail       153K   106G      153K  /var/mail
zroot/var/tmp        153K   106G      153K  /var/tmp
root@bsdtest:~ # zfs mount
zroot/ROOT/default              /
zroot/var/log                   /var/log
zroot/usr/src                   /usr/src
zroot/usr/ports                 /usr/ports
zroot/tmp                       /tmp
zroot/var/crash                 /var/crash
zroot/usr/home                  /usr/home
zroot/var/tmp                   /var/tmp
zroot/var/audit                 /var/audit
zroot                           /zroot
zroot/var/mail                  /var/mail

root@bsdtest:~ # zpool history
History for 'zroot':
2023-01-14.13:36:26 zpool create -o altroot=/mnt -O compress=lz4 -O atime=off -m none -f zroot raidz1 da0p4 da1p4 da2p4 da3p4 da4p4
2023-01-14.13:36:26 zfs create -o mountpoint=none zroot/ROOT
2023-01-14.13:36:27 zfs create -o mountpoint=/ zroot/ROOT/default
2023-01-14.13:36:28 zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp
2023-01-14.13:36:28 zfs create -o mountpoint=/usr -o canmount=off zroot/usr
2023-01-14.13:36:29 zfs create zroot/usr/home
2023-01-14.13:36:30 zfs create -o setuid=off zroot/usr/ports
2023-01-14.13:36:31 zfs create zroot/usr/src
2023-01-14.13:36:32 zfs create -o mountpoint=/var -o canmount=off zroot/var
2023-01-14.13:36:33 zfs create -o exec=off -o setuid=off zroot/var/audit
2023-01-14.13:36:34 zfs create -o exec=off -o setuid=off zroot/var/crash
2023-01-14.13:36:35 zfs create -o exec=off -o setuid=off zroot/var/log
2023-01-14.13:36:36 zfs create -o atime=on zroot/var/mail
2023-01-14.13:36:37 zfs create -o setuid=off zroot/var/tmp
2023-01-14.13:36:39 zfs set mountpoint=/zroot zroot
2023-01-14.13:36:39 zpool set bootfs=zroot/ROOT/default zroot
2023-01-14.13:36:39 zpool set cachefile=/mnt/boot/zfs/zpool.cache zroot
2023-01-14.13:36:46 zfs set canmount=noauto zroot/ROOT/default

 

[Erichans]

This is not the default dataset which the bsdinstall of 13.0-RELEASE is creating.

I agree.

legion5% zfs list -r -t all
NAME   USED  AVAIL     REFER  MOUNTPOINT
root   277G  27.6G      277G  none
legion5% zfs get all
NAME  PROPERTY              VALUE                  SOURCE
root  type                  filesystem             -
root  creation              Wed Apr 20  4:23 2022  -
root  used                  277G                   -
root  available             27.6G                  -
root  referenced            277G                   -
root  compressratio         1.00x                  -
root  mounted               yes                    -
<snap>
root  mountpoint            none                   local

zfsconcepts(7):

DESCRIPTION
[...]
   Mount Points
[...]
     A file system mountpoint property of none prevents	the file system from
     being mounted.

I cannot account for this.

 

[yha]

legion5# zpool history
History for 'root':
2022-04-20.04:23:04 zpool create -f -m none -o cachefile=/tmp/bsdinstall_boot/zfs/zpool.cache -O checksum=fletcher4 root /dev/nvd0p8
2023-01-07.17:32:10 zpool scrub root

legion5#

 

[yha]

Now that I've hit a dead-end, I was wondering if it's possible to create a different pool on an external drive, copy the files over, then boot a live CD and re-copy everything back to the original drive. I suppose a byte-by-byte copy won't work (a snapshot or whatever it's called), but perhaps a file-by-file copy would get rid of this issue and free up my "stolen" space. Again, I'm no ZFS expert so I would really appreciate if someone would give me the specific commands I need to use. Would this kind of solution work and get my system in the exact same state it was before (like with the bootloader and everything, because I do have other paritions on the drive for Windows and a shared parition)?

 

[Erichans]

It's all undiscovered territory but yes, you could try that, as I similarly suggested here

 

[rihad]

You may have snapshots.

What is the output of: zfs list -t snap

Hm, I do have them on my 13.1-13.2 machines. Is this unasked creation a new feature? Does freebsd-update create them and leaves them around?

[rihad@master ~]$ zfs list -t snap
NAME                                       USED  AVAIL     REFER  MOUNTPOINT
zroot/ROOT/default@2022-08-17-10:53:03-0   260M      -     4.56G  -
zroot/ROOT/default@2022-09-04-16:20:27-0   178M      -     4.56G  -
zroot/ROOT/default@2022-11-06-21:14:03-0   195M      -     4.71G  -
zroot/ROOT/default@2022-12-02-14:46:16-0   132M      -     4.75G  -
zroot/ROOT/default@2023-03-16-21:54:51-0   184M      -     4.87G  -
zroot/ROOT/default@2023-05-13-18:02:55-0  5.44M      -     5.53G  -
zroot/ROOT/default@2023-05-13-18:08:14-0     5M      -     5.61G  -

Any way to disable their creation anywhere? Is it safe to remove them?


UPD: Oh, I can see there's some sort of installation FS bound to a snapshot, so it can't be deleted due to the dependency. Adding -R flag to zfs destroy removes the snapshot and the FS.