Solved ZFS zpool Full

EDIT

I have solved my problem. I had an external drive (one of those Western Digital Easystores) that I was using to periodically back up some files. For whatever reason, the fstab entry for the easystore was commented out. I deleted all of the files that had been unknowingly copied to an unmounted filesystem, and then remounted drive. My disk usage dropped from 440+ GB, to less than 6GB, which is more accurate. Thanks for all the suggestions. Hopefully if someone else has problems with filesystems filling up, this will at least serve as a cautionary tale if nothing else.



I am new to FreeBSD and am trying to figure out why my zroot pool is almost completely full. I have 4 disks in my server: 2 are used for the root (zroot) drive, and the other 2 are used for everything else (NAS, iocage jails, etc.). I was getting errors when using sudo that my drives were full. I ended up clearing out /tmp and then, when the error went away, not looking into it any further. But today I decided to re-investigate and noticed that my zroot pool is nearly full, despite only having samba and a few other packages installed, and the user accounts for 4 people, who's home directories are completely empty (aside from mine, which has 2 git repos of a few megabytes a piece). How do I clear out the space? Do I need to clear out any space?

Code:
brian@server:~ % zpool list
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zroot      460G   444G  16.1G        -         -    68%    96%  1.00x  ONLINE  -
zstorage  5.44T   609G  4.84T        -         -     1%    10%  1.00x  ONLINE  -
brian@server:~ % zfs list zroot
NAME    USED  AVAIL  REFER  MOUNTPOINT
zroot   444G  1.69G    88K  /zroot
brian@server:~ % zfs list
NAME                                         USED  AVAIL  REFER  MOUNTPOINT
zroot                                        444G  1.69G    88K  /zroot
zroot/ROOT                                   441G  1.69G    88K  none
zroot/ROOT/default                           441G  1.69G   441G  /
zroot/tmp                                     88K  1.69G    88K  /tmp
zroot/usr                                   3.03G  1.69G    88K  /usr
zroot/usr/home                              1.67G  1.69G  1.67G  /usr/home
zroot/usr/ports                              691M  1.69G   691M  /usr/ports
zroot/usr/src                                704M  1.69G   704M  /usr/src
zroot/var                                    164M  1.69G    88K  /var
zroot/var/audit                               88K  1.69G    88K  /var/audit
zroot/var/crash                              161M  1.69G   161M  /var/crash
zroot/var/log                               2.10M  1.69G  2.10M  /var/log
zroot/var/mail                               732K  1.69G   732K  /var/mail
zroot/var/tmp                                 88K  1.69G    88K  /var/tmp
zstorage                                     609G  4.67T    88K  /zstorage
zstorage/iocage                             51.4G  4.67T   104K  /zstorage/iocage
zstorage/iocage/download                     371M  4.67T    88K  /zstorage/iocage/download
zstorage/iocage/download/12.1-RELEASE        371M  4.67T   371M  /zstorage/iocage/download/12.1-RELEASE
zstorage/iocage/images                        88K  4.67T    88K  /zstorage/iocage/images
zstorage/iocage/jails                       49.6G  4.67T    96K  /zstorage/iocage/jails
zstorage/iocage/jails/elasticsearch         1.22G  4.67T    92K  /zstorage/iocage/jails/elasticsearch
zstorage/iocage/jails/elasticsearch/data      88K  4.67T    88K  none
zstorage/iocage/jails/elasticsearch/root    1.22G  4.67T  2.54G  /zstorage/iocage/jails/elasticsearch/root
zstorage/iocage/jails/email                 11.2M  4.67T    92K  /zstorage/iocage/jails/email
zstorage/iocage/jails/email/root            11.1M  4.67T  1.36G  /zstorage/iocage/jails/email/root
zstorage/iocage/jails/gitea                 1.16G  4.67T    96K  /zstorage/iocage/jails/gitea
zstorage/iocage/jails/gitea/data              88K  4.67T    88K  none
zstorage/iocage/jails/gitea/root            1.16G  4.67T  2.50G  /zstorage/iocage/jails/gitea/root
zstorage/iocage/jails/http                  47.2G  4.67T    92K  /zstorage/iocage/jails/http
zstorage/iocage/jails/http/data               88K  4.67T    88K  none
zstorage/iocage/jails/http/root             47.2G  4.67T  46.8G  /zstorage/iocage/jails/http/root
zstorage/iocage/jails/vpn                   90.5M  4.67T    92K  /zstorage/iocage/jails/vpn
zstorage/iocage/jails/vpn/data                88K  4.67T    88K  none
zstorage/iocage/jails/vpn/root              90.4M  4.67T  1.43G  /zstorage/iocage/jails/vpn/root
zstorage/iocage/log                          124K  4.67T   124K  /zstorage/iocage/log
zstorage/iocage/releases                    1.39G  4.67T    88K  /zstorage/iocage/releases
zstorage/iocage/releases/12.1-RELEASE       1.39G  4.67T    88K  /zstorage/iocage/releases/12.1-RELEASE
zstorage/iocage/releases/12.1-RELEASE/root  1.39G  4.67T  1.36G  /zstorage/iocage/releases/12.1-RELEASE/root
zstorage/iocage/templates                     88K  4.67T    88K  /zstorage/iocage/templates
zstorage/nextcloud                            88K  4.67T    88K  /zstorage/nextcloud
zstorage/repositories                       1.99M  4.67T  1.99M  /zstorage/repositories
zstorage/shares                              558G  4.67T   558G  /zstorage/shares
brian@server:~ % geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 500107862016 (466G)
   Sectorsize: 512
   Mode: r2w2e3
   descr: WD5003ABYX-88 LEN
   lunid: 50014ee0ae782ff9
   ident: WD-WMAYP7980281
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 6001175126016 (5.5T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   descr: ST6000VN0033-2EE110
   lunid: 5000c500c54e63c2
   ident: ZADAKFPZ
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 6001175126016 (5.5T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   descr: ST6000VN0033-2EE110
   lunid: 5000c500c54ffe85
   ident: ZADAKE3G
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: ada3
Providers:
1. Name: ada3
   Mediasize: 500107862016 (466G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: WD5003ABYX-88 LEN
   lunid: 50014ee003cd6549
   ident: WD-WMAYP8051215
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: da0
Providers:
1. Name: da0
   Mediasize: 8001562869760 (7.3T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   descr: WD easystore 25FB
   lunname: WD      easystore 25FB  7SH21MBD
   lunid: 50014eef0ae4d721
   ident: 37534832314D4244
   rotationrate: 5400
   fwsectors: 63
   fwheads: 255
 
Last edited:
Check root's home directory, there's about 441GB (of the 460GB) stored in your root (/). If it's not stored in root's home directory you may want to look for other directories in your root filesystem (/), that's where most of the data is being stored. You can skip checking /usr/, /tmp, etc. as these are separated filesystems and don't take up much space (look closely at the USED column from your zfs list output).

Code:
zroot/ROOT/default                           441G  1.69G   441G  /
 
Hi, maybe snapshots? ,look with

Code:
zfs list -t snapshot

I do have snapshots but I don't think that's what's taking up all the space.

Code:
brian@server:~ % sudo zfs list -t snapshot
Password:
NAME                                                                   USED  AVAIL  REFER  MOUNTPOINT
zstorage/iocage/jails/elasticsearch@2020-09-24_19:28:06                 60K      -    92K  -
zstorage/iocage/jails/elasticsearch/data@2020-09-24_19:28:06              0      -    88K  -
zstorage/iocage/jails/elasticsearch/root@2020-09-24_19:28:06          20.4M      -  2.53G  -
zstorage/iocage/jails/http@2020-09-23_15:41:28                          60K      -    92K  -
zstorage/iocage/jails/http@c579f4b4-a114-4525-be19-0b4adf82a047         60K      -    92K  -
zstorage/iocage/jails/http@http1                                        56K      -    92K  -
zstorage/iocage/jails/http@http-gitea                                   56K      -    92K  -
zstorage/iocage/jails/http/data@2020-09-23_15:41:28                       0      -    88K  -
zstorage/iocage/jails/http/data@c579f4b4-a114-4525-be19-0b4adf82a047      0      -    88K  -
zstorage/iocage/jails/http/data@http1                                     0      -    88K  -
zstorage/iocage/jails/http/data@http-gitea                                0      -    88K  -
zstorage/iocage/jails/http/root@2020-09-23_15:41:28                    187M      -  22.8G  -
zstorage/iocage/jails/http/root@c579f4b4-a114-4525-be19-0b4adf82a047   172M      -  22.8G  -
zstorage/iocage/jails/http/root@http1                                     0      -  48.8G  -
zstorage/iocage/jails/http/root@http-gitea                                0      -  48.8G  -
zstorage/iocage/releases/12.1-RELEASE/root@http                       11.8M      -  1.36G  -
zstorage/iocage/releases/12.1-RELEASE/root@elasticsearch               184K      -  1.36G  -
zstorage/iocage/releases/12.1-RELEASE/root@email                       184K      -  1.36G  -
brian@server:~ %
 
or maybe a checkpoint? see zpool status for a checkpoint in "config"

Doesn't look like it.

Code:
brian@server:~ % sudo zspool status
sudo: zspool: command not found
brian@server:~ % sudo zpool status
  pool: zroot
 state: ONLINE
  scan: resilvered 36.1M in 0 days 00:00:05 with 0 errors on Wed May 13 19:34:10 2020
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p3  ONLINE       0     0     0
        ada3p3  ONLINE       0     0     0

errors: No known data errors

  pool: zstorage
 state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    zstorage    ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada1    ONLINE       0     0     0
        ada2    ONLINE       0     0     0

errors: No known data errors
 
Perhaps you stored some things in /zstorage when the pool wasn't mounted? The newly mounted pool would then "cover" those files, making them invisible but they would still take up space on the root filesystem.
 
Perhaps you stored some things in /zstorage when the pool wasn't mounted? The newly mounted pool would then "cover" those files, making them invisible but they would still take up space on the root filesystem.
How would I check that? Do I need to unmount the pool?
 
Back
Top