ZFS ZFS zpool Full

brianjm

New Member

Reaction score: 2
Messages: 11

I am new to FreeBSD and am trying to figure out why my zroot pool is almost completely full. I have 4 disks in my server: 2 are used for the root (zroot) drive, and the other 2 are used for everything else (NAS, iocage jails, etc.). I was getting errors when using sudo that my drives were full. I ended up clearing out /tmp and then, when the error went away, not looking into it any further. But today I decided to re-investigate and noticed that my zroot pool is nearly full, despite only having samba and a few other packages installed, and the user accounts for 4 people, who's home directories are completely empty (aside from mine, which has 2 git repos of a few megabytes a piece). How do I clear out the space? Do I need to clear out any space?

Code:
brian@server:~ % zpool list
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zroot      460G   444G  16.1G        -         -    68%    96%  1.00x  ONLINE  -
zstorage  5.44T   609G  4.84T        -         -     1%    10%  1.00x  ONLINE  -
brian@server:~ % zfs list zroot
NAME    USED  AVAIL  REFER  MOUNTPOINT
zroot   444G  1.69G    88K  /zroot
brian@server:~ % zfs list
NAME                                         USED  AVAIL  REFER  MOUNTPOINT
zroot                                        444G  1.69G    88K  /zroot
zroot/ROOT                                   441G  1.69G    88K  none
zroot/ROOT/default                           441G  1.69G   441G  /
zroot/tmp                                     88K  1.69G    88K  /tmp
zroot/usr                                   3.03G  1.69G    88K  /usr
zroot/usr/home                              1.67G  1.69G  1.67G  /usr/home
zroot/usr/ports                              691M  1.69G   691M  /usr/ports
zroot/usr/src                                704M  1.69G   704M  /usr/src
zroot/var                                    164M  1.69G    88K  /var
zroot/var/audit                               88K  1.69G    88K  /var/audit
zroot/var/crash                              161M  1.69G   161M  /var/crash
zroot/var/log                               2.10M  1.69G  2.10M  /var/log
zroot/var/mail                               732K  1.69G   732K  /var/mail
zroot/var/tmp                                 88K  1.69G    88K  /var/tmp
zstorage                                     609G  4.67T    88K  /zstorage
zstorage/iocage                             51.4G  4.67T   104K  /zstorage/iocage
zstorage/iocage/download                     371M  4.67T    88K  /zstorage/iocage/download
zstorage/iocage/download/12.1-RELEASE        371M  4.67T   371M  /zstorage/iocage/download/12.1-RELEASE
zstorage/iocage/images                        88K  4.67T    88K  /zstorage/iocage/images
zstorage/iocage/jails                       49.6G  4.67T    96K  /zstorage/iocage/jails
zstorage/iocage/jails/elasticsearch         1.22G  4.67T    92K  /zstorage/iocage/jails/elasticsearch
zstorage/iocage/jails/elasticsearch/data      88K  4.67T    88K  none
zstorage/iocage/jails/elasticsearch/root    1.22G  4.67T  2.54G  /zstorage/iocage/jails/elasticsearch/root
zstorage/iocage/jails/email                 11.2M  4.67T    92K  /zstorage/iocage/jails/email
zstorage/iocage/jails/email/root            11.1M  4.67T  1.36G  /zstorage/iocage/jails/email/root
zstorage/iocage/jails/gitea                 1.16G  4.67T    96K  /zstorage/iocage/jails/gitea
zstorage/iocage/jails/gitea/data              88K  4.67T    88K  none
zstorage/iocage/jails/gitea/root            1.16G  4.67T  2.50G  /zstorage/iocage/jails/gitea/root
zstorage/iocage/jails/http                  47.2G  4.67T    92K  /zstorage/iocage/jails/http
zstorage/iocage/jails/http/data               88K  4.67T    88K  none
zstorage/iocage/jails/http/root             47.2G  4.67T  46.8G  /zstorage/iocage/jails/http/root
zstorage/iocage/jails/vpn                   90.5M  4.67T    92K  /zstorage/iocage/jails/vpn
zstorage/iocage/jails/vpn/data                88K  4.67T    88K  none
zstorage/iocage/jails/vpn/root              90.4M  4.67T  1.43G  /zstorage/iocage/jails/vpn/root
zstorage/iocage/log                          124K  4.67T   124K  /zstorage/iocage/log
zstorage/iocage/releases                    1.39G  4.67T    88K  /zstorage/iocage/releases
zstorage/iocage/releases/12.1-RELEASE       1.39G  4.67T    88K  /zstorage/iocage/releases/12.1-RELEASE
zstorage/iocage/releases/12.1-RELEASE/root  1.39G  4.67T  1.36G  /zstorage/iocage/releases/12.1-RELEASE/root
zstorage/iocage/templates                     88K  4.67T    88K  /zstorage/iocage/templates
zstorage/nextcloud                            88K  4.67T    88K  /zstorage/nextcloud
zstorage/repositories                       1.99M  4.67T  1.99M  /zstorage/repositories
zstorage/shares                              558G  4.67T   558G  /zstorage/shares
brian@server:~ % geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 500107862016 (466G)
   Sectorsize: 512
   Mode: r2w2e3
   descr: WD5003ABYX-88 LEN
   lunid: 50014ee0ae782ff9
   ident: WD-WMAYP7980281
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 6001175126016 (5.5T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   descr: ST6000VN0033-2EE110
   lunid: 5000c500c54e63c2
   ident: ZADAKFPZ
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 6001175126016 (5.5T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   descr: ST6000VN0033-2EE110
   lunid: 5000c500c54ffe85
   ident: ZADAKE3G
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: ada3
Providers:
1. Name: ada3
   Mediasize: 500107862016 (466G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: WD5003ABYX-88 LEN
   lunid: 50014ee003cd6549
   ident: WD-WMAYP8051215
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: da0
Providers:
1. Name: da0
   Mediasize: 8001562869760 (7.3T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   descr: WD easystore 25FB
   lunname: WD      easystore 25FB  7SH21MBD
   lunid: 50014eef0ae4d721
   ident: 37534832314D4244
   rotationrate: 5400
   fwsectors: 63
   fwheads: 255
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 9,332
Messages: 33,894

Check root's home directory, there's about 441GB (of the 460GB) stored in your root (/). If it's not stored in root's home directory you may want to look for other directories in your root filesystem (/), that's where most of the data is being stored. You can skip checking /usr/, /tmp, etc. as these are separated filesystems and don't take up much space (look closely at the USED column from your zfs list output).

Code:
zroot/ROOT/default                           441G  1.69G   441G  /
 
OP
B

brianjm

New Member

Reaction score: 2
Messages: 11

Hi, maybe snapshots? ,look with

Code:
zfs list -t snapshot
I do have snapshots but I don't think that's what's taking up all the space.

Code:
brian@server:~ % sudo zfs list -t snapshot
Password:
NAME                                                                   USED  AVAIL  REFER  MOUNTPOINT
zstorage/iocage/jails/elasticsearch@2020-09-24_19:28:06                 60K      -    92K  -
zstorage/iocage/jails/elasticsearch/data@2020-09-24_19:28:06              0      -    88K  -
zstorage/iocage/jails/elasticsearch/root@2020-09-24_19:28:06          20.4M      -  2.53G  -
zstorage/iocage/jails/http@2020-09-23_15:41:28                          60K      -    92K  -
zstorage/iocage/jails/http@c579f4b4-a114-4525-be19-0b4adf82a047         60K      -    92K  -
zstorage/iocage/jails/http@http1                                        56K      -    92K  -
zstorage/iocage/jails/http@http-gitea                                   56K      -    92K  -
zstorage/iocage/jails/http/data@2020-09-23_15:41:28                       0      -    88K  -
zstorage/iocage/jails/http/data@c579f4b4-a114-4525-be19-0b4adf82a047      0      -    88K  -
zstorage/iocage/jails/http/data@http1                                     0      -    88K  -
zstorage/iocage/jails/http/data@http-gitea                                0      -    88K  -
zstorage/iocage/jails/http/root@2020-09-23_15:41:28                    187M      -  22.8G  -
zstorage/iocage/jails/http/root@c579f4b4-a114-4525-be19-0b4adf82a047   172M      -  22.8G  -
zstorage/iocage/jails/http/root@http1                                     0      -  48.8G  -
zstorage/iocage/jails/http/root@http-gitea                                0      -  48.8G  -
zstorage/iocage/releases/12.1-RELEASE/root@http                       11.8M      -  1.36G  -
zstorage/iocage/releases/12.1-RELEASE/root@elasticsearch               184K      -  1.36G  -
zstorage/iocage/releases/12.1-RELEASE/root@email                       184K      -  1.36G  -
brian@server:~ %
 
OP
B

brianjm

New Member

Reaction score: 2
Messages: 11

or maybe a checkpoint? see zpool status for a checkpoint in "config"
Doesn't look like it.

Code:
brian@server:~ % sudo zspool status
sudo: zspool: command not found
brian@server:~ % sudo zpool status
  pool: zroot
 state: ONLINE
  scan: resilvered 36.1M in 0 days 00:00:05 with 0 errors on Wed May 13 19:34:10 2020
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p3  ONLINE       0     0     0
        ada3p3  ONLINE       0     0     0

errors: No known data errors

  pool: zstorage
 state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    zstorage    ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada1    ONLINE       0     0     0
        ada2    ONLINE       0     0     0

errors: No known data errors
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 9,332
Messages: 33,894

Perhaps you stored some things in /zstorage when the pool wasn't mounted? The newly mounted pool would then "cover" those files, making them invisible but they would still take up space on the root filesystem.
 
Top