Solved What's using up my disk space?

E

et.fz

Hi,

Trying to wrap my head around all this. The disk is 500 GB. The 200 GB home directory figure should be correct, but there shouldn't be much more than that. I'm going by the output of zpool list as the absolute truth, as far as available disk space goes. Though I was unable to write a file earlier due to lack of disk space, despite the zpool only showing some 92% used. Extended the (virtual) disk after that. The size of chroot is looking mighty suspicious, but that should be all mount points. I suspect it's got something to do with the zroot/usr/home dataset somehow, but I don't know what all the data points mean.

Pasting everything I think might be of relevance.

Code: 
# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot   496G   404G  91.7G        -         -    58%    81%  1.00x    ONLINE  -

Code: 
# zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot                404G  76.3G    88K  none
zroot/ROOT          7.60G  76.3G    88K  none
zroot/ROOT/default  7.60G  76.3G  6.01G  /
zroot/backup          88K  76.3G    88K  none
zroot/chroot          88K  76.3G    88K  none
zroot/tmp           6.95M  76.3G  6.95M  /tmp
zroot/usr            392G  76.3G    88K  /usr
zroot/usr/home       391G  76.3G   210G  /usr/home
zroot/usr/local     1002M  76.3G   973M  /usr/local
zroot/usr/ports      140K  76.3G   140K  /usr/ports
zroot/usr/src         88K  76.3G    88K  /usr/src
zroot/var           4.10G  76.3G    88K  /var
zroot/var/audit      128K  76.3G   128K  /var/audit
zroot/var/crash       88K  76.3G    88K  /var/crash
zroot/var/db        4.01G  76.3G  2.79G  /var/db
zroot/var/db/mysql  1.21G  76.3G  1.21G  /var/db/mysql
zroot/var/log       98.2M  76.3G  98.2M  /var/log
zroot/var/mail       136K  76.3G   136K  /var/mail
zroot/var/tmp         88K  76.3G    88K  /var/tmp

Code: 
# zfs list -t snapshot
NAME                            USED  AVAIL  REFER  MOUNTPOINT
zroot/ROOT/default@2025-11-27   103M      -  6.26G  -
zroot/ROOT/default@2025-11-28   101M      -  6.26G  -
zroot/ROOT/default@2025-11-29   101M      -  6.26G  -
zroot/ROOT/default@2025-11-30   101M      -  6.26G  -
zroot/ROOT/default@2025-12-01   101M      -  6.26G  -
zroot/ROOT/default@2025-12-02   101M      -  6.26G  -
zroot/ROOT/default@2025-12-03   102M      -  6.26G  -
zroot/ROOT/default@2025-12-04   102M      -  6.26G  -
zroot/ROOT/default@2025-12-05   103M      -  6.26G  -
zroot/ROOT/default@2025-12-06   103M      -  6.26G  -
zroot/ROOT/default@2025-12-07   103M      -  6.26G  -
zroot/ROOT/default@2025-12-08   103M      -  6.26G  -
zroot/ROOT/default@2025-12-09   103M      -  6.26G  -
zroot/usr@2025-11-27              0B      -    88K  -
zroot/usr@2025-11-28              0B      -    88K  -
zroot/usr@2025-11-29              0B      -    88K  -
zroot/usr@2025-11-30              0B      -    88K  -
zroot/usr@2025-12-01              0B      -    88K  -
zroot/usr@2025-12-02              0B      -    88K  -
zroot/usr@2025-12-03              0B      -    88K  -
zroot/usr@2025-12-04              0B      -    88K  -
zroot/usr@2025-12-05              0B      -    88K  -
zroot/usr@2025-12-06              0B      -    88K  -
zroot/usr@2025-12-07              0B      -    88K  -
zroot/usr@2025-12-08              0B      -    88K  -
zroot/usr@2025-12-09              0B      -    88K  -
zroot/usr/home@2025-11-27         0B      -   180G  -
zroot/usr/home@2025-11-28         0B      -   180G  -
zroot/usr/home@2025-11-29         0B      -   180G  -
zroot/usr/home@2025-11-30         0B      -   180G  -
zroot/usr/home@2025-12-01         0B      -   180G  -
zroot/usr/home@2025-12-02      76.7M      -   180G  -
zroot/usr/home@2025-12-03      2.37M      -   180G  -
zroot/usr/home@2025-12-04      1.83M      -   180G  -
zroot/usr/home@2025-12-05       896K      -   180G  -
zroot/usr/home@2025-12-06       888K      -   180G  -
zroot/usr/home@2025-12-07       872K      -   180G  -
zroot/usr/home@2025-12-08       864K      -   180G  -
zroot/usr/home@2025-12-09      5.27G      -  39.1G  -
zroot/usr/local@2025-11-27        0B      -   973M  -
zroot/usr/local@2025-11-28        0B      -   973M  -
zroot/usr/local@2025-11-29        0B      -   973M  -
zroot/usr/local@2025-11-30        0B      -   973M  -
zroot/usr/local@2025-12-01        0B      -   973M  -
zroot/usr/local@2025-12-02        0B      -   973M  -
zroot/usr/local@2025-12-03        0B      -   973M  -
zroot/usr/local@2025-12-04        0B      -   973M  -
zroot/usr/local@2025-12-05        0B      -   973M  -
zroot/usr/local@2025-12-06        0B      -   973M  -
zroot/usr/local@2025-12-07        0B      -   973M  -
zroot/usr/local@2025-12-08        0B      -   973M  -
zroot/usr/local@2025-12-09        0B      -   973M  -

Code: 
# bectl list
BE      Active Mountpoint Space Created
default NR     /          7.60G 2019-09-26 16:35

Code: 
# df -h
Filesystem                               Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default                        82G    6.0G     76G     7%    /
devfs                                    1.0K      0B    1.0K     0%    /dev
zroot/tmp                                 76G    6.9M     76G     0%    /tmp
zroot/usr/local                           77G    973M     76G     1%    /usr/local
zroot/var/log                             76G     98M     76G     0%    /var/log
zroot/var/db                              79G    2.8G     76G     4%    /var/db
zroot/var/mail                            76G    136K     76G     0%    /var/mail
zroot/usr/ports                           76G    140K     76G     0%    /usr/ports
zroot/var/audit                           76G    128K     76G     0%    /var/audit
zroot/var/crash                           76G     88K     76G     0%    /var/crash
zroot/usr/src                             76G     88K     76G     0%    /usr/src
zroot/var/tmp                             76G     88K     76G     0%    /var/tmp
zroot/var/db/mysql                        77G    1.2G     76G     2%    /var/db/mysql
zroot/usr/home                           286G    210G     76G    73%    /usr/home
/usr/home/esp                            286G    210G     76G    73%    /chroot/esp/home/esp
/var/log/httpd/esp                        76G     98M     76G     0%    /chroot/esp/var/log/httpd
/var/log/php-fpm/esp                      76G     98M     76G     0%    /chroot/esp/var/log/php-fpm
[more chroot mount points...]

Code: 
# du -hd1 / | sort -h
512B    /media
512B    /net
512B    /proc
512B    /zroot
1.0K    /mnt
3.0K    /dev
189K    /libexec
989K    /bin
2.2M    /etc
3.9M    /sbin
7.0M    /tmp
 10M    /lib
 10M    /rescue
155M    /root
179M    /boot
3.5G    /backup
5.3G    /var
210G    /chroot
212G    /usr
431G    /
 
et.fz said:
The size of chroot is looking mighty suspicious, but that should be all mount points.
Click to expand...
Maybe some data got written to it when those mounts were absent? If you mount a filesystem over it that data is basically hidden underneath the mount.
 
I suppose I couldn't rule that out, but is there any way to verify without unmounting? Doing so is going to kill the users, isn't it?

These numbers also seem suspicious, though:
Code: 
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot/usr            392G  76.3G    88K  /usr
zroot/usr/home       391G  76.3G   210G  /usr/home
zroot/usr/local     1002M  76.3G   973M  /usr/local
zroot/usr/ports      140K  76.3G   140K  /usr/ports
zroot/usr/src         88K  76.3G    88K  /usr/src
Code: 
NAME                            USED  AVAIL  REFER  MOUNTPOINT
zroot/usr/home@2025-12-08       864K      -   180G  -
zroot/usr/home@2025-12-09      5.27G      -  39.1G  -
So the bulk of it seems to be attributed to the home dataset? But the snapshots should effectively only use as much space as is indicated by USED, as far as I understand. I don't know why today's snapshot's REFER value is so different also?
 
et.fz said:
but is there any way to verify without unmounting?
Click to expand...
Nope. Can't access the filesystem underneath the mount. That's why it's effectively "hidden", but it still takes up space.

et.fz said:
Doing so is going to kill the users, isn't it?
Click to expand...
You can't unmount the filesystem if it's in use.

Code: 
zroot/usr            392G  76.3G    88K  /usr
zroot/usr isn't mounted, it's just a 'placeholder' for the other datasets under it. It's unmounted on a default installation that is.
Not sure why you have snapshots of it, but those shouldn't use up much space (there are zero changes).
Code: 
# zfs list -o name,mountpoint,canmount zroot/usr
NAME       MOUNTPOINT  CANMOUNT
zroot/usr  /usr        off

et.fz said:
But the snapshots should effectively only use as much space as is indicated by USED, as far as I understand. I don't know why today's snapshot's REFER value is so different also?
Click to expand...
Snapshots are specific state of the dataset at that time, the more the original changes the bigger the snapshot will become.
 
Code: 
zfs list -ro space
Is always a good place to start.

For the individual snapshot USED parameter, it can be very confusing. The value only shows the data that is unique to that snapshot. So a large file present in two snapshots, but not on the current filesystem, won’t count against the USED value of either snapshot.

This is in part, I believe, so you can predict the space that will be made available if you were to delete a single snapshot. Once that snapshot is deleted, the USED parameters of other snapshots will update.

Sometimes the written and refer parameters are more useful in figuring out where space is being consumed.
 
Also note you can mount a snapshot in an alternate location to investigate its contents; can be useful when trying to figure out if a mount is hiding some data. (Or browse into .zfs/snapshot at the live filesytem’s mount point)
 
Thank you. zfs list -ro space made it more clear that space was in fact tied up in snapshots. I also neglected to take into account any changes that seems to have been made since the time of the most recent snapshot. (some time earlier that day)
 
You must log in or register to reply here.
Back
Top