ZFS Where is my disk space / zfs root ?

Hello,

I'm trying to understand what is using so much space on my disk :
Code:
# df -h
Filesystem            Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default    179G    150G     29G    84%    /
devfs                 1.0K    1.0K      0B   100%    /dev
zroot                  29G     88K     29G     0%    /zroot
Code:
# du -hsc /*
4.5K    /COPYRIGHT
1.5M    /bin
127M    /boot
3.5K    /dev
4.5K    /entropy
3.0M    /etc
512B    /home
 11M    /lib
317K    /libexec
512B    /media
1.0K    /mnt
512B    /net
512B    /proc
 11M    /rescue
 39M    /root
6.4M    /sbin
512B    /sys
 88M    /tmp
5.8G    /usr
 24G    /var
512B    /zroot
 30G    total
Code:
# zfs list -o space
NAME                AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zroot               28.9G   165G         0     88K              0       165G
zroot/ROOT          28.9G   164G         0     88K              0       164G
zroot/ROOT/default  28.9G   164G     14.2G    150G              0          0

As I understand I have 30G for files + 14.2G for snapshots, why is it claiming 150G used ?

Thank you for any help... box rebooted just before running commands above if that can help
 
And perhaps the output of zfs get all zroot/ROOT/default ?
Code:
zroot/ROOT/default  type                   filesystem             -
zroot/ROOT/default  creation               Sun Sep 23 13:21 2018  -
zroot/ROOT/default  used                   165G                   -
zroot/ROOT/default  available              28.1G                  -
zroot/ROOT/default  referenced             150G                   -
zroot/ROOT/default  compressratio          1.17x                  -
zroot/ROOT/default  mounted                yes                    -
zroot/ROOT/default  quota                  none                   default
zroot/ROOT/default  reservation            none                   default
zroot/ROOT/default  recordsize             8K                     inherited from zroot
zroot/ROOT/default  mountpoint             /                      local
zroot/ROOT/default  sharenfs               off                    default
zroot/ROOT/default  checksum               on                     default
zroot/ROOT/default  compression            lz4                    inherited from zroot
zroot/ROOT/default  atime                  off                    inherited from zroot
zroot/ROOT/default  devices                on                     default
zroot/ROOT/default  exec                   on                     default
zroot/ROOT/default  setuid                 on                     default
zroot/ROOT/default  readonly               off                    default
zroot/ROOT/default  jailed                 off                    default
zroot/ROOT/default  snapdir                hidden                 default
zroot/ROOT/default  aclmode                discard                default
zroot/ROOT/default  aclinherit             restricted             default
zroot/ROOT/default  createtxg              8                      -
zroot/ROOT/default  canmount               noauto                 local
zroot/ROOT/default  xattr                  off                    temporary
zroot/ROOT/default  copies                 1                      default
zroot/ROOT/default  version                5                      -
zroot/ROOT/default  utf8only               off                    -
zroot/ROOT/default  normalization          none                   -
zroot/ROOT/default  casesensitivity        sensitive              -
zroot/ROOT/default  vscan                  off                    default
zroot/ROOT/default  nbmand                 off                    default
zroot/ROOT/default  sharesmb               off                    default
zroot/ROOT/default  refquota               none                   default
zroot/ROOT/default  refreservation         none                   default
zroot/ROOT/default  guid                   9457849703070924973    -
zroot/ROOT/default  primarycache           all                    default
zroot/ROOT/default  secondarycache         all                    default
zroot/ROOT/default  usedbysnapshots        15.0G                  -
zroot/ROOT/default  usedbydataset          150G                   -
zroot/ROOT/default  usedbychildren         0                      -
zroot/ROOT/default  usedbyrefreservation   0                      -
zroot/ROOT/default  logbias                latency                default
zroot/ROOT/default  objsetid               51                     -
zroot/ROOT/default  dedup                  off                    default
zroot/ROOT/default  mlslabel                                      -
zroot/ROOT/default  sync                   standard               default
zroot/ROOT/default  dnodesize              legacy                 default
zroot/ROOT/default  refcompressratio       1.13x                  -
zroot/ROOT/default  written                699M                   -
zroot/ROOT/default  logicalused            192G                   -
zroot/ROOT/default  logicalreferenced      168G                   -
zroot/ROOT/default  volmode                default                default
zroot/ROOT/default  filesystem_limit       none                   default
zroot/ROOT/default  snapshot_limit         none                   default
zroot/ROOT/default  filesystem_count       none                   default
zroot/ROOT/default  snapshot_count         none                   default
zroot/ROOT/default  redundant_metadata     all                    default
zroot/ROOT/default  special_small_blocks   0                      default
zroot/ROOT/default  com.sun:auto-snapshot  true                   local
 
The output from your zfs list -o space doesn't seem complete. A default install would also have various zroot/var datasets. Was that everything it showed? If not, please post the whole thing.
 
The output from your zfs list -o space doesn't seem complete. A default install would also have various zroot/var datasets. Was that everything it showed? If not, please post the whole thing.
Well, this line
zroot/ROOT/default usedbydataset 150G -
tells us that those 150G are used by that dataset. Not snapshots, not child datasets, not other datasets.
And apparently that contradicts the output from du.
I don't have an explanation for that discrepancy and I do not think that it's something trivial or obvious.
 
Well, this line
zroot/ROOT/default usedbydataset 150G -
tells us that those 150G are used by that dataset. Not snapshots, not child datasets, not other datasets.
And apparently that contradicts the output from du.
I don't have an explanation for that discrepancy and I do not think that it's something trivial or obvious.

Exactly. I've asked him to try a different du command ( du -xcd1 /) as the one he used ( du -hsc /*) would have missed any directories starting with '/.' (like '/.foo'). I wouldn't expect there to be any/large ones, but trying to eliminate the obvious explanations.
 
The output from your zfs list -o space doesn't seem complete. A default install would also have various zroot/var datasets. Was that everything it showed? If not, please post the whole thing.
I confirm that the output above is all I have
 
Other than 25M being a little large (but at the same time reasonable depending on what services you are running) for var, I’m not sure where to go next.

You say it is like this immediately at boot? One could imagine anonymous (unlinked but still held open by a process) files leading to this state, but it should go away on reboot. You don’t have anything saving large amounts of data in extended attributes, do you?

[edit: typos]
 
You say it is like this immediately at boot? One could image anonymous (unlinked but still help open by a process) files leading to this state, but it should go away on reboot.
That was my first thought, that's why I rebooted before posting here. But I confirm that numbers were the same before the reboot (100+ days uptime). I'm not able to find when the situation starts unfortunately...

You don’t have anything saving large amounts of data in extended attributes, do you?
Maybe or maybe not, how to check that ?
 
Hum... what do you think ?

# du -hs /
26G /
# du -hsA /
46G /
# du -hsA /var/db/mysql
35G /var/db/mysql
# du -hs /var/db/mysql
17G /var/db/mysql

Could this be some sort of "too much modifications" because mysql is on constant load (many writes) ?
In which case how to tune more efficiently zfs/mysql stack ? or is it better to move mysql on another non-zfs drive ?
 
Hum... what do you think ?

# du -hs /
26G /
# du -hsA /
46G /
# du -hsA /var/db/mysql
35G /var/db/mysql
# du -hs /var/db/mysql
17G /var/db/mysql

Could this be some sort of "too much modifications" because mysql is on constant load (many writes) ?
In which case how to tune more efficiently zfs/mysql stack ? or is it better to move mysql on another non-zfs drive ?

No, nothing looks wrong here.
 
  • Like
Reactions: cks
It is not good idea to use small block size like 8kB for entire file system because it may reduce its performance.
If you want to optimize file system for database then make separate dataset for it with specific block size.

I would like to see the output of following commands:
freebsd-version -kru
zpool -V
zpool get all zroot
 
It is not good idea to use small block size like 8kB for entire file system because it may reduce its performance.
If you want to optimize file system for database then make separate dataset for it with specific block size.

I would like to see the output of following commands:
freebsd-version -kru
zpool -V
zpool get all zroot
# freebsd-version -kru
12.4-RELEASE-p1
12.4-RELEASE
12.4-RELEASE-p2

# zpool -V zroot
unrecognized command '-V'
usage: zpool command args ...

# zpool get all zroot
NAME PROPERTY VALUE SOURCE
zroot size 200G -
zroot capacity 82% -
zroot altroot - default
zroot health ONLINE -
zroot guid 2466126157835272322 default
zroot version - default
zroot bootfs zroot/ROOT/default local
zroot delegation on default
zroot autoreplace off default
zroot cachefile - default
zroot failmode wait default
zroot listsnapshots off default
zroot autoexpand on local
zroot dedupditto 0 default
zroot dedupratio 1.00x -
zroot free 34.9G -
zroot allocated 165G -
zroot readonly off -
zroot comment - default
zroot expandsize - -
zroot freeing 0 default
zroot fragmentation 82% -
zroot leaked 0 default
zroot bootsize - default
zroot checkpoint - -
zroot multihost off default
zroot load_guid 2806634813821750133 default
zroot feature@async_destroy enabled local
zroot feature@empty_bpobj active local
zroot feature@lz4_compress active local
zroot feature@multi_vdev_crash_dump enabled local
zroot feature@spacemap_histogram active local
zroot feature@enabled_txg active local
zroot feature@hole_birth active local
zroot feature@extensible_dataset enabled local
zroot feature@embedded_data active local
zroot feature@bookmarks enabled local
zroot feature@filesystem_limits enabled local
zroot feature@large_blocks enabled local
zroot feature@large_dnode disabled local
zroot feature@sha512 enabled local
zroot feature@skein enabled local
zroot feature@device_removal disabled local
zroot feature@obsolete_counts disabled local
zroot feature@zpool_checkpoint disabled local
zroot feature@spacemap_v2 disabled local
zroot feature@allocation_classes disabled local
 
sudo zdb -C zroot output? Grasping at straws.

I'll echo rashey's comments above; 8kb is not a desirable setting for a general-purpose pool. The default 128k is much more reasonable (note recordsize is an upper bound; it's not a requirement that all files/io be at least 128k). That said, I wouldn't think that alone would cause what you're seeing. (But I've been wrong before!)

A small recordsize size (matching the DB's transaction size) is generally discussed as a "maybe" option for databases, but there are lots of YMMV caveats to go with that.
 
I can add also that LZ4 compression algorithm is much less effective on 8K blocks compared to default 128K blocks.

Back to the main issue nothing explains the difference between output of zfs list and du commands.
Your ZFS pool was created on older version of operating system than you are currently using.
Maybe at some point for some reason the used space was not freed after snapshot deletion.
Destroy all snapshots wait some time because sometimes ZFS needs more time to reclaim free space and then check free space.

If it will not help then run scrub:
zpool scrub zroot

You can track the progress of operation via status command:
zpool status

When it finish show the output of following commands:
zpool status
zfs list -o space


Please put the output of commands between CODE blocks for better readability.
 
sudo zdb -C zroot output? Grasping at straws.

I'll echo rashey's comments above; 8kb is not a desirable setting for a general-purpose pool. The default 128k is much more reasonable (note recordsize is an upper bound; it's not a requirement that all files/io be at least 128k). That said, I wouldn't think that alone would cause what you're seeing. (But I've been wrong before!)

A small recordsize size (matching the DB's transaction size) is generally discussed as a "maybe" option for databases, but there are lots of YMMV caveats to go with that.
Code:
# zdb -C zroot

MOS Configuration:
        version: 5000
        name: 'zroot'
        state: 0
        txg: 133469338
        pool_guid: 2466126157835272322
        hostid: 2254278924
        hostname: 'mmc1.cks.host'
        com.delphix:has_per_vdev_zaps
        vdev_children: 1
        vdev_tree:
            type: 'root'
            id: 0
            guid: 2466126157835272322
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 12300303235505686309
                path: '/dev/vtbd0p2'
                whole_disk: 1
                metaslab_array: 37
                metaslab_shift: 25
                ashift: 12
                asize: 214742335488
                is_log: 0
                DTL: 846
                create_txg: 4
                com.delphix:vdev_zap_leaf: 35
                com.delphix:vdev_zap_top: 36
        features_for_read:
            com.delphix:hole_birth
            com.delphix:embedded_data
 
I can add also that LZ4 compression algorithm is much less effective on 8K blocks compared to default 128K blocks.

Back to the main issue nothing explains the difference between output of zfs list and du commands.
Your ZFS pool was created on older version of operating system than you are currently using.
Maybe at some point for some reason the used space was not freed after snapshot deletion.
Destroy all snapshots wait some time because sometimes ZFS needs more time to reclaim free space and then check free space.

If it will not help then run scrub:
zpool scrub zroot

You can track the progress of operation via status command:
zpool status

When it finish show the output of following commands:
zpool status
zfs list -o space


Please put the output of commands between CODE blocks for better readability.
Scrub is in progress, I'll post status once it is finished
I'm not very confident about scrub effects since last scrub ran in January and I discovered the issue months ago... wait and see 🤞

Code:
# zfs list -o space
NAME                AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zroot               27.5G   166G         0     88K              0       166G
zroot/ROOT          27.5G   166G         0     88K              0       166G
zroot/ROOT/default  27.5G   166G     10.7G    155G              0          0

Code:
 # du -hsxP /
 35G    /
 
You did not destroy all snapshots, check the list:
zfs list -rt snapshot zroot

First destroy all of them and then follow the steps in order as I said in my previous post.
 
Back
Top