ZFS Full Filesystem

Code: 
# zpool status zroot
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:04:37 with 0 errors on Tue Mar  7 14:24:47 2023
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          ada0p3    ONLINE       0     0     0

errors: No known data errors
 
Code: 
du -hAd1 /
512B    /net
2.1M    /etc
512B    /proc
8.7M    /root
1.3M    /bin
 18G    /usr
 14M    /lib
177M    /boot
512B    /media
6.6M    /sbin
512B    /mnt
632K    /tmp
512B    /zroot
244K    /libexec
 13M    /rescue
9.5K    /dev
999K    /var
 18G    /
 
We found much more data in /usr directory but still a lot is missing.

Try a secure TRIM if supported by SSD, if not use the following command without "-d" option:
zpool trim -d zroot

You can track the progress of TRIM operation via status command:
zpool status -t

Check the free space when it finish.
 
Code: 
# zpool trim -d zroot
cannot trim '/dev/ada0p3': trim operations are not supported by this device
# zpool status -t
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:04:37 with 0 errors on Tue Mar  7 14:24:47 2023
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      ada0p3    ONLINE       0     0     0  (100% trimmed, completed at Tue Mar  7 14:24:51 2023)

errors: No known data errors
 
I assume that the trim command without "-d" option also didn't help and there is still the issue with free space.

Use the following command to find unallocated space and zero it out:
zpool initialize zroot

You can track the progress of operation via status command with "-i" option:
zpool status -i zroot

Check the free space when it finish:
zfs list -o space
 
no successs...

Code: 
# zpool initialize zroot
# zpool status -i zroot
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:04:37 with 0 errors on Tue Mar  7 14:24:47 2023
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          ada0p3    ONLINE       0     0     0  (16% initialized, started at Mon Mar 20 07:13:08 2023)

errors: No known data errors

after finishing the initialization

Code: 
zfs list -o space
NAME                AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zroot                  0B  80.9G        0B     96K             0B      80.9G
zroot/ROOT             0B  80.8G        0B     96K             0B      80.8G
zroot/ROOT/default     0B  80.8G        0B   80.8G             0B         0B
zroot/tmp              0B   376K        0B    376K             0B         0B
zroot/usr              0B   384K        0B     96K             0B       288K
zroot/usr/home         0B    96K        0B     96K             0B         0B
zroot/usr/ports        0B    96K        0B     96K             0B         0B
zroot/usr/src          0B    96K        0B     96K             0B         0B
zroot/var              0B  71.3M        0B     96K             0B      71.2M
zroot/var/audit        0B    96K        0B     96K             0B         0B
zroot/var/crash        0B    96K        0B     96K             0B         0B
zroot/var/log          0B  70.8M        0B   70.8M             0B         0B
zroot/var/mail         0B   172K        0B    172K             0B         0B
zroot/var/tmp          0B   112K        0B    112K             0B         0B
 
Code: 
# zdb -L
cannot open '/etc/zfs/zpool.cache': No such file or directory
# cd /etc/zfs/
/etc/zfs # ls -l
total 1
drwxr-xr-x  2 root  wheel  2 May 12  2022 compatibility.d
-rw-r--r--  1 root  wheel  0 Feb 13 07:23 exports
 
You can use zdb and zpool utilities to roll back the pool to state before the system crash if possible.

But I strongly do not recommend this because it can be very dangerous to the pool and data stored there.

Let me know if you would like to try anyway.
 
It can be hardware or software failure, especially when 13.2 is in BETA/RC stage at the moment.
You need to debug kernel crash dump to figure out what really happened.

As you noticed before, the /var directory is empty.
Since zroot/var dataset is not writable because of lack of space, a new /var directory is stored in RAM.

Change mountpoint of zroot/var/crash to /mnt/crash:
zfs set mountpoint=/mnt/crash zroot/var/crash

Then check if there is something in /mnt/crash:
ls -lh /mnt/crash
 
I don't have anything positive to add to the diagnostic discussion, but I have seen file system damage on VMware hosted systems when some enterprise class SAN switches were running buggy software. Every data path was redundant. Every virtual "disk" on every VM was mirror'd. But every SAN switch was broken. Essentially write commands to storage were verified to the host, but not actually completed. The file system damage was horrendous, and widespread. We spent months years recovering.

All my instincts suggest that you should rescue anything of value on that system before you do anything else!
 
rashey said:
It can be hardware or software failure, especially when 13.2 is in BETA/RC stage at the moment.
You need to debug kernel crash dump to figure out what really happened.

As you noticed before, the /var directory is empty.
Since zroot/var dataset is not writable because of lack of space, a new /var directory is stored in RAM.

Change mountpoint of zroot/var/crash to /mnt/crash:
zfs set mountpoint=/mnt/crash zroot/var/crash

Then check if there is something in /mnt/crash:
ls -lh /mnt/crash
Click to expand...
Thanks for your inputs. It is a 13.1 Release so no BETA/RC Version in this point.
I will check the crash Folder tomorrow
 
Extending the suggestion of rashey I'd try mounting all datasets to a separate mountpoint like /mnt/dataset1, /mnt/dataset2, etc., and then investigate each of those directories for suspicious space sizes—perhaps hidden overlays? Perhaps the easiest/cleanest way would be to boot from a separate media distinct from the current zfs on root pool. This is likely to be a long shot but, considering all the options taken before, not much else remains that I can think of. Good luck.

Edit: if you have an extra (external) storage (even a non-redundant single disk) at hand, you could consider using zfs-send(8) to send datasets over, one at a time, and compare the space usage between the orginal and the copy.
 
rashey said:
I would like to see the output of following command as well:
zpool get all zroot
Click to expand...
sorry for my late replay:
Code: 
# zpool get all zroot
NAME   PROPERTY                       VALUE                          SOURCE
zroot  size                           35.5G                          -
zroot  capacity                       96%                            -
zroot  altroot                        -                              default
zroot  health                         ONLINE                         -
zroot  guid                           16380778464114236343           -
zroot  version                        -                              default
zroot  bootfs                         zroot/ROOT/default             local
zroot  delegation                     on                             default
zroot  autoreplace                    off                            default
zroot  cachefile                      -                              default
zroot  failmode                       wait                           default
zroot  listsnapshots                  off                            default
zroot  autoexpand                     off                            default
zroot  dedupratio                     1.00x                          -
zroot  free                           1.09G                          -
zroot  allocated                      34.4G                          -
zroot  readonly                       off                            -
zroot  ashift                         0                              default
zroot  comment                        -                              default
zroot  expandsize                     -                              -
zroot  freeing                        0                              -
zroot  fragmentation                  89%                            -
zroot  leaked                         0                              -
zroot  multihost                      off                            default
zroot  checkpoint                     -                              -
zroot  load_guid                      2895214603155864083            -
zroot  autotrim                       off                            default
zroot  compatibility                  off                            default
zroot  feature@async_destroy          enabled                        local
zroot  feature@empty_bpobj            active                         local
zroot  feature@lz4_compress           active                         local
zroot  feature@multi_vdev_crash_dump  enabled                        local
zroot  feature@spacemap_histogram     active                         local
zroot  feature@enabled_txg            active                         local
zroot  feature@hole_birth             active                         local
zroot  feature@extensible_dataset     active                         local
zroot  feature@embedded_data          active                         local
zroot  feature@bookmarks              enabled                        local
zroot  feature@filesystem_limits      enabled                        local
zroot  feature@large_blocks           enabled                        local
zroot  feature@large_dnode            enabled                        local
zroot  feature@sha512                 enabled                        local
zroot  feature@skein                  enabled                        local
zroot  feature@userobj_accounting     active                         local
zroot  feature@encryption             enabled                        local
zroot  feature@project_quota          active                         local
zroot  feature@device_removal         enabled                        local
zroot  feature@obsolete_counts        enabled                        local
zroot  feature@zpool_checkpoint       enabled                        local
zroot  feature@spacemap_v2            active                         local
zroot  feature@allocation_classes     enabled                        local
zroot  feature@resilver_defer         enabled                        local
zroot  feature@bookmark_v2            enabled                        local
zroot  feature@redaction_bookmarks    enabled                        local
zroot  feature@redacted_datasets      enabled                        local
zroot  feature@bookmark_written       enabled                        local
zroot  feature@log_spacemap           active                         local
zroot  feature@livelist               enabled                        local
zroot  feature@device_rebuild         enabled                        local
zroot  feature@zstd_compress          enabled                        local
zroot  feature@draid                  enabled                        local
 
Any consideration that the problem is an obnoxious number of tiny or zero-length files?

I would use the top rated comment on here to find which directory has a bunch of junk in it

unix.stackexchange.com

How do I count all the files recursively through directories

I want to see how many files are in subdirectories to find out where all the inode usage is on the system. Kind of like I would do this for space usage du -sh /* which will give me the space used...
unix.stackexchange.com unix.stackexchange.com
 
You must log in or register to reply here.
Back
Top