ZFS ZFS out of space

A

amnixed

Hello,

I have a machine which ran out of disk space. I could delete a few large files and I have also deleted a few of the oldest snapshots. However, I still have no space:

Code: 
# zfs list -t filesystem -o space
NAME                         AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zroot                            0  1.76T         0     88K              0      1.76T
zroot/ROOT                       0  15.9G         0     88K              0      15.9G
zroot/ROOT/default               0  15.9G     7.08G   8.78G              0          0
zroot/samba                      0   706G         0     88K              0       706G
zroot/samba/mdata                0   706G         0     88K              0       706G
zroot/samba/mdata/UPS            0  2.37G      235M   2.14G              0          0
zroot/samba/mdata/edgefiles      0   670G      123G    547G              0          0
zroot/samba/mdata/musers         0    88K         0     88K              0          0
zroot/samba/mdata/store          0  34.3G         0   34.3G              0          0
zroot/tmp                        0  37.9M     3.63M   34.3M              0          0
zroot/usr                        0   131G         0     88K              0       131G
zroot/usr/home                   0   121G     5.87M    121G              0          0
zroot/usr/ports                  0  7.71G     1.44G   6.27G              0          0
zroot/usr/src                    0  1.70G      214M   1.50G              0          0
zroot/var                        0   227M         0     88K              0       227M
zroot/var/audit                  0    88K         0     88K              0          0
zroot/var/crash                  0    88K         0     88K              0          0
zroot/var/log                    0   224M      173M   50.9M              0          0
zroot/var/mail                   0  1.91M     1.80M    112K              0          0
zroot/var/tmp                    0   440K      352K     88K              0          0
zroot/vbox                       0   944G      650G    294G              0          0

Would you please advise?
 
What kind of advice are you expecting? Do you need more space? If no then there's nothing to do. If yes then either remove stuff that can be removed or increase your pool size. The later seems more likely given that your root partition is out of space. This basically means that the system won't be able to store log files and so on.
Keep in mind that you might also free up some space by removing no-longer-needed snapshots. Unless I'm mistaken your zroot/vbox dataset is only 294G in actual size but has 650G of snapshots hanging around.

There are various notes on the fact that one wouldn't usually want to use more than 80% of a ZFS pool's capacity.
If I recall correctly ZFS switches from "performance mode" to "omfg space is scarce" mode above 80% (which may or may not be correct - do your research).

For the future, you might consider setting a quota on your datasets such as zroot/vbox and zroot/samba. This is a simple setting which allows to limit the maximum space a dataset is allowed to consume (setting is per-dataset). This is helpful in situations like yours as you can ensure that you always have a few gigabytes of space left for the actual OS.
 
Well, I'd like to regain some usable disk space.

I was able to remove some large files with rm [filename] and snapshots with zfs destroy [snapshot], but these apparently have not recovered any space.

I will surely pay attention to quotas - but need to recover space first.
 
make sure the files you deleted were not open. otherwise the space won't be reclaimed until the program that opened them terminates
 
  • Like
Reactions: mer
Snapshots will use space.
It may take a little time for the recovery to work through everything so the upper layers see free.
Samba edgefiles has quite a bit in snapshots; see if you can destroy some there.
vbox: see if there are vms you can get rid of.
samba edge snapshots 123G, vbox snapshots 650G That's 773G of snapshots on a 1.76T device (basically everything after the decimal point). There should be some in those you can reclaim.
 
jbodenmann said:
There are various notes on the fact that one wouldn't usually want to use more than 80% of a ZFS pool's capacity.
If I recall correctly ZFS switches from "performance mode" to "omfg space is scarce" mode above 80% (which may or may not be correct - do your research).
Click to expand...
To the best of my knowledge this is correct. It's the ZFS equivalent of UFS reserving 10% of space.
 
amnixed said:
I was able to remove some large files with rm [filename] and snapshots with zfs destroy [snapshot], but these apparently have not recovered any space.
Click to expand...
Regarding log files. Whenever possible you'd usually want to use a log management system like newsyslog(5). This allows to automatically/periodically rotate log files and more.

Regarding deleting snapshots not increasing free space: Keep in mind that they are deltas (i.e. a record of what changed). To actually free up space you have to delete snapshots "from the bottom up".
You can check how much "extra" space a snapshot occupies by zfs list -t all -r.
 
Well, destroyed 12 of 29 zroot/samba/mdata/edgefiles/* snapshots, but still get:
Code: 
# zfs list -t filesystem -o space
NAME                         AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zroot                            0  1.76T         0     88K              0      1.76T
zroot/ROOT                       0  15.9G         0     88K              0      15.9G
zroot/ROOT/default               0  15.9G     7.08G   8.78G              0          0
zroot/samba                      0   706G         0     88K              0       706G
zroot/samba/mdata                0   706G         0     88K              0       706G
zroot/samba/mdata/UPS            0  2.37G      235M   2.14G              0          0
zroot/samba/mdata/edgefiles      0   670G      123G    547G              0          0
zroot/samba/mdata/musers         0    88K         0     88K              0          0
zroot/samba/mdata/store          0  34.3G         0   34.3G              0          0
zroot/tmp                        0  37.9M     3.63M   34.3M              0          0
zroot/usr                        0   131G         0     88K              0       131G
zroot/usr/home                   0   121G     5.87M    121G              0          0
zroot/usr/ports                  0  7.71G     1.44G   6.27G              0          0
zroot/usr/src                    0  1.70G      214M   1.50G              0          0
zroot/var                        0   227M         0     88K              0       227M
zroot/var/audit                  0    88K         0     88K              0          0
zroot/var/crash                  0    88K         0     88K              0          0
zroot/var/log                    0   224M      173M   50.9M              0          0
zroot/var/mail                   0  1.91M     1.80M    112K              0          0
zroot/var/tmp                    0   440K      352K     88K              0          0
zroot/vbox                       0   944G      650G    294G              0          0
 
Snapshots are deltas. When you delete a snapshot, the space used by it will likely migrate to something else. The space will not be reclaimed until there is nothing referencing it. That is the one fact of snapshots that is overlooked or misunderstood.

Yes that means you may have to delete all the snapshots.

Your vbox overall is a huge chunk, but about 2/3rds is in snapshots. I would start there and simply zfs destroy all the vbox snapshots.

ETA:
Quick rough math:
samba + vbox is 706G + 944G = 1.65T or about 93% of 1.76T. The rest could easily add up to 100%.
 
amnixed Glad to hear that. ZFS snapshots are addictive: they make it easy to back up and rollback but they breed like rabbits. The key to remember is ZFS blocks are basically ref-counted and only freed when nothing is referencing them.
 
I take every 15minutes an incremental snapshot of my home directory and it literally takes no disk-space.
But a virtual machine, network share , or database, can be a very different kind of beast.
 
Alain De Vos said:
I take every 15minutes an incremental snapshot of my home directory and it literally takes no disk-space.
But a virtual machine, network share , or database, can be a very different kind of beast.
Click to expand...
ZFS is Copy On Write so when you take a snapshot, you effectively increment a use count on the blocks (this is not exactly what happens but it's close). If you do something that changes a block, then the block before the change is copied to the snapshot (COW) then changed. The snapshot has the old block, the dataset has the changed one.
In your home directory, how much is actually changing in 15 minutes? Maybe some days a lot, maybe most days not so much.

But yep VMs, network shares, databases are likely to change more often so snapshots take more space.
 
jbodenmann said:
For the future, you might consider setting a quota on your datasets such as zroot/vbox and zroot/samba. This is a simple setting which allows to limit the maximum space a dataset is allowed to consume (setting is per-dataset). This is helpful in situations like yours as you can ensure that you always have a few gigabytes of space left for the actual OS.
Click to expand...

Another approach (from a MOTD) is to create a dedicated reserved dataset using refreservation:

Don't let your zpool fill up completely by creating a dataset with
reservation.

# zfs create -o refreservation=<5% of total pool space> <poolname>/reserved

You can always shrink the reserve if you need the space, but your pool will
always have space left this way.

-- Benedict Reuschling
Click to expand...
 
You must log in or register to reply here.
Back
Top