Solved ZFS space leak. 3.4T disappeared.

[robot468]

Hi.
OS 13.1, Raidz1 on 4 disks for video surveillance data.
~8tb of mp4 files. No snapshots. Recently i'v rewrited all of this data with something like this:

ffmpeg -i old.file.mp4 new.file.mp4
mv new.file.mp4 old.file.mp4

After that ~2T of space on this pool disappeared.
How i can fix this?

zfs list -r stor
NAME        USED  AVAIL     REFER  MOUNTPOINT
stor       7.39T   415G     12.0G  /stor
stor/surv  7.38T   415G     7.38T  /stor/surv

zfs list -r -t snapshot stor
no datasets available

du -hs /stor/surv/
4.0T    /stor/surv/

 

[yuripv79]

Are the original files still opened by applications?

 

[robot468]

Are the original files still opened by applications?

No, checked by fstat and lsof. surveillance software keeps open only the latest files that are being recorded right now

 

[_martin]

What does zfs list -o space say ?

 

[robot468]

_martin yuripv79
Solved.
It was somehow used by another jail, to which this fs was mounted by nullfs.
unmount of nullfs freed all space.

Does multiple nullfs mounts affect the ability of zfs to free space?

 

[_martin]

I'm not aware of it but I can't say for sure. For me it would make sense the way yuripv79 mentioned above - you removed files that other application (possible using nullfs) had it still open.

 

[robot468]

I'm not aware of it but I can't say for sure. For me it would make sense the way yuripv79 mentioned above - you removed files that other application (possible using nullfs) had it still open.

i checked it with lsof and fstat.
Now I did an experiment.
I have two jails with nullfs mounts of /stor/surv fs.
I created 100G file in first jail. fstat tells nobody used it.
check another jail - fstat is empte. on host machine fstat is empty too.
deleted file in first jail.
no space freed.
restarted all applications in all jails - no space freed.
stopped second jail with unmounting nullfs - space freed.

It seems two nullfs mounts somehow get in the way of freeing up space.

 

[robot468]

ZFS - nullfs prevents zfs from freeing space

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270546 how to reproduce: 1) mount zfs filesystem in 2 separate places with nullfs, ex /mount1 and /mount2 2) create file in /mount1 3) ls /mount2 4) rm /mount1/file 5) space not freed by zfs. only umount of /mount2 frees space Has anyone...

forums.freebsd.org