How to monitor big directory hierarchies?

I need to monitor changes(file/dir creatinon/modification/renaming/etc.) of big directory hierarchies that live on ZFS. Do you have any good tips how I should do this? The number of files and directories cannot be estimated, so in theory all limits are very bad idea.

Here are some methods I have found for now. I've also listed their cons:

* My friend found something called FAM, but it can monitor only one directory level at a time, so this would mean lots of FAM instances. Also I think FAM is polling, so that is another con.

* My friend also found something called kqueue(), but I think it can also work at only one directory level at a time.

* Audit seemed to be a very good option. It is mainly for security monitoring, but I could use it if I catch only filesystem events. There are two cons:
1) Few times it got stuck to infinte loop. I wonder if it monitors also it's own writes to it's logfiles. Maybe it's result of that?
2) The information I got using praudit was not in humanreadable form. I couldn't find files or paths from it.
3) Also sometimes it didn't noticed some modifications (or at least it didn't told it to pipe).

* I also tried to modificate source code of ZFS-module and I managed to find the places where modifications are done, but unfortunately all information I got from files/directories was in vnode format, and I do not know how to find out what files/paths point to vnode I have. I'm also afraid of some concurrency stuff when straightly hacking kernel code (I have never before done anything with kernel code, but I have other programming experience).

So basically I'm asking if you have any tips to Audit or if you know how to get more information about vnodes in kernel code. Or if you know any other methods to do this.
 
Do you need a real-time solution, or something that collects information once a day/hour/some_interval? There are some file integrity checkers around, like security/yafic and security/tripwire that will report any modification to a file with timestamps.
 
The more real time, the better. Also the amount of data might reach to even few terabytes, so all kind of polling systems are bad idea. That's why it would be great if the notifying system could somehow work at kernel level where all the modifications could easily be seen immediately.

If I could just convert those vnodes in kernel code or log messages in Audit to full paths... :\
 
ieska328 said:

I think that software is polling, which is not good. The callback stuff that the document talked about, was not found from pnotify.h.

I tried it but ended up having just error messages "directory_scan failed: No such file or directory" both in my software and in make check.
 
Back
Top