Syncing 2 NFS storages

L

Luke

Hello,

I'm looking for ideas of a way to sync two storage NFS servers - while the second one is a "hot spare". HAST is not stable enough to use in that case. Rsync might be a way but it takes about one hour to complete. The idea is to rsync only files/directories that changed. I've been trying the auditd - but it wont trigger on server sharing NFS. As for the Kqueue - it creates a handle for each file - it won't work as there's millions of files. Any other ideas to simply syncs files to spare server?
 
I have tried net/csync2 and net/unison. Both maintain a database of what files they have and consult that to see what has changed. They also are capable of handling conflicts to some degree. Unison can be started on one host and handle two way updates while Csync2 must be started on each host. I ended up going with Unison since Csync2 didn't like the unstable VPN connection I was using it over. Unison uses the rsync protocol over SSH and seemed to be more robust for my particular use case.
 
We had a similar situation and our solution was to run multiple rsync commands simultaneously. For example, we have a couple thousand home directories and have use something along the lines of (over simplified example below):

Code: 
/usr/local/bin/rsync -az /home/[0-a]* remote.server.com:/home &
/usr/local/bin/rsync -az /home/[b-c]* remote.server.com:/home &
/usr/local/bin/rsync -az /home/[d-e]* remote.server.com:/home &

And so on...

You can have a simple script like above (probably a good idea to add logging/error handling, etc), or just run each rsync in its own cron job.

Maybe one of these days, we'll get a multi-threaded rsync.
 
Thank you guys for the ideas. I will check them out. More solutions are most welcome.
 
As for the HAST - its not stable enough to use it on production - during tests it caused few kernel panics.

Question - is there any way to log filepatch of any files modified/added/removed/ by NFS server on Freebsd FreeBSD? That kind of log would be great start point to rsync not traversing whole directory tree.
 
You must log in or register to reply here.
Back
Top