freebsd lsyncd fails

It's log file lsyncd working with /pool/www/yiiazertag/azertag/css/ folder.
Code:
Wed Nov  6 13:27:11 2019 Normal: --- Startup, daemonizing ---
Wed Nov  6 13:27:11 2019 Normal: --- Startup ---
Wed Nov  6 13:27:11 2019 Normal: recursive startup rsync: /pool/www/yiiazertag/azertag/css/ -> root@10.0.0.55:/pool/www/yiiazertag/azertag/css/
sending incremental file list

sent 1,697 bytes  received 15 bytes  3,424.00 bytes/sec
total size is 974,876  speedup is 569.44
Wed Nov  6 13:27:11 2019 Normal: Startup of /pool/www/yiiazertag/azertag/css/ -> root@10.0.0.55:/pool/www/yiiazertag/azertag/css/ finished.
It's ok.
Code:
root@main2:/ # service lsyncd status
lsyncd is running as pid 2044.
It's lsyncd working with /pool/www/yiiazertag/azertag/newsimages/ folder.
Code:
Wed Nov  6 15:01:38 2019 Normal: --- Startup, daemonizing ---
Wed Nov  6 15:01:38 2019 Normal: --- Startup ---
It's not ok. And nothing more. lsyncd is not running.
Code:
root@main2:/ # service lsyncd status
lsyncd is not running.
root@main2:/ # cat /usr/local/etc/lsyncd.conf
settings {
  logfile    = "/var/log/lsyncd.log",
  statusFile = "/var/log/lsyncd.status",
  statusInterval = 1,
  maxProcesses = 1,
  insist = 1,
}
sync{
  default.rsync,
  delay = 0,
  source = "/pool/www/yiiazertag/azertag/newsimages/",
  target = "root@10.0.0.55:/pool/www/yiiazertag/azertag/newsimages/",
  rsync = {
    archive  = true,
    compress = true,
    acls = true,
    verbose = true,
    owner = true,
    group = true,
    perms = true,
    temp_dir="/tmp/",


    -- binary = "/usr/local/bin/rsync",
    -- sparse = true,
    -- update = true,
    -- protect_args = false,
    -- whole_file = false,
    -- links = true,
    _extra = { "--omit-dir-times" },
    rsh = "ssh -p 44400 -o StrictHostKeyChecking=no",
    -- rsync_path = "/usr/local/bin/sudo /usr/local/bin/rsync"
  },
  delete = true
}
I can copy css folder but cannot copy newsimages folder. Lsyncd service stopping. Please help me.
Code:
root@main2:/ # sysctl kern.maxfiles kern.maxfilesperproc kern.openfiles
kern.maxfiles: 500000
kern.maxfilesperproc: 490000
kern.openfiles: 274
 
Same problem here. I theorize that FreeBSD's queue system cannot deal a large amount of handles/hooks, so LsyncD just dies without explanation. The developer of clsync made note of this on their conversion to FreeBSD as well. Even though there is an inotify library/adaption to FreeBSD, it's backend is kqueue... so the problem still exists even with inotify.

At this point I need to consider if I have to switch away from FreeBSD for this project. A critical piece of it is synching some large directories to across a web cluster (800k files, 6500 directories). It's not the size of the files themselves but rather just how many there are.

Would be great if there's a workaround for this but information is scarce. I think it's just a limitation in the style of file watching on FreeBSD... I hope I'm wrong. If anyone knows please speak up!

EDIT: If you run Lsyncd pointed at an empty directory and then mv a large dataset into it, you will get log errors about it. As opposed to trying to point it at a big directory, to which it'll run for a few seconds then just stop with no log out. For example

Code:
Fri Jan 22 19:03:53 2021 Normal: --- OVERFLOW in event queue ---
Fri Jan 22 19:03:53 2021 Normal: --- HUP signal, resetting ---
Fri Jan 22 19:03:53 2021 Normal: waiting for 1 more child processes.
Fri Jan 22 19:03:53 2021 Normal: --- OVERFLOW in event queue ---
Fri Jan 22 19:03:53 2021 Normal: --- HUP signal, resetting ---
Fri Jan 22 19:03:53 2021 Normal: --- OVERFLOW in event queue ---

If you run lsyncd command line against a large dataset, it'll run because I'm pretty sure it just calls up rsync and doesn't bother to "watch" the files.

However, everything works just fine on a "small" dataset of less than say 1024 files.
 
Alright, so I worked on this some more and here's what I found as a solution for those interested.

First, it's not FreeBSD. It is an issue with Lsyncd on freebsd; or at least the binary i got from pkg install.

If you install the inotify tools and do inotifywait yourself, it works great. IE:

Code:
inotifywait -m -r -e modify,create,delete,move,attrib /path/to/your/stuff

For reference I have 800k files, 6500 directories or so, and 60GB+ of data in this web directory. It works gravy with inotify but lsyncd just chokes. I can't spend the time to get into the port and look at it's implementation (maybe it doesn't try to use inotify if available?), but one could write a simple bash script / daemon to do what lsyncd does, more or less. Just feed the "changed" directories to rsync calls. Or just use it to rsync your whole directory if something changes, presumably rsync is pretty efficient at that since that's its purpose in life =)

Hope this helps someone!
 
Back
Top