Solved Delay or stagger running of periodic scripts randomly

Hi all,

I have 100 or so FreeBSD VMs on a host and when disk intensive periodic scripts run, they all run at the same time. Think security and locate.

I don't care, for example, when weekly/310.locate runs, so long as it runs weekly. I know I could change /etc/crontab on each and every VM and try to come up with a different time for each, but that would also change when the other weekly jobs run.

I have seen an anticongestion_sleeptime in periodic.conf but to use it requires (I believe) editing individual periodic scripts to call anticongestion(). Something I'd rather not do.

So my question is, is there a simple and centralised way of randomly delaying or staggering the invocation time of individual periodic scripts? Ideally it would go in /etc/rc.conf or /usr/local/etc/rc.conf (or periodic.conf).

Thanks,
Scott
 
Hi everybody,

jumping in, I've encountered the same issue on my (rather small and slow) FreeBSD server running 8 ezjails. The periodic scripts in all jails used to start at the same time and especially the disk-intensive ones used to bring the server down regularly.

Unfortunately, I didn't find a proper solution either. So I'm not really helpful in posting here, besides in emphasizing the fact that you're not alone with this issue. I, too, would love to see a good idea to delay the start of periodic scripts by a configurable or even random delay.

For my part, I now somewhat eased the issue by
  • installing a SSD for ZFS L2ARC. This seems to have helped quite a bit because the expensive find and locate runs in periodic are now much faster and the server is able to serve other processes with disk IO while they are running (my server is low on RAM, so ZFS is not able to cache a lot of ARC in actual RAM).
  • manipulating /etc/crontab in each and every jail by hand so that there's no more overlap in daily, weekly and monthly periodic runs. So exactly the same idea that you also wrote above. And yes, it was tedious and I'd have loved to have a better solution at hand :)
Regards,
Tobias
 
The filenames of the periodic scripts usually begin with a three-digit number, and they're executed in numerical order. So, a simple solution would be to create a new periodic script with number 000 (e. g. /etc/periodic/weekly/000.delay) that just sleeps for a random amount of time. It would be executed first because of the number 000, and all the other scripts would be delayed.

To give you some ideas:

The following script would sleep between 0 and 3600 seconds (one hour) randomly:
Code:
#!/bin/sh
DELAY=$(jot -r 1 0 3600)
echo "Delaying for $DELAY seconds."
sleep $DELAY
If you need the delay to be predictable, but still pseudo-random, you can use the last octet of the machine's primary IP address, for example. The following script takes that octet (range 0–255) and multiplies it with 14, resulting in range of 0–3570 which is roughly an hour, too.
Code:
#!/bin/sh
OCTET=$(/sbin/ifconfig | awk -F'[ \t.]' '$2=="inet" && $3!="127" {print $6}')
DELAY=$(( OCTET * 14 ))
echo "Delaying for $DELAY seconds."
sleep $DELAY
Another way is to make the delay depend on the hostname. The following script takes 3 hexadecimal digits from the MD5 hash of the hostname. This results in a number in the range 0–4095 (0xfff) which is little more than one hour. The number is “pseudo-random”, but it's always the same for the same hostname.
Code:
#!/bin/sh
HEXHASH=$(hostname | /sbin/md5 | head -c3)
DELAY=$(( 0x$HEXHASH ))
echo "Delaying for $DELAY seconds."
sleep $DELAY
 
anticongestion() is also built in to /etc/defaults/periodic.conf. With that, you can add a call to anticongestion in the periodic script. Take a look at /etc/periodic/daily/480.leapfile-ntpd for an example (FreeBSD_12.0-RELEASE).
 
One possibility is to adjust cron(8) rootjitter within 60 seconds by adding
cron_flags="-J 60"
to /etc/rc.conf.

This might help a little to make all the periodic scripts not to start at the same second.
 
  • Thanks
Reactions: grz
Beware of an unintended side effect, which applies to doing this with jails, not for VMs. The file system is very good at caching. So if a lot of identical processes start at the same time, only the first one will read things from disk (like scripts, configuration files, executables), and all other processes will get the cached copy from buffer RAM. This is much more efficient, since on most systems the disk is the bottleneck. In contrast, if they are all completely staggered, then the cached data from the first one will have long been flushed by all the activity of the first one, before the second would have had a chance to reuse the cache. So while running them synchronized may increase latency, it also improves throughput.

Now whether this effect is significant depends on where the caches are. In most systems, the invididual disks have caches that are vanishingly small (the ~10MB cache of a disk drive can handle 1/10th of a second of traffic), while the RAM of the server (typically many GB) can handle 10's of seconds worth of traffic. Since jails share the same file system, I think they can also reuse file system caches, so this should work well for them. On the other hand, VMs all have their own kernel and their own file system and their own buffer caches, so there should be no reuse there. That's unless the VMs are configured to use a file server outside the VMs.
 
One possibility is to adjust cron(8) rootjitter within 60 seconds by adding
cron_flags="-J 60"
to /etc/rc.conf.

This might help a little to make all the periodic scripts not to start at the same second.
The problem with that approach is that it affects all of root's crontab entries, including things like newsyslog(8) and atrun(8). This is usually not desirable.

anticongestion() is also built in to /etc/defaults/periodic.conf. With that, you can add a call to anticongestion in the periodic script. Take a look at /etc/periodic/daily/480.leapfile-ntpd for an example (FreeBSD_12.0-RELEASE).
The OP already mentioned that. The problem here is that you have to modify FreeBSD's original periodic scripts, which will make updating more difficult. It would be nice if there was a way to specify a delay within periodic.conf, or via a command line option of periodic(8). Alas, that hasn't been implemented yet.
 
Oh by the way, I have disabled weekly/310.locate on my machines (weekly_locate_enable="NO"). If you don't really need locate(1), that's an option to consider. There are often other ways to find the things that you're looking for. Commands like type (bourne shell) and where (zsh, csh, bash) can be used to locate commands, and then there's find(1), and some shells have similar functionality built-in (e.g. the special ** glob in zsh that searches subdirectories recursively). For searching ports, there's the make search feature, or you grep the INDEX directly, or you use one of the online search engines (e.g. Porgle). And for files in my home directory, locate(1) wouldn't work anyway because it has mode 700 (rwx------).

TL;DR: I stopped using locate(1) a long time ago.
 
That's a great example of good system administration, based on cost/benefit/risk analysis: On your system, you don't need locate very much (you have other ways of finding things), and you roughly know the cost of running it, so you disabled it. My server at home is another example of the same approach, but with different results: For some workflows, I rely on being able to use locate, and it is quite inconvenient to have to wait a week, so I run locate nightly now. Matter-of-fact, if I had time to measure the impact of locate, and configure it fine-grain enough, I would probably run locate hourly for certain directories or file systems.

The real point is: The system as shipped is a starting point. An experienced user will tailor it to their situation.
 
Matter-of-fact, if I had time to measure the impact of locate, and configure it fine-grain enough, I would probably run locate hourly for certain directories or file systems.
At that point, I would start wondering if that should be a feature of the file system itself, so the “locate database” would always be up to date automatically. Maybe a feature request for zsh …

The real point is: The system as shipped is a starting point. An experienced user will tailor it to their situation.
Very well put.
 
At that point, I would start wondering if that should be a feature of the file system itself, so the “locate database” would always be up to date automatically. Maybe a feature request for zsh …
Traditional file system access files by following the directory structure, then associating attributes (such as permissions, mtime, extended attributes ...) with individual files and directories, then reading file content once the file has been found. They don't have data structures that allow direct access to the live metadata and content of the files in parallel.

However, there are file systems available that allow much more interesting queries than the traditional approach. For example they allow quickly finding all files with certain attributes, with certain names (or regular expressions in names), and in some cases even based on content. In some cases, this can be done with a SQL-like query syntax, like: select all files that are owned by someone in the "elefant" group, are at least 1MiB in size, were created on a weekend, and contain the string "hippopotamus" in the file name. Or: select all files that have the string "hippopotamus" in the file name, but do not have the string "rhinoceros" in the content.

To do this with the more traditional file systems requires using offline tools, such as locate (for file names) and glimpse/agrep (for file content). By their nature, those tools are asynchronous, giving the user a stale (historical) view. That is the price of simplicity of the file system. Implementing the functionality of online search of the file system metadata (for attribute and file name / path queries) requires designing the file system with this goal in mind, and that is not commonly done for free file systems (free here in the sense of "free beer" or "gratis", not "libre"). Implementing online search of file system content has significant performance implications, and for that reason is not likely to be deployed among general-use systems. Look at how many people here on the forum complain every time they think FreeBSD is "bloated", and imagine what would happen if they found out that FreeBSD's file systems are continuously indexing their files.
 
Thanks everyone.

So, a simple solution would be to create a new periodic script with number 000 (e. g. /etc/periodic/weekly/000.delay)

I assume I could also put that script in /usr/local/etc/periodic/weekly/?

My quick answer was to use ansible to run this sed script against each of my VMs:

Code:
sed -I '' -r -e "s/.*(root[[:space:]]periodic daily.*)/`jot -r 1 0 59`"$'\t'"`jot -r 1 0 23`"$'\t'"*"$'\t'"*"$'\t'"*"$'\t'"\1/" -e "s/.*(root[[:space:]]periodic weekly.*)/`jot -r 1 0 59`"$'\t'"`jot -r 1 0 23`"$'\t'"*"$'\t'"*"$'\t'"`jot -r 1 0 6`"$'\t'"\1/" -e "s/.*(root[[:space:]]periodic monthly.*)/`jot -r 1 0 59`"$'\t'"`jot -r 1 0 23`"$'\t'"`jot -r 1 0 28`"$'\t'"*"$'\t'"*"$'\t'"\1/" /etc/crontab
(formatted as code so as not to lose the backticks)

Still, olli@ your solution is neater and won't affect upgrades because of the file changes. I will need to use the deterministic approach as I will delay each machine's scripts by up to one day/week/month.
 
I assume I could also put that script in /usr/local/etc/periodic/weekly/?
No, I'm afraid not, because the periodic scripts in local are run after the ones in the base system, as far as I know. But I'm not 100% sure.
 
No, I'm afraid not, because the periodic scripts in local are run after the ones in the base system, as far as I know. But I'm not 100% sure.

Ahh. I was hoping the directories would be merged before being run :(

I just ran some tests and the inter-directory order is that passed to /usr/sbin/periodic.

And /usr/sbin/periodic builds its dirlist by searching /etc/periodic before /usr/local/etc/periodic:
Code:
for top in /etc/periodic ${local_periodic}; do

I think the workaround (which would involve editing /etc/crontab) would be to change the lines from period daily to periodic /usr/local/etc/periodic/daily daily (and so forth for weekly, monthly).
Then the 000.delay.sh file would need to be in each.

Or create directories below /usr/local/etc/periodic/ with a different name to those in /etc/periodic. At least then the crontab looks neater:

Code:
57      14      *       *       *       root    periodic daily_delay daily
 
Or create directories below /usr/local/etc/periodic/ with a different name to those in /etc/periodic. At least then the crontab looks neater:

Code:
57      14      *       *       *       root    periodic daily_delay daily
Neat! I like that idea.

I think it should be sufficient to create just one directory called delay, and create one script inside. That can be shared by daily, weekly and monthly. I don't think you have to create separate scripts.
Then you would just have:
Code:
1       3       *       *       *       root    periodic delay daily
15      4       *       *       6       root    periodic delay weekly
30      5       1       *       *       root    periodic delay monthly
If you need to differentiate between daily, weekly and monthly (e.g. to have different sleep times), a little awk(1) magic could do that ...
Code:
#!/bin/sh

WHEN=$(ps -alxww | awk '
        {
                pid = $2
                ppid[pid] = $3
                line[pid] = $0
                when[pid] = $NF
        }
        END {
                pid = '$$'
                while (pid > 2) {
                        pid = ppid[pid]
                        if (line[pid] !~ /LOCKED/) {
                                print when[pid]
                                exit
                        }
                }
        }
')

case "$WHEN" in
        daily)          sleep ... ;;
        weekly)         sleep ... ;;
        monthly)        sleep ... ;;
esac
An even more advanced solution would be to make the whole thing configurable via periodic.conf or periodic.conf.local.
 
Neat! I like that idea.

I think it should be sufficient to create just one directory called delay, and create one script inside. That can be shared by daily, weekly and monthly. I don't think you have to create separate scripts.
Then you would just have:
Code:
1       3       *       *       *       root    periodic delay daily
15      4       *       *       6       root    periodic delay weekly
30      5       1       *       *       root    periodic delay monthly
If you need to differentiate between daily, weekly and monthly (e.g. to have different sleep times), a little awk(1) magic could do that ...
Code:
#!/bin/sh

WHEN=$(ps -alxww | awk '
        {
                pid = $2
                ppid[pid] = $3
                line[pid] = $0
                when[pid] = $NF
        }
        END {
                pid = '$$'
                while (pid > 2) {
                        pid = ppid[pid]
                        if (line[pid] !~ /LOCKED/) {
                                print when[pid]
                                exit
                        }
                }
        }
')

case "$WHEN" in
        daily)          sleep ... ;;
        weekly)         sleep ... ;;
        monthly)        sleep ... ;;
esac
An even more advanced solution would be to make the whole thing configurable via periodic.conf or periodic.conf.local.

I was wondering if would be possible to know how the script was called. Nice work. When I get a few minutes I'll decode it :)

Scott
 
Thanks for this thread, folks. I really like the elegance of the periodic delay daily in /etc/crontab. For anyone who wants it, I have made a handy script that creates the script and does the modification to /etc/crontab. It's available as a public gist: a public gist. It's interactive (invoking sudo many times). But if you wanted to automate it, you could.
 
Back
Top