make periodic not run while load=value

D

dev.null

Hi,

I'd like to know if it's possible for periodic to avoid running when load is high.
Periodically the server runs some jobs. Sometimes they might complete in 30 mins. Sometimes 3 hrs.
When jobs are running, load is over 3 (300%). I want periodic to look for the load and if it's over that value,
to delay for 1 hour. Is there already a tool for this? if not, how would I script it?

freebsd is 13.2 & 14.0
 
dev.null said:
I'd like to know if it's possible for periodic to avoid running when load is high.
Click to expand...
It is, in essence, a cron(8) job. It runs at a set time, regardless of the load.

If you have a bunch of jails or VMs running on the machine I would suggest staggering the time a bit. So you don't get all your jails (or VMs) running periodic(8) at the same time.

/etc/crontab
Code: 
# Perform daily/weekly/monthly maintenance.
1       3       *       *       *       root    periodic daily
15      4       *       *       6       root    periodic weekly
30      5       1       *       *       root    periodic monthly
 
SirDice said:
It is, in essence, a cron(8) job. It runs at a set time, regardless of the load.

If you have a bunch of jails or VMs running on the machine I would suggest staggering the time a bit. So you don't get all your jails (or VMs) running periodic(8) at the same time.

/etc/crontab
Code: 
# Perform daily/weekly/monthly maintenance.
1       3       *       *       *       root    periodic daily
15      4       *       *       6       root    periodic weekly
30      5       1       *       *       root    periodic monthly
Click to expand...

It's a cron job if it's in a crontab. The daily,weekly,monthly scripts don't have to be in a crontab.

What I want to do is to control execution of these scripts by automatically first looking at 5min load then taking an action if load is lower or looping back and waiting an hour if it isn't. I'm not describing a complete end-point. Just asking if logic in a script based on load was already available.
 
dev.null said:
It's a cron job if it's in a crontab. The daily,weekly,monthly scripts don't have to be in a crontab.
Click to expand...
periodic(8) is run from the crontab, that's what the periodic daily (runs the daily scripts), periodic weekly and periodic monthly do. /usr/sbin/periodic itself is a shell script. It just runs the scripts in /etc/periodic/{daily,weekly,monthly} based on its argument, if they're turned on (/etc/periodic.conf, /etc/defaults/periodic.conf)

Many periodic(8) scripts are quite I/O heavy though. That's why I suggested staggering them if you have several jails or VMs (each with its own periodic(8)).

dev.null said:
Just asking if logic in a script based on load was already available.
Click to expand...
Short answer to that is, no.
 
You could prepend sysctl -n vm.loadavg | awk '{exit $2>1}' && to the crontab lines (right before periodic).

For example:
Code: 
1 3 * * *  root    sysctl -n vm.loadavg | awk '{exit $2>1}' && periodic daily

Not the polarity is a little awkward. When the 1-minute load average is greater than 1 here ($2>1), it returns 1 ("error" exit code) so the command following the && isn't run when the condition ($2>1) is true.
 
SirDice said:
periodic(8) is run from the crontab, that's what the periodic daily (runs the daily scripts), periodic weekly and periodic monthly do. /usr/sbin/periodic itself is a shell script. It just runs the scripts in /etc/periodic/{daily,weekly,monthly} based on its argument, if they're turned on (/etc/periodic.conf, /etc/defaults/periodic.conf)
Click to expand...

My point here is that yes they are usually called from crontab, but they don't *have* to be. One can run 'periodic daily' manually as root. In my context this would mean that the periodic jobs in the default crontab would be commented out; the script in essence being "controlled" by load (and, say, time-after-last-load-trigger-value) and not by time itself. It would not make sense to control by time because the time-of-load-being-over-a-value is very variable. Unfortunately, staggering the times doesn't scale well.
SirDice said:
Short answer to that is, no.
Click to expand...
ah well
 
Eric A. Borisch said:
You could prepend sysctl -n vm.loadavg | awk '{exit $2>1}' && to the crontab lines (right before periodic).

For example:
Code: 
1 3 * * *  root    sysctl -n vm.loadavg | awk '{exit $2>1}' && periodic daily

Not the polarity is a little awkward. When the 1-minute load average is greater than 1 here ($2>1), it returns 1 ("error" exit code) so the command following the && isn't run when the condition ($2>1) is true.
Click to expand...
this is brilliant, thank you. I'd never have thought of running awk in a crontab :)
 
dev.null said:
One can run 'periodic daily' manually as root.
Click to expand...
Sure, that's certainly possible. The awk trick above won't help here either (since that's also just for cron).

That said, you could add it to /usr/sbin/periodic itself though. Like I said, it's a shell script, so you could easily modify it.
 
it helps because I can gauge the time where most of these high-load jobs finish, set the cron line to that time, and accept that on the odd occasion I won't get the daily report. The tradeoff is avoiding system thrashing.
 
dev.null said:
it helps because I can gauge the time where most of these high-load jobs finish, set the cron line to that time, and accept that on the odd occasion I won't get the daily report. The tradeoff is avoiding system thrashing.
Click to expand...

What about just running the periodic tasks under a high nice(1) value when detected loadavg is high?

Or anyway. Then they should have little to no detrimental effect on your high-load tasks, and you'll always get a report, eventually.
 
You could have your periodic cron job submit an at(1) job instead of running periodic itself.

at(1) is the UNX batch job facility. It's not nearly anything like we had on the mainframe nor is it like Sun Grid Engine, but it is UNIX and has been a part of UNIX since forever.

What at(1) does is submit a job into the at(1) queue. Cron spawns atrun(8), the batch job initiator, every five minutes, see /etc/cron.d/at. atrun(8) will only run jobs from the at(1) queue when the load average is less than 1.5 * number of CPUs. And you can change it using the -l keyword, see the atrun(8) man page.

Output from at(1) jobs is sent to you by email. Review the output like you would any other email.

This is certainly a detour of a workaround but, it will accomplish what you want without getting into the weeds programming.

atq(1) will list the jobs in the at(1) queue. While atrm(1) will remove jobs from the queue, as long as they haven't started yet; else you need to use kill(1) after they start

Check out the at(1), atq(1), atrm(1) and atrun(8) man pages.

This is about as UNIX as it gets.
 
cy@ this is what I went with - additionally, it can be the almost the same config on all servers.
The end point I wanted was "don't start periodic when load is over (say) 3.00 (with -l) cos it's busy" and this does just that :D
 
You must log in or register to reply here.
Back
Top