Solved a puzzle with missing file and cron

aragats

Daemon

Reaction score: 653
Messages: 1,557

An hour I started getting emails with errors reporting by a cron() task:
Code:
/usr/local/www/misc/eldora/get_image.sh: convert: not found
I logged in my VPS and found that indeed, the file convert doesn't exist. It's supposed to belong to graphics/ImageMagick7 or to one of its flavors. So, I checked pkg info and found it's not installed. Okay, it could happen during updates/upgrades whatever. But why I have never received those error messages before? I didn't touch anything in this VPS for long time. I checked the security reports – everything looks good.

I'm using Gmail to relay emails, so checked the "Sent" folder and found just those few messages from today.

What could happen? What am I missing? How to track this issue down?
FreeBSD 12.0-RELEASE-p10

Thanks for ideas!
 

ljboiler

Well-Known Member

Reaction score: 160
Messages: 440

pkg should log install/update/delete actions to /var/log/messages (or the compressed archived versions messages.X.bz). I'd start there...
Code:
grep 'pkg\['
 
OP
aragats

aragats

Daemon

Reaction score: 653
Messages: 1,557

pkg should log install/update/delete actions to /var/log/messages
That makes sense, thanks!
Code:
% grep -i magic /var/log/messages
Sep 12 21:02:35 eu pkg[53351]: ImageMagick-6.9.9.28_2,1 deinstalled
However, it doesn't explain why no emails were sent since Sept 12 until today (the cron task runs every 5 minutes).
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 10,914
Messages: 36,820

Check /var/log/cron to see if it actually ran or not. Then check /var/log/maillog, maybe the mail got stuck in the queue, I've had this happen a couple times after an upgrade (it messed up my mail configuration).
 
OP
aragats

aragats

Daemon

Reaction score: 653
Messages: 1,557

Check /var/log/cron to see if it actually ran or not. Then check /var/log/maillog, maybe the mail got stuck in the queue
Just checked: cron did run, the mail queue is empty, /var/log/maillog* files don't have any references besides those emails which suddenly appeared yesterday after 12 days of the non-existing file...
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 10,914
Messages: 36,820

/var/log/maillog* files don't have any references besides those emails which suddenly appeared yesterday after 12 days of the non-existing file...
Was sendmail running during that time? If there's no MTA there's nowhere for the cron output to go. You should, at the very least, see the daily periodic emails, unless you disabled those.
 
OP
aragats

aragats

Daemon

Reaction score: 653
Messages: 1,557

Yes, I receive the daily periodic emails every day, also there are other cron tasks which send me notifications/reminders etc on purpose (not stdout/stderr).
The MTA is mail/ssmtp.
 

ralphbsz

Son of Beastie

Reaction score: 2,014
Messages: 2,976

Suggestion: Put a known broken command into cron. For example something where the path is wrong (execute /usr/local/bin/foo, and make sure it doesn't exist). Also put something that "works" but fail, for example put this:
Code:
#/bin/sh
echo Hallo and goodbye > /dev/stderr
exit 1
and run that from your cron. Make sure you get e-mails each time.
 
OP
aragats

aragats

Daemon

Reaction score: 653
Messages: 1,557

Put a known broken command into cron ... Also put something that "works" but fail
I checked both and they worked, then realized it must be a problem in my script. And indeed – I found a condition which has never been triggered before!

That script was pulling screenshots from a web camera at a ski area, and was very useful in winter season. Then they stopped that webcam on one day in April. Now (yesterday) it started working again, but with different parameters which triggered that condition!

No magic – problem solved!
 

ralphbsz

Son of Beastie

Reaction score: 2,014
Messages: 2,976

Software broken by a snowflake. Seems like a common problem. Right up there with software that breaks because a butterfly flaps it wings.
 
OP
aragats

aragats

Daemon

Reaction score: 653
Messages: 1,557

Software broken by a snowflake.
Software is not broken. The issue itself appeared when a package was removed during update process – so it's an operator's omission...
I don't think it's a common practice to check for all executables' existence in shell scripts.
 

ralphbsz

Son of Beastie

Reaction score: 2,014
Messages: 2,976

No, I agree, and for the shells scripts I write at home I don't do that either.

For the stuff I write at work, we have tests. Both unit tests (where an automated test runs, with fake input data, and we make sure the output is 100% identical to what is expected, and we require that at least 90% of all lines are executed, both in scripts and in programs). And system tests, where human testers set up and run software and systems. They first try to run the system according to the manual, make sure the good case works correctly. Then they try to break it, for example with insane inputs.
 
Top