Solved Cron does not start at boot

VucanRidr

New Member

Reaction score: 2
Messages: 9

I have two desktop systems, both running fresh-ish installs (installation was about February-ish?) of 12.0-RELEASE. For some reason, cron does not start on boot. I can start it manually after boot, but just not on startup. My /etc/defaults/rc.conf has:

Code:
cron_enable="YES"       # Run the periodic job daemon.
cron_program="/usr/sbin/cron"   # Which cron executable to run (if enabled).
cron_dst="YES"          # Handle DST transitions intelligently (YES/NO)
cron_flags=""           # Which options to pass to the cron daemon.
entropy_dir="/var/db/entropy" # Set to NO to disable caching entropy via cron.
Has anybody seen this behavior and/or does anyone know what knob needs to be turned to get it to start? I have finally gotten zfs snapshots set up through cron, and it would lend a bit of peace of mind to not have to worry about whether it is running or not.

Thanks,
--vr
 

tingo

Daemon

Reaction score: 432
Messages: 2,110

And you don't have anything in /etc/rc.conf that would affect cron? (Probably not, but it is better to ask)
What does # service cron status show after boot (before manually starting cron)?
 

ralphbsz

Daemon

Reaction score: 1,270
Messages: 2,036

A: Look in your /etc/rc.conf (and if you have it, /usr/local/etc/rc.conf). Make sure you don't override any of the default settings for cron, and turn it off by mistake.

B: Did you edit /etc/default/rc.conf? You are not supposed to; instead you use /etc/rc.conf to override the default settings. If you did edit default/rc.conf, then look up above the section that configures cron, for a mistake that might disable things. For example, you might have put in one extra double quote above it, and that causes a syntax error and the whole rest of the file is not processed.

C: Perhaps you broke something else that causes cron to start, and immediately die. For example, you might have changed permissions on a log file or configuration file such that it can't run. To debug that, try the following. First, temporarily replace /usr/sbin/cron (in place!) with a tiny script, which does nothing other than to confirm that it got started. It might just have the line "echo Starting cron at `date` >> /tmp/cron.start.log" or similar. Second, start cron by hand, with a sufficient number of debug flags to see what it does right after it starts, and why it exits.
 

k.jacker

Aspiring Daemon

Reaction score: 434
Messages: 738

The fact that the problem exists on two machines, makes it look home made.

Do you pass any (faulty) arguments to cron from /etc/rc.conf ?
Is there something like the following in /var/log/messages.
Code:
% grep cron /var/log/messages                                                                                                                                                                                         
Jun 10 17:37:23 yogi root[990]: /etc/rc: WARNING: failed to start cron
 
OP
OP
V

VucanRidr

New Member

Reaction score: 2
Messages: 9

And you don't have anything in /etc/rc.conf that would affect cron? (Probably not, but it is better to ask)
What does # service cron status show after boot (before manually starting cron)?
Hi Tingo,

I do not have anything in rc.conf that would affect cron. Initially, neither of the machines had anything about cron in rc.conf. I took the liberty of adding
Code:
cron_enable="YES"
to rc.conf on the laptop to see if it made a difference. It didn't.

It shows

Code:
$ sudo service cron status
cron is not running.
until I start it manually.

Hey Ralphbsz,

A: Look in your /etc/rc.conf (and if you have it, /usr/local/etc/rc.conf). Make sure you don't override any of the default settings for cron, and turn it off by mistake.
No. Nothing in /etc/rc.conf (and no /usr/local/etc/rc.conf).

B: Did you edit /etc/default/rc.conf? You are not supposed to; instead you use /etc/rc.conf to override the default settings. If you did edit default/rc.conf, then look up above the section that configures cron, for a mistake that might disable things. For example, you might have put in one extra double quote above it, and that causes a syntax error and the whole rest of the file is not processed.
Did not touch/edit /etc/defaults/rc.conf on either machine.

C: Perhaps you broke something else that causes cron to start, and immediately die. For example, you might have changed permissions on a log file or configuration file such that it can't run. To debug that, try the following. First, temporarily replace /usr/sbin/cron (in place!) with a tiny script, which does nothing other than to confirm that it got started. It might just have the line "echo Starting cron at `date` >> /tmp/cron.start.log" or similar. Second, start cron by hand, with a sufficient number of debug flags to see what it does right after it starts, and why it exits.
I didn't. This is a base install of 12.0-RELEASE. I did not even notice that it wasn't starting on boot until I started setting up snapshots in zfstools. I will test with the small script and see what happens.

k.jacker.

The fact that the problem exists on two machines, makes it look home made.
No, because as I said above, I didn't change anything cron-related.

Is there something like the following in /var/log/messages.
Code:
% grep cron /var/log/messages                                                                                                                                                                                       
Jun 10 17:37:23 yogi root[990]: /etc/rc: WARNING: failed to start cron
Just rebooted the laptop, and there was nothing in /var/log/messages.

Thanks all,
--vr
 

putney

New Member


Messages: 3

There was a similar post "/etc/rc.d/cron does not start at boot" by scrappywan on Mar 2nd 2018. That turned out to most likely be caused by x11/pcdm and the permissions on /var/run/cron.pid.

FWIW, I'm running 12.0-RELEASE-p5 and I'm seeing behaviour similar to yours. I don't have pcdm installed but I do have a bunch of x11 packages (just trying stuff out).

You've inspired me to try setting rc_debug="YES" in /etc/rc.conf to try to get to the bottom of that.
 
OP
OP
V

VucanRidr

New Member

Reaction score: 2
Messages: 9

There was a similar post "/etc/rc.d/cron does not start at boot" by scrappywan on Mar 2nd 2018. That turned out to most likely be caused by x11/pcdm and the permissions on /var/run/cron.pid.
Wow. Not sure how I missed that. Actually, I just did a search, and the search on the forum does not include "does, not, or at". I just found it by searching for x11/pcdm. And from the look of it, he found the problem for the same reason I did...

FWIW, I'm running 12.0-RELEASE-p5 and I'm seeing behaviour similar to yours. I don't have pcdm installed but I do have a bunch of x11 packages (just trying stuff out).
I have pcdm installed...And I am only on 12.0-RELEASE-p3. I'll be upgrading tonight. FWIW. And I will follow the instructions on the other thread, though if it happens to you without pcdm, it's probably not the issue.

You've inspired me to try setting rc_debug="YES" in /etc/rc.conf to try to get to the bottom of that.
I will be looking into this as well.

Thanks!
 

ShelLuser

Son of Beastie

Reaction score: 1,752
Messages: 3,569

I find it hard to believe that cron wouldn't start without giving any reason(s) for it in the system logfile.

Still, this leads up to: what things did you change on your machine? Because this is most certainly not out of the box behavior, so you changed something which caused all this.

Did you, perhaps, change the root shell to something obscure? Or maybe you messed around in the base system? Something I've seen Linux users do: dump bash into /bin and then being surprised that some things stopped working when they replaced one of the default shells.

I could also imagine a situation in which only the base system is mounted and as a result the system doesn't have access to the root shell just yet. Not a common situation but I've experienced something similar to this before.

Either way... /var/log/cron is your friend here. Unless of course syslogd also isn't running; have you checked with service syslogd status? It would make sense if this wasn't limited to just cron. Or.... you changed stuff yourself in /etc/syslog.conf which could obscure some log messages.

So yeah, dozens of possible options here. One thing seems certain to me: this was self inflicted somehow.
 
OP
OP
V

VucanRidr

New Member

Reaction score: 2
Messages: 9

Thanks for your post, ShelLuser. Full disclosure, I did, in fact, come over from Linux early in the dumpster fire that calls itself systemd, approximately 2014. Prior to that, I was a Linux guy for almost 20 years. However, I have also been a professional sysadmin for 25 years. I have worked with not only Linux, but other Unixes, including SunOS, Solaris, DEC Unix, HP-UX, AIX, BSDi, PC-BSD/TrueOS, and now FreeBSD.

I did not intentionally break my environment, which makes me think that it is something subtle.

I find it hard to believe that cron wouldn't start without giving any reason(s) for it in the system logfile.
I wouldn't either...And yet, I rebooted my desktop machine at 16:57 EDT. The tail of my /var/log/messages:

Code:
Jun 11 16:55:00 defiant /usr/sbin/cron[48959]: (operator) CMD (/usr/libexec/save-entropy)
Jun 11 16:55:00 defiant /usr/sbin/cron[48960]: (root) CMD (/usr/libexec/atrun)
...And nothing in /var/log/messages:

Code:
root@defiant:/home/storm # grep -i cron /var/log/messages
root@defiant:/home/storm #
I see the same behavior on my laptop.

Still, this leads up to: what things did you change on your machine? Because this is most certainly not out of the box behavior, so you changed something which caused all this.
I did not change anything system-wise. I did the FreeBSD install, then I installed a selection of packages. The only "changes" I made were in my own user environment.

Did you, perhaps, change the root shell to something obscure? Or maybe you messed around in the base system? Something I've seen Linux users do: dump bash into /bin and then being surprised that some things stopped working when they replaced one of the default shells.
Absolutely not! I learned long ago that you never, ever mess with the root environment. I also left bash where it belongs in /usr/local/bin:

Code:
root@defiant:/home/storm # echo $SHELL
/bin/csh
root@defiant:/home/storm # which bash
/usr/local/bin/bash
I could also imagine a situation in which only the base system is mounted and as a result the system doesn't have access to the root shell just yet. Not a common situation but I've experienced something similar to this before.
On my desktop, I have two zpools. The main one, NX74205, is on a mirrored pair of SSDs, and has the bulk of the system on it. The second pool, NCC1764, is on a mirrored pair of 1TB spinning drives, and has stuff on it that is either non-standard, or would probably wear out the SSDs faster:

Code:
NCC1764/usr/git                             2.31G   821G  2.31G  /usr/git
NCC1764/usr/ports                           1.00G   821G   206M  /usr/ports
NCC1764/usr/src                               96K   821G    96K  /usr/src
NCC1764/usr/virtualbox                      74.2G   821G  43.7G  /usr/virtualbox
The laptop, OTOH, has a single pool across two SSDs.

Either way... /var/log/cron is your friend here. Unless of course syslogd also isn't running; have you checked with service syslogd status? It would make sense if this wasn't limited to just cron. Or.... you changed stuff yourself in /etc/syslog.conf which could obscure some log messages.

So yeah, dozens of possible options here. One thing seems certain to me: this was self inflicted somehow.
syslogd is running, while cron is not:

Code:
root@defiant:/home/storm # service syslogd status
syslogd is running as pid 2246.
root@defiant:/home/storm # service cron status
cron is not running.
And I have not changed /etc/syslog.conf on either system, nor have I added or subtracted anything from /etc/syslog.d or /usr/local/etc/syslog.d

I'm trying to weed through the possibilities, but this one was beyond me. The reason I posted is that I checked the basic things, but I am hoping someone can help me find the more obscure things that it might be. I am still a vanilla FreeBSD n00b, so hopefully the experts can help.

Thanks,
--vr
 
OP
OP
V

VucanRidr

New Member

Reaction score: 2
Messages: 9

A: Look in your /etc/rc.conf (and if you have it, /usr/local/etc/rc.conf). Make sure you don't override any of the default settings for cron, and turn it off by mistake.
No /usr/local/etc/rc.conf. And nothing in /etc/rc.conf. (I had
Code:
cron_enable="YES"
, but took it out since it was redundant.

B: Did you edit /etc/default/rc.conf? You are not supposed to; instead you use /etc/rc.conf to override the default settings. If you did edit default/rc.conf, then look up above the section that configures cron, for a mistake that might disable things. For example, you might have put in one extra double quote above it, and that causes a syntax error and the whole rest of the file is not processed.
Haven't touched /etc/defaults/rc.conf (on either system). Here are the cron-related entries from that file:
Code:
cron_enable="YES"           # Run the periodic job daemon.
cron_program="/usr/sbin/cron"   # Which cron executable to run (if enabled). cron_dst="YES"        # Handle DST transitions intelligently (YES/NO)
cron_flags=""           # Which options to pass to the cron daemon.
C: Perhaps you broke something else that causes cron to start, and immediately die. For example, you might have changed permissions on a log file or configuration file such that it can't run. To debug that, try the following. First, temporarily replace /usr/sbin/cron (in place!) with a tiny script, which does nothing other than to confirm that it got started. It might just have the line "echo Starting cron at `date` >> /tmp/cron.start.log" or similar. Second, start cron by hand, with a sufficient number of debug flags to see what it does right after it starts, and why it exits.
I took your advice and moved /usr/sbin/cron to /usr/sbin/cron.real. Rebooted the laptop and there was no /tmp/cron.start.log file. So apparently, whatever is supposed to be calling /usr/sbin/cron isn't. Looking at /etc/rc.d/cron (which I also have not modified):

Code:
#!/bin/sh
#
# $FreeBSD: releng/12.0/libexec/rc/rc.d/cron 298514 2016-04-23 16:10:54Z lme $
#

# PROVIDE: cron
# REQUIRE: LOGIN FILESYSTEMS
# BEFORE: securelevel
# KEYWORD: shutdown

. /etc/rc.subr

name="cron"
desc="Daemon to execute scheduled commands"
rcvar="cron_enable"
command="/usr/sbin/${name}"
pidfile="/var/run/${name}.pid"

load_rc_config $name
if checkyesno cron_dst
then
        cron_flags="$cron_flags -s"
fi
run_rc_command "$1"
So where should I be looking next?
 

ralphbsz

Daemon

Reaction score: 1,270
Messages: 2,036

Hmm. What are you are reporting can not happen ... yet it is.

Time to add debugging layer after layer. Start with /etc/rc.d/cron. At the very top, before any other commands are run, add:
Code:
echo /etc/rc.d/cron starting at `date` >> /tmp/my_cron_debug.log
echo arguments are $* >> /tmp/my_cron_debug.log
(obviously, before adding anything to it, make a backup of it, and after you're done debugging, put the backup back).

That proves that the rc file for cron is being called. If that gives no output, then it gets ugly: that means the rc subsystem is of the opinion that it doesn't want to run cron, even though it is clearly configured. In that case, you need to start editing /etc/rc, which is a rather complicated script that runs the various rc.conf files (to set parameters), and then based on the parameters in there runs the various /etc/rc.d/* files to do all the jobs. You just need to add debug echo statements to /etc/rc, until you see where it decides whether cron needs to be started or not.

On the other hand, if you see that /etc/rc.d/cron is being started, it gets easier. You can see what the parameters to the script are. Then you can start putting debugging statements into that script, in various places (you need to read and understand the script), until you find where it attempts to run cron. At that point, debug whether it really starts cron, and whether cron really does get started.

By the way, I'm sort of assuming that you know enough simple shell script programming to know how to add echo statements, you know how to follow the flow of a script, and so on.
 

k.jacker

Aspiring Daemon

Reaction score: 434
Messages: 738

Definitely weird.

According to your configuration, cron should start, that's clear so far.
Though, since that does not match what's happening, have you tried service -e | grep cron to see if an attempt to start cron is made at all?

I would also suggest to run freebsd-update IDS > ids (with the original /etc/rc.d/cron in place)
Maybe you'll spot anomalies in files that shouldn't normally be modified.

The above suggestions are just additions to ralphbsz 's debugging approach, but they may give some extra insight.
 
OP
OP
V

VucanRidr

New Member

Reaction score: 2
Messages: 9

Hmm. What are you are reporting can not happen ... yet it is.

Time to add debugging layer after layer. Start with /etc/rc.d/cron. At the very top, before any other commands are run, add:
Code:
echo /etc/rc.d/cron starting at `date` >> /tmp/my_cron_debug.log
echo arguments are $* >> /tmp/my_cron_debug.log
(obviously, before adding anything to it, make a backup of it, and after you're done debugging, put the backup back).
Yep. It got more complicated. Added the lines above. Rebooted. Checked the file. It didn't exist. Ran
Code:
service cron start
manually, and had the following:

Code:
/etc/rc.d/cron starting at Sun Jun 16 11:12:38 EDT 2019
arguments are start
That proves that the rc file for cron is being called. If that gives no output, then it gets ugly: that means the rc subsystem is of the opinion that it doesn't want to run cron, even though it is clearly configured. In that case, you need to start editing /etc/rc, which is a rather complicated script that runs the various rc.conf files (to set parameters), and then based on the parameters in there runs the various /etc/rc.d/* files to do all the jobs. You just need to add debug echo statements to /etc/rc, until you see where it decides whether cron needs to be started or not.
Yes, now k.jacker suggested looking at service -e. Here is what the laptop shows:

[root@danube ~]# service -e
/etc/rc.d/hostid
/etc/rc.d/zvol
/etc/rc.d/hostid_save
/etc/rc.d/zfsbe
/etc/rc.d/zfs
/etc/rc.d/cleanvar
/etc/rc.d/kldxref
/etc/rc.d/ip6addrctl
/etc/rc.d/netif
/etc/rc.d/devd
/etc/rc.d/resolv
/etc/rc.d/newsyslog
/etc/rc.d/syslogd
/etc/rc.d/savecore
/etc/rc.d/dmesg
/etc/rc.d/virecover
/usr/local/etc/rc.d/dbus
/etc/rc.d/motd
/etc/rc.d/rctl
/usr/local/etc/rc.d/powerdxx
/usr/local/etc/rc.d/sshd
/usr/local/etc/rc.d/pcdm
/etc/rc.d/sshd
/usr/local/etc/rc.d/hald
/etc/rc.d/sendmail
/etc/rc.d/cron
/etc/rc.d/mixer
/etc/rc.d/gptboot
/etc/rc.d/devmatch
/etc/rc.d/bgfsck

So as we can see, cron is in the service -e list, and other things in that list are getting started, and many (most?) of them do not have a status directive. The few I see that are not running are cron, sendmail, sendmail_msp_queue, and hald. I do see that sshd and powerdxx, and devd are getting started at boot. I can post the list of
Code:
basename `service -e` | while read x ; do service $x status ; done
if you think that will help.

On the other hand, if you see that /etc/rc.d/cron is being started, it gets easier. You can see what the parameters to the script are. Then you can start putting debugging statements into that script, in various places (you need to read and understand the script), until you find where it attempts to run cron. At that point, debug whether it really starts cron, and whether cron really does get started.

By the way, I'm sort of assuming that you know enough simple shell script programming to know how to add echo statements, you know how to follow the flow of a script, and so on.
I've done some scripting. The big thing for me is grokking how FreeBSD does startup scripts.
 

ralphbsz

Daemon

Reaction score: 1,270
Messages: 2,036

Good data. Sadly not the answer. Good news: "service -e" says that cron should be started. By the way, your output from that command looks very similar to mine, I just have a lot fewer services enabled (server, not desktop). Even more good news: If you start cron by hand, it actually gets started. Bad news: It does not actually get started automatically.

You say that sendmail, sendmail_msp_queue, and held are not getting started either. That might be misleading: sendmail itself changes its process name when running, so a simple look at the output of "ps" may miss it. On the other hand, "service sendmail status" should give a sensible answer. And I know nothing about hald, don't have it or need it.

Next debugging step I can see: Go into /etc/rc, and put tracing echo statements in where it actually starts services. Maybe for each service that gets started, before and after. Just to see what's going wrong.
 

PMc

Aspiring Daemon

Reaction score: 180
Messages: 514

Hmm.... I tend to find such things in my /var/log/console.log:

Code:
...
Jun 16 00:17:58 <console.info> edge kernel: Starting ntpd.
Jun 16 00:17:58 <console.info> edge kernel: Starting cron.
Jun 16 00:17:58 <console.info> edge kernel: Starting bareos_fd.
...
But, 1) mine is still 11.2, and 2) the console.log is not enabled by default (can be enabled in /etc/syslog.conf).

Certainly, the thing could also be started, and then die for whatever creepy reason.
If all else fails, I would boot into single-user, and then start the services one after another manually in the proper sequence, and see what happens.
 
OP
OP
V

VucanRidr

New Member

Reaction score: 2
Messages: 9

Sorry, it has been a while...I believe I found a solution to the problem. And it turns out it was not a "home made problem," unless you count installing the same X login manager on both. :)
The problem appears to be in the PCDM package, pcdm-201710031254_1. There are apparently two pcdm packages in FreeBSD-12, the aforementioned pcdm package and the one in pcbsd-utils-qt5. According to Allan, the one in pcdm-201710031254_1 comes from TrueOS:

the wrong version of pcdm, while supposed to work on FreeBSD, doesn't go into the background, it stays attached, assuming it is being started by openrc

which it is not

so the rest of the started scripts never run after it

so no sshd, no cron, etc
Since the pcbsd-utils-qt5 seems to have failed to build in the package updates today, I commented it out in rc.conf and am using startx from the command line. I can confirm on the latptop that cron and sendmail are now starting at boot.

So the solutions I have found are to uninstall pcdm-201710031254_1 and either
  • Install pcbsd-utils-qt5
  • Install SDDM
  • Boot to the command line and run startx

Thanks,
--vr
 
Top