Solved Sporadic automatic reboots - "rebooted by root"

mack3457 said:
Can there be an ssh configuration, which may reboot the machine due to some configuration?
Click to expand...
Yes. You can have a command in .ssh/authorized_keys that is directly invoked whenever a login with that key happens. In fact this is a common practice on headless machines to have a simple means to reboot them by operators who shall not get a login shell.

BTW, there should be some data in /var/log/auth.log about who is logging in from where via ssh.
 
I can wholeheartedly recommend security/tripwire.
After this thread I will be adding more things to watch.

Code: 
        /usr/local/etc/tripwire/twpolicy.txt    -> $(CheckAll);
}
#
(rulename = "Root config files", severity = 100 )
{
        /root                           -> $(SEC_CRIT);
        /root/.cshrc                    -> $(SEC_CONFIG);
        /root/.login                    -> $(SEC_CONFIG);
        /root/.history                  -> $(SEC_CONFIG);
        /root/.profile                  -> $(SEC_CONFIG);
        /root/.ssh/known_hosts          -> $(SEC_CONFIG);
}
#
(rulename = "megarouter config", severity = 100)
{
        /etc/rc.conf                                    -> $(CheckAll);
        /boot/loader.conf                               -> $(CheckAll);
        /etc/sysctl.conf                                -> $(CheckAll);
        /etc/pf.conf                                    -> $(CheckAll);
        /usr/local/etc/dnsmasq.conf                     -> $(CheckAll);
        /etc/ssh/sshd_config                            -> $(CheckAll);
        /etc/ssh/ssh_config                             -> $(CheckAll);
        /etc/fstab                                      -> $(CheckAll);
        /etc/ttys                                       -> $(CheckAll);
        /etc/resolv.conf                                -> $(CheckAll);
        /usr/local/etc/dnsmasq.d/dnsmasq.blacklist.txt  -> $(CheckAll);
}
 
So, in auth.log, indeed I find these activities logged every six minutes:
Code: 
Apr 22 02:18:33 othercomputer sshd[4094]: Accepted publickey for root from
aa.bb.cc.dd port 58376 ssh2: RSA SHA256:...somekey...
Apr 22 02:18:33 othercomputer sshd[4094]: Received disconnect from aa.bb.cc.dd port 58376:11: disconnected by user
Apr 22 02:18:33 othercomputer sshd[4094]: Disconnected from user root aa.bb.cc.dd port 58376

I now found the problem, it dates back to the time when I initially had Realtek ethernet cards built into the router: the router would sometimes simply stop (every maybe three or five weeks or similar) when traffic was too high.

So I had a script on another computer running per cron every two minutes, which did a ping to an external computer and when I could not reach the external computer three times in a row, I would assume, the router stopped working and I reboot the router.

I later replaced the Realtek cards by Intel brands and didn't have anymore problems since then, so I should have deleted that cron job, but I forgot about it.

Unfortunately, the external computer stopped being online obviously since April 19th, so my script assumed the router being down and it's time to reboot it every three unsuccessful pings == every six minutes.

Thanks for all your help and insights into methods and features and commands, I wasn't aware of.

Thomas Mack
 
mack3457 said:
[...]
I now found the problem, it dates back to the time when I initially had Realtek ethernet cards built into the router: the router would sometimes simply stop (every maybe three or five weeks or similar) when traffic was too high.

So I had a script on another computer running per cron every two minutes, which did a ping to an external computer and when I could not reach the external computer three times in a row, I would assume, the router stopped working and I reboot the router.

I later replaced the Realtek cards by Intel brands and didn't have anymore problems since then, so I should have deleted that cron job, but I forgot about it.

Unfortunately, the external computer stopped being online obviously since April 19th, so my script assumed the router being down and it's time to reboot it every three unsuccessful pings == every six minutes.
Click to expand...
You had the good fortune of being able (and enabled) to remember that specific piece of history. Imagine how hard it would have been for anyone else to come up with this.

When I would ever need to implement such measures like triggering a reboot via a cron script , which in and of itself is pretty substantial (even for a router) and base this on the (ping) availability of an external server, I would think long and hard about how to document it and/or generate emails for (some) reboots that take place, linking the contents to the specific circumstances. Thereby making it as discoverable as possible for a "total stranger" faced with this (sysadmin) problem. Good that you've solved it!
 
You must log in or register to reply here.
Back
Top