I started to investigate our server problem - as I can see it is restarting without any issues - checked dmesg, messages, other logs - no error and nothing why it can be rebooted
There used to be a number of things which prevented crash dumps from being generated. The ones that could be fixed in software were mostly addressed. However, if there is some catastrophic hardware problem (for example, power problems or some hardware that is resetting the system) then there's nothing that FreeBSD can do to write a crash dump.
In the "old days", the easiest way to hopefully see what was happening was to use a serial port console instead of the video console. But that has become harder to explain / accomplish over time.
Before going further, I'd like to recommend a little script I wrote called "updown". You can get it
here. You copy it into
/usr/local/etc/rc.d/updown (note that you remove the .txt extension) and add:
to
/etc/rc.conf. This will email the "root" user (which I assume is forwarded somewhere useful) whenever the system is shut down or boots. When it reports the system booting, it will tell you either how many seconds elapsed since an intentional shutdown command or tell you that this was likely a crash restart. That will let you eliminate the system being intentionally rebooted as well as emailing you right away if the system does a crash restart.
Once that has been eliminated, let's see if we can pinpoint something like a resource starvation issue. From a
different system (which doesn't have to be FreeBSD), log into the problem system. Run a command like
top -S or
systat -v. If the system does a crash restart, the last displayed information will show the system state just before it crashed. That may let you discover something like running out of some resource, excessive interrupts, or other problems. Note that you're doing this from a different system - if you do it on the problem system's console, you'll lose the data when the hardware is initialized as part of the restart. If the error happens without any useful information being shown by those utilities, you may have to find some way to record what is happening on the console screen, perhaps by pointing a cellphone camera at it and hitting "record". That probably won't be useful if the console is in graphics mode, so if you're running a desktop environment you would probably need to disable that. Panic messages are generally just kernel printf's and don't bother putting the screen back into text mode first.
Report back when you've collected this additional info and we'll see where to go next.
IMPORTANT: You should never just blindly install code that someone hands you, particularly if it will run as root (as my updown script does). Give it at least a cursory looking-over to make sure it won't do anything nasty. The script is pretty simple. The only parts that are un-needed are the tests for configuration files for various local utilities, and if it doesn't find those files, it will happily run without them.
NOTE: There is a limitation (I hesitate to call it a bug) in this script - it assumes that the local system can generate the shutdown email and hand it to another system within 20 seconds. If that timeout is too short for your environment, you either won't get the shutdown email until the system reboots, or you'll get a copy when it shuts down and a duplicate when it reboots.
