Solved FreeBSD 14.1-RELEASE /dev/null disappears

M

mikkol

Dears,

Second day in a row, while a FreeBSD 14.1-RELEASE is running, it suddenly loses /dev/null. Or that is how it appears to me.

The symptoms are that Dovecot will stop doing anything, sshd will stop accepting connections, /var/log/messages will contain errors from the mail transport agent regarding fd0 not being usable because /dev/null does not exist.

slow sm-msp-queue[10937]: 46P6ThmZ010937: SYSERR(root): fill_fd: disconnect: cannot open /dev/null: No such file or directory
Click to expand...

A reboot fixes this for now, but obviously is not a solution. I don't have enough information to figure out where to start debugging this. Does someone else here who is willing to share?

A note on the rebooting: the system does shut itself down every night and a remote system starts this up every morning, so there is definitely a daily reboot.
 

Attachments

  • Screenshot 2024-07-25 105544.png
    Screenshot 2024-07-25 105544.png
    418.8 KB · Views: 73
When the problem manifests, ls -lah /dev/null tells me (root) that there is no /dev/null. The number of drives is the same when the problem manifests as when the problem does not manifest.

At this moment, the problem is not manifesting, and I may have to wait until tomorrow to see if I can give you verbatim output of those (wait for the problem to manifest again), but, prior to that, I can tell you that zpool status said all zpools were OK, all 10 drives were present and accounted for, all passed a SMART assessment, and the system was working relatively OK for a system that doesn't have /dev/null. For example, samba4 worked without hiccup.

That's 10 drives doing 5 zpools in a mirrored configuration.
 
Usually, /dev is mounted on a separate file system, which is not actually backed by disks, so wondering about geom or zpools is pointless. The mount line should look roughly like this: "devfs on /dev (devfs)". I would start by checking the output of mount. Then I would check whether jails or chroot is involved in the affected processes.

It is possible for a process to delete /dev/null. You can easily recreate it with "minor /dev/null c 23 0", but nobody should have deleted it in the first place.
 
Changes to the system include an upgrade to 14.1-RELEASE, but more than these two days ago, and removing and reinstalling VirtualBox (in vain). This is a rather simple system that runs dnsmasq, samba4 as a simple file service, apache24, nut, dovecot, and sendmail. And possibly something else that may come as standard, but nothing ever-changing and nothing too complicated. I don't see any reason for anything suddenly deleting /dev/null.
 
mer said:
I've never heard of anything trying to delete /dev/null; perhaps something passed /dev/null as an argument?
Click to expand...
Well, now you have. That was, in fact, exactly what was happening. The one to blame is the one between the keyboard and the backrest. It's me. Well, a cron job edited by me.

When a script cronned to run once a day says
Code: 
rm /root/powerstat-data.txt /dev/null
and not
Code: 
rm /root/powerstat-data.txt > /dev/null
, it appears that once a day, /dev/null will disappear. Entirely by design and through no bug.

Thank you, Mer, for making me doubt myself and doublecheck.

I have fixed my darned typo in the script, which I modified two days ago.
 
mikkol said:
Well, now you have. That was, in fact, exactly what was happening. The one to blame is the one between the keyboard and the backrest. It's me. Well, a cron job edited by me.

When a script cronned to run once a day says
Code: 
rm /root/powerstat-data.txt /dev/null
and not
Code: 
rm /root/powerstat-data.txt > /dev/null
, it appears that once a day, /dev/null will disappear. Entirely by design and through no bug.

Thank you, Mer, for making me doubt myself and doublecheck.

I have fixed my darned typo in the script, which I modified two days ago.
Click to expand...
Typos happen...
 
You must log in or register to reply here.
Back
Top