What worries me more is that nobody does logical verification, and not so many people look at the bits and bytes at all. When I start to look at the bits and bytes -and I do so if somthing annoys me the way it works, and I for my personal taste want it different-, if I look close enough, I happen to find logical flaws.
On this planet, the above seems to cover most areas of life.
Seems to me we got decadent, and that does not concern Linux, it concerns all of the stuff. And I see three reasons:
- it is no longer necessary to strive hard to get even some basic functionality operative. Systems tend to work out of the box.
- The whole system has become just too big for anybody to get a deep understanding of more than only a small fraction of it.
- Everybody is now using computers, and it is not necessary to have much of an idea how the thing works. I'm not sure if that's a good thing. I don't think kiddies need computers in elementary school, and I still feel a campfire in the wilderness a lot more soothing.
Today, something happened to me. During the years I used Linux, it happened to me many times. Many nights, it made me sit in the car, and visit the server room, far away from the city to get a console. Until today, it had never happened to me under FreeBSD, even though I have way more FreeBSD servers running than how many Linux boxes I could ever touch, and I have been using it for more than ten years.
After I ran freebsd-upgrade, I could’t login via SSH. I did it the canonical way. I didn’t save myself or the server from any reboot I was told to do. I upgraded the packages between the second and third round of
freebsd-update install. I can’t see how it could have been my fault. Fortunately, it was a test server, and I had remote console, but in the past, I felt much safer doing the upgrade. It was the upgrade from 11.2 to 12.0. I tested it inside a VM first. (That’s also new. I used to be bolder with FreeBSD). Then, I tested it on a production server hosted somewhere else.
Then I did it on Google Compute Engine, and I couldn’t login any of the two servers.
Someone (not me but who created the install image on GCE many years ago) managed to put a few lines at the end of the sshd_config. One of the lines was a Chiper setting. I don’t remember seeing any warning about it anywhere. (I mean, logs, login screen, whatever). FreeBSD 12 said, "I don’t like this Chiper, goodbye".
I still have servers at places where I have either no remote console access, or it takes ages to get one. They are in other countries.
People commit mistakes. Or one could say this was not a mistake from the maintainer. I don’t know. But if I were to prepare an upgrade for an operating system, I would think, right after the kernel and the shell, the ssh daemon is the third most important thing that should work, no matter what (unless the user did something crazy).
Since I have no Chipher line in the
sshd_config anywhere else, even doing a few test upgrades elsewhere did not help me expect this.
The sad part is that the sole reason I had two additional servers in GCE today is that I spent five hours of my life on the google-cloud-engine daemons that have hardcoded paths from Linux, refer to non-existent files, don’t process the config templates they are supposed to process, don’t have such a basic error handling that if an executable is missing, the log should display the name of the executable (which was also from Linux), and I could go on. Another daemon of this package adds two lines to my
/etc/hosts file in every few minutes. Over time, my hosts files grow to a few megabytes size. In the end, I wrote a script to re-create the hosts files on my servers from templates because I had no mood to trim them by hand on fifteen servers again and again.
For the latter, I don’t blame anyone related to FreeBSD, because these scripts are written in Python and I believe they are from Google, and it fits well with the rest the company did in the last one or two years.
On the other hand, I can’t blame either Google for adding a Chiper line to the
sshd_config (or the person from FreeBSD if they did it). I think there should be an automated test in the upgrade process that tells the user, "Hello. I am the SSH Daemon. I have a bad news for you", during the upgrade, before the reboot happens. They wouldn’t have to think about every possible line that became unsupported. It would be enough to make the sshd daemon check the config file whether it is valid or not.
It was also this year when I had some shared library error messages in a jail after some upgrade. Maybe it was me. I’m not sure. But I have been upgrading that server for maybe eight years. It’s one of my first FreeBSD machines co-located.
The answer is simple. The technical evolution has become magnitudes faster than the human evolution. A few decades ago, one could understand maybe every type of computer they had around in near to 100%. Today, understanding certain config files may be a full-time job.
