The Tragedy of Systemd

Phishfry

Beastie's Twin

Reaction score: 2,644
Messages: 5,558

The systemD concept is actually very good and was not new
Yes but certain things like journald came with it.
Binary logs really make me sad. There is no way I should need a log reader to look at my logs.
Look at how FreeBSD handles logs.
They are rotated and archived. That is ideal to me.
BInary logs are not.
 

rigoletto@

Daemon
Developer

Reaction score: 1,249
Messages: 2,291

There is no way I should need a log reader to look at my logs.

I have no opion about binary logs, but you still need vim/cat/less/something to read logs on FreeBSD, and for staters I don't see a really practical difference into those.
 

Phishfry

Beastie's Twin

Reaction score: 2,644
Messages: 5,558

Human readable versus machine optimized.
I get your point.
cat /var/log/messages | tail -n 30
 

aragats

Daemon

Reaction score: 703
Messages: 1,618

In my opinion binary logs are unacceptable, what we can gain with them? Save a couple of bytes or a couple of milliseconds?
How would you watch a log in real time, like we do now?
tail -f /var/log/messages
 

rigoletto@

Daemon
Developer

Reaction score: 1,249
Messages: 2,291

In my opinion binary logs are unacceptable, what we can gain with them? Save a couple of bytes or a couple of milliseconds?
How would you watch a log in real time, like we do now?
tail -f /var/log/messages

There are other implementations I also don't know how they work, but since the talk is about journald I did a search over jounalctl:

journalctl -f

List all journal entry that is in the system will be displayed in real-time. Last 15 lines would be displayed, similar to running tail /var/log/messages -f.

This Digital Ocean tutorial bring some opinions.
 

aragats

Daemon

Reaction score: 703
Messages: 1,618

That's good they provide such options, but still, what's the point?
It's like in large companies you often receive a MS Word/Excel attachment with a simple list of actions or similar, or even worse: a single picture is inserted in MS Word document.
 

rigoletto@

Daemon
Developer

Reaction score: 1,249
Messages: 2,291

Based on the Digital Ocean article I pointed, that centralize all logs which bring some interesting logs manipulation opportunities; however that is not the only implementation...
 

Phishfry

Beastie's Twin

Reaction score: 2,644
Messages: 5,558

I like what I have. Good for them.
journalctl -n 20

After reading that DO article I could see where a programmer might think it is better.
Providing binary data for applications to access and manipulate.
 

rigoletto@

Daemon
Developer

Reaction score: 1,249
Messages: 2,291

Phishfry you made me remember a talk I had with a good Gentoo developer some time ago about the eternal fight between sysadmins and developers:

sysadmins doesn't want anything to change becuase they don't want to have to re-learn how to admin the system, and developers are always willing to do changes to improve with new features. Or something like that. ;)
 

rigoletto@

Daemon
Developer

Reaction score: 1,249
Messages: 2,291

Just amend, I don't know Benno, I never had any contact with him but I liked his presentation, specially the BSDCan one, but being a Ada/SPARK (and also OCaml at some extend) advocate I do really, really have to disagree when he says something about it imust be buggy because it is software (but that is a complete different story), and I do leave a VIDEO why.
 

rigoletto@

Daemon
Developer

Reaction score: 1,249
Messages: 2,291

Good point, IDK, but would be worthy to look on how that, if any, is implemented in Solaris SMF and launchd. ;)

A hint would be to run NextBSD (or simply look for a Mac) to look how launchd does that. I would do that if I had not invented to upgrade the whole OCaml stuff. :eek:
 

Spartrekus

Daemon

Reaction score: 163
Messages: 1,150

BINARY LOGS:

I use GZIP for many things. vim can even read a gzip document. vim log.gz will give nice log.
xz certainly not, zip certainly not, but gz is fair ok, if after some time the log will be "archived" with gz.
 

phoenix

Administrator
Staff member
Administrator
Moderator

Reaction score: 1,288
Messages: 4,099

The Linux discussion continues unabated and seemingly this isn’t off topic.

It's in the Off-Topic forum, but still tangentially related to FreeBSD.

While the discussion has taken a few wrong turns, it's remained mostly on-course for a useful discussion. We're watching this thread closely, though. So far, it's okay to continue.
 

Spartrekus

Daemon

Reaction score: 163
Messages: 1,150

Good point, IDK, but would be worthy to look on how that, if any, is implemented in Solaris SMF and launchd. ;)

A hint would be to run NextBSD (or simply look for a Mac) to look how launchd does that. I would do that if I had not invented to upgrade the whole OCaml stuff. :eek:

NextBSD could be nice, does it still go on?

In same direction, alternatives to BSD:
NetBSD is actually a superbe alternative to FreeBSD, especially for "high-performance" servers.
For sure, NetBSD will never get a Systemd inside ;)

Actually, who really knows about it for FreeBSD, maybe in 20 years freebsd + systemd ? no one is able to predict the future. There are large companies relying on BSD as well.
 

rigoletto@

Daemon
Developer

Reaction score: 1,249
Messages: 2,291

The reason I pointed to NextBSD was becuase it has/was Mach and Launchd implemented into it, and so would be a quick alternative to MacOS if someone would like to take a look on how those work (I don't have Apple products for instance).
 

forquare

Well-Known Member

Reaction score: 175
Messages: 325

But a valid question is how would you debug a system by looking at logs when they are in binary.
In FreeBSD I mount the downed computers filesystem from a USB rescue stick and cat the logs.
tail -n 30 cat /mnt/var/log/messages

journalctl a broken systems disks logs from a rescue system. I bet that ain't as easy.
https://unix.stackexchange.com/ques...ct-systemd-system-journal-from-another-system

Not particularly knowledgeable in this area so speaking only from experience. On the standard CentOS 7.x systems we have at work the regular log files are available. I frequently grep(1)/tail(1)/whatever /var/log/messages and other log files.
I can rescue a system from a CentOS USB stick and check out the logs using regular tools.

My guess is that journalctl gives the option to easily output in additional formats/methods, and optionally turn logging to a file off.

I find it intriguing that the cloud team at work rarely consider individual systems, there are hundreds of them running a few services. They care about service stability and uptime. Logging into a single machine to check out some logs is probably a waste of time because either the issues is happening everywhere, or the machine in question can be destroyed and rebuilt without majorly impacting the service.
For them they generally want a fairly high-level view of systems, centralised location. For the applications they generally want finer-grained views, but still in a central place.

BINARY LOGS:

I use GZIP for many things. vim can even read a gzip document. vim log.gz will give nice log.
xz certainly not, zip certainly not, but gz is fair ok, if after some time the log will be "archived" with gz.

I've seen people use vim to look at logs and they have always ended up making some mistake where the forget to pass the "-R" flag, or don't know about Read-Only mode, and modified the log (maybe because they were filtering the contents) and accidentally write the modifications out because of muscle memory.

Personally I usually use less(1) for regular log files (switching to view(1) if I want more vim-like controls, but often use multiple pipes before) , and zless(1) for bz2/gz/xz/zip compressed log files.
 

ralphbsz

Son of Beastie

Reaction score: 2,297
Messages: 3,207

If someone who has root permissions uses vim on the live log of a system, they need to be taken behind the barn and hit with a stick. OK, that was a joke. They need to get an education of how to safely operate as a system administrator (which includes not using root if it is not necessary), and how to use Unix commands safely. And the will get the root password back only after they have demonstrated safe work practices.

Another thing that gets forgotten in this whole discussion of "binary" log files in systemd: In the default configuration of systemd, it does create clear text log files in /var/log, including the log rotation behavior. My little machine at home that uses systemd for example has all the standard log files:
Code:
# fgrep -i eqmon /var/log/daemon.log*
/var/log/daemon.log.1:Feb 20 06:33:59 pumphouse systemd[1]: Started EQmon monitoring daemon.
/var/log/daemon.log.1:Feb 21 10:17:04 pumphouse systemd[1]: Started EQmon monitoring daemon.
# uname -a
Linux pumphouse.example.com 4.14.86-v7+ #1175 SMP Thu Dec 6 14:04:38 GMT 2018 armv7l GNU/Linux
# more /etc/debian_version
9.6
This machine is an RPi, running Raspbian, but that is nothing but Debian recompiled for Arm, and with systemd. Just to show how easy it is to use systemd, here is the whole config file for a service:
Code:
# more /etc/systemd/system/eqmon_pumphouse.service
[Unit]
Description=EQmon monitoring daemon

[Service]
User=plumber
WorkingDirectory=/home/plumber/lr_eqmon
ExecStart=/home/plumber/lr_eqmon/eqmon_pumphouse
Restart=on-failure

[Install]
WantedBy=multi-user.target
This replaces the /usr/local/etc/rc.d/eqmon file in FreeBSD, which is 27 lines long, plus the 1-liner in /etc/rc.conf.
 

blueCub

Member

Reaction score: 8
Messages: 62

In same direction, alternatives to BSD:
NetBSD is actually a superbe alternative to FreeBSD, especially for "high-performance" servers.
For sure, NetBSD will never get a Systemd inside ;)
I'm a bit intrigued, Can you please explain more why you think NetBSD is superb alternative to FreeBSD?
I like NetBSD and started playing with it for couple of weeks now but curious to know what makes you think it is superb to FreeBSD.
 

Spartrekus

Daemon

Reaction score: 163
Messages: 1,150

Not particularly knowledgeable in this area so speaking only from experience. On the standard CentOS 7.x systems we have at work the regular log files are available. I frequently grep(1)/tail(1)/whatever /var/log/messages and other log files.
I can rescue a system from a CentOS USB stick and check out the logs using regular tools.

My guess is that journalctl gives the option to easily output in additional formats/methods, and optionally turn logging to a file off.

I find it intriguing that the cloud team at work rarely consider individual systems, there are hundreds of them running a few services. They care about service stability and uptime. Logging into a single machine to check out some logs is probably a waste of time because either the issues is happening everywhere, or the machine in question can be destroyed and rebuilt without majorly impacting the service.
For them they generally want a fairly high-level view of systems, centralised location. For the applications they generally want finer-grained views, but still in a central place.



I've seen people use vim to look at logs and they have always ended up making some mistake where the forget to pass the "-R" flag, or don't know about Read-Only mode, and modified the log (maybe because they were filtering the contents) and accidentally write the modifications out because of muscle memory.

Personally I usually use less(1) for regular log files (switching to view(1) if I want more vim-like controls, but often use multiple pipes before) , and zless(1) for bz2/gz/xz/zip compressed log files.

zless might take a lot of memory because of pipe .. no?

Here is the description concerning this related journalctl systemd topic: https://wiki.ubuntuusers.de/systemd/journalctl/
 
OP
J

justinnoor

Active Member

Reaction score: 55
Messages: 116

the cloud team at work rarely consider individual systems

Logging into a single machine to check out some logs is probably a waste of time

Very astute observations. It is inefficient to go fishing for random log files on, for example, a container cluster. journalctl allows you to access log files of the host and of specific containers. It is difficult to avoid the conclusion that systemd was designed with microservices in mind. As far as why journalctl has to be part of the init system is question worth exploring.
 

Trihexagonal

Son of Beastie

Reaction score: 2,285
Messages: 2,852

I'm a bit intrigued, Can you please explain more why you think NetBSD is superb alternative to FreeBSD?
I like NetBSD and started playing with it for couple of weeks now but curious to know what makes you think it is superb to FreeBSD.

sensucht94 is a proponent of NetBSD. I suggest asking at DaemonForums where they have a NetBSD sub-forum and might possibly reply.

He has screenshots of his desktops on my site as do other FreeBSD forum members of their unbastardized BSD and UNIX boxen.
 

Spartrekus

Daemon

Reaction score: 163
Messages: 1,150

Systemd is very important to be discussed, because it affects much Unix systems (whatever if Linux can be Unix-like system, maybe it was).
 
Top