systemd vs launchd

beatgammit

beatgammit

I'm looking for a comparison between the architectures of systemd and launchd. I've heard both criticized as being anti-unix, but the only real criticism of launchd I've read is the merging of crond into init.

For example, the major feature systemd brought (in my opinion) is socket activation paired with dependency management, which both decreases boot time and simplifies process management. This provides init with more information, and many of the features naturally follow. It looks like launchd offers a similar feature set (registering daemons as opposed to immediately starting them), but I haven't used launchd, so I don't know what exactly it brings to the table. I have used upstart, so comparisons there would also be helpful.

I've read the launchd wiki page, which states what it replaces, but I still don't understand how it works. Specifically, I have these questions:

  • Does launchd require (or at least strongly encourage) changes to daemons like systemd?
  • Does launchd have enough information about processes to analyze boot times (i.e. does it replace bootchart)?
  • Is there a concept of runlevels (like systemd's targets)? If so, how do they work?

It looks like there's significant interest in launchd in the FreeBSD community, so I'd like to understand it before I form any long-term opinions/potentially help out.
 
I wouldn't say there's significant interest in launchd, the efforts to port it to FreeBSD seem to be someone's side project. The original porting effort seemed to stagnate around 2005, and only recently has someone else picked up the torch.

If you're interested in it, you'll probably find more information about it from people who know about OS X. There should be someone who's had to migrate stuff to it.
 
beatgammit said:
It looks like there's significant interest in launchd in the FreeBSD community, so I'd like to understand it before I form any long-term opinions/potentially help out.
Click to expand...

Understanding the existing rc system FreeBSD has in place (and likely will remain for our lifespans) will be a lot more useful to yourself and others if you want to help out in the future.

It may not be immediately trendy but I can guarentee that it will outlive both systemd and Apple's launchd. Heck, it has already outlived Upstart and that was brand new and shiny :D

If there is a particular feature you like from another OSes startup system, perhaps you could look at intergrating it with FreeBSD's?
 
  • Thanks
Reactions: Oko
beatgammit said:
... Specifically, I have these questions:

  1. Does launchd require (or at least strongly encourage) changes to daemons like systemd?
  2. Does launchd have enough information about processes to analyze boot times (i.e. does it replace bootchart)?
  3. Is there a concept of runlevels (like systemd's targets)? If so, how do they work?
Click to expand...

  1. A Daemon started by launchd must not daemonize itself, i.e. doing a foreground/background switch on its own. For example, Apache httpd should be started by launchd with the option -D FOREGROUND. Otherwise, launchd would loose its track on the daemon and if the keep-alive flag is set, it would simply start it again.
     
  2. The philosophy behind launchd is, that the daemon should know best what it needs, and if the requirements are still not met during the boot process, it should simply exit gracefully as quickly as possible (no core dump, no romances in the error log, no e-mail to the root user, no phone-call to the president). launchd would simply try to start it again on the next occasion. It is worth to note that launchd is not blocked by a daemon which is still deciding whether it should continue to run or should exit, launchd happily continues to fork(2) other processes, and re-launches those that failed.
     
  3. AFAIK, the notion of runlevels is provided by the OS. System V runlevel designation in RC scripts is sort of accepted by FreeBSD, but otherwise ignored. So, if it ever comes to a port of launchd to FreeBSD, I guess launchd would ignore runlevels either.
     
    On Mac OS X, launchd got the concept of user-daemons and it calls this agents. Agents are configured in the home directory and run with the credentials of the respective user. This does not exactly match the concept of runlevels, but it can be used to automatically start certain agents on the login of for example a special user maintenance.

For me the main unique selling point of launchd compared to the RC system is the included configurable process watchdog. For example, if httpd would crash for any reason, launchd would simply start it again within a few milliseconds. I wrote already RC startup scripts for some of my daemons, and I wrote launchd configuration files for others, the latter of which is way more straightforward.

Anyway, I would not suggest to port launchd as is to FreeBSD. Having configuration files instead of RC-scripts would already be nice, but I would prefer simple key/value tables instead of XML files. A configurable process watchdog included into the startup system should already bring 95 % of the functionality of launchd.
 
obsigna said:
  1. ...
  2. The philosophy behind launchd is, that the daemon should know best what it need, and if the requirements are still not met during the boot process, it should simply exit gracefully as quickly as possible (no core dump, no romances in the error log, no e-mail to the root user, no phone-call to the president). launchd would simply try to start it again on the next occasion. It is worth to note that launchd is not blocked by a daemon which is still deciding whether it should continue to run or should exit, launchd happily continues to fork(2) other processes, and re-launches those that failed.
Click to expand...

Interesting. I think that's a bit more unix-like than systemd because daemons don't have to make any assumptions about which init is starting them.

I still like socket-activation, but I think this is an acceptable solution as well, and better than the status quo.
 
obsigna said:
Anyway, I would not suggest to port launchd as is to FreeBSD. Having configuration files instead of RC-scripts would already be nice, but I would prefer simple key/value tables instead of XML files. A configurable process watchdog included into the startup system should already bring 95 % of the functionality of launchd.
Click to expand...

So, something like sysutils/monit?
 
beatgammit said:
...
I still like socket-activation, but I think this is an acceptable solution as well, and better than the status quo.
Click to expand...

IMHO, this should be handled within a daemon itself, not by the watchdog.
 
beatgammit said:
obsigna said:
Anyway, I would not suggest to port launchd as is to FreeBSD. Having configuration files instead of RC-scripts would already be nice, but I would prefer simple key/value tables instead of XML files. A configurable process watchdog included into the startup system should already bring 95 % of the functionality of launchd.
Click to expand...

So, something like sysutils/monit?
Click to expand...

No, monit is a polling watchdog. I had a quick look in the documents of it, and there was an example where it polls every 2 minutes if sendmail is active.

launchd creates process forks for each of its daemons, and therefore, once a daemon dies, launchd knows this immediately, in less than perhaps a millisecond. So, if the daemon behaves well on exit, there is minimal processing involved on re-spawning it once again. Since all this happens in a compiled binary, forking, exiting, and re-forking is much faster than anything, that can be done with shell scripts.

For an examples on how this might work, look on these 3 posts:
        http://forums.freebsd.org/viewtopic.php ... 38#p120709
        http://forums.freebsd.org/viewtopic.php ... 38#p122038
        http://forums.freebsd.org/viewtopic.php ... 38#p122050

I posted a more elaborated example of a non-configurable watchdog (tunnel guard) on my blog:
        http://blog.obsigna.net/?p=229#the_watchdog

So, a launchd equivalent would ideally be a daemon by itself, lauching its daemons using fork(2) and waiting for them using wait(2). Some of the above may be taken as a basic prototype of this. All the configuration stuff needs to be added of course.
 
  • Thanks
Reactions: Oko
obsigna said:
launchd creates process forks for each of its daemons, and therefore, once a daemon dies, launchd knows this immediately
Click to expand...

Ok, so launchd is like systemd and upstart in this regard.

If daemons don't daemonize themselves, does launchd define another way for the service to notify init that it's ready? It seems systemd provides several options for signaling this. Does launchd do something similar? If so, I expect that means that boot profiling will be possible, which is a big win for embedded systems (which I'm particularly interested in).

Also, what do you think is most likely of the following:

  • Update existing init with ideas taken from launchd, upstart and systemd
  • Use launchd, but with some changes
  • Use openrc, but with some changes

I imagine FreeBSD is unlikely to make any drastic changes to init in the near future, and most of this is just speculation, but the recent drama in the Linux community over systemd got me interested in where the FreeBSD community stands.
 
beatgammit said:
...
If daemons don't daemonize themselves, does launchd define another way for the service to notify init that it's ready?
Click to expand...

On Mac OS X, launchd is init, i.e. the process that gets PID 1. Foreground daemons may write .pid-files to /var/run. FG-daemons, may create sockets, and register IPC-ports, they can do anything that BG-daemons are able to do, and more. Only, FG-daemons must not move themselves into the background because launchd is supposed to do this.

Processes may query launchctl for the status of running jobs. For example launchctl list would give a complete list of jobs registered with launchd, and launchctl list [i]label[/i] would show the status of the job having the given label.

beatgammit said:
It seems systemd provides several options for signaling this. Does launchd do something similar?
Click to expand...

It doesn't seem to me that launchd is actively signaling anything, I may be wrong, though.

beatgammit said:
If so, I expect that means that boot profiling will be possible, which is a big win for embedded systems (which I'm particularly interested in).
Click to expand...

I don't think that signaling is necessary for boot profiling, because launchd could simply do this by itself, it is in command. It is the first one that knows once one of its subordinated daemons exits prematurely and upon the next boot it may want to start that daemon later in the boot sequence.

beatgammit said:
Also, what do you think is most likely of the following:

  • Update existing init with ideas taken from launchd, upstart and systemd
  • Use launchd, but with some changes
  • Use openrc, but with some changes
Click to expand...

I don't know, what people are pretending to do or even whether they are pretending anything. I know what I would do. I would keep init completely functional and I only would force it to launch immediately a second stage init daemon (let's call it sstid having PID 2).

This sstid inherits a watchdog, would crawl the configuration directories of daemons and agents in designated system and users locations, and would launch what it finds in there. It would analyze the boot process for failures and would re-order the sequence for the next boot. Perhaps, I would implement xinetd capabilities, so sstid would create inet/inet6 sockets on behalf of daemons/agents that don't need to be started immediately, but may be started on occasion.

With that in place, a system admin could opt for using the traditional rc system, or he could leave the /etc/rc.conf completely empty and place for everything that is needed sstid config files into the designated locations, or he/she could use a wild mixture of both systems.

beatgammit said:
I imagine FreeBSD is unlikely to make any drastic changes to init in the near future, and most of this is just speculation, but the recent drama in the Linux community over systemd got me interested in where the FreeBSD community stands.
Click to expand...

Agreed:
Erich Kästner said:
Es gibt nichts Gutes außer man tut es.
Click to expand...
 
  • Thanks
Reactions: Oko
obsigna said:
I don't know, what people are pretending to do or even whether they are pretending anything. I know what I would do. I would keep init completely functional and I only would force it to launch immediately a second stage init daemon (let's call it sstid having PID 2).

This sstid inherits a watchdog, would crawl the configuration directories of daemons and agents in designated system and users locations, and would launch what it finds in there. It would analyze the boot process for failures and would re-order the sequence for the next boot. Perhaps, I would implement xinetd capabilities, so sstid would create inet/inet6 sockets on behalf of daemons/agents that don't need to be started immediately, but may be started on occasion.

With that in place, a system admin could opt for using the traditional rc system, or he could leave the /etc/rc.conf completely empty and place for everything that is needed sstid config files into the designated locations, or he/she could use a wild mixture of both systems.
Click to expand...
This sounds a lot like what Daemon Tools (not to be confused with a virtual CD device of the same name) does. At least partly. Daemon tools is actually quite old already and runs on a multitude of UNIX systems. Perhaps it could be extended to include the other features?

http://cr.yp.to/daemontools.html
sysutils/daemontools
 
SirDice said:
This sounds a lot like what Daemon Tools (not to be confused with a virtual CD device of the same name) does. At least partly. Daemon tools is actually quite old already and runs on a multitude of UNIX systems. Perhaps it could be extended to include the other features?

http://cr.yp.to/daemontools.html
sysutils/daemontools
Click to expand...

At least for prototyping the concept this may be a good start. However, I think after this some optimization and development is necessary. For example, svscan expects shell scripts in the configuration directories. If we are looking for performance than it would be better to pass the actual command line directly to the launchd equivalent, so it could launch its subordinates directly and not by the way of another shell. I would also restrict the number of possible configuration locations, perhaps:

Daemons and per-user agents provided by the base system
/etc/daemons.d
/etc/agents.d

Daemons and per-user agents provided by other system installations
/usr/local/etc/daemons.d
/usr/local/etc/agents.d

User agents installed by users
/home/user1/etc/agents.d
/home/user2/etc/agents.d
/home/user3/etc/agents.d
...

The set of configuration options could be very similar to that what Apple provides for launchd (s. launchd.plist(5)). As already mentioned in a previous post, I would prefer simple key/value tables over XML files, though.
 
  • Thanks
Reactions: Oko
I came here by looking for a comparison of launchd and systemd. This is an old thread but I can hopefully add some more information on launchd (I mainly use OS X). launchd also has the ability to launch daemons on demand either when a new volume is mounted, or when a socket is bound to. The later replaces the need for explicit interdependency maps. daemons are expected to handle their own checks via IPC, when they do so launchd will launch the required daemons. This means everything can be lazily launched. It's quite elegant and well documented.
 
Since Hubbards talk I am very interested in init systems and process control daemons. Intuitively, for me, they can not be intelligent and modern and advanced enough :). If the Unix philosphy is that you do a particular thing and that as good as possible, then this should also hold for process control daemons. Imagine you have a very very smart process control deamon :), not as smart J. A. R. V. I. S. of course :), but quite smart.....
Important note: I do not know much about init systems, so my dream could be something that an expert would not want to have.
 
I have written daemons that run under launchd and the only real difference is not calling fork(2) and setsid(2). launchd will also allow you to continue to use stdout for logging and will route that output to asl.

On other unix systems, you needed to move your working directory to /, close your std streams, and all logging had to go to syslog(3) since all your std streams were closed. Anyone familiar with APUE knows the drill.

My biggest and probably only complaint about launchd is the use of XML files for configuration. Some of the keys are pretty ambiguous. I think someone could have spent a bit more time with a thesaurus defining better keywords in the config files.
 
Joseph Bruni said:
I have written daemons that run under launchd and the only real difference is not calling fork() and setsid(). launchd will also allow you to continue to use stdout for logging and will route that output to asl.

On other unix systems, you needed to move your working directory to /, close your std streams, and all logging had to go to syslog() since all your std streams were closed.
Click to expand...
Sounds all nice, but the problem is: THIS is how it always worked, so >90% of Unix daemons are written this way. If you pull in a dependency on a non-standard "service manager" in your daemon, this service manager should at least run on top of a standard init system. It wouldn't be a problem. In fact, inetd and friends do a similar (minimal) thing for network services. Implementing the main features of launchd and systemd in a service manager that is started by init would allow you to have your managed services as well as classical daemons side by side easily and it wouldn't force anything upon users who do not intend to run a "managed service". But the attempt to just replace init not only violates the "one job, one tool" paradigm. It also forces users to use it even if they don't need it.

If, in a far future, a standard for a new Unix service manager evolved, so no "classic" daemon of any relevance is left, THEN it would be the time to completely replace init without violating Unix design principles.
 
Zirias said:
Joseph Bruni said:
I have written daemons that run under launchd and the only real difference is not calling fork() and setsid(). ...
Click to expand...
Sounds all nice, but the problem is: THIS is how it always worked, so >90% of Unix daemons are written this way. If you pull in a dependency on a non-standard "service manager" in your daemon, ...
Click to expand...

I guess, you misunderstood what Joseph Bruni was saying. The actual point is, that launchd requires the "daemon" not to daemonize, i.e. it should stay in the foreground running mode. This is because launchd does fork() and setsid() on behalf of the daemon, like inetd does, and if you develop a daemon for inetd, then as well you must not call fork()/setsid(). Many well known daemons have their special flags for this:
  • -D for sshd
  • -DFOREGROUND for httpd (Apache)
  • without -D for ftpd
  • ...
And specifying the respective flags, said daemons would behave nice under launchd without any code changes.
 
This was exactly what I understood and my argument is that this is not how a daemon is constructed. It's still fine as long as the manager for this doesn't replace init but is launched from it.
 
Zirias said:
This was exactly what I understood and my argument is that this is not how a daemon is constructed. It's still fine as long as the manager for this doesn't replace init but is launched from it.
Click to expand...

Well, yeah, read my post from 2 years ago:
obsigna said:
I don't know, what people are pretending to do or even whether they are pretending anything. I know what I would do. I would keep init completely functional and I only would force it to launch immediately a second stage init daemon (let's call it sstid having PID 2).
Click to expand...
 
obsigna said:
Well, yeah, read my post from 2 years ago:
Click to expand...
There's nothing in this post on the subject of putting this "service manager" in PID 1. This has serious implications. Referring to Joseph Bruni's post again, one thing that comes to mind immediately is logging. If all you have is stdout, you can't easily give syslogd the information (facility and level) it needs to do it's job. Maybe that was the reason for systemd to pull logging into scope. As well as lots of other things. What you typically do in a daemon supporting to run without detaching is to have some abstraction for logging. This abstraction layer can also write to stdout, in this case putting meta info like the log level into the text in some form.
 
Zirias said:
Sounds all nice, but the problem is: THIS is how it always worked, so >90% of Unix daemons are written this way. If you pull in a dependency on a non-standard "service manager" in your daemon, ...
Click to expand...

There is no dependency on launchd pulled in. Most daemons that I have seen (and written) have a "debug" or "foreground" mode that is enabled using a command line switch. Many of those daemons can also be launched from inetd using the same method. If you have one that doesn't, launchd can accommodate that too using keys in the configuration. launchd will react faster if the daemon does not disconnect from its parent because, as the parent, it simply blocks on a call to waitpid(2).
 
You must log in or register to reply here.
Back
Top