Improving boot time & parallising the start of services.

Alain De Vos

Alain De Vos

In order to improve boot time i removed lines from rc.conf and put them in rc.local.

What do you think of my rc.local :
Code: 
export PATH=/bin:/usr/bin:/usr/local/bin

/bin/echo "Start rc.local"

cd /home/x/git/postgres_exporter
/usr/bin/nohup ./postgres_exporter &
cd /root

cd /home/x/git/nginx-prometheus-exporter
/usr/bin/nohup ./start &
cd /root

cd /usr/local/etc
/usr/bin/nohup /usr/local/bin/blackbox_exporter &
cd /root

cd /home/x/git/mysqld_exporter/
/usr/bin/nohup ./mysqld_exporter &
cd /root

/usr/bin/nohup  /usr/local/bin/nvidia_gpu_prometheus_exporter &

/usr/bin/nohup /usr/sbin/service  moused                onestart &
/usr/bin/nohup /usr/sbin/service  seatd                 onestart &
/usr/bin/nohup /usr/sbin/service  jail                  onestart &
/usr/bin/nohup /usr/sbin/service  nginx                 onestart &
/usr/bin/nohup /usr/sbin/service  apache24              onestart &
/usr/bin/nohup /usr/sbin/service  minidlna              onestart &
/usr/bin/nohup /usr/sbin/service  cupsd                 onestart &
/usr/bin/nohup /usr/sbin/service  blackbox_exporter     onestart &
/usr/bin/nohup /usr/sbin/service  node_exporter         onestart &
/usr/bin/nohup /usr/sbin/service  gstat_exporter        onestart &
/usr/bin/nohup /usr/sbin/service  blackbox_exporter     onestart &
/usr/bin/nohup /usr/sbin/service  telegraf              onestart &
/usr/bin/nohup /usr/sbin/service  prometheus            onestart &
/usr/bin/nohup /usr/sbin/service  grafana               onestart &
/usr/bin/nohup /usr/sbin/service  zabbix74_server       onestart &
/usr/bin/nohup /usr/sbin/service  zabbix_agentd         onestart &
/usr/bin/nohup /usr/sbin/service  zabbix_server         onestart &

/bin/echo "Done rc.local"
 
If it works for you, good, but you need to make sure that you account for any dependencies.
It looks like you are starting blackbox_exporter 3 times.
 
I was wondering if you should force set all cpu frequencies to max at the start of rc and then reset it back to be handled by powerd when the boot is over?
 
What is strange is that some services are started after rc.local , i don't know how to fix this. I want rc.local to be "latest".
 
latest or last run?
/etc/rc.d/local runs rc.local
Do you have other service files for things like blackbox_exporter? If so then you wind up with a specific service being told to start more than once, at different times, from different locations.

If you truly want a more parallel start of services, you need to look alternatives to the current init system. I think the one used by what used to be TrueOS (ixSystems) was OpenRC (which I think is also available on some Linux distros) which tried to do more processes in parallel.
You wind up with trying to codify sequences statically or take the "thundering herd" approach where you start everything at the same time, but each item needs to wait for something to be announced as available.

Example:
Mounting network filesystems. Can't proceed until network is up.
You can start the service but the first thing it does is wake until the network is available. Or you can wait until the network is up before you start the mount network filesystems service.

That's a simple example using only 2 high level items, now think about expanding it for everything and "everything generalized".

Sometimes thundering herd is better, but you need to understand the resource usage.
"service -e | wc -l" tells you how many services are enabled.
Lets say you have 40.
We start all 40 at the same time, but the first thing they do is wait on something, maybe more than 1 (resource locking). Huge CPU spike on reboot, then nothing until locks get released.

Don't forget that init also walks through stages: single user, single user network enabled, multi user with network, etc.
Some things need to be started in single user (zfs) so the resources are available for later stages.

All this is my opinion, but I think that what you are trying to do with rc.local is going to hurt you in the long run.
 
A long time ago there was talk about using make instead of sh for rc.

It is naturally good with dependency ordering and it has parallelism already built in.
 
  • Like
Reactions: mer
service -e
Code: 
/etc/rc.d/hostid
/etc/rc.d/hostid_save
/etc/rc.d/var_run
/usr/local/etc/rc.d/microcode_update
/etc/rc.d/zfsbe
/etc/rc.d/zpool
/etc/rc.d/kldxref
/etc/rc.d/zpoolupgrade
/etc/rc.d/zpoolreguid
/etc/rc.d/devmatch
/etc/rc.d/zvol
/etc/rc.d/zfs
/etc/rc.d/cleanvar
/etc/rc.d/rctl
/etc/rc.d/ip6addrctl
/etc/rc.d/mixer
/etc/rc.d/netif
/etc/rc.d/rtsold
/etc/rc.d/resolv
/etc/rc.d/devd
/etc/rc.d/local_unbound
/etc/rc.d/newsyslog
/etc/rc.d/dmesg
/etc/rc.d/virecover
/etc/rc.d/motd
/etc/rc.d/os-release
/etc/rc.d/gptboot
/etc/rc.d/syslogd
/etc/rc.d/savecore
/etc/rc.d/ntpdate
/etc/rc.d/utx
/usr/local/etc/rc.d/influxd
/usr/local/etc/rc.d/dbus
/usr/local/etc/rc.d/snmpd
/usr/local/etc/rc.d/avahi-daemon
/etc/rc.d/ntpd
/usr/local/etc/rc.d/postgresql
/usr/local/etc/rc.d/snmp_exporter
/etc/rc.d/cron
 
If I recall correctly, pararrel runs of rc.d script was introduced as experimental and non-default functionality before, and reverted with some unresolvable issue at the moment.
Does anyone remember it?
 
T-Aoki said:
If I recall correctly, pararrel runs of rc.d script was introduced as experimental and non-default functionality before, and reverted with some unresolvable issue at the moment.
Does anyone remember it?
Click to expand...
vaguely, but no specifics.
 
The script fires a whole bunch of services to start but does not wait for these to actually start, i.e.:
Code: 
nohup service foo &  # triggers foo-start in the background
nohup service bar &  # triggers bar-start in the background

# wait  # uncomment this to wait for foo- and bar- services to complete the start up

echo "Done.local"  # prints done-message while foo- and bar-services are still starting

Is it intended?

The rational is: the rc-scripts run in a well defined order, rcorder(8). Each script must complete before rc runs the next script in the queue. The rc.local from OP triggers N services in the background and exits before the services complete the start up.
 
About 25-20 years ago, this was a big topic of discussion in the Linux camp, and in the computer science research community. There must be Usenix papers about this topic. If I remember correctly, the upshot was roughly this: Running startup scripts in parallel is hard to do, because every script has to be 100% accurate in documenting pre- and post-conditions. That would be a good thing to do, but because existing init systems run them in a fixed order, there is a very large number of bugs and unknowns, and trying to experiment with parallel startup turns into mostly an exercise in bug hunting. From a performance standpoint, it usually helps, but interestingly much less than people expect. That's because most startup activity is not CPU limited (where parallelism, in particular with SMP or multiple cores really helps), but in reality it is either network or disk IO limited. And in particular the disk IO size is an area where parallelism can end up hurting, if different "thread of controls" are reading different directories or areas of the disk, leading to long seeks as workloads are interspersed. A sequential startup can actually end up being faster, because the disk arm stays in one area for a longer time, and gets a job completely done, before doing a single long seek and staying in another local area for a longer time.

This observation then turned into a design: What if we could just prefetch all the files that are required for startup, in their on-disk order, and keep them in RAM? Then things would go much faster. So if on disks the files are stored in the order "a b c ... x y z", and we know that a and x are read first, but we also know that b will be read after all, then the file system should always prefetch b and c before heading across the long gap to x y z. But how do you know that b will be needed after a? Instead of knowing it, we can just observe which files are typically read in which order, record that observation every time we boot, and dynamically turn that into a pre-fetching schedule. A guy I know turned that into a whole system and wrote a PhD thesis about it, which made for a nice career (he's now a professor at a good school).

Two things have in the meantime made the whole discussion much less relevant. First, boot disks have been mostly replaced by SSDs on computers that matter, so pre-fetching and location ordering is much less important. Second, systemd has automated parallel startup (it is the default there, I think it uses D-bus for synchronization), and has done the hard labor of formalizing all the pre- and post-conditions, so Linux now does this all the time.
 
ralphbsz said:
About 25-20 years ago, this was a big topic of discussion in the Linux camp, and in the computer science research community. There must be Usenix papers about this topic. If I remember correctly, the upshot was roughly this: Running startup scripts in parallel is hard to do, because every script has to be 100% accurate in documenting pre- and post-conditions. That would be a good thing to do, but because existing init systems run them in a fixed order, there is a very large number of bugs and unknowns, and trying to experiment with parallel startup turns into mostly an exercise in bug hunting. From a performance standpoint, it usually helps, but interestingly much less than people expect. That's because most startup activity is not CPU limited (where parallelism, in particular with SMP or multiple cores really helps), but in reality it is either network or disk IO limited. And in particular the disk IO size is an area where parallelism can end up hurting, if different "thread of controls" are reading different directories or areas of the disk, leading to long seeks as workloads are interspersed. A sequential startup can actually end up being faster, because the disk arm stays in one area for a longer time, and gets a job completely done, before doing a single long seek and staying in another local area for a longer time.

This observation then turned into a design: What if we could just prefetch all the files that are required for startup, in their on-disk order, and keep them in RAM? Then things would go much faster. So if on disks the files are stored in the order "a b c ... x y z", and we know that a and x are read first, but we also know that b will be read after all, then the file system should always prefetch b and c before heading across the long gap to x y z. But how do you know that b will be needed after a? Instead of knowing it, we can just observe which files are typically read in which order, record that observation every time we boot, and dynamically turn that into a pre-fetching schedule. A guy I know turned that into a whole system and wrote a PhD thesis about it, which made for a nice career (he's now a professor at a good school).

Two things have in the meantime made the whole discussion much less relevant. First, boot disks have been mostly replaced by SSDs on computers that matter, so pre-fetching and location ordering is much less important. Second, systemd has automated parallel startup (it is the default there, I think it uses D-bus for synchronization), and has done the hard labor of formalizing all the pre- and post-conditions, so Linux now does this all the time.
Click to expand...
My quite rough and should-be-incomplete guess:
Forcing to list any resources to be specified in each rc.d scripts (including ones supplied via env from /etc/defaults/rc.conf, /etc/rc.conf and /etc/rc.conf.local) could help rc subsystem to resolve undefined dependencies within each scripts.
For programmers of rc.d scripts, determining dependencies are quite hard, as determining all possible combinations of scripts including not only in-base ones but also ones installed via ports or even in the wild in the far future for the time the dev leaves maintainership is almost impossible.
On the other hand, determining the resources the script alone need is usually far easier.

Another concern would be initialization of devices in parallel.
This would strongly depend on the design of I/O bus resource managements. Here, initializing resources which are NOT conflict with others can be done in parallel safely, but it would be quite limited.
Maybe quite strict spec in resource managements in I/O bus would be wanted at first place.
 
Alain De Vos said:
You can start a postgresql database & mariadb database in parallel.
You can initialise a keyboard & a video card in parallel.
Click to expand...
Are you sure? What if one of those is required BEFORE starting the network (for example to set the network configuration), and the other requires the network to be up (at least enough for some function, such as getting a host ID from the IP address)? How can you be 100% sure that they can run in parallel, without going through any resource they need, and pre-condition that has to be met? Including all the libraries they call, perhaps due to a local setting (like the keyboard config file being on an NFS mount, or the video card requiring authentication)?
 
  • Like
Reactions: mer
ralphbsz my thoughts align with this.
Startup of a system has never been trivial.
Simplistically every service has a "required/needed before I run" and "When I complete this is what I provide".
Lots of times the required's cascade into lots of hidden dependencies.

I think part of the "problem" is laptops; where people may be shutting down and restarting a lot more. I use desktops at home/laptops for work (that I treat like desktops) where my uptime is measured in days. Reboot typically tied to updates or power loss."reboot, go refill coffee cup, take 2 sips, system is ready"

That means boot time is lost in the noise. Say "optimizing" boot parallelism gets you a 1 minute faster boot. Your system is up for 24 hours. That 1 minute is 0.07%, so noise. Uptime of 100 days? Boot time may become irrelevant.

Now commercially, if a system has SLA of "5 nines" boot time can impact that (but the system would likely be redundant).

So I'm all for faster boot, but not at the expense of "correct boot 100% of the time".
 
cracauer@ said:
Just delete all the factory rc.d crap and hack it into /etc/rc.local with & at the of every line.
Click to expand...
Friend of mine (office mate) did that in the early 2000s on Linux. Spent half a day on it, and then his system was back up and running. Turns out the savings were minimal.

Another colleague spend two days carefully optimizing how his (kernel development testing) system boots. Got it down to 12 seconds for crash and reboot, and that was in the days of spinning disk drives. The key is not in parallelism, but in removing everything that is not strictly necessary for the task at hand. That was in the days when Linux kernel debugging was still done with a serial cable between two systems.
 
You must log in or register to reply here.
Back
Top