Hi gang!
Editorial
FreeBSD is my all time favorite Unix(-like) operating system. It is both a means as well as a goal for me at times, meaning that it's both simply a tool to get things done (like writing this article right now) but it's also often a goal in itself; trying to make things 'better' or while upgrading the system. I actually take great joy out of maintaining and working with the source tree and updating my personal server is always a bit special because of that. So I figured, why not share?
Now, some of you will probably already know about varies items on my list; keep in mind that when I write things like these I try to cater to the whole community; both veteran and less experienced users alike.
#1 Why not use freebsd-update?
freebsd-update is the de-facto tool to help you keep your system up to date, and it does a pretty good job too. So why bother with all the hassle that a source tree (and its maintenance) brings you? Whats that, the source gives you more control? Really now... ever heard of /etc/freebsd-update.conf (see also freebsd-update.conf(5))?
See, if you don't want certain parts to get updates then you can simply remove those and then tell freebsd-update to ignore them. You can use Components to specify exactly which parts you want to include in your update (or upgrade) and if you need more control you can also specify IgnorePaths or IDSIgnorePaths (the latter uses regular expressions) and make sure that only the part(s) you care about get included.
Of course this option isn't perfect. For example the Components which make up a FreeBSD environment are basically world (userland), kernel (guess
) and src (the source code in /usr/src). But you can also 'sub divide', for example: src/sys or world/games (this used to be a thing anyway). And there are several problems here...
First: the Components section isn't very well documented. As I said: world/games used to be a thing, but I doubt it is now. I always figured that the components were based on the actual archives which you use to install FreeBSD: base.txz, doc.txz, kernel.txz, ports.txz, src.txz and tests.txz. I seem to recall that games.txz used to be included, but it also created a theoretical problem with world/games. I mean; games was a separate archive yet also part of the base system?
The second problem is that you can only include components but not exclude them. I can't for example say: "upgrade everything except the source code". With three components this obviously isn't a big issue, but what about all those sub-components like world/games, world/tests, etc.?
And finally: are you sure that you can leave some things out? If other components rely on the one you're excluding then that could create quite a bit of a problem because you'll only notice after you upgraded (and thus somewhat whacked) your system.
Still... if you really want to customize your system to such extensive levels then that might be a good indication that you're ready to use the source But otherwise freebsd-update is a very good way to maintain your system.
Always remember: FreeBSD isn't like Linux where it somehow became the norm to build your own kernel. Definitely don't expect better performance or such just because you build your system vs. installing a pre-build version.
#2 Lock it up!
Did you know that you can easily build a FreeBSD system from a readonly source tree? It's simple: all you need is a dedicated filesystem and then mounting that read-only. This can be a little tricky with UFS (determining how much space to reserve) but if you're using ZFS then this should be a no-brainer, especially if you're using a server which hosts plenty of other services.
For UFS this would boil down to:
Why go through all that trouble? Well, easy: to prevent accidents and/or attacks of course. What better way to attack a system than to try and sneak in some code of your own? And before we start arguing about having to know C, the way the code was structured and of course it being somewhat unlikely that an attacker just so happens to know about an exploit to abuse: what about all those install scripts? Would you really notice if I were to change 00555 into 04555? Or if I were to add a few extra "kits" of my own to install? I don't think so...
Lock up your tests too!
As of FreeBSD 11 we have /usr/tests which contains a large collection of Kyua scripts which can help you test all sorts of aspects of your FreeBSD system. Now, of course, these tests don't necessarily focus on intrusion but more so on "system sanity" as I like to call it, but why take the risk? This is why I also maintain zroot/src/tests which is only changed to a read-write state during a system upgrade (which is mostly done in single user mode anyway).
#3 Read the right documentation!
You can customize your FreeBSD installation by adding options which either include or exclude certain aspects of the base system into /etc/src.conf. So it makes sense to read src.conf(5) prior to upgrading, right? Well, maybe not. If you simply run
Instead try this:
#4 Just because you can remove parts doesn't also mean that you should!
Now, this obviously depends on the parts you're going to exclude. For example if your system doesn't have any floppy drives (which one do?) then it makes sense to use WITHOUT_FLOPPY=. And if you're certain that you'll be using the source tree from here on then WITHOUT_FREEBSD_UPDATE= would also make sense. Options like these don't have that much impact.
But don't start assuming things. For example... if you're using OpenSSL (or LibreSSL) from the ports collection then don't simply assume that you won't be needing the one from the base system (WITHOUT_OPENSSL=) because what about all those base services which rely on crypto engines? Not including OpenSSL also results in Kerberos and OpenSSH getting removed and it'll have even more consequences.
There's also the issue of some changes which results will only become apparent after the installation itself. While it might seem like a good idea to remove Clang from the base system in favor of those in the ports you probably end up having a hard time during your next source build.
#5 Makefile makes for a great "cheat sheet"
Forgot a specific required step to build your base system? Easy:
Fun fact: even though it isn't officially supported at the time of writing the packages target already exists in 11.2-P4 and even already does something (but doesn't fully work).
#6 Don't 'pollute' your sourcetree!
This should be somewhat obvious but I'm definitely not too proud to admit that I also fell victim to this one myself
So the official documentation says that a good way to start a new kernel configuration is to copy GENERIC and then customize your new file, I quote:
Word of the wise: don't. What would happen if you're upgrading to the next major release and during that process replace this source tree for the other? Then you'd lose all your customization.
But another major problem is that this methodology would 'separate' your workflow from the official one. What if... a new required feature got added to the system and you kept relying on your 'config copy' made several years ago?
So do this instead:
This will make sure that no matter what happens to your source tree you'll always have full control over your configuration, both the default GENERIC as well as your own. And adding KERNCONF to /etc/make.conf will prevent you from having to constantly reference your own configuration as well.
Then in your config file I strongly suggest using this approach instead of blindly copying GENERIC and editing that one:
If you want to remove stuff: no problem! Just use whatever you find in GENERIC and add a no in front of it: nodevice and nooptions.
Fun fact: you can 'group' devices together but you cannot do that for options.
Note: I'm well aware that this approach is also explained in the FreeBSD handbook but only as an extra tip and seems like an afterthought (in my opinion anyway). That's why I featured this so prominently despite of this item being somewhat of a copy.
#7 You can maintain 2 source trees next to each other
Update April 2021: FreeBSD has moved away from Subversion in favor of Git, which means that it no longer uses multiple repositories for the source tree. Therefor this section is now fully redundant. I am going to leave it in for everyone to see, but will scratch it out to emphasize on the fact that this is no longer supported.
For the record: I don't recommend doing this but I have done so in the past and it worked perfectly.
So... you are on FreeBSD 10.4 and have the source code installed in /usr/src. You're ready to upgrade to 11.2 but also want to keep a failsave. You could (and should!) make a solid backup so that you can revert your changes (maybe a ZFS snapshot can help?) but to keep a bit more control over the situation you'd rather hold onto the source tree as well so that you could force a downgrade if needed.
Warning: downgrading a FreeBSD system is highly inadvisable and can definitely cause plenty of damage!
What you could do is this: create /usr/src11 and then use that to store your new source tree. Rename your old tree to /usr/src10 and create a symlink which points to the right tree (to prevent accidents if something is hardcoded to using /usr/src). So:
Once you build your system you'll notice that you now have /usr/obj/src11 which contains your newly build system, right next to the previous /usr/obj/src. You can continue to install your system as usual, and afterwards you'll even notice mention of src11 when using
Keep well in mind that this is somewhat 'abusing' the system and to my knowledge it's not officially supported. So be careful and don't blindly rely on this one.
#8 Don't bother building jails
Jails are awesome, they allow you to maintain your own 'mini environment' within your main host environment. The jail(8) manual page suggests building them from source and the handbook suggests to use some archives from an official installation medium.
Building a jail from the source tree can be fun, and I can definitely recommend it as a learning experience or simply for experimentation. But if you merely want to set up a jail to get some work done then I suggest following none of the official suggestions:
Isn't FreeBSD cool?
Of course this does assume that you have an Internet connection and it might also be helpful to use a local mirror instead. Still, this is how I always build new jails: I stream the binaries directly from the FTP server.
Fun fact: this is probably going to become even easier (and more cool) in the near future when we get a full roll out of PkgBase. I base myself on assumptions mind you but still educated ones. You are aware that pkg-add(8) supports the --relocate option, right? This allows you to install a package without placing its files under the default /usr/local location but somewhere else. I can well imagine that this could become a thing for base packages as well.
#9 Take your time and RTFM
Honestly: I re-read chapter 25 of the FreeBSD handbook and chapters 9 and 5 of the developers handbook multiple times before I took my first shot at building my own customized system and I still wasn't fully sure about my options. But as a result I really knew what to expect out of the whole thing.
It can be an overwhelming experience, especially if you take this on for the first time, but always keep in mind that this whole thing is well documented in every step of the way. All you need to do is actually read and follow all that 'fine' documentation.
For example: it might seem like a good idea to blindly copy all the appealing options you find in src.conf(5) and dumping those into /etc/src.conf but that's a seriously bad idea. Especially if it's explained that a certain option is already a default on certain platforms. Overly complicating your setup can only result in more possible causes for problems during building.
And as already mentioned before: you might be tempted to remove everything which you don't need (or want), but are you sure that the system itself also doesn't really need certain components later?
But most of all: take your time.
Fun fact: There's a reason why I suggested to also keep a copy of the GENERIC configuration:
Not only does this show me the steps I took when upgrading my system in the past (you can see that I skipped 10.4), I can always trace these back if I need to. But most of all: using diff makes it really easy to search for any specific changes in the config file(s) during an upgrade. Config files which I won't lose whenever I replace my source tree.
Finally: I can also highly recommend to put such a config directory under version control, either using Git (personal preference) or Subversion (which would also provide some very solid advantages). But I'm going to address that topic in an upcoming post.
Thanks for reading!
Editorial
FreeBSD is my all time favorite Unix(-like) operating system. It is both a means as well as a goal for me at times, meaning that it's both simply a tool to get things done (like writing this article right now) but it's also often a goal in itself; trying to make things 'better' or while upgrading the system. I actually take great joy out of maintaining and working with the source tree and updating my personal server is always a bit special because of that. So I figured, why not share?
Now, some of you will probably already know about varies items on my list; keep in mind that when I write things like these I try to cater to the whole community; both veteran and less experienced users alike.
#1 Why not use freebsd-update?
freebsd-update is the de-facto tool to help you keep your system up to date, and it does a pretty good job too. So why bother with all the hassle that a source tree (and its maintenance) brings you? Whats that, the source gives you more control? Really now... ever heard of /etc/freebsd-update.conf (see also freebsd-update.conf(5))?
See, if you don't want certain parts to get updates then you can simply remove those and then tell freebsd-update to ignore them. You can use Components to specify exactly which parts you want to include in your update (or upgrade) and if you need more control you can also specify IgnorePaths or IDSIgnorePaths (the latter uses regular expressions) and make sure that only the part(s) you care about get included.
Of course this option isn't perfect. For example the Components which make up a FreeBSD environment are basically world (userland), kernel (guess
) and src (the source code in /usr/src). But you can also 'sub divide', for example: src/sys or world/games (this used to be a thing anyway). And there are several problems here...First: the Components section isn't very well documented. As I said: world/games used to be a thing, but I doubt it is now. I always figured that the components were based on the actual archives which you use to install FreeBSD: base.txz, doc.txz, kernel.txz, ports.txz, src.txz and tests.txz. I seem to recall that games.txz used to be included, but it also created a theoretical problem with world/games. I mean; games was a separate archive yet also part of the base system?
The second problem is that you can only include components but not exclude them. I can't for example say: "upgrade everything except the source code". With three components this obviously isn't a big issue, but what about all those sub-components like world/games, world/tests, etc.?
And finally: are you sure that you can leave some things out? If other components rely on the one you're excluding then that could create quite a bit of a problem because you'll only notice after you upgraded (and thus somewhat whacked) your system.
Still... if you really want to customize your system to such extensive levels then that might be a good indication that you're ready to use the source
But otherwise freebsd-update is a very good way to maintain your system.Always remember: FreeBSD isn't like Linux where it somehow became the norm to build your own kernel. Definitely don't expect better performance or such just because you build your system vs. installing a pre-build version.
#2 Lock it up!
Did you know that you can easily build a FreeBSD system from a readonly source tree? It's simple: all you need is a dedicated filesystem and then mounting that read-only. This can be a little tricky with UFS (determining how much space to reserve) but if you're using ZFS then this should be a no-brainer, especially if you're using a server which hosts plenty of other services.
# zfs create -o mountpoint=/usr/src -o compression=on zroot/src, followed by: # git clone https://git.freebsd.org/src.git /usr/src and then: # zfs set readonly=on zroot/src, done. You will only need to unlock it during the next upgrade.For UFS this would boil down to:
# mount -ur /usr/src to lock things and # mount -uw /usr/src to unlock.Why go through all that trouble? Well, easy: to prevent accidents and/or attacks of course. What better way to attack a system than to try and sneak in some code of your own? And before we start arguing about having to know C, the way the code was structured and of course it being somewhat unlikely that an attacker just so happens to know about an exploit to abuse: what about all those install scripts? Would you really notice if I were to change 00555 into 04555? Or if I were to add a few extra "kits" of my own to install? I don't think so...
Lock up your tests too!
As of FreeBSD 11 we have /usr/tests which contains a large collection of Kyua scripts which can help you test all sorts of aspects of your FreeBSD system. Now, of course, these tests don't necessarily focus on intrusion but more so on "system sanity" as I like to call it, but why take the risk? This is why I also maintain zroot/src/tests which is only changed to a read-write state during a system upgrade (which is mostly done in single user mode anyway).
#3 Read the right documentation!
You can customize your FreeBSD installation by adding options which either include or exclude certain aspects of the base system into /etc/src.conf. So it makes sense to read src.conf(5) prior to upgrading, right? Well, maybe not. If you simply run
vi /etc/src.conf on one console and man src.conf on the other then you probably won't get the results you expect. Because why assume that your current installation knows about all the (changed?) options of the version you're about to install?Instead try this:
cd /usr/src ; man -M ./share/man src.conf, this will present you with the src.conf(5) manualpage for the system you're about to install, thus you can be sure that it will contain all the relevant options.#4 Just because you can remove parts doesn't also mean that you should!
Now, this obviously depends on the parts you're going to exclude. For example if your system doesn't have any floppy drives (which one do?) then it makes sense to use WITHOUT_FLOPPY=. And if you're certain that you'll be using the source tree from here on then WITHOUT_FREEBSD_UPDATE= would also make sense. Options like these don't have that much impact.
But don't start assuming things. For example... if you're using OpenSSL (or LibreSSL) from the ports collection then don't simply assume that you won't be needing the one from the base system (WITHOUT_OPENSSL=) because what about all those base services which rely on crypto engines? Not including OpenSSL also results in Kerberos and OpenSSH getting removed and it'll have even more consequences.
There's also the issue of some changes which results will only become apparent after the installation itself. While it might seem like a good idea to remove Clang from the base system in favor of those in the ports you probably end up having a hard time during your next source build.
#5 Makefile makes for a great "cheat sheet"
Forgot a specific required step to build your base system? Easy:
less /usr/src/Makefile, it's all in there. For me this is a classic example which separates hobby projects from more professional ones ("Enterprise"). And if you like tinkering then be sure to study the (massive) lists of build targets as well.Fun fact: even though it isn't officially supported at the time of writing the packages target already exists in 11.2-P4 and even already does something (but doesn't fully work).
#6 Don't 'pollute' your sourcetree!
This should be somewhat obvious but I'm definitely not too proud to admit that I also fell victim to this one myself
So the official documentation says that a good way to start a new kernel configuration is to copy GENERIC and then customize your new file, I quote:
Code:
# cd /usr/src/sys/amd64/conf
# cp GENERIC MYKERNELBut another major problem is that this methodology would 'separate' your workflow from the official one. What if... a new required feature got added to the system and you kept relying on your 'config copy' made several years ago?
So do this instead:
Code:
# mkdir /root/kernel
# cp /usr/src/sys/amd64/GENERIC /root/kernel/generic.11.2
# touch /root/kernel/conf && ln -s /root/kernel/conf /usr/src/sys/amd64/MYCONF
# echo KERNCONF=MYCONF >> /etc/make.conf
(mount /usr/src readonly again)Then in your config file I strongly suggest using this approach instead of blindly copying GENERIC and editing that one:
Code:
include GENERIC
ident YOURKERNELNAME
<your options & settings here>Fun fact: you can 'group' devices together but you cannot do that for options.
Note: I'm well aware that this approach is also explained in the FreeBSD handbook but only as an extra tip and seems like an afterthought (in my opinion anyway). That's why I featured this so prominently despite of this item being somewhat of a copy.
#7 You can maintain 2 source trees next to each other
Update April 2021: FreeBSD has moved away from Subversion in favor of Git, which means that it no longer uses multiple repositories for the source tree. Therefor this section is now fully redundant. I am going to leave it in for everyone to see, but will scratch it out to emphasize on the fact that this is no longer supported.
For the record: I don't recommend doing this but I have done so in the past and it worked perfectly.
Warning: downgrading a FreeBSD system is highly inadvisable and can definitely cause plenty of damage!
What you could do is this: create /usr/src11 and then use that to store your new source tree. Rename your old tree to /usr/src10 and create a symlink which points to the right tree (to prevent accidents if something is hardcoded to using /usr/src). So:
ln /usr/src11 /usr/src.Once you build your system you'll notice that you now have /usr/obj/src11 which contains your newly build system, right next to the previous /usr/obj/src. You can continue to install your system as usual, and afterwards you'll even notice mention of src11 when using
uname -v.Keep well in mind that this is somewhat 'abusing' the system and to my knowledge it's not officially supported. So be careful and don't blindly rely on this one.
#8 Don't bother building jails
Jails are awesome, they allow you to maintain your own 'mini environment' within your main host environment. The jail(8) manual page suggests building them from source and the handbook suggests to use some archives from an official installation medium.
Building a jail from the source tree can be fun, and I can definitely recommend it as a learning experience or simply for experimentation. But if you merely want to set up a jail to get some work done then I suggest following none of the official suggestions:
Code:
$ mkdir jail && cd jail
$ fetch -o - ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.2-RELEASE/base.txz | tar xvf -Of course this does assume that you have an Internet connection and it might also be helpful to use a local mirror instead. Still, this is how I always build new jails: I stream the binaries directly from the FTP server.
Fun fact: this is probably going to become even easier (and more cool) in the near future when we get a full roll out of PkgBase. I base myself on assumptions mind you but still educated ones. You are aware that pkg-add(8) supports the --relocate option, right? This allows you to install a package without placing its files under the default /usr/local location but somewhere else. I can well imagine that this could become a thing for base packages as well.
#9 Take your time and RTFM
Honestly: I re-read chapter 25 of the FreeBSD handbook and chapters 9 and 5 of the developers handbook multiple times before I took my first shot at building my own customized system and I still wasn't fully sure about my options. But as a result I really knew what to expect out of the whole thing.
It can be an overwhelming experience, especially if you take this on for the first time, but always keep in mind that this whole thing is well documented in every step of the way. All you need to do is actually read and follow all that 'fine' documentation.
For example: it might seem like a good idea to blindly copy all the appealing options you find in src.conf(5) and dumping those into /etc/src.conf but that's a seriously bad idea. Especially if it's explained that a certain option is already a default on certain platforms. Overly complicating your setup can only result in more possible causes for problems during building.
And as already mentioned before: you might be tempted to remove everything which you don't need (or want), but are you sure that the system itself also doesn't really need certain components later?
But most of all: take your time.
Fun fact: There's a reason why I suggested to also keep a copy of the GENERIC configuration:
Code:
peter@zefiris:/root/kernels $ ls
GENERIC.10.3 GENERIC.11.1 GENERIC.11.2 breve.conf zefiris.conf
peter@zefiris:/root/kernels $ diff GENERIC.11.1 GENERIC.11.2
19c19
< # $FreeBSD: releng/11.1/sys/amd64/conf/GENERIC 318763 2017-05-24 00:00:55Z jhb $
---
> # $FreeBSD: releng/11.2/sys/amd64/conf/GENERIC 333417 2018-05-09 16:14:12Z sbruno $
31a32
> options IPSEC_SUPPORT # Allow kldload of ipsec and tcpmd5
131a133
> device ocs_fc # Emulex FC adapters
155a158
> device smartpqi # Microsemi smartpqi driver
peter@zefiris:/root/kernels $Finally: I can also highly recommend to put such a config directory under version control, either using Git (personal preference) or Subversion (which would also provide some very solid advantages). But I'm going to address that topic in an upcoming post.
Thanks for reading!
Last edited: