Compiling a kernel - a walkthrough

It's easy to become fixated on only having what's necessary. But it's a lot easier to get a working custom kernel by just removing unnecessary devices. For example, look at SCSI controllers and Ethernet cards in GENERIC. They're pretty easy to identify, and most people won't have more than a few models of either. All the others can be removed. Repeat the process, paring away devices and options until it breaks and you can't fix it* or boredom or diminishing returns set in.

* For bicycle repairs, I express this as "tighten it until you pull the threads out, then back it off a quarter-turn."
 
Just wondering if this is it... this dmesg excerpt:
Code: 
ada1 at ahcich4 bus 0 target 0 lun 0
Would that mean I need to enable the following?
Code: 
device ahc # AHA2940 and onboard AIC7xxx devices
 
wblock said:
It's easy to become fixated on only having what's necessary. But it's a lot easier to get a working custom kernel by just removing unnecessary devices. For example, look at SCSI controllers and Ethernet cards in GENERIC. They're pretty easy to identify, and most people won't have more than a few models of either. All the others can be removed. Repeat the process, paring away devices and options until it breaks and you can't fix it* or boredom or diminishing returns set in.
Click to expand...
Yes indeed. I've removed the bulk of the entries and it is working still. Keeping on going.
* For bicycle repairs, I express this as "tighten it until you pull the threads out, then back it off a quarter-turn."
Click to expand...
Lol.
 
carlton_draught said:
Just wondering if this is it... this dmesg excerpt:
Code: 
ada1 at ahcich4 bus 0 target 0 lun 0
Would that mean I need to enable the following?
Code: 
device ahc # AHA2940 and onboard AIC7xxx devices
Click to expand...

There is a man page for every device. Look at man ahc(4)() and see if that helps you.
 
carlton_draught said:
Just wondering if this is it... this dmesg excerpt:
Code: 
ada1 at ahcich4 bus 0 target 0 lun 0
Would that mean I need to enable the following?
Code: 
device ahc # AHA2940 and onboard AIC7xxx devices
Click to expand...

No, adaX on ahci is SATA. ahci channel 4, I suppose.
 
What I implied but didn't outright say in #26 was "But it's a lot easier to get a working custom kernel by starting with GENERIC and just removing unnecessary devices."

In other words, start with a known-good kernel config file and mercilessly hack bits out of it rather than start with an empty file.
 
wblock said:
What I implied but didn't outright say in #26 was "But it's a lot easier to get a working custom kernel by starting with GENERIC and just removing unnecessary devices."

In other words, start with a known-good kernel config file and mercilessly hack bits out of it rather than start with an empty file.
Click to expand...
Oh definitely. I never start from scratch with anything if I have a choice. Much easier to start with something that works (i.e. GENERIC) than try and put stuff in you think will work. You would never know what the extra addition would be that makes it work, but it's always the last thing you took away that broke it.
 
UNIXgod said:
There is a man page for every device. Look at man ahc(4)() and see if that helps you.
Click to expand...
Thanks. I realized after I read the man page that I had the following in /boot/loader.conf
Code: 
ahci_load="YES"
which is (duh) the alternative to compiling it in the kernel. I think I prefer to do it that way (in loader.conf), because I like things to still work with GENERIC, if possible.

91 active lines of working kernel so far. So far so good.

Edit:
81 lines now. Difference between previous WORKSTATION config:
Code: 
compat_freebsd32
hwpmc_hooks
ataraid
pass
ses
psm
kbdmux
splash
I think we've hit the point of diminishing returns.
 
looking at the list I know that splash can be removed. It's just a console based screensaver. basically it's McKusick's BSD Daemon picture moving around the screen.

kbdmux Is only necessary if you use more than one keyboard( strangely I love this hack), I have a laptop which I use a happy hacking keyboard and it works well there and I also have a system I watch vids on one side and program on the other. where I move the screen 180 degrees and the system has on keyboard on either side based on how I'm using the machine.

If you have a PS/2 mouse port leave psm in

I have no idea what ses is but I haven't owned any real scsi gear in a long time so I can't comment on this one.

if you plan on cd and dvd burning the handbook suggests pass, cd, scbus and ata with atapicam_load="YES" in loader.conf

ataraid is "fakeraid" if your motherboard supports it. It can be removed if you never plan on enabling it.

I'm gonna be gone for a couple days. I am interested in in the success of your project. It sounds like your pretty much there.
 
UNIXgod said:
I'm gonna be gone for a couple days. I am interested in in the success of your project. It sounds like your pretty much there.
Click to expand...
Thanks for the help. I'll have a look at it a bit later and maybe remove a few more lines, posting the final kernel config here. As you say, I'm pretty much there.

My immediate goal is to get gnome2 to finish compiling (just did!), so that I can live in gnome again rather than doing everything from 1 tiny console. Easier to cut and paste and stuff etc to forums. From there I'll hark back to my original goals:
  • Compile a custom kernel, and learn how to update that (and world I guess) so that I can do it regularly and easily, so as to have a properly secure and reliable system. Still need to document what is involved with that. And also figure out and document my own practice for how to cope with updates and new additions to GENERIC, so that I can keep the holy trinity of kernel, world and ports fully patched/updated, including on when to do each or how to keep abreast of when.
  • Get system able to boot from 2nd disk, which is apparently a problem with 8.1-RELEASE and ZFS root mirror setups. Test it.
  • Fix my broken gnome (no gnome panel). DONE!
  • Figure out how to properly backup and restore the updated root mirror (which probably involves figuring out the minimum set of directories/files to backup and the order in which to do so).
 
Without going into much details about what options/devices in the kernel to leave/remove I would like to point out just few things:

1. There are times, when you MUST update the kernel AND world at the same time. These are documented in the /usr/src/UPDATING file. There are not many such cases recently, for in the past changes in the networking code for example would leave your without network if you update only the kernel. Make it a habit to check that file each time you update sources.

2. There is not much point to snapshot the root if you only are installing new kernel. The installkernel target will move the old kernel and modules to kernel.old. Of course, if you are going to iterate a lot, it's good idea to have safe working root anyway. But if things break, you will have to boot off some other media anyway :)

3. Take a look at nextboot. Using nextboot is advisable for experiments like yours, where you add/remove random parts from the kernel.

4. There is no point in having separate place for kernel config files and having symlinks in /sys/{arch}/conf -- you can leave the config files in there -- nothing is going to overwrite those. Or, if it does, it will overwrite the files via the symlinks as well -- little value.

5. As you have multicore CPU, run (you said you have 4 core CPU):

# make -j8 buildworld
# make -j8 buildkernel

do not use -j for other targets!

6. It is always smart to do buildworld after changing your sources. It's also smart to do cleanworld before that :)

There are of course zillion ways to shoot yourself in the leg. What I described is actually what is documented as 'The FreeBSD way' and you could imagine, for good reason.

PS:
7. As you are playing with the kernel, you may wish to discover hardware that the GENERIC kernel didn't care about, but your system has present. The easiest and funny way is to load kernel modules, like

# kldload ichsmb

If this produces kernel output about new device(s) on the console, you have just hit one :)
Study the man page and if you like/want it, add the respective load statement in /boot/loader.conf. Or, add the device to your kernel config.

Have fun!
 
danbi said:
Without going into much details about what options/devices in the kernel to leave/remove I would like to point out just few things:

1. There are times, when you MUST update the kernel AND world at the same time. These are documented in the /usr/src/UPDATING file. There are not many such cases recently, for in the past changes in the networking code for example would leave your without network if you update only the kernel. Make it a habit to check that file each time you update sources.
Click to expand...
Thanks. Even though it upsets my desire (and sensibilities?) that everything possible should be automated, I am learning to read and not skim or scan both /usr/src/UPDATING and /usr/ports/UPDATING.

2. There is not much point to snapshot the root if you only are installing new kernel. The installkernel target will move the old kernel and modules to kernel.old. Of course, if you are going to iterate a lot, it's good idea to have safe working root anyway. But if things break, you will have to boot off some other media anyway :)
Click to expand...
While that is true, at least I have written out procedures for that so that in the worst case I can get back what I once had. If you can do that, you feel more freedom to experiment (whether the fears are warranted or not), which is ultimately good IMO. And snapshots are easily destroyed, so no harm done.

3. Take a look at nextboot. Using nextboot is advisable for experiments like yours, where you add/remove random parts from the kernel.
Click to expand...
Sweet! Just so people know, this is included in the system and not a port. Just type
# man nextboot

4. There is no point in having separate place for kernel config files and having symlinks in /sys/{arch}/conf -- you can leave the config files in there -- nothing is going to overwrite those. Or, if it does, it will overwrite the files via the symlinks as well -- little value.
Click to expand...
I don't think anything is going to overwrite those files other than the admin. However, you might delete /usr/src in a fit of anger, which is why I did what is done in the first tip in that section of the handbook.
5. As you have multicore CPU, run (you said you have 4 core CPU):

# make -j8 buildworld
# make -j8 buildkernel

do not use -j for other targets!
Click to expand...
Hey, thanks for that!

6. It is always smart to do buildworld after changing your sources. It's also smart to do cleanworld before that :)
Click to expand...
Again, thanks for the tip.

PS:
7. As you are playing with the kernel, you may wish to discover hardware that the GENERIC kernel didn't care about, but your system has present. The easiest and funny way is to load kernel modules, like

# kldload ichsmb

If this produces kernel output about new device(s) on the console, you have just hit one :)
Study the man page and if you like/want it, add the respective load statement in /boot/loader.conf. Or, add the device to your kernel config.
Click to expand...
This is cool. I'm just playing around with a way to automate this discovery of devices. Hopefully I can post a script when done.

I've got LINT pared away to just the names of the modules, to be iterated through in a loop. However, I'm trying to do the following in order to see whether there is any output, and so far seeing nothing in some things I know should be there.

e.g.
# kldunload snd_hda
# kldload snd_hda
No output.
# kldunload speaker
# kldload speaker
Nothing. Any comments?
 
Following on from that last one:
Code: 
#!/bin/sh
# By scrolling through your terminal output, you should see which kernel module
# load detects some hardware on your system you didn't know about/enable.
ARCH=amd64
# Note that LINT includes everything that is NOT in GENERIC
LINT_DEVICES=$(cat /sys/$ARCH/conf/LINT | grep "^device" | cut -f 3 | sort | uniq)
for device in $LINT_DEVICES
do
  echo "$device"
  kldload "$device"
  sleep 1
  echo
done
Note that I did NOT find anything I did not have already enabled. Not a great surprise, as I have a server/workstation mobo and I'm basically using what I want to already. This may be of use to someone.
 
UNIXgod said:
looking at the list I know that splash can be removed. It's just a console based screensaver. basically it's McKusick's BSD Daemon picture moving around the screen.

kbdmux Is only necessary if you use more than one keyboard( strangely I love this hack), I have a laptop which I use a happy hacking keyboard and it works well there and I also have a system I watch vids on one side and program on the other. where I move the screen 180 degrees and the system has on keyboard on either side based on how I'm using the machine.

If you have a PS/2 mouse port leave psm in

I have no idea what ses is but I haven't owned any real scsi gear in a long time so I can't comment on this one.

if you plan on cd and dvd burning the handbook suggests pass, cd, scbus and ata with atapicam_load="YES" in loader.conf

ataraid is "fakeraid" if your motherboard supports it. It can be removed if you never plan on enabling it.

I'm gonna be gone for a couple days. I am interested in in the success of your project. It sounds like your pretty much there.
Click to expand...
Ok, I'm commenting out ataraid and splash, the others can stay. Here is what I'm doing now:
# cp /root/kernels/WORKSTATION /root/kernels/WORKSTATION.works_20101218
# make -j8 buildkernel KERNCONF=WORKSTATION
And while we are at it:
# make cleanworld
# make -j8 buildworld
Next time, boot the system using the new kernel without checking to see if it exists.
# nextboot -f -k kernel
Reboot.
# shutdown -r now
 
Well, that worked. I've pared it down as far as I want to. Now to tick the rest of the items off the todo list. For those interested, here is the final configuration file, WORKSTATION.
 
Considering that I've had a several month gap in being able to work on anything FreeBSD, I've consequently forgotten a lot of this stuff, or at least, it has grown vague. I've spent a few hours reading through this thread, the handbook, various other posts on the forum etc, to try and understand where I was and where I need to go.

So now I'm in a position where I want to upgrade my system to 8.2-RELEASE, and I have a custom kernel (which IIRC was built primarily because I wanted to include some patches that fixed some ZFS issues). I thought I might continue documenting that process in this thread, for my own benefit and the benefit of others hopefully.

Upgrading your system with a custom kernel to a new RELEASE - a walkthrough

One thing I needed to make clear (correct me if I'm wrong) was that you can have a custom kernel and still follow RELEASE (rather than STABLE or CURRENT). This seems to be backed up here. i.e. - the fact that you can upgrade from source indicates that this should be possible.

Such a system that tracks RELEASE will still receive security fixes. e.g.
While it is true that security fixes also go into the FreeBSD-STABLE branch, you do not need to track FreeBSD-STABLE to do this. Every security advisory for FreeBSD explains how to fix the problem for the releases it affects [1], and tracking an entire development branch just for security reasons is likely to bring in a lot of unwanted changes as well.
Click to expand...

So it looks like I will need to first sync my sources, and then make/rebuild world, so to speak. I'll research that and then begin updating this thread.
 
Make a backup

Before we do anything, it's a good idea to backup your system. So I'm doing it here (using ZFS pools). I'll first make recursive snapshots on zroot and storage, so I can rollback if necessary.

# zfs snapshot -r zroot@before_updating20110318
# zfs snapshot -r storage@before_updating20110318

I'm backing up my storage pool to a HDD in an e-SATA HDD dock. I backup my SSD root mirror zroot to storage regularly, so all I need to do is backup what's on storage.

Insert the HDD, which has had a pool created on it called backup06, and two filesystems, backup06/filesystems and backup06/pools. Turn it on, and the HDD is detected by FreeBSD 8.1.
# zpool import backup06

Now here's where I show off the script in my sig (which I really need to get included into ports, but one thing at a time). To backup everything including snapshots and properties, and force the destination to have 2 copies to guard against bad sectors, and lzjb compression in order to reduce the size, all while allowing restore of the original properties easily, all I do is the following (whether it is the first time, second time, or nth time):
# zxfer -dFkPvb -o copies=2,compression=lzjb -N storage/home backup06/filesystems
# zxfer -dFkPvb -o copies=2 -N storage/distfiles backup06/filesystems
# zxfer -dFkPvb -o copies=2 -N storage/packages backup06/filesystems
# zxfer -dFkPvb -o copies=2,compression=lzjb -R storage/zrootbackup backup06/pools

Note that I could probably just do:
# zxfer -dFkPvb -o copies=2,compression=lzjb -R storage backup06/pools
Not sure why I'm not, to be honest. I will figure that out when I finish my guide to installing, backing up and restoring a FreeBSD root mirror system.

Export the backup pool, and turn the HDD dock off.
# zpool export backup06

Syncing my sources

Before we can build from source, we need to get that source from somewhere. I'll check the handbook. Note that I'm somewhat following on from dcbdbis's post as well.

I am going to be using csup, which is a faster reimplementation of net/cvsup. We are using it because
  1. it's efficient in terms of only getting the files we need compared to anonymous CVS
  2. using email, and PGP to prevent forged CTM deltas seems complicated. I also can't really find anything on it in the forums, googling
    Code: 
    site:forums.freebsd.org ctm
    comes up with nothing.
  3. unlike net/cvsup, there are no dependencies.
  4. If we use a host mirror, there is no load on FreeBSD core infrastructure.

The only potential downside I can see is that AFAIK using a local mirror means that you trust their checksums, which would have higher probability of being compromised than using the original FreeBSD site. It would be nice to get the checksums from there and fetch the other files locally, not sure if that is possible. Would appreciate any enlightenment about this from someone who knows more than I do.

So, let's follow the cvsup link (and we'll just use "csup" instead in commands).

We are going to start with the standard supfile, and modify if necessary.

# cp /usr/share/examples/cvsup/standard-supfile /root/supfile_20110318

Now edit /root/supfile_20110318.

Lines changed:
Code: 
*default release=cvs tag=RELENG_8_2
*default host={local [url=http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/cvsup.html#CVSUP-MIRRORS]mirror[/URL]}

Now we run csup. No need to specify -g as csup doesn't appear to have a gui.

# csup /root/supfile_20110318

It took about 10 minutes or so to complete. Do it again, just for kicks, and you'll find that it should take about 5 seconds, if that. So we are finished syncing source. Now it's time to "rebuild".
 
Note that these are my notes to myself, I'll print them out and try and out, updating this post with what I should have done if I make a mistake. Edit: I've now finished, and fleshed out any bits that I had trouble with. In particular, the following post covers the mergemaster in detail, which was the trickiest for a newbie.

Rebuilding world

  1. Check /usr/src/UPDATING.
    # vim /usr/src/UPDATING
    Nothing of particular note. Note that the mailing list for RELEASE appears not to exist, so checking UPDATING should be fine.
  2. Check /etc/make.conf. Ok, did some googling, this thread was good. Seems the consensus is to not add anything to /etc/make.conf. (Perhaps the handbook needs updating, as they suggest a typical user wants to copy CFLAGS and NO_PROFILE lines.) So left as is.
  3. Get to single user mode. I'm in gnome, so I need to get out of that first, and then get to single user mode.
    # /usr/local/etc/rc.d/gdm stop
    # shutdown now
  4. Remove object files for a clean build.
    # cd /usr/obj
    # chflags -R noschg *
    # rm -rf *
  5. Make buildworld (using script to get a record of what we have done).
    # cd /usr/src
    # script /root/make_buildworld_20110319.out
    (Note, we use -j8 to speed up the build process, I have a 4 core (8 with HT) CPU, so we use -j8.)
    # make -j8 buildworld
    This took 17 minutes.
  6. Make and install the GENERIC kernel (don't worry, we will make our custom kernel later). Defaults to generic, so no need to put buildkernel KERNCONF=GENERIC.
    # make -j8 buildkernel - this took 9 minutes.
    # make installkernel
  7. Make and install the custom kernel, in my case, WORKSTATION.
    # make -j8 buildkernel KERNCONF=WORKSTATION - this took 6 minutes
    # make installkernel KERNCONF=WORKSTATION
  8. Backup /etc. Not really necessary, as we will have a snapshot to look at that if we need to. However, just to be needlessly ultra-anal,
    # cp -Rp /etc /etc.old
  9. Reboot the system, and at the boot prompt, select the “single user” option (or option 6, "boot -s". The system will then boot single user. At the shell prompt you should then run:
    # fsck -p
    # mount -u /
    # mount -a -t ufs
    # swapon -a
    Note we also need to mount all our ZFS file systems. (The handbook needs updating here.)
    # zfs mount -a
  10. Mergemaster. Read here for how to use. Search for "24.7.11.1".
    # mergemaster -p
  11. Install the system binaries.
    # cd /usr/src
    # make installworld - takes 1 minute
  12. Mergemaster. Read here for how to use. Search for "24.7.11.1". Hmmm. This will be a good thing for the next post to cover. I see dcdbis has already trodden down this path before... Edit: Whoops. I really, really, really should use the options -Ui
    # mergemaster -Ui
  13. Reboot.
    # shutdown -r now
  14. Should now have an updated world and kernel.

Addendum
  1. Check that the upgrade looks ok, and that you are running your custom kernel.
    # uname -iv
  2. Upgrade pools and ZFS file systems.
    # zpool upgrade -a
    # zfs upgrade -a
    Strangely, the system crashed immediately after/during the zfs upgrade, but after a reboot it says everything is upgraded. A bit scary.
  3. Upgrade bootcode on the root mirror. Used dmesg to determine device nodes, then issued the command that was given after the zpool upgrade. (not in my history unfortunately, due to crash).
  4. Taken from here. Clean up and delete obsolete files, directories and libraries.
    # cd /usr/src
    # make check-old
    # yes|make delete-old
 
Dealing with mergemaster -p
Ok, the first up is /etc/group.

I select "m" for merge, then "l" several times until done. Then "i" to install the merged file.

/etc/master.passwd
"m", "l" until finished, then "i".

Then it comes up and says:
Code: 
You installed a new master.passwd file, so make sure that you run /usr/sbin/pwd_mkdb -p /etc/master.passwd to rebuild your password files. Would you like it to run now?
Hit "y".

Ok, that is it. It compares make variables from /etc/make.conf with /usr/src/share/examples/etc/make.conf and bails back to the command prompt.

I guess it's time to go to the next step... (previous post)

Dealing with mergemaster
Ok, now we have installed world, we are looking at...
./.cshrc
"r", "i", and, uh "l", for create a link.

.profile
"r", "l" (the "l" is for my changed path), "i", and "l"

COPYRIGHT
"i" (it's new, just changes the dates basically, so install the new one)

./boot/device.hints
"i" (it's new, just changes the dates basically, so install the new one)

./etc/amd.map
"i"

./etc/apmd.conf
"i"

./etc/auth.conf
"i"

... more files I know I have not touched, just hit "i"

./etc/crontab
"i" (thought this was the file generated by # crontab -e, must be wrong?

... more files I know I have not touched, just hit "i"

./etc/rc.conf
m,r,i (I think this must be src/etc/defaults/rc.conf)


... more files I know I have not touched, just hit "i"

./etc/group
r,l,l,i

... more files I know I have not touched, just hit "i"

./etc/master.passwd
m,r,l,l,i


... more files I know I have not touched, just hit "i"


./etc/shells
m,r,l,i


... more files I know I have not touched, just hit "i", rebuild aliases database by typing "y", and several other things.

Finished.
 
After a release, the version control strings have changed in many of those files.
# mergemaster -Ui
helps with that.
 
You must log in or register to reply here.
Back
Top