(TL;DR: At the bottom are instructions on my suggestion for fixing it.)
Ah, what probably happens is that the disk numbering changes. Your FreeBSD root disk itself is present and happy, but when you plug the other two disks in, it isn't ada0p2 any more, it's probably ada1p2 or ada2p2. Same problem often occurs in Linux: When there is a single (SCSI or SATA) disk, it's /dev/sda; you plug in another disk or two, and it becomes /dev/sdb or /dev/sdc, and the boot loader can't find the root file system any longer. Oops. The stupidity here is obvious: Operating systems and file systems are dumb (I'm allowed to say that, I get my paycheck for developing such stuff, so I'm part of the problem). The traditional Unix way of mounting file systems is: Let some early hardware discovery enumerate the disks it finds (for example with ada0/ada1/ada2, ... or sda/sdb/sdc), and then write the hardware-dependent disk address into the
/etc/fstab file; that information is guaranteed to become wrong when disks are moved.
(Side remark: You understand the way partitions are numbered? Partition ada0p2 is the 2nd partition on ada0; the Linux equivalent would be sda2. When the disk ada0 moves to for example ada1, then that partition moves with it to ada1p2.)
To understand the theory behind this, one also has to consider the boot loader. When the hardware comes up, the BIOS in the machine also enumerates all disk drives, and tries to load the OS from one of them. How did it manage to find the right one, if they had been renumbered due to plugging things in or out? Most likely, the BIOS stores (in non-volatile RAM) the hardware identity of the disk it boots from, and finds it even when other hardware is plugged in. If you want to have some fun and see the hardware identity of your boot disk, try
camcontrol identify ada0; here's the output from one of my disks at home:
Code:
device model Hitachi HDS5C3030ALA630
firmware revision MEAOAA10
serial number MJ0351YNG9RZ6A
WWN 5000cca228c46d95
In particular the WWN (= world wide name) is a guaranteed unique identifier for this particular disk, guaranteed to not change during the life of the disk, and guaranteed to not be shared with any other disk on this planet (it contains encoded the manufacturer of the disk).
So how to fix this in theory? One option would be: In
/etc/fstab, instead of the disk's (variable) device name ada0p2, store the hardware identity of the disk, for example "wwn=5000cca228c46d95/partition=2". Unfortunately, that is not implemented. Another suggestion would be: Tell the file system to write a descriptor on the disk itself, which is unique, and which says "boot me, I'm the root disk", and then modify the boot loader code to search all disks for this descriptor. Unfortunately, that solution is very complex, because the descriptor (like all data on disks) is unreliable, and we would need multiple copies, and code to deal with corrupted, damaged, out-of-date, and malicious copies, and would need to be able to recognize when some disks are missing. Very hard to do.
Fortunately, FreeBSD has a simpler version of this which works well enough for single-disk file systems. On modern disks, the partition table at the beginning of the disk (the thing that defines what and where the partition ada0p2 is) can also store explicit names for partitions, and then you can put that explicit name into
/etc/fstab. All you need to do is this: Pick sensible but unique names for your file system partitions. For example, on my machine "house", they are called "house_root", "house_var", and so on. Then, to give partition ada0p2 the name "foo_root", you issue this command:
gpart modify -i 2 -l foo_root ada0. After explicitly naming all your partitions, check with
gpart show -l ada0 that the partition names match your expectations. Then you go and edit
/etc/fstab, and replace the explicit partition device names (like ada0p2) with your explicit names. From now on, mounting them at boot time will work, even when the devices themselves move around.
