After using FreeBSD irregularly for almost 10 years, I've started using it quite a bit more recently. Over the past couple of days I've put together a FreeBSD 9.1 amd64 image I'll be using for various tasks on VMWare ESXi.
The image should satisfy the usual requirements of an enterprise: SNMP, remote syslog-ing, NTP, VMWare Tools, UTF-8 character set, etc. If you see anything crucial I forgot, please comment below.
While trying to be as generic as possible for many possible use cases, the main goal of the setup, however, is security. Thus,
There are two reasons I'm putting this together
I try to give credit where it's due. If I forgot to mention somebody's work, please let me know.
Without further ado, let's get our hands dirty.
Basic installation
Credit: http://namor.userpage.fu-berlin.de/howto_fbsd9_encrypted_ufs.html
During the installation we deselect all distribution sets and when asked about partitioning we select "Shell". This is what we are trying to achieve
and this is how we do it
Notes:
Shortcomings and residual risk:
The image should satisfy the usual requirements of an enterprise: SNMP, remote syslog-ing, NTP, VMWare Tools, UTF-8 character set, etc. If you see anything crucial I forgot, please comment below.
While trying to be as generic as possible for many possible use cases, the main goal of the setup, however, is security. Thus,
- the image is fully encrypted using geli() (except the bootloader) and requests one single passphrase at boot, the /var/log partition is separate and, of course, encrypted,
- the pf() firewall is set to block log all by default for incoming and outgoing connections,
- even with firewall turned off, the only two open ports are TCP/22 and UPD/161 for authenticated SNMPv3 users. (This is quite different from the default installation of some services that listen, often under root, on TCP/25 (sendmail), UDP/514 (syslogd), UDP/123 (ntpd).) sshd is the single service listening on network under the root account.
- When the installation is finished, the system is fully up-to-date and ready to be updated using the standard tools: freebsd-update(), portmaster(8).
There are two reasons I'm putting this together
- I've spent quite a bit of time creating the image and came across several non-trivial situations. I might soon forget how I solved them and/or in what exact order. FreeBSD Forums post allows me to document everything (with a very nice formatting) for future reference to me and to anyone that might find it useful.
- I'm hoping that the readers will suggest improvements, correct errors and help the image get better over time. I'd appreciate your feedback!
I try to give credit where it's due. If I forgot to mention somebody's work, please let me know.
Without further ado, let's get our hands dirty.
Basic installation
Credit: http://namor.userpage.fu-berlin.de/howto_fbsd9_encrypted_ufs.html
During the installation we deselect all distribution sets and when asked about partitioning we select "Shell". This is what we are trying to achieve
Code:
# gpart show -l
=> 34 41942973 da0 GPT (20G)
34 6 - free - (3.0k)
40 128 1 (null) (64k)
168 1048576 2 boot (512M)
1048744 2097152 3 swap (1.0G)
3145896 3145728 4 varlog (1.5G)
6291624 35651376 5 root (17G)
41943000 7 - free - (3.5k)
# cat /etc/fstab
/dev/gpt/boot /mnt/boot_unencrypted ufs rw,noatime 1 1
/dev/gpt/swap.eli none swap sw 0 0
/dev/da0p5.eli / ufs rw,noatime 1 1
/dev/gpt/varlog.eli /var/log ufs rw,noatime 1 1and this is how we do it
Code:
#CREATE BOOT BLOCK
gpart destroy -F da0 #it is OK to receive an error here, if drive is new
gpart create -s gpt da0
gpart add -t freebsd-boot -s 64k -a 4k da0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 da0
#CREATE AND FORMAT BOOT PARTITION
gpart add -t freebsd-ufs -s 512m -a 4k -l boot da0
newfs -U gpt/boot
#CREATE PARTITION FOR ENCRYPTED SWAP
gpart add -t freebsd-swap -s 1g -a 4k -l swap da0
#CREATE PARTITIONS FOR ENCRYPTED NONROOT FILESYSTEMS
gpart add -t freebsd-ufs -s 1536m -a 4k -l varlog da0
#CREATE AND INIT ENCRYPTED ROOT VOLUME
gpart add -t freebsd-ufs -a 4k -l root da0
geli init -b -l 256 -e AES-XTS gpt/root
# insert complex passphrase, [color="Red"]if it is lost, all setup is lost[/color]
geli attach gpt/root
#FORMAT ENCRYPTED ROOT VOLUME
newfs -U gpt/root.eli
#MOUNT ROOT AND BOOT FILESYSTEMS
mount /dev/gpt/root.eli /mnt
mkdir -p /mnt/mnt/boot_unencrypted
mount /dev/gpt/boot /mnt/mnt/boot_unencrypted
cd /mnt
ln -s boot_unencrypted boot
cd ..
#PREPARE KEYS FOR NONROOT PARTITIONS
mkdir -p /mnt/root/geli_keys
chmod 0400 /mnt/root/geli_keys
head -c 32 /dev/random > /mnt/root/geli_keys/varlog.key
#INIT ENCRYPTED NONROOT PARTITIONS
geli init -P -l 256 -K /mnt/root/geli_keys/varlog.key gpt/varlog
geli attach -p -k /mnt/root/geli_keys/varlog.key gpt/varlog
#FORMAT ENCRYPTED NONROOT VOLUMES
newfs -U gpt/varlog.eli
#MOUNT ENCRYPTED NONROOT VOLUMES
mkdir -p /mnt/var/log
mount /dev/gpt/varlog.eli /mnt/var/log
#CONTINUE INSTALLATION
#POST INSTALLATION
Choose "Shell" when installation finished for "final modifications"
#CONFIG BOOT LOADER
echo >> /boot/loader.conf << __EOF__
aesni_load="YES"
geom_eli_load="YES"
vfs.root.mountfrom="ufs:/dev/da0p5.eli"
__EOF__
#CONFIG FSTAB
echo >> /etc/fstab << __EOF__
/dev/gpt/boot /mnt/boot_unencrypted ufs rw,noatime 1 1
/dev/gpt/swap.eli none swap sw 0 0
/dev/da0p5.eli / ufs rw,noatime 1 1
/dev/gpt/varlog.eli /var/log ufs rw,noatime 1 1
__EOF__
#MAKE SWAP ENCRYPTED
echo >> /etc/rc.conf << __EOF__
geli_swap_flags="-e AES-XTS -l 256 -s 4096 -d"
__EOF__
#MAKE SURE ENCRYPTED NONROOT PARTITIONS ARE AUTOATTACHED
echo >> /etc/rc.conf << __EOF__
geli_gpt_varlog_flags="-p -k /root/geli_keys/varlog.key"
__EOF__Notes:
- For some reason, /dev/da0p5 can not be specified as gpt/boot in /boot/loader.conf nor in /etc/fstab. It seems there's some trouble with the order of the start up scripts.
- swap is encrypted using a random password.
- Hardware acceleration of AES is enabled using the kernel module aesni. It should fail safe even if the CPU doesn't support it.
- The flag "-a 4k" for the gpart add command makes sure the partitions are alligned for 4kB disks. It doesn't hurt even if we don't have them, since 512B * 8 = 4kB
. - We are not setting up any kind of redundancy at this level (e.g. gmirror(), nor zfs()). We're setting the template for the enterprise ESX environment where the virtual harddrives (vmdks) are placed on the datastores on top of highly reliable RAID arrays.
- Also, we're not setting up any kind of backups. We're trying the set everything in such way, that an accidental power failure won't be a problem. Thus, backing up raw VMWare vmkd while the machine is running shouldn't be a problem either. From the security point of view, it is no problem an "untrusted" backup solutions handle our disk image since it is encrypted (except the bootloader) with the complex passphrase known only to us.
Shortcomings and residual risk:
- We're silently assuming nobody messes up our unencrypted bootloader and make us enter the passphrase to such untrusted environment.
- We are again silently assuming the path "from our fingers to the bootloader" is secure. That is, we're assuming there are no emanation receptors :h, hardware keyloggers, software keyloggers, SSL connections broken into, malicious administrators of the ESX hypervisors, etc.
- Well, we are silently assuming many things .
- We could place the bootloader on a thumbdrive and insert it with every boot (again silently assuming noone will overwrite it from the inside before booting). Also, we would have to remember to mount the thumbdrive everytime we run
# freebsd-update installwhich is not that ofen so we'd probably forget. - Ultimately, the "security" is always a trade off. Also, the name of the post doesn't promise a "secure installation" but a "secured installation" .