I want to create small fast clone-able servers to spawn many worker systems that respond to HTTP servlet requests and to SQS messages. Yes, yes, I know that Amazon has this auto-scaling group stuff, and I will use that of course, but I don't have to use Amazon Linux for that. I want a small boot disk of the minimal 1 GB size, and then mount /usr from EFS (NFSv4).
Why do I want that? Because I want to scale up to possibly 100s of worker systems, each being a t2.micro instance, brought up on demand. Possibly in hibernating state to come up quickly, while not costing anything. Each such server needs one boot disk. But if I can do with 1 GB instead of 10 GB (the minimum size of the FreeBSD 12.0 AMI) I can save on disk space. The bulk of the /usr file system would be mounted through NFS. I can make the bootable root even smaller than 1 GB (500 MB) and read-only, so I can quickly clone that, and with the other 500 MB available for writable stuff.
My life has drifted away from FreeBSD since most cloud hosting providers push some CentOS Linux version on me. So I got used to that. But the issue with Amazon Linux / CentOS is that they have done away with the multi-stage boot design of original UNIX (or may be only Ultrix and BSD only ever really had it, I don't know). In this design
I have been able to push this idea up to a point. It is pretty easy do a decent boot and mount NFS without the /usr file system tree present if I make limiting assumptions, especially I need to configure the local IPv4 address statically in rc.conf (ifconfig_xn0="inet 172.x.y.z/24". And then still the boot is not complete, because after the /usr system has finally been mounted through NFS, it doesn't seem to be processing the AWS specific initializations. Unfortunately it seems that for an AWS instance to be connectable through the public IP address I must have the IPv6 interfaces configured, with dual-dhcp, and all of that depends on stuff installed in /usr/local and also a lot in /usr. These rc scripts seem to be making promiscuous use of /usr tools, such as /usr/bin/find and /usr/bin/sed, etc. That is not good to do a minimal bootable configuration.
I wonder if there is interest in reorganizing the AWS AMI a bit and push the essentials down into the minimal root file system. Also would like to know how it works for a new box to read configuration parameters from EC2 config resource. May be if I could just read the private IPv4 and IPv6 address from that configuration resource, then I wouldn't even need DHCP at all.
To whom it may concern, I give here my recipe for what I have:
mount the EFS through NFSv4:
mount_nfs -o nfsv4,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-fXXXXXXX.efs.RRRRRR.amazonaws.com:/ /mnt
and check:
ls /mnt
it's empty at first.
Now fill it:
cd /mnt
(cd /usr ; tar cf - .) |dd bs=10m status=progress |tar xf -
Then I have added the /etc/fstab entry:
/dev/gpt/rootfs / ufs rw 1 1
/dev/gpt/usrfs /usr ufs rw 1 1
fs-fXXXXXX.efs.RRRRRR.amazonaws.com:/ /mnt nfs rw,nfsv4,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport 0 0
and the server into the /etc/hosts
172.X.Y.Z fs-fXXXXXX fs-fXXXXXX.efs.RRRRRR.amazonaws.com
But then this never works because the network doesn't come up.
I try
netwait_enable=YES
netwait_if="xn0"
netwait_ip="172.A.B.C"
noting that the EFS server can never be pinged. But all that is of no use. Until I give:
ifconfig_xn0="inet 172.31.58.248 netmask 0xfffff000"
then it will boot, but I can only connect through that local IPv4 address, not through the global IP address. I am pretty sure AWS maps the global IPv4 address to the server via IPv6.
With the system basically up, I could now manually add the /usr/local/etc/rc.d work or may be just set up the IPv6 in rc.local.
But wouldn't it be much nicer if we moved the very essential stuff into the rootfs partition leaving all that Python stuff behind that the AWS CLI needs. Anyone interested in fleshing this out with me?
Why do I want that? Because I want to scale up to possibly 100s of worker systems, each being a t2.micro instance, brought up on demand. Possibly in hibernating state to come up quickly, while not costing anything. Each such server needs one boot disk. But if I can do with 1 GB instead of 10 GB (the minimum size of the FreeBSD 12.0 AMI) I can save on disk space. The bulk of the /usr file system would be mounted through NFS. I can make the bootable root even smaller than 1 GB (500 MB) and read-only, so I can quickly clone that, and with the other 500 MB available for writable stuff.
My life has drifted away from FreeBSD since most cloud hosting providers push some CentOS Linux version on me. So I got used to that. But the issue with Amazon Linux / CentOS is that they have done away with the multi-stage boot design of original UNIX (or may be only Ultrix and BSD only ever really had it, I don't know). In this design
- /sbin, /bin, /lib and of course /etc were on the root disk
- /usr could be mounted later during the boot process,
- and we intend to mount /usr from NFS
I have been able to push this idea up to a point. It is pretty easy do a decent boot and mount NFS without the /usr file system tree present if I make limiting assumptions, especially I need to configure the local IPv4 address statically in rc.conf (ifconfig_xn0="inet 172.x.y.z/24". And then still the boot is not complete, because after the /usr system has finally been mounted through NFS, it doesn't seem to be processing the AWS specific initializations. Unfortunately it seems that for an AWS instance to be connectable through the public IP address I must have the IPv6 interfaces configured, with dual-dhcp, and all of that depends on stuff installed in /usr/local and also a lot in /usr. These rc scripts seem to be making promiscuous use of /usr tools, such as /usr/bin/find and /usr/bin/sed, etc. That is not good to do a minimal bootable configuration.
I wonder if there is interest in reorganizing the AWS AMI a bit and push the essentials down into the minimal root file system. Also would like to know how it works for a new box to read configuration parameters from EC2 config resource. May be if I could just read the private IPv4 and IPv6 address from that configuration resource, then I wouldn't even need DHCP at all.
To whom it may concern, I give here my recipe for what I have:
- create an initial instance from the latest official FreeBSD AMI (12.0 in my case)
- initially already add 2 more disks:
- one 1 GB (/dev/sdb) will show as /dev/xdb1
- one 4 GB (/dev/sdc) will show as /dev/xbd2
- boot
- enter with ec2-user and then to root
- su - to become root
- obviously in the end a root password needs to be set
- FreeBSD doesn't rely so heavily on sudo
- geom disk list shows all disk devices
- ada0
- xbd1
- xbd2
- gpart create -s GPT xbd1 to create a GPT label on the xbd1 device
- gpart bootcode -b /boot/pmbr xbd1 to write the master boot record
- gpart add -b 40 -s 472 -t freebsd-boot -l bootfs0 xbd1 to create the 2nd stage boot partition
- NOTICE the use of the -l option to set a label (name) for the partition, here bootfs0
- we are recreating the setup of the AMI boot disk gpart show -l ada0
- bootfs
- rootfs
- we must then use any of these partitions through its named handle
- /dev/gpt/bootfs - currently booted system
- /dev/gpt/rootfs - currently booted system
- /dev/gpt/bootfs0 - new boot partition
- /dev/gpt/rootfs0 - new root partition (see below)
- /dev/gpt/usrfs - new usr file system (see below)
- we are recreating the setup of the AMI boot disk gpart show -l ada0
- NOTICE the use of the -l option to set a label (name) for the partition, here bootfs0
- copy the boot code onto the boot partition
- gpart bootcode -p /boot/gptboot -i 1 xbd1 is how its done, but I didn't know, so
- dd if=/dev/gpt/bootfs of=/dev/gpt/bootfs0 status=progress is what I did.
- If you follow man gpart, don't set up swap partition
- gpart add -t freebsd-ufs -l rootfs0 xbd1 - to set up /dev/gpt/rootfs0 using the entire rest of the disk
- newfs -U /dev/gpt/rootfs0 - to create a file system on the new partition
- gpart create -s GPT xbd2 - label for the usr disk
- gpart add -t freebsd-ufs -l usrfs xbd2 - to set up /dev/gpt/usrfs using the entire disk
- newfs -U /dev/gpt/usrfs - to create a file system on the new partition
- mount /dev/gpt/rootfs0 /mnt - to mount what will be the new minimal root disk
- echo * - list all items on /
- cd /mnt - and go to our new root
- (cd / ; tar cf - boot bin sbin ... all items from / except for mnt and usr) |tar xvf - - copy everything from / to /mnt, except /usr and /mnt itself
- ls -l / . - compare
- cp the dot-files that you didn't see initially, .snap, .profile, .cshrc
- mkdir /mnt /usr - make the 2 directories not copied
- compare once more that you have everything
- mount /dev/gpt/usrfs /mnt/usr - to mount what will be the /usr file system
- (cd / ; tar cf - usr) |tar xvf - - copy the entire /usr tree on the new file system
- now we are almost ready
- umount /mnt/usr - already unmount the new /usr file system
- vi /mnt/etc/fstab - add /usr mount usrfs /usr ufs 1 1
- umount /mnt - unmount the new / root.
- gpart modify -i 2 -l rootfs xbd1 - change label from rootfs0 to rootfs as it will be the only disk attached with that label on next boot
- gpart modify -i 0 -l bootfs xbd1 - change label from bootfs0 to bootfs as it will be the only disk attached with that label on next boot
- not sure where it is used, but we want to mirror exactly the set up of the AMI, only using 2 different disks.
- shutdown -p now - shut down the system and power down
- set up the disk configuration with the two new disks now
- detach all 3 disks from EC2 instance
- attach the 1 GB root disk to EC2 instance as /dev/sda1 - will become ada0
- attach the 4 GB usr disk to EC2 instance as /dev/sdb - will become xbd1
- start EC2 instance
- see it boot!
mount the EFS through NFSv4:
mount_nfs -o nfsv4,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-fXXXXXXX.efs.RRRRRR.amazonaws.com:/ /mnt
and check:
ls /mnt
it's empty at first.
Now fill it:
cd /mnt
(cd /usr ; tar cf - .) |dd bs=10m status=progress |tar xf -
Then I have added the /etc/fstab entry:
/dev/gpt/rootfs / ufs rw 1 1
/dev/gpt/usrfs /usr ufs rw 1 1
fs-fXXXXXX.efs.RRRRRR.amazonaws.com:/ /mnt nfs rw,nfsv4,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport 0 0
and the server into the /etc/hosts
172.X.Y.Z fs-fXXXXXX fs-fXXXXXX.efs.RRRRRR.amazonaws.com
But then this never works because the network doesn't come up.
I try
netwait_enable=YES
netwait_if="xn0"
netwait_ip="172.A.B.C"
noting that the EFS server can never be pinged. But all that is of no use. Until I give:
ifconfig_xn0="inet 172.31.58.248 netmask 0xfffff000"
then it will boot, but I can only connect through that local IPv4 address, not through the global IP address. I am pretty sure AWS maps the global IPv4 address to the server via IPv6.
With the system basically up, I could now manually add the /usr/local/etc/rc.d work or may be just set up the IPv6 in rc.local.
But wouldn't it be much nicer if we moved the very essential stuff into the rootfs partition leaving all that Python stuff behind that the AWS CLI needs. Anyone interested in fleshing this out with me?
Last edited: