Opinions requested: FreeBSD on AWS EC2 vs Amazon Linux etc

Rafal Lukawiecki

Rafal Lukawiecki

As a relative newcomer to FreeBSD, but something of a lapsed veteran of the Unix/System V era, I am curious about returning to my Unix roots by running FreeBSD in the Amazon and/or Azure clouds. My company online training business will be upgrading soon, and we have an option, one that arises only once every few years, to change the underlying OS for our cloud-based web and app servers. We are not ready to go the Docker way yet, so a good OS is a must. So far, we have happily run PHP-based Drupal on CentOS on EC2 for over 6 years, and we have benchmarked AWS Linux satisfactorily with a view towards this upcoming upgrade.

However, I see several reasons why FreeBSD would be an attractive option: it has a quality ports system; it is a little more ahead of AWS Linux packaging (ex. support for newer OpenSSL versions); naturally, it is more vendor-neutral and so more portable between the cloud providers; and above all, it allows me to connect to this community, and its active developers on the mail lists. At the same time, being able to contribute, and to offer some financial and human support to the Foundation, would also provide me with the pleasure of knowing that my company and I have been able to help others in some small way.

I realise there has been a couple of years of FreeBSD releases of AWS AMIs labelled under the name of Colin Percival, so I would hope that by now there was some community experience with it. I would appreciate if anyone who has run FreeBSD on AWS and/or Azure could share their opinion about their experience, especially in terms of overall performance (especially vs Amazon Linux or others) primarily as a web server (Apache or Nginx), reliability (hangs requiring reboots, other major issues), security and overall integration woes. If you read this and you want to suggest this is all a great or a bad idea, please let me know.

Thank you for sharing your thoughts.
 
I was testing FreeBSD last year on AWS. I don't recall any problems. In my understanding Colin Percival is the sole developer of FreeBSD AWM image due to the needs of his own company (Tarsnap). He paid for all the development out of his pocket. Many XEN drivers are still being ported from OpenBSD. The image is quite massive (but FreeBSD has 3 times more lines of code (close to 9 millions) than Open (about 2.5 millions) so it might be expected. For the record Colin Percival super intelligent (PhD in pure mathematics) and extremely nice person. I have a very high opinion of Xen hypervisor used by Amazon. I think it is the most mature open source hypervisor technology.

Azure port of FreeBSD was paid by Microsoft. I think I meet their developers last year in Ottawa at BSDCan. They paid for OpenBSD port too. I have not tried Azure yet but my boss was pushing me to try. I am tired of trying new stuff.


To be frank with you my feeling is that you are not going to have problems with either one. The biggest thing is contracting and price which is extremely complicated in both cases. Also your experience with particular underlining platform (I have 0 experience with Microsoft product while running Xen Dom0 on Alpine Linux at home).

I am very conservative and I tend to stick to the devils I know. If you have many years of experience using CentOS I would stay with it. FreeBSD has nothing to brag about when it comes to SSL. When and if BearSSL gets done and becomes the part of FreeBSD core than maybe. Right now FreeBSD is shipped with OpenSSL and compiling things against LibreSSL is non trivial. So it is basically the same OpenSSL code you get on CentOS.
 
We have several FreeBSD hosts running on AWS. Note that working on AWS is a little different (compared to a more traditional VPS provider). It is assumed the instances are easily deleted and recreated. There's no console option for example. So if you run into problems there's no way to login via a console and fix issues. So you're going to have to destroy the instance and bring up a new one. It's therefor imperative to never save any data on the instances themselves.
 
Many thanks for your kind opinions. I have been in touch with Colin, and indeed, he has been very helpful.

Oko said:
If you have many years of experience using CentOS I would stay with it. FreeBSD has nothing to brag about when it comes to SSL. When and if BearSSL gets done and becomes the part of FreeBSD core than maybe. Right now FreeBSD is shipped with OpenSSL and compiling things against LibreSSL is non trivial. So it is basically the same OpenSSL code you get on CentOS.
Click to expand...

Indeed, I agree with not changing unless needed. However, I dislike the haphazard nature of how much of Linux userland has been put together. Managing all the bits, with their inconsistencies, taxes the time. I also have to rely on what distro maintainers want me to have in it. Currently, Amazon Linux does not come with OpenSSL newer than 1.0.1k-fips 8 Jan 2015 and our apps need 1.0.2k. Their engineers will not let us know when they are upgrading. CentOS is stuck with old PHP... Sure, I can manage it all manually, but then I might just as well do it with FreeBSD where it is more elegantly put together—I feel (and hope). In general, having a more elegant base OS, where things as small as config files or start-up scripts are in their right places has much value to me and to my company.

Also: the community. I have been several times in a situation where getting a sensible reply from a developer of a Linux/Drupal/AWS component is not possible or leads to expensive frustration. My month-old experience of FreeBSD suggests that its approachable, knowledgeable, and above all, more mature community may be one of its key qualities that I would appreciate for a long enough time to come.

SirDice said:
We have several FreeBSD hosts running on AWS. Note that working on AWS is a little different (compared to a more traditional VPS provider). It is assumed the instances are easily deleted and recreated. There's no console option for example. So if you run into problems there's no way to login via a console and fix issues. So you're going to have to destroy the instance and bring up a new one. It's therefor imperative to never save any data on the instances themselves.
Click to expand...

Thank you for pointing this out. I would not be too concerned as we have run our CentOS/Amazon Linux on AWS for 6 years now, and we have plenty of own scripts to tear down and rebuild (small) fleets of machines as needed. It was harder in 2010 when we had to build our own AMI images, things are pretty automatic now. I have similar experience with Azure, but we prefer to use it for running cloud-based analytics, ML and databases. One way or another, I do not fear the cloud anymore. :) I am more fishing for insight on the stability and overall longer-term experience of FreeBSD in the cloud. From the last few days of testing I can already see that FreeBSD takes longer to launch a new machine or reboot than our existing machines in AWS, however, it also makes some of the instance-launch config cleaner and simpler. I hope to have more experience in due course, and I will gladly share it when I have gathered enough reliable experience.
 
SirDice said:
[...]So if you run into problems there's no way to login via a console and fix issues. So you're going to have to destroy the instance and bring up a new one. It's therefor imperative to never save any data on the instances themselves.
Click to expand...
PS. If you we are ever in that situation, what we do, is to attach the ESB volume of the affected instance to another, healthy one, at which point you can just mount it inside it. That gives you access to data and you can perform simple fixes. However, it is generally easier to bring up a new instance if you can have all of its config applied at launch, with the data, as you suggested, elsewhere: a database or NFS-like EFS.
 
Rafal Lukawiecki said:
However, it is generally easier to bring up a new instance if you can have all of its config applied at launch, with the data, as you suggested, elsewhere: a database or NFS-like EFS.
Click to expand...
Yes, for us it's not a problem. Our data is stored in MySQL databases. The webservers are managed and set up with Puppet. There's nothing worth saving on them so it's easier to just throw away the instance and spawn a new one. The load-balancers at the front will cope with any number of backend webservers. As long as I keep at least one web server alive nobody will notice. The whole way of working with cloud based services was new to me and not being able to fallback to fixing things via a console was the biggest hurdle I had to cope with.
 
Rafal Lukawiecki said:
have you managed to install the AWS CloudWatch Logs agent (awslogs) on your FreeBSD instances?
Click to expand...
Never played with it. We have two (rather old) servers, CPUs are slow (compared to everything else) but there's a bunch of storage on them and running sysutils/syslog-ng to receive all syslogs. Fairly old school way of logging :)

The AWS setup is mainly for disaster recovery, most of the stuff is off by default but can be switched into high gear when needed. Our day-to-day stuff is all physical hardware housed at a hosting company.
 
Adiel Ribeiro said:
Hello. I've published hardened FreeBSD solutions that you can use on AWS.
First is an AMI available on AWS Market Place, here: https://aws.amazon.com/marketplace/pp/B08T6Q1R5Q
Another is the "code" used to made the FreeBSD AMI available on GitHub, here: https://github.com/Adiel-Ribeiro/FreeBSD/tree/master
Click to expand...
I have checked out your image quickly to see if you do better in disk IO throughput than the official AMI, and I could not do a direct raw dd write to /dev/nvd2 (freshly attached EBS disk). But it gave me "operation not permitted" (yes I am root of course). I suppose this has to do with all that "hardening". But write test to the existing file system showed me I get 170 MB/s in and that's not great. So I conclude that you have not really been able to improve the bad I/O performance of FreeBSD on AWS.

I'm curious what you think about the poor IO performance of FreeBSD on AWS (it's by several factors if not an order of magnitude, not just percents, slower than stock Amazon Linux). I think unless FreeBSD people break through this, the role of FreeBSD in the market of serious computing will quickly fade as AWS is picking up more and more market share.

forums.freebsd.org

FreeBSD was once "the power to server" but in an AWS world we have fallen way, waaay behind and there seems no interest to fix it!

I probably shouldn't but, oh well.... Why would this even be an issue in the first place? If FreeBSD doesn't meet your demands: move on! There are apparently other options available, pick those. Done! FreeBSD isn't the all powerful #1 solution to everything + world hunger. SO... if something...
forums.freebsd.org forums.freebsd.org
 
You must log in or register to reply here.
Back
Top