Home made load testing. Under VMWare FreeBSD outperforms Centos7. I am missing something.

Dear all,

I am a hobbyist using FreeBSD on the desktop for my daily computing needs and for two websites I work on during my free time.
I have been reading a few things on virtualization and have found jails to be very useful for my particular needs. However, working in a big environment (I work as a UNIX operator in a bank) has let me thinking on commercial solutions.
Solaris is fading away and with it zones/containers, which are derived from jails, too. Since I have a few pc's at home I decided to check some different solutions aside of jails. The first into my home made test is VMWare. Before you read any longer, things to consider:

  • The purpose of this is merely discovery.
  • The main goal was/is load the system as much as possible with VMWare playing with 3-4 vm's, all different OS.
  • The secondary goal is to see how each different system behaves in this scenario.
  • The hardware is an HP Proliant ML110 G7 with 14GB of ram.
  • The VMWare version is 6.5 community edition.
  • The setup is at the time of writing:
    • A vm with FreeBSD 11.1 with one core assigned and 3GB of ram.
    • Another vm with CentOS 7 with one core and 3GB of ram.
    • A third vm with Solaris 11.3 with one core and 3 gigs of ram as well.
    • The BSD and the Linux vm's "play" a wordpress webiste with real data.
    • The FreeBSD vm uses stock system configuration plus vmware tools and MySQL 5.7, PHP 5.6, as well as Apache 2.4. The site is configured with the MPM Event module so php-fpm is mandatory.
    • The CentOS vm uses similar config but since the stock software offering is much older I set it up to use: Apache 2.4, MySQL 5.7 and PHP 7.1. Again Apache is configured to use the MPM Event module.
    • The Solaris machine will be in charge of another website using a different CMS, in this case a Magnolia installation (it's made on Java).
  • The load testing software in use is siege and ab (apache benchmark).
  • The test came from a jailed environment on an spare laptop I connect from.
  • The networking gear is a home router. No more, no less.
  • Mind my knowledge is little. Veeeery little.
I have just fired the FreeBSD and the CentOS vm's with everything configured and ready to rock. The Solaris install is still lacking a working website on the Magnolia CMS (I am still learning how to publish something with that).

My first impression is although I see the advantages of this kind of software (I mean VMWare) the vm's run much slower than on bare metal with not so powerful hardware. My second impression is I have made some mistake somewhere (or maybe I lack some knowledge) because FreeBSD outperforms CentOS 7 by a lot when benchmarking the wordpress sites.

I have used siege as follows:

[INDENT]siege -c150 -d10 -t3M URL[/INDENT]

And the results have been:

For FreeBSD:

Transactions: 10716 hits
Availability: 96.51 %
Elapsed time: 179.48 secs
Data transferred: 163.70 MB
Response time: 1.91 secs
Transaction rate: 59.71 trans/sec
Throughput: 0.91 MB/sec
Concurrency: 113.97
Successful transactions: 10716
Failed transactions: 388
Longest transaction: 74.65
Shortest transaction: 0.00

For CentOS 7:

Transactions: 854 hits
Availability: 99.53 %
Elapsed time: 179.91 secs
Data transferred: 19.40 MB
Response time: 1.78 secs
Transaction rate: 4.75 trans/sec
Throughput: 0.11 MB/sec
Concurrency: 8.44
Successful transactions: 854
Failed transactions: 4
Longest transaction: 14.98
Shortest transaction: 0.00

The FreeBSD vm only reached about 700MB of ram usage while testing and opened three more httpd processes to handle the request. On the other hand the CentOS vm went up to 2.8 GB of ram use and swapped a few MB. It opened four "extra" httpd processes to handle the load. Plus after the test the Linux machine was not responding to commands and new ssh connections failed having to reboot the vm.

I know the results are irrelevant and anyone waiting 70 sc for a webpage to load will be very far away (although 1.91 sc is not that bad). At least as far as the 388 requesters who didn't receive their page. However I cannot explain by any means the performance difference between the two vm's.

Again, both systems play default OS configurations, both websites are equal, testing the same URL (the home page which loads a ton of things), although using different PHP versions. And I imagine the CentOS version being newer should perform at least a bit better than the one in the BSD vm.

Any idea of what have I missed? I've checked everything, from the VM's configuration in VMWare, to the Apache's modules, etc. Everything. And aside different PHP versions they are both on the same page. I even checked if the command was the same. It was.

PS: Sorry for my long post.
Dear all,

I am back. I have found some time to do some more load tests. The first problem has been I was testing a load that couldn't be handled. It was too much. The siege command I was launching was:

siege -c150 -d10 -t3M URL

150 concurrent users, hitting the url in ranges in between 1sc and 10sc, for three minutes.

Modifying this load I have extracted the best performance each system can handle without almost no request loss. That said the results are interesting. At least from my point of view.

The Centos7 system has been a bit tricky. I had to reduce the concurrency of users quite considerably to gain significant results. One thing has to be said. If the system swaps you're done. The performance slows down noticeably and some times it's difficult to obtain a responsive system after the test.

The best result for the Centos7 system has been under these circumstances:

siege -c18 -d10 -t3M URL

which is the equivalent of saying 18 concurrent users, hitting the url in ranges in between 1sc and 10sc, for three minutes.

Transactions: 8119 hits
Availability: 100.00 %
Elapsed time: 179.98 secs
Data transferred: 126.45 MB
Response time: 0.33 secs
Transaction rate: 45.11 trans/sec
Throughput: 0.70 MB/sec
Concurrency: 15.06
Successful transactions: 8020
Failed transactions: 0
Longest transaction: 12.68
Shortest transaction: 0.01

Now on the FreeBSD 11.1 side. The load has also been dramatically reduced to obtain sane results. But not as much as on the Centos7 system.

The best result for FreeBSD 11.1 has been under this load:

siege -c47 -d10 -t3M URL

which is like saying 47 concurrent users, hitting the url in ranges in between 1sc and 10sc, for three minutes.

Transactions: 12063 hits
Availability: 100.00 %
Elapsed time: 179.94 secs
Data transferred: 187.27 MB
Response time: 0.63 secs
Transaction rate: 67.04 trans/sec
Throughput: 1.04 MB/sec
Concurrency: 42.25
Successful transactions: 11911
Failed transactions: 0
Longest transaction: 20.88
Shortest transaction: 0.01

For both systems increasing the user concurrency by just one user has an impact. On the FreeBSD side there is a slight loss in transactions although availability remains intact. On Linux the system swaps and it starts having an odd behavior.

As I said in the beginning, my knowledge is small and I just "play" with this systems. However I think it can be said benchmarking anything is not an easy thing and many results can be interpreted in different ways. And some answers to some questions lead to some other questions, so you end up with more questions than before.

At first sight 12.000 hits look better than 8.100. Concurrency is at least some significant value in my eyes. 42 concurrent users are a better value than 15. However the response time doubles in the FreeBSD system, meaning the Linux system has been twice as fast on serving the results. Which is weird because at the end it has served like 35-40% less content. Yes, there you have it. Less concurrent users has let the system to work. The data transfer result confirms this as well, at least in my opinion.

All this said I still have doubts on why this has been the case and how can I, or anyone, interpret these kind of tests. I don't believe a Centos system is as slower as FreeBSD as the results have shown. And all this has made me think of... why on Earth does the Centos system get a much better value reducing the users concurrency? What does people do, just throw more hardware to high concurrency sites? Ok, obviously people tune systems, apache's, databases, all that. I ask myself what do people who does not have the time or the knowledge and get default system configurations.

If anyone can point out any mistakes or address anything I have said or done which could improve the results of both systems or just share their view I am more than willing to read it.
I found the big mistake in my CentOS configuration.

The php-fpm module was not correctly configured.

The results are now more sane and there is very little difference in using both systems. Almost the same number of transactions, concurrency, etc.

There is a difference though in memory usage.

CentOS still climbs up quite high, 2.4 GB and beyond. FreeBSD still gets near 800 MB but doesn't get further.

I will continue on this home made test and will post results soon.

Again, if anyone wants to comment feel free to do it. I am waiting for someone more versed on these topics to shed some light or thoughts.