FreeBSD was once "the power to server" but in an AWS world we have fallen way, waaay behind and there seems no interest to fix it!

gschadow

Member

Reaction score: 11
Messages: 36

I have been with FreeBSD since 386BSD 0.0new and always was going for FreeBSD as my first choice for every server I built. Device drivers being a little later than for Linux was OK. Somehow I muddled through.

But now I feel like a stomach punch. The future of serving is AWS and the like.

There are so many things out there that say FreeBSD on AWS is slow.

Network: https://forums.freebsd.org/threads/aws-ec2-ena-poor-network-performance-low-pps.77093/#post-492744 and some reports from 2016.

But even disk: incredible:
Code:
[root@freebsd ~]# dd if=/dev/zero of=/dev/nvd1 bs=1G status=progress
^C28991029248 bytes (29 GB, 27 GiB) transferred 405.900s, 71 MB/s
28+0 records in
27+0 records out
28991029248 bytes transferred in 420.583038 secs (68930572 bytes/sec)
compared to Amazon Linux:
Code:
[root@linux ~]#  dd if=/dev/zero of=/dev/xvdbt bs=1G status=progress
209379655680 bytes (209 GB) copied, 794.254631 s, 264 MB/s
Maybe I'm doing something wrong? Or is this really so bad?
 
OP
G

gschadow

Member

Reaction score: 11
Messages: 36

Here is another example of just how nobody seems to care any more.


There was some suggestion to add to /boot/loader.conf

hw.nvme.use_nvd=0

and it makes no difference.
 
OP
G

gschadow

Member

Reaction score: 11
Messages: 36

I also verified that the instance type and volume is exactly the same (m5a.large), and I am not running into any CPU or IO burst balance exhaustion issues either. 80 MB/s dd write on FreeBSD 12.2 RELEASE on m5a.large vs. 300 MB/s on Amazon Linux (Linux ip-172-31-54-217.ec2.internal 4.14.106-97.85.amzn2.x86_64).
 
OP
G

gschadow

Member

Reaction score: 11
Messages: 36

I tried FreeBSD 10 RELEASE because someone said performance had degraded, but no, it yielded 61 MiB/s, same thing.
 

Speedy

Well-Known Member

Reaction score: 27
Messages: 312

I've never chosen FreeBSD for speed. I recall in one of predecessors of these forums about 12-15 years ago someone complained their Linux web server is on the limit (they were running some sort of porn site) and tried FreeBSD instead. To their disappointment FreeBSD wasn't able to get even 50% of Linux performance. Damn penguin, they said, and switched back to Gentoo. I wonder if those old forums are still stored somewhere, it would be interesting to read.
 
D

Deleted member 66267

Guest


Could you test with other Linux, e.g: Debian or Ubuntu? IMHO, running Amazon's OS on Amazon's platform and comparing the performance with other OS is a bit unfair.
 

ralphbsz

Son of Beastie

Reaction score: 2,692
Messages: 3,554

A: You understand that Amazon EBS is not just a device? In reality, the cloud providers create fake devices, which underneath use interesting RAID-like techniques to provide backup, snapshots, redundancy for durability and availability, and so on. Accessing those fake devices requires considerable support from the layer that virtualizes them. My educated guess is that someone at Amazon has not gone through the work of optimizing that virtualization layer for the specifics of FreeBSD's NVME driver.

B: If you have a performance problem with Amazon, you should bring it up with Amazon. Rumor has it that they have excellent technical support, and for a small amount of money, you can purchase support contracts with them.

C: Have you tried this with other big providers that support FreeBSD? I know for a fact that FreeBSD instances are available in Microsoft Azure and in Google Cloud.

D: Venting to the FreeBSD forum about something that's mostly under Amazon's control is ... silly, malicious, pointless, I'm sure we can find some other adjectives.
 
OP
G

gschadow

Member

Reaction score: 11
Messages: 36

Thanks for the replies. But I find them strange.

* FreeBSD not for speed .... but for what? Power to serve?
* Amazon EBS, what it is and what it isn't, that's irrelevant since I am running both Linux and FreeBSD on that exact same hardware.

But tine, I have some good news.

I really want this not to be a problem. I am too invested in FreeBSD. Could it be that raw dd to disk is a special case?

After I created a ZFS pool from just that one disk and then do the dd test, I get better results:
Code:
[root@db00 ~]# dd if=/dev/zero of=/dbtank/test bs=1G status=progress
  9663676416 bytes (9664 MB, 9216 MiB) transferred 33.873s, 285 MB/s
But how? Raw writing should be faster!?

I have done a second test with a UFS file system and again dd test to a file is fast. There seems to be an issue when writing to the raw device? Maybe it's a blocksize issue? Not sure, since I gave dd's bs=1G, maybe it assumed some internal fragmentation?

Here is raw once again:

Code:
# dd if=/dev/zero of=/dev/nvd2 bs=1G status=progress
  4294967296 bytes (4295 MB, 4096 MiB) transferred 56.234s, 76 MB/s
  9663676416 bytes (9664 MB, 9216 MiB) transferred 124.146s, 78 MB/s

device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
nvd0           0       0      0.0      0.0     0     0     0     0    0   0
nvd1           0       0      0.0      0.0     0     0     0     0    0   0
nvd2           0     650      0.0  83324.1     0     1     0     1    0  97

dT: 1.008s  w: 1.000s
L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    1    507      0      0    0.0    507  64864    1.7   85.9| nvd2

So the write request per second w/s is really just 650 for some reason dd is weird here, it's not fragmenting the blocks.

But with a UFS file system:

Code:
newfs /dev/nvd0
mount /dev/nvd2 /mnt
dd if=/dev/zero of=/mnt/tet bs=1G status=progress
  9663676416 bytes (9664 MB, 9216 MiB) transferred 26.192s, 369 MB/s
44023414784 bytes (44 GB, 41 GiB) transferred 121.471s, 362 MB/s

device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
nvd0           0       0      0.0      0.0     0     0     0     0    0   0
nvd1           0       0      0.0      0.0     0     0     0     0    0   0
nvd2           3    2741     96.0 350646.0     2    26     0    26   80  91

dT: 1.005s  w: 1.000s
L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
   86   2761      2     64    1.4   2759 352940   27.0   94.0| nvd2

umount /mnt

Here we write with 2741 IOPS which is about the limit that the virtual machine bus hardware allows.

So, I will say that perhaps this is OK. Perhaps only dd is somehow flawed. But for the use that dd gets, to make raw disk copies usually, it's weird that dd would perform so badly.
 

drhowarddrfine

Son of Beastie

Reaction score: 2,731
Messages: 4,645

I've never chosen FreeBSD for speed. To their disappointment FreeBSD wasn't able to get even 50% of Linux performance.
Well, what did you choose because for networking/internet/etc FreeBSD is provably faster and that's a well-known fact.
 

6502

Active Member

Reaction score: 47
Messages: 145

OK, FreeBSD on AWS is slow. But what if we compare FreeBSD on good server (bare metal) vs Amazon Linux on AWS?
 

Beastie7

Aspiring Daemon

Reaction score: 688
Messages: 769

It's amazing how marketing buzzwords sways entire industries. Cloud Computing? You mean my entire infrastructure on your computer with a fancy UI?

I mean, gosh, VPS's are so yesteryear.
 

Snurg

Daemon

Reaction score: 598
Messages: 1,349

I find funny the similarities of "the cloud" with the analog telephone network in Communist Eastern Germany.
Actually it consisted of the 1930s telephone network, and so in the 1980s due to extreme wear and overloading, it was common to sometimes to have to dial several times until one got a line with quality transmission.
It was sometimes completely unpredictable whether and when it would be possible to get a stable connection at all.

On the other hand, it was awesome how much they managed to squeeze out of that meager infrastructure.

Or, take the typical 80A fuse for a 20-flat house, with every flat having an individual fusing of 32A.
This only works well due to the simultaneity factor. If only a small number of flats would draw their full rated current, it would result in a brownout or even blackout for all.

Anyway, so "cloud" in reality means, it allows me to sell you 4 real amperes for the price of 32A 👍
 
OP
G

gschadow

Member

Reaction score: 11
Messages: 36

I love you guys, those are the dismissive "I don't care" and "It's amazon's fault" and "cloud's a buzzword" replies that I totally sympathize with. I'm a conservative old grump too. Haha!

I used to have a rack of DL585s and an MSA whatnot in a cage in a data center. Yeah, been there, done that. FreeBSD all the way.

But then I became a digital nomad. And then my state wanted to charge tax on "business personal property" and it became a nightmare, being 5000 miles on the other side of the planet and have something go "poof". And also, the savings on the data center rent. And then the flexibility of the AWS stuff. Need a TB of disk for an hour to run some report updates? No problem! Costs almost nothing! The awesome capability to scale resources in seconds and pay only for the time I use them, that's total game changer. It really changes the way you design data and transaction processing.

So if Linux is fast on one box and FreeBSD is slow on the same box, I should complain with the box manufacturer? Now your grumpy-old-man is taking the better of you.

Furthermore, it is established the FreeBSD has issues on network too. As for disk IO, my tests show some 20% worse. Perhaps. As long as the output goes to a file system.

That's a puzzling thing more than a show-stopper now with the dd to raw device. There is some technical reason that we cannot write fast through dd to raw disk. I don't know if that is AWS specific or if that is true for bare metal too. I have no bare metal left. My DL585 rack is in a garage collecting dust.
 

Mjölnir

Daemon

Reaction score: 1,530
Messages: 2,114

Here is raw once again:
Code:
# dd if=/dev/zero of=/dev/nvd2 bs=1G status=progress
  4294967296 bytes (4295 MB, 4096 MiB) transferred 56.234s, 76 MB/s
  9663676416 bytes (9664 MB, 9216 MiB) transferred 124.146s, 78 MB/s

device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
nvd0           0       0      0.0      0.0     0     0     0     0    0   0
nvd1           0       0      0.0      0.0     0     0     0     0    0   0
nvd2           0     650      0.0  83324.1     0     1     0     1    0  97

dT: 1.008s  w: 1.000s
L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    1    507      0      0    0.0    507  64864    1.7   85.9| nvd2
Code:
# zfs create -V 8G -o volmode=dev -o compression=off -o checksum=off t450s/RAW
# dd if=/dev/zero of=/dev/zvol/t450s/RAW bs=1G count=8 status=progress
  8589934592 bytes (8590 MB, 8192 MiB) transferred 33.416s, 257 MB/s
8+0 records in
8+0 records out
8589934592 bytes transferred in 36.933352 secs (232579336 bytes/sec)
# zfs destroy t450s/RAW
FreeBSD 12.1-p13 on a 6 year old laptop w/o NVD is faster than AWS with NVD...
 

drhowarddrfine

Son of Beastie

Reaction score: 2,731
Messages: 4,645

gschadow Your individual circumstances do not apply to everyone. Your links or quotes from other forums or posts mean nothing as anyone can do the same for the opposite. Are those Linux tests on a server optimized to the fullest but the FreeBSD tests are non-optimized? And a million questions like that.

These threads pop up here a couple of times a year by people complaining about "Why isn't FreeBSD like...?" and it's against forum rules. (Why is this thread still here?) You will get nowhere about the desktop because FreeBSD is a professional system for professionals and serious computer enthusiasts who are more interested in performance and not pretty blinking lights...ACK!

Now you got me started again. Why is this thread still open?!
 

Speedy

Well-Known Member

Reaction score: 27
Messages: 312

FreeBSD is provably faster and that's a well-known fact.
I don't think this is true. Wall Street wouldn't be running on Linux then, they count microseconds there.
Anyhow, you ask why. It happened in previous century. I forgot my Debian Potato mail server running without upgrading and it got hacked. I fixed that, and then started looking around what else is out there. This is how I found FreeBSD and I fell in love with it. The organization, everything made sense to me, more than Linux. Linux kernel is nowadays like a huge puzzle, at least for those who do not use custom kernel and upgrade it by hand routinely. There is a plethora of options which are performance vs. security and the diff at least for Intel processors is in excess of 20% when you go all-performance or all-security. There is nothing like this in FreeBSD kernel.
I don't want to start some Linux vs. FreeBSD discussion here, in any case I will not participate.
 

wolffnx

Aspiring Daemon

Reaction score: 237
Messages: 679

Now you got me started again. Why is this thread still open?!

The same ask I do.
Maybe because everyone deserve the oportunity to give their opinion
but..the point at this time of the post is lost and everyone know the answer
 
OP
G

gschadow

Member

Reaction score: 11
Messages: 36

Actually, I think you are completely mistaken now. I tried to take it in jest, but now you're getting over the top.

AWS is like a major hardware vendor.
As of October 2020, Canalys reports that the worldwide cloud market grew 33% this quarter to $36.5 billion. AWS has 32% of the market and generated more revenue than the next three largest combined, Azure is at 19% of the market, Google Cloud at 7%, Alibaba Cloud close behind at 6%, and other clouds with 37%.
You're not just big mouthing being defensive. I have probably been longer with FreeBSD than most of you, we should be talking like friends not you taking any criticism like it's the enemy. This is nuts!

We have a MAJOR problem with the AWS performance, and if you have worked on Azure and you think that's better, then that means very little, because AWS has twice the market share. Whatever is going on, has to have a fix. It's not rocket science, but the problem is that there seems to be a culture of complacency around here, where nothing ever gets fixed because people just rant how irrelevant the issue is.

Slowness on AWS has been reported here, not on some other places, for many years (sine 2016 at least) and nobody seems to have any fucks to give, or gets right out nasty when the issue is confronted, that is worse even than whatever shortness of resources might be causing us to be behind on this matter.
 

Mjölnir

Daemon

Reaction score: 1,530
Messages: 2,114

You could write a polite e-mail to the appropiate mailing list, asking kindly for advice on how to debug your issue. With a little bit of DTrace scripting you'll be able to narrow where the problem is & identify a small set of possible root causes.
 
D

Deleted member 66267

Guest


I have read FreeBSD is "the power to serve" but not "the power to server" yet. Please correct me if I'm wrong. Because the two is very different.

If it's simply "the power to serve", then it doesn't imply "best performance" but "able to serve without disruption" IMHO.

BTW, FreeBSD has the best performance after Linux. It's the best performance non-Linux OS. You can find nothing better than it.
 

mark_j

Daemon

Reaction score: 870
Messages: 1,449

Actually, I think you are completely mistaken now. I tried to take it in jest, but now you're getting over the top.

AWS is like a major hardware vendor.

You're not just big mouthing being defensive. I have probably been longer with FreeBSD than most of you, we should be talking like friends not you taking any criticism like it's the enemy. This is nuts!

We have a MAJOR problem with the AWS performance, and if you have worked on Azure and you think that's better, then that means very little, because AWS has twice the market share. Whatever is going on, has to have a fix. It's not rocket science, but the problem is that there seems to be a culture of complacency around here, where nothing ever gets fixed because people just rant how irrelevant the issue is.

Slowness on AWS has been reported here, not on some other places, for many years (sine 2016 at least) and nobody seems to have any fucks to give, or gets right out nasty when the issue is confronted, that is worse even than whatever shortness of resources might be causing us to be behind on this matter.
I understand your grief and, unfortunately, you are approaching this the wrong way.

You are targeting users with your dismay and anger rather than the developers. You need to join the appropriate mailing lists and hassle the developers (and one in particular, I might point you to, but someone else has already, I believe).

Your anger, annoyance, frustration and downright dismissive approach is TOTALLY misplaced here.

This is not feedback, this is an example of venting.
 
Top