FreeBSD was once "the power to server" but in an AWS world we have fallen way, waaay behind and there seems no interest to fix it!

ShelLuser

Son of Beastie

Reaction score: 2,123
Messages: 3,797

I probably shouldn't but, oh well....

Why would this even be an issue in the first place? If FreeBSD doesn't meet your demands: move on! There are apparently other options available, pick those. Done! FreeBSD isn't the all powerful #1 solution to everything + world hunger. SO... if something else fits your needs better: use it!

Want to really make a difference? Then why not stop whining and instead start researching. Into: how things got so bad (personally doubt they did but...), how to improve on things (IMO this post is a solid lead) and then actually start working on things. Instead of posting rants which look like you're attacking the hand that fed you why not turn stuff around and actually try to make a real difference in the problem you perceived?

Yah, it takes more effort than just writing up a forum post, but at least you'll accomplish something.

... unless of course that was never the goal, which is something I kinda suspect to be the issue here.

</vent>
 

ralphbsz

Son of Beastie

Reaction score: 2,516
Messages: 3,377

Amazon EBS, what it is and what it isn't, that's irrelevant since I am running both Linux and FreeBSD on that exact same hardware.
Wrong. You are not running FreeBSD and Linux on hardware. You are running then on a highly complex emulated thing, which is mostly software. I have no idea how many lines of code there are in Amazon EBS (I have no access to Amazon's code base, and if I did, I wouldn't tell you). But knowing in general how complex it is to build such a thing, there are probably a few million lines of code behind EBS. Those lines of code have been designed and tuned. It seems to me that whoever did that tuning didn't worry very much about the case of FreeBSD running on it.

So what is needed is some Amazon engineers (and perhaps a FreeBSD kernel developer, but Amazon might have some of those on their staff) to get together and spent some time fixing it. The first step to that is to open a support ticket with Amazon. If enough people do, the issue might get some attention.

Could it be that raw dd to disk is a special case? ...
After I created a ZFS pool from just that one disk and then do the dd test, I get better results: ...
But how? Raw writing should be faster!? ...
Wrong. You need to learn about file system caches. Writing to a file system is very often MUCH faster than writing to the underlying device.

Finally: You keep harping on "the power to serve". There is much more to building a system than simple speed. Before we take your silly dd raw benchmark result seriously, you need to explain to us exactly how affects your workload, and how you want to use the system. Start by explaining to us the requirements for your system, and why FreeBSD doesn't meet them.
 

drhowarddrfine

Son of Beastie

Reaction score: 2,499
Messages: 4,446

I don't think this is true. Wall Street wouldn't be running on Linux then, they count microseconds there.
It's true and well known. Ask Netflix who use it for video delivery and contribute to that very action. Your Wall Street example is based on their custom applications that don't use standard tcp/ip and http.

In any case, why did they choose Linux? Possibly for the same reason Google initially chose Linux. Because it was what they were used to! Sergey and whats-his-name were asked that question long ago--which I can't find cause it's lost in the PR releases instead--where they said they chose Linux because that's what they used in school.
 

Mjölnir

Daemon

Reaction score: 1,519
Messages: 2,114

It's true and well known. Ask Netflix who use it for video delivery and contribute to that very action. Your Wall Street example is based on their custom applications that don't use standard tcp/ip and http.
Besides that, it's also well known that decisions taken are often sub-optimal (to say it politely), and do not solely rely on technical expertise. A decision reflects the optinion of the one (or group) that takes it. Very often decision-makers are not immune to following a fashionable hipe.
 

drhowarddrfine

Son of Beastie

Reaction score: 2,499
Messages: 4,446

Mjölnir I was just editing my comment to include that about why Google chose Linux. It was only because it was what they used in school so they were used to it--no other reason though much newer PR releases claim differently than the interview I read a few years after Google started.
 

MarcoB

Well-Known Member

Reaction score: 117
Messages: 398

I think Colin Percival has worked a lot with FreeBSD on AWS (EC2), so perhaps gschadow should contact him. I'm not an expert though so correct me if I'm wrong.
 

kpedersen

Son of Beastie

Reaction score: 2,240
Messages: 3,069

Isn't much of the processing of Wall Street done on ASIC or custom FPGA stuff? This could just as easily be controlled by MS-DOS.

Though this awesome advert (admittedly old and biased) would have us believe it was Solaris 10. I am inclined to believe it still is because everyone is too terrified to re-engineer this stuff and it can't always be done incrementally.

View: https://www.youtube.com/watch?v=Z-Bt3ylCMIU
 

reddy

Active Member

Reaction score: 83
Messages: 123

I understand your grief and, unfortunately, you are approaching this the wrong way.

You are targeting users with your dismay and anger rather than the developers. You need to join the appropriate mailing lists and hassle the developers (and one in particular, I might point you to, but someone else has already, I believe).

Your anger, annoyance, frustration and downright dismissive approach is TOTALLY misplaced here.

This is not feedback, this is an example of venting.

While I agree with you for the rest, I doubt that this attitude would fly well on the dev mailing list. I think he has received a friendlier feedback here than the one he would have received on the developers' mailing list.

I personally agree with the OP that the problem is legitimate (even though I am not a big fan of the cloud), but I feel the only issue is the way you have decided to engage the community on this problem. Just look at the title of this thread, this spends any goodwill people may have before they even read the gist your question. But despite this, I would note that many people have provided answers with technical substance before this thread got into politics.

If you look on the forum, you can see a thread where the community has spent days trying to help someone who created a "dangerously dedicated" usb stick 5 or 10 years ago on an unsupported version of FreeBSD. In light of this, thinking that FreeBSD folks have no interest in fixing issues they dismiss as being irrelevant corner cases probably isn't the right conclusion to reach. I would really encourage you to improve your communication style, because to me, it is the main issue preventing you from getting the help you need here
 

Mjölnir

Daemon

Reaction score: 1,519
Messages: 2,114

drhowarddrfine, you're supporting my mantra: To recruit more developers & increase FreeBSD's deployment # -- thus enhancing it's standing in the industry -- FreeBSD must enhance it's user-friendliness & ease of installation (incl. optional GUI out-of-the-box, instantly, on a commodity PC/laptop/tablet). FreeBSD Kommunity Edition. Xommunity/Qommunity (XfCE/LxQt), too, with a shared toolkit for FreeBSD-specific (admin) tasks, like what OpenDesktop.org is to GUI toolkits. TrueOS was close to this, but they drifted apart from native FreeBSD.
 
OP
G

gschadow

Member

Reaction score: 11
Messages: 36

This is nuts. That's all I have to say. Not worth even responding to any details.
 

eldaemon

Active Member

Reaction score: 28
Messages: 144

FreeBSD does not have any buffering/caching with /dev/ block device access. Linux does.

The reason UFS writes were so much faster is probably because there was buffering and caching. Hence, the speed disrepancies.

About network, that may be another issue. But generally speaking, you can tune FreeBSD to have as good of network performance as anything, if not better.
 
OP
G

gschadow

Member

Reaction score: 11
Messages: 36

FreeBSD does not have any buffering/caching with /dev/ block device access. Linux does.

The reason UFS writes were so much faster is probably because there was buffering and caching. Hence, the speed disrepancies.
Thank you for your on-topic reply.

If it was a buffer caching issue, then in the long run (many GB written) the caching issue should disappear, shouldn't it? I can see how you can have high IO throughput in bursts, especially for reading what's already cached, but as you write and read beyond the cache limit, you will end up with the raw throughput.

Please correct me if I am wrong.

What I think must rather underlie what we can see with dd is that the UFS or ZFS write will have some sort of better understanding how to use the hardware efficiently beyond single threaded sequential writes. Something like that. But OTOH, whatever this solves for a single threaded dd, it will disappear when you actually hit the system with realistic multiple threads of reading and writing as you would have it in a database.

So that single dd problem is still a possible window in understanding the much worse performance even through the file system. As my PostgreSQL database on ZFS is several times slower and IO throughput is so much less. There is nothing subtle about it.
 

eldaemon

Active Member

Reaction score: 28
Messages: 144

Thank you for your on-topic reply.

If it was a buffer caching issue, then in the long run (many GB written) the caching issue should disappear, shouldn't it? I can see how you can have high IO throughput in bursts, especially for reading what's already cached, but as you write and read beyond the cache limit, you will end up with the raw throughput.

Please correct me if I am wrong.

What I think must rather underlie what we can see with dd is that the UFS or ZFS write will have some sort of better understanding how to use the hardware efficiently beyond single threaded sequential writes. Something like that. But OTOH, whatever this solves for a single threaded dd, it will disappear when you actually hit the system with realistic multiple threads of reading and writing as you would have it in a database.

So that single dd problem is still a possible window in understanding the much worse performance even through the file system. As my PostgreSQL database on ZFS is several times slower and IO throughput is so much less. There is nothing subtle about it.
Yes, the buffer will fill up at a certain point for sure and the performance should level out.

Have you tried to do any database-style benchmarks? With your database under load, what does iostat -x 1 look like?

You should see I/O wait if you have I/O contention.

Phoronix has lots of different benchmarks to give you an idea, and regularly compares FreeBSD, Linux, and so forth. Now it's possible these slowdowns are only so evident on AWS.

Another possibility is AWS having different virtualization settings for Linux and FreeBSD. I don't know if this is the case or not, but it's quite possible depending on how it's done.
 
OP
G

gschadow

Member

Reaction score: 11
Messages: 36

Yes, the buffer will fill up at a certain point for sure and the performance should level out.

Have you tried to do any database-style benchmarks? With your database under load, what does iostat -x 1 look like?

You should see I/O wait if you have I/O contention.

...

Another possibility is AWS having different virtualization settings for Linux and FreeBSD. I don't know if this is the case or not, but it's quite possible depending on how it's done.
I do not need any database benchmark when I notice right off the bat that as I am doing

g_dumpall -h pg-on-linux |psql postgres

and I see that the transfer occurs at 32 MB/s and then when it's time to rebuild the indexes it take 10 min with lots of contention, wait, slow throughput when that exact same index rebuilt on the linux server races at 200 MB/s read and 200 MB/s and finishing in 1 or 2 minutes. It's not even subtle. It is obvious that FreeBSD out of the box is 5-10 times slower. You need a benchmark if the difference is subtle, but it is not.

Given all these dismissive answers above (not yours), I would pose the challenge: is ANYONE known to run a larger than a toy database server on FreeBSD AWS EC2?

And database isn't the only thing. Even just AWS CLI (which is a beast cobbled together from Python scripts) is horribly slow on FreeBSD. Like it takes 30 seconds for the Python stuff to load all its modules to do your command.

All these things are obvious for anyone who even tries this more seriously for a day. I made the mistake that I was so loyal to FreeBSD that I would spend days and weeks to create a really cool small image which would be cloneable and super neat and organized, only to find out that I probably won't be able to use any of it in earnest.
 

eldaemon

Active Member

Reaction score: 28
Messages: 144

I do not need any database benchmark when I notice right off the bat that as I am doing

g_dumpall -h pg-on-linux |psql postgres

and I see that the transfer occurs at 32 MB/s and then when it's time to rebuild the indexes it take 10 min with lots of contention, wait, slow throughput when that exact same index rebuilt on the linux server races at 200 MB/s read and 200 MB/s and finishing in 1 or 2 minutes. It's not even subtle. It is obvious that FreeBSD out of the box is 5-10 times slower. You need a benchmark if the difference is subtle, but it is not.

Given all these dismissive answers above (not yours), I would pose the challenge: is ANYONE known to run a larger than a toy database server on FreeBSD AWS EC2?

And database isn't the only thing. Even just AWS CLI (which is a beast cobbled together from Python scripts) is horribly slow on FreeBSD. Like it takes 30 seconds for the Python stuff to load all its modules to do your command.

All these things are obvious for anyone who even tries this more seriously for a day. I made the mistake that I was so loyal to FreeBSD that I would spend days and weeks to create a really cool small image which would be cloneable and super neat and organized, only to find out that I probably won't be able to use any of it in earnest.
If it's that slow, it should probably be investigated. I try to use AWS as little as possible. In general, I don't care for the extreme centralization of hosting services.

I will say that FreeBSD on Digital Ocean, Vultr, and baremetal behave just fine.

Have you confirmed this on a different EC2 node? It is possible you have a noisy neighbor or a host with poor disk performance. I'm assuming the EC2 flavors are the same, right?
 
D

Deleted member 66267

Guest


I do not need any database benchmark when I notice right off the bat that as I am doing

g_dumpall -h pg-on-linux |psql postgres

and I see that the transfer occurs at 32 MB/s and then when it's time to rebuild the indexes it take 10 min with lots of contention, wait, slow throughput when that exact same index rebuilt on the linux server races at 200 MB/s read and 200 MB/s and finishing in 1 or 2 minutes. It's not even subtle. It is obvious that FreeBSD out of the box is 5-10 times slower. You need a benchmark if the difference is subtle, but it is not.

Given all these dismissive answers above (not yours), I would pose the challenge: is ANYONE known to run a larger than a toy database server on FreeBSD AWS EC2?

And database isn't the only thing. Even just AWS CLI (which is a beast cobbled together from Python scripts) is horribly slow on FreeBSD. Like it takes 30 seconds for the Python stuff to load all its modules to do your command.
I have no knowledge with the cloud. What I do at most is a VPS.

If FreeBSD is that bad, could you try other BSDs, too?

You could try with other non-BSD OS, too.

Tribblix (Illumos) available on AWS as it stated here

And could be other Illumos if they are available on AWS.

Giving you said you are too passionate on FreeBSD, I think you definitely want to know how other OSes perform on AWS.

After that, you should have your own conclusion.

IMHO, FreeBSD is the best non-Linux OS. If it's bad, other OSes will be much worse.

It's nothing wrong if Linux is the best suites your needs, just use it.
 
D

Deleted member 66267

Guest


All these things are obvious for anyone who even tries this more seriously for a day. I made the mistake that I was so loyal to FreeBSD that I would spend days and weeks to create a really cool small image which would be cloneable and super neat and organized, only to find out that I probably won't be able to use any of it in earnest.
You don't have to. You don't need to stay loyal to any OS. Just use the best suite your needs. Why have to bother things that you can't control?

Having people working for you for free is very hard. They will ask you to do many investigations yourself and even after that it's not sure if they will fix it at all. If you are a company having financial ability I suggest just hire paid developers to do the job and uses your own patched version, optionally upstream your patches to the project and hopefully they will merge it. Or you could try to find some developers sponsored by AWS themselves and ask them for help. It's all depending on luck after all. But does it worth at all?

If you have money things will be very easy because you don't have to depend on the willingness of other people to do the job for you. If you don't have money, you must be very clever and of course you need luck.

This is my experience of various open source projects and it's the reason why I don't think you will have any more luck on the mailing lists.

Honestly, if I have money, I will go for commercial supported products. It's not assurance of good support, but at least we can sue or have someone responsible for us to sue.

But we don't have it, do we? This is the reason why we use free and open source software after all.
 

Mjölnir

Daemon

Reaction score: 1,519
Messages: 2,114

You could insert a gcache(8) level. If it speeds up your average workload, you have another pointer where to investigate further. EDIT The manpage gives a hint: there might (will?) be some redundant disk array behind your virtual NVD htat splits up the read chunks, while the FreeBSD drivers are written to talk to bare metal.
 

scottro

Daemon

Reaction score: 925
Messages: 2,071

I understand that you're frustrated and want to vent here. However, as has been said, this forum is really more one of users helping users, and like a message on the Fedora forums, The developers ain't here. Though some may at times see things, this isn't the way to reach them. The best way to reach developers is to get on a mailing list and state your concerns there. Filing bug reports, especially if you can be specific, may also be useful. On a FreeBSD oriented forum, coming on and saying FreeBSD is worse than Linux at X will arouse all those who are emotionally invested (and there's nothing wrong with that in my opinion), rather than lead to useful solutions. After awhile, you can even predict who will respond in what way, at least in some cases. (I like to think I'm the one who responds calmly and wisely, but of course, I am prejudiced.) :)
 

kpedersen

Son of Beastie

Reaction score: 2,240
Messages: 3,069

But we don't have it, do we? This is the reason why we use free and open source software after all.
Honestly, since COVID is preventing me from blowing my wage on expensive Starbucks coffee, I feel I have all the money in the world. However, I still use open-source software to avoid DRM which I simply will not engage with.

Up until now, did you really believe that we are all on these forums because we cannot afford a MacBook? ;)
 

mark_j

Daemon

Reaction score: 796
Messages: 1,371

I do not need any database benchmark when I notice right off the bat that as I am doing

g_dumpall -h pg-on-linux |psql postgres

and I see that the transfer occurs at 32 MB/s and then when it's time to rebuild the indexes it take 10 min with lots of contention, wait, slow throughput when that exact same index rebuilt on the linux server races at 200 MB/s read and 200 MB/s and finishing in 1 or 2 minutes. It's not even subtle. It is obvious that FreeBSD out of the box is 5-10 times slower. You need a benchmark if the difference is subtle, but it is not.

Given all these dismissive answers above (not yours), I would pose the challenge: is ANYONE known to run a larger than a toy database server on FreeBSD AWS EC2?

And database isn't the only thing. Even just AWS CLI (which is a beast cobbled together from Python scripts) is horribly slow on FreeBSD. Like it takes 30 seconds for the Python stuff to load all its modules to do your command.

All these things are obvious for anyone who even tries this more seriously for a day. I made the mistake that I was so loyal to FreeBSD that I would spend days and weeks to create a really cool small image which would be cloneable and super neat and organized, only to find out that I probably won't be able to use any of it in earnest.
It's getting tiresome.
Accumulate all the relevant information and raise a bug report. Everything else is just a waste of bytes.
Mike drop....
 
D

Deleted member 66267

Guest


Honestly, since COVID is preventing me from blowing my wage on expensive Starbucks coffee, I feel I have all the money in the world. However, I still use open-source software to avoid DRM which I simply will not engage with.

Up until now, did you really believe that we are all on these forums because we cannot afford a MacBook? ;)
No. Using free and open source software also because of reducing the cost. Or you have specific reasons of your own, mostly ideology, technical, or endeavor/adventurous.

But for me, if I have money, I just bought a local 'Enterprise Linux" which they took the CentOS/Redhat sources and recompile/rebrand it to make their own and offer commercial support targeting just the local market inside my country. I want a worry-free life when my server just up and running after I followed the tutorials and paying monthly fees to keep everything on going until I no longer want it. This company also provided hosting. But the only problem is, their price is... insane!

I don't care if it's FreeBSD or Linux. I only care if it just works and I don't give me much headache.

I choose FreeBSD because it's the best performance non-Linux OS and the community here is very helpful when support is completely free.

BTW, I think when using free and open source software, you are already living on the mercy of a few people. Don't want to be dependent? Money is the key! Unless you are a very talented person having the ability to do everything your own, writing and maintaining your own patches... which I think many people here can but unfortunately I can't. There must be reason why I choose the name 'failure' for my account name ;)
 

kpedersen

Son of Beastie

Reaction score: 2,240
Messages: 3,069

BTW, I think when using free and open source software, you are already living on the mercy of a few people. Don't want to be dependent? Money is the key!
I do agree that open-source does depend on some key individuals (this is also often true behind the scenes in proprietary products). However I don't think money really helps in many cases. The really important innovations I notice come from personal interest and passion. Whereas, for example, if I chucked an infinite amount of money at Microsoft, they would still provide shite.

You obviously have had a much better experience with "commercial support" than I ever have!

offtopic: I always chuckle when I encounter a tech-net help forum. Their little Microsoft verified qualifications along with their introductions are often longer than their precanned "solutions" telling the user that is not supported at this time.
 

mark_j

Daemon

Reaction score: 796
Messages: 1,371

You obviously have had a much better experience with "commercial support" than I ever have!
(Off-topic, but then this topic is off topic!)

In my experience, commercial support is more a management state-of-mind, in that, if they are risk averse and have the funds, they will always go the route of commercial support. That is, take Oracle over MySQL or postgreSQL (yes, I know there's commercial support available, but they're not a "name" like Oracle).

Having said that, management then do not care what actual support you get and whether it's useful; all they know is they paid for something and should the excrement hit the oscillation device, they've got someone to blame.

I have experienced first hand a senior manager who, upon learning of a catastrophic bug (multixact in v9.4 (I think)) in postgres was absolutely catatonic when he found out we didn't have paid support. He actually said "Well, who can I blame then? You? Or you?..." (as his eyes moved around the room ... it was hilarious).
 

PMc

Daemon

Reaction score: 694
Messages: 1,394

But for me, if I have money, I just bought a local 'Enterprise Linux" which they took the CentOS/Redhat sources and recompile/rebrand it to make their own and offer commercial support targeting just the local market inside my country. I want a worry-free life when my server just up and running after I followed the tutorials and paying monthly fees to keep everything on going until I no longer want it. This company also provided hosting. But the only problem is, their price is... insane!
Yeah, having money is one thing. Throwing it into the greedy mouths of others is entirely a different one.

BTW, I think when using free and open source software, you are already living on the mercy of a few people. Don't want to be dependent? Money is the key!
What? Buying support? Buying support gives you the opportunity to discuss with lawyers about the definition of the word "bug". What makes that better?

Unless you are a very talented person having the ability to do everything your own, writing and maintaining your own patches...
And that's exactly the definition of independence. The only one. You cannot buy your independence - that's a contradiction in itself.
 
Top