The fight for FreeBSD at work

[drhowarddrfine]

So it's finally happening to me. I took a job to revamp a company web site which is about 10 years out of date. They pretty much just handed me the key and said "go for it". I told them I'd like to change hosting companies, use a new programming language, restructure the whole thing and they were fine with that.

And then ...

I told the IT guy I wanted to change it all to FreeBSD.

"Isn't that like Linux?". This is a Windows shop but their site is on CentOS so they have an Ubuntu box where they wrote everything, copy it to a Windows server for backup and then upload it to the CentOS server. They use CuteFTP and Putty to accomplish this so it's all a bit of a mish-mash.

I decided I wanted to use nginx as a server but, when I tried to install it on the Ubuntu box, the repository is pretty old. That's when I found out you have to jump through a few hoops to get the latest version. This is also true on CentOS which makes me wonder what else the future holds. Thus, my quest to switch everything to FreeBSD.

Everyone here admits to being *nix novices and don't really know their way around though the IT guy can figure things out eventually but he shook his head when I said I wanted to switch hosting companies and "where would we get support?" as if he gets better support from our host (HostGator) now.

I have the higher up's ears so I'm pretty sure that what I say goes. I just didn't come back to the IT guy strong enough as I should have cause, frankly, it's the first time I had a choice and didn't have to answer to someone else. (Plus I ran into him unexpectedly in the coffee room when I brought it up.)

So there. I spilled my guts. I guess I'm looking for someone to say something.

 

[scottro]

Who will maintain it after you're gone? Will the new hosting company give support?
You're getting paid to do it, and will be the one they call if it breaks, so you should be the one to decide--phrase that more politely of course.

Without knowing you or them, one can only state the fairly obvious....you can say in your experience, you've found more stable and giving better performance, find an article about that Facebook thing that they're looking for developers to make Linux's networking as good as FreeBSD's as evidence.

Oh, one more thing, point out that it has an upgrade path, and CentOS' is currently not well supported--they can look at CentOS forums for proof of that. In Fedora, the Fedup thing is working somewhat better, but I believe that unless one has an almost unaltered installation, they still recommend reinstalling. (Debian used to, anyway, have a pretty smooth upgrade path, so don't mention Ubuntu in that respect).

Although in fairness, a FreeBSD install running, say, apache-2.2 can start making you work harder than you'd want, as everything now defaults to 2.4, and anyway, you're using nginx.

 

[Dies_Irae]

They pretty much just handed me the key and said "go for it"

I told them I'd like to ... and they were fine with that.

... but he shook his head ...

If I were you, I would tell them: "Ok guys, let's start over. Tell me clearly what I can do and what I can't. Otherwise, we're talking about nothing."

(Don't worry, you're not alone. At my workplace I say this exact phrase about twice a day! )

 

[SirDice]

I suggest you use Apache as that's what they are used to (regardless of the OS, that's what they 'see'). So for those creating the content nothing would change (you don't want discussions about things not working as expected afterwards, they'll be quick to blame everything they don't know). Next up write everything down in easy to understand language how to update, how to change settings, how to add a web site, etc. The key here is documentation. It doesn't matter much what OS you use as long as they can do simple modifications on it. Use a Wiki or something similar to document the system(s).

And best of all, put your plan to paper. Write it out, yes, it's a lot of work. But the "bosses" really like that kind of stuff. If your plan looks good on paper they're much more inclined to let you have a go. Try and make the plan as elaborate as possible. Incorporate everything they want and might want in the future. Find out exactly how much support they're paying for. Find out if they can have more support and on what. Incorporate that into your plan too.

Last but not least, don't be afraid to keep everything on CentOS. It's their money, it's their systems. And truth of the matter is, it's really easy to find someone with Linux knowledge. It's a lot more difficult to get someone with good FreeBSD knowledge.

 

[drhowarddrfine]

And truth of the matter is, it's really easy to find someone with Linux knowledge. It's a lot more difficult to get someone with good FreeBSD knowledge.

See, that could be an argument for them, and it's something I thought about, but I hate to bypass FreeBSD for that reason alone. It's almost a Mac vs Windows machine argument.

Which brings up scottro 's point about who will work on it if I leave. These guys admit they don't know anything about *nix/BSD beyond the stumbling and fumbling to bring their site up in the first place.

And then there's this thought. They make devices controlled by Windows applications and they want to move some of that to the web so it can be controlled from a web app. What happens if they want to have it work on Linux machines or Linux mobile devices in the future? The web is agnostic along those lines but am I going to face a "we need to develop drivers for Linux hardware" at some point?

Add to that, I hate the direction Linux has been taking.

I see the technical reasons for using FreeBSD and not wanting to look at the practical ones for continuing with Linux.

This has been keeping me up at night.

EDIT: You know, now that I think of it, is this any more of a drastic change than switching them to nginx for server software from Apache? I feel far more confident in that choice and can put up a good fight for it.

 

[protocelt]

I think SirDice covered most of my thoughts. Loyalty to a particular tool or operating system should really not be factor in this decision at all unless or until all else has been considered on technical merit and customer feedback first.

I would also add after discussing things in depth with your customer, maybe even go one step further and come up with competing plans using different tools,then bring that to the customer's attention. In my experience a lot of the time the people running the show will appreciate it when they can make comparisons and see the differences. It would also give you a better understanding of what tools or operating system would work best in this particular situation. My two cents.

 

[scottro]

I would consider it less of a change. If I were in position to hire--or were job hunting--if I have a DNS server that's running CentOS, I'm more concerned about whether the person knows BIND than whether they're on CentOS or FreeBSD. The applications are usually the more troublesome thing.

If you're going to be the main go to person for a period of more than a couple of months, then you should have the say in the O/S. As you describe these people, they're not experienced in Linux either. If they are relatively skilled in CentOS, then they have, IMHO, a more valid reason to argue. Even if you are going to be responsible---though as I think of it, they'd have a bigger changed from CentOS-6 to CentOS-7 than from CentOS to FreeBSD. (I'm exaggerating a bit, but as an example, the service command that is used in CentOS also works in FreeBSD--right now, it might work in CentOS-7, I don't have one around, but it's deprecated in favor of systemctl start whatever.service.

 

[drhowarddrfine]

I think SirDice covered most of my thoughts. Loyalty to a particular tool or operating system should really not be factor in this decision at all unless or until all else has been considered on technical merit and customer feedback first.

What got to me wasn't so much loyalty to FreeBSD as much as the things I tripped over trying to set up one of the workstations they develop on and trying to update. I'm definitely biased toward FreeBSD but we all know the reasons for that.

I talked it over with the head of engineering and his lead programmer and they don't seem to have a problem with switching but, as I said earlier, they really don't know enough about it to make the decision. I'm getting convinced, after talking to them, that the IT guy is only pushing back cause he's in the "it works, why change it?" mode and, possibly, an "I had so much trouble getting it to work I'm afraid to do anything".

But I'm the one in charge so I'm going to go as far as I can with building the site and then swap it out.

 

[Crivens]

What helped me a lot in my current job is the documentation. The plan was to develop the system we currently build on Linux, but I had a good argument for FreeBSD. I simply put my stash in books on the table, starting with "The design and implementation..." (old edition) and ending with the "Device driver development" book. With about one foot of books stacked up on the table the question of "who will work on this later" can be answered with "anyone who can read".

This is one of the big "selling points" here. The documentation is awesome and the system is not changing for the sake of the change. It is stable, and so the calculation for the moment when finally someone has to take over may still have to include a learning ramp up, but after that you are mostly set. Compare that with the changes and incompatibilities you find from one version of Windows to the next, or one version of some Linux distribution to the next.

 

[drhowarddrfine]

I was caught off guard. The director of engineering came up to me and said we were having a meeting with the IT guy right then and my mind was elsewhere. He did some research on his own, had printed out 10 pages of stuff, but basically said, no one uses FreeBSD and everyone uses Linux so he sees no reason to use FreeBSD.

"Where will we get support?" "Who do we call when something goes wrong?" You could tell he was a Windows guy cause he went on to ask how much nginx costs, how much it would cost to use Golang, and even how much FreeBSD costs. The real capper was when he said they needed a GUI to work with cause none of these guys are used to working from the command line.

The surprising part was, the IT guy showed some interest and even declared that it didn't matter to him.

The only thing I couldn't argue against was that they want to start developing some drivers for Linux systems they're creating and wanted to keep everything on one platform.

I argued that I had trouble getting up-to-date versions of the software I wanted to use out of the repositories for Ubuntu and CentOS while FreeBSD had everything I wanted in ports, Netflix used it, derived directly from ATT Unix, but, to tell the truth, I felt all stressed out cause I thought the IT guy would rake me over the coals and I just wasn't prepared and I don't think I handled it as good as I otherwise would have.

While I think we'll end up with Linux, I still got them to compromise that I'll continue with what I'm doing and, at the first sign of trouble, we'll revisit the whole thing.

 

[jrm@]

The only thing I couldn't argue against was that they want to start developing some drivers for Linux systems they're creating and wanted to keep everything on one platform. ...
I argued that I had trouble getting up-to-date versions of the software I wanted to use out of the repositories for Ubuntu and CentOS while FreeBSD had everything I wanted in ports

Has bhyve come up in your discussions?

I work in a research group that builds/uses models of evolution. Most of these evolutionary biologists write code that's thrown together with lots of Linuxism. For a long time I battled with the windmills trying to get this software to work on FreeBSD. Sometimes I succeeded, but in the end it was just too time consuming. bhyve is making my life so much easier. I get to run FreeBSD, but I just keep a Linux bhyve instance. It's the best of both worlds for us. Maybe the transition for this company would be easier if, when necessary, Linux could be around to fall back on.

 

[NewGuy]

drhowarddrfine, you seem to be coming at this project from the point of view of which tools you like to use and what environment makes you comfortable. However, in my experience, it is best to provide the tools and services the client wants and what the client feels comfortable maintaining.

You may be able to upgrade them a bit, perhaps streamline their processes a little, but ask yourself if it really benefits _them_ to switch web servers, if it really benefits them (long term) to switch to FreeBSD. Their manager raised some intelligent points and, depending on who ends up maintaining this system once the job is done, it may make sense for them to stick with technology they are familiar with.

Also, keep in mind that while CentOS and Ubuntu may have older versions of web servers in their repositories, those packages are supported by the distribution and may receive backports. FreeBSD, while a great operating system, has (in my experience) flaky support for some ports/packages. That's probably part of the manager's concers, if the new tools break, who will fix it? Will you, will upstream, will their IT guy be stuck fixing it once the job is done?

I get a lot of business from people who had their systems changed in ways they don't understand by the last IT guy. Often times I either end up explaining things to them, or ripping out the old work and replacing it with something they better understand.

 

[jrm@]

On the other side of the coin, when do you attempt to steer the customer away from what they're comfortable with when you feel it's in their best long-term interest? For example, it might be comfortable (easier in the short term) to not use jails, to not hash passwords, or install Windows. These are obvious; I'm just pointing out that we can lean on our biases in the interest of the customers too. If we're not going to (gently) push FreeBSD when we truly feel it's the best tool for the job, then who will? Package management has come a long way and the documentation is excellent, so even a Linux admin shouldn't have that much trouble getting up to speed these days, no?

ADDED: I'm reconsidering my last statement, recalling that some Linux distributions don't even have ifconfig any more.

 

[NewGuy]

Yeah, the move from Linux to FreeBSD can still be a bit of a climb if the admin is just thrown into the new system. I'm all for offering training, for suggesting alternative/better technologies to the client. But, at the end of the day, the client needs to be comfortable using and/or maintaining the technology.

 

[drhowarddrfine]

it may make sense for them to stick with technology they are familiar with.

That's part of my point. They aren't familiar with anything outside of Windows. In fact, tomorrow, I'm having lunch with the engineering head and his son, who's a web developer with some Microsoft shop in town (!), who keeps asking what tools we're going to use and he doesn't understand anything his son is telling him and he can't figure out half the stuff I'm telling him. So I guess he's going to be an interpreter of sorts.

This is strictly a Microsoft shop but their web server is Linux and the set up is a mess. If it weren't for the fact that they want to get into inter-operating with some Linux devices in the future, I wouldn't have any concern for doing the switch. I'm the one who will be maintaining this for them. If they want to get someone else, fine, but it's my job to provide the best system and FreeBSD is it.

I still don't think it matters what we go with but, as I said, the "one platform" idea is compelling. Of course, they're not one platform cause they're running Windows, too, but that's a different story.

So far, it looks like we'll just stick to the Linux thing, and I'm OK with that. I was just hoping to make this another FreeBSD win, too.

 

[roddierod]

My question would be what is the application written in and what kind of technologies or other apps does it have to interact with?

I working in a 95% Microsoft shop on the server side and 99.9% on the desktop (might be one or two Macs because Doctors like them). I have two website that wrote and I also maintain them. One of these was originally programmed in pure Java and running on MS IIS 5 or something. Since I was doing all the maintenance I rewrote the entire thing in Python and have it running on Apache on a Windows server. The only thing it interacts with is an MS SQL database, which is pretty simple using COM objects in Python.

Now the other web app is written in C# and MS MVC... this thing has to connect to MS Active Directory, MS SharePoint Server, MS SQL and even MS Exchange... although I could do all this with Python it far more simple and easier to maintain by using C# and IIS than it would be to do so host this on Apache or writing this in Python/Django... especially when MS puts out one of there none backward compatible updates that everyone has to have because it's really shiny!

 

[drhowarddrfine]

I'm looking forward. The current application is a Windows desktop only thing. The new product will have a built-in Linux server with application that will work locally or over the internet. None of that is written yet.

 

[da1]

We should make a thread on how to respond to anti-FreeBSD arguments.

 

[gkontos]

We need to see this from the clients prospective. This is not an anti-FreeBSD attitude really. This is clearly the attitude of people who have no idea about *nix. Keep in mind that the only reason why they use Linux now is because they have to. That explains why their systems are a mess. I am pretty sure that if you dig around a bit, you are going to find outdated software with many security vulnerabilities.

No matter what OS you are going to use, if it is *nix based then you are going to be the only one who can maintain it. If you are gone then chaos comes and believe me, the IT guy will say that "I know a bit about Linux but this thing is different and I have no clue on what to do". Of course, the IT guy has no clue about Linux either but that is a different story. Then comes the director who will try to contact FreeBSD for paid support like he would do with Microsoft, I think you catch my drift.

Whatever you decide to do, make sure you keep it very simple. Should you choose to go with Linux then avoid CentOS 7 and stick with 6.6.

 

[drhowarddrfine]

gkontos hit the nail on the head with how the company is. I need to update this with the CEO's rant, just two days ago, about being fed up with having to pay so much money to update Microsoft Word and Windows and how he wants to have an ongoing discussion about switching the whole company to Linux. Especially after not being able to open a .doc file with Word and, after installing LibreOffice, that opened it "without argument" as he said.

However, he has the typical issues with doing such a thing in that sales and accounting all use Quickbooks among two or three other major Windows-only programs.

 

[da1]

So then Ubuntu on the server because any twat knows how to clickety-click

 

[scottro]

Ubuntu server, by default, doesn't have a GUI. Depending upon what they want and need, there are a few turnkey solutions with Linux and GUIs for servers. I can't think of their names offhand, but there's one based on CentOS with GUI controls for various aspects.

 

[Beastie7]

gkontos hit the nail on the head with how the company is. I need to update this with the CEO's rant, just two days ago, about being fed up with having to pay so much money to update Microsoft Word and Windows and how he wants to have an ongoing discussion about switching the whole company to Linux. Especially after not being able to open a .doc file with Word and, after installing LibreOffice, that opened it "without argument" as he said.

However, he has the typical issues with doing such a thing in that sales and accounting all use Quickbooks among two or three other major Windows-only programs.

If he's looking for a good Unix solution without paying high upfront licensing costs, I would recommend Mac OS X deployments with the Casper Suite from JAMF software. It's literally AD/GPO + some other awesome stuff for iOS/OS X deployments. I'd look into it, it's pretty cool software. They even offer commercial support also.

 

[roddierod]

Doesn't iX System offer FreeBSD support services? I think the used to. Maybe if the FreeBSD project had and official or recommended company for support you could point the higher ups in that direction.

 

[gkontos]

Doesn't iX System offer FreeBSD support services? I think the used to. Maybe if the FreeBSD project had and official or recommended company for support you could point the higher ups in that direction.

There is actually a list with commercial vendors which also includes consulting services. Having one recommended company for support is something that most people using FreeBSD would certainly be against. Monopolies is definitely not the FreeBSD way...