Startup hosting and sleeping well (encryption)

[Mage]

I’ve never been in a server room from which I could not steal a random hard drive without getting caught, if I wanted. I have been in server hosting companies’ rooms in more than one countries.

Should I find one that employs guards with machine guns, still there is a point from which it isn’t smart to keep your unencrypted data on disks that can fail and be replaced by people who are not you.

I have been considering Geli for a while and collecting questions about ZFS + Geli. Not because there aren’t tutorials online. There are also tutorials online how to climb onto 1000 meters tall buildings in Russia without any kind of protection. Yet I don’t want to do that either. Just because people with good will post guides one should not put a method into production right after it didn’t throw any error in the command line. Especially not without being expert in that area. 4 of the 5 guides top on my Google search don’t mention that TRIM won’t be supported. (The one that does.)

Actually, I used to have Geli + ZFS on my home desktop computer with zero issues, but I know that the setup harms ZFS’s self-healing features. Should it be below the metadata, a few bad sectors that would be corrected by native ZFS may eliminate the whole disk with ZFS + Geli. Another nuance rarely mentioned in random guides.

I’m not opening the thread to blame guides that helped me hundreds of times. I’m wondering what experts do for this kind of security. I wonder what companies that aren’t big enough yet to hire gunmen but aren’t small enough to not worry about their data can do.

The sole alternative crossed my mind so far was Amazon EC2. I have never tried it but I read they support FreeBSD. They also provide encryption. I can only hope they are big enough to not try to steal my encryption keys. I’m not talking about the company but the employees.

Should you wonder whether I am paranoid or not, I’m opening this topic from the country that has been famous for its bank secrecy for decades until a few individuals stole data from their employers ruining the whole industry permanently and causing more damage than you might think.

Any information based on real experience that makes a young company’s people sleep well would be awesome.

 

[Murph]

As far as real world professional data centres go, it all comes down how much you are willing to pay.

At the top end, I've seen per-customer locked cages behind biometric doors behind armed guards inside a category 5 tornado-proof building/bunker. Inch-thick bulletproof glass at reception and extremely resistant doors as the only means of external entry. All covered by huge overkill levels of CCTV. Good luck getting anywhere near someone else's server in that type of place, let alone getting out with anything. Short of a military assault (or a court order, or government spooks), it was physically secure. You don't have to be a big company to be in a place like that, only able to justify the high cost of renting space there.

At the bottom end, I've seen miscellaneous office/industrial units with very little security and a relatively disinterested receptionist.

You have to balance the cost against realistically how valuable your data is (mostly the value to an attacker, but also the value to you), and how big a target you really are. Factor into that the difficulty for someone to quickly find a specific server in a large hosting building (on the basis of it needing to be quick for them to have a chance of getting away with it).

Unless you choose a really terrible hosting company, you normally have a far greater risk of an attack over the network where encrypted drives won't help you. Good hosting companies will sell you failed drives from your servers, so that you can either destroy them yourself or send them to a data recovery company; others offer a drive destruction service.

One last thought. The world has survived for decades without routinely encrypting drives (mostly due to the performance hit prior to modern CPUs and hardware encryption, or just lack of OS support for it), and it's generally been just fine for low to medium levels of confidentiality. It's still relatively abnormal to encrypt drives in data centres, other than quite specific situations where it's required.

 

[Mage]

Thank you for the input. I’m going to consider everything you wrote regardless of my comments. It wouldn’t be the first time I valued an advice I also argued with.

The psychological factor is huge. I mean, eating in a restaurant has its risks. Flying on a plane has its risk. Some drive their cars with hundreds of miles per hour yet they manage to stay alive. Some drive slowly and end up in the graveyard unplanned. Some believe in karma. Some believe in luck. Some believe in good will of others. I’m anything but sarcastic here. At the end it’s all about the differences of the personalities and the personal experiences.

Indeed, the world has survived without encrypting everything. That’s why it wasn’t the future of the human kind I worried about. Humans also survived the plague. In case one day they fail to survive, there will be the rain forests there, having a reason to cheer up. One way or another, the story will have a happy ending.

130 years ago people fought over the recipe and the license of Coca Cola. There were casualties. The reason we know about this isn’t that the company is evil. We know it because the company still exist and grew big. Which implies two things:

1. We don’t know the thousands of similar stories because those companies didn’t grow big and even ceased to exist.

2. It doesn’t matter whether one has the so called big thing or not. In the beginning, thing’s aren’t big. It was just a drink of many unhealthy drinks. Yet people already fought.

Back to the servers, physical protection helps against attacks from outside. Attacks from outside, as you told, more likely happen over the network. I’m more concerned about people from inside and random visitors.

Anyway, the hosting company I trust and has my servers is 1000km away from me. It would make sense to have the next server oversea. I can’t replace the drives by myself in any of them.

The hosting company won’t sell me the drives. The drives are already mine. However, the time when they could sell me the drives is exactly the time when a person could make a copy of the data, without me noticing it. The only question is whether it would take more than ten minutes or less. Especially when the drive didn’t fail but was replaced as an upgrade.

I (or you) don’t have to be a specific target. He could do this with many drives. As a hobby. Actually, people do these things all the time. Yes, there are many who would never do such a thing. It doesn’t matter what the ratio is.

Since I am not a network expert, I have put my trust into pf, the FreeBSD engineers and a cloud-web protection company. I also have trust in my application design skills. If none of us made a big mistake, the weakest link might be the physical access.

There is a story on Quora about a young and talented guy who stole world class compression solution in the age of 33k phone modems. He did it during his summer work. Unfortunately for him, the buyer from China was an undercover FBI agent.

Sony got their users’ data stolen, including the unencrypted passwords. Yahoo had its search data leaked. Few weeks ago, Denmark managed to send China the medical data of more than 5 millions Danish people. They sent the unencrypted CDs to a wrong address. (And wow, China came up again.)

Yahoo and Sony are struggling. There might be hundreds of reasons of that. Still, who knows when a huge company starts losing the trust?

Also, many websites had the email addresses and credit card data stolen. Are we sure all of them are still online?

Once I knew a huge company that was the biggest player in the local area. For many, it was like the goose who laid golden eggs.

Over the years several employees stole the company’s database. It didn’t take more than:

pg_dumpall > let_me_take_this_home_thank_you.dump.sql

The company doesn’t exist anymore. Not only because of the theft. It wasn’t even the biggest reason of the crash. However, users aren’t stupid. Some people sign up with an email address they don’t use anywhere else. For example, should your website be called Blue Moon, some will sign up with blue_moon_436476 at whatever.domain.

The random number makes the address impossible to guess by spammers. As soon as they start receiving spam, or what’s even worse, they receive personalized emails, including their real names, from other companies, they’ll know that their personal data was either sold by you or stolen from you. That’s exactly what happened to this company more than once.

One can argue that the data was stolen over the network. It’s true. It was possible to do it. Too many people had access to the database.

Now let’s imagine that the manager did his job well. It’s easy to setup a query interface for testing and for analysis that doesn’t let any employee make a dump.

The question is: could the data still be stolen? In case of this real-life example, the answer is yes. Some had a good relationship with at least one of the administrators who didn’t care. Some were in a position high enough to convince. When making a remote dump is not possible, why not to copy the whole disk? Or why not to bring a snapshot home?

Now let’s imagine that the query interface was set up, the access system was set up in a sane way, and the disks were encrypted. Only two persons could mount them. All connections use SSL. How would you steal the data?

The point is: the data is often stolen over the network not because the network protocol is insecure but because the configuration is insecure. However, as soon as you use a sane configuration, it’s not unlikely that the physical access becomes the weakest link. And it’s hard to have control over it, especially when they are geologically distributed.

It won’t be the hacker from Malaysia but it can be one of administrators who’ll just put a box of pizza onto his laptop and your previous drive while your data it is being copied.

And yes, the world will probably survive that.