ZFS zfs snapshots protection against ransomware

L

last1

I am building a new ZFS cluster: a live system and backup system.

If I setup ssh keys, then a potential hacker can delete snapshots from my backup system as well if they gain root access on the live system.

I'm thinking of two possible approaches to ensure my backups can't be touched:

1. bkp server holds keys to live server and does something like:

ssh root@live 'zfs send -i zfs/testindex@now5 zfs/testindex@now6 | sshpass -p test2023 ssh root@bkp zfs recv zfs/testindex'

- the password would be changed after each invocation on the bkp system, although it would still offer a brief window where it could be grabbed and used

2. on live server, I zfs send the snapshot to a file
on bkp, I rsync that file via a simple pull and do zfs recv from it

#2 is much more secure but a bit less elegant I find. Is there a way to leverage send/receive without compromising access to the backup system ?
 
Instead of pushing to the backup server the backup server could be the one with an ssh key to the primary system. I'm not sure why you are complicating it in #1.

Code: 
ssh root@live 'zfs send -i zfs/testindex@now5 zfs/testindex@now6' | zfs recv zfs/testindex
 
cracauer@ said:
Instead of pushing to the backup server the backup server could be the one with an ssh key to the primary system. I'm not sure why you are complicating it in #1.

Code: 
ssh root@live 'zfs send -i zfs/testindex@now5 zfs/testindex@now6' | zfs recv zfs/testindex
Click to expand...

oh wow, feel like an idiot. Thank you sir
 
now, if it's not too much to ask, do you know of any snapshot managers that can operate from the backup server and not from the live system ?

all that I've seen run from live -> backup
 
last1 said:
now, if it's not too much to ask, do you know of any snapshot managers that can operate from the backup server and not from the live system ?

all that I've seen run from live -> backup
Click to expand...

None that I know of. I'm interested, too.
 
You can lock the keys only to specified hosts, to run only some command, and to make only permitted TCP-forwarding.
So it may prevent any access to the server other than configured for the key.

/home/sshbackup/.ssh/authorized_keys
Code: 
from="1.1.9.4/32,9.2.1.1/32,9.1.1.8/32,1.1.1.2/32",no-pty,command="sleep 10",permitopen="127.0.0.1:873" ssh-rsa KEY-HERE

sshd(8)
section: AUTHORIZED_KEYS FILE FORMAT

I use the similar config to forward rsyncd over ssh. Of course, every rsyncd module has its own password. And I have configured one dedicated rsyncd module per external host.
 
One of my credit card providers (GE Capital) just got hacked, and yesterday I got the email telling me that they have leaked a lot of my personal data. So you should certainly plan to defend external attacks.

But, most catastrophes have an internal cause. Electronic things break, sometime in the most inconvenient ways. People do dumb things, sometimes really dumb. Sometimes accidents just happen. Occasionally you see malice.

Keeping backups on-line is pretty much de rigueur at the big end of town. It facilitates rapid diagnosis of problems, and minimises time to recover for the most common problems.

Is your backup ZFS server off site? If not, you need additional protection against (undetected) hacks, fires, sprinklers, earthquakes, nuclear attack, etc.

I'm also a strong believer in complete physical isolation of the off-site backups, to resist "contamination" from any likely source.
 
last1 said:
If I setup ssh keys, then a potential hacker can delete snapshots from my backup system as well if they gain root access on the live system.

I'm thinking of two possible approaches to ensure my backups can't be touched:
Click to expand...
Why are you authenticating with root over ssh? It's a security breach. I understand that you want to do all that to provide extra security, but it may not have been necessary.

Why not create a user that is able to replicate snapshots and mount them?

Example local machine:

pw user add senderuser -m -s /bin/sh #create user for send and create snaps.
zfs allow -u senderuser send,snapshot zroot/test #allow permissions in the senderuser to send and create snap in the zroot/pool
Click to expand...


Example remote machine:

pw user add recvuser -m -s /bin/sh
zfs allow -u recvuser compression,mountpoint,create,mount,receive zroot/backup
Click to expand...

I would have to create a mount point and if I'm not mistaken activate vfs.usermount.
But wouldn't this method be more valid? So you could bypass root, and your snaps would be more secure.

regards.
 
freejlr said:
Why are you authenticating with root over ssh? It's a security breach. I understand that you want to do all that to provide extra security, but it may not have been necessary.

Why not create a user that is able to replicate snapshots and mount them?

Example local machine:




Example remote machine:



I would have to create a mount point and if I'm not mistaken activate vfs.usermount.
But wouldn't this method be more valid? So you could bypass root, and your snaps would be more secure.

regards.
Click to expand...

I used root for brevity of testing, but even if I use another user, that user's credentials will be known on the production server, and could be used to destroy/encrypt my snapshots. The only real solution is for the backup system to have different credentials than the other systems, credentials which aren't inserted into any automation scripts anywhere.

cracauer@ said:
Instead of pushing to the backup server the backup server could be the one with an ssh key to the primary system. I'm not sure why you are complicating it in #1.

Code: 
ssh root@live 'zfs send -i zfs/testindex@now5 zfs/testindex@now6' | zfs recv zfs/testindex
Click to expand...

To up the ante: if I wanted to run the send/recv via nc instead of ssh, would that be possible with a one-liner login from the backup system ?
 
gpw928 said:
One of my credit card providers (GE Capital) just got hacked, and yesterday I got the email telling me that they have leaked a lot of my personal data. So you should certainly plan to defend external attacks.

But, most catastrophes have an internal cause. Electronic things break, sometime in the most inconvenient ways. People do dumb things, sometimes really dumb. Sometimes accidents just happen. Occasionally you see malice.

Keeping backups on-line is pretty much de rigueur at the big end of town. It facilitates rapid diagnosis of problems, and minimises time to recover for the most common problems.

Is your backup ZFS server off site? If not, you need additional protection against (undetected) hacks, fires, sprinklers, earthquakes, nuclear attack, etc.

I'm also a strong believer in complete physical isolation of the off-site backups, to resist "contamination" from any likely source.
Click to expand...

yes, it is offsite in another DC, however, I look at all these big shops and most of the time their backups also get encrypted in such attacks, because...automation. And now big storage vendors are selling 'immutable storage' which costs an arm and a leg, for what is in fact not that hard of a problem to solve. Their issue is still that they use central authentication with push backups, vs separate auth and pull
 
last1 said:
I used root for brevity of testing, but even if I use another user, that user's credentials will be known on the production server, and could be used to destroy/encrypt my snapshots. The only real solution is for the backup system to have different credentials than the other systems, credentials which aren't inserted into any automation scripts anywhere.
Click to expand...

I understand, I think it was my mistake not to ask before, but what you said is not true.

Snaps cannot be encrypted, they are a read-only copy of a volume or file system. As was well said, they can only be destroyed by a user with privileges such as root, who has full permission.

I understand your position, I simply gave you the example above, to create a user with the necessary permissions to do what you want, without being able to destroy them, because if an attacker manages to exploit your system, he will do so as that user with certain limited permissions on your snapshots, if you do it as root you are lost.

Well, I'm glad to know that the root user was just an example.
 
freejlr said:
I understand, I think it was my mistake not to ask before, but what you said is not true.

Snaps cannot be encrypted, they are a read-only copy of a volume or file system. As was well said, they can only be destroyed by a user with privileges such as root, who has full permission.

I understand your position, I simply gave you the example above, to create a user with the necessary permissions to do what you want, without being able to destroy them, because if an attacker manages to exploit your system, he will do so as that user with certain limited permissions on your snapshots, if you do it as root you are lost.

Well, I'm glad to know that the root user was just an example.
Click to expand...

yes, I had seen this in the FreeBSD ZFS document, however, that's still a potentially compromised user that an attacker could have access to on the backup system, no ? Do I really want to worry about what potential 0-day exploits an attacker might have in order to gain full control of my backup system ? My experience tells me to limit the attack surface as much as possible, and leaving even unprivileged user access is a big no-no. Is this wrong ?
 
I think that you did not fully understand what I said, I refer to your main message:

last1 said:
I'm thinking of two possible approaches to ensure my backups can't be touched:
Click to expand...

If there is more, possibly the one I told you, if I'm not wrong, I'm not telling you that your points are wrong, they seem fine to me.

If I use root as an example, just forget everything I said, I just wanted to tell you that if you are authenticating as root on a remote machine, you run that risk, you run the risk of your snapshots being touched by a super user as root, and they can be removed.

Not the other way, it would be a user without privileges, with the right permissions to receive and send create snapshots. Yes, it's equally dangerous for an attacker to break into your system and run a 0-day, but don't you think that's better than logging in as root? It already granted full permissions, without being able to execute a 0-day.

At no time did I tell him to abandon his proposal and go for mine, I believe or at least I told him this, because it would be another point to take into account to reinforce his security.

But if you put root, and do not say that it is an example, some users can misinterpret it, and think that you are simply doing that directly.

Leaving this , I think that apart from the user's message im I don't know if it would be correct, to create new keys after each login/snap , I exchanged them for the old ones on both machines.
 
freejlr said:
I think that you did not fully understand what I said, I refer to your main message:



If there is more, possibly the one I told you, if I'm not wrong, I'm not telling you that your points are wrong, they seem fine to me.

If I use root as an example, just forget everything I said, I just wanted to tell you that if you are authenticating as root on a remote machine, you run that risk, you run the risk of your snapshots being touched by a super user as root, and they can be removed.

Not the other way, it would be a user without privileges, with the right permissions to receive and send create snapshots. Yes, it's equally dangerous for an attacker to break into your system and run a 0-day, but don't you think that's better than logging in as root? It already granted full permissions, without being able to execute a 0-day.

At no time did I tell him to abandon his proposal and go for mine, I believe or at least I told him this, because it would be another point to take into account to reinforce his security.

But if you put root, and do not say that it is an example, some users can misinterpret it, and think that you are simply doing that directly.

Leaving this , I think that apart from the user's message im I don't know if it would be correct, to create new keys after each login/snap , I exchanged them for the old ones on both machines.
Click to expand...

It is better of course to not run the snaps as root. But between these two situations, which would you choose as more secure:

1. Live has stored credentials for a non-root user to connect to backup server and send snapshots

2. Live has no stored credentials. Backup connects to live to send/receive the snapshots
 
last1 said:
... I look at all these big shops and most of the time their backups also get encrypted in such attacks, because...automation.
Click to expand...
Then their "defense in depth" was defective. Deeply defective!

No competent backup scheme can rely solely on on-line data. It's simply too vulnerable to both internal and external threats.

You should get copies of your transaction logs and backups secured off-line to defend against (potentially undetected) data contamination or loss. This will provide the linchpin of your capacity to undertake confident forensic examination and re-construction, following any "incident". I have been there (with defective SAN directors that contaminated all the on-line storage).

I am not saying that your backup ZFS server is a bad idea. Indeed it's a really good idea -- and the questions you are asking are spot on!

But, for a competent defence, you need more depth... and that needs to be integrated into the architectural design from the outset, because it could change that design.
 
last1 said:
It is better of course to not run the snaps as root. But between these two situations, which would you choose as more secure:

1. Live has stored credentials for a non-root user to connect to backup server and send snapshots

2. Live has no stored credentials. Backup connects to live to send/receive the snapshots
Click to expand...
I have a case similar to yours, what I did was the following, I have a machine A where I have a mirror of 3 disks where I share files with some colleagues. I make the snapshots on a remote machine B where I have an ssh server. What I did was create some keys on machine A and copy them to machine B.

On the server of machine B, disable in sshd the possibility of authenticating by user and PAM password etc... just leave the option of authenticating by public keys/certificates enabled.

I have a configuration with PF, and some security policies such as kern.securelevel, on the other hand, as I told you before, a user without privileges to carry out these tasks without exposing root.

Surely there are a thousand better ways to do it, in any case I'm all ears, but if we refer to ssh security, there are only 3 authentication methods: user/pass, publickey, certificates. Possibly certificates are a better option but I don't know to what extent, if there are simply keys on that machine that are mine, there is no one else.
 
You must log in or register to reply here.
Back
Top