Solved ZFS Snapshots and Ransomware Attacks

A while ago, I was called to help by someone who had been a victim of some kind of ransomware. I told him I could not help him because he did not have any backup. I told him he needed a Samba server...

But then I thought, if a Samba share were online on a Windows system either as a link or a network disk at the time of a ransomware attack, it would infect those files too. That is how I would program the cursed thing!

So, I thought I could write some sort of script that would run when the server is turned on and take periodic snapshots of the Samba share (mapped to a ZFS file system), but then I found out that all such ransomware would encrypt a portion of a file in place – it would not simply create new encrypted files from existing ones and then delete them.

So my question is:
If I took a snapshot of a ZFS file system and a few minutes later got my files encrypted by some ransomware, would I still have the unencrypted versions of my files in the snapshot? In other words, can I recover my files from a snapshot no matter what happens to the the ones in the ZFS file system?
 

I have already seen it. The one thing in that article that I am doubtful of is this:
CryptoLocker attacks work on the file level rather than the block level, ...

I read from another article that that is not the case. Well, maybe some work on file level and others on block level. What we are sure about is that a snapshot would keep our files safe if the ransomware attacked on file level. But what would happen if it were on block level?
 
The ideal solution is to send the snapshots to a remote machine (or at very least an external HD). So, doing that doesn't matter how the attack works, one would just need to reinstall the system and rollback the snapshot. :)
 
The ideal solution is to send the snapshots to a remote machine (or at very least an external HD). So, doing that doesn't matter how the attack works, one would just need to reinstall the system and rollback the snapshot. :)

Even more ideally, it would be better to use rsync to regularly backup all the data to some other location inaccessible to the ransomware. But I still can't help wondering, you know. What would happen to a file in a snapshot if the file it refers to were to be tampered with at the block level? Is there any safe way we can try that? Without risking our VM host machine, that is...
 
Last edited by a moderator:
I see little reason using rsync instead of send/recv when using ZFS. (Unless you specifically want to store your backups on a non-ZFS file system). They effectively accomplish the same thing but ZFS send is FAR more efficient and without any additional effort you get versioned backups on the receiving system; You don't have to bother taking snapshots on the destination - each snapshot you send continues to exist on the backup unless you delete it.

If ransomware is file-based then it will not affect snapshots. There's always the possibility that the author could write their utility to do a simple zfs destroy of snapshots if it detects ZFS file systems, but that would need root access.

If it were block level, as in it encrypted blocks on /dev/ada0 directly for example, then it would encrypt absolutely everything. This would also need root access of course. I don't know if any ransomware actually does this - The major issue I see is that they usually want to give you some option to pay for decryption, and target only user data (images, documents, etc); Usually achieved by simply traversing folders looking for certain file extensions. When accessing block devices directly, unless the ransomware had its own built in file system (NTFS/UFS/ZFS/whatever) implementation, it wouldn't really know what it was encrypting and would be likely to render the system unusable by encrypting core system files, which isn't that useful for them. They want the base system to continue to run normally and provide some sort of warning that the user's data is encrypted, along with some sort of contact/payment information for decryption.

This is all based on the ransomware attacking the ZFS server directly. If running on a Windows client, it will have no write access to ZFS snapshots. Any encryption of file or block data on Windows would be irrelevant to the snapshots. A backup after the malware will backup encrypted files, but the previous snapshots will be untouched.

In general a ZFS machine running Samba, taking regular snapshots, is a perfectly reasonable way to backup data from client machines. For the full belt-and-braces you'd probably want a second ZFS machine storing a copy via send/recv as well, ideally offsite (especially if it stores data from a large number of clients). If possible, when a machine containing important data fails, you don't want to be left relying on a single copy of that data. If the backup machine fails, you don't want to have all the clients with no backup.
 
In general a ZFS machine running Samba, taking regular snapshots, is a perfectly reasonable way to backup data from client machines. For the full belt-and-braces you'd probably want a second ZFS machine storing a copy via send/recv as well, ideally offsite (especially if it stores data from a large number of clients). If possible, when a machine containing important data fails, you don't want to be left relying on a single copy of that data. If the backup machine fails, you don't want to have all the clients with no backup.
Additionally (for any server running SAMBA, not just FreeBSD + ZFS), you can simply set read-only files and directories to have an ownership that doesn't map to Windows clients. As long as the files have group (or world) read (but not write) clients in the appropriate groups (or world) will be able to read them but not modify them. Most of my data consists of static photographs, videos, software distribution files, etc. where world-R is appropriate.
 
This is all based on the ransomware attacking the ZFS server directly. If running on a Windows client, it will have no write access to ZFS snapshots. Any encryption of file or block data on Windows would be irrelevant to the snapshots. A backup after the malware will backup encrypted files, but the previous snapshots will be untouched.

In general a ZFS machine running Samba, taking regular snapshots, is a perfectly reasonable way to backup data from client machines. For the full belt-and-braces you'd probably want a second ZFS machine storing a copy via send/recv as well, ideally offsite (especially if it stores data from a large number of clients). If possible, when a machine containing important data fails, you don't want to be left relying on a single copy of that data. If the backup machine fails, you don't want to have all the clients with no backup.

Well, I think that settles the matter. Thank you very much indeed.
 
There is one other thing to consider as well...

Some of the more sophisticated ransomware will install itself as a rootkit and quitely encrypt your files, decrypting them on the fly when opened so the operation is completely transparent. After...say a month, it springs the trap on you. They do this so the backups are also useless because by the time it springs the trap, the backup tapes would have started to have been rotated...at least that's the theory. Ransomware usually only targets Windows because that is what most people are running. I think there's one or two Mac OSX ones out there, and some that run on your smartphone, but the big target is Windows. They wouldn't be going after the servers unless they were going to steal the data.
 
Some of the more sophisticated ransomware will install itself as a rootkit and quitely encrypt your files, decrypting them on the fly when opened so the operation is completely transparent. After...say a month, it springs the trap on you. They do this so the backups are also useless because by the time it springs the trap, the backup tapes would have started to have been rotated...at least that's the theory.

What do you suggest?
 
I started writing a reply a few days ago to suggest that if you use ZFS snapshots, you may as well store months if not years worth. For most use cases the storage required would likely not be an issue as snapshots really don't use that much space. However I realised that being able to restore an old copy of a file that's been corrupted for a month or longer may not be that useful if it's something worked on regularly (possibly fine for archives of images/etc though).

Does raise the interesting question of how does it know you're using the file and not backing up though? If it decrypts on the fly when I open a file so I don't know it's been encrypted, what happens if I use robocopy to copy that to another machine? Is it able to see the command being used and not decrypt when it sees certain commands, or would my robocopy process get the decrypted version?

All makes me a bit dubious about how much these block-level or super clever "on-the-fly decryption" variants actually exist in the wild; A few Google searches doesn't show any obvious ransomware using these tactics.
 
What do you suggest?

On Windows, a good antivirus is essential.

I started writing a reply a few days ago to suggest that if you use ZFS snapshots, you may as well store months if not years worth. For most use cases the storage required would likely not be an issue as snapshots really don't use that much space. However I realised that being able to restore an old copy of a file that's been corrupted for a month or longer may not be that useful if it's something worked on regularly (possibly fine for archives of images/etc though).

Does raise the interesting question of how does it know you're using the file and not backing up though? If it decrypts on the fly when I open a file so I don't know it's been encrypted, what happens if I use robocopy to copy that to another machine? Is it able to see the command being used and not decrypt when it sees certain commands, or would my robocopy process get the decrypted version?

All makes me a bit dubious about how much these block-level or super clever "on-the-fly decryption" variants actually exist in the wild; A few Google searches doesn't show any obvious ransomware using these tactics.

If *I* was writing these things, there are two functions in the Windows API called ReadProcessMemory and GetWindowText to find out what's running on the system. If it's a backup program, then don't decrypt. It it's say, Office then decrypt. If it's Sytos or NTBackup, then do not decrypt.

One way to get around this is to have a script run an MD5 comparison on the original file and the backup file on a daily basis. If the hashes differ, then you have a problem. Malware is evolving all the time. You have to stay 2 steps ahead of it.
 
On Windows, a good antivirus is essential.

No antivirus suite would be able to protect a machine from a careless user, right? And this is not a discussion about Windows protection anyway.

If *I* was writing these things, there are two functions in the Windows API called ReadProcessMemory and GetWindowText to find out what's running on the system. If it's a backup program, then don't decrypt. It it's say, Office then decrypt. If it's Sytos or NTBackup, then do not decrypt.

In order to be able to do that, the bloody thing would have to know the names of at least all the major file backup/move applications, if not all. In addition, it would have to remember which portions of the file to decrypt. I am not looking for a solution on the Windows platform anyway. I want to make ZFS take care of the back-up process.

One way to get around this is to have a script run an MD5 comparison on the original file and the backup file on a daily basis. If the hashes differ, then you have a problem.

That scheme would fail if the user edited the file. My script would run on the Samba server and be taking regular snapshots only – it would be too complicated and time-consuming a task to go through all the files in a share. How would I hashcheck a file on a ZFS file system anyway?


Malware is evolving all the time. You have to stay 2 steps ahead of it.

That is what I am trying to achieve in the long term. The solution must be simple and reliable, no matter how clever the malware might be; and also absolutely NOT Windows-based.
 
Well, I realize it's bad form to discuss Windows security in this forum. The best way to protect yourself I guess would be to keep backups offline. Bring them online for the backup, and then take them off. There are no easy solutions that I have think of.

No antivirus suite would be able to protect a machine from a careless user, right? And this is not a discussion about Windows protection anyway.

Agreed on both counts.

That scheme would fail if the user edited the file. My script would run on the Samba server and be taking regular snapshots only – it would be too complicated and time-consuming a task to go through all the files in a share. How would I hashcheck a file on a ZFS file system anyway?

I see your point. You would have a problem if the user was editing a file on a shared resource. What I was thinking was if the user had the file locally, and then a backup program ran on the Windows machine and uploaded it to the server, the server could log into the Windows machine and do MD5 comparisons to make sure the backup was done with no errors. That goes out the...window...if the user is editing a shared resource.

To answer your question, the same way that you do on all filesystems. md5 <filename> or sha256 <filename>.

That is what I am trying to achieve in the long term. The solution must be simple and reliable, no matter how clever the malware might be; and also absolutely NOT Windows-based.

No easy solution, I agree...but I think I might have come up with something. I just checked it out. FreeBSD has something called a file command that will actually look at the file types. From my testing, it will properly recognize filetypes of MS Office, Open/LibreOffice, PDF, and others. So you could run a script or program that checks the filetype. So if you have a .doc file that comes back as data instead of a MS Word document, then something is wrong.

That way, if the ransomware is doing on-the-fly crypto, then it will be discovered during the backup...if nobody complains about the file being corrupted first.
 
To answer your question, the same way that you do on all filesystems. md5 <filename> or sha256 <filename>.

I know how to issue the command. But for that to work, I would have to have a string to compare against. I cannot keep an SHA256 checksum for each and every file. Doing so would complicate matters.

No easy solution, I agree...but I think I might have come up with something. I just checked it out. FreeBSD has something called a file command that will actually look at the file types. From my testing, it will properly recognize filetypes of MS Office, Open/LibreOffice, PDF, and others. So you could run a script or program that checks the filetype. So if you have a .doc file that comes back as data instead of a MS Word document, then something is wrong.

Such a utility would have to open each file to check its contents. Again, that would be a radical deviation from the KISS principle.

That way, if the ransomware is doing on-the-fly crypto, then it will be discovered during the backup...if nobody complains about the file being corrupted first.

That is exactly what I want to prevent. What use is a protection system if you find out that it does not do its job only after the loss is suffered?


Well... After giving the problem some thought, I have come to the conclusion that there is no %100 fail-proof solution. It seems my best option is to make the system keep as many snapshots as possible. I am in the process of learning Python 3.6 and I am working on a configurable service that will take regular snapshots of list of given ZFS file systems. I would love to share it with interested parties and get some professional feedback.
 
Back
Top