Restoring deleted file from ZFS file system

Hi All,

This morning I have accidentally deleted some files in a certain directory on my ZFS storage system. I am wondering if there is a way to get these back? I didn't have an off line backup of these files, so I am wondering if I can use some file system restore tools?

I am not sure if I had auto snapshots setup, however is there a way to check and if i didn't have snapshots setup can i recover the files from the ZFS file system?

Thanks.
 
Last edited by a moderator:
Check the output of $ zfs list -t snapshot poolname/filesystem to see if there are any snapshots of the filesystem. If there are, you can copy the file from /path/to/filesystem/.zfs/snapshot/snapshotname/path/to/file/ (replace /path/to/filesystem and /path/to/file).

If you do not have snapshots, the file is gone.
 
Thanks Freddie,

Here is the output:
Code:
[root@server-01]# zfs list -t snapshot
no datasets available
[root@server-01]#

I guess I am out of luck as I had no snapshots setup on my system. :(
 
SirDice - well, it was an accident. We are 2 users with rwx on a /www/test.site.com directory and when I was transferring a 3 Gb ISO file at half of it transferr was stopped and I saw that entire directory structure of test.site.com including the /tmp directory where I was copying the file was deleted by my coleague to upload the site with an updated platform of it. But I lost the *.filepart of 1.5Gb...:( This was a ocassionally file transferr so no backup was made to it.

In other order of ideeas: in Windows OS (any) you have many free or paid software for recovery of data, under Linux/Unix-like OS I don't know or hear/saw anything about it.
 
Seems incredibly strange to me to resort to looking up recovery solutions just for a file transfer that didn't complete...

Regarding accidentally deleted files, snapshots are one of the primary features of ZFS, and if you're running ZFS, you may as well take regular snapshots even if you don't really have a use for them. At the least it allows you to go back and retrieve removed files, or something you've managed to mess-up/overwrite (assuming the file existed when a snapshot was taken).

Also, there is no auto-snapshoting built in. ZFS or FreeBSD have no interest in managing snapshots for you. You either have to take snapshots manually, write your own cron script, or use one of the many ports.
 
In other order of ideeas: in Windows OS (any) you have many free or paid software for recovery of data, under Linux/Unix-like OS I don't know or hear/saw anything about it.
Because, generally speaking, the Linux/Unix-like systems are administered by people that actually understand the significance of proper backups. So there's very little need for it.

That said, there is some recovery software like sysutils/autopsy/sysutils/sleuthkit. But as far as I know it doesn't support ZFS. In any case it's going to be a long and tedious exercise with zero guarantees you'll be able to recover anything. Having to spend several days on analyzing is the prime reason why everybody else simply restores from backup, as that usually only takes a few minutes.

https://www.joyent.com/blog/zfs-forensics-recovering-files-from-a-destroyed-zpool
 
Real operating systems don't have training wheels. You fall off, you get back on again and (hopefully) you learn not to make the same mistakes again.
It really boggles the mind that anyone can know enough about computers to go through the trouble of installing ZFS (or an entire OS that comes with it), and not know enough to back up their data! How exactly does that work?

To be perfectly honest, the number of people asking about "data recovery" online is really frustrating. They screw up, and then it becomes everyone else's problem to get their data back.
 
"...and not know enough to back up their data! How exactly does that work?"

Because things still happen some times, that is how.
I am a life-time server operator and have at times lost files, and googled for if there is a way to get it back.

Does somebody claim that they could not delete files by mistake and thereby having lost them?

"They screw up, and then it becomes everyone else's problem to get their data back."
So, if I have a problem and ask if there is a way, then it 'it becomes everyone else's problem'?

I can't believe the unfriendlyness in this forum.
But thanks to phoenix, usdmatt and after all SirDice who wised up and gave a helpful reply.
 
First of all please keep in mind that you're responding to a thread which is nearly one year old. Some of the messages you responded to are even older, so there's really not much sense to it because it remains to be seen if those users are still active.

Also: if you want to comment on someone it helps to use the quote feature (or multi quote if you want to reply to multiple messages). This way the users will actually get an alert that someone commented (which increases the chance of getting a response).

Does somebody claim that they could not delete files by mistake and thereby having lost them?
That's really not the point. The point is mostly about people not thinking ahead. A filesystem like ZFS makes it really easy to avoid problems like these, but if people don't use them then yeah... You can't really "blame" the filesystem or its features for that, which is what most people tend to do.

I can't believe the unfriendlyness in this forum.
Being direct is not the same as being unfriendly. This thread didn't even come close in my opinion. With technical stuff it's even preferred to be direct in order to avoid the possibility of creating extra confusion.

There's also something as perception and intent. Just because someone is being direct doesn't mean he intended to be unfriendly. Instead of immediately assuming the worst you could also pick it up for what it was: a mere comment on a forum.

There's also something as being over sensitive and incapable to handle criticism. If that applies then, sorry to say, but the technical Internet as a whole (referring to technical fora) are probably not the best places to roam. Because people tend to be direct for reasons mentioned above.
 
That's really not the point. The point is mostly about people not thinking ahead. A filesystem like ZFS makes it really easy to avoid problems like these, but if people don't use them then yeah... You can't really "blame" the filesystem or its features for that, which is what most people tend to do.

As for me, you can be as much direct as you wish, but the point is you are wrong! in your way of thinking!

NOBODY!!! can think ahead of situation which is just happened. In my >20 years experience with *nix I've never had an issue with
files I deleted years ago. Even I would fall into it I would probably never remember, that something has been deleted.

In 99.99% of cases you just delete files without intention and that moment you recognize that this has happened.
Also usually you did something with that file or group of files at that very last moment, before deleting them.

For example it just has happened to me now. I never meant delete my PG config files instead I did this:
Code:
pg_dropcluster --stop 9.1 main

Now all my config files from.... /etc/posgressql just gone. To me it's a bug in software it cannot delete any config files without implicit confirmation from user. But now it became ZFS issue.

If I cannot restore all these files which is very easy on very old FAT, whom I have to blame?

Your position to blame me for not doing a backup. Are you personally doing backups every 2 minutes??? - Probably not.

Therefore you are totally wrong! Or better to say you have no clue. I am not too direct?

So I suggest you don't talk to people like that. You aren't more clever than others. Therefore you either help or stay calm = shut up.

P.S. If all those who blamed people for not doing backups would know just a little how really almost all FS work, they would fully understand
the wish to undelete just deleted file. Almost none of them really delete files. Therefore FAT and EXT files systems do have undelete feauture as long as you don't use freed blocks again. And that IS the case if you just deleted a file.
 
Well, I find it sad that HAMMER is available only on Dragonfly BSD :)
With HAMMER you just could look up the version history of the file, and take the one you like best...

With a cronjob doing regular ZFS snapshots one can approach that HAMMER functionality somewhat, though.
 
Not only HAMMER. Absolutely EVERY file system has this capability. Only encrypted file systems with additional security do really delete files.
And that it a nightmare in terms of performance. Nobody did it for ZFS, just because of ignorance.
I am not happy, but can live without it. Have now to spend another hour or more to recover text file I deleted.

I just cannot tolerate answers of idiots who think of themselves like they are guru and capable to teach people.
Instead they just have no clue!

The proper answer in such a case would be -"Hey, sorry for what has happened, but ZFS has no such utility to recover your data."
Can be used as template I provide it on BSD License:p

P.S. Also sorry for my style. I was pissed off by ZFS. Have installed it specially on a MIRROR to improve storage safety. Have spent several days
to make it right. And have got a surprize, which I would never see on my EXT4 I have everywhere else. Argh..:rude:
 
Didn't know that restoring of deleted files would be easy on ext4 or any unix filesystem... thought deleted=gone.
Thought that this would be one of the things HAMMER introduced... one can view it as sort of filesystem version control system, even as forensic tool to view what malware on VMs changes, for example :)
 
Yeah, it should be possible to get approximate functionality from ZFS by careful snapshot and prune policy. With that being said I think it would be extremely beneficial having HAMMER in FreeBSD as it has some tools which are simply better for this kind of thing. Not to mention a better implementaton of deduplication.
 
Well, this thread seems to be getting more hilarious by the day. But sorry to say but the days of MS-DOS' FAT16 which provided a pretty static File Allocation Table (which made it practically impossible not to spot the location where missing files could have gone) is long after us.

Yet even with FAT16/FAT32 it's pretty easy to lose your files, for the simple reason that once the FAT gets updated then best of luck to you finding all the bits and pieces of your data, especially when they're scattered on a fragmented filesystem.

Hmm, what was that saying again? Talking without being hindered by any factual knowledge? Something like that :D

Bottom line: think ahead, make backups, don't blame the system for your own short comings. Unless of course you're working for GitLab, in which case I guess it's part of your job.
 
It seems that some people without a clue in everything are most aggressive to teach others! They just can't stop do it even
after they were proven of being wrong!

Please, answer messages only if you wish to HELP! not to blame other people. You definitely do not know their situation
and can not charge on it. To do that one need to have certain level of qualification in the topic. Obviously it's absent in your case.

It seems FBSD has a real problem to survive in that world. Linux users seems to be much more educated what concerns computers, OS and FS.

Just links: https://wiki.archlinux.org/index.php/file_recovery https://www.cgsecurity.org/wiki/Advanced_FAT_Repair
Key words: extundelete, testdisk

Such tools are existed for almost every FS, if it has a certain level of popularity. They all are based on simple principle - none of FS do really delete files!
If immediately after deleting a file one would try to recover it the success probability is practically 100%.

Almost anyone who were using Linux more than a year usually know about them. They work perfectly well!
Both helped me to save many files over long time.

P.S. Now I am ready to listen either a prove, that I am wrong, or an excuse about being not well informed.
 
Didn't know that restoring of deleted files would be easy on ext4 or any unix filesystem... thought deleted=gone.
Thought that this would be one of the things HAMMER introduced... one can view it as sort of filesystem version control system, even as forensic tool to view what malware on VMs changes, for example :)

Snurg, please, see my previous message in that thread. I have provided links for programs you can use.
Such tools are existed since 90x! Possibly even earlier. I had no PC in 80x, therefore do not know.

The tool "extundelete" has it's name based on DOS tool "undelete" which was very poplular many years ago.

Unfortunately, those local experts seems to be totally unqualified, so if you wish to learn something real, try to avoid them. Good luck.
 
It seems FBSD has a real problem to survive in that world. Linux users seems to be much more educated what concerns computers, OS and FS.

ZFS is irrelevant from FreeBSD. ZFS was ported to FreeBSD and is currently being also used in Linux too.
Perhaps it would be better for you to just use ZFS on Linux.
 
Such tools are existed for almost every FS, if it has a certain level of popularity. They all are based on simple principle - none of FS do really delete files!
If immediately after deleting a file one would try to recover it the success probability is practically 100%.
Have you actually tried using any of them in a real world situation? I have. From my own experience I can tell you the success rate is far less, and is probably closer to 50/50.

In a real world situation you can't just turn off a production system because some id-10-t removed a file they shouldn't have. And just the act of shutting down a system could cause writes (caches being flushed to disk) that could overwrite the old file data. Yeah, you can risk it and forcefully power down the machine. But that could result in even bigger problems (databases for example) than just a lost file.

The advice is simple, don't rely on magical tools. Rely on your backups. Because at the end of the day, that's going to save you.
 
Hammer is by far not the first file system with snapshots. It's snapshot mechanism is more fine-grained than common, but it pays a price for that in that delete operations are handled badly (slowly, inefficiently). The other well-known file system with fine-grained snapshots is WAFL (which is only available inside NetApp filers, meaning NFS appliances); WAFL is so well-engineered that it gets very good performance and a complete CoW (copy-on-write) deisgn. Yet it does not have undelete support. The most extreme example of a file system with snapshots was "Elephant - the file system that never forgets" (look up the research papers about it). Alas, Elephant was never made into a full production file system that shipped with any operating system, and remained a (glorious) research project.

All engineering is an art of making compromises. Implementing a file system is a compromise between performance, durability, and features. Snapshots are one ingredient in such a compromise. It is possible to make extremely fine-grained snapshots, such as in Elephant, but the result is not production worthy. As SirDice said: If you want protection against unintentional delete, use backup and snapshots.

Tools that try to undelete files after the fact are voodoo. They may work some of the time, but they are just unsafe. If they could be implemented efficiently and safely, they would be, but the fact that file system developers do not release those tools to the public should tell you that they should not be used. With expert help, it is sometimes possible to retrieve files that have been lost and verify their correctness. I've had to do that for file systems I worked on, but I would refuse unless I get a large paycheck, and there is a strong indemnification agreement that makes sure I'm not liable if something goes wrong.

A poster above claimed that Linux is better than FreeBSD because it has such tools. That poster is so wrong that it would be better for FreeBSD if he goes back to using and hyping Linux.
 
I'm also wondering why this 8 year old thread needs to be hijacked anyway. Its obvious that the current topic is totally unrelated to that of the OP because many things changed in 8 years time. Instead of hijacking an existing thread and making a nuisance of yourself (which, on a personal impression, makes you really stand out as a troll) it might be a better to start a new one to fully explain the current issue at hand.

Provided this is actually related to FreeBSD of course, something which I now seriously question.
 
The advice is simple, don't rely on magical tools. Rely on your backups. Because at the end of the day, that's going to save you.

1. The main point of discussion was how to properly answer to questions people are asking.
No backup, no preparation could do anything about what people are asking. None of us is able to predict the future and what would happen next.

And I insist it's a bad attitude trying to teach people, who were not asking any one to teach them!!!
Is it so difficult to understand? Reading this forum or Stack Exchange/Overflow I am always wondering where those people come from?

Somebody is asking "how to perform a certain task" and very common answer is "You don't need it" or "Why do you need it?"
or "It's a bad idea - never do it" etc.

Who ever told you, that you do have enough expertise to give such suggestions? In many cases the first answer is something like it doesn't work and the second one somebody who really knows provide the solution.

Why it's so difficult to keep quite if you don't know the answer? Or at least you are willing to say something nobody is asking about???

Imagine some one would ask you what is better - doing backups or use an "undelete" program for files recovery.
Then it would be OK. You provide your opinion. Although we all know those questions are not allowed at SO/SE... Think why it is so?
Because too many so called experts are trying to convince people in some stupid things.

2. Regarding backups! That was not a topic! Nobody is denying the need of backups, but they aren't useful in many cases!
It's not without a reason that Microsoft has introduced the "Recycle Bin" technology which is mad simple to prevent loosing files in exactly
such circumstances. And now after 20+ years I am reading that is called "a Magic Tool"...

Do you really wish to say, that Windows 3.11 is more advanced than today's FreeBSD?????? - OMG...
 
Back
Top