ZFS Anyone attmpted data recovery?

mefizto

Aspiring Daemon

Reaction score: 13
Messages: 535

Greetings all,

due to several unfortunate events (a.k.a. stupidity), one filesystem was accidentally left out of backup. I was, therefore, wondering if anybody has some experience with any of the recovery tools, e.g., https://github.com/Stefan311/ZfsSpy, Klennet ZFS Recovery software, and the like?

As I do not want to attempt the recovery on the active system, I would first like to replicate the affected filesystem onto a different drive. How do I do that so that I save all (even the missing) data? Use dd?

If worst comes to worst, I can attempt to regenerate the data, but it will be a long and arduous process. So even if I could restore only some of the data, it would be a great help.

Kindest regards,

M
 

ralphbsz

Son of Beastie

Reaction score: 1,721
Messages: 2,690

Copy: Use dd. Probably best to do it one partition at a time, unless the disk is unpartitioned. I would copy onto a blank disk; in theory, you could copy into a file in a different file system, but getting ZFS to mount that later might be gnarly.

Recovery tools for ZFS: I have no idea. In general, doing file system recovery is REALLY hard, and the only good solutions come from the actual developers of the file system, since they tend to be the only ones who actually know what the metadata really means.
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 13
Messages: 535

Hi ralphbsz,

I know that it is close to impossible, I have read a lot about it. But, what do I have to lose, a few hours to try.

Thank you for confirming the use of dd.

Kindest regards,

M
 

wolffnx

Well-Known Member

Reaction score: 122
Messages: 482

Greetings all,

due to several unfortunate events (a.k.a. stupidity), one filesystem was accidentally left out of backup. I was, therefore, wondering if anybody has some experience with any of the recovery tools, e.g., https://github.com/Stefan311/ZfsSpy, Klennet ZFS Recovery software, and the like?

As I do not want to attempt the recovery on the active system, I would first like to replicate the affected filesystem onto a different drive. How do I do that so that I save all (even the missing) data? Use dd?

If worst comes to worst, I can attempt to regenerate the data, but it will be a long and arduous process. So even if I could restore only some of the data, it would be a great help.

Kindest regards,

M

Hi mefizto, at the point I know ZFS dont have a recovery tool
I lost a lot of personal data (for mistake) and try every tool I find
even the comercials on Windows
and nothing...
If you find a solution please post it
good luck
 

sko

Well-Known Member

Reaction score: 251
Messages: 460

Hi mefizto, at the point I know ZFS dont have a recovery tool
I lost a lot of personal data (for mistake) and try every tool I find
even the comercials on Windows
and nothing...
If you find a solution please post it
good luck

How should any windows software help in recovering ZFS data? This toy-OS doesn't even support any proper filesystem, let alone ZFS (yes, there is a OpenZFS port, but it is still highly experimental...)

The ZFS debugger zdb has some diagnostics and recovery capabilities. E.g. it can roll back transactions, which usually is _the_ way to go when trying to recover a pool that has gone sideways and lost data. (of course, if you completely lost the data e.g. by some drives failing, even zdb can't get back data from thin air...)
Lots of functions and options are said to be undocumented and the best way to go when every documented feature has been exhausted are the developer mailing lists.
Most - if not all - functionality ZfsSpy claims to be able to perform should be already covered by zdb and then some. For diagnostics there are also a ton of probes available (way over 6000...) to observe each and any aspect of ZFS to figure out what went wrong and why. Again, the best source of information for debugging ZFS with dtrace is also the ZFS dev mailing list.


Edit: I just remembered; regarding zdb there was a nice post on the delphix blog on some basic recovery functions it can perform:
This post was written at a very early state of zdb, so I don't know how much of it applies today and what has changed or improved since....
 

wolffnx

Well-Known Member

Reaction score: 122
Messages: 482

How should any windows software help in recovering ZFS data? This toy-OS doesn't even support any proper filesystem, let alone ZFS (yes, there is a OpenZFS port, but it is still highly experimental...)

The ZFS debugger zdb has some diagnostics and recovery capabilities. E.g. it can roll back transactions, which usually is _the_ way to go when trying to recover a pool that has gone sideways and lost data. (of course, if you completely lost the data e.g. by some drives failing, even zdb can't get back data from thin air...)
Lots of functions and options are said to be undocumented and the best way to go when every documented feature has been exhausted are the developer mailing lists.
Most - if not all - functionality ZfsSpy claims to be able to perform should be already covered by zdb and then some. For diagnostics there are also a ton of probes available (way over 6000...) to observe each and any aspect of ZFS to figure out what went wrong and why. Again, the best source of information for debugging ZFS with dtrace is also the ZFS dev mailing list.


Edit: I just remembered; regarding zdb there was a nice post on the delphix blog on some basic recovery functions it can perform:
This post was written at a very early state of zdb, so I don't know how much of it applies today and what has changed or improved since....

for my too,is a toy OS but in emergency i look anywere,and aee paid tools
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 13
Messages: 535

Greetings all,

thank you for all the suggestions. I will try the zdb based on sko's recommendation.

Kindest regards,

M
 

achanler

New Member


Messages: 4

[USER=8676]mefizto[/USER]
+1 for making copies of the raw drives or partitions with dd first.

Then attempt to debug and recover on the copies so the original is safe. You can read about recovery I did recently in this post:

 
OP
M

mefizto

Aspiring Daemon

Reaction score: 13
Messages: 535

Hi achanler,

thank you very much for the link, I will study it.

I have been actually stuck on the dd. My (single) pool is raidz comprising three drives, and it has two datasets: music and backup. The one from which data was lost is music, but if I understand correctly, I need to image the entire pool. As I understand, dd works on hard drive, and I read that some people had problems restoring the pool after doing dd on the separate drives.

So, what I decided to do is add a second (separate) pool, take snapshot of the original pool and restore the snapshot on the second pool. I can then work on the first pool. If I am unsuccessful, well, I will have to restore from the original sources, but it will be a lot of effort.

Kindest regards,

M
 

tyson

Member

Reaction score: 16
Messages: 47

I use ZFS for linux/FreeBSD shared pool. Last year thanks to linux inconsistency and my fast fingers I did dd the pool drive with like 1GB of data :D. Was going to write image to usb stick, but it changed letter(go figure linux...).
Photorec from sysutils/testdisk was good as any other free tool out there. Recovering photos works very good, getting back other files could be tricky, but depends how damaged your pool is. Mine was rather badly damaged, but still i could get back some of it.
 

Crivens

Moderator
Staff member
Moderator

Reaction score: 1,325
Messages: 2,287

.

So, what I decided to do is add a second (separate) pool, take snapshot of the original pool and restore the snapshot on the second pool. I can then work on the first pool. If I am unsuccessful, well, I will have to restore from the original sources, but it will be a lot of effort.

Kindest regards,

M
Taking a snapshot and transmitting that one only gives you the data ZFS thinks of as allocated. You will not be able to recover things that way. If it helps, you may dd the drivves into files and then try to recover the drives. If needed, you can get back to before by dd-ing the files back to their source drives.
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 13
Messages: 535

Hi tyson,

thank you for your help.

Hi Crivens,

from my post (emphasis supplied):

"So, what I decided to do is add a second (separate) pool, take snapshot of the original pool and restore the snapshot on the second pool. I can then work on the first pool."

Kindest regards,

M
 

Crivens

Moderator
Staff member
Moderator

Reaction score: 1,325
Messages: 2,287

No you can't do it like that. When you damage some data during recovery that is no longer part of the active file system, but once was (deleted file f.e.) that will not be backed up and thus can not be restored for a second attempt. You then only have one shot.
 

Ordoban

Active Member

Reaction score: 33
Messages: 113

If you want do data recovery, it is strongly advised to mount the file system read only! This is true for all file systems, but especially for ZFS.
The ZFS Uberblock contain up to 64 pointers to filesystem states. One of this filesystem states may still contain your deleted files.
Everything you do with your ZFS pool creates a new filesystem state, and possible overwrites an older state.
So, if you just mount/unmount the pool several times, you may make data recovery much harder!
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 13
Messages: 535

Hi Crivens,

you are, of course, correct that, presuming that I make a mistake, I have one and only one attempt. But, what is my option? I do not have additional several hard-drives to do dd on.

I was wondering if I could, do dd onto a partition. I have one larger hard-drive, on which I could create three partitions that would hold the three hard-drives' contents of the original pool. If something happens during the recovery, I could then dd on the hard-drives from the corresponding partitions. I will look into this.

Hi Ordoban,

that is a good point. Should I mount read-only the pool or the individual data-sets?

On a lightly different topic, it appears that only files with a certain extension (*.flac have been deleted.

Kindest regards,

M
 

Crivens

Moderator
Staff member
Moderator

Reaction score: 1,325
Messages: 2,287

You can do a dd into a file on a file system. If that FS has dedup, you may end up with surprisingly low usage. You may even use a remote storage system.
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 13
Messages: 535

Hi Alexey, Crivens,

seems like great minds (you) think alike. I have reviewed dd(1), however, before I make a mistake, is this the correct command to copy from /dev/ada1 - a drive in the array, to FILE]/dev/ada2[/FILE] - a drive to copy to into a file ada1_img:

dd if=/dev/ada1 of=/dev/ada2/ada1_img

Alternatively, (and better from my perspective) ddonto a server:

account@server:/backup # dd if=/dev/ada1 of=/backup/ada1_img

Does the bs value matter?

Kindest regards,

M
 

Alexey V. Gubin

New Member

Reaction score: 2
Messages: 4

Set bs to 128K, should be close to optimal. It affects speed. Increasing bs improves speed, up to a point.

do not write of= to /dev/disk/filename
/dev is for devices.
specifying /dev/disk/filename is a very bad idea

second command, where you write to /backup, looks correct.
 

Ordoban

Active Member

Reaction score: 33
Messages: 113

that is a good point. Should I mount read-only the pool or the individual data-sets?
You should mount the whole pool as read only.

Does the bs value matter?
Yes bs matters, but only in speed.
dd reads a pice of (bs)bytes, writes a pice of (bs)bytes, reads a pice of (bs)bytes, writes... ...until everything is copied. Every read and every write process has some overhead. So if you chose bigger bs values, you have less overhead. Also, if you dd data on the same harddisk, the disk's head motor has to move less.
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 13
Messages: 535

Hi Alexey,

thank you for the suggestion about bs. Since my zfs recordsize is 1M, I thought that it would be a good idea to match it.

I am little confused about your comment that "specifying /dev/disk/filename is a very bad idea". Why? How do I specify the hard drive onto which I want to dd the pool's hard-drive?

Hi Ordoban,

thank you, I have mounted the entire pool as read only.

Thank you for the bs explanation. It appears that it has no relationship to the zfs recordsize.

Kindest regards,

M
 

Alexey V. Gubin

New Member

Reaction score: 2
Messages: 4

If you want to dd into a file, you don't specify a drive.

See, your source is a drive. So you specify if= as a drive, if=/dev/diskID

Next, your target is a file. So you specify of= as a path to file. /dev/whatever is not a path to file, it is path to a device, because of, well, /dev/. If you are copying disk to disk, then of=/dev/anotherDiskID is good, but you are not doing that. You need something along the lines of ~/DiskImageFileName if you want to place it in your home directory, or whatever other location where you would normally place a file.

and you need to make sure your pool is best not mounted at all at the time of copying, or at least mounted read-only, and you are not writing onto drives in the pool.

For a dd process to work, you need to have your pool not modified between start of the process when the first drive is started to image, and the end of the process, when the last drive is imaged. If something is written onto the pool in the middle of the imaging, you end up with an inconsistent image.
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 13
Messages: 535

Hi Alexey,

thank you for the answer. I have s false impression that I now understand.

Kindest regards,

M
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 13
Messages: 535

Greetings all,

a gentleman contacted me via p.m. and recommended to try the above-mentioned Klennet ZFS Recovery software. Although [USER=49061]sko[/USER] did dismiss it as a "toy-OS" appeared to analyze the pool and, more importantly, appear to indicate TXG groups that appear to have about 90%+ of the files in the non-backed filesystem.

The software, being paid for, does not - appropriately - let one select and copy the recoverable files. Since I do not feel that the data is worth the license price, I have done some reading, and it appears that it is possible to mount a specific TXG group: https://lists.freebsd.org/pipermail/freebsd-hackers/2013-July/043131.html. However, I cannot find the -T option in zpool(8).

Does anyone have any experience with it? Where would I ask?

Kindest regards,

M
 
Top