ZFS data loss -- trying to stay calm

gregben

gregben

OK, I'm new to FreeBSD, but I did use BSD back in the old days
around '82. I came to FreeBSD 8.0 for ZFS. Set up a Core duo 2
box with three disks in early March of 2010. One for OS/boot,
the other two are:

Code: 
fbsd40# zpool status
  pool: apool
 state: ONLINE
 scrub: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	apool       ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    ad2     ONLINE       0     0     0
	    ad6     ONLINE       0     0     0

errors: No known data errors

Code: 
fbsd40# zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
apool               276G   180G  71.0G  /apool
apool/family       19.5K   180G  19.5K  /apool/family
apool/home          205G   180G    23K  /apool/home
apool/home/andyb   2.91G   180G  2.91G  /apool/home/andyb
apool/home/averyb  16.4G   180G  16.4G  /apool/home/averyb
apool/home/gregb    186G   180G   186G  /apool/home/gregb
apool/home/thib      18K   180G    18K  /apool/home/thib

Notice that apool/family is only a few kilobytes. It should be
approx 80GB (filled with photos and short home-movie clips).
I was running this machine (fbsd40) for a few months with no
problem (and had no trouble accessing everything on apool/family)
but recently rebooted due to samba setup woes (probably unrelated).
When the machine came back up and I attempted to NFS mount apool/family
the only thing there was single text file I had created at some point.

My hope is that through my own lack of knowledge/forgetfulness I've simply
mounted incorrectly, but I cannot find the data anywhere. Only thing I can
think of is a) data was actually being stored on another disk somewhere
on my network, or b) data was in RAM, or at least the little I looked at.
My fear is that due to some bug in ZFS 13 that the data was written to disk but
is now inaccessible. Help! Oh, yes, I have some of the stuff backed up off-site,
but not the last few months. And another thing, I haven't done any snapshots.
Here's the history:

Code: 
fbsd40# zpool history
History for 'apool':
2010-03-08.10:30:49 zpool create -f apool mirror ad2 ad6
2010-03-08.10:31:24 zfs create apool/family
2010-03-08.22:46:32 zfs set sharenfs=on apool/family
2010-03-08.22:49:55 zfs set sharenfs=-alldirs -network 192.168.1/24 apool/family
2010-03-08.22:51:51 zfs set sharenfs=-maproot=root -alldirs -network 192.168.1/24 apool/family
2010-03-09.00:05:15 zfs sharenfs=on apool/family
2010-03-09.21:23:15 zfs sharenfs=on apool
2010-03-09.21:24:17 zfs sharenfs=off apool/family
2010-03-09.21:28:29 zfs sharenfs=on apool/family
2010-03-09.21:30:02 zfs sharenfs=off apool
2010-03-14.09:40:00 zpool scrub apool
2010-03-16.23:09:56 zfs create apool/home
2010-03-16.23:10:00 zfs create apool/home/gregb
2010-03-16.23:17:25 zfs sharenfs=on apool/home/gregb
2010-03-17.13:29:06 zpool scrub apool
2010-03-21.18:41:34 zfs create apool/home/averyb
2010-03-21.18:41:48 zfs create apool/home/andyb
2010-03-21.18:41:53 zfs create apool/home/thib
2010-03-21.19:09:49 zfs sharenfs=on apool/home/averyb
2010-03-21.19:09:57 zfs sharenfs=on apool/home/andyb
2010-03-21.19:10:01 zfs sharenfs=on apool/home/thib
2010-03-21.23:15:01 zpool scrub apool
2010-03-29.15:54:12 zpool scrub apool
2010-04-05.21:17:34 zpool scrub apool
2010-04-23.09:54:08 zpool scrub apool
2010-05-05.21:39:02 zpool scrub apool
2010-05-05.22:05:16 zpool scrub apool
2010-05-30.22:57:12 zpool scrub apool
2010-06-21.08:25:55 zpool scrub apool
2010-06-29.09:28:00 zpool scrub apool
2010-07-14.21:15:28 zfs sharenfs=on apool/home
2010-07-14.23:28:55 zpool scrub apool
2010-07-28.22:02:03 zfs sharesmb=on apool/home
2010-07-28.22:05:13 zfs sharesmb=on apool/home
 
How did you access the ZFS pool via Samba and NFS?
I was under the impression that those filesharing protocols in ZFS were removed in FreeBSD, but I see you enabled them (or tried to) in ZFS?

Am I wrong??

EDIT:
According to the site below, NFS can be enabled from ZFS but Samba needs to be shared the old way.

http://submesa.com/data/bsd/zfs
 
I would love to know what bsd was like in the 80's =)

I am also new to zfs. I know that it is a good idea to set 2 or more copies of each pool. on pools that have important data set more. my pools are set to 3 copies. I believe I read that the copies also help the fs deal with corrupted data as it goes and pulls from good bits from the copies and replaces the bad bits.

I see your scrubbing often. this a also recommended for at least once a week. even if you don't have data change that often taking snapshots will allow you access to another set of copies which you can `cd` into a rooted .zfs/ dir(in your pool) and see your file structure.

I wish I could help you further.
 
Code: 
fbsd40# zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
apool               276G   180G  71.0G  /apool
apool/family       19.5K   180G  19.5K  /apool/family
apool/home          205G   180G    23K  /apool/home
apool/home/andyb   2.91G   180G  2.91G  /apool/home/andyb
apool/home/averyb  16.4G   180G  16.4G  /apool/home/averyb
apool/home/gregb    186G   180G   186G  /apool/home/gregb
apool/home/thib      18K   180G    18K  /apool/home/thib
Click to expand...

From the above list, there are 71G reside on /apool. Is it possible that your files are on /apool?
 
USED: Size of space used by zfs-dataset and sub-zfs-datasets
AVAIL: size of space available
REFER: Size of space used by zfs-dataset without sub-zfs-datasets

Can you put the output (and remove sensible information) of following commands on pastebin?
[CMD=""]zpool() get all apool[/CMD]
[CMD=""]zfs() get -rH all[/CMD]
[CMD=""]mount[/CMD]
[CMD=""]ls -Hall /apool[/CMD]

Also the samba configuration could be interesting, why did you issue two times # zfs sharesmb=on apool/home and when did the reboot into trouble occur?

Maybe apool/family is mounted on top of /apool/family and hides the family directory that resides in the apool zfs-dataset (as t1066 suggested).

Try to unmount apool/family and mount apool/family to /randomfamily, then check if /apool shows a directory family.
One way to do so is to run # zfs set mountpoint=/randomfamily apool/family.
 
Thanks, and an in-progress update

First, thanks to each of you for replying. This is obviously a wonderful community. :)

I spent most of the weekend resorting to brute-force methods of getting my precious pictures back. The most interesting of which was shutting down the FBSD machine, and running the utility photorec_static (from testdisk-6.11.3) on one of the raw ZFS disks from a machine running Ubuntu 10.04. This recovered most everything, except that a few files were corrupted due to out-of-order-block-allocation. This program scans filesystems/partitions block-by-block looking for file headers and reassembles them. I also used it on some CF and SD cards and it was able to recover some files on CF cards
that had been reformatted.

When I came back here to see if anyone had responded, I was surprised to see so many
nice posts. Anyway, I moved the disk back from the Ubuntu machine into the FreeBSD
machine and rebooted. Everything was the same as expected. /apool/family was basically
empty. I then unmounted, and remounted /apool since t1066 pointed out that /apool
was taking up 71G.

Code: 
# zfs unmount apool
# zfs mount apool

This fixed the problem. Apparently, I had mounted /apool/family atop /apool,
and /apool is where the data I cared about was stored. The thing I have to do now
is study mounting/unmounting of zfs filesystems to make sure I have a correct
setup.

As for Samba, that is another project (trying to get a running Virtualbox VM image to
properly access files on the BSD/ZFS box).
 
You must log in or register to reply here.
Back
Top