snapshot/clone doesn't seem to duplicate filesystem

J

jef

I think I'm doing the right thing here, but perhaps I have misunderstood the process to duplicate a filesystem using ZFS. The intent is to have an exact duplicate of a given ZFS filesystem at a given point in time, when a snapshot was taken. When I follow a snapshot-clone path, the new filesystem does not "match" the original filesystem. It appears that the clone is the same as the snapshot of the original filesystem, but the snapshot does not "agree" with the original filesystem.

Nothing is writing the filesystem in question during this process.

Any suggestions as to how to resolve this problem would be very welcome.

i386 build as of: 9.0-STABLE FreeBSD 9.0-STABLE #2 r237164: Sat Jun 16 17:49:32 PDT 2012
Code: 
[jeff@port16 ~]$ zfs upgrade
This system is currently running ZFS filesystem version 5.
All filesystems are formatted with the current version.

Code: 
[jeff@port16 ~]$ zfs list -r test/sgs4g-rsync/sdcard/Trans16Gc10
NAME                                  USED  AVAIL  REFER  MOUNTPOINT
test/sgs4g-rsync/sdcard/Trans16Gc10  22.9G   153G  4.24G  /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10
[jeff@port16 ~]$ sudo zfs snapshot test/sgs4g-rsync/sdcard/Trans16Gc10@pre-clone
[jeff@port16 ~]$ sudo zfs clone -o mountpoint=/home/jeff/tmpmnt/ test/sgs4g-rsync/sdcard/Trans16Gc10@pre-clone test/cloned
ZFS believes the clone is the same as the snapshot, and the live filesystem is the same as the snapshot
Code: 
[jeff@port16 ~]$ sudo zfs diff test/sgs4g-rsync/sdcard/Trans16Gc10@pre-clone test/cloned
[jeff@port16 ~]$ sudo zfs diff test/sgs4g-rsync/sdcard/Trans16Gc10@pre-clone
[jeff@port16 ~]$
diff, however, tells a different story.

Compare one specific directory in the original filesystem to clone
Code: 
[jeff@port16 ~]$ sudo diff -r --brief /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/ \
/home/jeff/tmpmnt/.android_secure/
Only in /home/jeff/tmpmnt/.android_secure/: com.adobe.reader-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.adobe.reader-2.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.boatbrowser.free-1.asec
Only in /home/jeff/tmpmnt/.android_secure/: com.boatbrowser.free-2.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.clov4r.android.nil-1.asec
Only in /home/jeff/tmpmnt/.android_secure/: com.clov4r.android.nil-2.asec
Only in /home/jeff/tmpmnt/.android_secure/: com.evernote-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.evernote-2.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.farsight.AndroidPinball.javaProject-2.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.flufflydelusions.app.enotesclassiclite-1.asec
Only in /home/jeff/tmpmnt/.android_secure/: com.flufflydelusions.app.enotesclassiclite-2.asec
Only in /home/jeff/tmpmnt/.android_secure/: com.mxtech.videoplayer.ad-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.mxtech.videoplayer.ad-2.asec
Only in /home/jeff/tmpmnt/.android_secure/: com.riteshsahu.SMSBackupRestore-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.riteshsahu.SMSBackupRestore-2.asec
Files /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/com.socialnmobile.dictapps.notepad.color.note-1.asec and /home/jeff/tmpmnt/.android_secure/com.socialnmobile.dictapps.notepad.color.note-1.asec differ
Only in /home/jeff/tmpmnt/.android_secure/: com.usaa.mobile.android.usaa-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.usaa.mobile.android.usaa-2.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.xtakagi.android.memopad-1.asec
Files /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/menion.android.locus.pro-1.asec and /home/jeff/tmpmnt/.android_secure/menion.android.locus.pro-1.asec differ
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: net.androgames.level-1.asec
Only in /home/jeff/tmpmnt/.android_secure/: net.skyscanner.android.main-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: net.skyscanner.android.main-2.asec
Compare snapshot to original filesystem:
Code: 
[jeff@port16 ~]$ sudo diff -r --brief /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.zfs/snapshot/pre-clone/.android_secure/ \
/usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.zfs/snapshot/pre-clone/.android_secure/: com.adobe.reader-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.adobe.reader-2.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.boatbrowser.free-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.zfs/snapshot/pre-clone/.android_secure/: com.boatbrowser.free-2.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.clov4r.android.nil-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.zfs/snapshot/pre-clone/.android_secure/: com.clov4r.android.nil-2.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.zfs/snapshot/pre-clone/.android_secure/: com.evernote-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.evernote-2.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.farsight.AndroidPinball.javaProject-2.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.flufflydelusions.app.enotesclassiclite-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.zfs/snapshot/pre-clone/.android_secure/: com.flufflydelusions.app.enotesclassiclite-2.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.zfs/snapshot/pre-clone/.android_secure/: com.mxtech.videoplayer.ad-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.mxtech.videoplayer.ad-2.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.zfs/snapshot/pre-clone/.android_secure/: com.riteshsahu.SMSBackupRestore-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.riteshsahu.SMSBackupRestore-2.asec
Files /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.zfs/snapshot/pre-clone/.android_secure/com.socialnmobile.dictapps.notepad.color.note-1.asec and /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/com.socialnmobile.dictapps.notepad.color.note-1.asec differ
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.zfs/snapshot/pre-clone/.android_secure/: com.usaa.mobile.android.usaa-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.usaa.mobile.android.usaa-2.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: com.xtakagi.android.memopad-1.asec
Files /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.zfs/snapshot/pre-clone/.android_secure/menion.android.locus.pro-1.asec and /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/menion.android.locus.pro-1.asec differ
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: net.androgames.level-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.zfs/snapshot/pre-clone/.android_secure/: net.skyscanner.android.main-1.asec
Only in /usr/home/jeff/sgs4g-rsync/sdcard/Trans16Gc10/.android_secure/: net.skyscanner.android.main-2.asec
 
So ignoring the clone entirely, it appears that the snapshot is different to the source file system, which as far as I'm aware should be impossible. Snapshots are not at all writable and should be effectively displaying data from the exact same directory trees as the live system.

Can we confirm exactly what's changing:

  • List the contents of the directory on the live file system and store the output
  • Take the snapshot
  • List the contents of the same directory in snapshot

Do they differ at this point, and which is different? Does the snapshot contain different filenames or does the snapshot match your original listing and it's the live file system that's changed?

At the moment my only ideas are something is actually changing the live file system although unlikely as it seems to happen in sync with you taking snapshots, or this folder is some sort of symlink which ends up pointing somewhere else - which I'm not sure is even possible.
 
Code: 
[jeff@port16 ~/sgs4g-rsync/sdcard/Trans16Gc10]$ sudo sudo find -ds . -not -path './.zfs/*' -not -name '.zfs' -ls > ~/sdcard.live

[jeff@port16 ~/sgs4g-rsync/sdcard/Trans16Gc10]$ sudo zfs snapshot test/sgs4g-rsync/sdcard/Trans16Gc10@new-snapshot

[jeff@port16 ~/sgs4g-rsync/sdcard/Trans16Gc10/.zfs/snapshot/new-snapshot]$ sudo sudo find -ds . -not -path './.zfs/*' -not -name '.zfs' -ls > ~/sdcard.new-snapshot


[jeff@port16 ~/sgs4g-rsync/sdcard/Trans16Gc10]$ sudo sudo find -ds . -not -path './.zfs/*' -not -name '.zfs' -ls > ~/sdcard.after-snapshot

[jeff@port16 ~]$ diff sdcard.live sdcard.after-snapshot 
[jeff@port16 ~]$

[jeff@port16 ~]$ diff sdcard.live sdcard.new-snapshot 
3,51c3,48
<      3     4224 ----rwxr-x    1 1000             1015              2161664 Jul 24 16:47 ./.android_secure/aws.apps.networkInfoIi-2.asec
<     49     4224 ----rwxr-x    1 1000             1015              2161664 Jul  8 20:50 ./.android_secure/bbc.mobile.news.ww-1.asec
<     48    31256 ----rwxr-x    1 1000             1015             15999488 Sep 20 09:22 ./.android_secure/com.adobe.reader-2.asec
<     47    14624 ----rwxr-x    1 1000             1015              7483904 Aug  1 12:23 ./.android_secure/com.amazon.mShop.android-2.asec
<     46     8384 ----rwxr-x    1 1000             1015              4290560 Sep  6 10:46 ./.android_secure/com.boatbrowser.free-1.asec
[...]
---
>    143     1550 -rwxr-xr-x    1 root             wheel             2161664 Jul 24 09:47 ./.android_secure/aws.apps.networkInfoIi-2.asec
>    144     1956 -rwxr-xr-x    1 root             wheel             2161664 Jul  8 13:50 ./.android_secure/bbc.mobile.news.ww-1.asec
>    145    27720 -rwxr-xr-x    1 root             wheel            15999488 Jul 31 02:10 ./.android_secure/com.adobe.reader-1.asec
>    146    11608 -rwxr-xr-x    1 root             wheel             7483904 Aug  1 05:23 ./.android_secure/com.amazon.mShop.android-2.asec
>    147     4308 -rwxr-xr-x    1 root             wheel             4290560 Aug 14 04:55 ./.android_secure/com.boatbrowser.free-2.asec
[...]

Only the .android_secure/ directory seems to be impacted.

The files were originally written using rsync from an Android phone. The original filesystem for the files not in .android_secure/ is FAT. rsyncd.conf options include
  • use chroot = true
  • uid = 0
  • gid = 0
  • munge symlinks = true
  • numeric ids = true
At least as I understand it, it shouldn't matter to the filesystem how the files got there, but this may be useful information for understanding the issue.

The 075 permissions and 1000:1015 ownership correspond to those seen on the phone's Linux filesystem.

From the change in dates, as well as the puzzling shift in UID and GID, it suggests to me that somehow the filesystem is holding on to "old" versions in the snapshot. This specific directory is written at a different time than the rest of the directory due to security masking in the Linux underlying Android system (Android mounts a tempfs on /mnt/sdcard/.android_secure/, then mounts .android_secure/ at /mnt/secure/asec/.

No matter how the files are getting there, it is troublesome that the snapshot is not a copy of the live filesystem.
 
So you basically touch files (access them). That's why they are different from the snapshot. Do you have
Code: 
atime=on
on your ZFS systems?
 
You must log in or register to reply here.
Back
Top