ZFS Question - How do I remove excess file copies

R

rajl

I recently built a FreeBSD 9.0 box to use as a fileserver with ZFS. I chose ZFS for its data integrity and self-healing features. ZFS really is the "killer app" for FreeBSD if there ever was one.

After much initial research, I built the system and created a zpool mirror spanning two 1TB hard drives named "zdrive." Within zdrive, I created a zfs filesystem named "home" and set copies=2. I erroneously assumed that I needed to have ZFS create and store extra copies of the files that I saved in order to take advantage of the self-healing properties of ZFS.

I now know that due to the mirror, zfs will self-heal by replacing the corrupted copy on one disk with the non-corrupted copy on the mirrored disk. There is no requirement to have copies=2 on a mirror for zfs to self-heal. If both disks have corrupted copies, something so catastrophic has happened that I will be restoring from an off-site backup anyway.

What is now happening is that I am needlessly saving 4 copies of each file (two copies to each drive in the mirror). This is needlessly redundant for my purposes, and it effectively halves the capacity of my 1TB drives to 500GB. I am running out of space fast.

My question is, how do I fix this without nuking the whole thing and starting from scratch? I tried to remove the excess copies to no avail by issuing the command "zfs set copies=1 zdrive/home". While that eliminates saving redundant copies moving forward, the redundant copies already saved are still on my zfs filesystem. I would like to reclaim this space.

Does anyone have any suggestions?
 
You can create a snapshot of /home and send it to a backup device. You will then destroy your initial pool and recreate it. After that you are ready to receive back the snapshot.

A few things you have to watch out for. When ZFS properties are changed, they only affect new data. Therefore, you might change the copies but as you saw by yourself this didn't affect your existing data.

When you send the snapshot don't use the -R switch because you don't want the properties to be also sent. See zfs() for more information.

If you don't have the necessary space for this procedure, you can still do it by degrading your mirror and using the second disk as a new pool for the transfer. However, you should already know that a mirror is not a backup strategy.
 
Thanks for the direction. I read the man page and think I know how to do what you recommend. Could you please double check what I plan to do to make sure I understand it correctly.

After reading the man pages, I believe this will be a 6 step process:

Step 1: Create zpool on backup device named "zbackup."
Step 2: Issue command - zfs snapshot zdrive/home@backupsnapshot
Step 3: Issue command - zfs send zdrive/home@backupsnapshot | zfs receive zbackup
Step 4: Issue command - zpool destroy zdrive
Step 5: Issue command - zpool create zdrive
Step 6: Issue command - zfs send zbackup/home | zfs receive zdrive

I have two questions about these steps. First, should there be a seventh step in middle where I create a snapshot on the backup device before resending it back to the recreated pool? Also, even if the copies property is not replicated with zfs send, will the additional copies of the files still be sent? I'm a little confused about the difference between data and properties. When the copies property is set to copies=2, are the additional versions of the file that are saved considered a "property" or are they "data."

Thanks for your help, time, and patience.

I am sorry for the confusion.
 
rajl said:
I have two questions about these steps. First, should there be a seventh step in middle where I create a snapshot on the backup device before resending it back to the recreated pool?
Click to expand...

No, because you sent a snapshot of your pool. You will send back the same snapshot.

rajl said:
Also, even if the copies property is not replicated with zfs send, will the additional copies of the files still be sent? I'm a little confused about the difference between data and properties. When the copies property is set to copies=2, are the additional versions of the file that are saved considered a "property" or are they "data."
Click to expand...

They are properties of the pool. Even if you use the -R switch when sending, only one copy will be sent over. The difference is that you would eventually end up by storing two copies on the receiving pool. That is why I said that you shouldn't use it.
 
You can also copy your files into the existing pool and erase the old copies, assuming you have enough space left.

Code: 
zfs set copies=1 zpool
cd zpool
mkdir copy
cp -pR * copy
find -d -name !copy (or whatever)
 
I just had a (maybe) brilliant idea? Would this work:

Code: 
# zfs create -o copies=1 zdrive/temp
# mv /zdrive/home/* /zdrive/temp/
# zfs set copies=1 zdrive/home
# mv /zdrive/temp/ /zdrive/home

Basically, I'm creating a new file-system with copies=1, moving everything over to it, setting copies=1 on the old file-system, then moving everything back in order to reclaim space. Since zfs is copy-on-write, I could see this being extremely fast as no data is actually changing location. Also, I think this would remove the properties attached to the previously saved versions on the file-system.

Am I crazy, or would this work to solve my problem?
 
Datasets are filesystems. The operation would be lengthy but you've grasped the concept that you need to rewrite the data in order to eliminate the duplicated data.

According to your paths and granted the space, you could:

Code: 
# zfs set copies=1 zdrive/home
# cd /zdrive/home
# mkdir temp
# mv * temp
# cd temp
# mv * ..

But essentially the process is the same: the files need to be copied, the duplicated files removed along with their shadow copies and the copied files restored to their intended path.

I'm actually not sure about ZFS' internals, though I would hypothesize that each ZFS' version of an inode points to two physical registers on the drive. There's likely a way to rewrite the ZFS inode table, though I also think that this could cause an interlacement of data on the drive. As much as possible, I like my data to be contiguous, or at least not shredded in such a way. This is likely the reason why it's preferable for the issue to be resolved in this manner.
 
Sceak,

I did exactly as you suggested, based on my idea. The operation took about 5-seconds and did absolutely nothing to remove the shadow copies. My guess is that it merely shifted the inode pointers of the directories inside zdrive/home to /zdrive/temp. Since the operation did not touch the files inside the directories, it did not rewrite data and thus no shadow copies were removed.

I will try the other two suggestions and report back which one, if either, worked.
 
I'm almost positive that you will have to:
  1. create a new pool (or even just format an external drive as UFS)
  2. copy the data from the existing pool to the new one
  3. verify the copied data
  4. delete the data from the existing pool (or even just delete the entire filesystem)
  5. set copies=1 on the filesystem
  6. copy the data back
  7. verify the copied data
  8. delete the new pool
You have to move the data out of the pool, set the zfs property, then move the data back to the pool to force it to write it out with the new properties.

Copy/move inside the pool won't be enough, although crossing ZFS filesystems may do it. But removing it from the pool entirely will guarantee it.
 
rajl said:
Sceak,

I did exactly as you suggested, based on my idea. The operation took about 5-seconds and did absolutely nothing to remove the shadow copies. My guess is that it merely shifted the inode pointers of the directories inside zdrive/home to /zdrive/temp. Since the operation did not touch the files inside the directories, it did not rewrite data and thus no shadow copies were removed.

I will try the other two suggestions and report back which one, if either, worked.
Click to expand...

I was relying on mv's copy/delete mechanism... not sure what happened. I did the test myself and while a move operation didn't perform as I expected a copy/delete/move fixed the issue on my test dataset.

Edit: A quick look at the source reveals that mv, as the epitome of efficiency, will attempt to rewrite file descriptors rather than perform an actual copy operation when possible... who would've thought?
 
^^ well, mv = equivalent of dos RENAME.

I suspect you could perhaps write a script to do something like (psuedocode):

Code: 
for all files
  cp $file $file.bak
  rm $file
  mv $file.bak $file
end for

That way you only need the space for 2 copies of your largest file.

Any reason that wouldn't work? Maybe take a snapshot of your ZFS first? :D
 
What I did was not the best solution, but it was one that worked. I wish throAU had posted earlier. His idea is sheer genius and probably would have been the best in hindsight.

I had copied most of the data previously to a UFS formatted external harddrive. What I did was issue the command [cmd=]cp -an /zdrive/home/ /mnt[/cmd] and let it copy new files to the external harddrive. I then deleted the filesystem with [cmd=]zfs destroy zdrive/home[/cmd] and recreated it with [cmd=]zfs create -o copies=1 compression=on atime=off zdrive/home[/cmd] After that, I issued the command [cmd=]cp -a /mnt/home /zdrive/home[/cmd] to copy everything back.

A few points to clarify:
1) This does notguarantee data integrity. Statistically speaking, the odds of having your data be corrupted during this procedure are slim to none.
2) I used the -n option to avoid overwriting previously backed up files on my external harddrive to save time. However, I found out that the behavior is to avoid overwriting files with the same name and does not take into account datestamps for when the file last modified. It is possible, but unlikely, that I may have lost the most recent version of a previously saved file. Use the -n flag at your peril.
3) I tried to create a snapshot and pipe zfs send into zfs receive. I then realized I did not have enough drive space to do what I was attempting to do, so I canceled the operation. Given more harddrive space, it may work.

In hindsight, if I were not so rusty on the command line or with *nix (i.e. if the thought had occured to me), I probably would have done something like what throAU proposed. Even simpler, since my fileserver only has two users, might be:

Code: 
cp -a /zdrive/home/user1/ /zdrive/home/user1tempdir
cp -a /zdrive/home/user2/ /zdrive/home/user2tempdir
rm -r /zdrive/home/user1
rm -r /zdrive/home/user2
mv /zdrive/home/user1tempdir /zdrive/home/user1
mv /zdrive/home/user2tempdir /zdrive/home/user2

Since my problem is solved, I am marking this thread as solved (or if I do not have permission, I ask that a moderator mark it as solved). Hopefully my experiences help everyone else new to FreeBSD and ZFS.

Thanks for all the help.
 
You must log in or register to reply here.
Back
Top