Content-type: text/plain; CHARSET=US-ASCII; format=flowed
References: <4B9D3FF9.40109@aristo.tau.ac.il>
<3BEB5F9B6605A140847DEEEDF7439C5D03D2A4E5@CORPUSMX50A.corp.emc.com>
User-Agent: Thunderbird 2.0.0.24 (Macintosh/20100228)
Message-ID: <4BA27FDA.40605@sun.com>
Date: Thu, 18 Mar 2010 20:32:42 +0100
Reply-To: EMC NetWorker discussion <NETWORKER@LISTSERV.TEMPLE.EDU>,
Attila.Mester@SUN.COM
Sender: EMC NetWorker discussion <NETWORKER@LISTSERV.TEMPLE.EDU>
From: Attila Mester <Attila.Mester@SUN.COM>
Subject: Re: ZFS deduplication.
Comments: To:
lemons_terry@EMC.COM
In-Reply-To: <3BEB5F9B6605A140847DEEEDF7439C5D03D2A4E5@CORPUSMX50A.corp.emc.com>
I have recently made similar tests to see what dedup ratios I can get
when writing to dedup enabled ZFS filesystems as a B2D target.
I have discovered the same behavior, e.g. NetWorker saveset streams do
not really get deduplicated. Changing the underlying ZFS dedup blocksize
doesn't have any positiv effect, even if I go down to 8k blocksize.
The only explanation I can think of is, as Yaron already pointed out,
block boundaries are aligned differently because of the different
meta-data which is part of the savesets.
It seems, at current point of the code implementation the dedup can not
deal with different block bounderies and as such, not effectively usable
for deduplicating such streams.
BTW I experienced the same thing when using NetBackup as the backup application.
regards -attila
********************************************************************
Attila Mester 5 Digit Sun internal: x62534
Data Protection Architect Tel: (+49 89) 46 008 2534
Sun Microsystems GmbH Fax: (+49 89) 46 008 2583
Sonnenallee 1 Mobil: +49 172 812 5947
85551 Heimstetten / Germany mail:
attila.mester@sun.com
********************************************************************
Terry Lemons schrieb:
> Hi Yaron
>
> I know that NetWorker has a default block size for each of its output devices. This default block size can be overridden; details on this are in the NetWorker Administration Guide.
>
> Does ZFS have a specific block size that it deduplicates? If so, could it be that the NetWorker default block size for the AFTD device is not the same as, or a multiple of, the ZFS default block size?
>
> tl
>
> -----Original Message-----
> From: EMC NetWorker discussion [mailto:NETWORKER@LISTSERV.TEMPLE.EDU] On Behalf Of Yaron Zabary
> Sent: Sunday, March 14, 2010 3:59 PM
> To:
NETWORKER@LISTSERV.TEMPLE.EDU
> Subject: [Networker] ZFS deduplication.
>
>
> A few days ago I read a post to EMC's Networker forum by Nicholas
> Bone (
https://community.emc.com/thread/99839?tstart=0). He reported a
> test he performed with AFTD which was running on top of ZFS with dedup.
> Unfortunately, he wasn't able to get any reasonable dedup ratios (1.03
> for three full savesets of the the same file system). My conclusion was
> that Networker does not align files at block level, which confuses the
> ZFS dedup code. Is anyone familiar with some flag or any configuration
> option which will convince save or AFTD to do the right thing so that
> ZFS will be able to find identical blocks ?
>