ZFS High fragmentation cp(i)ing to empty volume?!

adams · Apr 15, 2015

Much like ethoms in this thread: https://forums.freebsd.org/threads/high-fragmentation-after-zfs-send-recv.50604/#post-283684 I am re-doing my pool by swapping thing around to fix block sizes and also to hopefully resolve some performance issues I was seeing.

My original pool was created in FreeBSD 9.1 many years ago before the 'gnop' trick was known (at least by me). I believe it was reporting around 3% fragmentation before I destroyed it earlier today. Before I did that I did a ZFS send/receive to another volume (zstripe). After the copy I noticed that the fragmentation was out of control:

Code:

NAME  SIZE  ALLOC  FREE  FRAG  EXPANDSZ  CAP  DEDUP  HEALTH  ALTROOT
zstripe  7.25T  5.04T  2.21T  42%  -  69%  1.00x  ONLINE  -

So, thinking that something probably was messing up with send/receive (or per the last comment in the above post, perhaps that fragmentation was always there but under reported). .. I decided to just use cp(1) to copy back to the newly formed zroot vol. However the copy is about to finish (after hours of running) and I'm seeing what I would still consider to be massive fragmentation:

Code:

NAME  SIZE  ALLOC  FREE  FRAG  EXPANDSZ  CAP  DEDUP  HEALTH  ALTROOT
zroot  16.2T  7.58T  8.67T  23%  -  46%  1.00x  ONLINE  -

Especially given that there should be none (right?) since I'm just writing data in a very very serial fashion with cp. Yes it's less than the copy on zstripe but ... why is there any?

I'm at a loss as to what is going on here, any thoughts/insights would be helpful as I'm stuck here wondering if I should re-do the copy or what. Meanwhile my NAS is offline.

P.S. I enabled compression on both zpools but it was not enabled on the original volume.

protocelt · Apr 17, 2015

Hi,

From what I recall 10.1-RELEASE added a new zpool feature called spacemap_histogram which is enabled by default. The zpool-features(7) man page has an explanation of this feature:

Code:

spacemap_histogram

               GUID                    com.delphix:spacemap_histogram
               READ-ONLY COMPATIBLE    yes
               DEPENDENCIES            none

               This features allows ZFS to maintain more information about how
               free space is organized within the pool. If this feature is
               enabled, ZFS will set this feature to active when a new space
               map object is created or an existing space map is upgraded to
               the new format.  Once the feature is active, it will remain in
               that state until the pool is destroyed.

I think what your seeing is expected. The FRAG percentage your seeing in your zpool list output isn't really the percentage of the pool that's fragmented but rather the percentage of free blocks that are less than a particular size. What that size is I don't know. The difference in percentage your seeing before and after the data was copied over is due to the feature only working on new or moved data on the pool. Memory is foggy at the moment, but I believe I got this information from one of the mailing lists a few months ago. It caught me by surprise as well. I'll see if I can't find the mailing list, edit my post and add a link when I get a chance. Hope this helps.

Edit: Here is the link: https://lists.freebsd.org/pipermail/freebsd-current/2014-September/052301.html

ZFS High fragmentation cp(i)ing to empty volume?!

adams

protocelt