ZFS ZFS and xattr=sa

CyberCr33p

CyberCr33p

I create a new dataset in a FreeBSD server using OpenZFS 2.1.4.

After creation I set acltype=nfsv4 and xattr=sa

I create a new file and setfacl -m u:www:x:allow file

I zdb the file inode number but it doesn't show anything about xattr.

Shouldn't zdb show something about xattr?
 
Most references (if not all) about xattr=sa were related to Linux and I confused.
So "extended attributes" are extra (custom) metadata. For example the man page shows how to add a "md5" metadata in /boot/kernel/kernel.

But both "extended attributes" and "ACLs" (setfacl) are stored as "extra files" when xattr=on, right?

I was searching about xattr=sa because I read these:

1) http://www.nerdblog.com/2013/10/zfs-xattr-tuning-on-linux.html

2) https://www.reddit.com/r/zfs/comments/sgg9iu/quirk_on_using_zfs_dataset_as_rsync_target_xattr/

Does a dataset with files not using "extended attributes" and only using few directory ACLs take a performance hit with xattr=on ? Does for every file system has to stat the ACLs even with untouched ACLs which will result in performance reduction ?
 
I also did this:

Code: 
zfs create -o xattr=sa zroot/home/test
cd /home/test
touch file
setextattr system md5 test file

And zdb -ddddddddddd zroot/home/test 2 shows it:

Code: 
Dataset zroot/home/test [ZPL], ID 1075, cr_txg 418982, 96K, 8 objects, rootbp DVA[0]=<0:87c03170000:1000> DVA[1]=<0:8e802ed0000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=419014L/419014P fill=8 cksum=10215d460e:2d33cd3ac2ba:43ec6880b01f17:4866bdefa171fdd8

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
         2    1   128K    512      0     512    512    0.00  ZFS plain file (K=inherit) (Z=inherit=lz4)
                                               256   bonus  System attributes
        dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED
        dnode maxblkid: 0
        path    /file
        uid     0
        gid     0
        atime   Mon Jan 30 22:13:44 2023
        mtime   Mon Jan 30 22:13:44 2023
        ctime   Mon Jan 30 22:16:07 2023
        crtime  Mon Jan 30 22:13:44 2023
        gen     418986
        mode    100644
        size    0
        parent  34
        links   1
        pflags  40800000000
        SA xattrs: 64 bytes, 1 entries

                freebsd:system:md5 = test
Indirect blocks:

And also:

Code: 
zfs create -o xattr=on zroot/home/test2
cd /home/test2
touch file
setextattr system md5 test file

And zdb -ddddddddddd zroot/home/test2 2:

Code: 
Dataset zroot/home/test2 [ZPL], ID 586, cr_txg 419051, 104K, 10 objects, rootbp DVA[0]=<0:85425a1e000:1000> DVA[1]=<0:8640b5eb000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=419067L/419067P fill=10 cksum=e000b8e11:27bf3e355e6d:3c35a9a2eb5714:4066bd613d85b338

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
         2    1   128K    512      0     512    512    0.00  ZFS plain file (K=inherit) (Z=inherit=lz4)
                                               176   bonus  System attributes
        dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED
        dnode maxblkid: 0
        path    /file
        uid     0
        gid     0
        atime   Mon Jan 30 22:19:26 2023
        mtime   Mon Jan 30 22:19:26 2023
        ctime   Mon Jan 30 22:19:26 2023
        crtime  Mon Jan 30 22:19:26 2023
        gen     419053
        mode    100644
        size    0
        parent  34
        links   1
        pflags  40800000004
        xattr   128
Indirect blocks:

And then zdb -ddddddddddd zroot/home/test2 128:

Code: 
Dataset zroot/home/test2 [ZPL], ID 586, cr_txg 419051, 104K, 10 objects, rootbp DVA[0]=<0:85425a1e000:1000> DVA[1]=<0:8640b5eb000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=419067L/419067P fill=10 cksum=e000b8e11:27bf3e355e6d:3c35a9a2eb5714:4066bd613d85b338

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
       128    1   128K    512      0     512    512  100.00  ZFS directory (K=inherit) (Z=inherit=lz4)
                                               168   bonus  System attributes
        dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED
        dnode maxblkid: 0
        path    /file/<xattrdir>
        uid     0
        gid     0
        atime   Mon Jan 30 22:20:39 2023
        mtime   Mon Jan 30 22:20:39 2023
        ctime   Mon Jan 30 22:20:39 2023
        crtime  Mon Jan 30 22:20:39 2023
        gen     419067
        mode    41777
        size    3
        parent  2
        links   2
        pflags  40800000145
        microzap: 512 bytes, 1 entries

                freebsd:system:md5 = 129 (type: Regular File)
Indirect blocks:
               0 L0 EMBEDDED et=0 200L/3eP B=419067

                segment [0000000000000000, 0000000000000200) size   512

From what I understand in the case of xattr=sa there is only a stat during file read, and with xattr=on there are two stat. Am I right?
 
on ufs setfacl creates an extattr in the system namespace named
posix1e.acl_access but does not seem the same on zfs

i have nfs v4 acls on some data sets but they are not busy enough to make any difference
 
And here is zdb for a file without extra attributes:

Code: 
zdb -ddddddddddd zroot/home/test2 384
Dataset zroot/home/test2 [ZPL], ID 586, cr_txg 419051, 120K, 12 objects, rootbp DVA[0]=<0:85425d0f000:1000> DVA[1]=<0:8640ba2d000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=419131L/419131P fill=12 cksum=e034d9a4a:27c85b837ae0:3c40c6526078f1:406ce3886fe98f67

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
       384    1   128K    512      0     512    512    0.00  ZFS plain file (K=inherit) (Z=inherit=lz4)
                                               168   bonus  System attributes
        dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED
        dnode maxblkid: 0
        path    /file2
        uid     0
        gid     0
        atime   Mon Jan 30 22:26:09 2023
        mtime   Mon Jan 30 22:26:09 2023
        ctime   Mon Jan 30 22:26:09 2023
        crtime  Mon Jan 30 22:26:09 2023
        gen     419131
        mode    100644
        size    0
        parent  34
        links   1
        pflags  40800000004
Indirect blocks:
 
CyberCr33p said:
Does a dataset with files not using "extended attributes" and only using few directory ACLs take a performance hit with xattr=on ? Does for every file system has to stat the ACLs even with untouched ACLs which will result in performance reduction ?
Click to expand...
In the file with no extra attributes & xattr=on zdb shows no inode that contains any "extra attributes" info, so I believe that if the files don't use extended attributes then it needs only one stat and in this case xattr=on and xatrr=sa performance is the same, am I right?
 
And now 2 tests with ACLs, after doing setfacl -m u:www:x:allow newfile:

dataset with xattr=sa:

Code: 
zdb -ddddddddddd zroot/home/test 128
Dataset zroot/home/test [ZPL], ID 1075, cr_txg 418982, 104K, 9 objects, rootbp DVA[0]=<0:87c036c0000:1000> DVA[1]=<0:8e803420000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=419289L/419289P fill=9 cksum=efd7b72c5:29c208391fef:3e5cb3dd57b207:4218c25eeb184bf2

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
       128    1   128K    512      0     512    512    0.00  ZFS plain file (K=inherit) (Z=inherit=lz4)
                                               184   bonus  System attributes
        dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED
        dnode maxblkid: 0
        path    /newfile
        uid     0
        gid     0
        atime   Mon Jan 30 22:39:22 2023
        mtime   Mon Jan 30 22:39:22 2023
        ctime   Mon Jan 30 22:39:53 2023
        crtime  Mon Jan 30 22:39:22 2023
        gen     419282
        mode    100644
        size    0
        parent  34
        links   1
        pflags  40800000000
Indirect blocks:

dataset with xattr=on:

Code: 
zdb -ddddddddddd zroot/home/test2 512
Dataset zroot/home/test2 [ZPL], ID 586, cr_txg 419051, 128K, 13 objects, rootbp DVA[0]=<0:8542669c000:1000> DVA[1]=<0:8640c324000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=419274L/419274P fill=13 cksum=e0ab19cdd:27e6d285e770:3c7f59c6e30951:40c24f83daef2bfd

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
       512    1   128K    512      0     512    512    0.00  ZFS plain file (K=inherit) (Z=inherit=lz4)
                                               184   bonus  System attributes
        dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED
        dnode maxblkid: 0
        path    /newfile
        uid     0
        gid     0
        atime   Mon Jan 30 22:38:35 2023
        mtime   Mon Jan 30 22:38:35 2023
        ctime   Mon Jan 30 22:38:40 2023
        crtime  Mon Jan 30 22:38:35 2023
        gen     419274
        mode    100644
        size    0
        parent  34
        links   1
        pflags  40800000000
Indirect blocks:

So where the ACLs data is stored? Does xattr only works for "extended attributes" and not ACLs? And if doesn't work for ACLs then why zfsprops() talks about xattr under acltype=off|nfsv4|posix section?
 
Eric A. Borisch said:
It does not appear that 2.1.4 has this commit to actually implement xattr=sa on FreeBSD
Click to expand...

It's added in 2.0.7:

github.com

Release zfs-2.0.7 · openzfs/zfs

Supported Platforms Linux: compatible with 3.10 - 5.15 kernels FreeBSD: compatible with releases starting from 12.2-RELEASE Changes Fix zvol_open() lock inversion #12863 FreeBSD: Add vop_standar...
github.com github.com

And also from my previous tests xattr=sa works as expected for "extended attributes".

My only question now is where are the ACLs stored.
 
CyberCr33p said:
My only question now is where are the ACLs stored.
Click to expand...
May I suggest reading the book? Kirk McKusick et al, "Design and Implementation of the FreeBSD Operating System", make sure to get the new edition (cover is mostly black with a red dot-like daemon), it has a chapter on ZFS.

Also, Kirk teaches ZFS internals classes; the slides from his class may be available online.
 
ralphbsz said:
May I suggest reading the book? Kirk McKusick et al, "Design and Implementation of the FreeBSD Operating System", make sure to get the new edition (cover is mostly black with a red dot-like daemon), it has a chapter on ZFS.

Also, Kirk teaches ZFS internals classes; the slides from his class may be available online.
Click to expand...
In general, reading that book is worth your while, especially where it concerns ZFS. It contains probably the sole detailed description of ZFS' internal structures in one place, informed by the source (that is: Matthew Ahrens, consulted by Marshall Kirk McKusick).

However, taking into account that that book's current second edition is from 2014 and that the relevant commit is released in zfs-2.0.7 by Dec 23, 2021, I think that the requested details about the nfsv4 ACLs as implemented in ZFS are not in the book; a quick search of the book suggests as much.

Looking at zfsprops(7):
Code: 
 acltype=off|nfsv4|posix
[...]
	   nfsv4     default on FreeBSD, indicates that NFSv4-style ZFS ACLs
		     should be used.  These ACLs can be	managed with the
		     getfacl(1)	and setfacl(1).	 The nfsv4 ZFS ACL type	is not
		     yet supported on Linux.
	   posix     indicates POSIX ACLs should be used.  POSIX ACLs are spe-
		     cific to Linux and are not functional on other platforms.
		     POSIX ACLs are stored as an extended attribute and	there-
		     fore will not overwrite any existing NFSv4 ACLs which may
		     be set.
	   posixacl  an	alias for posix

       To obtain the best performance when setting posix users are strongly
       encouraged to set the xattr=sa property. This will result in the POSIX
       ACL being stored more efficiently on disk.
only mentions specifically the performance impact of "xattr=sa" in relation to POSIX ACLs, implicitly suggesting that this does not apply to nfsv4 ACLs in ZFS. This aspect is mentioned again later on:
Code: 
     xattr=on|off|sa
[...]
       System attribute	based xattrs can be enabled by setting the value to
       sa.  The	key advantage of this type of xattr is improved	performance.
       Storing extended	attributes as system attributes	significantly de-
       creases the amount of disk IO required.	
[...]
       The use of system attribute based xattrs is strongly encouraged for
       users of SELinux or POSIX ACLs. Both of	these features heavily rely on
       extended	attributes and benefit significantly from the reduced access
       time.
This does not explicitly answer your question where and how nfsv4 ACLs are stored and if there might be a perfomance impact depending on what settings are used. Regarding your question "Does xattr only works for "extended attributes" and not ACLs?": it could also be that nfsv4 ACLs in FreeBSD ZFS are stored like system attributes (i.e., efficiently) but nevertheless do not show up in the output of zdb(8).

Being that nfsv4 ACLs in FreeBSD is a somewhat esoteric topic—i.e. not commonly used, but when needed rather important—the answer should probably be sought at the source. Besides the actual source code, that would probably be at the OpenZFS github. Your other option would be to contact Edward Tomasz Napierała: see FreeBSD Wiki - NFSv4_ACLs.
 
I'm not exactly up to date with OpenZFS development, but last time I've looked the NFSv4 ACLs in ZFS were not stored as extended attributes, but rather as "their own thing" assigned to the inode, and IIRC they didn't incur a performance penalty.
I think this whole issue is deeply Linux-specific - in Linux the kernel doesn't provide a proper API for ACLs, instead there is a convention on the ACL being stored in a certain xattr, which userspace manipulates directly. So when the filesystem natively stores them in some other way, like ZFS does, it needs to be hacked around somehow.
 
You must log in or register to reply here.
Back
Top