ZFS ; partial data encryption

[_martin]

I'm building a 8TB raidz2 data pool (4D+2P). I'm looking for a solution to encrypt some of the data, preferably expanding the encryption part of the pool dynamically on demand. I went through some posts here and over the internet. I found one solution which I like and would like to hear your opinion on that. I was inspired on this 0xfeedface.org blog.

I'm still waiting for my sff-8087 to sata cables for my controller so I was only testing this in virtual machine. Point is to build ZFS pool as usual (not paying attention to 4k block here in VM) and encrypt it later:

root@fbsd01:/root # zpool create tpool raidz2 /dev/da1 /dev/da2 /dev/da3 /dev/da4 /dev/da5 /dev/da6
root@fbsd01:/root # zpool list
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
tpool  3.03G   218K  3.03G     0%  1.00x  ONLINE  -
root@fbsd01:/root #

And now create a geli provider on ZFS:

root@fbsd01:/root # zfs create -V 1G tpool/shadow
root@fbsd01:/root # dd if=/dev/random of=enc.key bs=64 count=1
1+0 records in
1+0 records out
64 bytes transferred in 0.000179 secs (357437 bytes/sec)
root@fbsd01:/root # 
root@fbsd01:/root # geli init -s 4096 -K enc.key /dev/zvol/tpool/shadow
Enter new passphrase:
Reenter new passphrase:

Metadata backup can be found in /var/backups/zvol_tpool_shadow.eli and
can be restored with the following command:

        # geli restore /var/backups/zvol_tpool_shadow.eli /dev/zvol/tpool/shadow

root@fbsd01:/root # geli attach -k enc.key /dev/zvol/tpool/shadow
Enter passphrase:
root@fbsd01:/root # zpool create evilstuff /dev/zvol/tpool/shadow.eli
root@fbsd01:/root # zpool list
NAME        SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
evilstuff  1016M   440K  1016M     0%  1.00x  ONLINE  -
tpool      3.03G   344K  3.03G     0%  1.00x  ONLINE  -
root@fbsd01:/root #

It might be interesting to check what performance impact it might have on a pool if a create tpool/shadow in sparse mode. I don't think it's worth doing more vdevs for evilstuff pool, though I might be mistaken.

Now if my evilstuff pool gets full I can easily expand it:

root@fbsd01:/root # zfs create -V 512M tpool/shadow2
root@fbsd01:/root # geli init -s 4096 -K enc.key /dev/zvol/tpool/shadow2
Enter new passphrase:
Reenter new passphrase:

Metadata backup can be found in /var/backups/zvol_tpool_shadow2.eli and
can be restored with the following command:

        # geli restore /var/backups/zvol_tpool_shadow2.eli /dev/zvol/tpool/shadow2

root@fbsd01:/root # geli attach -k enc.key /dev/zvol/tpool/shadow2
Enter passphrase:
root@fbsd01:/root # zpool add evilstuff /dev/zvol/tpool/shadow2.eli
root@fbsd01:/root # zpool list
NAME        SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
evilstuff  1.48G   832K  1.48G     0%  1.00x  ONLINE  -
tpool      3.03G  4.49M  3.03G     0%  1.00x  ONLINE  -
root@fbsd01:/root # zpool status evilstuff
  pool: evilstuff
 state: ONLINE
  scan: none requested
config:

        NAME                      STATE     READ WRITE CKSUM
        evilstuff                 ONLINE       0     0     0
          zvol/tpool/shadow.eli   ONLINE       0     0     0
          zvol/tpool/shadow2.eli  ONLINE       0     0     0

errors: No known data errors
root@fbsd01:/root #

Which seems to be an interesting way to go. I will do some test myself, but I was wondering: is anybody using this scenario already? If so, can you share your findings / performance results?

 

[gkontos]

Which seems to be an interesting way to go. I will do some test myself, but I was wondering: is anybody using this scenario already? If so, can you share your findings / performance results?

I do. I have created 2 sparse bundles, used geli to encrypt them and on top of them I have created 2 pools.

First example:

[CMD=""]dd if=/dev/zero of=/root/pool1.md bs=1k seek=819200k count=0[/CMD]
[CMD=""]mdconfig -a -t vnode -f /root/pool1.md[/CMD]
[CMD=""]geli init /dev/md0[/CMD]
[CMD=""]geli attach /dev/md0[/CMD]
[CMD=""]zpool create -o altroot=/media/pool1 pool1 /dev/md0.eli[/CMD]

Note, the underlying file system is a raidz1 pool. The reason for this is that we simply needed to create different encrypted pools that would be able to contain full replicas of other servers.

The performance is not bad given the nature of the server. It only accepts differential snapshots from different servers as full clones for DR purposes.

 

[Sebulon]

Interesting method, very resourceful! If it was me, I would just use plain UFS on the encrypted part, expand the zvol and use growfs to enlarge the filesystem, to get away from the overhead of having zfs on zfs. Got me thinking of that movie Inception, having zfs on zfs, on zfs, on zfs... Wonder if time passes slower inside of that as well

/Sebulon

 

[gkontos]

Interesting method, very resourceful! If it was me, I would just use plain UFS on the encrypted part, expand the zvol and use growfs to enlarge the filesystem, to get away from the overhead of having zfs on zfs. Got me thinking of that movie Inception, having zfs on zfs, on zfs, on zfs... Wonder if time passes slower inside of that as well

/Sebulon

You are right, it does pass slower and reminds of Inception! But remember that the reason that we have those pools is to receive full differential snapshots of different root on ZFS servers. So, UFS was really out of the question. But like I said before, performance does not really matter here. And the bottleneck is the Internet speed which maxes at 4.5Mbit.

 

[_martin]

Yes, it strongly reminds me:

Yo dawg, we heard you like ZFS so we put zpool on your zpool so you can zfs while you zfs ;-)

On serious note though, it seems like a good solution to have partially encrypted FS.

@gkontos: thanks for sharing. I was referring to sparse mode (-s) during ZFS fileset creation to avoid additional layer (md). But I guess it's not that big performance hit.

It might be worth to use weaker checksum algorithm if possible on upper pool. But I'm just guessing here - I'll do some tests and will post the results once that cable finally arrives.