1889e GELI Benchmarks - The FreeBSD Forums
The FreeBSD Forums  

Go Back   The FreeBSD Forums > Base System > Storage
Reload this Page GELI Benchmarks
Register FAQ Calendar Search Today's Posts Mark Forums Read

Storage Place to ask questions about partitioning, labelling, filesystems, encryption or anything else related to storage area.

Reply
 
Thread Tools Display Modes
  #1  
Old April 17th, 2012, 12:09
Sebulon's Avatar
Sebulon Sebulon is offline
Member
  
Join Date: Nov 2010
Location: Uppsala, Sweden
Posts: 559
Thanks: 24
Thanked 93 Times in 78 Posts
Default GELI Benchmarks
Code:
HW
CHA: HP DL180 G6
CPU: Xeon E5620 @ 2.40GHz
RAM: 32GB DDR3 REG ECC
HBA: LSI 9211 (PH13 FW)
HDD: HP(WD) MB2000EAZNL
Code:
SW
# uname -a
FreeBSD hostname 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan  3 07:46:30 UTC 2012     root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
# kldstat
Id Refs Address            Size     Name
 1   19 0xffffffff80200000 11cd9b0  kernel
 2    1 0xffffffff813ce000 203d70   zfs.ko
 3    2 0xffffffff815d2000 5c50     opensolaris.ko
 4    1 0xffffffff81812000 51b3     tmpfs.ko
 5    1 0xffffffff81818000 ce78     geom_eli.ko
 6    2 0xffffffff81825000 1b11e    crypto.ko
 7    1 0xffffffff81841000 a4d9     zlib.ko
 8    1 0xffffffff8184c000 1a3f     aesni.ko
Code:
PART
# gpart create -s gpt da(0,1,2,3,4)
# gpart add -t freebsd-zfs -l disk(1,2,3,4,5) -b 2048 -a 4k da(0,1,2,3,4)
Code:
GELI
# dd if=/dev/random of=/boot/geli/disks.key bs=64 count=1
# geli init -s 4096 -K /boot/geli/disks.key -P -l (128,192,256) -e (AES-XTS,AES-CBC,Blowfish-CBC,Camellia-CBC,3DES-CBC) /dev/gpt/disk(1,2,3,4,5)
# geli attach -p -k /boot/geli/disks.key /dev/gpt/disk(1,2,3,4,5)
Code:
MO
# mdmfs -s 2048m md0 /mnt/ram
# umount /mnt/ram
(Because I don´t know the mdconfig-syntax to do the same thing)
# dd if=/dev/random of=/dev/md0 bs=1024000 count=2048

# dd if=/dev/md0 of=/dev/gpt/disk(1,2,3,4,5).eli bs=1024000 count=2048
# dd if=/dev/md0 of=/dev/gpt/disk(1,2,3,4,5).eli bs=1024000 count=2048
# dd if=/dev/md0 of=/dev/gpt/disk(1,2,3,4,5).eli bs=1024000 count=2048
Code:
GELI SCORE
              Bit  MB/s
Raw                146
AES-XTS       128  70,5
AES-CBC       128  114,4 (65,5 without aesni.ko loaded)
Blowfish-CBC  128  27,8
Camellia-CBC  128  43,0

3DES-CBC      192  14,6

AES-XTS       256  67,7
AES-CBC       256  106,5
Blowfish-CBC  256  27,8
Camellia-CBC  256  37,6

Proceeding by choosing the fastest GELI option (AES-CBC 128bit) and testing the performance of a filesystem on top of that.
Code:
ZFS/GELI MO:

# zpool create -O mountpoint=legacy -O compress=on tank mirror gpt/disk{1.eli,2.eli} mirror gpt/disk{3.eli,4.eli}
mirror gpt/disk{5.eli,6.eli} mirror gpt/disk{7.eli,8.eli}
# mount -t zfs tank /mnt/tank/
# bonnie++ -d /mnt/tank/ -u 0 -s 64g
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
hostname          64G   138  99 383394  69 292000  63   337  98 891332  82 506.4  28
Latency               399ms    5960ms    8642ms     168ms   31746us     182ms
Version  1.96       ------Sequential Create------ --------Random Create--------
hostname              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 24120  95 +++++ +++ 22920  96 21911  94 +++++ +++ 24014  97
Latency             13852us     139us    2164us   14589us      92us     174us
1.96,1.96,hostname,1,1334621912,64G,,138,99,383394,69,292000,63,337,98,891332,82,506.4,28,1
6,,,,,24120,95,+++++,+++,22920,96,21911,94,+++++,+++,24014,97,399ms,5960ms,8642ms,168ms,317
46us,182ms,13852us,139us,2164us,14589us,92us,174us

# zpool create -O mountpoint=legacy -O compress=on tank raidz2 gpt/disk{1.eli,2.eli,3.eli,4.eli} raidz2 gpt/disk{5.eli,6.eli,7.eli,8.eli}
# mount -t zfs tank /mnt/tank/
# bonnie++ -d /mnt/tank/ -u 0 -s 64g
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
hostname          64G   135  99 407474  74 297102  65   334  99 797934  72 372.7   7
Latency             77224us    1074ms    2944ms   79237us   62840us     305ms
Version  1.96       ------Sequential Create------ --------Random Create--------
hostname              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 24776  98 +++++ +++ 23919  97 24210  95 15289  99  4257  98
Latency             13907us     140us     169us   17596us     324us    7133us
1.96,1.96,hostname,1,1334627599,64G,,135,99,407474,74,297102,65,334,99,797934,72,372.7,7,16
,,,,,24776,98,+++++,+++,23919,97,24210,95,15289,99,4257,98,77224us,1074ms,2944ms,79237us,62
840us,305ms,13907us,140us,169us,17596us,324us,7133us

# zpool create -O mountpoint=legacy -O compress=on tank raidz2 gpt/disk{1.eli,2.eli,3.eli,4.eli,5.eli,6.eli,7.eli,8.eli}
# mount -t zfs tank /mnt/tank/
# bonnie++ -d /mnt/tank/ -u 0 -s 64g
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
hostname          64G   136  99 426207  76 296597  65   331  98 784113  71 313.5  19
Latency             64337us     493ms    2187ms   94173us   53133us     286ms
Version  1.96       ------Sequential Create------ --------Random Create--------
hostname              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23478  93 +++++ +++ 24146  97 22300  97 +++++ +++ 23719  97
Latency             14004us     139us     171us   29080us     142us     170us
1.96,1.96,hostname,1,1334624351,64G,,136,99,426207,76,296597,65,331,98,784113,71,313.5,19,1
6,,,,,23478,93,+++++,+++,24146,97,22300,97,+++++,+++,23719,97,64337us,493ms,2187ms,94173us,
53133us,286ms,14004us,139us,171us,29080us,142us,170us
Code:
ZFS/GELI SCORE
          Write  Rewrite  Read  (MB/s)
4xmirror  374    285      870
2xraidz2  397    290      779
1xraidz2  416    289      765

/Sebulon
Reply With Quote
The Following 7 Users Say Thank You to Sebulon For This Useful Post:
bbzz (May 2nd, 2012), Crivens (April 17th, 2012), HarryE (April 18th, 2012), MorgothV8 (October 28th, 2012), t0ken (February 13th, 2013), wblock@ (October 25th, 2012), Zare (April 17th, 2012)
  #2  
Old April 17th, 2012, 15:56
t1066 t1066 is offline
Member
  
Join Date: Jun 2010
Posts: 142
Thanks: 3
Thanked 25 Times in 24 Posts
Default
Have you try AES-XTS with aesni.ko loaded? It should also be hardware accelerated.
Reply With Quote
  #3  
Old April 17th, 2012, 18:48
Sebulon's Avatar
Sebulon Sebulon is offline
Member
  
Join Date: Nov 2010
Location: Uppsala, Sweden
Posts: 559
Thanks: 24
Thanked 93 Times in 78 Posts
Default
Quote:
Originally Posted by t1066 View Post
Have you try AES-XTS with aesni.ko loaded? It should also be hardware accelerated.
It was. It was loaded during the whole first suite of tests. After that, I tried one more time only with AES-CBC (because it was the fastest) with the driver unloaded, just to know the difference between soft- and hardware crypto with the same algo.

/Sebulon
Reply With Quote
The Following User Says Thank You to Sebulon For This Useful Post:
t1066 (April 18th, 2012)
  #4  
Old October 26th, 2012, 07:34
lockdoc lockdoc is offline
Member
  
Join Date: Jul 2009
Posts: 122
Thanks: 3
Thanked 6 Times in 5 Posts
Default
Quote:
Originally Posted by Sebulon View Post
Code:
GELI
# dd if=/dev/random of=/boot/geli/disks.key bs=64 count=1
# geli init -s 4096 -K /boot/geli/disks.key -P -l (128,192,256) -e (AES-XTS,AES-CBC,Blowfish-CBC,Camellia-CBC,3DES-CBC) /dev/gpt/disk(1,2,3,4,5)
Does the size of the key you use (in your case 64) actually affect the performance of the encryption itself?
Reply With Quote
  #5  
Old October 26th, 2012, 12:36
Sebulon's Avatar
Sebulon Sebulon is offline
Member
  
Join Date: Nov 2010
Location: Uppsala, Sweden
Posts: 559
Thanks: 24
Thanked 93 Times in 78 Posts
Default
@lockdoc

I think someone did ask me that before as well, but I only ever tried using the same sized key, as documented in the Handbook for setting up GELI. I figured it was best to go by the book

/Sebulon
Reply With Quote
  #6  
Old October 27th, 2012, 08:33
MorgothV8's Avatar
MorgothV8 MorgothV8 is offline
Junior Member
  
Join Date: Nov 2008
Location: Poland, Minsk Mazowiecki
Posts: 74
Thanks: 1
Thanked 4 Times in 4 Posts
Default
Benchmarks are benchmarks: (copied from my post)

I'm using geli on 10-CURRENT - processor Core i5 3450, geli in hardware
I have mirror of two identical discs (!TB each): zpool create zmirr mirror ada1.eli ada2.eli
Also sync=disabled, atime=off
RAM: 16 GB, system CURRENT-10.0 from one week ago.
Geli: AES-CBC 128bit, aesni.ko loaded

Write/read speed in practice the same about 150-180 Mb/s
dd bs=16M if=./some_8GB_file of=/dev/zero gives 175 MB/s
dd bs=16M if=/dev/zero of=./some_8GB_file count=512 gives 158 Mb/s

Second dd from the same file givers about 2,5 Gb/s (but it is fetched from ZFS ARC then)
__________________
The sun has cast its final rays over the Northen Skies...
Reply With Quote
  #7  
Old October 27th, 2012, 12:15
Sebulon's Avatar
Sebulon Sebulon is offline
Member
  
Join Date: Nov 2010
Location: Uppsala, Sweden
Posts: 559
Thanks: 24
Thanked 93 Times in 78 Posts
Default
Quote:
Originally Posted by MorgothV8 View Post
Benchmarks are benchmarks
Amen to that brother, thanks for sharing! Although, I would really like better comparing bonnie results. Would you please install benchmarks/bonnie++ and run:
# bonnie++ -d /some/zfs/dir -u 0(if root) -s 32g

/Sebulon
Reply With Quote
  #8  
Old October 27th, 2012, 20:52
MorgothV8's Avatar
MorgothV8 MorgothV8 is offline
Junior Member
  
Join Date: Nov 2008
Location: Poland, Minsk Mazowiecki
Posts: 74
Thanks: 1
Thanked 4 Times in 4 Posts
Default
OK this is output:

install -o root -g wheel -m 444 bonnie++.8 zcav.8 getc_putc.8 /usr/local/man/man8
install -o root -g wheel -m 444 /usr/ports/benchmarks/bonnie++/work/bonnie++-1.96/readme.html /u sr/local/share/doc/bonnie++
===> Compressing manual pages for bonnie++-1.96_1
===> Registering installation for bonnie++-1.96_1
Installing bonnie++-1.96_1... done
===> Cleaning for bonnie++-1.96_1
root@darkstar /usr/ports/benchmarks/bonnie++$ cd /data/
root@darkstar /data$ mkdir tmp
root@darkstar /data$ cd tmp
root@darkstar /data/tmp$ bonnie++ -d /data/tmp/ -u 0 -s 32g
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
darkstar 32G 208 99 132266 15 78895 10 458 92 220835 9 228.9 1
Latency 44213us 721ms 1107ms 474ms 310ms 275ms
Version 1.96 ------Sequential Create------ --------Random Create--------
darkstar -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 26956 68 7332 9 18348 99 +++++ +++ +++++ +++ +++++ +++
Latency 57745us 767ms 339us 11919us 87us 216us
1.96,1.96,darkstar,1,1351361104,32G,,208,99,132266 ,15,78895,10,458,92,220835,9,228.9,1,16,,,,,26956, 68,7332,9,18348,99,+++++,+++,+++++,+++,+++++,+++,4 4213us,721ms,1107ms,474ms,310ms,275ms,57745us,767m s,339us,11919us,87us,216us
root@darkstar /data/tmp$
__________________
The sun has cast its final rays over the Northen Skies...
Reply With Quote
  #9  
Old October 27th, 2012, 21:04
MorgothV8's Avatar
MorgothV8 MorgothV8 is offline
Junior Member
  
Join Date: Nov 2008
Location: Poland, Minsk Mazowiecki
Posts: 74
Thanks: 1
Thanked 4 Times in 4 Posts
Default
BTW: is there any chance to work for FreeBSD? I've been working for Poland, Norway, USA up today, I've got own Company - and I I can work for My favorit OS (from '99s) as a voulntier....
__________________
The sun has cast its final rays over the Northen Skies...
Reply With Quote
  #10  
Old October 27th, 2012, 21:08
jb_fvwm2 jb_fvwm2 is offline
Senior Member
  
Join Date: Nov 2008
Posts: 1,386
Thanks: 60
Thanked 145 Times in 130 Posts
Default
Fixup unmaintained ports that have open PR's ?? (The Handbook may give a more precise answer...or even freebsd.org, a link from there.)
__________________
Using /lookat/ with zsh/grep/find/aliases/pipes/portmaster and /var/db/pkg/ flat files to meteorically speed port installs/upgrades forever hopefully...
Reply With Quote
  #11  
Old October 28th, 2012, 11:00
Sebulon's Avatar
Sebulon Sebulon is offline
Member
  
Join Date: Nov 2010
Location: Uppsala, Sweden
Posts: 559
Thanks: 24
Thanked 93 Times in 78 Posts
Default
@MorgothV8

Code:
            Write   Rewrite    Read    (MB/s)
1x mirror   129     77         215
That is awesome! You get write performance of 1x HD and read performance of x2. No penalty, either from ZFS, or GELI.

/Sebulon
Reply With Quote
  #12  
Old October 28th, 2012, 11:25
MorgothV8's Avatar
MorgothV8 MorgothV8 is offline
Junior Member
  
Join Date: Nov 2008
Location: Poland, Minsk Mazowiecki
Posts: 74
Thanks: 1
Thanked 4 Times in 4 Posts
Default
Looks so.
It rally works great on my current setup.
ARC consumes 12 GB of RAM, but seems to be quite clever, fast and efficient
And if I need M$ Windoze - VirtualBox runs Win 7 & like a charm I think faster than native - Windows VDI file is just in ARC and *rocks* !
__________________
The sun has cast its final rays over the Northen Skies...
Reply With Quote
Reply

« Previous Thread | Next Thread »
Thread Tools
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
ZFS benchmarks traustitj Storage 9 February 17th, 2012 08:49
My ZFS V28 benchmarks Goose997 System Hardware 18 January 20th, 2012 07:23
(Benchmarks) To clang or not to clang falkman General 2 July 15th, 2011 08:10
FreeBSD benchmarks (56K warning) cracauer@ General 6 July 1st, 2010 17:01
GELI Benchmarks (AES/BLOWFISH/CAMELLIA/3DES) eyebone Installing & Upgrading 0 April 4th, 2010 12:13


All times are GMT +1. The time now is 12:40.

Contact Us - FreeBSD - Archive - Top

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2013, vBulletin Solutions, Inc.
The mark FreeBSD is a registered trademark of The FreeBSD Foundation and is used by The FreeBSD Project with the permission of The FreeBSD Foundation.
Web protection and acceleration provided by CloudFlare
0