When I create a FreeBSD-AMD64 8.2 VM with a disk residing on a datastore on an iSCSI attached SAN I observe some strange performance quirks related to disk I/O.
I have two Cisco C200 servers with 48G of RAM and 10gigE NICs connected to a Cisco Nexus 5548UP which connects via a 60gig etherchannel to a stack of Cisco 3750X switches which connects to an EMC VNXe 3300 SAN via iSCSI over 4 aggregated 1gigE links.
I'm aware of the fact that iSCSI over such a configuration is limited to 1 gig of bandwidth per LUN.
I know the disk array can saturate the 1 gig of bandwidth it has. I've observed this by testing at the application layer with other VMs and via performance monitoring of the switchports/etherchannels which connect my configuration.
If I run a very basic disk test with dd:
This looks about right. I can write to the disk at about 108MB/sec. This is a reasonable number for my configuration, I think. Other benchmarks like bonnie++ and iozone show lots of variability in their results, sometimes over 30%.
Here's `mount`
If I try to transfer data from a very fast workstation to a Samba share residing on /dev/da0 (which in reality lives on the iSCSI SAN) on that machine, I get about 5 megabytes of second throughput.
If I create another disk on a datastore that's locally attached to the host running my VM and export that via Samba, I can read and write to it from my workstation at a bit over 100 megabytes/second.
For what it's worth, iperf shows that my bandwidth between the VM and that workstation I'm using for testing is 933mbit/sec. That's about what I'd expect from my config.
If I clone this VM and put the resulting clone's disk on a locally attached datastore, I can read/write to the Samba shares at over 100 megabytes/second.
If I watch systat -v during the writes or reads from the "slow" share I see that when the transfer starts, em0 and mpt0 (my ethernet and SCSI devices on the vm) initially generate lots of interrupts, as they should. VM CPU usage goes up and things look normal. Soon, observed transfer speeds drop and em0 and mpt0 stop generating interrupts for periods of time and CPU usage drops to 0%.
While watching systat -v during a "fast" transfer, things look normal. em0 and mpt0 generate lots of interrupts and CPU usage is appropriate for the workload.
I've experimented with manipulating many different sysctl variables and turning on and off various features and tuning options in smb.conf. Over the course of a few days, I've determined that none of these had any appreciable impact other than reducing the maximum transfer speed of the "fast" shares. For example, sysctl vfs.write_behind=0 reduces my maximum throughput to the fast share by about 5%.
I only observe this slow transfer speed when the exported share is on a datastore which is located on my iSCSI device. NFS has similar performance oddities.
It seems to me that somewhere, something is getting starved, the buffers fill up and things go south from there. I just can't figure out where the problem is. Anyone have any ideas?
Here are the relevant config files:
I have two Cisco C200 servers with 48G of RAM and 10gigE NICs connected to a Cisco Nexus 5548UP which connects via a 60gig etherchannel to a stack of Cisco 3750X switches which connects to an EMC VNXe 3300 SAN via iSCSI over 4 aggregated 1gigE links.
I'm aware of the fact that iSCSI over such a configuration is limited to 1 gig of bandwidth per LUN.
I know the disk array can saturate the 1 gig of bandwidth it has. I've observed this by testing at the application layer with other VMs and via performance monitoring of the switchports/etherchannels which connect my configuration.
If I run a very basic disk test with dd:
Code:
dd if=/dev/zero of=/root/foo bs=512
^C4459648+0 records in
4459647+0 records out
2283339264 bytes transferred in 21.095505 secs (108238189 bytes/sec)This looks about right. I can write to the disk at about 108MB/sec. This is a reasonable number for my configuration, I think. Other benchmarks like bonnie++ and iozone show lots of variability in their results, sometimes over 30%.
Here's `mount`
Code:
root@iscsi ~]# mount
/dev/da0s1a on / (ufs, local, soft-updates)
devfs on /dev (devfs, local, multilabel)
/dev/da0s2d on /usr (ufs, local, soft-updates)
/dev/da1 on /opt (ufs, local, soft-updates)If I try to transfer data from a very fast workstation to a Samba share residing on /dev/da0 (which in reality lives on the iSCSI SAN) on that machine, I get about 5 megabytes of second throughput.
If I create another disk on a datastore that's locally attached to the host running my VM and export that via Samba, I can read and write to it from my workstation at a bit over 100 megabytes/second.
For what it's worth, iperf shows that my bandwidth between the VM and that workstation I'm using for testing is 933mbit/sec. That's about what I'd expect from my config.
If I clone this VM and put the resulting clone's disk on a locally attached datastore, I can read/write to the Samba shares at over 100 megabytes/second.
If I watch systat -v during the writes or reads from the "slow" share I see that when the transfer starts, em0 and mpt0 (my ethernet and SCSI devices on the vm) initially generate lots of interrupts, as they should. VM CPU usage goes up and things look normal. Soon, observed transfer speeds drop and em0 and mpt0 stop generating interrupts for periods of time and CPU usage drops to 0%.
While watching systat -v during a "fast" transfer, things look normal. em0 and mpt0 generate lots of interrupts and CPU usage is appropriate for the workload.
I've experimented with manipulating many different sysctl variables and turning on and off various features and tuning options in smb.conf. Over the course of a few days, I've determined that none of these had any appreciable impact other than reducing the maximum transfer speed of the "fast" shares. For example, sysctl vfs.write_behind=0 reduces my maximum throughput to the fast share by about 5%.
I only observe this slow transfer speed when the exported share is on a datastore which is located on my iSCSI device. NFS has similar performance oddities.
It seems to me that somewhere, something is getting starved, the buffers fill up and things go south from there. I just can't figure out where the problem is. Anyone have any ideas?
Here are the relevant config files:
Code:
RC.CONF
hostname="iscsi.supersecret.testdomain.net"
sshd_enable="YES"
ifconfig_em0="DHCP mtu 9000"
samba_enable="YES"
rpcbind_enable="YES"
nfs_server_enable="YES"
mountd_enable="YES"
mountd_flags="-r"
SYSCTL.CONF
# $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1.6.1 2010/12/21 17:09:25 kensmith Exp $
#
# This file is read when going to multi-user and its contents piped thru
# ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details.
#
# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
vfs.read_max=256
LOADER.CONF
#kern.hz=1000 /*this was a silly thing to try*/
# Beginning of the block added by the VMware software
vmxnet_load="YES"
# End of the block added by the VMware software
# Beginning of the block added by the VMware software
vmxnet3_load="YES"
# End of the block added by the VMware software
aio_load="YES"
SMB.CONF
[global]
netbios name = JUPITER
workgroup = INSIGHT
encrypt passwords = yes
preferred master = yes
domain master = no
local master = yes
security = user
domain logons = no
inherit permissions = yes
follow symlinks = yes
unix extensions = no
dns proxy = no
[test]
path = /usr/export/test
writeable = yes
write list = me
[opt]
path = /opt
writeable = yes
write list = me
[root@iscsi ~]#
