Slow network upload. Please, help.

B

bsd-user

Hi!
I use 9.1-RC2 amd64 version on my server.
Server hardware:
Code: 
IBM X3630
CPU: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz (2400.13-MHz K8-class CPU)
real memory  = 17179869184 (16384 MB)
avail memory = 16471982080 (15708 MB)
igb0: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port 0x2020-0x203f mem 0x92a60000-0x92a7ffff,0x92a40000-0x92a5ffff,0x92a84000-0x92a87fff irq 28 at device 0.0 on pci11
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: 5c:f3:fc:xxxxxxx
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3

# mfiutil version
mfiutil version 1.0.14
# mfiutil show firmware
mfi0 Firmware Package Version: 12.12.0-0056
mfi0 Firmware Images:
Name  Version                        Date         Time         Status
APP   2.120.113-1307                 Jun 23 2011  18:19:36     active
BIOS  3.22.00_4.11.05.00_0x0501A000   6/21/2011
   6/21/2011
  active
PCLI  04.04-017:#%00008              Oct 12 2010  11:32:58     active
BCON  6.0-40-e_35-Rel                May 24 2011  14:27:56     active
NVDT  2.09.03-0015                   May 18 2011  05:25:33     active
BTBL  2.02.00.00-0000                Sep 16 2009  21:37:06     active
BOOT  09.250.01.219                  4/4/2011     15:58:38     active

# mfiutil show volumes
mfi0 Volumes:
  Id     Size    Level   Stripe  State   Cache   Name
 mfid0 ( 2794G) RAID-0     128k OPTIMAL Disabled
 mfid1 ( 2794G) RAID-0     128k OPTIMAL Disabled
 mfid2 ( 2794G) RAID-0     128k OPTIMAL Disabled
 mfid3 ( 2794G) RAID-0     128k OPTIMAL Disabled
 mfid4 ( 2794G) RAID-0     128k OPTIMAL Disabled
 mfid5 ( 2794G) RAID-0     128k OPTIMAL Disabled
 mfid6 ( 2794G) RAID-0     128k OPTIMAL Disabled
 mfid7 ( 2794G) RAID-0     128k OPTIMAL Disabled
 mfid8 ( 2794G) RAID-0     128k OPTIMAL Disabled
 mfid9 ( 2794G) RAID-0     128k OPTIMAL Disabled
mfid10 ( 2794G) RAID-0     128k OPTIMAL Disabled
mfid11 ( 2794G) RAID-0     128k OPTIMAL Disabled

I use zfs:
Code: 
# zpool status
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0 in 4h11m with 0 errors on Fri Feb 15 07:13:04 2013
config:

        NAME          STATE     READ WRITE CKSUM
        zroot         ONLINE       0     0     0
          raidz1-0    ONLINE       0     0     0
            mfid0p2   ONLINE       0     0     0
            mfid1p2   ONLINE       0     0     0
            mfid2p2   ONLINE       0     0     0
            mfid3p2   ONLINE       0     0     0
            mfid4p2   ONLINE       0     0     0
            mfid5p2   ONLINE       0     0     0
          raidz1-1    ONLINE       0     0     0
            mfid6p2   ONLINE       0     0     0
            mfid7p2   ONLINE       0     0     0
            mfid11p2  ONLINE       0     0     0
            mfid8p2   ONLINE       0     0     0
            mfid9p2   ONLINE       0     0     0
            mfid10p2  ONLINE       0     0     0

errors: No known data errors

# zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
zroot                      9.86T  15.7T   384M  legacy
zroot/hosting              9.84T  15.7T  9.84T  /hosting
zroot/swap                 4.13G  15.7T   763M  -
zroot/tmp                   307K  15.7T   307K  /tmp
zroot/usr                  6.06G  15.7T  3.70G  /usr
zroot/usr/ports            1.35G  15.7T  1.22G  /usr/ports
zroot/usr/ports/distfiles   132M  15.7T   132M  /usr/ports/distfiles
zroot/usr/src              1.01G  15.7T  1.01G  /usr/src
zroot/var                  1.91G  15.7T  1.69G  /var
zroot/var/db                218M  15.7T   218M  /var/db

Network configuration:
Code: 
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 5c:f3:fc:xx:xx:xx
        inet6 xxxxxxxxxxxx%igb0 prefixlen 64 scopeid 0x1
        inet xxxxxxxxxx netmask 0xffffffxx broadcast yyyyyyyyyy
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 1000baseT <full-duplex>
        status: active

/boot/loader.conf:
Code: 
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"

aio_load="YES"
autoboot_delay="2"

accf_http_load="YES"
accf_data_load="YES"

hw.igb.txd=4096
hw.igb.rxd=4096

net.inet.tcp.tcbhashsize=2048

/etc/sysctl.conf:
Code: 
net.raw.recvspace=256000
net.raw.sendspace=256000
net.inet.tcp.sendspace=256000
net.inet.raw.recvspace=256000
net.inet.raw.maxdgram=16384

net.local.stream.recvspace=131072
net.local.stream.sendspace=131072

net.route.netisr_maxqlen=8192

net.inet.ip.random_id=1
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1

#net.inet.tcp.rfc1323=0
net.inet.tcp.drop_synfin=1

net.inet.ip.ttl=128

security.bsd.see_other_gids=0
security.bsd.see_other_uids=0

dev.igb.0.rx_processing_limit=2048
kern.ipc.somaxconn=4096
kern.ipc.nmbclusters=256000
This server use apache22+php+sql+nginx and serve various upload scripts.

When we try to upload test files we get a maximum speed of 20-22MB/s and then goes down to 4-4.3 MB until the end. We try to remove apache+php+sql and use only nginx with little pure html file for uploading - the same result. We try to upload via pure-ftp - same result.

If we try to make upload to another host from that datacenter with Debian inside we have 40-90 MB/s speed. So, there are no upload problems with our testing PC and datacenter which serve this server.

We check many times connections settings, cables, switch ports, etc. I try to change various sysctl settings with no luck.

Please, help me to get higher upload speed from my freebsd server.
Thanks.
 
This is zfs read and then write perfomance:
Code: 
# dd if=1000mb.bin of=1000mb-2.bin
1953125+0 records in
1953125+0 records out
1000000000 bytes transferred in 27.268206 secs (36672746 bytes/sec)
 
Sorry to write new message, but this zfs read and write test was made with total network upload about 20-24MB/s during dd test.
 
You must log in or register to reply here.
Back
Top