Nginx-FreeBSD slow download!

shahzaib

Active Member

Reaction score: 1
Messages: 110

Hi,

We've recently shifted to FreeBSD 10.1-RELEASE due to its robust asynchronous performance for big storage based on .mp4 files. Here is the server specs :

2 x Intel Xeon X5690
96GB DDR3 Memory
12 x 3TB SATA Raid-10 (HBA LSI-9211)
ZFS FileSystem with 18TB usable space
2 x 1Gbps LACP (2Gbps Throughput)

Things are working quite well, no high I/O due to Big Ram cache and AIO performance but once network port started to go over 1Gbps, performance begins to lag, download speed started to stuck around 60-100Kbps on a 4Mbps connection (using wget) which works quite efficient under 800Mbps port (450kbps on 4Mbps). We first thought it could be network issue or LACP issue but doesn't looks like it is, we tested with download/upolading from this server on 2Gbps throughput. We also checked that if requests are in queue using command netstat -Lan but it was '0' .We also checked with Disk Busy rate using gstat(8) which was quite stable as well.

So it looks like either the sysctl(1) values need to tweak or NGINX configurations are not optimized.

Following is the link with all configs performed till now :

http://pastebin.com/EHUHV7SH
====================================================

Please iI need guidance to handle with this problem, iI am sure that some value needs to tweak.

Thanks in advance !!
 
OP
OP
S

shahzaib

Active Member

Reaction score: 1
Messages: 110

Did you create ZFS pool on the top of the hardware RAID:confused:?
No, we created using mirroring of each pair (2 x Sata) followed by rest of the drives i.e.

zpool create storage mirror da0 da1 mirror da2 da3 mirror da4 da5 mirror da6 da7 mirror da8 da9 mirror da10 da11
 
OP
OP
S

shahzaib

Active Member

Reaction score: 1
Messages: 110

I've also observed lots of retransmitted packets in netstat -s output. Here is the status :

http://prntscr.com/9xa6z2

When iI try to download file using wget on 4Mbps . This is what the behavior of download is:

- Download starts --> speed goes up --> download stucks for 2-3 seconds (much like a pause, no progress in download) --> process resume and speed goes up --> stucks again --> resume etc.
 

kpa

Beastie's Twin

Reaction score: 1,806
Messages: 6,318

I would first disable all TSO options on the interfaces and see if it makes a difference.
 
OP
OP
S

shahzaib

Active Member

Reaction score: 1
Messages: 110

I would first disable all TSO options on the interfaces and see if it makes a difference.
Could you please let me know if there would be downtime for disabling TSO? If you can provide me command to do that, it'd be very kind.

Do iI need to disable TSO in all Active interfaces ? i.e Igb0, Igb2, lagg0.
 
Last edited by a moderator:

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,542
Messages: 30,245

It should be done on the physical interfaces (igb0,igb1) and shouldn't create downtime.

ifconfig igb0 -tso
 
OP
OP
S

shahzaib

Active Member

Reaction score: 1
Messages: 110

It should be done on the physical interfaces (igb0,igb1) and shouldn't create downtime.

ifconfig igb0 -tso
Thanks, II've disabled tso on two servers to see how it goes. Will keep you updated. Thanks !!
 
Last edited by a moderator:
OP
OP
S

shahzaib

Active Member

Reaction score: 1
Messages: 110

Hi guys, one more thing worth mentioning is: we've encountered bit high %interrupt rate over 1.2+Gbps realtime port :

http://prntscr.com/a2d884

Supermicro server has built-in 4 x 1Gbps NIC ports. Following are there details, please let me know if these NIC are good enough ?

http://prntscr.com/a2e0vc

igb0 & igb2 are emerged to spawn 2Gbps Lacp
 

kpa

Beastie's Twin

Reaction score: 1,806
Messages: 6,318

You could ask on the freebsd-net mailing list, many of the NIC driver developers are there including Jack Vogel who (as I understand) is the main contact at Intel for FreeBSD driver issues.
 
Top