FreeBSD 8.2 High Capacity HTTP Downloads

C

cmanns

Basic system stats. 8CPU Xeon, 8gB ram, Raid10 hardware BBU
Code: 
FreeBSD box2.local.net 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Wed Jun 29 05:26:20 CDT 2011
     root@host398106:/usr/obj/usr/src/sys/host398106  amd64
root@FreeBSD box2 /root ->top

last pid: 44812;  load averages:  0.27,  0.20,  0.17   
                                        up 12+15:06:49  06:08:46
1671 processes:2 running, 1665 sleeping, 3 stopped, 1 zombie
CPU:  5.1% user,  0.0% nice,  3.1% system,  0.0% interrupt, 91.8% idle
Mem: 1114M Active, 5503M Inact, 958M Wired, 171M Cache, 511M Buf, 152M Free
Swap: 2048M Total, 320K Used, 2048M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 1260 root       35  46    0  1230M   100M accept  0  28:58  0.00% java
10116 root        1  50    0 44336K  9064K select  3   9:47  0.00% sshd
13799 www         1  44    0 69260K  5872K accept  0   1:23  0.00% httpd
13798 root        1  44    0 69760K  6188K select  0   0:47  0.00% httpd
 1302 root        1  44    0 26260K  3036K select  7   0:33  0.00% sshd
79864 root        1  44    0 19828K  7916K select  0   0:30  0.00% perl5.10.1
40769 www        60   4    0 89600K 15428K select  0   0:04  0.00% httpd
43962 www        63  44    0 89984K 15600K select  4   0:04  0.00% httpd
41258 www        63  44    0 87936K 15424K select  1   0:03  0.00% httpd
43955 www        61   4    0 89728K 15588K select  4   0:03  0.00% httpd
44211 www        63  44    0 89984K 15532K select  0   0:03  0.00% httpd
44422 www        62   4    0 89856K 15488K select  3   0:02  0.00% httpd
44306 www        62  44    0 87808K 15332K select  4   0:02  0.00% httpd
44335 www        55   4    0 86912K 14112K select  0   0:02  0.00% httpd
44352 www        58  44    0 87296K 14844K select  3   0:02  0.00% httpd
44495 www        65   4    0 92288K 15728K select  4   0:02  0.00% httpd
44560 www        61  44    0 87680K 14784K select  4   0:02  0.00% httpd
44559 www        60   4    0 87552K 14732K select  3   0:02  0.00% httpd
41253 www        31   4    0 92288K 15528K select  6   0:01  0.00% httpd
 1321 root        1  44    0  7976K  1232K nanslp  6   0:01  0.00% cron
44621 www        56   4    0 87040K 13856K select  6   0:01  0.00% httpd
44612 www        57   4    0 87168K 14276K select  4   0:01  0.00% httpd
40119 www        14   4    0 94372K 15872K select  6   0:01  0.00% httpd
44597 www        50   4    0 84224K 12992K select  4   0:01  0.00% httpd
44632 www        53   4    0 86656K 13424K select  4   0:01  0.00% httpd
39850 www        14   4    0 92416K 15532K select  0   0:01  0.00% httpd
44603 www        50  44    0 84224K 13148K select  5   0:01  0.00% httpd
37761 www        10   4    0 96932K 15364K select  4   0:01  0.00% httpd
44599 www        52   4    0 84480K 13160K select  4   0:01  0.00% httpd
44204 www        31   4    0 87552K 14480K select  0   0:01  0.00% httpd
44652 www        53   4    0 86656K 13400K select  0   0:01  0.00% httpd
44728 www        66  44    0 90496K 15384K ucond   6   0:01  0.00% httpd
//
      tty           aacd0            aacd1            pass0             cpu
 tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
   0    62 51.45   5  0.26  92.45  40  3.60   0.00   0  0.00   0  0  3  0 97
//
dT: 1.007s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      1      0      0    0.0      1     48    0.1    0.0| aacd0
    0     45     44   3861    0.4      1      2    0.0    1.9| aacd1
    0      1      0      0    0.0      1     48    0.1    0.0| aacd0s1
    0      0      0      0    0.0      0      0    0.0    0.0| aacd0s1a
    0      0      0      0    0.0      0      0    0.0    0.0| aacd0s1b
    0      0      0      0    0.0      0      0    0.0    0.0| aacd0s1d
    0      1      0      0    0.0      1     48    0.1    0.0| aacd0s1e
    0      0      0      0    0.0      0      0    0.0    0.0| aacd0s1f
//bandwidth lagg1
TX:             cumm:  2.04GB   peak:    240Mb                                          rates:    230Mb   231Mb   215Mb
RX:                    33.5MB           3.95Mb                                                   3.57Mb  3.69Mb  3.45Mb
TOTAL:                 2.07GB            244Mb                                                    234Mb   235Mb   218Mb

Iostat, dmesg, df -h

http://pastebin.com/PNSQzXze

Basic tuning

Code: 
CPUTYPE?=nocona
MASTER_SITE_OVERRIDE?="http://mirrors.service.softlayer.com/freebsd/distfiles/${DIST_SUBDIR}/"
#CFLAGS=-O2 -pipe -fno-strict-aliasing
#COPTFLAGS=-O2 -pipe -funroll-loops -ffast-math -fno-strict-aliasing
OPTIMIZED_CFLAGS=YES
WITHOUT_X11=YES
#BUILD_OPTIMIZED=YES
WITH_CPUFLAGS=YES
WITH_OPTIMIZED_CFLAGS=YES
WITHOUT_DEBUG=YES
BUILD_STATIC=YES
MAKE_JOBS_NUMBER=4
# added by use.perl 2011-06-29 11:11:56
PERL_VERSION=5.10.1
SUEXEC_DOCROOT=/home
.if !empty(.CURDIR:M/usr/ports/www/*) && exists(/usr/local/bin/gcc45)
.if empty(.CURDIR:M/usr/ports/*)
CC=gcc45
CXX=g++45
CPP=cpp45
CPUTYPE=nocona
CFLAGS+=-march=nocona -mssse3
.endif
.endif

No loader settings changed yet, I've yet to reboot since partitioning the rest of the raid as /home. Apache has sendfile enabled, GCC45 did seem to make a boost, only apache is set with gcc45.

Only issues noticed so far is random upload (To the server) speeds while it's only doing a few mbit upload, and we notice it hasn't really pumped more then 45mByte/sec over apache while the linux box's have done alittle more on same network.

I have an identical system to transfer 2 more nodes too just like this one had done, hoping to find some kernel tweaks or something to test on the new system before deploying onto the one I posted stats of above.

Oh and sysctl http://pastebin.com/Yb6we3eT

Work load?

99% static downloads, 5-10mB, some nodes got 50-100mB files.

Goal, up to 1gbps saturation reliable.

We may test drive varnish to relieve apache.
 
SirDice said:
Not sure what the question actually is, I assume it's about tuning? Have a read in tuning(7).
Click to expand...

Have had many reads. Currently nothing really changes the load but it's mostly over 90% idle 24/7.

Not sure where to tune for large data/1gbps through network. :e
 
I'll look into polling however wouldn't that help if my ethernet cards needed it?

They don't seem to be lagging at all.


Code: 
>netstat -m
18628/4412/23040 mbufs in use (current/cache/total)
4183/2501/6684/32768 mbuf clusters in use (current/cache/total/max)
4183/1193 mbuf+clusters out of packet secondary zone in use (current/cache)
0/848/848/8192 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/4096 9k jumbo clusters in use (current/cache/total/max)
0/0/0/2048 16k jumbo clusters in use (current/cache/total/max)
13023K/9497K/22520K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
12844477 requests for I/O initiated by sendfile
0 calls to protocol drain routines

Over 200mbps atm

CPU: 0.0% user, 0.0% nice, 2.7% system, 0.0% interrupt, 97.3% idle
as far as I know if the nics were buggy it'd have a high interrupt %
 
I would recommend replacing apache with gatling for static files over HTTP. Gatling saturates 1gbit/s ethernet with an 1.3GHz N36L and 4 WD20EARS as RAID-Z1 in a LAN environment. Just experiment with the buffer sizes required for your latency. Gatling is a single threaded HTTP and FTP server using kqueue/event for non-blocking I/O. Serving static content over HTTP or FTP is disk or network bound. Your expensive CPU is just idling.

Watch gstat if your disk performance is trashed by too high i/o parallelism. Gatling can help the disk scheduler by performing blocking reads of up to gatling -P${PREFETCH_MEGABYTES}M bytes at once.
 
Hello,

As Crest wrote use gstat(8) for monitoring HDD - depending on its output you can play with vfs.read_max (32 is suggested as "best value" on RAID1).

drive.png
-
yearly graph with vfs.read_max=32 on RAID1+0 (32 is set in m. May).

Also you can take a look (and/or paste) the output from # vmstat -z
 
You must log in or register to reply here.
Back
Top