My server keeps running out of RAM

User23 said:
You could run the ab test on a static html page. This will show how the apache perform without mysql.

As i said, i guess it is the wordpress statistic plugin. Keep an eye on the mysql slow queries log and use
Code:
show full processlist;
on the mysql console, while stress testing. If the statistics inserts are the bottleneck it should be easy to identify them in the processlist.

Thanks :)

I had tried without the statistics plugin (it is called Wassup) and the result was the same.

Yet, when I run the same test it on a static webpage with a fairly large flash element, it rips though it so fast that I don't even see new threads opening in top.

Though it seems it is related to the database I can't immediatly tell much from the results of your query. I am looking into mgm_scheduling which makes a large part of the log:

show full processlist;
^click for results in pastebin.^

Is it worth looking into moving the database to MyISAM?
 
Sir Dice suggested I disable caching on the database filesystem on an previous thread but he was unaware of how to do it on FreeBSD.

I have just found the vfs.vmiodirenable variable for sysctl which I think is what I am looking for and if I am reading the manual correctly I should add this:

/etc/sysctl.conf
Code:
vfs.vmiodirenable=0

Does this sound reasonable? I doubt it will break anything if I just give it a go but on the off chance that it corrupts everything, I wanted to ask first.
 
ghostcorps said:
I had tried without the statistics plugin (it is called Wassup) and the result was the same.

Yet, when I run the same test it on a static webpage with a fairly large flash element, it rips though it so fast that I don't even see new threads opening in top.

Ok, good.

ghostcorps said:
Though it seems it is related to the database I can't immediatly tell much from the results of your query. I am looking into mgm_scheduling which makes a large part of the log:

Code:
"...  WHERE `option_name` = 'cron' "
"...  WHERE `option_name` = '_transient_doing_cron' "

The queries are all from the (stupid?) wordpress cron solution. This script runs every time a page is loaded. So it can burn CPU time if there are like 20 or more request per second, pretty useless. But it shouldnt result in such a bad performance. How many request per second do you got now, using
Code:
ab -c20 -n1000
?

ghostcorps said:
Is it worth looking into moving the database to MyISAM?

Well, I think this should not be the first step to do now.

Please post your
Code:
show variables;
and
Code:
show status;
output from mysql console. Remove hostnames and paths if you like.
 
Thanks for your patience! :)

Disabling cache with the below config did not help:

/etc/sysctl.conf
Code:
vfs.vmiodirenable=0


20 concurrent threads is enough to lock up apache so I ran this:
ab -c10 -n100

And got the results below:
Code:
HTML transferred:       703494 bytes
Requests per second:    1.30 [#/sec] (mean)
Time per request:       7686.952 [ms] (mean)
Time per request:       768.695 [ms] (mean, across all concurrent requests)
Transfer rate:          9.49 [Kbytes/sec] received


If I run the same command on the static page I get his:
Code:
HTML transferred:       219700 bytes
Requests per second:    6.67 [#/sec] (mean)
Time per request:       1499.014 [ms] (mean)
Time per request:       149.901 [ms] (mean, across all concurrent requests)
Transfer rate:          16.13 [Kbytes/sec] received


[url=http://pastebin.com/X26e0dpi]show variables;[/url]
^Pastebin^

[url=http://pastebin.com/df4rps7U]show status;[/url]
^Pastebin^
 
aa said:
try nginx + php-fpm

Thanks :)

I have never heard of these, what is the advantage of ngnix over apache?

As far as I can tell, apache is working fine as long as it doesn't have to query the database. I will look at php-fpm + ngnix later, but for now I don't want to introduce any potential new issues.
 
Yes, you're better off fixing new issues related to nginx than fixing old issues related to apache. The fact you said you don't even know nginx is scary! You're really behind man.
 
Lol Ok I give in I'll do it.

But I really want to fix the current problem first. All evidence points away from the webserver actually causing the lockup and fixing that is more important than getting with the times :p

I'll have a play with it on my test server and drop it in once the database has been fixed.
 
einthusan said:
Yes, you're better off fixing new issues related to nginx than fixing old issues related to apache. The fact you said you don't even know nginx is scary! You're really behind man.
Be patient, please. I haven't even read your problem when posting here.
 
First thing I notice about nginx, is that it is optimised for serving static pages. We do not use static pages and if we did we currently have no issue with doing so using apache.

nginx looks to be sold on the idea of doing a few basic things as fast as possible, which is great for static pages and ultra light websites. But our site is dynamic and content heavy. On a day to day basis we do not expect to have a huge amount of traffic and certainly do not expect to be serving 1000s of pages at a time. It is nice to have the overhead availability but I use a few of the advanced apache functions that nginx does not support, one specificly is 'digest access authentication' which we use for our secured streaming sessions. Without that I will not be able to get the webmaster to sign off on the Change.

I have also read that there is no point looking at php-fpm without nginx because mod_php serves the same function within apache.

Our webmaster is doing his own investigation, but I doubt we will be doing nginx or php-fpm until it supports the more complex tasks we need from a webserver.



Back to the real task at hand...
 
ghostcorps said:
20 concurrent threads is enough to lock up apache so I ran this:
ab -c10 -n100

And got the results below:
Code:
HTML transferred:       703494 bytes
Requests per second:    1.30 [#/sec] (mean)
Time per request:       7686.952 [ms] (mean)
Time per request:       768.695 [ms] (mean, across all concurrent requests)
Transfer rate:          9.49 [Kbytes/sec] received


If I run the same command on the static page I get his:
Code:
HTML transferred:       219700 bytes
Requests per second:    6.67 [#/sec] (mean)
Time per request:       1499.014 [ms] (mean)
Time per request:       149.901 [ms] (mean, across all concurrent requests)
Transfer rate:          16.13 [Kbytes/sec] received

6.6 Requests per second, on a static page ... horrible. What kind of hardware is that server (CPU type, RAM size ...)? What is the usual load on this machine? It looks like your problem is the apache config and/or your load/hardware.

For example, I got 10 request per second testing a Wordpress website, using -c20 -n1000 on a slow 4 core Opteron 4130 (2.6 GHz). The server usually has a load around 1.0 without ab testing.
 
Hmmm, ok, I am not surprised if it ends up being a config issue on my end, as you can tell I an groping in the dark with a lot of this stuff.

The following are taken from the Host server not the jailed webserver.

# dmesg | grep CPU
Code:
CPU: Intel(R) Xeon(R) CPU           X5680  @ 3.33GHz (3324.51-MHz K8-class CPU)
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0

# dmesg | grep memory
Code:
real memory  = 805306368 (768 MB)
avail memory = 759275520 (724 MB)


# swapinfo -k
Code:
Device          1K-blocks     Used    Avail Capacity
/dev/da0s1b       1011592    64864   946728     6%


#top (at idle)
Code:
last pid: 38797;  load averages:  0.03,  0.04,  0.01                                                                                                                         up 3+19:10:58  23:21:25
98 processes:  1 running, 97 sleeping
CPU:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
Mem: 64M Active, 114M Inact, 143M Wired, 1136K Cache, 85M Buf, 408M Free
Swap: 988M Total, 63M Used, 925M Free, 6% Inuse

# sysctl -a | less
^Pastebin^

---------

Below are my apache config files on the jailed webserver:

/usr/local/etc/apache22/httpd.conf
Code:
ServerRoot "/usr/local"

Listen 80

LoadModule authn_file_module libexec/apache22/mod_authn_file.so
LoadModule authz_host_module libexec/apache22/mod_authz_host.so
LoadModule authz_groupfile_module libexec/apache22/mod_authz_groupfile.so
LoadModule authz_user_module libexec/apache22/mod_authz_user.so
LoadModule auth_basic_module libexec/apache22/mod_auth_basic.so
LoadModule reqtimeout_module libexec/apache22/mod_reqtimeout.so
LoadModule include_module libexec/apache22/mod_include.so
LoadModule deflate_module libexec/apache22/mod_deflate.so
LoadModule log_config_module libexec/apache22/mod_log_config.so
LoadModule env_module libexec/apache22/mod_env.so
LoadModule expires_module libexec/apache22/mod_expires.so
LoadModule headers_module libexec/apache22/mod_headers.so
LoadModule usertrack_module libexec/apache22/mod_usertrack.so
LoadModule unique_id_module libexec/apache22/mod_unique_id.so
LoadModule setenvif_module libexec/apache22/mod_setenvif.so
LoadModule ssl_module libexec/apache22/mod_ssl.so
LoadModule mime_module libexec/apache22/mod_mime.so
LoadModule cgi_module libexec/apache22/mod_cgi.so
LoadModule vhost_alias_module libexec/apache22/mod_vhost_alias.so
LoadModule dir_module libexec/apache22/mod_dir.so
LoadModule actions_module libexec/apache22/mod_actions.so
LoadModule speling_module libexec/apache22/mod_speling.so
LoadModule alias_module libexec/apache22/mod_alias.so
LoadModule rewrite_module libexec/apache22/mod_rewrite.so
LoadModule unique_id_module libexec/apache22/mod_unique_id.so
LoadModule security2_module libexec/apache22/mod_security2.so
LoadModule php5_module        libexec/apache22/libphp5.so

AddType application/x-httpd-php .php .phtml
AddType application/x-httpd-php-source .phps

ServerAdmin ME@gmail.com

ServerName www.URL.com:80

DocumentRoot "/usr/local/www/apache22/data"

ServerSignature Off
ServerTokens Prod

<Directory />
    AllowOverride All
    Order deny,allow
    Deny from all
</Directory>

Alias /errors/ "/usr/local/www/apache22/errors/"

<Directory "/usr/local/www/apache22/errors/">
	Order Deny,Allow
	Allow from 192.168.0.0/24
	Deny from all
</Directory>

<Directory "/usr/local/www/apache22/offline/">
	Order deny,allow
	Deny from all
    Allow from 192.168.0.0/24
	AllowOverride All
</Directory>

<Directory "/usr/local/www/apache22/data">
    Options Indexes +FollowSymLinks MultiViews
    AllowOverride All
    Order allow,deny
    Allow from all
</Directory>

<IfModule dir_module>
    DirectoryIndex index.html index.php
</IfModule>

<FilesMatch "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy All
</FilesMatch>

ErrorLog "/var/log/httpd-error.log"

LogLevel error

<IfModule log_config_module>

    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    CustomLog "/var/log/httpd-access.log" combined
</IfModule>

<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/usr/local/www/apache22/cgi-bin/"
</IfModule>


<Directory "/usr/local/www/apache22/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

DefaultType text/plain

<IfModule mime_module>
    TypesConfig etc/apache22/mime.types
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
</IfModule>

ErrorDocument 401 /errors/autherror.html

Include etc/apache22/extra/httpd-mpm.conf

Include etc/apache22/extra/httpd-vhosts.conf

Include etc/apache22/extra/httpd-default.conf

Include etc/apache22/extra/httpd-ssl.conf

<IfModule ssl_module>
	SSLRandomSeed startup builtin
	SSLRandomSeed connect builtin
</IfModule>

Include etc/apache22/Includes/*.conf
TraceEnable off

RequestReadTimeout header=1-3,MinRate=500


/usr/local/etc/apache22/extras/httpd-ssl.conf
Code:
Listen 443

AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl

SSLPassPhraseDialog  builtin

SSLSessionCache        "shmcb:/var/run/ssl_scache(512000)"
SSLSessionCacheTimeout  300

SSLMutex  "file:/var/run/ssl_mutex"

<VirtualHost _default_:443>

DocumentRoot "/usr/local/www/apache22/data"
ServerName www.URL.com
ServerAdmin ME@gmail.com
ErrorLog "/var/log/httpd-error.log"
TransferLog "/var/log/httpd-access.log"

SSLEngine on
SSLProtocol -all +SSLv3 +TLSv1

SSLCipherSuite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM

SSLCertificateFile "/usr/local/etc/apache22/ssl/certificate.crt"
SSLCertificateKeyFile "/usr/local/etc/apache22/ssl/server.key"
SSLCACertificateFile "/usr/local/etc/apache22/ssl/RapidSSL_CA_bundle.pem"

<FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
</FilesMatch>
<Directory "/usr/local/www/apache22/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>

BrowserMatch ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

CustomLog "/var/log/httpd-ssl_request.log" \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

</VirtualHost>


/usr/local/etc/apache22/extras/httpd-vhosts.conf
Code:
NameVirtualHost HOST.EXTERNAL.IP:80

<VirtualHost WEBSERVER.JAIL.EXTERNAL.IP:80>
    DocumentRoot "/usr/local/www/apache22/data"
    ServerName URL.com
    ServerAlias URL.com
</VirtualHost>

<VirtualHost WEBSERVER.JAIL.EXTERNAL.IP:80>
    ServerAdmin webmaster@URL.com
    DocumentRoot "/usr/local/www/apache22/offline"
    ServerName offline.URL.com
    ServerAlias offline.URL.com
</VirtualHost>

<VirtualHost WEBSERVER.JAIL.VPN.IP:80>
    DocumentRoot "/usr/local/www/apache22/test"
    ServerName test.URL.com
    ServerAlias test.URL.com
</VirtualHost>


/usr/local/etc/apache22/extras/httpd-default.conf
Code:
Timeout 10

KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 1
UseCanonicalName Off
AccessFileName .htaccess
ServerTokens Prod
ServerSignature Off
HostnameLookups Off

/usr/local/etc/apache22/extra/httpd-mpm.conf
Code:
<IfModule mpm_prefork_module>
StartServers 3
MinSpareServers 2
MaxSpareServers 5
ServerLimit 10
MaxClients 10
MaxRequestsPerChild 100
</IfModule>

<IfModule mpm_worker_module>
    StartServers          2
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>

And finally the jailed webservers /usr/local/etc/my.cnf
Code:
[client]
host		= JAILED.DATABASE.VPN.IP
port            = 3306

[mysqld]
port		= 3306
host		= JAILED.DATABASE.VPN.IP
skip-external-locking
key_buffer = 16K
max_allowed_packet = 1M
table_cache = 4
sort_buffer_size = 64K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
net_buffer_length = 2K
thread_stack = 64K

server-id	= 1

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[isamchk]
key_buffer = 8M
sort_buffer_size = 8M

[myisamchk]
key_buffer = 8M
sort_buffer_size = 8M

[mysqlhotcopy]
interactive-timeout
 
I have been using the performance tuning tool HERE to tune the MySQL install. So far I have increased the table cache to 512 but I need to wait a couple days to get a good report on some of the other settings.
 
ghostcorps said:
First thing I notice about nginx, is that it is optimised for serving static pages. We do not use static pages and if we did we currently have no issue with doing so using apache.

nginx looks to be sold on the idea of doing a few basic things as fast as possible, which is great for static pages and ultra light websites. But our site is dynamic and content heavy. On a day to day basis we do not expect to have a huge amount of traffic and certainly do not expect to be serving 1000s of pages at a time. It is nice to have the overhead availability but I use a few of the advanced apache functions that nginx does not support, one specificly is 'digest access authentication' which we use for our secured streaming sessions. Without that I will not be able to get the webmaster to sign off on the Change.

I have also read that there is no point looking at php-fpm without nginx because mod_php serves the same function within apache.

Our webmaster is doing his own investigation, but I doubt we will be doing nginx or php-fpm until it supports the more complex tasks we need from a webserver.



Back to the real task at hand...

Don't be so misinformed.
http://wiki.nginx.org/HttpAuthDigestModule

These isn't anything that you're doing which can't be supported by nginx. A few years back, nginx WAS as you described, but that's a long time ago.

This link should be VERY helpful for you.
A faster Web server: ripping out Apache for Nginx

It was built for static content but that doesn't mean it is slow or doesn't work with dynamic pages. There are many content-rich sites using nginx. Ultimately, APACHE eats your memory like hell.

In fact I run a video site using nginx and serves dynamic php pages. We have tons of content.
http://www.einthusan.com

Anyways, sorry for going off topic.
 
Thanks for clarifying, I guess I was reading old docs.

I spoke to our webmaster and it turns out we don't use the digest access auth any more anyway. I'm going to do some testing with nginx over the next few days.

Thanks again
 
First benchmark on nginx, just the same static page I used for apache:


Code:
Requests per second:    2130.88 [#/sec] (mean)

This is starting to look very worthwhile :D
 
ghostcorps said:
First benchmark on nginx, just the same static page I used for apache:


Code:
Requests per second:    2130.88 [#/sec] (mean)

This is starting to look very worthwhile :D

I'm glad to see that you're testing nginx now and seeing good results. As you tune it further, along with FreeBSD tuning using sysctl.conf and loader.conf, you will see further improvements.

There are many modules (add-ons) for nginx that will help you accomplish your goals.

Have fun! :D
 
einthusan said:
I'm glad to see that you're testing nginx now and seeing good results. As you tune it further, along with FreeBSD tuning using sysctl.conf and loader.conf, you will see further improvements.

There are many modules (add-ons) for nginx that will help you accomplish your goals.

Have fun! :D


Yeh, so far so good. :)

I would still like to know why apache was being such a dog, but oh well it will be behind me soon enough :)
 
Hi guys,

nginx is killing me :( I can get html pages to load, but php files return 'no input files'.

I have been through a tonne of tutes and confirmed the path to the webroot and to fastcgi_params are correct.

I confirmed php-fpm is listening on port 9000 with sockstat | grep 9000

Would anyone mind posting their /usr/local/etc/nginx/nginx.conf and /usr/local/etc/php-fpm.conf?

This is what I have in my gninx.conf so far:

/usr/local/etc/nginx/nginx.conf
Code:
worker_processes  1;

error_log                       /var/log/nginx-error.log;

events  {
        worker_connections      1024;
        }

http    {
        include                 mime.types;
        default_type            application/octet-stream;
        log_format      main    '$remote_addr - $remote_user [$time_local] "$request" '
                                '$status $body_bytes_sent "$http_referer" '
                                '"$http_user_agent" "$http_x_forwarded_for"';

        access_log              /var/log/nginx-access.log  main;

        sendfile                on;
        keepalive_timeout       65;
        gzip                    on;

        server {
                listen                  192.168.0.2:90;
                server_name             URL.com;

                location /
                        {
                        root            /usr/local/www/nginx;
                        index           index.php index.html index.htm;
                        }
                error_page              500 502 503 504  /50x.html;

                location = /50x.html
                        {
                        root            /usr/local/www/nginx-dist;
                        }

                location ~ \.php
                        {
                        fastcgi_pass    127.0.0.1:9000;
                        fastcgi_index   index.php;
                        fastcgi_param   SCRIPT_FILENAME         /scripts$fastcgi_script_name;
                        include         /usr/local/etc/nginx/fastcgi_params;
                        }

                location ~ /\.ht
                        {
                        deny            all;
                        }
                }
      }

/usr/local/etc/php-fpm.conf
Code:
[global]
pid = run/php-fpm.pid
error_log = /var/log/php-fpm.log
log_level = notice
daemonize = yes

[www]
user = www
group = www
listen = 127.0.0.1:9000
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3


Thanks :)
 
Here is parts of my config, this is on a ubuntu box so the paths are different than yours. I have my config files separated but you could just include them all in one file.

nginx.conf
Code:
user www-data;
worker_processes 1;
pid /var/run/nginx.pid;

events {
  worker_connections 1024;
  # multi_accept on;
}

http {

  ##
  # Basic Settings
  ##
  sendfile off;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 2;
  types_hash_max_size 2048;
  charset utf-8;
 
  # server_tokens off;
  # server_names_hash_bucket_size 64;
  server_name_in_redirect off;

  include /etc/nginx/mime.types;
  default_type application/octet-stream;
  index index.php index.htm index.html;

  ##
  # Gzip Settings
  ##
  gzip on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied any;
  gzip_comp_level 9;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

  ##
  # If HTTPS, then set a variable so it can be passed along.
  ##
  #map $scheme $server_https {
  #  default off;
  #  https on;
  #}

  ##
  # Virtual Host Configs
  ##
  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/sites-enabled/*;

}

Notice the lines, include /etc/nginx/conf.d/*.conf; and include /etc/nginx/sites-enabled/*;

/etc/nginx/sites-enabled/mysite.conf
Code:
server {
  listen   80;
  server_name  mysite.com;
  rewrite ^ http://www.mysite.com$request_uri? permanent;
}

server {
  listen   80;
  server_name www.mysite.com;

  access_log off;
  error_log /var/www/public_html/www.mysite.com/log/error.log error;

  root   /var/www/public_html/www.mysite.com/public;
  try_files $uri $uri/ /index.php?q=$uri&$args;

  error_page  401  /errorpages/index.php?error=401;
  error_page  403  /errorpages/index.php?error=403;
  error_page  404  /errorpages/index.php?error=404;
  error_page  500 502 503 504  /errorpages/index.php?error=500;

  # serve static files directly
  location ~* ^.+\.(js|css|jpg|jpeg|gif|png|ico|swf|eot|woff|ttf|svg)$ {
    expires 30d;
  }

  # Deny all attempts to access hidden files such as .htaccess, .htpasswd
  location ~ /\. {
    deny all;
    access_log off;
    log_not_found off;
  }

  location ~ \.php$ {
    try_files $uri =404;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    include /etc/nginx/fastcgi_params;
    fastcgi_pass unix:/var/run/php5-fpm.sock;
  }
}
 
Argh I forgot to sanitise the configs. einthusan, would you mind removing my domain from your post? [ done -- Mod.] Thanks :)

I am using port 90 as it is running side by side with apache on 80. Once it is working on my test server I will move the configs over to the production box, but will keep port 90 until it is running exactly as I need. Then I will stop apache for good and move nginx to 80.

I will follow up on your suggestions when I get home from work later tonight.
 
It was a silly mistake.

I had the root dir set in the wrong section. I moved it from:

/usr/local/etc/nginx.conf
Code:
...
        server {
                listen                  192.168.0.2:90;
                server_name             URL.com;

                location /
                        {
                        root            /usr/local/www/nginx;
...

To

/usr/local/etc/nginx.conf
Code:
...
http    {
        root            /usr/local/www/nginx;
...





:)
 
Back
Top