HAProxy Linux kernel 2.6 tuning for high load/connections

Hi,

 
I have played around with HAproxy and I must say that it is a great product.

My tests were conducted on a Xen hypervisor with 3 Virtual Machines running 
Ubuntu Server 8.04 64bit:

#1: HAPROXY 1.3.18 compiled from source 128 MB, 1 CPU
#2: Apache  256 MB, 1 CPU
#3 Lighttpd 512 MB, 2 CPU

This is the HAProxy configuration that I used:

dra...@haproxyubu:~$ cat /etc/haproxy/haproxy.cfg
global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 notice
        #log loghost    local0 info
        maxconn 131070
        #debug
        #quiet
        user haproxy
        group haproxy

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option redispatch
        retries 3
        maxconn 65535
        contimeout      5000
        clitimeout      50000
        srvtimeout      50000

listen webfarm 192.168.1.79:80
       mode http
       stats enable
       stats auth someuser:somepassword
       balance roundrobin
       cookie JSESSIONID prefix
       option httpclose
       option forwardfor
       option httpchk HEAD /check.txt HTTP/1.0
       server webA 192.168.1.83:80 cookie A check
       server webB 192.168.1.84:80 cookie B check

Both Apache2-mpm-worker and lighttpd were serving a very simple index.html
<html><body>It works!</body></html>

I started some tests and I soon figured out that all 3 operating systems were 
running out of resources:

dra...@apache1ubu:~$  sudo ab  -n 10000 -c 100 -t 20 
http://192.168.1.79/index.html

This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.79 (be patient)
Finished 1991 requests

Server Software:        lighttpd/1.4.19
Server Hostname:        192.168.1.79
Server Port:            80

Document Path:          /index.html
Document Length:        45 bytes

Concurrency Level:      100
Time taken for tests:   20.769130 seconds
Complete requests:      1991
Failed requests:        997
   (Connect: 0, Length: 997, Exceptions: 0)
Write errors:           0
Non-2xx responses:      3
Total transferred:      572186 bytes
HTML transferred:       90775 bytes
Requests per second:    95.86 [#/sec] (mean)
Time per request:       1043.151 [ms] (mean)
Time per request:       10.432 [ms] (mean, across all concurrent requests)
Transfer rate:          26.87 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0  112 532.6     21    8998
Processing:     1  178 594.4     83   13040
Waiting:        0  163 594.6     72   13039
Total:          1  291 876.6    113   14007

Percentage of the requests served within a certain time (ms)
  50%    113
  66%    135
  75%    156
  80%    181
  90%    331
  95%    532
  98%   3113
  99%   3409
 100%  14007 (longest request)

It seems that the connections stack fills very fast and then the webservers 
responds very slow or timesout.

I don't know if this relates to Xen's hypervisor own TCP/IP implementation or 
the ubuntu server tcp/ip settings.

Based on this
My question is: what settings do you recommend for the Linux kernel/system in 
order that
HAProxy and the webservers to be able to serve many connections at a time ? 

I am looking for answers that relates to these 3 components:

* linux kernel TCP/IP configrations
* ulimit settings
* system wide settings

-------------------------------------------------------------------

What I already did:

sysctl.conf:

# These ensure that TIME_WAIT ports either get reused or closed fast.
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_tw_recycle = 1

# TCP memory
#  net.core.rmem_max = 16777216
#  net.core.rmem_default = 16777216
#  net.core.netdev_max_backlog = 262144
#  net.core.somaxconn = 262144

# net.ipv4.tcp_syncookies = 1
#  net.ipv4.tcp_max_orphans = 262144
#  net.ipv4.tcp_max_syn_backlog = 262144  // this is the connection queue ?
#  net.ipv4.tcp_synack_retries = 2
#  net.ipv4.tcp_syn_retries = 2

dra...@apache1ubu:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1951
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 2000000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1951
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I don't know how to corelate the sysctl settings with my desired simultaneous 
connections/second.

I am sure that here lurks some benevolent Linux kernel experts that are willing 
to help with this :)

So, to get to the point what Kernel, TCP/IP, ulimit settings do you 
have/suggest for high traffic(10000+ simultaneous
connections/second) HAProxy/webservers
Advertisements
Leave a comment

1 Comment

  1. After the kernel tuning, how was the response time with ab?

    Reply

Leave your opinion

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: