Archive for December, 2011

The task was simple. Create a load balancing solution in an attempt at creating high availability on a crucial service. The solution must support SSL and must support cookie-based persistence so that clients will always be sent to the same backend server.

Firstly I’d like to credit Bob Feldbauer of CompleteFusion whose instructions provide the basis for this solution. However Bob’s instructions were slightly lacking in that communication between the load balancer and the application servers is in the clear. Here I attempt to show how to encrypt all network traffic.

HAProxy is an extremely powerful load balancer and is up to the job for the most part. It can insert its own cookies for persistence, however it does not support SSL. This is not a show stopper, but is the reason why I felt the need to document my set up as it is a little complicated.

In order to ensure that haproxy was not involved in SSL I used stunnel. However stunnel can operate in client mode, or server mode, but not both. Creating two instances of stunnel could get messy so I decided to use stunnel in client mode to talk to the application servers, and to use pound on the load balancer to receive connections from the clients.

This system is running on Debian 6.0 (squeeze). Please adjust accordingly if you are using a different system.

Firstly make sure that all the software you need is installed.

apt-get install pound haproxy stunnel4

If you need your application servers to know the IP address of the originating client then check out Bob Feldbauer’s instructions on building your own stunnel including the xforwarded-for patch. Should you choose to build your own stunnel then the simplest way to make sure you are running it is to edit /etc/init.d/stunnel4 and set the DAEMON variable to the location of your hand-built stunnel binary.

Assumptions

I am using three servers here. The load balancer at 192.168.0.1, app server 1 at 192.168.0.2 and app server 2 at 192.168.0.3. Please substitute your appropriate IP addresses (but you knew that anyway, if that was not obvious to you then you shouldn’t be attempting any of this).

Get a Certificate

One of the first things you should do is get a SSL certificate. Have a look at Paul Bramscher’s instructions on how to create SSL certificates, but instead of self-signing it you probably want to get it signed by a recognised certificate authority. When you receive your certificate you need to set it up in the appropriate format for pound to accept it. The certificate file that pound will read needs your unencrypted key at the top of the file, followed by your signed certificate, then any intermediate certificates that your CA may have sent you. In this example I have placed the full certificate file in /etc/ssl/certs/fullcertificate.crt.

While you are in certificate mode I recommend creating self-signed certificates for each of the web servers. These should be installed appropriately on your web server software and will be used by stunnel to verify that it is talking directly to the web servers.

Configuring Pound

Once you have a certificate you can configure pound to receive SSL connections, decrypt them and send them on to haproxy (which we will configure to listen on port 81) in the clear.

Create the following /etc/pound/pound.cfg:

User "www-data"
Group "www-data"
LogLevel 1
Alive 30
Control "/var/run/pound/poundctl.socket"

ListenHTTPS
    Address 192.168.0.1
    Port    443
    Cert    "/etc/ssl/certs/fullcertificate.crt"
    Service
        BackEnd
            Address 127.0.0.1
            Port    81
        End
    End
End

Also (on Debian) you need to edit /etc/default/pound to set startup=1. You can then run pound:

/etc/init.d/pound start

Configuring HAProxy

We will configure two aspects of haproxy. Firstly we can tell it to simply forward requests on port 80 to port 80 on the application servers. Secondly we tell it to take requests on port 81 (ie from pound) and forward them onto stunnel (which we will configure to forward the requests via SSL to the application servers).

Here are the contents of /etc/haproxy/haproxy.cfg (shamelessly copied from Bob Feldbauer and tweaked).

    global
        log 127.0.0.1 local0 debug
        user haproxy
        group haproxy
        daemon
        maxconn 20000

    defaults
        log global
        option dontlognull
        balance leastconn
        clitimeout 60000
        srvtimeout 60000
        contimeout 5000
        retries 3
        option redispatch

    listen http 192.168.0.1:80
        mode http
        cookie WEBSERVERID insert
        option httplog
        balance source
        option forwardfor except 192.168.0.1
        option httpclose
        option redispatch
        maxconn 10000
        reqadd X-Forwarded-Proto:\ http
        server webserver1 192.168.0.2 cookie webserver1 maxconn 5000
        server webserver2 192.168.0.3 cookie webserver2 maxconn 5000

    listen https 127.0.0.1:81
        mode http
        cookie WEBSERVERID insert
        option httplog
        balance source
        option forwardfor except 192.168.0.1
        option httpclose
        option redispatch
        maxconn 10000
        reqadd X-Forwarded-Proto:\ https
        server webserver1 127.0.0.1:82 cookie webserver1 maxconn 5000
        server webserver2 127.0.0.1:83 cookie webserver2 maxconn 5000

Here’s a quick run through of what is going on here.

In the global section we set users, daemon mode and logging. If you want haproxy to log to syslog then you’ll need to switch on UDP port 514 in rsyslog. Find the MODULES section in /etc/rsyslog.conf and add the following lines.

$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 127.0.0.1

Then restart rsyslog (/etc/init.d/rsyslog restart)

Once you have finished and got everything working you may wish to turn logging down to “notice” instead of “debug”.

The listen http 192.168.0.1:80 section is telling haproxy to load balance port 80 between the two webservers and to insert its own WEBSERVERID cookie that it can use for webserver persistence.

The listen https 127.0.0.1:81 section is telling haproxy to receive data on port 81 (from pound) and forward it on to either of the two webservers, but to do so via stunnel (which will be configured to listen on ports 82 and 83 and forward them on via SSL to the webservers). It also sets and uses the WEBSERVERID cookie.

Edit /etc/default/haproxy and set ENABLED=1 before starting haproxy (/etc/init.d/haproxy start).

Configuring stunnel

We configure stunnel (in /etc/stunnel/stunnel.conf) to receive data on port 82 and send it to port 443 over SSL to webserver 1 and to receive data on port 83 and send it to webserver 2. This version uses certificates to verify that it is talking to the web servers and not being intercepted by a man-in-the-middle. If you don’t care to verify the webservers then set verify = 0 and don’t bother with the CAfile lines.

You will need to create the certificate files in the appropriate directory (in this case /etc/stunnel/certs). Each certificate file should contain (in this order) your signed certificate, any intermediate certificates, and finally your private key.

client = yes
verify = 1

#sslVersion = SSLv3

chroot = /var/lib/stunnel4/
setuid = stunnel4
setgid = stunnel4
pid = /stunnel4.pid

socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1

debug = 7
output = /var/log/stunnel4/stunnel.log

[webserver1]
accept = 82
connect = 192.168.0.2:443
CAfile = /etc/stunnel/certs/webserver1.crt

[webserver2]
accept = 83
connect = 192.168.0.3:443
CAfile = /etc/stunnel/certs/webserver2.crt

End of Part One

If you have followed this guide so far you should have a single IP address that load balances across two servers using SSL, both on the front end and on the back end. In Part Two I will look at creating a second load balancer that can take over should your first one fail, and monitoring the web servers so that we can automatically update the balancing rules should one of them fail.