Archive for January, 2012

In part 1 I showed you how to use a combination of pound, haproxy and stunnel to create a cookie based load balancing solution on Debian GNU/Linux 6.0. In part two I will show you how to make the system more resilient.

Aims

If you have followed the instructions in part 1 you should have two web servers and a server that is acting as a load balancer between them. This solution will work well, but in the event of an unexpected failure in one of the nodes the system will cease to function properly. Either half your users will have their connection forwarded to a broken web server, or (if the load balancer fails) the whole system will be unavailable.

In this part we will set up two new features.

  1. A second stand-by load balancer that can take over load balancing should the primary one fail.
  2. A system that monitors the web servers and adjusts the load balancing rules to remove servers that have failed.

Outwith the scope of this document is the monitoring that you should be doing anyway so that you can react to failed services. My preference is Nagios but other monitoring systems are available.

Assumptions

The IP addresses will change slightly here. Since 192.168.0.1 was the address of the load balancer, it will remain the address of the load balancer, but only from a user’s perspective. Here are the addresses as they should stand now:

192.168.0.1 – Load balancer IP address that will float between the two load balancers.

192.168.0.2 – Web server 1

192.168.0.3 – Web server 2

192.168.0.4 – Load Balancer 1

192.168.0.5 – Load Balancer 2

Initial Set Up

Change the IP address of your current load balancer to 192.168.0.4, then head over to part 1 and set up your second load balancer with pound, haproxy and stunnel. Make sure to configure pound to listen on address 192.168.0.1, and also to tell haproxy to listen on 192.168.0.1:80 if you are load balancing http as well as https.

Heartbeat

Heartbeat is the software that will allow the secondary load balancer to take over if the primary one fails. Install it with:

apt-get install heartbeat

Then, on both load balancers, configure it by creating the following files:

/etc/ha.d/ha.cf

logfacility     local0
keepalive 2
deadtime 10
warntime 10
initdead 20
udpport 694
auto_failback on
node    balancer1
node    balancer2

ucast   eth0 <other nodes ip address>

In <other nodes ip address> insert the ip address of the other load balancer on each one. This tells heartbeat the IP address of the other node in the cluster. balancer1 and balancer2 are the resolvable host names of the two load balancers.

/etc/ha.d/haresources

balancer1 192.168.0.1 stunnel4 haproxy pound

This tells heartbeat to manage the 192.168.0.1 ip address and also to start/stop stunnel, haproxy and pound. It also specifies balancer1 as the primary load balancer. This, plus the auto_failback on setting tells heartbeat to always use balancer1 if possible and to revert to using it as soon as it comes back to health.

Once you have configured heartbeat, restart it on both load balancers and check the ip address with ifconfig. You should see eth0:0 on the active node with the ip address of 192.168.0.1. Shut down heartbeat on balancer1 and run ifconfig on balancer2 and you should see it take over 192.168.0.1. Start heartbeat on balancer1 and it should take the ip address back.

Monitoring the Web Servers

Monitoring the web servers is done using mon. Mon will constantly monitor the web servers. If one goes down mon will trigger an alert that will adjust the load balancing configuration. The alert program works by maintaining a sqlite database in which it records the “state of the world” and then uses the contents of that database to regenerate the haproxy configuration before restarting haproxy.

From our perspective an SQL database is the simplest way to maintain the “state of the world” so that we do not have to write our own faffy flat file handling code, but the amount of data and amount of access is too small to require a full blown MySQL, Postgres or (God forbid) Oracle installation. For this reason I chose sqlite.

All the steps shown below in this section should be executed on both load balancers.

Install the software:

apt-get install mon sqlite3 libdbd-sqlite3-perl

SQLite Database

Next we need to create our sqlite database as follows:

sqlite3 /etc/mon/balance.db
CREATE TABLE balance (type text, checkurl text, targeturl text, status text);
INSERT INTO balance
VALUES('plain','webserver1','server webserver1 192.168.0.2:80 cookie webserver1 maxconn 5000','up');
INSERT INTO balance
VALUES('plain','webserver2','server webserver2 192.168.0.3:80 cookie webserver2 maxconn 5000','up');
INSERT INTO balance
VALUES('ssl','webserver1','server webserver1 127.0.0.1:82 cookie webserver1 maxconn 5000','up');
INSERT INTO balance
VALUES('ssl','webserver2','server webserver2 127.0.0.1:83 cookie webserver2 maxconn 5000','up');
.exit

That will create your sqlite database. In order for mon to be able to maintain it ensure that balance.db is owned by mon and that mon has read/write access to it. You should also ensure that the /etc/mon directory  has mon as its gid and has group write permissions.

The event handler

By default mon event handlers (or alerts as they are known) live in /usr/lib/mon/alert.d

In this directory we need to create the balance.alert program as follows:

#!/usr/bin/perl
use DBI;
use Getopt::Std;
getopts ("g:u");

my $dbargs = {AutoCommit => 1,
              PrintError => 1};

my $dbh = DBI->connect("dbi:SQLite:dbname=/etc/mon/balance.db","","",$dbargs);

if ($opt_u)
{
        $dbh->do("UPDATE balance SET status='up' WHERE checkurl='$opt_g'");
}
else
{
        $dbh->do("UPDATE balance SET status='down' WHERE checkurl='$opt_g'");
}

open HAP, ">/etc/haproxy/haproxy.cfg";
print HAP << "EOT"     global         log 127.0.0.1 local0 notice         user haproxy         group haproxy         daemon         maxconn 20000     defaults         log global         option dontlognull         balance leastconn         clitimeout 60000          srvtimeout 60000         contimeout 5000         retries 3         option redispatch      listen http 192.168.0.1:80          mode http          cookie WEBSERVERID insert          option httplog          balance source          option forwardfor except 192.168.0.1          option httpclose          option redispatch          maxconn 10000 EOT ; my $statcursor=$dbh->prepare("SELECT * FROM balance WHERE type='plain' AND status='up'");
$statcursor->execute();
while(my $statrow=$statcursor->fetchrow_hashref())
{
        my $targeturl=$statrow->{'targeturl'};
        print HAP "     $targeturl\n";
}
$statcursor->finish();

print HAP << "EOT"     listen https 127.0.0.1:81          mode http          cookie WEBSERVERID insert          option httplog          balance source          option forwardfor except 192.168.0.1          option httpclose          option redispatch          maxconn 10000 EOT ; my $statcursor=$dbh->prepare("SELECT * FROM balance WHERE type='ssl' AND status='up'");
$statcursor->execute();
while(my $statrow=$statcursor->fetchrow_hashref())
{
        my $targeturl=$statrow->{'targeturl'};
        print HAP "     $targeturl\n";
}
$statcursor->finish();

close HAP;
$dbh->disconnect();

my $heartbeat = system("/sbin/ifconfig | grep 192.168.0.1");
if($heartbeat == 0)
{
        `sudo /etc/init.d/haproxy restart`
}

Mon runs this program using options -g to specify the web server on which it is alerting, and -u  if it is alerting that the server has recovered.

In order for mon to be able to run this program you need to ensure that the mon user can write haproxy.conf in the /etc/haproxy directory. You also need to tell sudo that mon can run the /etc/init.d/haproxy script without needing a password. In /etc/sudoers add the following (preferably by using visudo to edit it):

mon ALL = NOPASSWD: /etc/init.d/haproxy

This program, firstly updates the sqlite database with the status of the web server on which it is alerting. It then reads from the database and creates /etc/haproxy/haproxy.conf according to the servers that are registered as “up” and then (if running on the active load balancer) restarts haproxy.

mon.cf

Configure mon as follows:

cfbasedir   = /etc/mon
alertdir   = /usr/lib/mon/alert.d
mondir   = /usr/lib/mon/mon.d
maxprocs        = 20
histlength = 100
randstart = 30s
logdir = /var/log/mon
dtlogging = yes
dtlogfile = dtlog

hostgroup webserver1 webserver1.stir.ac.uk
hostgroup webserver2 webserver2.stir.ac.uk

watch webserver1
        service apache
        interval 10s
        monitor http.monitor
        period wd {Sun-Sat}
        numalerts 1
        alert balance.alert
        upalert balance.alert

watch webserver2
        service apache
        interval 10s
        monitor http.monitor
        period wd {Sun-Sat}
        numalerts 1
        alert balance.alert
        upalert balance.alert

Once done, restart mon.

Extra Resilience

Pound, haproxy and stunnel are fairly robust but I have known stunnel to crash before now. If you want to guard against any of the key processes crashing then you can configure mon to watch for them, and to shutdown heartbeat if any of them die (thus migrating load balancing to the secondary server which will start its own processes up at that point.

Mon does not provide an easy to use monitor to check if processes are up so I chose to use nagios for this purpose since the systems I use are already set up for full nagios monitoring and event handling.

Please get in touch if you have any comments or ways to improve this.

I am really sorry if I have angered you, as angering motorists is not something that I set out to do today when I got on my bicycle.
However I’d like to try and reason some of your anger away and respond to some of the typical issues.

Common Myths Regarding Cyclists

You Don’t Pay Road Tax

Please do not use this as an excuse for believing that cyclists have fewer rights on the road than you. “Road Tax” was abolished in 1937 because the government did not want those who pay it to believe that they have more rights than those who do not. The money you spend on your paper disc is not the sole funder of the roads. Roads are funded from general taxation, typically from central government for the main highways, and from local authorities for local roads. Therefore as a payer of income tax, VAT and council tax I pay my contribution towards the upkeep of the roads.

What most people call “road tax” is actually “vehicle excise duty” and is levied on the basis of emissions. There are many classes of road users who do not pay VED but still have the right to use the roads. Cyclists are one such class. In fact cyclists and horse riders have an automatic right to use the public highway. Motor vehicles are only allowed there by license after a series of stringent tests have been passed. Before flaming cyclists you need to ask if your driving was up to the standard required from the DVLA.

Please see http://ipayroadtax.com for further details.

You should be in the Cycle Lane

Cycle lanes are provided for the convenience and safety of the cyclists that choose to use them, however their use is not compulsory. The highway code has this to say about their use.

 

61

Cycle Routes and Other Facilities. Use cycle routes, advanced stop lines, cycle boxes and toucan crossings unless at the time it is unsafe to do so. Use of these facilities is not compulsory and will depend on your experience and skills, but they can make your journey safer.
62

Cycle Tracks. These are normally located away from the road, but may occasionally be found alongside footpaths or pavements. Cyclists and pedestrians may be segregated or they may share the same space (unsegregated). When using segregated tracks you MUST keep to the side intended for cyclists as the pedestrian side remains a pavement or footpath. Take care when passing pedestrians, especially children, older or disabled people, and allow them plenty of room. Always be prepared to slow down and stop if necessary. Take care near road junctions as you may have difficulty seeing other road users, who might not notice you.

[Law HA 1835 sect 72]
63

Cycle Lanes. These are marked by a white line (which may be broken) along the carriageway (see Rule 140). Keep within the lane when practicable. When leaving a cycle lane check before pulling out that it is safe to do so and signal your intention clearly to other road users. Use of cycle lanes is not compulsory and will depend on your experience and skills, but they can make your journey safer.

Cycle lanes are not always the safest places to ride. Sometimes they are too narrow and encourage cars to overtake too close. They may contain debris, potholes, snow, or sunken drains. They may take you into the “door zone” of parked cars, that area where a carelessly flung open car door could cause a fatal accident. They may be at the side of bus lanes and encourage cyclists to go up the left hand side of a larger vehicle which is extremely dangerous. Some of them are just downright stupid in their design. See here for some examples of brainless cycle lane design.

As mentioned earlier bicycles are allowed on the road by right, so the existence of a cycle lane does not mean a cyclist has to use it.

This brings us on to…

You’re in the middle of the road

By this I assume the middle of the lane, rather than cycling down the white line? It is a common misconception that cyclists should hug the gutter. There are several reasons why hugging the gutter is a bad idea.

  1. No escape route. If a pothole, drain or other obstruction appears you can only swerve right to avoid it, possibly into the path of the car that is overtaking you too closely.
  2. It encourages close passes. Some drivers are unaware how much space they should give a cyclist. If there is room to overtake without crossing the central line some drivers will do so.
  3. Visibility. You are more visible to traffic turning onto the road out of junctions or parking spaces the further out from the kerb you are.
  4. You are turning right. You should never overtake a vehicle that is signalling to turn right, and this includes cyclists. A right turning cyclist will move across the lane and should be able to do so unimpeded by overtaking motor vehicles.

The highway code has this to say about overtaking.

 

163

Overtake only when it is safe and legal to do so. You should

not get too close to the vehicle you intend to overtake
use your mirrors, signal when it is safe to do so, take a quick sideways glance if necessary into the blind spot area and then start to move out
not assume that you can simply follow a vehicle ahead which is overtaking; there may only be enough room for one vehicle
move quickly past the vehicle you are overtaking, once you have started to overtake. Allow plenty of room. Move back to the left as soon as you can but do not cut in

Give vulnerable road users at least as much space as you would a car

take extra care at night and in poor visibility when it is harder to judge speed and distance
give way to oncoming vehicles before passing parked vehicles or other obstructions on your side of the road
only overtake on the left if the vehicle in front is signalling to turn right, and there is room to do so
stay in your lane if traffic is moving slowly in queues. If the queue on your right is moving more slowly than you are, you may pass on the left
give motorcyclists, cyclists and horse riders at least as much room as you would when overtaking a car (see Rules 211-215)

Remember: Mirrors – Signal – Manoeuvre

 

It also includes this image to demonstrate how to overtake a cyclist.
How to safely overtake a cyclist

You can see from this that a cyclist has the right to use as much of the lane as they see fit. If you cannot overtake safely then you should wait patiently until you can.

You should not be cycling two abreast

The highway code states that we should not cycle more than two abreast, and should cycle single file on busy or narrow roads and when cornering.

Particularly when out with a group it is often advisable to cycle two abreast. It means that the train of cyclists is shorter and so easier to overtake. It also means that you cannot overtake when there is oncoming traffic and so encourages safer overtaking.

You’re a bunch of red-light-jumping pavement-cycling hooligans

It is true that some cyclists are idiots that jump red lights and break other highway code rules. However, the same is true for some motorists. It is not intelligent to tar us all with the same brush and use this as an excuse for anger towards a cyclist whom you have not witnessed cycling in this manner.

Common driving errors that irk cyclists

If you are guilty of one of these manoeuvres then you can expect a cyclist to berate you and possibly post a video of your sin on youtube, since some cyclists have started using video cameras while riding.

The close pass

Not much needs to be said about this since I show above how to overtake cyclists. Please do so safely as no one likes to be cut up.

The left hook

If you are turning left you should not be overtaking just before you do so. Overtaking a cyclist and then immediately slowing to turn left is extremely dangerous. You should hang back for a few seconds and let them clear the junction before you turn.

SMIDSY

“Sorry Mate, I Didn’t See You”.

The misjudged pull-out

A cyclist can be travelling at up to 40MPH, typically in the high 20s on the flat. Don’t assume that you have loads of time to pull out in front of one.