HAProxy is a well known, powerful load balancer. Best known for use with HTTP servers, but it can work with any kind of TCP traffic, including MySQL. Upcoming versions will support native MySQL monitoring, so we’ll be able to retire the custom scripts we created in the previous step, when they’re available in apt-get.

1. Install HAProxy using the following apt-get command: apt-get install haproxy

2. Delete the file /etc/haproxy/haproxy.cfg and create a new one with the following contents:

global
log 127.0.0.1 local0
maxconn 4096
user haproxy
group haproxy
daemon

defaults
log global
mode tcp
option tcplog
option dontlognull
retries 3
option redispatch
maxconn 2000
contimeout 4000
clitimeout 50000
srvtimeout 30000
stats enable
stats scope .

frontend mysql_cluster
bind {load-balancer-ip}:3306
default_backend mysql_cluster

backend mysql_cluster
mode tcp
balance roundrobin
stats enable
option tcpka
option httpchk
server {node-name} {ip-address}:3306 weight 1 check port 9200 inter 5s rise 2 fall 2
server {node-name} {ip-address}:3306 weight 1 check port 9200 inter 5s rise 2 fall 2

listen stats {load-balancer-ip}:80
mode http
option httpclose
balance roundrobin
stats uri /
stats realm Haproxy\ Statistics
stats auth {username}:{password}


Once you’ve modified this file with settings appropriate to your environment, you’ll have a ready to go load balancer for MySQL. It’ll open port 3306 on the load balancer and forward those connections using round-robin load balancing to your two cluster servers. If either server goes down, it’ll be removed from the pool and no connections will be sent to it until it comes back up.

Additionally, it’ll create a statistics interface similar to the following, which you can access by going to your load balancing server’s address in your browser.

Each cluster node should be listed under the first “mysql_cluster” section. They should all appear in green, as they should all be live. If a server is down, it’ll appear red, as explained in the key at the top of the page. From this interface you can also see how long the load balancer has been up, how long each server has been up, how much downtime each have had, as well as the individual number of downtime incidents. As you can see in the image above, our ‘cluster-1′ server has had 21 separate downtime incidents, totalling just over 9 minutes of downtime.

So what do you need to change in your haproxy.cfg file? Here’s a quick rundown:

• Firstly, you’ll need to change the line starting ‘bind’ to use your load balancer’s public IP address.
• Secondly, make the same change on the line starting ‘listen stats’. You may also choose to put this service on a different port than port 80.
• Next, change the {username} and {password} section of the ‘stats auth’ line to ones of your choosing. This provides password protection to your web interface.
• Finally, for each of your MySQL cluster nodes, you’ll need an appropriate ‘server’ line in the ‘mysql_cluster’ backend group. The things you need to change are {node-name}, to the name of the cluster node you defined earlier (e.g. cluster-1, cluster-2) and {ip-address} to the cluster node’s internal IP address.

You may also wish to change the interval at which the load balancer will poll the MySQL servers, by changing ‘inter 5′ to a different number. For more information on HAProxy configuration, their detailed – if a little unfriendly – documentation is available here.

The final step to get this running, is to restart haproxy: /etc/init.d/haproxy restart - You should now be able to access both your web interface on port 80 and your brand new MySQL cluster on port 3306!

If you’ve got any questions about these guides or setting up a similar cluster, please leave a comment below and I’ll do my best to answer it.

On every one of your MySQL cluster nodes, follow the steps below to get it ready for status reporting:

1. Run the following apt-get command: apt-get install xinetd php5-cli php5-mysql – We need xinetd to host our status server, and php5 for the status checking script.

2. Put the following code into a new file: /etc/xinetd.d/mysqlchk – This creates an xinetd “server” on port 9200, using the /opt/mysql-status.php file to generate the response. Our proxy will connect to this server to check the status of the node.

#
# /etc/xinetd.d/mysqlchk
#
service mysqlchk
{
flags = REUSE
socket_type = stream
protocol = tcp
port = 9200
wait = no
user = mysql
server = /opt/mysql-status.php
log_on_failure += USERID
disable = no
}

3. Add the following line at the bottom of /etc/services – This registers your newly created xinetd service with the system to allow it to function.
mysqlchk 9200/tcp # MySQL Monitoring

4. Create the PHP file /opt/mysql-status.php with the code below.
This script tries to connect to your MySQL server, and if it succeeds, checks the ‘Seconds_Behind_Master’ value of the slave status to make sure it is not too far behind it’s master. If either of these checks fail, it will return a 503 error to the client, telling it that the server is down.
#!/usr/bin/php
< ?php
/**
* MySQL Replication Monitor
*/

// Connection details:
$_host = 'localhost';
$_username = '{username}';
$_password = '{password}';
$_timeout = 60; // Number of seconds behind master this node can fall before being marked as "down".

function serverIsUp()
{
$output = 'Server is up. All is okay.';
print "HTTP/1.1 200 OK\r\n";
print "Date: " . date('r') . "\r\n";
print "Server: MySQL Monitor/1.0\r\n";
print "Connection: close\r\n";
print "Content-Type: text/plain\r\n";
print "Content-Length: " . strlen($output) . "\r\n";
print "\r\n";
print $output;
die;
}

5. chown -fv mysql:mysql /opt/mysql-status.php – Give ownership of the status checking script to mysql.
6. chmod -fv +x /opt/mysql-status.php – Give execute permissions on the script.
7. /etc/init.d/xinetd restart – Restart xinetd, to make all of the changes we’ve just made live.

Having completed the steps above, you will now have a script that checks the current status of your MySQL server, responding to connections on port 9200 of your server. In the next article, I’ll describe how to set up your load balancing server with HAProxy, and utilise the server we’ve just created to check which nodes are live at any given time.

This first post is going to cover setting up the servers themselves and configuring MySQL replication.

1. Set up the following three servers:

• Cluster Load Balancer – One Public IP, One Internal IP – Ubuntu Jaunty Basic installation
• Cluster MySQL 1 – One Public IP, One Internal IP – Ubuntu Jaunty Basic + MySQL Server 5.x
• Cluster MySQL 2 – One Public IP, One Internal IP – Ubuntu Jaunty Basic + MySQL Server 5.x

We’re using VPS.NET. If you’re going to do the same, use the ‘Ubuntu 8.04 LTS 64-bit Basic Installation’ template for the Load Balancer and ‘Ubuntu 8.04 x64 MySQL’ for the MySQL nodes. You’ll then need to do the following to upgrade to Jaunty:

• Edit the file ‘/etc/update-manager/release-upgrades‘ and change the line ‘Prompt=lts‘ to ‘Prompt=normal‘.
• Run the command ‘do-release-upgrade‘. Let it go through the entire process of installing Intrepid, then reboot.
• Repeat the step above again to install of Jaunty.

2. On the two MySQL servers, edit /etc/mysql/my.cnf and comment out the line ‘bind-address 127.0.0.1‘ – the servers need to allow remote TCP/IP connections.

3. On each server, add the following to the /etc/mysql/my.cnf file:

server-id = 1 # Increment this number by one for each server, so 1 for cluster-1, 2 for cluster-2
auto_increment_increment = 2 # Set to the number of nodes you have (or are likely to have)
auto_increment_offset = 1 # Set to the same as the server-id
replicate-same-server-id = 0 # To ensure the slave thread doesn't try to write updates that this node has produced.
log-bin # Turn on binary logging (neccessary for replication)
log-slave-updates # Neccessary for chain or circular replication
relay-log # As above
relay-log-index # As above

Note: On our servers, we’ve used an auto_increment_increment of 4, to allow us to add two more nodes to our cluster at a later date, without hitting auto-increment problems.

4. Restart MySQL on both servers.

5. On Cluster 1, log into the MySQL Console and run the following queries:

STOP SLAVE;
CHANGE master TO master_host="cluster-2.yourdomain.com", master_user="{username}", master_password="{password}", master_log_file="mysqld-bin.000001", master_log_pos=98;
START SLAVE;

Note: To verify the master_log_pos value, execute the query: “SHOW MASTER STATUS;” and check the Position column. Also, you may wish to use your internal IP addresses for the master_host option.

6. On Cluster 2, log into the MySQL Console and run the following queries:

STOP SLAVE;
CHANGE master TO master_host="cluster-1.yourdomain.com", master_user="{username}", master_password="{password}", master_log_file="mysqld-bin.000001", master_log_pos=98;
START SLAVE;

Note: If on either server you get an error message at this point, verify that:

• a) the master_log_file name in the query above matches the ones your servers are generating
• b) your username and password for the other server are correct
• c) that your servers can communicate with each other using the host names you specified.
• Additionally, you may wish to remove AppArmor, as that can interfere with MySQL replication and cause all kinds of miscellaneous problems. To do so, run ‘apt-get remove apparmor‘ followed by ‘apt-get purge apparmor‘.

You should now have master to master replication set up and running across your two MySQL servers. To test, create a database on cluster-1, then create a table within that database and insert some data into it. Next, log on to cluster-2 and you should be able to see that newly created table, complete with data.

If you want to ensure that it works both ways, insert some data on cluster-2 and you should see it on cluster-1. You will also want to test that your auto-increment settings are working correctly.

If you’ve got any questions about this article, or have any problems setting up the replication, please leave a comment below and I’ll do my best to help. The next article in this guide will cover setting up a proxy on the load balancer and hopefully, implementing failover.

• Next Step: Setting up MySQL Monitoring on the cluster nodes.