HAProxy is a well known, powerful load balancer. Best known for use with HTTP servers, but it can work with any kind of TCP traffic, including MySQL. Upcoming versions will support native MySQL monitoring, so we’ll be able to retire the custom scripts we created in the previous step, when they’re available in apt-get.

1. Install HAProxy using the following apt-get command: apt-get install haproxy

2. Delete the file /etc/haproxy/haproxy.cfg and create a new one with the following contents:

global
log 127.0.0.1 local0
maxconn 4096
user haproxy
group haproxy
daemon

defaults
log global
mode tcp
option tcplog
option dontlognull
retries 3
option redispatch
maxconn 2000
contimeout 4000
clitimeout 50000
srvtimeout 30000
stats enable
stats scope .

frontend mysql_cluster
bind {load-balancer-ip}:3306
default_backend mysql_cluster

backend mysql_cluster
mode tcp
balance roundrobin
stats enable
option tcpka
option httpchk
server {node-name} {ip-address}:3306 weight 1 check port 9200 inter 5s rise 2 fall 2
server {node-name} {ip-address}:3306 weight 1 check port 9200 inter 5s rise 2 fall 2

listen stats {load-balancer-ip}:80
mode http
option httpclose
balance roundrobin
stats uri /
stats realm Haproxy\ Statistics
stats auth {username}:{password}


Once you’ve modified this file with settings appropriate to your environment, you’ll have a ready to go load balancer for MySQL. It’ll open port 3306 on the load balancer and forward those connections using round-robin load balancing to your two cluster servers. If either server goes down, it’ll be removed from the pool and no connections will be sent to it until it comes back up.

Additionally, it’ll create a statistics interface similar to the following, which you can access by going to your load balancing server’s address in your browser.

Each cluster node should be listed under the first “mysql_cluster” section. They should all appear in green, as they should all be live. If a server is down, it’ll appear red, as explained in the key at the top of the page. From this interface you can also see how long the load balancer has been up, how long each server has been up, how much downtime each have had, as well as the individual number of downtime incidents. As you can see in the image above, our ‘cluster-1′ server has had 21 separate downtime incidents, totalling just over 9 minutes of downtime.

So what do you need to change in your haproxy.cfg file? Here’s a quick rundown:

• Firstly, you’ll need to change the line starting ‘bind’ to use your load balancer’s public IP address.
• Secondly, make the same change on the line starting ‘listen stats’. You may also choose to put this service on a different port than port 80.
• Next, change the {username} and {password} section of the ‘stats auth’ line to ones of your choosing. This provides password protection to your web interface.
• Finally, for each of your MySQL cluster nodes, you’ll need an appropriate ‘server’ line in the ‘mysql_cluster’ backend group. The things you need to change are {node-name}, to the name of the cluster node you defined earlier (e.g. cluster-1, cluster-2) and {ip-address} to the cluster node’s internal IP address.

You may also wish to change the interval at which the load balancer will poll the MySQL servers, by changing ‘inter 5′ to a different number. For more information on HAProxy configuration, their detailed – if a little unfriendly – documentation is available here.

The final step to get this running, is to restart haproxy: /etc/init.d/haproxy restart - You should now be able to access both your web interface on port 80 and your brand new MySQL cluster on port 3306!

If you’ve got any questions about these guides or setting up a similar cluster, please leave a comment below and I’ll do my best to answer it.

On every one of your MySQL cluster nodes, follow the steps below to get it ready for status reporting:

1. Run the following apt-get command: apt-get install xinetd php5-cli php5-mysql – We need xinetd to host our status server, and php5 for the status checking script.

2. Put the following code into a new file: /etc/xinetd.d/mysqlchk – This creates an xinetd “server” on port 9200, using the /opt/mysql-status.php file to generate the response. Our proxy will connect to this server to check the status of the node.

#
# /etc/xinetd.d/mysqlchk
#
service mysqlchk
{
flags = REUSE
socket_type = stream
protocol = tcp
port = 9200
wait = no
user = mysql
server = /opt/mysql-status.php
log_on_failure += USERID
disable = no
}

3. Add the following line at the bottom of /etc/services – This registers your newly created xinetd service with the system to allow it to function.
mysqlchk 9200/tcp # MySQL Monitoring

4. Create the PHP file /opt/mysql-status.php with the code below.
This script tries to connect to your MySQL server, and if it succeeds, checks the ‘Seconds_Behind_Master’ value of the slave status to make sure it is not too far behind it’s master. If either of these checks fail, it will return a 503 error to the client, telling it that the server is down.
#!/usr/bin/php
< ?php
/**
* MySQL Replication Monitor
*/

// Connection details:
$_host = 'localhost';
$_username = '{username}';
$_password = '{password}';
$_timeout = 60; // Number of seconds behind master this node can fall before being marked as "down".

function serverIsUp()
{
$output = 'Server is up. All is okay.';
print "HTTP/1.1 200 OK\r\n";
print "Date: " . date('r') . "\r\n";
print "Server: MySQL Monitor/1.0\r\n";
print "Connection: close\r\n";
print "Content-Type: text/plain\r\n";
print "Content-Length: " . strlen($output) . "\r\n";
print "\r\n";
print $output;
die;
}

5. chown -fv mysql:mysql /opt/mysql-status.php – Give ownership of the status checking script to mysql.
6. chmod -fv +x /opt/mysql-status.php – Give execute permissions on the script.
7. /etc/init.d/xinetd restart – Restart xinetd, to make all of the changes we’ve just made live.

Having completed the steps above, you will now have a script that checks the current status of your MySQL server, responding to connections on port 9200 of your server. In the next article, I’ll describe how to set up your load balancing server with HAProxy, and utilise the server we’ve just created to check which nodes are live at any given time.

This first post is going to cover setting up the servers themselves and configuring MySQL replication.

1. Set up the following three servers:

• Cluster Load Balancer – One Public IP, One Internal IP – Ubuntu Jaunty Basic installation
• Cluster MySQL 1 – One Public IP, One Internal IP – Ubuntu Jaunty Basic + MySQL Server 5.x
• Cluster MySQL 2 – One Public IP, One Internal IP – Ubuntu Jaunty Basic + MySQL Server 5.x

We’re using VPS.NET. If you’re going to do the same, use the ‘Ubuntu 8.04 LTS 64-bit Basic Installation’ template for the Load Balancer and ‘Ubuntu 8.04 x64 MySQL’ for the MySQL nodes. You’ll then need to do the following to upgrade to Jaunty:

• Edit the file ‘/etc/update-manager/release-upgrades‘ and change the line ‘Prompt=lts‘ to ‘Prompt=normal‘.
• Run the command ‘do-release-upgrade‘. Let it go through the entire process of installing Intrepid, then reboot.
• Repeat the step above again to install of Jaunty.

2. On the two MySQL servers, edit /etc/mysql/my.cnf and comment out the line ‘bind-address 127.0.0.1‘ – the servers need to allow remote TCP/IP connections.

3. On each server, add the following to the /etc/mysql/my.cnf file:

server-id = 1 # Increment this number by one for each server, so 1 for cluster-1, 2 for cluster-2
auto_increment_increment = 2 # Set to the number of nodes you have (or are likely to have)
auto_increment_offset = 1 # Set to the same as the server-id
replicate-same-server-id = 0 # To ensure the slave thread doesn't try to write updates that this node has produced.
log-bin # Turn on binary logging (neccessary for replication)
log-slave-updates # Neccessary for chain or circular replication
relay-log # As above
relay-log-index # As above

Note: On our servers, we’ve used an auto_increment_increment of 4, to allow us to add two more nodes to our cluster at a later date, without hitting auto-increment problems.

4. Restart MySQL on both servers.

5. On Cluster 1, log into the MySQL Console and run the following queries:

STOP SLAVE;
CHANGE master TO master_host="cluster-2.yourdomain.com", master_user="{username}", master_password="{password}", master_log_file="mysqld-bin.000001", master_log_pos=98;
START SLAVE;

Note: To verify the master_log_pos value, execute the query: “SHOW MASTER STATUS;” and check the Position column. Also, you may wish to use your internal IP addresses for the master_host option.

6. On Cluster 2, log into the MySQL Console and run the following queries:

STOP SLAVE;
CHANGE master TO master_host="cluster-1.yourdomain.com", master_user="{username}", master_password="{password}", master_log_file="mysqld-bin.000001", master_log_pos=98;
START SLAVE;

Note: If on either server you get an error message at this point, verify that:

• a) the master_log_file name in the query above matches the ones your servers are generating
• b) your username and password for the other server are correct
• c) that your servers can communicate with each other using the host names you specified.
• Additionally, you may wish to remove AppArmor, as that can interfere with MySQL replication and cause all kinds of miscellaneous problems. To do so, run ‘apt-get remove apparmor‘ followed by ‘apt-get purge apparmor‘.

You should now have master to master replication set up and running across your two MySQL servers. To test, create a database on cluster-1, then create a table within that database and insert some data into it. Next, log on to cluster-2 and you should be able to see that newly created table, complete with data.

If you want to ensure that it works both ways, insert some data on cluster-2 and you should see it on cluster-1. You will also want to test that your auto-increment settings are working correctly.

If you’ve got any questions about this article, or have any problems setting up the replication, please leave a comment below and I’ll do my best to help. The next article in this guide will cover setting up a proxy on the load balancer and hopefully, implementing failover.

• Next Step: Setting up MySQL Monitoring on the cluster nodes.

From what we’ve read so far, the Spider Storage Engine is a drop-in replacement for InnoDB. The engine transparently handles both sharding of storage and distributed query processing across a pool of MySQL nodes. Upon creation, you define the column it should look at, and then the range to distribute to each server, and everything else is handled for you. It also appears to be Drizzle compatible. However, apart from that, we know very little else.

I’m hoping that someone reading this will have used Spider in the past and be able to give their opinions. If you have used it, please drop a comment below with your experiences. We want to know about both positives and negatives, any benchmarks, any glaring pitfalls. We’ll likely post about our experiences if we end up testing it, but until then, whatever knowledge people have would be very much appreciated.

• One master – All writes go to this server.
• Two slaves:
  • Slave One runs in our office, is a somewhat less powerful machine than the master, but usually keeps up. This slave’s job is to do a daily backup, as such, it can be up to six hours behind, as the backup takes that long.
  • Slave Two runs in the same rack as the master at our data centre, this box is of equal spec to the master and is usually 0-2 seconds behind. This slave’s job is to serve our reporting systems.
• Row-based replication.

When we arrived, we were hit with Slave Two throwing errors such as: “Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event’s master log”. Slave One was, and has remained, fine. Remo, Ben, and I spent quite some time Googling for a solution, but that turned up absolutely nothing. There were a few suggestions, such as setting “slave_exec_mode” to “IDEMPOTENT”. This is merely silent error suppression for such errors, and seems to be suggested for environments using master to master replication and/or cluster-based replication.  We were very uncomfortable with using this, as it meant that our slaves could slowly slip completely out of sync on all tables, and we’d never know about it.

Luckily for us, the affected table is one that we never need to read from on the slaves. It’s a queue table, which we update on the master, and then need to immediately read those rows from, so we do that on the master too. Our solution became clear later in the afternoon. As MySQL allows you to use different storage engines on your slaves to the master, we went the dirty route. The queue tables on the slaves are now powered by the BlackHole storage engine.

Getting the slave back up and running was a whole task in itself, however, as thousands upon thousands of queries had failed in the time this problem had been occuring. The commonly accepted solution is to do the following:

STOP SLAVE;

SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1;

START SLAVE;

SHOW SLAVE STATUS\G;

However, manually running this thousands of times was going to become tedius, so one of the team wrote a script to do it for us. It’ll automatically check that it’s only bypassing errors on a specific table (look out for “database.table” in the script), and bypass them all. It’s not perfect, but it seems to work pretty well. Here’s the code: http://pastie.org/private/mcxnw1oienrlfswciix4w