Remo’s Bio:

Remo is Head of Research & Development at Stickyeyes, a search marketing agency based in Leeds. The R&D team works on a mixture of client site builds, internal reporting tools and Market Defender a search marketing intelligence tool. His team works mainly with PHP, MySQL and Apache. Remo graduated with an MSc in pure mathematics having covered programming in Fortran, Pascal, Lisp & C; he joined BT’s R&D Labs at Martlesham Heath working on large scale systems (a dozen mainframes, a few hundred Unix servers…) before getting an MBA and leaving to set up his own web development business. Remo sold the business in 2008 and joined Stickyeyes. Remo is passionate about object-orientated coding and can talk for hours about the Liskov Subsitution Principle, test driven development and solid software engineering.

The Talk – Database Optimisation:

A real life example getting more throughput with fewer queries.

Over the last year we’ve grown a database from a few hundred megabytes to just over one terabyte. The database is reported on and populated by a network of servers using PHP. As the database has grown we’ve had to look again our initial assumptions and ways of working. One table has over 2billion rows; 2.5 million rows every day are added to another table. This talk will cover how we use explain, foreign keys, normalising data without sacrificing performance, queuing and using memcache. And, how we’ve made the system run faster now than it did with a much smaller database.

Without going into too much detail, the talk basically centres around the work that myself, Wade and the rest of our team have been doing over the course of the last year and a half.

Much of the content will be familiar to those who read my blog. It’s about scaling MySQL from a couple of hundred megabyte data store in our office, to a two server set up holding over a terabyte of data, fed by tens of servers, all the whileincreasing the speed at which we could store and retrieve data. It’ll cover how we implemented beanstalkd and memcached to lessen database load, and I’d imagine it will also touch on our ongoing implementation of HAProxy load balancing.

If you’re attending the conference and this sounds interesting to you, make sure you get to the talk. Wade and I will be around as well, to answer any questions that might come up throughout the day. As always, you can also ask the questions here, and if the answer is interesting enough, it might make it into the talk (though no promises!)

HAProxy is a well known, powerful load balancer. Best known for use with HTTP servers, but it can work with any kind of TCP traffic, including MySQL. Upcoming versions will support native MySQL monitoring, so we’ll be able to retire the custom scripts we created in the previous step, when they’re available in apt-get.

1. Install HAProxy using the following apt-get command: apt-get install haproxy

2. Delete the file /etc/haproxy/haproxy.cfg and create a new one with the following contents:

global
log 127.0.0.1 local0
maxconn 4096
user haproxy
group haproxy
daemon

defaults
log global
mode tcp
option tcplog
option dontlognull
retries 3
option redispatch
maxconn 2000
contimeout 4000
clitimeout 50000
srvtimeout 30000
stats enable
stats scope .

frontend mysql_cluster
bind {load-balancer-ip}:3306
default_backend mysql_cluster

backend mysql_cluster
mode tcp
balance roundrobin
stats enable
option tcpka
option httpchk
server {node-name} {ip-address}:3306 weight 1 check port 9200 inter 5s rise 2 fall 2
server {node-name} {ip-address}:3306 weight 1 check port 9200 inter 5s rise 2 fall 2

listen stats {load-balancer-ip}:80
mode http
option httpclose
balance roundrobin
stats uri /
stats realm Haproxy\ Statistics
stats auth {username}:{password}


Once you’ve modified this file with settings appropriate to your environment, you’ll have a ready to go load balancer for MySQL. It’ll open port 3306 on the load balancer and forward those connections using round-robin load balancing to your two cluster servers. If either server goes down, it’ll be removed from the pool and no connections will be sent to it until it comes back up.

Additionally, it’ll create a statistics interface similar to the following, which you can access by going to your load balancing server’s address in your browser.

Each cluster node should be listed under the first “mysql_cluster” section. They should all appear in green, as they should all be live. If a server is down, it’ll appear red, as explained in the key at the top of the page. From this interface you can also see how long the load balancer has been up, how long each server has been up, how much downtime each have had, as well as the individual number of downtime incidents. As you can see in the image above, our ‘cluster-1′ server has had 21 separate downtime incidents, totalling just over 9 minutes of downtime.

So what do you need to change in your haproxy.cfg file? Here’s a quick rundown:

• Firstly, you’ll need to change the line starting ‘bind’ to use your load balancer’s public IP address.
• Secondly, make the same change on the line starting ‘listen stats’. You may also choose to put this service on a different port than port 80.
• Next, change the {username} and {password} section of the ‘stats auth’ line to ones of your choosing. This provides password protection to your web interface.
• Finally, for each of your MySQL cluster nodes, you’ll need an appropriate ‘server’ line in the ‘mysql_cluster’ backend group. The things you need to change are {node-name}, to the name of the cluster node you defined earlier (e.g. cluster-1, cluster-2) and {ip-address} to the cluster node’s internal IP address.

You may also wish to change the interval at which the load balancer will poll the MySQL servers, by changing ‘inter 5′ to a different number. For more information on HAProxy configuration, their detailed – if a little unfriendly – documentation is available here.

The final step to get this running, is to restart haproxy: /etc/init.d/haproxy restart - You should now be able to access both your web interface on port 80 and your brand new MySQL cluster on port 3306!

If you’ve got any questions about these guides or setting up a similar cluster, please leave a comment below and I’ll do my best to answer it.

On every one of your MySQL cluster nodes, follow the steps below to get it ready for status reporting:

1. Run the following apt-get command: apt-get install xinetd php5-cli php5-mysql – We need xinetd to host our status server, and php5 for the status checking script.

2. Put the following code into a new file: /etc/xinetd.d/mysqlchk – This creates an xinetd “server” on port 9200, using the /opt/mysql-status.php file to generate the response. Our proxy will connect to this server to check the status of the node.

#
# /etc/xinetd.d/mysqlchk
#
service mysqlchk
{
flags = REUSE
socket_type = stream
protocol = tcp
port = 9200
wait = no
user = mysql
server = /opt/mysql-status.php
log_on_failure += USERID
disable = no
}

3. Add the following line at the bottom of /etc/services – This registers your newly created xinetd service with the system to allow it to function.
mysqlchk 9200/tcp # MySQL Monitoring

4. Create the PHP file /opt/mysql-status.php with the code below.
This script tries to connect to your MySQL server, and if it succeeds, checks the ‘Seconds_Behind_Master’ value of the slave status to make sure it is not too far behind it’s master. If either of these checks fail, it will return a 503 error to the client, telling it that the server is down.
#!/usr/bin/php
< ?php
/**
* MySQL Replication Monitor
*/

// Connection details:
$_host = 'localhost';
$_username = '{username}';
$_password = '{password}';
$_timeout = 60; // Number of seconds behind master this node can fall before being marked as "down".

function serverIsUp()
{
$output = 'Server is up. All is okay.';
print "HTTP/1.1 200 OK\r\n";
print "Date: " . date('r') . "\r\n";
print "Server: MySQL Monitor/1.0\r\n";
print "Connection: close\r\n";
print "Content-Type: text/plain\r\n";
print "Content-Length: " . strlen($output) . "\r\n";
print "\r\n";
print $output;
die;
}

5. chown -fv mysql:mysql /opt/mysql-status.php – Give ownership of the status checking script to mysql.
6. chmod -fv +x /opt/mysql-status.php – Give execute permissions on the script.
7. /etc/init.d/xinetd restart – Restart xinetd, to make all of the changes we’ve just made live.

Having completed the steps above, you will now have a script that checks the current status of your MySQL server, responding to connections on port 9200 of your server. In the next article, I’ll describe how to set up your load balancing server with HAProxy, and utilise the server we’ve just created to check which nodes are live at any given time.

This first post is going to cover setting up the servers themselves and configuring MySQL replication.

1. Set up the following three servers:

• Cluster Load Balancer – One Public IP, One Internal IP – Ubuntu Jaunty Basic installation
• Cluster MySQL 1 – One Public IP, One Internal IP – Ubuntu Jaunty Basic + MySQL Server 5.x
• Cluster MySQL 2 – One Public IP, One Internal IP – Ubuntu Jaunty Basic + MySQL Server 5.x

We’re using VPS.NET. If you’re going to do the same, use the ‘Ubuntu 8.04 LTS 64-bit Basic Installation’ template for the Load Balancer and ‘Ubuntu 8.04 x64 MySQL’ for the MySQL nodes. You’ll then need to do the following to upgrade to Jaunty:

• Edit the file ‘/etc/update-manager/release-upgrades‘ and change the line ‘Prompt=lts‘ to ‘Prompt=normal‘.
• Run the command ‘do-release-upgrade‘. Let it go through the entire process of installing Intrepid, then reboot.
• Repeat the step above again to install of Jaunty.

2. On the two MySQL servers, edit /etc/mysql/my.cnf and comment out the line ‘bind-address 127.0.0.1‘ – the servers need to allow remote TCP/IP connections.

3. On each server, add the following to the /etc/mysql/my.cnf file:

server-id = 1 # Increment this number by one for each server, so 1 for cluster-1, 2 for cluster-2
auto_increment_increment = 2 # Set to the number of nodes you have (or are likely to have)
auto_increment_offset = 1 # Set to the same as the server-id
replicate-same-server-id = 0 # To ensure the slave thread doesn't try to write updates that this node has produced.
log-bin # Turn on binary logging (neccessary for replication)
log-slave-updates # Neccessary for chain or circular replication
relay-log # As above
relay-log-index # As above

Note: On our servers, we’ve used an auto_increment_increment of 4, to allow us to add two more nodes to our cluster at a later date, without hitting auto-increment problems.

4. Restart MySQL on both servers.

5. On Cluster 1, log into the MySQL Console and run the following queries:

STOP SLAVE;
CHANGE master TO master_host="cluster-2.yourdomain.com", master_user="{username}", master_password="{password}", master_log_file="mysqld-bin.000001", master_log_pos=98;
START SLAVE;

Note: To verify the master_log_pos value, execute the query: “SHOW MASTER STATUS;” and check the Position column. Also, you may wish to use your internal IP addresses for the master_host option.

6. On Cluster 2, log into the MySQL Console and run the following queries:

STOP SLAVE;
CHANGE master TO master_host="cluster-1.yourdomain.com", master_user="{username}", master_password="{password}", master_log_file="mysqld-bin.000001", master_log_pos=98;
START SLAVE;

Note: If on either server you get an error message at this point, verify that:

• a) the master_log_file name in the query above matches the ones your servers are generating
• b) your username and password for the other server are correct
• c) that your servers can communicate with each other using the host names you specified.
• Additionally, you may wish to remove AppArmor, as that can interfere with MySQL replication and cause all kinds of miscellaneous problems. To do so, run ‘apt-get remove apparmor‘ followed by ‘apt-get purge apparmor‘.

You should now have master to master replication set up and running across your two MySQL servers. To test, create a database on cluster-1, then create a table within that database and insert some data into it. Next, log on to cluster-2 and you should be able to see that newly created table, complete with data.

If you want to ensure that it works both ways, insert some data on cluster-2 and you should see it on cluster-1. You will also want to test that your auto-increment settings are working correctly.

If you’ve got any questions about this article, or have any problems setting up the replication, please leave a comment below and I’ll do my best to help. The next article in this guide will cover setting up a proxy on the load balancer and hopefully, implementing failover.

• Next Step: Setting up MySQL Monitoring on the cluster nodes.

Having run the code with the changes we made previously for a day or so, in order to give it a fair trial, it was obvious it wasn’t going to work. We were still doing an average of 5,000 pages an hour, way below the original code and even further below the targets we’d set ourselves.

The first change we made was to move the queue table back to InnoDB from MEMORY. The memory engine wasn’t providing the benefits we’d hoped, and it was using far more RAM than it was worth. It was also locking, a lot. I have to admit, it was probably unwise to assume that we could expect memory tables to perform any better, when working with over a gigabyte of data. Next, we turned our attention to what had become an obvious bottleneck: statement based binary logging. We’d made this choice early on, based on reasons unbeknownst to me. After much research and a little panic, we flipped the switch and turned our crawlers back on, seeing an immediate and significant performance improvement.

We’d finally done it, we were now running an average of 25,000 pages an hour, peaking at 45,000. We’re still working on it, of course, as we’d like to hit 100,000 pages an hour, but we’re very happy to have finally gotten past the bottlenecks.

As an additional interesting tidbit, I re-ran the graphs for our Memcache utilisation. This time over the period 12:00pm Thursday to 12:00pm Monday. As you can see, the hits to misses ratio is starting to level out over time, averaging around 70% hits, 30% misses. Here’s the charts:

As usual, please let me know if you’ve got any questions or comments, I (and I’m sure the others on my team) would be happy to help if I can.

• One master – All writes go to this server.
• Two slaves:
  • Slave One runs in our office, is a somewhat less powerful machine than the master, but usually keeps up. This slave’s job is to do a daily backup, as such, it can be up to six hours behind, as the backup takes that long.
  • Slave Two runs in the same rack as the master at our data centre, this box is of equal spec to the master and is usually 0-2 seconds behind. This slave’s job is to serve our reporting systems.
• Row-based replication.

When we arrived, we were hit with Slave Two throwing errors such as: “Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event’s master log”. Slave One was, and has remained, fine. Remo, Ben, and I spent quite some time Googling for a solution, but that turned up absolutely nothing. There were a few suggestions, such as setting “slave_exec_mode” to “IDEMPOTENT”. This is merely silent error suppression for such errors, and seems to be suggested for environments using master to master replication and/or cluster-based replication.  We were very uncomfortable with using this, as it meant that our slaves could slowly slip completely out of sync on all tables, and we’d never know about it.

Luckily for us, the affected table is one that we never need to read from on the slaves. It’s a queue table, which we update on the master, and then need to immediately read those rows from, so we do that on the master too. Our solution became clear later in the afternoon. As MySQL allows you to use different storage engines on your slaves to the master, we went the dirty route. The queue tables on the slaves are now powered by the BlackHole storage engine.

Getting the slave back up and running was a whole task in itself, however, as thousands upon thousands of queries had failed in the time this problem had been occuring. The commonly accepted solution is to do the following:

STOP SLAVE;

SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1;

START SLAVE;

SHOW SLAVE STATUS\G;

However, manually running this thousands of times was going to become tedius, so one of the team wrote a script to do it for us. It’ll automatically check that it’s only bypassing errors on a specific table (look out for “database.table” in the script), and bypass them all. It’s not perfect, but it seems to work pretty well. Here’s the code: http://pastie.org/private/mcxnw1oienrlfswciix4w

This all started on Wednesday, we’d been looking at the queue and decided that it was far too large. We’d hit 12.5M pages waiting to be checked, and were processing about 200,000 a day, tops. The initial reaction, as this is a distributed system, was to scale it horizontally. We rented six more VPSs (to add to the existing 5) and planned to run them at capacity until the queue was empty. After setting all the boxes up, we left them over night, in hope of completing 750,000 or more pages. Boy, were we wrong. On Thursday morning, we came in to discover it was running at a peak of 5,000 pages an hour (144,000 a day). We’d slowed it down, and not only that, we’d slowed everything else down too.

This is when the optimisation work started. We looked at the queries we were running, and noticed that an obvious bottleneck appeared to be the queue itself. Each VPS ran 15 instances of our application at any given time, and each one did a query for 250 pages to check when it was starting up, updated them as in progress, and deleting each from the queue on completion. The first change we made was to change the indexing on the table, which merely served to slow us down further. The second, more interesting change, was to change the entire table from innodb to the memory/heap table engine. This meant changing both max_heap_table_size and tmp_table_size to over 2.5GB to accomodate the 1GB table at the same time as allowing temporary tables to be created. Getting this change replicated onto our two slaves was a delight.

The memory table “solution” worked well after turning all of our VPSs back on, for about 10 minutes. At which point, they were up to capacity and the queue table started locking, a lot. As did all of the other tables, inexplicably. In addition to inducing panic, this highlighted to us that another table was starting to cause problems, our anchor text table. This is responsible for storing only the words that are used to link from one site to another. This table was seeing anything up to 2 million inserts an hour, and the same number of selects. After several failed attempts to remedy this, we finally settled on selecting all of the anchor texts we’d just collected, any that came back from the database were then dropped from our insert, thus only inserting those that were not present. Those inserts were done with batches of 10 per query. This cut the number of queries down significantly. Whilst this has improved the locking situation, it has, unfortunately, led to hanging selects. These selects are using potentially gigantic ‘IN(one, two, three)’ conditions, and are taking anything from 0.01 to 120 seconds to complete. Completely unpredictably.

Next priority was fixing the table locking on our queue. We’ve done this by marking all deletes from the queue as LOW_PRIORITY. In doing that, we’re allowing MySQL to queue those deletes for when no other clients are accessing the table, preventing unnecessary locking.

At the time of this writing, 113 instances of our application are running without significant conflicts on either of the tables we’ve optimised. However, we’re still processing just 5,000 pages an hour. I’m sure I’ll be working on this problem again on Monday, and will update here with anything we do discover.

So to summarise this somewhat rambley and unhelpful post a little, don’t assume that because you’ve built your system to run across multiple servers, that it can infact run across multiple servers. Ours scaled up to 4, and then the scaling started working against us. If you found it interesting, have any MySQL questions, want to offer advice, or are doing similar work yourself… drop me a comment below. I’d love to know that people are reading this!