Posts tagged: mysql

We’re doing some research into how to scale our MySQL database from the current Master and Two Slaves set up, to a scalable cluster, ideally without going down the master to master replication route MySQL offer. In doing so, Wade stumbled across the “Spider” storage engine, but information seems sorely lacking and the home page [...]

Alongside my work on our current crawler, Wade has been rewriting the system to make better use of what we’ve learned so far, and make a number of changes: The current codebase is a mess, many cron jobs running terribly written classes, across two separate crawler systems. The new system is neatly organised with models [...]

I posted last week about our work on scaling our crawler application and it’s corresponding MySQL database, but left it with very little conclusion, apart from that Memcached is good. I wanted to follow up with some of the changes we made on Thursday and Friday. Having run the code with the changes we made [...]

In our ongoing, joyous adventures with scaling MySQL, we hit another stumbling block this morning. We arrived to find out that one of our two MySQL slaves had stopped replicating at 6AM on Saturday morning. This meant that our reporting systems were all running on two day old data, even though we’d been diligently collecting [...]

If you read my previous post about our scaling issues, you’ll know that I was planning on trying out a number of other ideas to see if we could improve the performance of our system. Well, one of the things I tried was to test out Memcached. I set up a six node VPS (2.4 [...]