The move over has been incredibly positive so far. I feel better about the company, the work is interesting and the people are great. I had my first client meeting this week and it went exceptionally well, we’re now starting to build their new web site, which I hope is going to be a vast improvement on their current one.

I’ve also been getting some things set up here for us as a company, including our new hosting environment, rebuilding the company site in WordPress, and a few other bits and bobs. The office itself and the area it’s in is pretty cool too, I’ve included a couple of pictures – one of the office building and one of my desk.

It’s an exciting time and I think I’m going to be busy for quite some time, but I’m feeling good about it and looking forward to the challenge.

However, Google has now announced that it will be dropping support for Internet Explorer 6, starting with Google Apps as early as March 1st. Google Mail and their other properties will also phase out support shortly afterwards. This strikes me as the moment that Google has decided that IE6 market share is no longer big enough to allow it to hold back progress any longer. This announcement is big enough that it’s one of the top three stories on the BBC News homepage.

So one question comes up. Does this mean that we, as regular every day web developers, can finally drop support for Internet Explorer 6 in our own work? My site doesn’t work in the browser currently, but purely due to laziness on my part, can I now say that actually, it never will? It feels like it could be that moment.

I’ve developed a Google Chrome extension that strips out everything in the results area apart from the “ten blue links”, your search results. This includes Images, Maps, Video, Twitter, Site Links and Ads. You can download the extension from the Google Chrome Extensions site.

If you’ve got any feedback, please let me know in the comments.

• Renamed to Page Profiler – makes more sense, as it only checks individual pages
• Speed improved, also no longer an AJAX loaded page (result pages can also be linked to)
• Presentation improved
• Added data size (Plus GZIP compression checking and GZIP data size)
• Added HTTP response header
• Added title and meta description
• Added internal link reporting
• Added number of redirects encountered when loading URL
• Added request time
• Improved whois checking
• Added warnings for key SEOMoz Web Developer SEO Cheat Sheet Recommendations.

The user agent you may see in your access logs for this tool is now:
Mozilla/5.0 (compatible; PageProfiler/2.0; +http://www.dancryer.com/page-profiler)

If you’re using the tool, or have any suggestions for improvements, please let me know in the comments below!

I’m offering out these projects as I do not have the time to complete them myself at the moment. They are projects of vastly varying types, scale and complexity.

If you are interested in taking on some of this work, please email me at dan.cryer@veris.co.uk. If you know anyone else who may be interested, please encourage them to read this post and email me.

I’ve been using Google Wave for a while now, as I was in the first round of invites, and the developer preview before that… but I say using in the loosest sense of the word, as I can’t actually find a use for it. It may be simply because barely anyone is on it yet, but nothing has struck me as a clear purpose.

That’s not to say I don’t like it, it’s a great application and it’s technologically outstanding. Features like drag and drop photo uploading and live typing are incredibly impressive, but I feel like I’m missing the point. I feel like I’m not using it properly, or that I’m expecting the wrong things from it.

What is Google Wave, and why should I be using it? I’d love to know what you all think!

P.s. Thank you to everyone who downloaded the Twitter extension yesterday, including the Googlers… I assume some of the Chrome team.

You can download the extension here, automatic updates are enabled, so you should get new versions as and when I put them up. Please be aware that this extension is quite rough around the edges, (and quite spectacularly ugly,) so please let me know if you manage to break it!

If you choose to use the extension, please let me know, either by commenting below… or by tweeting this page!

a[rel~=nofollow]
{
background-color: #FCC !important;
border: 1px dashed #F55 !important;
color: #600 !important;
}

I should really have thought of that in the first place! Want the extension? Download it here.

Being an epic nerd, the first thing I checked was whether or not it validates. Of course, it does.

It’s worth noting that the tool crawls roughly five pages on the target site each time it is run, so you may notice the following user agent in server logs:

Mozilla/5.0 (compatible; SiteProfiler/1.0; +http://www.dancryer.com/site-profiler)

I thought I’d make it available on this site, so others can use it. If you like it, or you’ve got any feedback, please let me know.

Having run the code with the changes we made previously for a day or so, in order to give it a fair trial, it was obvious it wasn’t going to work. We were still doing an average of 5,000 pages an hour, way below the original code and even further below the targets we’d set ourselves.

The first change we made was to move the queue table back to InnoDB from MEMORY. The memory engine wasn’t providing the benefits we’d hoped, and it was using far more RAM than it was worth. It was also locking, a lot. I have to admit, it was probably unwise to assume that we could expect memory tables to perform any better, when working with over a gigabyte of data. Next, we turned our attention to what had become an obvious bottleneck: statement based binary logging. We’d made this choice early on, based on reasons unbeknownst to me. After much research and a little panic, we flipped the switch and turned our crawlers back on, seeing an immediate and significant performance improvement.

We’d finally done it, we were now running an average of 25,000 pages an hour, peaking at 45,000. We’re still working on it, of course, as we’d like to hit 100,000 pages an hour, but we’re very happy to have finally gotten past the bottlenecks.

As an additional interesting tidbit, I re-ran the graphs for our Memcache utilisation. This time over the period 12:00pm Thursday to 12:00pm Monday. As you can see, the hits to misses ratio is starting to level out over time, averaging around 70% hits, 30% misses. Here’s the charts:

As usual, please let me know if you’ve got any questions or comments, I (and I’m sure the others on my team) would be happy to help if I can.

• One master – All writes go to this server.
• Two slaves:
  • Slave One runs in our office, is a somewhat less powerful machine than the master, but usually keeps up. This slave’s job is to do a daily backup, as such, it can be up to six hours behind, as the backup takes that long.
  • Slave Two runs in the same rack as the master at our data centre, this box is of equal spec to the master and is usually 0-2 seconds behind. This slave’s job is to serve our reporting systems.
• Row-based replication.

When we arrived, we were hit with Slave Two throwing errors such as: “Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event’s master log”. Slave One was, and has remained, fine. Remo, Ben, and I spent quite some time Googling for a solution, but that turned up absolutely nothing. There were a few suggestions, such as setting “slave_exec_mode” to “IDEMPOTENT”. This is merely silent error suppression for such errors, and seems to be suggested for environments using master to master replication and/or cluster-based replication.  We were very uncomfortable with using this, as it meant that our slaves could slowly slip completely out of sync on all tables, and we’d never know about it.

Luckily for us, the affected table is one that we never need to read from on the slaves. It’s a queue table, which we update on the master, and then need to immediately read those rows from, so we do that on the master too. Our solution became clear later in the afternoon. As MySQL allows you to use different storage engines on your slaves to the master, we went the dirty route. The queue tables on the slaves are now powered by the BlackHole storage engine.

Getting the slave back up and running was a whole task in itself, however, as thousands upon thousands of queries had failed in the time this problem had been occuring. The commonly accepted solution is to do the following:

STOP SLAVE;

SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1;

START SLAVE;

SHOW SLAVE STATUS\G;

However, manually running this thousands of times was going to become tedius, so one of the team wrote a script to do it for us. It’ll automatically check that it’s only bypassing errors on a specific table (look out for “database.table” in the script), and bypass them all. It’s not perfect, but it seems to work pretty well. Here’s the code: http://pastie.org/private/mcxnw1oienrlfswciix4w

• http://www.hotelsin-york.co.uk/
• http://www.hotelsin-durham.co.uk/
• http://www.hotelsin-edinburgh.co.uk/
• http://www.hotelsin-glasgow.co.uk/
• http://www.cheaphotels-blackpool.co.uk/
• http://www.cheaphotels-liverpool.co.uk/
• http://www.cheaphotels-bristol.co.uk/

As you’ll be able to see if you look at them, they’re all running on the same, custom written, platform. They’re self-managing affiliate sites for Laterooms. None rank particularly well for any terms, and don’t come close to ranking for any competitive terms, and thus traffic is quite low. They also lack any real links from outside.

For a while, I had some Google Local listings up for each of them, and during that time, the sites were converting well on the traffic they got. Unfortunately, I had to take those listings down. Now I’m unsure what to do with them. I don’t really have the time nor the funds to do link building on them myself, but don’t particularly want them to stagnate.

Any suggestions anyone can give would be much appreciated! As before, either leave me a comment, or drop me an email to dan@dancryer.com.

• leedswebdeveloper.co.uk (and .com)
• leedswebservices.com
• webdesignerleeds.co.uk (and .com)

Obviously, these are pretty niche, but if anyone has any suggestions, let me know. Either leave me a comment here, or email to dan@dancryer.com.

I modified our applications to start checking Memcached first, using the PECL ‘Memcache’ extension, and if no results were found, access the database, get the data and put it into Memcached. Below are a couple of graphs of the first hour of our rollout:

As you can see, the number of requests started out pretty low, as the number of application instances ramped up… This caused a massive spike in misses whilst the cache warmed up. However, as soon as the number of requests started to rise, the balance shifted, and since the rollout, the ratio has remained a fairly stable 60/40 split in favour of hits.

As you may also be able to tell from the graphs, this means that of the 2.5 million hits an hour, we’re managing to catch 1.5 million before they ever make it to the database. This is a huge saving for us, and we’re now looking at where else we can roll this out with similar effect.

On the note of our scaling issues, I’ll be writing up some more of what we’ve done with the databases later. We’re at a point now where the improvements are actually starting to make a difference!