The nicely integrated Server Density monitoring does a great job of informing you if a server is down, but it can’t do anything about it. So I’ve created a script that I run on an external server to check all of the servers are up (via HTTP) and reboot those that are not. It’s fairly simple, the process is as follows:

• Get a list of VMs from VPS.NET.
• For each VM, send a GET request to http://{hostname}.{domainname}/ and give it (by default) 5 seconds to respond.
• If the server does not respond, it will check if it is marked as “running” on VPS.NET – by default it will then assume that there is simply a routing issue.
• If it is not, it will check if the server is already in a power change state (i.e. startup, reboot).
• Finally, if neither of the above are true, it will send a power on request to VPS.NET.
• After all of that, it’ll (optionally) send you an email to let you know.

The code is below. You can also download it here. Let me know if you find it useful!

Notes:

• This script needs setting up as a cron job, on an off-site server / computer. I recommend running it every 5 minutes or so.
• I don’t use the VPS.NET API PHP classes because they seem to be out of date.
• This script *will* spam you to oblivion if there is a cloud issue and you set it up to email in all cases.

< ?php

//----
// Set up the monitoring script:
//----

define('MONITOR_MAIL_TO', '');     // Email address to send alerts to.
define('MONITOR_API_EMAIL', ''); // Email address associated with VPS.NET API account.
define('MONITOR_API_KEY', '');     // VPS.NET API key.
define('MONITOR_SEND_EMAILS', true);               // Whether or not to send emails if a server is flagged as down.
define('MONITOR_SEND_EMAILS_ON_NO_ACTION', true);  // Whether or not to send emails if no action is taken.
define('MONITOR_REBOOT_IF_RUNNING', true);         // If HTTP check fails, but VPS.NET says the server is up, reboot?
define('MONITOR_HTTP_TIMEOUT', 5);                 // Number of seconds to allow for a server response. 

//----
// Initialise API Class and get VMs:
//----
$api = new SimpleVpsApi();
$api->setApiEmail(MONITOR_API_EMAIL);
$api->setApiKey(MONITOR_API_KEY);
$vms = $api->getVMs();

//----
// Loop through all the VMs and check their status:
//----
$out     = '';
foreach($vms as $vm)
{
	$server = $vm['virtual_machine']['hostname'] . '.' . $vm['virtual_machine']['domain_name'];
	$id     = $vm['virtual_machine']['id'];

	$get = false;
	$get = @file_get_contents('http://'.$server.'/', false, stream_context_create(array('http' => array('timeout' => MONITOR_HTTP_TIMEOUT))));

	if(!$get)
	{
		$_out .= 'Server ' . $server . ' appears to be DOWN!'.PHP_EOL;

		$vm = $api->getVM($id);

		if($vm['power_action_pending'])
		{
			$_out .= '- VPS.NET is reporting that the server is rebooting already. Taking no action.'.PHP_EOL;

			if(!MONITOR_SEND_EMAILS_ON_NO_ACTION)
			{
				$_out = '';
			}
		}
		elseif($vm['running'] && !MONITOR_REBOOT_IF_RUNNING)
		{
			$_out .= '- VPS.NET is reporting the server as running. Taking no action.'.PHP_EOL;

			if(!MONITOR_SEND_EMAILS_ON_NO_ACTION)
			{
				$_out = '';
			}
		}
		elseif($vm['running'] && MONITOR_REBOOT_IF_RUNNING)
		{
			$api->rebootVM($id);
			$_out .= '- VPS.NET is reporting the server as running. Rebooting.'.PHP_EOL;
		}
		else
		{
			$api->bootVM($id);
			$_out .= '- Submitted an automated power on request to VPS.NET.'.PHP_EOL;
		}

		// If we've logged anything for this VM, append it to the log:
		if($_out)
		{
			if($out != '')
			{
				$out .= PHP_EOL;
			}

			$out .= $_out;
			$_out = '';
		}
	}
}

//----
// Emailing functionality, for notification of server downtime:
//----

$email = '
	

		Server Monitoring Notice

			
		


	
';

if($out != '' && MONITOR_SEND_EMAILS)
{
	$email = str_replace('', nl2br($out), $email);
	mail(MONITOR_MAIL_TO, 'Server Monitoring Notice', $email, 'Content-Type: text/html'."\r\n");
}

//----
// Custom class for interacting with the VPS.NET API:
//----

class SimpleVpsApi
{
	const TYPE_GET  = 1;
	const TYPE_POST = 2;

	protected $apiUrl   = 'api.vps.net/';
	protected $apiEmail = null;
	protected $apiKey   = null;

	/**
	* Set the email address to use in requests:
	*/
	public function setApiEmail($email)
	{
		$this->apiEmail = $email;
	}

	/**
	* Set the API key to use in requests:
	*/
	public function setApiKey($key)
	{
		$this->apiKey = $key;
	}

	/**
	* Get a list of all VMs:
	*/
	public function getVMs()
	{
		$vm = $this->sendApiRequest('virtual_machines', 'GET');

		if($vm)
		{
			return $vm;
		}
		else
		{
			return array();
		}
	}

	/**
	* Sends a request to VPS.NET to load details about a VM:
	*/
	public function getVM($id)
	{
		$vm = $this->sendApiRequest('virtual_machines/'.$id, 'GET');

		if($vm)
		{
			return $vm['virtual_machine'];
		}
		else
		{
			return null;
		}
	}

	/**
	* Sends a request to VPS.NET to reboot a VM:
	*/
	public function rebootVM($id)
	{
		return $this->sendApiRequest('virtual_machines/'.$id.'/reboot', 'POST');
	}

	/**
	* Sends a request to VPS.NET to boot a VM:
	*/
	public function bootVM($id)
	{
		return $this->sendApiRequest('virtual_machines/'.$id.'/power_on', 'POST');
	}

	/**
	* Send an API request to VPS.NET
	*/
	protected function sendApiRequest($uri, $type = 'GET')
	{
		if(!$this->apiEmail || !$this->apiKey)
		{
			throw new Exception('You must set an API key and email address first.');
		}

		$opts = array(
		  'http'=>array(
		    'method'  => $type,
		    'header'  => 'Accept: application/json' . "\r\n" .
		                 'Content-Type: application/json' . "\r\n" .
						 'Authorization: Basic ' . base64_encode($this->apiEmail.':'.$this->apiKey) . "\r\n",

			'timeout' => 3,
		  )
		);

		$url      = 'https://'.$this->apiUrl.$uri.'.api10json';
		$context  = stream_context_create($opts);
		$response = @file_get_contents($url, false, $context);
		return json_decode($response, true);
	}
}

edc17388fb0743a4910e77ba4c1b8171

Remo’s Bio:

Remo is Head of Research & Development at Stickyeyes, a search marketing agency based in Leeds. The R&D team works on a mixture of client site builds, internal reporting tools and Market Defender a search marketing intelligence tool. His team works mainly with PHP, MySQL and Apache. Remo graduated with an MSc in pure mathematics having covered programming in Fortran, Pascal, Lisp & C; he joined BT’s R&D Labs at Martlesham Heath working on large scale systems (a dozen mainframes, a few hundred Unix servers…) before getting an MBA and leaving to set up his own web development business. Remo sold the business in 2008 and joined Stickyeyes. Remo is passionate about object-orientated coding and can talk for hours about the Liskov Subsitution Principle, test driven development and solid software engineering.

The Talk – Database Optimisation:

A real life example getting more throughput with fewer queries.

Over the last year we’ve grown a database from a few hundred megabytes to just over one terabyte. The database is reported on and populated by a network of servers using PHP. As the database has grown we’ve had to look again our initial assumptions and ways of working. One table has over 2billion rows; 2.5 million rows every day are added to another table. This talk will cover how we use explain, foreign keys, normalising data without sacrificing performance, queuing and using memcache. And, how we’ve made the system run faster now than it did with a much smaller database.

Without going into too much detail, the talk basically centres around the work that myself, Wade and the rest of our team have been doing over the course of the last year and a half.

Much of the content will be familiar to those who read my blog. It’s about scaling MySQL from a couple of hundred megabyte data store in our office, to a two server set up holding over a terabyte of data, fed by tens of servers, all the whileincreasing the speed at which we could store and retrieve data. It’ll cover how we implemented beanstalkd and memcached to lessen database load, and I’d imagine it will also touch on our ongoing implementation of HAProxy load balancing.

If you’re attending the conference and this sounds interesting to you, make sure you get to the talk. Wade and I will be around as well, to answer any questions that might come up throughout the day. As always, you can also ask the questions here, and if the answer is interesting enough, it might make it into the talk (though no promises!)

The announcement states that running HipHop has reduced the CPU footprint of the Facebook code base by 50%, over using the standard PHP runtime. Whilst those improvements may not be reflected by all PHP applications, due to the special nature of the Facebook codebase, it represents an undeniably vast improvement to what was previously an engine known for being somewhat hacked together.

As the improved engine is seemingly so much more efficient, I assume a significant portion of the PHP developer community will now start tinkering with HipHop for PHP and the crazier amongst us will start making the switch over on our personal sites and projects. This is going to have a huge impact on the PHP community, web hosting industry and those who work in either. I can only assume that some hosts are already receiving their first “When will we get HipHop for PHP?” emails.

I’m going to set it up on a sub-domain of this site when it’s released this evening, assuming it can run under Apache at the moment, and see how much it affects performance running simple web sites powered by software like WordPress and IP.Board. If it is compatible and makes much of a performance difference, I’ll switch the whole server over here, and start looking at it more seriously for our systems at work.

For more information, see the official announcement by Facebook.

I’ll set your web server up for you, for free, after you sign up to VPS.NET. I’ll get it up and running with Ubuntu Linux, Apache 2.2, PHP 5.2 and MySQL 5.1. I’ll also set up FTP (if desired) and create up to five user accounts and Apache virtual hosts for domains you already have.

All you have to do to take advantage of this is the following:

1. Sign up to VPS.NET using this link (see note 1) - but don’t create a VPS
2. Forward the account details link you get from VPS.NET (including your account username and password) to vps@dancryer.com
3. In the email, include the number of VPS.NET “nodes” you want me to allocate to your VPS, a list of up to five domains you would like to be set up, the server name you want to use (if you have a preference), and whether or not you would like to be able to access it via FTP.

I’ll set up the server as soon as I can and email you back with a list of Ubuntu user names and passwords, MySQL usernames and passwords, and various other server details.

• Renamed to Page Profiler – makes more sense, as it only checks individual pages
• Speed improved, also no longer an AJAX loaded page (result pages can also be linked to)
• Presentation improved
• Added data size (Plus GZIP compression checking and GZIP data size)
• Added HTTP response header
• Added title and meta description
• Added internal link reporting
• Added number of redirects encountered when loading URL
• Added request time
• Improved whois checking
• Added warnings for key SEOMoz Web Developer SEO Cheat Sheet Recommendations.

The user agent you may see in your access logs for this tool is now:
Mozilla/5.0 (compatible; PageProfiler/2.0; +http://www.dancryer.com/page-profiler)

If you’re using the tool, or have any suggestions for improvements, please let me know in the comments below!

I’m offering out these projects as I do not have the time to complete them myself at the moment. They are projects of vastly varying types, scale and complexity.

If you are interested in taking on some of this work, please email me at dan.cryer@veris.co.uk. If you know anyone else who may be interested, please encourage them to read this post and email me.

It’s worth noting that the tool crawls roughly five pages on the target site each time it is run, so you may notice the following user agent in server logs:

Mozilla/5.0 (compatible; SiteProfiler/1.0; +http://www.dancryer.com/site-profiler)

I thought I’d make it available on this site, so others can use it. If you like it, or you’ve got any feedback, please let me know.

If anyone reading this is coming, drop me a note and we’ll look out for you at the conference, especially if you’re currently looking for a developer job in Leeds!

• The current codebase is a mess, many cron jobs running terribly written classes, across two separate crawler systems. The new system is neatly organised with models and controllers, and all run through one central job manager.
• Our MySQL-based job queueing system is being replaced with beanstalkd.
• We’re dropping all auto_increment primary keys in the database, replacing them with pre-calculated binary hashes, a BINARY(16) column with the UNHEX() value of an MD5.
• Doing the above allows us to drop most of the SELECT queries we do, as we don’t need to know if we’ve already got a reference to that page/url/backlink, we can INSERT IGNORE new ones, and it’ll simply skip over it on those we already have. All references to that data will also use the binary hash.
• Our crawler will make all requests under the user agent MarketDefenderBot/2.0 (some of our requests currently use MarketDefenderBot/1.0, but not all), and will respect robots.txt properly.
• We’re putting indexes on what we need and use, and dropping ones we don’t… Crazy, I know.

However, due to the rate at which we’re growing, I’m already beginning to investigate where we can go next. We’re beginning to push some of MySQL’s limits, and we don’t want to simply migrate wholesale to another RDBMS. We’ve got a few options, obviously, including moving specific problems out of MySQL and into other solutions, as we’ve done with beanstalkd for queueing… but my preference is headed rapidly towards migrating to a Hadoop cluster, along with HDFS, MapReduce and HBase.

I know it may seem like simply jumping on the NoSQL bandwagon, but actually, I think that for something of the type and scale we’re working towards, it’s the only option. I realise that there are organisations with much larger datasets running on relational databases, but a consideration a lot of people miss when they make these comparisons, is the use cases.  We’re constantly querying across our entire dataset in a variety of different ways, many of which can’t be optimised with simple indexes.

I’ve also just discovered that Hadoop’s MapReduce can run any application for both the Map and Reduce tasks, meaning we can continue to use PHP until we absolutely have to switch to another language for performance, at which point, we can do it on a task by task basis.

Having run the code with the changes we made previously for a day or so, in order to give it a fair trial, it was obvious it wasn’t going to work. We were still doing an average of 5,000 pages an hour, way below the original code and even further below the targets we’d set ourselves.

The first change we made was to move the queue table back to InnoDB from MEMORY. The memory engine wasn’t providing the benefits we’d hoped, and it was using far more RAM than it was worth. It was also locking, a lot. I have to admit, it was probably unwise to assume that we could expect memory tables to perform any better, when working with over a gigabyte of data. Next, we turned our attention to what had become an obvious bottleneck: statement based binary logging. We’d made this choice early on, based on reasons unbeknownst to me. After much research and a little panic, we flipped the switch and turned our crawlers back on, seeing an immediate and significant performance improvement.

We’d finally done it, we were now running an average of 25,000 pages an hour, peaking at 45,000. We’re still working on it, of course, as we’d like to hit 100,000 pages an hour, but we’re very happy to have finally gotten past the bottlenecks.

As an additional interesting tidbit, I re-ran the graphs for our Memcache utilisation. This time over the period 12:00pm Thursday to 12:00pm Monday. As you can see, the hits to misses ratio is starting to level out over time, averaging around 70% hits, 30% misses. Here’s the charts:

As usual, please let me know if you’ve got any questions or comments, I (and I’m sure the others on my team) would be happy to help if I can.

I modified our applications to start checking Memcached first, using the PECL ‘Memcache’ extension, and if no results were found, access the database, get the data and put it into Memcached. Below are a couple of graphs of the first hour of our rollout:

As you can see, the number of requests started out pretty low, as the number of application instances ramped up… This caused a massive spike in misses whilst the cache warmed up. However, as soon as the number of requests started to rise, the balance shifted, and since the rollout, the ratio has remained a fairly stable 60/40 split in favour of hits.

As you may also be able to tell from the graphs, this means that of the 2.5 million hits an hour, we’re managing to catch 1.5 million before they ever make it to the database. This is a huge saving for us, and we’re now looking at where else we can roll this out with similar effect.

On the note of our scaling issues, I’ll be writing up some more of what we’ve done with the databases later. We’re at a point now where the improvements are actually starting to make a difference!

The first file it generates is a very simple, empty, model class in ./app/model/. This is where you’d put your custom methods, such as hashing for passwords, etc. By default, this file will only be generated the first time you run generate models script.

The second file it generates is the database model itself. This will contain, as properties, all of the columns in your database table. For each of those properties, you’ll also get a nicely formatted getter and setter method. Optionally, the generator will also write you some code to validate input in the setters by their null/not null status, column type and length.

Finally, for now, the generator will optionally generate you a static getter (with SQL) for your primary key, allowing you to do, for example: Model_Whatever::getByID(234);

Usage for the script is as follows:

./generate-models -s=myhost.com [-u=user] [-p=password] -d=database

ADDITIONAL PARAMETERS:
-p MySQL Port. Default: 3306
-sockets Use MySQL Sockets? Default: off, values: on | off
-validation Generate validation code for basic SQL types. Default: on, values: on | off
-exceptions Generate exceptions in validation code. Default: on, values: on | off
-sql Generate basic SQL in database models. Default: on, values: on | off
-prefix Prefix models with Model_. Default: on, values: on | off
-idcase getId() becomes getID(). Default: on, values: on | off
-tables Tables are a multiple of their columns, e.g. reviews.review_id. Default: on, values: on | off
-overwrite Overwrite existing model files (database model files are automatically overwritten). Default: off, values: on | off

EXPERIMENTAL FEATURES:
-php53 Generate PHP 5.3 code (use namespaces). Default: off, values: on | off

It’s worth noting that this does come with some caveats:

• Assumes that table names are in either format: user or users (will make Model_User, Model_Users, respectively)
• Assumes that column names are in the format: user_id (will make getID() in the User[s] model)
• If you use -sql, assumes you have a class named Database that gives back instances of PDO. An example class is bundled.

Over time, I plan to add much more to this. Next on the list are a static create() method, and a save() method on instances, when running with the -sql parameter on.

One last note: At present, this code is not open source, nor public domain. You’re free to use it however you wish, but please don’t redistribute. I do plan to open source this when I’m happy with it, but it’s just not quite ready for that yet.

And finally, here’s the code: Download. Please let me know what you think, if you’re using it, if you think it’s useful, or even if you think it’s rubbish.

This all started on Wednesday, we’d been looking at the queue and decided that it was far too large. We’d hit 12.5M pages waiting to be checked, and were processing about 200,000 a day, tops. The initial reaction, as this is a distributed system, was to scale it horizontally. We rented six more VPSs (to add to the existing 5) and planned to run them at capacity until the queue was empty. After setting all the boxes up, we left them over night, in hope of completing 750,000 or more pages. Boy, were we wrong. On Thursday morning, we came in to discover it was running at a peak of 5,000 pages an hour (144,000 a day). We’d slowed it down, and not only that, we’d slowed everything else down too.

This is when the optimisation work started. We looked at the queries we were running, and noticed that an obvious bottleneck appeared to be the queue itself. Each VPS ran 15 instances of our application at any given time, and each one did a query for 250 pages to check when it was starting up, updated them as in progress, and deleting each from the queue on completion. The first change we made was to change the indexing on the table, which merely served to slow us down further. The second, more interesting change, was to change the entire table from innodb to the memory/heap table engine. This meant changing both max_heap_table_size and tmp_table_size to over 2.5GB to accomodate the 1GB table at the same time as allowing temporary tables to be created. Getting this change replicated onto our two slaves was a delight.

The memory table “solution” worked well after turning all of our VPSs back on, for about 10 minutes. At which point, they were up to capacity and the queue table started locking, a lot. As did all of the other tables, inexplicably. In addition to inducing panic, this highlighted to us that another table was starting to cause problems, our anchor text table. This is responsible for storing only the words that are used to link from one site to another. This table was seeing anything up to 2 million inserts an hour, and the same number of selects. After several failed attempts to remedy this, we finally settled on selecting all of the anchor texts we’d just collected, any that came back from the database were then dropped from our insert, thus only inserting those that were not present. Those inserts were done with batches of 10 per query. This cut the number of queries down significantly. Whilst this has improved the locking situation, it has, unfortunately, led to hanging selects. These selects are using potentially gigantic ‘IN(one, two, three)’ conditions, and are taking anything from 0.01 to 120 seconds to complete. Completely unpredictably.

Next priority was fixing the table locking on our queue. We’ve done this by marking all deletes from the queue as LOW_PRIORITY. In doing that, we’re allowing MySQL to queue those deletes for when no other clients are accessing the table, preventing unnecessary locking.

At the time of this writing, 113 instances of our application are running without significant conflicts on either of the tables we’ve optimised. However, we’re still processing just 5,000 pages an hour. I’m sure I’ll be working on this problem again on Monday, and will update here with anything we do discover.

So to summarise this somewhat rambley and unhelpful post a little, don’t assume that because you’ve built your system to run across multiple servers, that it can infact run across multiple servers. Ours scaled up to 4, and then the scaling started working against us. If you found it interesting, have any MySQL questions, want to offer advice, or are doing similar work yourself… drop me a comment below. I’d love to know that people are reading this!

I’ve been wanting to change the Ineo interface, to generally improve and streamline what we’ve got. IPB 2.2 gave me a perfect excuse to do so.

The first target was the ‘overview’ page. This has been totally revamped and is now powered entirely by plugins, which can be displayed alone, or as tabs at the top of the page. The tabs take on styles used in areas of IPB 2.2.

The second target was the sidebar. Previously, this used three separate template bits, one for the menu, one for ‘blocks’ and one for ‘items’. The majority of the menu-building was done in code, which, as I found out when trying to skin it, is not good. This has been reworked to use one template bit, and the entire menu is generated in-templates. The menu has also been redesigned to use new IPB styles and also to integrate the quick-search results area.

Finally, quite possibly the most unpopular part of the whole Ineo interface, ratings! They’ve been completely redone. No more numbers, no more “x ( x total votes )”. I think you’ll be pleasantly surprised.

This wouldn’t be an interface entry without an accompanying set of screenshots, or a video. So here you are:

Video used to be linked here.

Any feedback you have would be greatly appreciated.

The biggest change I’ve made recently is that I have stopped doing temporary work for Adecco, at least for now. I’m going to try and make a go of the freelance design and development thing. On that note, I recently finished my latest project, which I am pretty proud of. The site is E29 Incorporated, which is Adam Kinder’s new venture. Between us, we’ve got some pretty exciting things going on, but I have to keep those quiet for now.

I’ve also been loving the heat these past couple of weeks. Spending a lot of time away from the computer, in my garden and in the pool has been great, it’s one of the many reasons I love working for myself. Hopefully some day, it’ll be one of the many reasons other people love working for me, too.

That’s about all i’ve got, sorry. I guess I need to think of something new to cover on this site. In the meantime, if any of you want any custom work doing… Drop me a line.