VPS.NET Auto Recovery Monitor

Posted June 29 by Dan Cryer

I’ve been using VPS.NET for months now, both for myself and for Media Wow. Whilst their platform is incredibly stable, and their automated failover is great, there are still occasionally times when a server will go down and not come back up by itself.

The nicely integrated Server Density monitoring does a great job of informing you if a server is down, but it can’t do anything about it. So I’ve created a script that I run on an external server to check all of the servers are up (via HTTP) and reboot those that are not. It’s fairly simple, the process is as follows:

  • Get a list of VMs from VPS.NET.
  • For each VM, send a GET request to http://{hostname}.{domainname}/ and give it (by default) 5 seconds to respond.
  • If the server does not respond, it will check if it is marked as “running” on VPS.NET – by default it will then assume that there is simply a routing issue.
  • If it is not, it will check if the server is already in a power change state (i.e. startup, reboot).
  • Finally, if neither of the above are true, it will send a power on request to VPS.NET.
  • After all of that, it’ll (optionally) send you an email to let you know.

The code is below. You can also download it here. Let me know if you find it useful!

Notes:

  • This script needs setting up as a cron job, on an off-site server / computer. I recommend running it every 5 minutes or so.
  • I don’t use the VPS.NET API PHP classes because they seem to be out of date.
  • This script *will* spam you to oblivion if there is a cloud issue and you set it up to email in all cases.
< ?php

//----
// Set up the monitoring script:
//----

define('MONITOR_MAIL_TO', '');     // Email address to send alerts to.
define('MONITOR_API_EMAIL', ''); // Email address associated with VPS.NET API account.
define('MONITOR_API_KEY', '');     // VPS.NET API key.
define('MONITOR_SEND_EMAILS', true);               // Whether or not to send emails if a server is flagged as down.
define('MONITOR_SEND_EMAILS_ON_NO_ACTION', true);  // Whether or not to send emails if no action is taken.
define('MONITOR_REBOOT_IF_RUNNING', true);         // If HTTP check fails, but VPS.NET says the server is up, reboot?
define('MONITOR_HTTP_TIMEOUT', 5);                 // Number of seconds to allow for a server response. 

//----
// Initialise API Class and get VMs:
//----
$api = new SimpleVpsApi();
$api->setApiEmail(MONITOR_API_EMAIL);
$api->setApiKey(MONITOR_API_KEY);
$vms = $api->getVMs();

//----
// Loop through all the VMs and check their status:
//----
$out     = '';
foreach($vms as $vm)
{
	$server = $vm['virtual_machine']['hostname'] . '.' . $vm['virtual_machine']['domain_name'];
	$id     = $vm['virtual_machine']['id'];

	$get = false;
	$get = @file_get_contents('http://'.$server.'/', false, stream_context_create(array('http' => array('timeout' => MONITOR_HTTP_TIMEOUT))));

	if(!$get)
	{
		$_out .= 'Server ' . $server . ' appears to be DOWN!'.PHP_EOL;

		$vm = $api->getVM($id);

		if($vm['power_action_pending'])
		{
			$_out .= '- VPS.NET is reporting that the server is rebooting already. Taking no action.'.PHP_EOL;

			if(!MONITOR_SEND_EMAILS_ON_NO_ACTION)
			{
				$_out = '';
			}
		}
		elseif($vm['running'] && !MONITOR_REBOOT_IF_RUNNING)
		{
			$_out .= '- VPS.NET is reporting the server as running. Taking no action.'.PHP_EOL;

			if(!MONITOR_SEND_EMAILS_ON_NO_ACTION)
			{
				$_out = '';
			}
		}
		elseif($vm['running'] && MONITOR_REBOOT_IF_RUNNING)
		{
			$api->rebootVM($id);
			$_out .= '- VPS.NET is reporting the server as running. Rebooting.'.PHP_EOL;
		}
		else
		{
			$api->bootVM($id);
			$_out .= '- Submitted an automated power on request to VPS.NET.'.PHP_EOL;
		}

		// If we've logged anything for this VM, append it to the log:
		if($_out)
		{
			if($out != '')
			{
				$out .= PHP_EOL;
			}

			$out .= $_out;
			$_out = '';
		}
	}
}

//----
// Emailing functionality, for notification of server downtime:
//----

$email = '
	
Server Monitoring Notice
'; if($out != '' && MONITOR_SEND_EMAILS) { $email = str_replace('', nl2br($out), $email); mail(MONITOR_MAIL_TO, 'Server Monitoring Notice', $email, 'Content-Type: text/html'."\r\n"); } //---- // Custom class for interacting with the VPS.NET API: //---- class SimpleVpsApi { const TYPE_GET = 1; const TYPE_POST = 2; protected $apiUrl = 'api.vps.net/'; protected $apiEmail = null; protected $apiKey = null; /** * Set the email address to use in requests: */ public function setApiEmail($email) { $this->apiEmail = $email; } /** * Set the API key to use in requests: */ public function setApiKey($key) { $this->apiKey = $key; } /** * Get a list of all VMs: */ public function getVMs() { $vm = $this->sendApiRequest('virtual_machines', 'GET'); if($vm) { return $vm; } else { return array(); } } /** * Sends a request to VPS.NET to load details about a VM: */ public function getVM($id) { $vm = $this->sendApiRequest('virtual_machines/'.$id, 'GET'); if($vm) { return $vm['virtual_machine']; } else { return null; } } /** * Sends a request to VPS.NET to reboot a VM: */ public function rebootVM($id) { return $this->sendApiRequest('virtual_machines/'.$id.'/reboot', 'POST'); } /** * Sends a request to VPS.NET to boot a VM: */ public function bootVM($id) { return $this->sendApiRequest('virtual_machines/'.$id.'/power_on', 'POST'); } /** * Send an API request to VPS.NET */ protected function sendApiRequest($uri, $type = 'GET') { if(!$this->apiEmail || !$this->apiKey) { throw new Exception('You must set an API key and email address first.'); } $opts = array( 'http'=>array( 'method' => $type, 'header' => 'Accept: application/json' . "\r\n" . 'Content-Type: application/json' . "\r\n" . 'Authorization: Basic ' . base64_encode($this->apiEmail.':'.$this->apiKey) . "\r\n", 'timeout' => 3, ) ); $url = 'https://'.$this->apiUrl.$uri.'.api10json'; $context = stream_context_create($opts); $response = @file_get_contents($url, false, $context); return json_decode($response, true); } }

1 Comment »

NoFollow Checker For Safari 5

Posted June 9 by Dan Cryer

A while ago, I wrote a very simple nofollow extension for Google Chrome. When enabled, it marks all no-follow links with a red highlight. Despite it’s simplicity, it has been downloaded a few thousand times! So, with the release of Safari 5 and it’s new extensions system, I thought I’d knock together something that does the...

Read More »

Where Have I Been? Work, IPS and Car Issues, Oh My.

Posted June 8 by Dan Cryer

I went through quite a long phase of updating this blog with almost surprising regularity, so anyone that actually reads it may have noticed I’ve been somewhat absent as of late. So where have I been? Well, the short answer is I’ve just been busy! Media Wow Since I started at Media Wow, I’ve been busy there...

Read More »

GWT Top Search Queries

Posted May 4 by Dan Cryer

We had an interesting conversation about Google Webmaster Tools, and it’s “top search queries” section, at work today. We’d read a lot of articles suggesting that the updated version of this page shows the overall impressions you could expect for each of the positions for keywords you rank for, but that didn’t quite sit right...

Read More »

My New Job at Media Wow

Posted April 8 by Dan Cryer

I posted last week about my last day at Stickyeyes, and as of today, I’ve been at Media Wow for a whole week. I’m not sure quite where the week has gone, it has absolutely flown. It could be due to the fact that I’m already knee deep in a couple of projects with a...

Read More »