Dan Cryer - Leeds Web Developer » servers

VPS.NET Auto Recovery Monitor

Dan Cryer — Tue, 29 Jun 2010 18:26:47 +0000

I’ve been using VPS.NET for months now, both for myself and for Media Wow. Whilst their platform is incredibly stable, and their automated failover is great, there are still occasionally times when a server will go down and not come back up by itself.

The nicely integrated Server Density monitoring does a great job of informing you if a server is down, but it can’t do anything about it. So I’ve created a script that I run on an external server to check all of the servers are up (via HTTP) and reboot those that are not. It’s fairly simple, the process is as follows:

Get a list of VMs from VPS.NET.
For each VM, send a GET request to http://{hostname}.{domainname}/ and give it (by default) 5 seconds to respond.
If the server does not respond, it will check if it is marked as “running” on VPS.NET – by default it will then assume that there is simply a routing issue.
If it is not, it will check if the server is already in a power change state (i.e. startup, reboot).
Finally, if neither of the above are true, it will send a power on request to VPS.NET.
After all of that, it’ll (optionally) send you an email to let you know.

The code is below. You can also download it here. Let me know if you find it useful!

Notes:

This script needs setting up as a cron job, on an off-site server / computer. I recommend running it every 5 minutes or so.
I don’t use the VPS.NET API PHP classes because they seem to be out of date.
This script *will* spam you to oblivion if there is a cloud issue and you set it up to email in all cases.

< ?php

//----
// Set up the monitoring script:
//----

define('MONITOR_MAIL_TO', '');     // Email address to send alerts to.
define('MONITOR_API_EMAIL', ''); // Email address associated with VPS.NET API account.
define('MONITOR_API_KEY', '');     // VPS.NET API key.
define('MONITOR_SEND_EMAILS', true);               // Whether or not to send emails if a server is flagged as down.
define('MONITOR_SEND_EMAILS_ON_NO_ACTION', true);  // Whether or not to send emails if no action is taken.
define('MONITOR_REBOOT_IF_RUNNING', true);         // If HTTP check fails, but VPS.NET says the server is up, reboot?
define('MONITOR_HTTP_TIMEOUT', 5);                 // Number of seconds to allow for a server response. 

//----
// Initialise API Class and get VMs:
//----
$api = new SimpleVpsApi();
$api->setApiEmail(MONITOR_API_EMAIL);
$api->setApiKey(MONITOR_API_KEY);
$vms = $api->getVMs();

//----
// Loop through all the VMs and check their status:
//----
$out     = '';
foreach($vms as $vm)
{
	$server = $vm['virtual_machine']['hostname'] . '.' . $vm['virtual_machine']['domain_name'];
	$id     = $vm['virtual_machine']['id'];

	$get = false;
	$get = @file_get_contents('http://'.$server.'/', false, stream_context_create(array('http' => array('timeout' => MONITOR_HTTP_TIMEOUT))));

	if(!$get)
	{
		$_out .= 'Server ' . $server . ' appears to be DOWN!'.PHP_EOL;

		$vm = $api->getVM($id);

		if($vm['power_action_pending'])
		{
			$_out .= '- VPS.NET is reporting that the server is rebooting already. Taking no action.'.PHP_EOL;

			if(!MONITOR_SEND_EMAILS_ON_NO_ACTION)
			{
				$_out = '';
			}
		}
		elseif($vm['running'] && !MONITOR_REBOOT_IF_RUNNING)
		{
			$_out .= '- VPS.NET is reporting the server as running. Taking no action.'.PHP_EOL;

			if(!MONITOR_SEND_EMAILS_ON_NO_ACTION)
			{
				$_out = '';
			}
		}
		elseif($vm['running'] && MONITOR_REBOOT_IF_RUNNING)
		{
			$api->rebootVM($id);
			$_out .= '- VPS.NET is reporting the server as running. Rebooting.'.PHP_EOL;
		}
		else
		{
			$api->bootVM($id);
			$_out .= '- Submitted an automated power on request to VPS.NET.'.PHP_EOL;
		}

		// If we've logged anything for this VM, append it to the log:
		if($_out)
		{
			if($out != '')
			{
				$out .= PHP_EOL;
			}

			$out .= $_out;
			$_out = '';
		}
	}
}

//----
// Emailing functionality, for notification of server downtime:
//----

$email = '
	

		Server Monitoring Notice

			
		


	
';

if($out != '' && MONITOR_SEND_EMAILS)
{
	$email = str_replace('', nl2br($out), $email);
	mail(MONITOR_MAIL_TO, 'Server Monitoring Notice', $email, 'Content-Type: text/html'."\r\n");
}

//----
// Custom class for interacting with the VPS.NET API:
//----

class SimpleVpsApi
{
	const TYPE_GET  = 1;
	const TYPE_POST = 2;

	protected $apiUrl   = 'api.vps.net/';
	protected $apiEmail = null;
	protected $apiKey   = null;

	/**
	* Set the email address to use in requests:
	*/
	public function setApiEmail($email)
	{
		$this->apiEmail = $email;
	}

	/**
	* Set the API key to use in requests:
	*/
	public function setApiKey($key)
	{
		$this->apiKey = $key;
	}

	/**
	* Get a list of all VMs:
	*/
	public function getVMs()
	{
		$vm = $this->sendApiRequest('virtual_machines', 'GET');

		if($vm)
		{
			return $vm;
		}
		else
		{
			return array();
		}
	}

	/**
	* Sends a request to VPS.NET to load details about a VM:
	*/
	public function getVM($id)
	{
		$vm = $this->sendApiRequest('virtual_machines/'.$id, 'GET');

		if($vm)
		{
			return $vm['virtual_machine'];
		}
		else
		{
			return null;
		}
	}

	/**
	* Sends a request to VPS.NET to reboot a VM:
	*/
	public function rebootVM($id)
	{
		return $this->sendApiRequest('virtual_machines/'.$id.'/reboot', 'POST');
	}

	/**
	* Sends a request to VPS.NET to boot a VM:
	*/
	public function bootVM($id)
	{
		return $this->sendApiRequest('virtual_machines/'.$id.'/power_on', 'POST');
	}

	/**
	* Send an API request to VPS.NET
	*/
	protected function sendApiRequest($uri, $type = 'GET')
	{
		if(!$this->apiEmail || !$this->apiKey)
		{
			throw new Exception('You must set an API key and email address first.');
		}

		$opts = array(
		  'http'=>array(
		    'method'  => $type,
		    'header'  => 'Accept: application/json' . "\r\n" .
		                 'Content-Type: application/json' . "\r\n" .
						 'Authorization: Basic ' . base64_encode($this->apiEmail.':'.$this->apiKey) . "\r\n",

			'timeout' => 3,
		  )
		);

		$url      = 'https://'.$this->apiUrl.$uri.'.api10json';
		$context  = stream_context_create($opts);
		$response = @file_get_contents($url, false, $context);
		return json_decode($response, true);
	}
}

edc17388fb0743a4910e77ba4c1b8171

MySQL Load-Balanced Cluster Guide – Part 3

Dan Cryer — Thu, 28 Jan 2010 17:14:00 +0000

This article is the last in my MySQL Load-Balanced Cluster Guide series, following on from my earlier post today on setting up MySQL Monitoring on the cluster nodes, this article is a guide to setting up the load balancer with HAProxy, using the monitoring scripts we’ve created to check the node status.

HAProxy is a well known, powerful load balancer. Best known for use with HTTP servers, but it can work with any kind of TCP traffic, including MySQL. Upcoming versions will support native MySQL monitoring, so we’ll be able to retire the custom scripts we created in the previous step, when they’re available in apt-get.

1. Install HAProxy using the following apt-get command: apt-get install haproxy

2. Delete the file /etc/haproxy/haproxy.cfg and create a new one with the following contents:

global log 127.0.0.1 local0 maxconn 4096 user haproxy group haproxy daemon

defaults log global mode tcp option tcplog option dontlognull retries 3 option redispatch maxconn 2000 contimeout 4000 clitimeout 50000 srvtimeout 30000 stats enable stats scope .

frontend mysql_cluster bind {load-balancer-ip}:3306 default_backend mysql_cluster

backend mysql_cluster mode tcp balance roundrobin stats enable option tcpka option httpchk server {node-name} {ip-address}:3306 weight 1 check port 9200 inter 5s rise 2 fall 2 server {node-name} {ip-address}:3306 weight 1 check port 9200 inter 5s rise 2 fall 2

listen stats {load-balancer-ip}:80 mode http option httpclose balance roundrobin stats uri / stats realm Haproxy\ Statistics stats auth {username}:{password}

Once you’ve modified this file with settings appropriate to your environment, you’ll have a ready to go load balancer for MySQL. It’ll open port 3306 on the load balancer and forward those connections using round-robin load balancing to your two cluster servers. If either server goes down, it’ll be removed from the pool and no connections will be sent to it until it comes back up.

Additionally, it’ll create a statistics interface similar to the following, which you can access by going to your load balancing server’s address in your browser.

Each cluster node should be listed under the first “mysql_cluster” section. They should all appear in green, as they should all be live. If a server is down, it’ll appear red, as explained in the key at the top of the page. From this interface you can also see how long the load balancer has been up, how long each server has been up, how much downtime each have had, as well as the individual number of downtime incidents. As you can see in the image above, our ‘cluster-1′ server has had 21 separate downtime incidents, totalling just over 9 minutes of downtime.

So what do you need to change in your haproxy.cfg file? Here’s a quick rundown:

Firstly, you’ll need to change the line starting ‘bind’ to use your load balancer’s public IP address.
Secondly, make the same change on the line starting ‘listen stats’. You may also choose to put this service on a different port than port 80.
Next, change the {username} and {password} section of the ‘stats auth’ line to ones of your choosing. This provides password protection to your web interface.
Finally, for each of your MySQL cluster nodes, you’ll need an appropriate ‘server’ line in the ‘mysql_cluster’ backend group. The things you need to change are {node-name}, to the name of the cluster node you defined earlier (e.g. cluster-1, cluster-2) and {ip-address} to the cluster node’s internal IP address.

You may also wish to change the interval at which the load balancer will poll the MySQL servers, by changing ‘inter 5′ to a different number. For more information on HAProxy configuration, their detailed – if a little unfriendly – documentation is available here.

The final step to get this running, is to restart haproxy: /etc/init.d/haproxy restart - You should now be able to access both your web interface on port 80 and your brand new MySQL cluster on port 3306!

If you’ve got any questions about these guides or setting up a similar cluster, please leave a comment below and I’ll do my best to answer it.

MySQL Load-Balanced Cluster Guide – Part 2

Dan Cryer — Thu, 28 Jan 2010 16:28:35 +0000

Following on from yesterday’s guide to setting up MySQL master-master replication, the first of today’s guides is how to set up a script to monitor the status of your MySQL cluster nodes, which we’ll use in the next guide to set up our proxy.

On every one of your MySQL cluster nodes, follow the steps below to get it ready for status reporting:

1. Run the following apt-get command: apt-get install xinetd php5-cli php5-mysql – We need xinetd to host our status server, and php5 for the status checking script.

2. Put the following code into a new file: /etc/xinetd.d/mysqlchk – This creates an xinetd “server” on port 9200, using the /opt/mysql-status.php file to generate the response. Our proxy will connect to this server to check the status of the node.

# # /etc/xinetd.d/mysqlchk # service mysqlchk { flags = REUSE socket_type = stream protocol = tcp port = 9200 wait = no user = mysql server = /opt/mysql-status.php log_on_failure += USERID disable = no }

3. Add the following line at the bottom of /etc/services – This registers your newly created xinetd service with the system to allow it to function.
mysqlchk 9200/tcp # MySQL Monitoring

4. Create the PHP file /opt/mysql-status.php with the code below.
This script tries to connect to your MySQL server, and if it succeeds, checks the ‘Seconds_Behind_Master’ value of the slave status to make sure it is not too far behind it’s master. If either of these checks fail, it will return a 503 error to the client, telling it that the server is down.
#!/usr/bin/php < ?php /** * MySQL Replication Monitor */

// Connection details: $_host = 'localhost'; $_username = '{username}'; $_password = '{password}'; $_timeout = 60; // Number of seconds behind master this node can fall before being marked as "down".


try

{

$pdo = new PDO('mysql:host='.$_host, $_username, $_password);

$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$stmt   = $pdo->query('SHOW SLAVE STATUS');

$status = $stmt->fetch(PDO::FETCH_ASSOC);
if($status == false)

{

serverIsDown('Querying slave status failed.');

}
if(intval($status['Seconds_Behind_Master']) > $_timeout)

{

serverIsDown('Slave is behind.');

}

}

catch(Exception $ex)

{

if(!is_null($pdo))

unset($pdo);
serverIsDown('Failed to connect to MySQL.');

}
unset($pdo);
serverIsUp();
function serverIsDown($message)

{

$output = 'Server is currently down. Message:' . $message;

print "HTTP/1.1 503 Service Unavailable\r\n";

print "Date: " . date('r') . "\r\n";

print "Server: MySQL Monitor/1.0\r\n";

print "Connection: close\r\n";

print "Content-Type: text/plain\r\n";

print "Content-Length: " . strlen($output) . "\r\n";

print "\r\n";

print $output;

die;

}

function serverIsUp() { $output = 'Server is up. All is okay.'; print "HTTP/1.1 200 OK\r\n"; print "Date: " . date('r') . "\r\n"; print "Server: MySQL Monitor/1.0\r\n"; print "Connection: close\r\n"; print "Content-Type: text/plain\r\n"; print "Content-Length: " . strlen($output) . "\r\n"; print "\r\n"; print $output; die; }

5. chown -fv mysql:mysql /opt/mysql-status.php – Give ownership of the status checking script to mysql.
6. chmod -fv +x /opt/mysql-status.php – Give execute permissions on the script.
7. /etc/init.d/xinetd restart – Restart xinetd, to make all of the changes we’ve just made live.

Having completed the steps above, you will now have a script that checks the current status of your MySQL server, responding to connections on port 9200 of your server. In the next article, I’ll describe how to set up your load balancing server with HAProxy, and utilise the server we’ve just created to check which nodes are live at any given time.

MySQL Load-Balanced Cluster Guide – Part 1

Dan Cryer — Wed, 27 Jan 2010 17:35:21 +0000

I’m currently working on setting up a load-balanced many-master MySQL cluster at work, without using MySQL Cluster as it’s incompatible with our needs. Finding good guides on how to do this has proven incredibly difficult, so I thought I would document the process here.

This first post is going to cover setting up the servers themselves and configuring MySQL replication.

1. Set up the following three servers:

Cluster Load Balancer – One Public IP, One Internal IP – Ubuntu Jaunty Basic installation
Cluster MySQL 1 – One Public IP, One Internal IP – Ubuntu Jaunty Basic + MySQL Server 5.x
Cluster MySQL 2 – One Public IP, One Internal IP – Ubuntu Jaunty Basic + MySQL Server 5.x

We’re using VPS.NET. If you’re going to do the same, use the ‘Ubuntu 8.04 LTS 64-bit Basic Installation’ template for the Load Balancer and ‘Ubuntu 8.04 x64 MySQL’ for the MySQL nodes. You’ll then need to do the following to upgrade to Jaunty:

Edit the file ‘/etc/update-manager/release-upgrades‘ and change the line ‘Prompt=lts‘ to ‘Prompt=normal‘.
Run the command ‘do-release-upgrade‘. Let it go through the entire process of installing Intrepid, then reboot.
Repeat the step above again to install of Jaunty.

2. On the two MySQL servers, edit /etc/mysql/my.cnf and comment out the line ‘bind-address 127.0.0.1‘ – the servers need to allow remote TCP/IP connections.

3. On each server, add the following to the /etc/mysql/my.cnf file:

server-id = 1 # Increment this number by one for each server, so 1 for cluster-1, 2 for cluster-2 auto_increment_increment = 2 # Set to the number of nodes you have (or are likely to have) auto_increment_offset = 1 # Set to the same as the server-id replicate-same-server-id = 0 # To ensure the slave thread doesn't try to write updates that this node has produced. log-bin # Turn on binary logging (neccessary for replication) log-slave-updates # Neccessary for chain or circular replication relay-log # As above relay-log-index # As above

Note: On our servers, we’ve used an auto_increment_increment of 4, to allow us to add two more nodes to our cluster at a later date, without hitting auto-increment problems.

4. Restart MySQL on both servers.

5. On Cluster 1, log into the MySQL Console and run the following queries:

STOP SLAVE; CHANGE master TO master_host="cluster-2.yourdomain.com", master_user="{username}", master_password="{password}", master_log_file="mysqld-bin.000001", master_log_pos=98; START SLAVE;

Note: To verify the master_log_pos value, execute the query: “SHOW MASTER STATUS;” and check the Position column. Also, you may wish to use your internal IP addresses for the master_host option.

6. On Cluster 2, log into the MySQL Console and run the following queries:

STOP SLAVE; CHANGE master TO master_host="cluster-1.yourdomain.com", master_user="{username}", master_password="{password}", master_log_file="mysqld-bin.000001", master_log_pos=98; START SLAVE;

Note: If on either server you get an error message at this point, verify that:

a) the master_log_file name in the query above matches the ones your servers are generating
b) your username and password for the other server are correct
c) that your servers can communicate with each other using the host names you specified.
Additionally, you may wish to remove AppArmor, as that can interfere with MySQL replication and cause all kinds of miscellaneous problems. To do so, run ‘apt-get remove apparmor‘ followed by ‘apt-get purge apparmor‘.

You should now have master to master replication set up and running across your two MySQL servers. To test, create a database on cluster-1, then create a table within that database and insert some data into it. Next, log on to cluster-2 and you should be able to see that newly created table, complete with data.

If you want to ensure that it works both ways, insert some data on cluster-2 and you should see it on cluster-1. You will also want to test that your auto-increment settings are working correctly.

If you’ve got any questions about this article, or have any problems setting up the replication, please leave a comment below and I’ll do my best to help. The next article in this guide will cover setting up a proxy on the load balancer and hopefully, implementing failover.

Next Step: Setting up MySQL Monitoring on the cluster nodes.

Free VPS.NET Server Setup

Dan Cryer — Mon, 18 Jan 2010 17:40:31 +0000

I’ve been getting a lot of traffic lately from people looking for information about VPS.NET and it’s cloud platform, and I get the impression that people might think it’s difficult to get started. It’s not. As such, I’ve decided to make an offer to people wanting to use it for their web sites.

I’ll set your web server up for you, for free, after you sign up to VPS.NET. I’ll get it up and running with Ubuntu Linux, Apache 2.2, PHP 5.2 and MySQL 5.1. I’ll also set up FTP (if desired) and create up to five user accounts and Apache virtual hosts for domains you already have.

All you have to do to take advantage of this is the following:

Sign up to VPS.NET using this link (see note 1) - but don’t create a VPS
Forward the account details link you get from VPS.NET (including your account username and password) to vps@dancryer.com
In the email, include the number of VPS.NET “nodes” you want me to allocate to your VPS, a list of up to five domains you would like to be set up, the server name you want to use (if you have a preference), and whether or not you would like to be able to access it via FTP.

I’ll set up the server as soon as I can and email you back with a list of Ubuntu user names and passwords, MySQL usernames and passwords, and various other server details.

Notes:

In order to qualify for the free set up, you must use the link I provided to sign up. VPS.NET offers a generous affiliate program that covers the cost for my time to set up the server, which allows me to offer it to you for no cost.
If demand is high, I cannot guarantee to be able to set your server up for you. I’ll email you back at the earliest possible time to let you know.
Your account details will be deleted from my email account as soon as the work is completed, you are of course welcome to change all passwords after completion of the work.
I cannot offer ongoing server support for free, such requests will be met with a request for payment.
I will not install any web applications as part of this offer, nor migrate data from existing hosting accounts, simply install the server software itself, further configuration is up to you.