You can use “df -h” to check how much free disk space you have remaining (man page).

No disk space!

So I cant just mount another disk because all my work is in /dev/sda1 (root).

By work I mean a PostgreSQL database, my Python app and ProFTPD. Normally I could just admit defeat and start a new EC2 instance but then I wouldn’t have a blog post. There could be times when you wouldn’t want to do that anyway.

So.. I’m not at the magical level where I can unmount and re-mount the disk the OS resides on while it is running.

The solution is to create a new EBS volume, copy the file system using the EBS ‘Snapshot’ feature and then attach it to your server instance.

How to fix

It’s an easy job and the downtime is only a few minutes.

• Stop the EC2 instance
  
• Take a ‘Snapshot’ of the EC2 instance (in the EBS section)

• Create a new larger EBS volume and load the ‘Snapshot’
  
  
• Detach the EBS volume currently being used by the EC2 instance
• Attach the larger EBS volume, containing the snapshot
• Start the EC2 instance
• Assign any Elastic Ip’s you need
• Log in via SSH
• Reclaim the new space (The OS won’t do this on its own), see below

Claim the new free space

The command to claim unused space on your hard drive is resize2fs. Read about it first because you can specify some things like which disk to use. Run time may be a few minutes and there wont be any output to stdout until its complete. Allow it to finish or append the command with an &.

As root just execute:

resize2fd /dev/sda1

If that doesn’t work then try:

resize2fs /dev/xvda1

quick note: “xvda” – this is a Xen Virtual Disk, explained

Finally, execute df to check it worked. Output should be like the top picture with a small % under use.

In future

In my haste I used the wizard to set up a Debian ‘Squeeze’ instance. The Amazon Machine Image (AMI) came with a 1GB hard drive (EBS). It’s at that point you could attach a larger EBS just for the job you are working on. That would also be cool as you could detach (unmount) it and attach (mount) on another EC2 instance without having to take servers offline.

More info

This article saved me:
http://alestic.com/2010/02/ec2-resize-running-ebs-root

This issue seams pretty common:
http://stackoverflow.com/questions/6151695/ec2-instace-on-amazon-and-i-am-greeted-with-no-space-left-on-the-disk

Amazon official documentation:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/using.html

1. Click the pencil at the top right (Personalize your news)
2. Scroll down to ‘Adjust Sources’ and move the slider(s)

data = {'name':'Alex','colour':'Green','number':42,'foo':'bar'}
data = json.dumps(data)

url = "http://somewebsite.zzz/webservice"

req = urllib2.Request(url, data, {'Content-Type': 'application/json'})

f = urllib2.urlopen(req)
response = f.read()
f.close()

Excellent guide to Python and REST webservices: http://developer.yahoo.com/python/python-rest.html

However for hosting email, I’d like something more permanent. Amazon could potentially be wide open to spammers (make a server, send spam, shutdown) so you first have to get permission.

Instead of EC2 I host my email on a managed hosting plan. I use DNS so that an  EC2 instance is my web server and my managed hosting (not Amazon) is my email server.

Sending email to another server is really simple to set up:

@ 	MX 	10 	mail.your-website.com.
mail 	A 		111.222.333.444

The MX record says “send email to mail.your-website.com”. Then the A record says “send traffic to mail.your-website.com to 111.222.333.444″.

This is a great guide explaining each of the DNS record types: https://www.hover.com/help/edit-dns-records-cname-mx-txt-and-srv

So in total I have:

@	A		444.333.222.111
www	A		444.333.222.111
@ 	MX 	10 	mail.your-website.com.
mail 	A 		111.222.333.444

444.333.222.111 is the web server.
111.222.333.444 is the email server.

Everthing works perfectly but this is a very simple set up. You could have multiple email servers and machines for different applications. Learn DNS!

First where is “the cloud”? Thats not like asking “where is the internet?”. The cloud is the network of data centers owned by companies like Google, Amazon and IBM. Like always they have emergency fail safe systems and load balancing but are all vulnerable to attack, tyranny, censorship, natural disaster and blackspots. We’ve had data centers since the 1950′s! This is just the rebranding of main frame computing.

However what is new and great is the services that are now easily and cheaply available to the masses.

Cloud computing is just webhosting++

At my current job I manage three Linux servers. The first is next to my desk and runs the BI software we created to manage the company.  The second is from before my time, its on a dedicated hosting plan, runs CentOS and it costs us £60 per month. The third is an Ubuntu box hosted on Amazon EC2. For the first year its free and there after its predicted to be under £30 per month.

Getting a Linux box (any distro/version) live on Amazon is ridiculously easy.

1. Go to aws.amazon.com and sign up
2. Go to the aws management console, select EC2 and which data center you want to use (geographic location)
3. Go to instances, create new, choose the distro + version
4. Grab the .pem file and ssh into your new server

You can have a server live with root access running your choice of distro, for nothing, in less than 15 minutes. This is called “Infrastructure as a Service”.

The £60/month server I talked about is now totally obsolete. Thats all cloud computing means to me.

 More info

Vineet Nayar talks absolute sense! – “Cloud is bullsh*t”

Botclouds: a cyberattacker’s dream

 Ubuntu docs for getting live on Amazon EC2

The ‘newest’ of two files will have the highest ‘last modified’ time.
 

Using os.path to find a file modified time


import os

 
os.path.getmtime() returns a time stamp. To turn it into a readable date you can use datetime.datetime.fromtimestamp().


import os
from datetime import datetime

Get the ‘last modified’ time of a file on an FTP server

I’ve always used ftplib for handling FTP in my scripts. Over FTP, the command for getting a files modified time is MDTM. We can send this in Python using the code below.


import ftplib

connection.quit()


The modifiedTime comes back with “213 ” at the start, this means the command was successful. The rest of the data is year + month + day + hour + minute + seconds or “%Y%m%d%H%M%S”. Using datetime we can make MDTM more readable.


from datetime import datetime

# prints like 01 January 1970 09:30:01


ftplib.FTP and os.path are both great. Python wins!

I found this on TED.com. Richard John has interviewed 500 successful people (like Bill Gates and Rupert Murdoch) and asked them “What leads to success?”.

He explains in 3.5 minutes that to be successful you need these 8 points:

1. Passion, do it for love not money.
2. Work, work really hard.
3. Good, you have to be damn good at what you do.
4. Focus, focus on one thing.
5. Push, you must be pushed physically and mentally.
6. Serve, serve others something of value.
7. Ideas, follow a good one.
8. Persist, even through failure.

More info on ‘Success Analyst’ Richard St. John:
http://www.ted.com/speakers/richard_st_john.html

Excellent talk I found on TED.com on why things are broken by Seth Godin ~ http://www.sethgodin.com

Seven reasons why things suck

1. Not my job
2. Selfish jerks
3. The world changed
4. I didn’t know
5. I’m not a fish
6. Contradictions
7. Broken on purpose

More info:

http://www.ted.com/speakers/seth_godin.html

This Is Broken website (new version): http://goodexperience.com

What is a Client Access License?

“A CAL is not a software product; rather, it is a license that gives a user the right to access the services of the server.” – Definition from Microsoft

A CAL is a piece of paper you are legally required to have purchased for each individual user which will be connected to your Windows Domain at one time. They came in brown envelopes and are size A4.

When we upgraded in early 2011 they cost us £60 each. 5 came “free” with SBS and the rest we bought from a Microsoft reseller. Currently $168 for a pack of 5 on Amazon.

There is a second kind of license, Device Client Access License. This is so that if you have more computers than people then you save by getting CAL’s otherwise if there are more people than computers its cheaper to use DCAL’s.

Paradigm shift

Our most recent upgrade was the first time we were required to purchase licenses in this way. It marks a paradigm shift; the software, even though it has a financial cost, has no value. The value (for Microsoft) is in licensing on a per user or device basis. Whats messed up is that both Small Business Server and the Licenses cost money despite one being totally worthless.

This came as a surprise. For the previous version this was not required and there were no extra costs. But thats tough luck for anyone dependent on propitiatory tech.

Even better the latest version of SBS comes with a “Vista Home Premium” style Essentials edition which only supports 25 clients but doesn’t require CAL’s. Thats pretty dumb because as soon as you’re business grows above 25 then you need an upgrade and at least 20 CAL’S to go with it.

Proprietary Madness

By purchasing SBS 2008 we got Windows Server, Exchange (office file sharing and email) and 5 CAL’s. We purchased more packs of CALS so now 30 computers can join the domain. Legally that is, technically nothing stops you.

The majority of companies are dependent on Microsoft products. So are all our schools (in the UK) and government.. but why?! The most common reason I’ve heard when debating/arguing a switch is that “its the industry standard” or you get technical support with Windows and any employee we hire will know how to use it.

Alternative Operating Systems

Windows is not the only OS and it is not the most fit for purpose. Seriously, if you are willing to buy software and licenses for each user/device every 4-5 years, just so your company can share files and use email, then good luck to you.

I recommend the Ubuntu distribution of Linux.

More info:

http://www.neowin.net/forum/topic/633184-cals-wtf/

http://www.microsoft.com/licensing/about-licensing/client-access-license.aspx

http://blog.montopolis.com/2008/03/05/microsoft-windows-server-user-cal-versus-device-cal/

http://www.techrepublic.com/blog/10things/10-things-you-should-know-about-microsoft-small-business-server-2011/2339

http://www.eweekeurope.co.uk/news/government-cost-cutting-strategy-embraces-open-source-25311

Hundreds of their lectures, entire computer science modules, have been uploaded to Youtube!

Programming Paradigms (CS 107)
27 Lectures
Languages: C, C++, assembly, Python, Scheme.
http://www.youtube.com/playlist?list=PL9D558D49CA734A02

Programming Abstractions (CS 106B)
27 Lectures
Languages: C++
http://www.youtube.com/playlist?list=PLFE6E58F856038C69

Programming Methodology (CS 106A)
28 Lectures
Languages: Java
http://www.youtube.com/playlist?list=PL84A56BC7F4A1F852

Also of interest:

Introduction to Robotics (CS 223A)
16 Lectures
http://www.youtube.com/playlist?list=PL65CC0384A1798ADF

Machine Learning (CS 229)
20 Lectures
http://www.youtube.com/playlist?list=PLA89DCFA6ADACE599

Above are just the play lists related to writing software, theres a few more subjects but also talks and debates. One good talk is The Future of Robotics and Artificial Intelligence.

Stanford University Youtube channel: http://www.youtube.com/user/StanfordUniversity