Spouting Shite

Scalability on EC2…for the masses.

About a year ago I was tasked with the job of building a spam scanning environment on EC2. Due to the unpredictability of email traffic the system had to be scalable, redundant and self curing…. all the things that would have been unaffordable in the pre-EC2 eara. The process took months (around three months in total) and we released our spam scanning service in September 2007. The process was hugely successful…we are now very happy with the results…but to achieve the task we had to cut a few corners and use some ‘glue’.

If I was tasked with the job today I would do things a little differently. The differences in design are related to some of the lessons I learned along the journey, some new services from AWS (fixed IP’s) and Scalr.

The tricks I learned would allow me to run a mysql database on EC2, the fixed IP’s would give me the confidence to run a web server and Scalr would allow me to use an open source scalability platform…instead of re-inventing the wheel. Anybody who is interested in scalability should have a look at Scalr.

I am thinking of starting the platform from scratch again…something that would now take me much less than a week due to the power of the tools available.