Healthcare as a Startup

Nimbus AWS Case Study & Auto Scaling Tips

March 16, 2010 at 10:38 pm Justin Wilcox Leave a comment

Amazon, our bff when it comes to hosting, published on a case study of our use of AWS yesterday.  In particular, they discuss why we chose AWS over their competitors and how we use Auto-Scaling.  More info in the study: http://bit.ly/97yOfN.

I did want to expand a bit on the lessons we learned during our Auto-Scaling experience.  If you’re thinking about going that route, which I highly recommend for the right kind of problems, be able to answer these questions about your architecture before diving in too deep:

How will your machines “start their work” when they come up?

You need to have a way for the machines to know what to do and when, once they come to life. We use TopShelf to run services that periodically ping some web services asking for records to process.

How will your auto-scaled machine get your latest bits?

You don’t want to have to rebundle your AMIs every time you make a change that’s to be deployed on an auto-scaled server. Instead, build a way for auto-scaled machines to auto-update themselves when they come to life.

For example, our AS machines have a simple batch script that downloads a zip file with the latest processing bits, deploys them, and restarts the appropriate services.

Can you save money by leveraging different size machines?

We have one Auto-Scaling group that’s a single small machine with a min and max count of one machine. That means no matter what, we always have one small instance running. Let’s call this the “small auto-scaling group.”

We also have another Auto-Scaling group that consists of medium instances with a min of 0 and a relatively large max. Let’s call this the “medium auto-scaling group.”

The number of machines Auto-Scaling spins up within the “medium” auto-scaling group is actually determined by the performance of the “small” AS group. That enables us to keep the cost of our idling servers low with a single small instance, but provide extra power in the form of 0-N medium instances when records are available for processing.

Note: since small & medium instances are both 32-bit servers, we can use the same AMIs for both groups. We do have custom code in our processing service however that changes the number of processes we run based on the number of processors the machine has. You should consider doing the same.

What happens when your machine is terminated?

Make sure whatever processing you’re doing on your auto-scaled machines isn’t affected by having a particular machine terminate at any point. Since Auto-Scaling is determined by machine performance metrics, once you’re below your load threshold, machines will start terminating automatically. Make sure your infrastructure is prepared to cope with processing services that terminate mid-job.

Entry filed under: Uncategorized. Tags: .