Expert interview: How to scale Django

Eventbrite's John Shuping and Simon Willison reveal all about scaling Django, the Python Web framework developers love

1 2 3 4 5 Page 5
Page 5 of 5

Shuping: On the caching note, I'd love to give a shout-out for Redis. On the Eventbrite side, we use a ton of Redis. We do a lot of back-end analytics in Hadoop to calculate things like events recommended for you based on your previous purchases. Instead of letting Django query Hadoop directly, which might be slow, we have Hadoop upload the results and keep them updated in Redis. The site ends up querying Redis, which is superfast and we love it. We're definitely making use of Redis Sentinel, which is the automated master-slave failover system. We've written our own Redis Sentinel client that also supports sharding of clusters, which we rely on heavily.

One thing I wanted to touch on from way back was the scale question -- requests per second on the systems side. What's interesting about Eventbrite is that we don't always know when there's going to be an onsale for a huge event where 500,000 people are going to show up at our site to buy tickets for the Black Eyed Peas or whatever it may be. On the ops side, we're using Puppet and everything's automated. We actually have, within our ops team, our own Django site to manage our instances. We basically have our site built out to four times the capacity at any moment, so if a big onsale comes along and a ton of people hit the site, we can absorb that traffic and handle it.

Freeman: So you hold your capacity padded four times out. Is that ever not enough? Do you ever have to scale out beyond that?

Shuping: No. It gives us the max performance from the order side for where we're built out now. We've devised a system to handle excess traffic called the "waiting room." It's part of our core Django application, running on a separate set of servers. Basically what happens is we keep track of how many people are registering for a particular event at any given time. If it exceeds a certain threshold, we let the first 5,000 people start typing their credit card number in. When customer 5,001 comes in, we send them to the waiting room. It's a static page that's AJAXing to a separate set of servers asking, "Is it my turn? Is it my turn? Is it my turn?" It effectively lets all these people get in line, literally, for the available tickets. This lets us absorb the extra load to a different set of servers.

Freeman: So you're really not dealing with elastic clusters. You have a steady setup, and for some overflow, it spills over.

Shuping: Yeah, most people do autoscaling, but we take advantage of that more in QA and stage environments. We'll say, "Let's build up this cluster and see what happens if we do this," but we don't do autoscaling in production.

This article, "Expert interview: How to scale Django," was originally published at InfoWorld.com. Keep up on the latest news in application development and read more of Andrew Oliver's Strategic Developer blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.

1 2 3 4 5 Page 5
Page 5 of 5