Expert interview: How to scale Django

Eventbrite's John Shuping and Simon Willison reveal all about scaling Django, the Python Web framework developers love

1 2 3 4 5 Page 4
Page 4 of 5

Freeman: For people who are thinking about building this into their process, what sort of investment does it take?

Willison: It can be really simple. At Lanyrd we had a custom implementation, and the initial version took half a day to build, and it benefited us everyday from then onward. For Eventbrite, I don't know how long it took us to implement the first version, but again it was using this OS package. It can be a case of dropping in Gargoyle and starting to use it, especially if you have a standard Django setup.

Freeman: That's great. That touches on both how to scale your code base and your engineering team. Are there any other tools or techniques that come to mind?

Willison: A topic you mentioned earlier is Celery. Both Lanyrd and Eventbrite use Celery. I think some kind of offline processing should be part of the default stack for any website that you build these days. The moment something takes more than a few seconds to run, it should be running asynchronously.

Shuping: I could ramble about Celery for the whole 30 minutes. A lot of my time at Eventbrite has been making Celery awesome. It really helps get the job done while keeping the actual page response times fast for the end-user. We use RabbitMQ as our message broker, and we use multiple clusters of it. On each Web server, we have a local HAProxy running that routes tasks across those multiple RabbitMQ clusters onward to Celery servers.

Freeman: Can you give me an example or two of longer-running tasks you guys see frequently?

Shuping: Your order confirmation email. When you buy a ticket, you don't actually have to wait while the Web server generates the PDF of your ticket. We farm that off to Celery, and you get the page that says, "Here's your order confirmation number." By that time, your email has already come in because Celery's done it in the background.

Freeman: That's on the Eventbrite side. On the Lanyrd side, you're using Celery also, but backed by Redis.

Willison: That's right. We use Redis as our message broker because it's much more lightweight. We didn't need to do the full, clustered RabbitMQ setup. I think if we'd had time we might have gone with the RabbitMQ setup, but Redis is shockingly powerful out of the box for this kind of thing.

Freeman: Is Varnish part of your stack?

Willison: At Eventbrite we've used it for a few small things and we're looking at increasing our usage. At Lanyrd we primarily used it to deal with unexpected traffic spikes. If you're not signed in, we serve your request via Varnish with a 60-second expiration. If you are logged in, it skipped the caching layer entirely. That means if somebody with 3 million followers tweets a link to a Lanyrd page, we don't even notice the uptick in traffic.

Freeman: I had read that Disqus uses Varnish in its architecture, and it really does a lot, but it sounds like it's not a key component to what you're doing.

Willison: It's not yet. Disqus is doing really interesting stuff with it, and we're keen on doing more with Varnish. It's an incredibly powerful tool once you get into the details of VCL and its different capabilities.

1 2 3 4 5 Page 4
Page 4 of 5