September 02, 2009

Gmail Gfails, Internet survives again

Yet another Google e-mail outage casts more doubt on the viability of cloud computing. But Cringely says there's always a trade-off

Maybe you didn't notice, but Gmail went down for about 100 minutes Tuesday, depriving millions of the Gfaithful of their beloved e-mail service.

[ Google has its hands full with a variety of issues these days, including Internet anonymity: "Skanks for nothing: Google must identify 'anonymous' blogger" | Stay up to date on Robert X. Cringely's musings and observations with InfoWorld's Notes from the Underground newsletter. ]

The gnashing of teeth and the renting of garments could be heard clear across Twitterville. Like these representative twits -- err, tweets:

Words and phrases like “apocalypse” and “digital terrorism” really did come to mind when I realized the #gfail.

gmail is down? im having an anxiety attack. i cant function without it.

No Gmail, Facebook is twitchy and Twitter is slow. My life may end just now…

Actually, as several Tweeters pointed out, Google's POP and IMAP servers were working just fine. It was the Web interface that did a face plant. But to the Twitterati who couldn't quite grasp that concept, it seemed the world had ended, at least temporarily.

Turns out the problem was fairly prosaic, though at the scale Google operates, even a hangnail can look like a fatal condition. The Google team took a few Gmail Web servers offline for routine maintenance, the servers that remained online got too slow and shut themselves down, and the resulting traffic overwhelmed the machines still left standing.

In a post to The Official Gmail Blog, "Site Reliability Czar" Ben Treynor writes:

The Gmail engineering team was alerted to the failures within seconds (we take monitoring very seriously). After establishing that the core problem was insufficient available capacity, the team brought a LOT of additional request routers online (flexible capacity is one of the advantages of Google's architecture), distributed the traffic across the request routers, and the Gmail web interface came back online.

additional resources
White Paper - How to Improve Delivery of Advanced Web Applications

White Paper

Virtual Workforce: The Key to Expanding The Business While Cutting Costs

Get the independent advice and expertise you need to support a virtual workforce.

Go inside:
The three-step approach to making a virtual workforce a reality.
The four flavors of client virtualization technologies.
The three key initiatives that solve IT challenges.
Download now »
White Paper: Successfully Secure Your Wireless LAN With Wi-Fi firewalls.

White Paper

Addressing Linux Threats Leveraging Fewer Resources

The increase in Linux popularity has increased the frequency and sophistication of malware attacks. Read this 2 page white paper now to learn how you can protect your Linux environment with real-time protection that is certified by all major Linux vendors.

Download now »
White Paper - The 2009 Handbook of Application Delivery

White Paper

The 2009 Handbook of Application Delivery

Ensuring acceptable application delivery will become even more difficult over the next few years. As a result, IT organizations need to ensure that the approach that they take to resolving the current application delivery challenges can scale to support the emerging challenges. This handbook elaborates on the key tasks associated with planning, optimization, management and control and provides decision criteria to help IT organizations choose appropriate solutions.

Download now »
White Paper - Is Your Backup System Outdated?

White Paper

Mid-range Storage Considerations

A common misconception is that mid-range storage requirements are dramatically different than that of a larger enterprise. Mid-range storage users may require less capacity, but they have similar functionality and management requirements. This ESG paper examines mid-range storage needs and reviews a new solution that adjusts size while retaining value, performance and functionality.

Download now »
tupahotu 2-Sep-09 3:25pm
1 reply
"The gnashing of teeth and the renting of garments," eh? What is the rate for renting a garment these days?
hoohah 8-Sep-09 4:08pm
Some of the Gmail-addicted probably did have to rent garments, having not ventured outdoors for so long, but yes-- rend, Cringe?
CodeZombie 3-Sep-09 6:18am
1 reply
Some people really, REALLY need to get a life if they're that dependent on the internet for survival.
With regards to the question about the gmail czar it does make me wonder: do Obama's czars where furry hats and drink a bunch of vodka as well? It would definitely explain alot.
zornwil 3-Sep-09 11:19pm
1 reply
A lot of people are conducting business, it's not a matter of "getting a life." Many businesses *are* dependent on email for commercial health. To be fair, a lot of the posts on Twitter and such were jokes, too. And I would say email is more critical to more people these days than phones, so I can see a degree of anxiety. But to your larger point, yes, people get themselves too wrapped up in communication and instant gratification.
CodeZombie 4-Sep-09 8:30am
Oh, yes, most definitely, businesses are very dependent on E-mail nowadays. I spend a fair amount of my day in Outlook reading and replying to E-mail messages. But if my E-mail goes down I sure don't feel like it's the end of the world. Sometimes I feel a sense of relief when the electronic things in my life go away for a while. Kind of a micro vacation. Yes, folks, believe it or not, once upon a time we got along quite well without E-mail and the other electronic "necessities" of life.
philosopher 3-Sep-09 9:11am
1 reply

Gmail goes down for 100 minutes every once in a proverbial blue moon.

Internet service, whether DSL from the phone company, broadband from the cable company, or satellite from DirecTV, has a failure rate that is vastly worse by many, many, many orders of magnitude.

And let's not forget the outages due to rebooting or reinstalling Windows to cure its latest ill-du-jour.

Or is your diatribe based on feedback from people who only use Blackberry, iPhone, Linux, or Mac to access Gmail and have 100.000% perfect internet connections?

Just curious.

zornwil 3-Sep-09 11:15pm
We have not experienced this failure rate, in terms of number of incidents at least but in terms of effective personal total downtime even, on Comcast (yes, somewhat surprisingly given older history) in the last year. But that said, the total hours over the past year aren't so bad, but unacceptable for any email-dependent enterprise (even if it is happening in real-life email-dependent enterprises, that doesn't make it unacceptable, and many enterprises *do* perform better than we've seen Google in the last year in terms of email - of course there's plenty of good reasons, this is no indictment of Google nor a praise of enterprise internal email, as each are operating at different scales and needs). I don't see what you mean by Cringely posting a "diatribe". He said quite clearly as well that there's tradeoffs, and at this point in time those tradeoffs are fairly stated, I believe. He didn't accuse Google of anything untoward, he just pointed out the issues, if slightly critically. He didn't claim people should abandon Google - in fact his closing statements say quite the opposite. I would hardly call this a "diatribe."
focused 3-Sep-09 1:09pm
Gmail does go down every once in a while. No service is perfect. However, I've only heard of Hotmail users actually loosing emails forever, never Gmail.
helentang 4-Sep-09 3:50pm
Hardly the end of the world indeed. But this latest Gmail Fail does point to a larger issue: networks are getting more complex, increasingly distributed, and harder to troubleshoot. What has repeatedly happened to GMail can and does happen to plenty of other major service providers. However, when sites like PayPal go down, the costs are real (and really expensive). But this is less about routers being overwhelmed with traffic than it is about the need for real-time capacity planning and getting a more complete picture of your entire network architecture (especially the critical L7 application layer). More thoughts on the Gmail Fail and role of capacity planning can be found here: http://bit.ly/B7vP8

Sign up to receive Web Services Resource Alerts

Subscribe to the Robert X. Cringely: Notes from the Trenches Newsletter

The one-stop resource center for IT professionals.

©1994-2010 Infoworld, Inc.