Cloud audits often don't mean what you think they do

Some cloud audits provide little actual assurance -- demystifying SAS 70 (SSAE 16), SOC-1, SOC-2, and SOC-3 claims

Of all of the potential shortcomings of the cloud, trust is perhaps the largest. "Seeing is believing" is a truism that certainly applies to IT. Although you could have the worst-run internal IT shop ever, there's a comfort in being able to walk down to the data center and put your hands on what makes it tick. Moving critical pieces of your application infrastructure into the cloud removes that (sometimes false) sense of security and leaves many people feeling exposed.

Of course, that's completely natural -- any time you trust anyone to do anything for you, you are exposing yourself to risk. If a provider tells you it's taking nightly backups of your cloud-hosted ERP application and it turns out not to be true, you're the one who'll suffer in the event of a failure. The same is true with a traditional IT department, but at least you can wander down and ask to see tapes and backup logs if you're concerned that your people might not be on the ball. That's not so easy to do when systems might be hosted hundreds or even thousands of miles away in a nameless data center and you're customer No. 20,000 out of 100,000.

From the cloud provider's perspective, this lack of trust is a tough problem to solve. Assuming you're doing a thorough job, how exactly do you get a potential client to trust your operation well enough to give you its business? Aside from having a good track record with existing clients that might recommend you, publishing audit reports such as SAS-70, SOC-1 (aka SSAE 16), SOC-2, and SOC-3 has been a great way to go about instilling that trust by having an independent third party vouch that you're doing what you say you're doing.

However, this has led to cloud consumers treating these audit standards as checkoff items without knowing what they actually mean. In this post, I disassemble some of the alphabet soup and describe what each report might mean and what you could expect to learn from them.

The basics about cloud audits
First, you must understand that audit report types are not certifications. Although you might see a company say it's "SSAE 16 Certified," it means nothing. An audit isn't a certification with a set of specific tests you must pass, though some audit standards are more prescriptive than others.

Instead, the organization being audited gives the auditor a list of "controls" -- processes, procedures, statements about capabilities, and so on -- and the auditor validates those controls would achieve their goals and, optionally, ensure the controls are effectively implemented. In other words, the organization decides what it wants audited and hires the auditor to verify it's doing what it's claiming -- one company's SAS-70, for example, can cover very different attributes than another's.

Say I'm a cloud provider. I'd like to advertise to my client base that I maintain a data center that has a redundant power infrastructure, among many other features. Stating so in a marketing glossy is one thing, but I want to be able to prove it to a prospective customer. Thus, I engage a CPA to perform a SOC-1 (SSAE 16) audit (more on what that means later).

I work with that auditor to build a list of controls that might include such likes: "XYZ Cloud Provider operates two mutually redundant generators that are each independently capable of providing emergency power to the facility," "XYZ Cloud Provider maintains the facilities generators under a maintenance contract that ensures that all manufacturer-recommended preventative maintenance is undertaken in a timely manner," and so on. The goal is for me to describe everything I'm doing to ensure that my customers' data and operations are well protected, so I can have the auditor vouch for what I'm claiming.

Type I vs. Type II audits
What exactly is being vouched for depends on the subtype of audit I've decided to undergo. For the two most common report types (SOC-1 and SOC-2), there are two subtypes: Type I and Type II. The Type I report simply states that the auditor has reviewed the controls and agrees they're designed to achieve the goals the provider has defined -- but doesn't say the provider is actually doing any of the things it claims to be doing.

The Type II report validates both that the controls are properly designed and effectively implemented. A Type II audit is more difficult for me (if I were a cloud provider) to undergo. Not only must I describe the controls for the goals I'm claiming (as in a Type I audit), I need to prove they work as claimed. For example, the auditor might want to see the two generators I say I have and verify they have enough capacity to run my data center.

SOC-1, SOC-2, and SOC-3 audits
The type of audit I choose will determine how the report is assembled and the final product I can offer to my clients. Although lots of audits are out there, the most common are the SOC-1 (Types I and II), SOC-2 (Types I and II), and SOC-3, where "SOC" stands for "Service Organization Controls." All these audit standards are defined by the American Institute of CPAs (AICPA) and are designed specifically for service organizations such as cloud providers, data centers, and managed services providers.

The SOC-1 (also called SSAE 16) Type I and II audit has its roots in the old SAS 70 audit standard. In fact, it's an adaptation of the SAS 70 standard to bring it in line with the internationally recognized ISAE 3402 standard. Like both ISAE 3402 and SAS 70, the SOC-1 (SSAE 16) standard provides guidance to the CPA on how best to audit an organization's financial controls.

However, the SAS 70 and the follow-on SOC-1 (SSAE 16) standards don't directly speak to IT-related concepts such as security or privacy at all -- it's entirely up to the auditor and audited body to work out what they believe is sufficient, and it's totally up to a potential customer reading the report to judge whether they did a good job. In other words, a SAS 70 or SOC-1 (SSAE 16) is highly variable and subjective when it comes to IT concerns.

Thus, a SAS 70 or SOC-1 (SSAE 16) requires the customer to do a lot of digging to figure out whether the right controls are in place and are being validated. If it's a Type I audit, there's no validation that the claimed controls are actually implemented and work.

By contrast, the SOC-2 and SOC-3 reports include specific guidance (and thus standardization) to the auditor for how to evaluate an organization's security, availability, processing integrity, confidentiality, and privacy as they relate to IT. This provides a better baseline for assessing a cloud provider, as a provider and its auditor can remove fewer attributes from the audit, and you're theoretically guaranteed a more consistent set of basic controls.

The big difference between an SOC-2 and SOC-3 report is that the SOC-3 report tries to hide from the customer all the gory details that might be included in an SOC-1 or SOC-2 report -- details many customers have no understanding of -- and instead issues an overall conclusion as to the effectiveness of the controls in place. It's layman-friendly and easier for the provider to distribute widely. The operational details in SOC-1 and SOC-2 reports typically mean they're distributed to prospective customers only under a nondisclosure agreement. For example, you can see an example of Amazon's AWS SOC-3 report, but you'll have to get in touch with its sales folks to see AWS's SOC-1 or SOC-2 reports.

Putting it all together
At the end of the day, it's a good thing that cloud providers are investing in audits -- that means they're (likely) putting the processes and procedures in place to demonstrate they're worthy of your trust. The audit standards require these organizations to spell out what they do and, for Type II audits, prove to some degree that they follow their procedures -- which is nothing but a benefit.

Just don't think seeing a SAS 70 or SOC label on a cloud provider's website constitutes due diligence on your part. These audits are not certifications, and they require some degree of interpretation as to whether the provider's measures are sufficient for your situation.

If you're considering a cloud provider that has undergone one of these audits, ask to see it and compare the measures the provider is taking to protect your data and operations to the measures you already do in-house. If you're considering dealing with a provider that hasn't been audited, you should ask yourself whether you're comfortable trusting an organization that hasn't been willing to let an auditor review its claims at even a base level.

This article, "Cloud audits often don't mean what you think they do," originally appeared at Read more of Matt Prigge's Information Overload blog and follow the latest developments in storage at For the latest business technology news, follow on Twitter.

Copyright © 2013 IDG Communications, Inc.