The Truth About Audits

August 8, 2022

This post talks about the good, bad and ugly that we see around cybersecurity audits. It is informed by 4 years working for a company deeply embedded with PCI Compliance, and then about 8-10 years of experience helping dozens of companies with SOC 2, ISO 27001, NIST 800-171A and other similar audits.

The TL;DR is that audits can be helpful for improving an organization's security posture. On the other hand, an audit doesn't prove security and the more you game the audit the less likely it is that the audit improves your security. Furthermore, unfortunately, most audits we have seen are inconsistent across auditing firms and don't mean what you might think they mean.

The Good

Audits force an organization to do some introspection. If you identify appropriate "controls" to check, then the audit activity can help you to make sure you are doing the right things.

For example, if you say you have endpoint protection in place then the auditor may ask for proof. While producing the proof, you may realize that Mac and Linux devices were not actually covered by the tool you had in mind. That might lead you to use a solution that works with those devices, reducing your actual risk materially.

In some cases, the audit introduce controls you didn't know you needed to have in place! This can be extremely helpful for organizations that haven't oriented to a standard (see: why we use NIST 800-53 or which security standard should I use? for more information) and may not have identified broad sets of controls they want to have in place.

Sometimes an audit just tells you what you need to know, but from an outside party. Well, if the respected expert XYZ firm says we need to do it, then we can justify the expense to upper management. This just happens sometimes.

So in some ways, audits can really help organizations improve their security.

The Bad

For every customer we have that has had good experiences with audits, we have at least one that has had bad experiences with audits. That's < 50% positive feedback.

One huge problem is defining the audit Scope. I've seen audits where the auditors review all of the internal corporate IT as if internet facing applications didn't exist and sign off on an audit because the correct controls were in place for internal email and file sharing.

I've seen firms scope an audit a certain way and then come in and bleed customers after an initial audit agreement based on their lack of alignment to the controls.

I've also seen the opposite, audit firms that come in and charge a very low fee and basically check nothing.

Another problem with audits is that most of them aren't pass / fail and nobody reads audit reports. This means that so long as you can produce an audit report (eg. SOC 2), you can usually win business, even if the report says you didn't cover the main application the reviewer cares about or even if you had significant findings. Only very invested companies with strong teams are able to do this justice - and even then, they are subject to the whims of the auditor.

The Ugly

We have worked with a number of audit firms. You might be surprised at how inconsistent the process is. Some firms are quite technical and understand what controls fit in a given environment. Other firms have a list of controls and they are going to stick to them no matter what. Maybe the worst thing is that we've even seen significant variation in experience with different teams from the same firm!

Did you ever stop and think about how some of the most famous hacks were conducted against organizations that had been audited? Consider Target and PCI. Consider almost any recent hacking incident and SOC 2 Type 2. Was Heroku SOC 2 audited? GitHub? Okta?

It reminds me a bit of when I worked for a security company that helped build the PCI Compliance standard. I invite you to think for a moment about how good a business it is to define the standard for compliance, get the card brands to agree to your standard, then define the certification process for assessors, then be the main company that gates compliance to that standard! Can you say boondoggle!? But even PCI was at least a little bit prescriptive.

There is also a growing industry of tools to accelerate compliance. They typically integrate with a bunch of data sources and consolidate information. Whenever I see this, and the idea of automating compliance, I have to say - I do not think that works the way you think it works.

I do not think that means what you think it means.
Security and Compliance

For example, you may have a tool that claims to confirm that you are using a password manager. Great. Does it know which passwords are in scope and that those passwords are stored ONLY in the password manager? Can it tell if you are actively using the password manager?

Let's take another example, suppose a tool tells you your AWS environment is SOC 2 compliant. What does that even mean? Does the tool understand anything about what your dataflows, user lists, and other details are? Do they think they can confirm ALL of your security settings are applied properly AUTOMATICALLY? Consider the question of who has access to what data? I've basically never seen a tool in any security domain that can effectively automate that.

So this whole idea that we can automate compliance is either jaw droppingly naive to me or reflects on the inadequacy of the audits - OR BOTH! And by the way, we build security tools that are directly adjacent to (but different from) both audit compliance (securityprogram.io) and AWS Compliance (jasp.cloud). One thing I will say, I'm all for using API's and open tools like steampipe.io that let you get the data you need for compliance. I will just insist that you have to be able to ask and answer questions and not just run a tool. Some auditors we work with won't accept CLI output as evidence, they need to see the AWS console. How do we reconcile that with a fully automated acceleration tool that is checking a bunch of things that may or may not matter?

So the idea that any audit (and particularly SOC 2) consistently demonstrates security is so obviously far from the truth that I can't believe that "the community" is using it as a standardized baseline these days.

The worst thing is, I don't believe it is ignorance so much as avarice on the part of the auditing community that we see this. Certainly many in the security community know that these audits are fundamentally flawed.

Conclusion

We work with some audit firms that we respect and like a lot. We believe they help our customers improve and hold themselves accountable for their internal security.

We also work with firms and tools that we think don't add a lot of value - let alone security - and which make a lot of work for the organization.

We recommend that you go into these relationships with eyes wide open about what you are trying to get and what your audit actually means based on the price and type of auditor that you have.

That being said, maybe it is time for a new audit standard. Maybe it is time for a better security standard. In either case, we believe collaboration and people will be a foundational part of it.

Share this article with colleagues

Popular Posts

Ready to get started?

Build a comprehensive security program using our proven model.
© 2012-2024 Jemurai. All rights reserved.
linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram