Engineering

Announcing Jemurai Security Automation

Matt Konda No Comments

Its been an exhilarating few weeks.  I had to remind myself to take a breath and blog today.

What’s new-ish is that, we have a core team working on a new platform for security automation.  It extends the work we’ve done with Glue and other ideas taken from a great sample of client consulting engagements.  Across all of those engagements it just never felt like any solution was complete or as intuitive as it should have been.  We were always bending rules to live in Jenkins or to write custom scripts to check things that maybe aren’t that unique.

I’ve tried to always remind my team:  get meta.  What’s the problem behind the problem?  That vuln isn’t the problem.  Its the SDLC that allows the vuln to be introduced and not reviewed or noticed that is the problem.

Frankly, as a team we all started to think that all the tooling we’re using for DevOps and Security Automation is all too opinionated.  None of it is at its root there to be the backbone of a flexible solution.  Usually the existing tooling is pushing you into a particular vendors security tool.

We don’t have a combination of a rich inventory, accessible through an API, combined with a flexible way to do things with source code, containers, live systems, clouds in an abstracted way that can be assembled to build a better security automation solution.  The opportunity presented itself to start working on it and as a team we couldn’t be more excited to be building the next generation of tooling.  We’ve got prototypes flying that look at this problem in a whole new way.

If you’re interested in hearing more, are struggling with security automation, or have ideas you want to share to help us, I would personally love to hear from you.  We’re also actively looking for beta testers.  Catch me at matt at the company domain dot com.

Impersonation Failures

Matt Konda No Comments

Several times in the last few weeks we have looked at applications that have significant issues with “impersonation” features.  What is impersonation?

an act of pretending to be another person for the purpose of entertainment or fraud.  (From Google)

In practice, impersonation is a feature that is commonly implemented to allow Customer Support to “see what the user sees”.  Typically, it involves the support user effectively logging in as the actual user.  As such, as a feature it can be invaluable.

It can also go quite wrong. 

We reviewed one application that allowed any user to impersonate any other user.  Only certain users should be able to perform impersonation.  This is an issue of authorization.  Rarely do we want regular users to be able to impersonate internal users.

It is almost a surprise if we see an application that has proper auditing around impersonation and actions during the impersonated session.

Things that should be auditable include:

  • When the impersonation started and who initiated it.
  • What actions were taken on behalf of the user by the impersonating user.
  • When the impersonation stopped.

Extending on auditing just a bit, it is generally advisable that logging facilities record both the acting user and the logged in user.  Imagine that Joe wants to impersonate Matt’s account to help troubleshoot an issue.  All of the actions he takes should show changes to Matt’s data but there should also be a record that Joe is the acting / impersonating user.

This is also a good reminder that we often don’t need to show sensitive data, even to users themselves.  If we avoid showing specific sensitive data without an extra event (click), we can ensure that during impersonation sessions support personnel don’t see data that they don’t need to.

Another fail mode for impersonation is to have a common account with a “bypass” password, kind of like a shared super user for impersonation.  We’ve recently seen systems that allow users to get a bypass password/session and then do whatever they want without any audit trail as to what the users were looking at.  Using a shared user for this means that all of the audit trail will converge on a service account that we can’t now tie to a particular user.  It is always a good exercise to walk through negative abuse case scenarios and ensure that the information needed is being collected to make sense of them.

Let’s think about the risk model for a moment:

  • Malicious internal user wants to collect information from application users
  • Non-malicious internal user helps lots of people and ends up with their data on their system – then they become a victim of a phishing or other type of attack
  • Malicious external user can elevate privileges by assuming another identity

Since impersonation is a way to bypass authentication and authorization, it should receive significant scrutiny.  If you have impersonation code, it may be worth reviewing code to ensure that only the users that you want to do it can, and that when they do their actions are tracked.

Dependency Management for Developers

Matt Konda No Comments

I recently got asked about best practices for dependency management by an old colleague.  In response, I wrote the following which I realized might be useful to a broader audience.


So … things like github’s new notification on this have been automated via platforms like CodeClimate or via Jenkins internally at dev shops for a long time using tools such as:

  • Retire.js (JavaScript)
  • Bundler Audit (Ruby)
  • Safety Check (Python)
  • Dependency Check (Java)

Generally, we write these checks into a build so that the build fails if there are vulnerabilities in libraries.  We have contributed to an open source project called Glue that makes it easy to run tools such as these and then push the results into JIRA or a CSV for easier integration with normal workflows.

Note that there are also commercial tools that do that ranging from Sonatype to BlackDuck to Veracode’s Software Composition Analysis.  I generally recommend starting with an open source thing to prove you can do the process around the thing and then improving the thing.

At a higher level, I pretty much always recommend that companies adopt a tiered response system such as the following:

  1. Critical – This needs to get fixed ASAP, interrupts current development and represents a new branch of whatever is in Prod.  Typical target turnaround is < 24 hours.  Examples of vulnerabilities in this category might be remote code execution in a library – especially if it is weaponized.
  2. Flow – This should get fixed in the next Sprint or a near term unit of work, within the flow of normal development.  These types of issues might need to get addressed within a week or two.  Typical examples are XSS (in non major sites, major sites treat XSS like an emergency).
  3. Hygiene – These are things that really aren’t severe issues but if we don’t step back and handle them, bad things could happen.  If we don’t update libraries when they come out with minor updates then we get behind.  The problem with being far behind (eg. 3 year old JQuery) is that if there is an issue and the best fix is to update to current, the actual work to remediate could involve API changes that require substantial development work.  So philosophically, I think of this as being the equivalent of keeping ourselves in a position where we could realistically meet our Critical SLA (say 24 hours) on updating to address any given framework bug.

An important part of this is figuring out how a new identified issue gets handled in the context of the tiering system.  In other words, if a new issue arises, we want to know who (a team?) gets to determine which tier it should be handled as.  We definitely do not want to be defining the process and figuring this all out while an active major issue looms in the background.

Of course, with all of this and particularly the hygiene part, we need to be pragmatic and have a way to negotiate with dev teams.  We can weigh the cost of major updates to a system against the cost of keeping it running.  Planned retirement of applications can be the right answer for reducing overall risk.

Ultimately, we want to translate the risk we see with this into terms stakeholders can understand so that they don’t balk at the hygiene work and they are prepared when we need to drop current work to accommodate critical updates.

Tend Your Digital Garden

Matt Konda No Comments

Something that is really hard about application security is that it isn’t something you can just point a tool at and be finished at some point in time.  It is always going to take ongoing work.  I like to use the analogy of a garden.  Both the plants in the garden and the conditions around them change no matter what we do.  Maintaining a beautiful garden is a labor of love and an ongoing investment in time.  We could think of our applications in the same way.

Unfortunately, many applications look more like this example of an overgrown garden.  The original intent of many applications tends to get bent, expanded or even lost as systems evolve.  In some cases, the original beauty and architecture are lost in the complexity and difficulty managing the result.

When we think about application security, we are always looking for ways to make it a habit – something that people naturally think about and take care of.  I’d even go so far as to say that tending our security garden needs to be a labor of love.

So what do we do?  There are many layers to these examples that we can learn from:

  • We get tools to help us: clippers, weed whackers, fertilizer, hoses, wheelbarrows, etc.  We learn how to use the tools.
  • We plan to work in the garden periodically.  If we don’t, we know it is going to take more work dedicated to clean up.
  • We plan the garden and take out the plants that aren’t working.
  • We balance our time around different areas.  One wildly overgrown plant can make the whole garden less pleasant.  We know some plants take more work than others.
  • We aren’t afraid to get dirty.  We know it is work.  We’re satisfied when we’re done.

Unfortunately, with software, outside of the development team, it is often difficult to tell whether the garden looks great and is well tended or if it is a bit of a mess…

That’s one of the key reasons Jemurai is built the way we are – around expert software developers that know good software – only very strong developers can look at systems and help make them beautiful like the Japanese garden pictured above.

 

Announcing “Inside Out” Tech Talks

Matt Konda No Comments

As a small, growing and disruptive company we place a major focus on training our employees.

We’ve tried a lot of different things:  Capture the Flag games, internal videos, weekly tech talks, etc.  It’s an ongoing challenge and a continually improving process.  In a recent team discussion, we realized that there might be an interesting value to making some of those tech talks public.  It’s a way for us to provide something valuable to our community while giving our team a platform to present and cross training on technology and software security problems we’re facing.  For example, we’re seeing Hashicorp Vault and Marathon at Client X, or we’re using OWASP Glue with Jenkins at Client Y.

Somehow we came up with the idea of Inside Out Tech Talks, where we take one of our regular tech talks and make it open to the public.

The first will be 12/13 at 1:00 PM CST.

Join us on Zoom:  https://zoom.us/meeting/register/cd9408314686923e7510d14dfea9e911.

The topic is Security Automation.

 

Thinking About Secrets

Matt Konda No Comments

Introduction

We have two types of projects that often uncover secrets being shared in ways that aren’t well thought through.

  1. During code review, it is actually rare that we do not find some sort of secret.  Maybe a database password or maybe an ssh key.  Sometimes, it is AWS credentials.  We’ve even built our own code review assist tool to check for all the ones we commonly see.
  2. During security engineering or appsec automation projects we end up wiring static analysis tools to source code and JIRA and this often uncovers plaintext secrets in git.

So, generally, plaintext secrets are everywhere.

Best Practice

I said above that people are sharing secrets in ways that aren’t well thought through.  Let me expand on that.  If all of your developers have access to secrets, that’s a problem.  Of course, most developers aren’t going to do anything nefarious but they might, especially after they leave.  Most companies have a further challenge that it is very difficult to change system passwords.  So .. developer leaves the company and chances are low any secrets are changing.  Suppose a laptop with source code on it gets stolen?

The other problem with having secrets around is that it makes it easy for an attacker to pivot and find other things they can target.  Suppose I get code execution on Server 1.  If all of the secrets Server 1 uses are stored in files on the server that the code uses, it makes that pivot to get on Server 2 via SSH or pull data from DB Server 3 trivial.

Testing

Here are two instant ways to look for secrets in your code:

docker run owasp/glue -t sfl https://github.com/Jemurai/triage.git

This runs a check for sensitive files that are often in the source code.

docker run owasp/glue -t trufflehog https://github.com/Jemurai/triage.git

This looks for entropy in the files in a project.  It can take a while to run but is a good way to find things like keys or generated passwords.

Nothing beats a person that is looking carefully and knows what to look for but grep is your friend for sure.  We try to find these and offer alternatives for storing secrets.

The Alternatives

Generally, we want to keep secrets in a place where:

  • People can’t just go read them
  • It is easy to change them
  • We know if anyone uses them (audit)

We see a lot of projects adopting Vault and tools like it to store secrets.

Even better is a scenario where credentials don’t even exist but get generated for a particular action and then automatically expired.  Ideally, we require MFA.  99-Design’s AWS-Vault does this with AWS and its sessions in an elegant way.  This pattern, in general, allows us to know that there aren’t existing passwords out there that people could use without our necessarily realizing.  It also reduces the challenge of responding to a stolen laptop for example.

References

An older reference from a ThoughtWorker:  https://danielsomerfield.github.io/turtles/

A tool:  https://www.vaultproject.io/

Another tool:  https://github.com/99designs/aws-vault

An upcoming Jemurai Tech Talk:  https://www.jemurai.com/webinar/3topopensourcetoolsforsecretsmanagement

Glue 0.9.4 and Scout2

Matt Konda No Comments

Glue

We spend a fair amount of time building and using OWASP Glue to improve security automation at clients.  The idea is generally to make it easy to run tools from CI/CD (eg.  Jenkins) and collect results in JIRA.  In a way, Glue is like ThreadFix or other frameworks that collect results from different tools.  Recently, we thought it would be cool to extend some of what we were doing to AWS.  We have our own scripts we use to examine AWS via APIs but we realized that Scout2 was probably ahead of us and it would be a good place to start.

Scout2

The fine folks at NCCGroup wrote and open sourced a tool for inspecting AWS security called Scout2.  You can use it directly, and we recommend it, based on the description here:  https://github.com/nccgroup/Scout2.  It produces an HTML report like this:

For most programmers, running Scout2 is easy.  It just requires a little bit of python setup and an appropriate AWS profile.  So it wasn’t so much the barrier to entry that made us want to integrate it into Glue so much as the idea that we could take the results and integrate them into the workflow (JIRA) that we are using for other findings from other tools.  We thought that having an easy way to pull the results together and publish them based on Jenkins would be pretty useful.

What’s Coming with Glue

Glue has been a fun project that we’ve used opportunistically.  The next set of goals with Glue is to clean it up, improve tests and documentation and prepare for a legitimate 1.0 release.  At that point, we’ll probably also try to get Glue submitted for Lab status as an OWASP project.

Signal, Audit and Logging – Introduction

Matt Konda No Comments

At clients, we work to make sure the best information is available to:

  1. Debug an application
  2. Track what happens in an application
  3. Produce security signal for monitoring

Often, developers or security folks think of these as overlapping.  We hear:  “We’re using Log4J wrapped with SLF4J, it seems really redundant to do anything else.”

In practice, we believe the information is different and needs to be stored and reviewed in different ways by different people.  That’s why we build libraries to help integrate these things into applications easily – each as a first class piece of information.  As we examine each in further detail, we’ll call out the technology, audience involved and typical content.

Logging

Let’s start with logging because every developer knows about logging, right?  We work with some companies that log every request that they process.  That seems like a lot and should start to trigger alarm bells about what information lives in the logs – but let’s not be mad at logging.  For the most part, there are established solutions for doing it.  The logs need to get aggregated or centralized somewhere and then we can try to see what happened.

We would be remiss here not to point out that it is really important to keep sensitive data out of logs.  We’ve seen everything from card numbers to passwords to reset tokens to session ids …

But the point is, there isn’t anything wrong with a little log.debug(“XYZ”); or log.warn(“Data is stale”);.  From a maintenance and debugging perspective, this information is valuable – generally to operations.

Technology:  Typically file based, then aggregated.  Need text search.  High volume.  Retained for relatively short periods of time (weeks).

Audience:  Developers, Operations

Content:  Freeform – anything a developer might think is useful.

Audit

Some applications explicitly need to be able to produce an audit record for the objects they manage.  This might be who created it, when it changed and how – at who’s direction.  It might even be who accessed this data?  Consider the Stripe interface where they let you access your secret.  The secret is obscured and you have to take an extra action to see it.  Pretty sure they audit that so they know who saw it when.

Technically, you could write audit messages to logs.  This results in tedious work getting the detail back out of the logs and in any system where logs are not smoothly aggregated or can’t be managed at scale, this approach falls down.  Furthermore, someone looking for the messages needs to sift through lots of unrelated data.

A deeper issue is that if you want to produce a true audit record, like for a partner or a customer or an auditor, you can’t just give them all your dev logs!  We want to be able to produce a report tailored for the question they are asking and containing only data for the users they should be able to ask about.  Also, audit records need to be stored for a lot longer than log messages.

Technology:  Queryable interface, centralized long term storage, retained “for a long time”

Audience:  Compliance, Partners, Auditors

Content:  Specific object reads and changes (ideally with before / after visibility) associated to users & time.

Two deeper notes here:

  1. There is a lot more that you can do with structured auditing to be proactive, but that is Aaron Bedra magic so we’ll leave that to him to describe in some venue of his choosing.
  2. Audit information is richer and more valuable the closer it is to actions.  So we discourage general filters that indicate which endpoint got called, for example, because it can’t provide the rich context that a deeper integration can.

Signal

When I say signal, I really mean security signal.  As in, the opposite of noise.  Let’s face it, most of what is in logs is noise.  Even cutting edge technology built to collect and analyze logs produces a ton of noise.  When we get into the application and signal specific events there, we can break out of the noise and give security monitoring teams lots of rich data.

For example, we may want to know about failed logins.  A stream of failed logins looks bad.  A similar stream followed by a successful login looks worse.  (Exercise for reader)  Either way, this is information the security team probably doesn’t see right now.  Go a step deeper – what if input validation fails?  What if someone tries to do something they shouldn’t have permission to do?  Should security get notified?  Obviously the goal would be to take some defensive actions autonomously and in systems we’ve worked on, this works best when you can capture the actual events you care the most about.  Where can you do that?  In the application.

Another key thing with Signal is that it needs to go to the right place.  Often security operations teams are using their own SEIM that is different than the log collector used by developers.  That is smart.  They are optimized for different things.  But we need to help developers get the security events to the SEIM.

Technology:  Push signal to syslog + SEIM, ideally not retained for more than weeks but aggregated and processed for future context.

Audience:  Security Operations Team (The Watchers), Automated Security Response

Content:  Specific security events only.

Our Conclusion

At companies that have the capability and resources (say they have compliance and security monitoring teams) separating these application generated log stream messages has value because they are used by different people for different things in different tools.

We may circle back in the future with another post about our libraries for doing these things and some of the more extended benefits or specific examples of data to track.  Let us know if you are interested!

Jemurai Newsletter

Privacy Policy

Recent Comments