Services
We had a customer ask us to dig for some indicators of compromise in their AWS account. We are already using our JASP tool to help them to check security configurations in general, so we took the opportunity to formalize some of what we’re doing into a tool which we plan to open source once we clean it up. This post presents some of the types of things that are challenging to just check in JASP and how we’re thinking about the tool.
It may help to start by explaining what JASP does and doesn’t do. It does not look at actual CloudTrail events. It also doesn’t help an organization to identify users that are provioned that shouldn’t be. It looks for things like weak IAMIAM (Identity and Access Management) is a set of processes, policies, and technologies that enable organizations to manage and control access to their systems and data. IAM solutions help to ensure that only authorized individuals can access sensitive resources, and that access is granted on a "need-to-know" basis. password policies or other configuration issues. JASP is invaluable in detecting things like ports open to the world that are in new VPC’s. We think it makes a lot of sense to use it to kind of keep tabs on your AWS environment.
But in this case, we’re looking for contextual data that would suggest something unusual is happening. Its kind of a different case.
Generally we’re looking for anything that suggests unusual behavior. Suppose an internal engineer were misusing the environment or one of our admins got compromised
Note that while many people use the AWS Console (The web GUI), we heavily focus on using the API’s and recommend clients do as well because the UI is so rich with data that it can be tricky to be sure that you’re looking at all of the relevant information. Also, a script to click through x,y,z different ways doesn’t scale (or allow repeated quick usage and tweaking) the same way a script that uses an API does.
Some examples of the things we’re looking for didn’t even come from cloudtrail. They were accessible directly from other AWS API’s eg:
aws-vault exec jemurai -- aws iam list-users --output text | cut -f 7 > iam_users.txtaws iam list-groups --output textaws iam list-rolesaws iam list-policies --scope Local --output text | cut -f 4,9,10aws ec2 describe-vpcs --output textaws ec2 describe-vpn-gatewaysaws ec2 describe-subnets --output textaws s3api list-buckets --output textHere is a quick script to check user access keys:
aws iam list-users --output text | cut -f 7 > iam_users.txt
allUsers=$(cat ./iam_users.txt)
for userName in $allUsers; do
aws iam list-access-keys --user-name $userName --output text | cut -f 3,4,5
doneOur CloudTrail tool ended up looking for:
There are a lot more events we can specifically look for. This is just an initial taste.
We stumbled across the GorillaStack slack bot (see references) that can help you keep track of events as they go by. That’s a neat approach.
Certainly using CloudWatch to create alerts on certain things makes sense.
Our approach was to write a Go program to collect a bunch of the events and data for view. We’re imagining that DevOps teams will want to modify this or adapt it to their use case. We also see the tool as finding candidate data that someone has to review. In theory, this could get incorporated into a UI or tool where these events could be filtered or responded to. For us, having a simple way to slice in to the data was the starting point.
We’re trying to figure out the best way for the tool to grab these and make them accessible to the right people in an organization. Should this be run through CI/CD as a nightly thing? Should it be hourly? We’d like to get the tool to be easy to set up as a Lambda. Once we have this set up as a first class Jemurai Open Source project we’ll be able to discuss it on Gitter and point to it on GitHub. Now we just need time!
This was a foray into Go and more dynamic AWS management. I expect we’ll continue to build custom tooling around AWS in Go now.
Our AWS cloud tool:
Tools and other pertinent references:
