Cloud-Base Host Discovery Is Easier Than You Think!

During a recent conversation at DerbyCon it occurred to me that some security folks who are just dipping their toes into AWS are struggling a lot with the idea that cloud (EC2) instances keep popping up spontaneously. Developers and their agile / devops / continuous deployment methodologies are creating a chaotic mess of the network that has some security folks feeling like they are constantly in a game of whack-a-mole.  “I have to keep telling the developers to send me the IP address of new instances so I can scan them!” one security pro told me.  Running port scans across  the entire network every day may seem impractical.

If this sounds like your current situation, have no fear!  Amazon has a pretty slick solution that makes the use of port-scanning for discovering hosts obsolete.  It is called the AWS CLI (Command Line Interface).  When used with an API key configured with appropriate permissions, the CLI enables a significant degree of control over an AWS environment.  To get started with the CLI just visit the Amazon guide (https://aws.amazon.com/cli/).  You will find the downloads on the right-hand side, as indicated below:

Notice for Windows there is an actual installer but for Mac and Linux you can just use Python’s package installer “pip”.  If you don’t yet have pip installed, then you will need to do this first.  On my Debian/Ubuntu based system I ran:

apt-get install python-pip

On my Mac I find the easiest way to keep up with Python and pip (and several other things) is to install it through homebrew (see http://brew.sh/).

So assuming you have gotten this far and have the AWS CLI installed, you won’t be able to use it right away.  First you need an Access Key.  To get this, sign in to your AWS account, navigate to Identity and Access Management (IAM) and either:

1. Select your account; or,
2. Create a service account.

Note that for the service account you will also have to set up which subset of permissions the account will have.  Once you have decided which account is going to take advantage of the CLI, create a new access key from the Security Credentials tab:

Once you create the key you will be presented with a dialog that displays the Access Key ID and Secret Access Key as follows:

See that Download Credentials button prominently displayed in the bottom right-hand of the dialog?  Will this is the only time your Secret Access Key will ever be displayed so press that button and download your Access Key ID and Secret Access Key to a safe location.  You’re going to need them.

Once you have these in place, type from the command-line:

aws configure

This will prompt you for your keys as well as the AWS region you are communicating with, which will look something like:

AWS Access Key ID [None]: <Insert your key ID here>
AWS Secret Access Key [None]: <Insert your secret key here>
Default region name [None]: us-west-2
Default output format [None]:

Once this is complete you should be able to use the AWS CLI.  Of immediate interest is the ability to return a list of currently running hosts (e.g. the describe-instances command), which essentially skips the discovery step of a port scan completely.  Yep, that’s right.  Instead of scanning for instances, we just ask AWS for a list!  One way to return information is as follows:

aws ec2 describe-instances

You will see immediately that although the AWS CLI is very flexible, it tends to return more information than we really need.  To simplify things further I prefer to write Python scripts using the boto3 module.  This module provides relatively easy access to the AWS CLI functionality but within the Python interpreter.  To demonstrate just how easy things get I threw together a simple Python script called aws-list-hosts.py (on GitHub: https://github.com/JGillam/aws/tree/master/scripts).  Once you have AWS CLI running locally and boto3 installed (i.e. pip install boto3), you can run aws-list-hosts to return a list of ip addresses… convenient for feeding directly into NMap or your favorite vulnerability scanning tool.

By default it will just list internal IP addresses of running hosts in the default region, but there are options to filter on other run-states and to view public IP addresses.  You can see all the available options but by simply running

 python aws-list-hosts.py --help

usage: aws-list-hosts.py [-h] [--region REGION] [--profile PROFILE]
[--state {pending,running,shutting-down,terminated,stopped,stopping,all}]
[--group GROUP] [--public]

This really is just a tiny taste of  what can be done through the AWS CLI and Boto3.  If you are getting started with AWS I hope this will help you realize that managing assets in the AWS cloud can actually be less stressful and more automated than trying to manage them in a traditional network where host discovery through port scanning is the only way to find them.  View my github repo for the latest version of the script, and please let me know if you have ideas for other ways of filtering hosts or automating security scanning tasks through AWS CLI and Boto3.

Jason Gillam is a Principal Security Consultant with Secure Ideas. If you are in need of a penetration test or other security consulting services you can contact him at jgillam@secureideas.com, on Twitter @JGillam, or visit the Secure Ideas – ProfessionallyEvil site for services provided.