Saturday, January 25, 2014

Decoding Your AWS Bill (Part 1)

As you begin to adopt AWS you will likely be asked to report on both usage and cost. One way to do this is using the Monthly Billing report. In this post I will show you how to download your bill and analyze it using PowerShell.
AWS offers a feature called Programmatic Billing Access. When programmatic billing access is enabled, AWS periodically saves a copy of your bill to an S3 bucket. To enable programmatic billing access click here. Be sure to enable the Monthly Report.
Once programmatic billing access is enabled you can download your bill using PowerShell. The function below will download the monthly report and load it in to memory.

Function Get-MonthlyReport {
    If($BucketName -eq $Null){
        #If no BucketName was specified, assume it is the same as the account alias
        $BucketName = Get-IAMAccountAlias
    If($AccountID -eq $Null){
        #If no AccountId was specified, use the account of the current user
        $AccountID = (Get-IAMUser).ARN.Replace('arn:aws:iam::','').Substring(0,12)
    #If no month and year were specified, use last month
    If([System.String]::IsNullOrEmpty($Month)) {$Month = If((Get-Date).Month -eq 1){12}Else{(Get-Date).Month}}
    If([System.String]::IsNullOrEmpty($Year)) {$Year = If($Month -eq 12){(Get-Date).Year - 1}Else{(Get-Date).Year}}
    $Month = "{0:D2}" -f [int]$Month #Pad single digit with 0
    #Download the report from S3 and save to the temp directory
    $Key = "$AccountId-aws-billing-csv-$Year-$Month.csv"
    $FileName = "$env:TEMP\$AccountId-aws-billing-csv-$Year-$Month.csv"
    If(Test-Path $FileName) {Remove-Item $FileName}
    $Null = Read-S3Object -BucketName $BucketName -Key $Key -File $FileName
    #Import the file from the temp directory
    Import-Csv $FileName
The monthly report is a CSV file with all of the line items in the bill you receive each month. In addition to the line items, the bill includes a few total lines. If you have consolidated billing enabled, there is an invoice total for each account and a statement total that includes the overall total. To get the total of your bill, you simply find the StatementTotal line. For example:

$Report = Get-MonthlyReport
$Report | Where-Object {$_.RecordType -eq 'StatementTotal'}
Alternatively you could sum up the PayerLineItems using Measure-Object.
($Report | Where-Object {$_.RecordType -eq 'PayerLineItem'} | Measure-Object TotalCost -Sum ).Sum
You can also find specific line items. For example, the following script will find the total number of on-demand instance hours.
($Report | Where-Object {$_.UsageType -like 'BoxUsage*'} | Measure-Object UsageQuantity -Sum ).Sum
And this line will find the total cost of the on-demand instances.
($Report | Where-Object {$_.UsageType -like 'BoxUsage*'} | Measure-Object TotalCost -Sum ).Sum
These will find the usage and cost of EBS storage.
($Report | Where-Object {$_.UsageType -like 'EBS:VolumeUsage*'} | Measure-Object UsageQuantity -Sum ).Sum
($Report | Where-Object {$_.UsageType -like 'EBS:VolumeUsage*'} | Measure-Object TotalCost -Sum ).Sum
These will find the usage and cost of S3.
($Report | Where-Object {$_.UsageType -like 'TimedStorage*'} | Measure-Object UsageQuantity -Sum ).Sum
($Report | Where-Object {$_.UsageType -like 'TimedStorage*'} | Measure-Object TotalCost -Sum ).Sum
And this one will show you snapshots.
($Report | Where-Object {$_.UsageType -like 'EBS:SnapshotUsage*'} | Measure-Object UsageQuantity -Sum ).Sum
($Report | Where-Object {$_.UsageType -like 'EBS:SnapshotUsage*'} | Measure-Object TotalCost -Sum ).Sum
As you can see there is a lot of interesting information in your bill that you can use to report on both usage and costs. In the next post I will use cost allocation report to calculate chargebacks.

Monday, January 20, 2014

Fun with AWS CloudTrail and SQS

CloudTrail is new service that logs all AWS API calls to an S3 bucket. While the obvious use case is creating an audit trail for security compliance, there are many other purposes. For example, we might use the CloudTrail logs to keep a Change Management Database (CMDB) up date by looking for all API calls that create, modify or delete an instance. In this exercise I’ll use CloudTrail, Simple Storage Service (S3), Simple Notifications Services (SNS), Simple Queue Service (SQS) and PowerShell to parse CloudTrail logs looking for new events.
The picture below describes the solution. CloudTrail periodically writes log files to an S3 bucket (1). When each file is written, CloudTrail also sends out an SNS notification (2). SQS is subscribing to the notification (3) and will hold it until we get around to processing it. When the PowerShell script runs, it pools the queue (4) for new CloudTrail notifications. If there are new notifications, the script downloads the log file (5) and processes it. If the script finds interesting events in the log file, it writes them to another queue (6). Now, other applications (like our CMDB) can subscribe to just the events it needs and does not have bother processing the log files.

Let’s start by Configuring CloudTrail. I just created a new S3 bucket and enabled SNS notifications creating a new topic named “CloudTrail”.

Now let’s create a new queue called “CloudTrail”. I just left the default values. This queue will hold notifications that a new CloudTrail log file has been written. You should also create queues for each of the events you care about. I created a queue for instances (to update the CMDB) and one for users (to notify the security team of new users).

Next, we need to subscribe our “CloudTrail” SQS queue to the “CloudTrail” SNS topic. Right click on the CloudTrail queue and choose “Subscribe Queue to SNS Topic.” Then choose the “CloudTrail” topic from the dropdown and click Subscribe.

The messages in the queue will look like the example below. The CloudTrail message (yellow) is wrapped in a SNS notification (green) which in turn is wrapped in an SQS message (blue). Our script will need to unwrap this structure to get to the CloudTrail message.

Let’s begin our PowerShell script by defining the queues. First we need the URL of our CloudTrail queue.
$CloudTrailQueue = ''
In addition, we need a list of CloudTrail log events we are interested in, along with which Queue to write them to. I used a hash table for this.
$InterestingEvents = @{
    'RunInstances'            = '';
    'ModifyInstanceAttribute' = '';
    'TerminateInstances'      = '';
    'CreateUser'              = '';
    'DeleteUser'              = '';
Now we can get a batch of messages from the queue and use a loop to process them one by one.
$SQSMessages = Receive-SQSMessage $CloudTrailQueue 
$SQSMessages | % { $SQSMessage = $_  …
Remember that the message we are interested in is wrapped in both an SNS and SQS message. Therefore, we have to unpack the message which is stored as JSON.
$SNSMessage = $SQSMessage.Body | ConvertFrom-Json
$CloudTrailMessage = $SNSMessage.Message | ConvertFrom-Json
Also remember that the CloudTrail message does not contain the log file. Rather the log file is stored in S3 and the message contains the name of the bucket and path to the log file. We next have to download the cloud trail log file from S3 and save it to the temp folder.
Read-S3Object -BucketName $CloudTrailMessage.s3Bucket -Key $CloudTrailMessage.s3ObjectKey[0] -File "$env:TEMP\CloudTrail.json.gz"
The log file is JSON format, but compressed using gzip. Therefore, I am using WinZip to uncompress the JSON file. If you don’t have WinZip, you can replace this line with your favorite tool.
Start-Process -Wait -FilePath 'C:\Program Files\WinZip\winzip32.exe' '-min -e -o CloudTrail.json.gz' -WorkingDirectory $env:TEMP
Now we finally have the detailed log file. Load it and loop over the records.
$CloudTrailFile = Get-Content "$env:TEMP \CloudTrail.json" -Raw |  ConvertFrom-Json
$CloudTrailFile.Records | % { $CloudTrailRecord = $_ …
I check the event type of each record against the hash table of events we are interested in.
$QueueUrl = $InterestingEvents[$CloudTrailRecord.eventName]
If($QueueUrl -ne $null){
                $Response = Send-SQSMessage -QueueUrl $QueueUrl -MessageBody ($CloudTrailRecord | ConvertTo-Json)
Finally, we remove the message from the queue so we don't process it again
Remove-SQSMessage -QueueUrl $CloudTrailQueue -ReceiptHandle $SQSMessage.ReceiptHandle –Force
Here is the full script.
Set-AWSCredentials LAB

$CloudTrailQueue = ''

$InterestingEvents = @{
    'RunInstances'            = '';
    'ModifyInstanceAttribute' = '';
    'TerminateInstances'      = '';
    'CreateUser'              = '';
    'DeleteUser'              = '';

#First, let's get a batch of up to 10 messages from the queue
$SQSMessages = Receive-SQSMessage $CloudTrailQueue -VisibilityTimeout 60 -MaxNumberOfMessages 10
Write-Host "Found" $SQSMessages.Count "messages in the queue."

$SQSMessages | % {
    Try {

        $SQSMessage = $_

        #Second, let's unpack the SQS message to get the SNS message
        $SNSMessage = $SQSMessage.Body | ConvertFrom-Json

        #Third, we unpack the SNS message to get the original CloudTrail message
        $CloudTrailMessage = $SNSMessage.Message | ConvertFrom-Json

        #Fourth, we download the cloud trail log file from S3 and save it to the temp folder
        $Null = Read-S3Object -BucketName $CloudTrailMessage.s3Bucket -Key $CloudTrailMessage.s3ObjectKey[0] -File "$env:TEMP\CloudTrail.json.gz"

        #Fifth, we uncompress the CloudTrail JSON file.  I'm using winzip here.
        Start-Process -Wait -FilePath 'C:\Program Files\WinZip\winzip32.exe' '-min -e -o CloudTrail.json.gz' -WorkingDirectory $env:TEMP

        #Read the JSON file from disk
        $CloudTrailFile = Get-Content "$env:TEMP\\CloudTrail.json" -Raw |  ConvertFrom-Json

        #Loop over all the records in the log file
        $CloudTrailFile.Records | % {

            $CloudTrailRecord = $_
            #Check each event against our hash table of interesting events 
            $QueueUrl = $InterestingEvents[$CloudTrailRecord.eventName]
            If($QueueUrl -ne $null){
                Write-Host "Found event " $CloudTrailRecord.eventName
                #If this event is interesting, write to the corresponding queue
                $Response = Send-SQSMessage -QueueUrl $QueueUrl -MessageBody ($CloudTrailRecord | ConvertTo-Json)

        #Finally, remove the message from the queue so we don't process it again
        Remove-SQSMessage -QueueUrl $CloudTrailQueue -ReceiptHandle $SQSMessage.ReceiptHandle -Force
        #Log errors to the console
        Write-Host "Oh No!" $_
        #Clean up the temp folder
        If(Test-Path "$env:TEMP\CloudTrail.json.gz") {Remove-Item "$env:TEMP\CloudTrail.json.gz"}
        If(Test-Path "$env:TEMP\CloudTrail.json") {Remove-Item "$env:TEMP\CloudTrail.json"}

Tuesday, January 14, 2014

I've Been Published

My book, "Pro PowerShell for Amazon Web Services," was published today.  It's been a long road, but a great experience.

I took down anything related to AWS a few months ago to avoid a conflict of interest.  Now I can start blogging about AWS again.