Saturday, April 8, 2017

Elastic Beanstalk Worker Environment Timeouts

I have been working with Worker Environments in AWS Elastic Beanstalk. I found all the timeouts confusing at first, so I share my findings here.

The instances in your Worker Environment have a demon that reads messages from an SQS Queue. That queue has a Default Visibility Timeout and Message Retention Period. In addition, the Elastic Beanstalk Worker Configuration has its own Visibility Timeout and Retention Period in addition to a Connection Timeout, Error Visibility Timeout and Inactivity Timeout.

The process works like this (see diagram below). The SQS demon polls the queue. When it reads a message, it sets the Visibility Timeout overriding the queue's Default Visibility Timeout. The demon then checks if the message is older than the Retention Period. If it is, it explicitly deletes the message, effectively overriding the queue's Message Retention Period. In other words, the Worker Environment's Visibility Timeout and Retention Period replace the queue's Default Visibility Timeout and Message Retention Period respectively.


Assuming the demon finds a message that has not exceeded the Retention Period, it does an HTTP POST with the message in the body to your application which should be listening on If the demon cannot create a connection to your application within the Connection Timeout it sets the message's visibility to the Error Visibility Timeout. The message will be retied after the Error Visibility Timeout.

If the demon can create a connection, it waits for a response. If the Inactivity Timeout is exceeded before the demon receives a response, it aborts the request and sets the message's Visibility to the Error Visibility Timeout. The message will be retied after the Error Visibility Timeout.

Note that your entire run does need to complete within the Inactivity Timeout (max 30 mins). Each time your application sends data the counter is reset. In other words you can hold the HTTP connection open for longer than 30 minutes by streaming data back in small increments. You could extend this up to the Visibility Timeout (max 12 hours). While SQS allows you to reset the visibility timeout, Elastic Beanstalk does provide the receipt handle to your code.

At this point we have addressed all seven of the timeouts you can configure (2 on the queue and 5 in worker configuration), but we came this far so let's see this through to completion. If the demon receives a response from your application, it checks the return code. If the response indicates success (i.e. 200) it explicitly deletes the message from SQS and the process completes. If the response indicates failure, the demon sets the message's Visibility to the Error Visibility Timeout. The message will be retied after the Error Visibility Timeout.

Thursday, November 10, 2016

Simple Email Service (SES) Sample Application

I could not find a simple example to send email using SES in Python. Turns out it is really easy. If you have a MIME formatted message you can simply call send_raw_message.

client = boto3.client("ses")
client.send_raw_email(RawMessage = {'Data': mime_formatted_message})

Of course the tricky part is the MIME formatting. Turns out that is really easy in Python. Here is a simple example.

message = MIMEText("Testing 123\nTesting 123\nTesting 123")
message['From'] = ""
message['To'] = ["", ""]
message['Date'] = formatdate(localtime=True)
message['Subject'] = "Testing"

Then you can simply call as_string() and pass it to SES.

client = boto3.client("ses")
response = client.send_raw_email(RawMessage = {'Data': message.as_string()})

I messed around for a little while and created a few helper functions to handle HTML formatting and attachments. You can find the complete code in GitHub. I hope that helps someone.

Sunday, November 6, 2016

Linked Account Template

It is common for an AWS customer to have many accounts.  Often a central IT team will own the payer account and have oversight over all accounts. The IT team will create a linked account for each project or business unit. When you create a new linked account, it's helpful to have a template Cloud Formation template to ensure the configuration of the linked accounts are all identical.

This template takes the account number of the payer account and a bucket to write CloudTrail logs to (Note: best practice is to write logs to the payer account to ensure separation of duties.) It will create:

  1. CloudTrail - Configures a trail that writes to the bucket specified. This bucket should be in the payer account to assure that users in the linked accounts cannot alter the log.
  2. CrossAccountOversight - A cross account role that users in the parent account can assume when they need access to the linked account.
  3. SystemAdministrators - Add users to this group if they need to manage resources in the linked account. This is just a template and you can alter it to include the subset of services you allow the account owners to use. Note that this group gives users read only access to everything so they do not get errors navigating around the console.
  4. SecurityAdministrators - Add users to this group if you want them to manage their own permissions. Note that if you do, they can delete your oversight role so only add users you trust.
  5. ChangeYourPassword - A managed policy that allows users to change their own password. Note that this policy is already associated with the SystemAdministrators group.
  6. DefaultInstanceRole - An instance role users can assign to an EC2 instance. I allows read only access to EC2 so instances can discover information about the environment they are running in for auto configuration at runtime.

Sunday, October 4, 2015

My Cloud EX2 Backup to Amazon S3

With all the devices in the house it was finally time to invest in a NAS. I settled on the Western Digital My Cloud EX2. I picked this specifically because it supported back up to Amazon S3. In practice, the backup software sucks and I had to work around a few issues to get it working reliably and inexpensively.  

Overall I really like the EX2. It has great features for the price. My version came with two 4TB drives which I configured to mirror for redundancy (you can forgo redundancy and get 8TB of storage).  The EX2 supports SMB and NFS. It can act a DLNA (I use an app called Vimu Player on my Fire TV) or iTunes server (unprotected audio only). For the more advanced user, you can also join Active Directory, act as an iSCSI target, and mount ISO images. The EX2 can backup to another EX2, Elephant Drive or Amazon S3. The rest of this post focuses on backup to S3 which is less than perfect, but with a little effort I have it running reliably.


At a high level, I want the back to protect me from three things: 1) Hardware failure. The EX2 has two disks, but I still want a more protection. 2) My own stupidity. I might accidentally delete or overwrite something. 3) Malware. Most notably CryptoLocker or similar ransom ware. The backup agent built into the EX2 offers three backup types (taken from here):

  • Overwriting existing file(s): Overwrites files in the target folder that have the identical name as your source file.
  • Full Backup: Creates a separate folder containing all of the backup data each time the backup is performed.
  • Incremental Backup: Overwrites files with source files that are newer then the target files.

I wanted the third option, and this is what I am running. Unfortunately, it does not work as advertised. Every once in a while it overwrites files that have not changed. This would be not a big deal, but I want to run versioning to protect against malicious malware overwriting my files. With versioning enabled, S3 stores every version of your files so you can always roll back to an old copy.

The problem is that the EX2 keeps adding versions.  Over the past six months it has created as many as 10 copies of a file that has never changed. This has driven my bill up dramatically. To keep my bill in check I resorted to a lifecycle policy that moves my files to glacier and removes old versions after 30 days.  Glacier is much cheaper and and 30 days gives me enough time to fix a mistake.


The first thing I created was an S3 bucket. There is noting special here, just accept the defaults.  Then, I created the lifecycle policy described above. The configuration looks like this:

Next, I needed an IAM user for the backup job on the EX2. I created a user policy that had only those rights needed by the backup job.  This way, even if my EX2 were compromised, the attacker could never delete from my bucket or access other resources in my account. My policy looks like this.  

    "Version": "2012-10-17",
    "Statement": [
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
            "Effect": "Allow",
            "Action": [
            "Resource": "arn:aws:s3:::BUCKETNAME"
            "Effect": "Allow",
            "Action": [
            "Resource": "arn:aws:s3:::BUCKETNAME/*"

Finally, I could configure the backup job on the EX2. The configuration above has been running for a while now. It still overwrites files that have not changed, but the lifecycle policy keeps them under control.

Monday, September 14, 2015

NYC PowerShell User Group

I'll be presenting on AWS tonight at the NYC PowerShell User Group.  The presentation is available here.

Wednesday, April 15, 2015

Configuring an AWS Customer Gateway Behind a NAT

I have been wanting to configure a VPN Connection from AWS to my house, but my cheap Netgear router does not support IPSec. So, I picked up an old Cisco 871 router that does. I didn’t want to sacrifice the speed (it supports 802.11ac while the 871 is an old 802.11g device) and features of my Netgear router, so I put the 871 behind the Netgear and modified the VPN configuration for NAT traversal.
The 871 (or a similar device) is a great way to get some hands on experience configuring a Virtual Private Gateway. Despite its age, the 871 is actually a capable device and it’s available on eBay for less than $100. While most production implementations will not require NAT traversal, this is also good experience. You may want to peer two VPCs (in the same or different regions) and one common solution is to use two Cisco CSR1000V (available in the AWS Marketplace). In this configuration both CSR100V devices will require an Elastic IP, which uses NAT.

My configuration looks like the diagram below. I am using a Netgear router, but any router will work. My cable provider has assigned the address to my Netgear router which also has a private address of I have assigned the Cisco 871 a static IP address of (make sure to exclude this from the DHCP pool on the router). My AWS VPC has the CIDR block I have configured a static route on the Netgear that forwards all traffic destined for to In addition, I added a port forwarding rule to the Netgear that forwards UDP port 500 to

In the AWS VPC console, I created a VPN Connection as shown below. Note that I have entered the public IP address of the Netgear router ( as the IP address of a new Customer Gateway. I also configured static routing and entered the CIDR block of my home network (

Once the VPN connection is created you can download the router configuration. I choose a Cisco Systems ISR Series Router. In order to support NAT traversal you will need to modify the configuration slightly. You need to find the six places where the public IP address appears and replace it with the private IP address of the IPSec router. Not that there will two of each of the highlighted sections below, one for Tunnel1 and one for Tunnel2.
crypto keyring keyring-vpn-d67b98bf-0  
  local-address !Replace with
pre-shared-key address key XXXXXXXXXXXXXXXXX
crypto isakmp profile isakmp-vpn-d67b98bf-0
  local-address !Replace with
  match identity address
  keyring keyring-vpn-d67b89bf-0

crypto ipsec fragmentation before-encryption
interface Tunnel1
  ip address
  ip virtual-reassembly !Protects against Fragment attacks
  tunnel source !Replace with
  tunnel destination 
  tunnel mode ipsec ipv4
  tunnel protection ipsec profile ipsec-vpn-d67b98bf-0
  ! This option causes the router to reduce the Maximum Segment Size of
  ! TCP packets to prevent packet fragmentation.
  ip tcp adjust-mss 1387 
  no shutdown
In addition, I needed to change the ip sla timeout because the 871 is too old to support the default value, but this is unique to the 871.
ip sla 100
   icmp-echo source-interface Tunnel1
   timeout 1000 !AWS uses 1000.  Min on 871 is 5000.
   frequency 5
With static routing AWS is not advertising routes over the VPN tunnel. Therefore, I had to add the route statements to the 871 manually. The first statement uses the default administrative distance of 1 and therefore tell the router to prefer Tunnel1. If SLA 100 fails, this route will be removed and Tunnel2 with administrative distance of 10 will take over.
ip route Tunnel1 track 100
ip route Tunnel2 10 track 200

Extra Credit: Securing the Home Network

In order to protect my home network from nefarious traffic from AWS, I added a “firewall” policy using inspect statements on the 871. The ACL defines what is allowed from AWS. In this case, just ping for testing. All traffic to AWS is allowed and the inspect rules open the return path for any traffic initiated from my house. SSH and FTP defines high level inspect rules specific to these protocols.
ip inspect name TrafficToAWS tcp
ip inspect name TrafficToAWS udp
ip inspect name TrafficToAWS icmp
ip inspect name TrafficToAWS ssh
ip inspect name TrafficToAWS ftp

ip access-list extended TrafficFromAWS
 permit icmp any any echo
 permit icmp any any echo-reply
 !NOTE: echo-reply needed for sla ping

interface Tunnel1
 ip access-group TrafficFromAWS in
 ip inspect TrafficToAWS out
interface Tunnel2
 ip access-group TrafficFromAWS in
 ip inspect TrafficToAWS out
This is when it gets interesting. While my configuration gives priority to Tunnel1 when sending traffic to AWS, AWS uses both tunnels for return traffic. The issue is that the inspect rules only allow return traffic on the tunnel it exited. Therefore, if a request goes out Tunnel1 but the response is received on Tunnel2, the router will block it. I simply disabled Tunnel2, sacrificing redundancy, but plan to dig into this a bit deeper when I get a chance. If you beat me to it, let me know how you fixed it.

Wednesday, March 4, 2015

ElastiCache as an ASP.NET Session Store

NOTE: This article is reposted from the AWS .Net Blog.  Please post comments here.

Are you hosting an ASP.NET application on AWS? Do you want the benefits of Elastic Load Balancing (ELB) and Auto Scaling, but feel limited by a dependency on ASP.NET session state? Rather than rely on sticky sessions, you can use an out-of-process session state provider to share session state between multiple web servers. In this post, I will show you how to configure ElastiCache and the RedisSessionStateProvider from Microsoft to eliminate the dependency on sticky sessions.


An ASP.NET session state provider maintains a user’s session between requests to an ASP.NET application. For example, you might store the contents of a shopping cart in session state. The default provider stores the user’s session in memory on the web server that received the request.

Using the default provider, your ELB must send every request from a specific user to the same web server. This is known as sticky sessions and greatly limits your elasticity. First, the ELB cannot distribute traffic evenly, often sending a disproportionate amount of traffic to one server. Second, Auto Scaling cannot terminate web servers without losing some user’s session state.

By moving the session state to a central location, all the web servers can share a single copy of session state. This allows the ELB to send requests to any web server, better distributing load across all the web servers. In addition, Auto Scaling can terminate individual web servers without losing session state information.

There are numerous providers available that allow multiple web servers to share session state. One option is use the DynamoDB Session State Provider that ships with the AWS SDK for .NET. This post introduces another option, storing session state in an ElastiCache cluster.

ElastiCache is a web service that makes it easy to deploy, operate, and scale an in-memory cache in the cloud. ElastiCache supports both Memcached and Redis cache clusters. While either technology can store ASP.NET session state, Microsoft offers a provider for Redis, and I will focus on Redis here.

Launch an Elasticache for Redis Cluster

Let us begin by launching a new Elasticache for Redis cluster in the default VPC using PowerShell. Note that you can use the ElastiCache console if you prefer.

First, get a reference to the default VPC and create a new security group for the cluster. The security group must allow inbound requests to Redis, which uses TCP port 6379.

$VPC = Get-EC2Vpc -Filter @{name='isDefault'; value='true'}
$Group = New-EC2SecurityGroup -GroupName 'ElastiCacheRedis' -Description 'Allows TCP Port 6379'
Grant-EC2SecurityGroupIngress -GroupId $Group -IpPermission  @{ IpProtocol="tcp"; FromPort="6379"; ToPort="6379"; IpRanges=$VPC.CidrBlock }

Second, launch a new Redis cluster. In the example below, I launch a single node cluster named “aspnet” running on a t2.micro. Make sure you specify the security group you created above.

New-ECCacheCluster -CacheClusterId 'aspnet' -Engine 'redis' -CacheNodeType 'cache.t2.micro' -NumCacheNode 1 -SecurityGroupId $Group

Finally, get the endpoint address of the instance you just created. Note that you must wait a few minutes for the cluster to launch before the address is available.

(Get-ECCacheCluster -CacheClusterId 'aspnet' -ShowCacheNodeInfo $true).CacheNodes[0].Endpoint.Address

The endpoint address is a fully qualified domain name that ends in and resolves to a private IP address in the VPC. For example, ElastiCache assigned my cluster the address below.

Configuring the Redis Session State Provider

With the Redis cluster running, you are ready to add the RedisSessionStateProvider to your ASP.NET application. Open your project in Visual Studio. First, right click on the project in Solution Explorer and select Manage NuGet Packages. Then, search for “RedisSessionStateProvider” and click the Install button as show below.

Manage NuGet Packages

NuGet will add a custom session state provider to your project’s web.config file. Open the web.config file and locate the Microsoft.Web.Redis.RedisSessionStateProvider shown below.

<sessionState mode="Custom" customProvider="MySessionStateStore">
    <add name="MySessionStateStore" type="Microsoft.Web.Redis.RedisSessionStateProvider" host="" accessKey="" ssl="false" />

Now replace the host attribute with the endpoint address you received from Get-ECCacheCluster. For example, my configuration looks like this.

<sessionState mode="Custom" customProvider="MySessionStateStore">
    <add name="MySessionStateStore" type="Microsoft.Web.Redis.RedisSessionStateProvider" host="" accessKey="" ssl="false" />

You are now ready to deploy and test your application. Wasn’t that easy?


You can use ElastiCache to share ASP.NET session information with multiple web servers and eliminate the dependency on ELB stick sessions. ElastiCache is simple to use and integrates with ASP.NET using the RedisSessionStateProvider available as a NuGet package. For more information about ElastiCache, see the ElastiCache documentation.