Sunday, October 4, 2015

My Cloud EX2 Backup to Amazon S3

With all the devices in the house it was finally time to invest in a NAS. I settled on the Western Digital My Cloud EX2. I picked this specifically because it supported back up to Amazon S3. In practice, the backup software sucks and I had to work around a few issues to get it working reliably and inexpensively.  


Overall I really like the EX2. It has great features for the price. My version came with two 4TB drives which I configured to mirror for redundancy (you can forgo redundancy and get 8TB of storage).  The EX2 supports SMB and NFS. It can act a DLNA (I use an app called Vimu Player on my Fire TV) or iTunes server (unprotected audio only). For the more advanced user, you can also join Active Directory, act as an iSCSI target, and mount ISO images. The EX2 can backup to another EX2, Elephant Drive or Amazon S3. The rest of this post focuses on backup to S3 which is less than perfect, but with a little effort I have it running reliably.

Backup

At a high level, I want the back to protect me from three things: 1) Hardware failure. The EX2 has two disks, but I still want a more protection. 2) My own stupidity. I might accidentally delete or overwrite something. 3) Malware. Most notably CryptoLocker or similar ransom ware. The backup agent built into the EX2 offers three backup types (taken from here):

  • Overwriting existing file(s): Overwrites files in the target folder that have the identical name as your source file.
  • Full Backup: Creates a separate folder containing all of the backup data each time the backup is performed.
  • Incremental Backup: Overwrites files with source files that are newer then the target files.

I wanted the third option, and this is what I am running. Unfortunately, it does not work as advertised. Every once in a while it overwrites files that have not changed. This would be not a big deal, but I want to run versioning to protect against malicious malware overwriting my files. With versioning enabled, S3 stores every version of your files so you can always roll back to an old copy.

The problem is that the EX2 keeps adding versions.  Over the past six months it has created as many as 10 copies of a file that has never changed. This has driven my bill up dramatically. To keep my bill in check I resorted to a lifecycle policy that moves my files to glacier and removes old versions after 30 days.  Glacier is much cheaper and and 30 days gives me enough time to fix a mistake.

Configuration

The first thing I created was an S3 bucket. There is noting special here, just accept the defaults.  Then, I created the lifecycle policy described above. The configuration looks like this:


Next, I needed an IAM user for the backup job on the EX2. I created a user policy that had only those rights needed by the backup job.  This way, even if my EX2 were compromised, the attacker could never delete from my bucket or access other resources in my account. My policy looks like this.  

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::BUCKETNAME"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::BUCKETNAME/*"
        }
    ]
}

Finally, I could configure the backup job on the EX2. The configuration above has been running for a while now. It still overwrites files that have not changed, but the lifecycle policy keeps them under control.


Monday, September 14, 2015

NYC PowerShell User Group

I'll be presenting on AWS tonight at the NYC PowerShell User Group.  The presentation is available here.

Wednesday, April 15, 2015

Configuring an AWS Customer Gateway Behind a NAT

I have been wanting to configure a VPN Connection from AWS to my house, but my cheap Netgear router does not support IPSec. So, I picked up an old Cisco 871 router that does. I didn’t want to sacrifice the speed (it supports 802.11ac while the 871 is an old 802.11g device) and features of my Netgear router, so I put the 871 behind the Netgear and modified the VPN configuration for NAT traversal.
The 871 (or a similar device) is a great way to get some hands on experience configuring a Virtual Private Gateway. Despite its age, the 871 is actually a capable device and it’s available on eBay for less than $100. While most production implementations will not require NAT traversal, this is also good experience. You may want to peer two VPCs (in the same or different regions) and one common solution is to use two Cisco CSR1000V (available in the AWS Marketplace). In this configuration both CSR100V devices will require an Elastic IP, which uses NAT.

My configuration looks like the diagram below. I am using a Netgear router, but any router will work. My cable provider has assigned the address 203.0.113.123 to my Netgear router which also has a private address of 192.168.1.1. I have assigned the Cisco 871 a static IP address of 192.128.1.2 (make sure to exclude this from the DHCP pool on the router). My AWS VPC has the CIDR block 172.16.0.0/16. I have configured a static route on the Netgear that forwards all traffic destined for 172.16.0.0/16 to 192.168.1.2. In addition, I added a port forwarding rule to the Netgear that forwards UDP port 500 to 192.168.1.2.

In the AWS VPC console, I created a VPN Connection as shown below. Note that I have entered the public IP address of the Netgear router (203.0.113.123) as the IP address of a new Customer Gateway. I also configured static routing and entered the CIDR block of my home network (192.168.0.0/16).

Once the VPN connection is created you can download the router configuration. I choose a Cisco Systems ISR Series Router. In order to support NAT traversal you will need to modify the configuration slightly. You need to find the six places where the public IP address appears and replace it with the private IP address of the IPSec router. Not that there will two of each of the highlighted sections below, one for Tunnel1 and one for Tunnel2.
crypto keyring keyring-vpn-d67b98bf-0  
  local-address 203.0.113.123 !Replace with 192.168.1.2
pre-shared-key address 54.240.217.162 key XXXXXXXXXXXXXXXXX
exit
crypto isakmp profile isakmp-vpn-d67b98bf-0
  local-address 203.0.113.123 !Replace with 192.168.1.2
  match identity address 54.240.217.162
  keyring keyring-vpn-d67b89bf-0
exit

crypto ipsec fragmentation before-encryption
interface Tunnel1
  ip address 169.254.255.2 255.255.255.252
  ip virtual-reassembly !Protects against Fragment attacks
  tunnel source 203.0.113.123 !Replace with 192.168.1.2
  tunnel destination 54.240.217.162 
  tunnel mode ipsec ipv4
  tunnel protection ipsec profile ipsec-vpn-d67b98bf-0
  ! This option causes the router to reduce the Maximum Segment Size of
  ! TCP packets to prevent packet fragmentation.
  ip tcp adjust-mss 1387 
  no shutdown
exit
In addition, I needed to change the ip sla timeout because the 871 is too old to support the default value, but this is unique to the 871.
ip sla 100
   icmp-echo 169.254.255.1 source-interface Tunnel1
   timeout 1000 !AWS uses 1000.  Min on 871 is 5000.
   frequency 5
exit
With static routing AWS is not advertising routes over the VPN tunnel. Therefore, I had to add the route statements to the 871 manually. The first statement uses the default administrative distance of 1 and therefore tell the router to prefer Tunnel1. If SLA 100 fails, this route will be removed and Tunnel2 with administrative distance of 10 will take over.
ip route 172.16.0.0 255.255.0.0 Tunnel1 track 100
ip route 172.16.0.0 255.255.0.0 Tunnel2 10 track 200

Extra Credit: Securing the Home Network

In order to protect my home network from nefarious traffic from AWS, I added a “firewall” policy using inspect statements on the 871. The ACL defines what is allowed from AWS. In this case, just ping for testing. All traffic to AWS is allowed and the inspect rules open the return path for any traffic initiated from my house. SSH and FTP defines high level inspect rules specific to these protocols.
ip inspect name TrafficToAWS tcp
ip inspect name TrafficToAWS udp
ip inspect name TrafficToAWS icmp
ip inspect name TrafficToAWS ssh
ip inspect name TrafficToAWS ftp

ip access-list extended TrafficFromAWS
 permit icmp any any echo
 permit icmp any any echo-reply
 !NOTE: echo-reply needed for sla ping
exit

interface Tunnel1
 ip access-group TrafficFromAWS in
 ip inspect TrafficToAWS out
exit
 
interface Tunnel2
 ip access-group TrafficFromAWS in
 ip inspect TrafficToAWS out
exit
This is when it gets interesting. While my configuration gives priority to Tunnel1 when sending traffic to AWS, AWS uses both tunnels for return traffic. The issue is that the inspect rules only allow return traffic on the tunnel it exited. Therefore, if a request goes out Tunnel1 but the response is received on Tunnel2, the router will block it. I simply disabled Tunnel2, sacrificing redundancy, but plan to dig into this a bit deeper when I get a chance. If you beat me to it, let me know how you fixed it.

Wednesday, March 4, 2015

ElastiCache as an ASP.NET Session Store

NOTE: This article is reposted from the AWS .Net Blog.  Please post comments here.

Are you hosting an ASP.NET application on AWS? Do you want the benefits of Elastic Load Balancing (ELB) and Auto Scaling, but feel limited by a dependency on ASP.NET session state? Rather than rely on sticky sessions, you can use an out-of-process session state provider to share session state between multiple web servers. In this post, I will show you how to configure ElastiCache and the RedisSessionStateProvider from Microsoft to eliminate the dependency on sticky sessions.

Background

An ASP.NET session state provider maintains a user’s session between requests to an ASP.NET application. For example, you might store the contents of a shopping cart in session state. The default provider stores the user’s session in memory on the web server that received the request.

Using the default provider, your ELB must send every request from a specific user to the same web server. This is known as sticky sessions and greatly limits your elasticity. First, the ELB cannot distribute traffic evenly, often sending a disproportionate amount of traffic to one server. Second, Auto Scaling cannot terminate web servers without losing some user’s session state.

By moving the session state to a central location, all the web servers can share a single copy of session state. This allows the ELB to send requests to any web server, better distributing load across all the web servers. In addition, Auto Scaling can terminate individual web servers without losing session state information.

There are numerous providers available that allow multiple web servers to share session state. One option is use the DynamoDB Session State Provider that ships with the AWS SDK for .NET. This post introduces another option, storing session state in an ElastiCache cluster.

ElastiCache is a web service that makes it easy to deploy, operate, and scale an in-memory cache in the cloud. ElastiCache supports both Memcached and Redis cache clusters. While either technology can store ASP.NET session state, Microsoft offers a provider for Redis, and I will focus on Redis here.

Launch an Elasticache for Redis Cluster

Let us begin by launching a new Elasticache for Redis cluster in the default VPC using PowerShell. Note that you can use the ElastiCache console if you prefer.

First, get a reference to the default VPC and create a new security group for the cluster. The security group must allow inbound requests to Redis, which uses TCP port 6379.

$VPC = Get-EC2Vpc -Filter @{name='isDefault'; value='true'}
$Group = New-EC2SecurityGroup -GroupName 'ElastiCacheRedis' -Description 'Allows TCP Port 6379'
Grant-EC2SecurityGroupIngress -GroupId $Group -IpPermission  @{ IpProtocol="tcp"; FromPort="6379"; ToPort="6379"; IpRanges=$VPC.CidrBlock }

Second, launch a new Redis cluster. In the example below, I launch a single node cluster named “aspnet” running on a t2.micro. Make sure you specify the security group you created above.

New-ECCacheCluster -CacheClusterId 'aspnet' -Engine 'redis' -CacheNodeType 'cache.t2.micro' -NumCacheNode 1 -SecurityGroupId $Group

Finally, get the endpoint address of the instance you just created. Note that you must wait a few minutes for the cluster to launch before the address is available.

(Get-ECCacheCluster -CacheClusterId 'aspnet' -ShowCacheNodeInfo $true).CacheNodes[0].Endpoint.Address

The endpoint address is a fully qualified domain name that ends in cache.amazon.com and resolves to a private IP address in the VPC. For example, ElastiCache assigned my cluster the address below.

aspnet.k30h8n.0001.use1.cache.amazonaws.com

Configuring the Redis Session State Provider

With the Redis cluster running, you are ready to add the RedisSessionStateProvider to your ASP.NET application. Open your project in Visual Studio. First, right click on the project in Solution Explorer and select Manage NuGet Packages. Then, search for “RedisSessionStateProvider” and click the Install button as show below.

Manage NuGet Packages

NuGet will add a custom session state provider to your project’s web.config file. Open the web.config file and locate the Microsoft.Web.Redis.RedisSessionStateProvider shown below.

<sessionState mode="Custom" customProvider="MySessionStateStore">
  <providers>
    <add name="MySessionStateStore" type="Microsoft.Web.Redis.RedisSessionStateProvider" host="127.0.0.1" accessKey="" ssl="false" />
  </providers>
</sessionState>

Now replace the host attribute with the endpoint address you received from Get-ECCacheCluster. For example, my configuration looks like this.

<sessionState mode="Custom" customProvider="MySessionStateStore">
  <providers>
    <add name="MySessionStateStore" type="Microsoft.Web.Redis.RedisSessionStateProvider" host="aspnet.k30h8n.0001.use1.cache.amazonaws.com" accessKey="" ssl="false" />
  </providers>
</sessionState>

You are now ready to deploy and test your application. Wasn’t that easy?

Summary

You can use ElastiCache to share ASP.NET session information with multiple web servers and eliminate the dependency on ELB stick sessions. ElastiCache is simple to use and integrates with ASP.NET using the RedisSessionStateProvider available as a NuGet package. For more information about ElastiCache, see the ElastiCache documentation.

Thursday, February 19, 2015

Enforcing a Squid Access Policy for Amazon S3 and Yum

NOTE: This article is reposted from the AWS Site. The original is located here.

In this article, we will set up an example situation showing how to use the open source Squid proxy to control access to Amazon Simple Storage Service (S3) from within an Amazon Virtual Private Cloud (VPC). First, you will configure Squid to allow access to Linux Yum repositories. Next, you will configure Squid to restrict access to a list of approved Amazon S3 buckets. Then, you will configure Squid to direct traffic based on the URL, sending some requests to an Internet gateway (IGW) and other traffic to a virtual private gateway (VGW). Finally, you will explore options for making Squid highly available.

In our example, Alice is a Chief Technology Officer (CTO) at a small consulting firm. Despite its small size, her company has many high-profile customers, and Alice has worked hard to gain their trust. Alice employs a set of strict firewall policies and has deployed numerous security appliances over the past few years.

As the company begins to migrate applications to the cloud, Alice's team is discussing how to implement similar polices using Amazon Web Services (AWS). The first order of business is blocking access to the Internet. Developers should not be able to download files from the Internet except for a few approved scenarios. These scenarios include accessing Yum repositories to update Amazon Linux, and using AWS services such as Amazon S3. Alice plans to implement this policy by using an IP address restriction in an Amazon Elastic Compute Cloud (EC2) security group.

Alice finds numerous posts in the AWS forums from people asking for the IP address ranges of the Yum repositories and Amazon S3. However, Amazon does not publish this list. Why? In the cloud, resources are highly elastic. Applications grow and shrink in response to demand. An IP address assigned to one application today might be assigned to another application tomorrow.

As applications expand and contract, instances are added and removed, and the Domain Name Service (DNS) is constantly updated with new IP addresses. In the cloud, you cannot rely on IP address-based security rules; therefore, you must base security policies on domain names because they will not change as the application scales. However, Amazon EC2 security groups and network access control lists (ACLs) do not support rules based on domain names. Alice needs to find another solution to implement her security policy.

Deploying and Configuring Squid

Alice decides to use Squid, an open source web proxy, to implement her policy. Squid will allow access to an approved list of services, but deny all other Internet access. (Note that Alice chose Squid, but there are numerous solutions that she could have chosen.)

She begins by creating the VPC shown in Figure 1.

Figure 1 - VPC configured to allow Internet access through a Squid proxy

As shown in Figure 1, Alice wants to block direct access to the Internet from the application instances. Instead, application instances must access the Internet through the Squid proxy. To ensure that all application instances use the proxy, Alice creates a new network ACL for the Application Subnet with the rules shown in the table in Figure 2.

Note that AWS offers both security groups and network ACLs to secure your application. A security group is applied to an instance; a network ACL is applied to the entire subnet. Alice uses a network ACL to ensure that the rules apply to all instances deployed in the application subnet. For more information about security groups and network ACLs see the Amazon VPC documentation.

Figure 2: The application subnet ACL

The ACL in Figure 2 allows HTTP/S within the VPC (rules 100 and 101), but blocks HTTP/S to the Internet (rules 200 and 201). Therefore, the only way for instances in the application subnet to get access to the Internet is through the Squid proxy.

Note that because the application instances access the Internet through the proxy, the application subnet can be private. A private subnet does not have a route to the Internet. For more information about public and private subnets, see the VPC documentation.

Next, Alice launches a new Amazon Linux AMI (Amazon Machine Image) in the DMZ subnet and assigns it an Elastic IP address. Then she installs Squid using the following commands.

sudo yum update -y
sudo yum install -y squid

After she installs Squid, she begins to configure it. The configuration is stored in a text file located at /etc/squid/squid.conf. Alice uses vim to edit the file.

sudo vim /etc/squid/squid.conf

Squid uses rules, called ACLs, to identify traffic. Do not confuse Squid's ACLs with the Amazon EC2 network ACLs we created above. The first rule Alice encounters is src used to identify traffic by the source IP address of a request. In other words, the proxy will only allow requests from these addresses. By default, Squid will allow requests from any private address. This is the default configuration:

acl localnet src 10.0.0.0/8     
acl localnet src 172.16.0.0/12  
acl localnet src 192.168.0.0/16 
acl localnet src fc00::/7       
acl localnet src fe80::/10  

Alice wants to further limit access to only include instances within the VPC, so she deletes these rules and creates a single rule that allows requests from 10.1.0.0/16, the Classless Inter-Domain Routing (CIDR) range of her VPC.

acl localnet src 10.1.0.0/16   #Only allow requests from within the VPC

With only the source defined, Squid will allow access to any URL. This is a good time to test. Alice saves her changes and starts the Squid daemon.

$ sudo service squid start

She opens an SSH session to one of the application servers and configures it to use the proxy.

$ export http_proxy=http://10.1.1.10:3128
$ export https_proxy=http://10.1.1.10:3128
$ export no_proxy="169.254.169.254"

There are a few important things you need to know about the prior command:

  • The proxy configuration used here is only valid for the current session. This is fine for testing, but to persist these entries you should add them to /etc/profile.d/proxy.sh
  • Most, but not all, applications will use these environment variables. Check the documentation of your application for details on configuring a proxy server.
  • Squid listens on port 3128 by default. You can change the port in the squid.conf file.
  • 169.254.169.254 is the Amazon EC2 metadata service. We are excluding this because we want our instances to hit the metadata service directly. If we proxy these requests, the metadata service will return information about the proxy instance rather than the instance that made the request

At this point the proxy will allow access to any URL. To ensure everything is working, Alice uses curl to load www.google.com.

$ curl -I http://www.google.com 
HTTP/1.1 200 OK 
...
Via: 1.0 ip-10-1-1-10 (squid/3.1.10)

The response code of 200 indicates that everything is configured as expected, and the Via header indicates that the application instance is using the proxy to access the Internet. Everything appears to working as expected.

Granting Access to Yum

With Squid installed and working, Alice continues to implement her security policy. She moves on to the Yum repository. As shown in Figure 3, Alice wants to allow access to the Yum repository, and deny all other Internet access.

Figure 3 - Squid denying access to everything except the Yum repository

Alice returns to the Squid instance and opens the Squid configuration file.

sudo vim /etc/squid/squid.conf

Next, she creates a set of destination rules just after the source rule she created in the last step. These rules define what resources the instances can access. Alice uses a "dstdomain" rule to match a DNS name.

There are two Yum URLs for each region. If one region is not available, Yum will try to contact another region. Therefore, Alice adds all regions to her configuration.

acl yum dstdomain repo.us-east-1.amazonaws.com
acl yum dstdomain repo.us-west-1.amazonaws.com
acl yum dstdomain repo.us-west-2.amazonaws.com
acl yum dstdomain repo.eu-west-1.amazonaws.com
acl yum dstdomain repo.eu-central-1.amazonaws.com
acl yum dstdomain repo.ap-southeast-1.amazonaws.com
acl yum dstdomain repo.ap-southeast-2.amazonaws.com
acl yum dstdomain repo.ap-northeast-1.amazonaws.com
acl yum dstdomain repo.sa-east-1.amazonaws.com
acl yum dstdomain packages.us-east-1.amazonaws.com
acl yum dstdomain packages.us-west-1.amazonaws.com
acl yum dstdomain packages.us-west-2.amazonaws.com
acl yum dstdomain packages.eu-west-1.amazonaws.com
acl yum dstdomain packages.eu-central-1.amazonaws.com
acl yum dstdomain packages.ap-southeast-1.amazonaws.com
acl yum dstdomain packages.ap-northeast-1.amazonaws.com
acl yum dstdomain packages.sa-east-1.amazonaws.com
acl yum dstdomain packages.ap-southeast-2.amazonaws.com

Now that the ACLs are defined to match both source and destination, Alice can update the access rule. As we saw earlier, the default access rule only checks that the request came from the local network (in this case the VPC).

http_access allow localnet

Alice wants to check both the source and destination; therefore, she changes the access rule to check that the request came from the VPC and is going to the Yum repository. All other requests will be denied. Her rule looks like this:

http_access allow localnet yum 

Alice saves her changes and restarts the Squid daemon.

$ sudo service squid restart

Alice is ready to test the new configuration. She returns to the application instance. Note: be sure the proxy is still configured.

Alice once again tests access to Google, and this time she gets the expected 403 forbidden error. Note the X-Squid-Error header below. This indicates that Squid denied the request rather than the web server.

$ curl -I www.google.com 
HTTP/1.0 403 Forbidden
...
X-Squid-Error: ERR_ACCESS_DENIED 0

Next, Alice tries a Yum URL to ensure it is working.

$ curl -I http://repo.us-east-1.amazonaws.com/latest/main/mirror.list
HTTP/1.1 200 OK 

Finally, Alice tests Yum and shows it is working as expected.

$ yum check-update
Loaded plugins: priorities, update-motd, upgrade-helper
Security: kernel-3.14.20-20.44.amzn1.x86_64 is the currently running version

Granting Access to Amazon S3

With Yum working, Alice moves on to Amazon S3. As shown in Figure 4, she wants to allow access to both the Yum repository and Amazon S3. Squid will continue to block access to all other URLs.

Figure 4 - Squid allowing access to Yum repository and Amazon S3 buckets

Amazon S3 supports two types of URLs, path and virtual host. A path URL is in the format https://s3.amazonaws.com/mybucket and a virtual host URL is in the format https://mybucket.s3.amazonaws.com/. See the Amazon S3 documentation for more information.

In order to support both URL types, Alice uses a regular expression. For example, all domain names in US Standard will end with "s3.amazon.com" regardless of the URL type.

Returning the Squid instance, Alice opens the configuration file.

sudo vim /etc/squid/squid.conf

Alice plans to support all AWS regions, so she adds a line for each region.

acl s3 dstdom_regex .*s3\.amazonaws\.com
acl s3 dstdom_regex .*s3\.eu-central-1\.amazonaws\.com
acl s3 dstdom_regex .*s3\.sa-east-1\.amazonaws\.com
acl s3 dstdom_regex .*s3\.ap-northeast-1\.amazonaws\.com
acl s3 dstdom_regex .*s3\.eu-west-1\.amazonaws\.com
acl s3 dstdom_regex .*s3\.us-west-1\.amazonaws\.com
acl s3 dstdom_regex .*s3\.us-west-2\.amazonaws\.com
acl s3 dstdom_regex .*s3\.ap-southeast-2\.amazonaws\.com
acl s3 dstdom_regex .*s3\.ap-southeast-1\.amazonaws\.com

Alice also adds a new access rule that allows requests from the VPC to Amazon S3. With the Yum rule still in place, the access rules look like this:

http_access allow localnet yum 
http_access allow localnet s3

Alice saves her changes and restarts the Squid daemon.

$ sudo service squid restart

Returning to the application instance, Alice tries an Amazon S3 bucket using both the path and virtual host URLs and sees that both are working as expected. Remember to configure the environment variables if you are starting a new SSH session.

$ curl -I https://mybucket.s3.amazonaws.com/test.txt 
HTTP/1.1 200 OK 
$ curl -I https://s3.amazonaws.com/mybucket/text.txt 
HTTP/1.1 200 OK 

Finally, Alice tests access from the AWS CLI. Everything works great.

$  aws s3 ls s3://mybucket
2014-10-22 21:32:48          0 
2014-10-22 21:38:27         15 test.txt

Whitelisting Buckets

Alice is really happy with the results, but she wants to take it a step further. Currently, Squid allows access to any Amazon S3 bucket owned by any AWS customer. As shown in Figure 5, Alice would like to limit access toonly the buckets the team needs access to (e.g., mybucket) and block access to any other buckets.

Figure 5 - Squid allowing access to specific S3 buckets

Alice returns to the Squid instance and opens the configuration file again. She creates two new ACLs that identify "mybucket" which is stored in the US Standard region. She must create two rules, one for each of the URL types discussed above.

acl virtual_host_urls dstdomain mybucket.s3.amazonaws.com
acl path_urls url_regex s3\.amazonaws\.com/mybucket/.*

The first ACL identifies the virtual host style URL and uses dstdomain that we saw earlier. No regular expression is needed here because Alice knows the exact host header. The second ACL identifies the path style URL and uses url_regex to match URLs that begin with "s3.amazonaws.com/mybucket/".

Now, Alice locates the rule she created earlier.

http_access allow localnet s3

And then she replaces the rule with two new rules (one for each ACL). The complete list of access rules now looks like this:

http_access allow localnet yum 
http_access allow localnet virtual_host_urls
http_access allow localnet path_urls

Next, she restarts the Squid daemon so she can test again.

$ sudo service squid restart

Alice returns to her application instance and tests access to mybucket. Everything appears to be working as expected.

$  aws s3 ls s3://mybucket
2014-10-22 21:32:48          0 
2014-10-22 21:38:27         15 test.txt

At this point we want to discuss an issue with HTTPS URLs. The AWS CLI and most other tools use HTTPS. Notice that when testing with HTTPS, the virtual host-style URL works:

$ curl -I https://mybucket.s3.amazonaws.com/test.txt 
HTTP/1.1 200 OK 

However, the path style URL returns a 403. Why?

$ curl -I https://s3.amazonaws.com/mybucket/text.txt 
HTTP/1.0 403 Forbidden
X-Squid-Error: ERR_ACCESS_DENIED 0

This fails because the path style ACL Alice created needs to see the entire URL, part of which is inside the encrypted HTTPS packet. In the virtual host URL all the information is in the host name (see figure 6). In the path style URL, the path-which includes the bucket name-is encrypted (see figure 7).

Figure 6 - Virtual host style URL sent over SSL

Figure 7 - Path style URL sent over SSL

Squid uses a feature called SSL Bump to decrypt requests. SSL Bump is outside the scope for this article, but you can read more on the Squid website. Luckily, the AWS CLI uses virtual host URLs and works as expected without needing to decrypt SSL.

Controlling Squid's Outbound Interface

Alice is really excited about the security she has put in place so far. But, over the first few weeks of the project, the team identifies other resources that they need access to. Updating the rules is getting tedious. All of these exceptions were identified in the data center a long time ago. Alice wants to leverage the existing infrastructure.

Alice decides to add a virtual private gateway (VGW) to connect the VPC to her company's data center. With the VGW in place, she can configure the VPC to send all HTTP/S requests through her data center where existing security policies have already been defined. But, Alice does not want to add latency to the application by sending Amazon S3 requests over the VPN tunnel.

Alice wants a solution that leverages the VGW to send most requests to the data center, but allows her to identify special cases that should use the Internet gateway for low-latency access to specific services. Therefore, she reconfigures her VPC as shown in Figure 8.

Figure 8 - Squid directing traffic to an IGW or VGW

In this new design, Alice adds a resources subnet. This subnet has a new route table with a default route that points to the VGW instead of than the IGW. She also adds an Elastic Network Interface (ENI) to the Squid proxy (shown in Figure 8 with the IP address 10.1.2.10) and places it in the resources subnet. When the Squid proxy sends requests out the 10.1.1.10 interface, the VPC will route the request out the IGW. When the Squid proxy sends requests out the 10.1.2.10 interface, the VPC will route the request out the VGW.

Rather than deny requests, Alice reconfigures the Squid proxy to allow all requests, but send them out one of the two interfaces based on the URL. Requests for Yum and S3 will exit the 10.1.1.10 interface, and be routed out the IGW. All other traffic will exit the 10.1.2.10 interface, and be routed over the VPN tunnel to the data center. After the request is in the data center, the existing infrastructure can determine how to handle each request (indicated by the two yellow lines labeled "TBD" in Figure 8).

Alice once again returns to the Squid configuration file. First, she replaces the access rules with one that allows all traffic from the VPC. She removes the following items:

http_access allow localnet yum 
http_access allow localnet virtual_host_urls
http_access allow localnet path_urls

And she adds the following item:

http_access allow localnet

Note that this is the same rule we used at the beginning of this article. Now the proxy will once again allow any traffic, from anywhere in the VPC, regardless of the destination. Rather than deny this traffic, Squid will forward it her company's data center and allow the existing infrastructure to decide what to do with it.

Next, Alice configures the outgoing address. If a request is destined for a Yum repository or her Amazon S3 bucket, it will be sent to the Internet gateway using the interface with an IP address of 10.1.1.10. If not, it will be sent to the virtual private gateway using the interface with an IP address of 10.1.2.10.

tcp_outgoing_address 10.1.1.10 localnet yum
tcp_outgoing_address 10.1.1.10 localnet virtual_host_urls
tcp_outgoing_address 10.1.1.10 localnet path_urls
tcp_outgoing_address 10.1.2.10 localnet

With these rules in place, Alice has enabled low-latency access to Amazon S3 while ensuring that cloud-based applications are subject to the same security policy as the applications hosted in the data center.

High Availability

Squid has become an integral part of Alice's applications, which depend on it for access to data stored in Amazon S3. Alice wants to ensure that the Squid solution is highly available. There are few ways to approach this.

One solution, discussed in a prior article, is to host multiple Squid instances in an Auto Scaling group behind a private Elastic Load Balancer (ELB). Unfortunately, Alice's company is small, and she is working on a tight budget. She does not want to pay for multiple Squid instances and an ELB.

Alice decides to use a single Squid instance as shown in Figure 9. She puts this instance in an Auto Scaling group with a min and max of one. If the Squid instance - or even an entire Availability Zone - fails, the Auto Scaling group will replace it with a new instance.

Figure 9 - Making Squid highly available

This adds obvious complexity. When the Auto Scaling group replaces a Squid instance, the application instances need to begin using the new Squid instance. Alice uses Amazon Route 53 to create a DNS entry (e.g., proxy.example.com) to refer to the proxy instance. Amazon Route 53 is Amazon's highly available and scalable DNS service.

The application instances will reference the Squid instance using the DNS name rather than an IP address as shown below. Now, when a Squid instance fails, Alice only needs to update the DNS entry, and the application instances will all begin using the new Squid instance.

$ export http_proxy=http://proxy.example.com:3128
$ export https_proxy=http://proxy.example.com:3128
$ export no_proxy="169.254.169.254"

To make this process automatic, Alice creates the simple shell script shown below. This script will update Amazon Route 53 whenever a new Squid instance is launched. She adds this script to the user-data section of the AutoScaling group's launch configuration.

#!/bin/bash
cat <<EOF > ~/dns.conf
{
  "Comment": "string",
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "proxy.example.com",
        "Type": "A",
        "TTL": 60,
        "ResourceRecords": [
          {
            "Value": "$(curl http://169.254.169.254/latest/meta-data/local-ipv4/)"
          }
        ]
      }
    }
  ]
}
EOF
aws route53 change-resource-record-sets -hosted-zone-id "ZONEID" -change-batch "file://~/dns.conf"

This script will ensure that the DNS entry always points to the most recently launched Squid instance. The script uses the Amazon EC2 metadata service to discover the IP address of the instance it is running on. Then it calls the Amazon Route 53 API to update the DNS entry.

Note that you must replace ZONEID with the ID of the Amazon Route 53 hosted zone you want to update. In addition, your instance must use an Amazon EC2 role that has permission to update Route 53.

With the Auto Scaling group configured, Alice can be sure that her application can recover from the failure of a Squid instance.

Conclusion

Alice has learned that the cloud is inherently elastic and she cannot depend on IP addresses remaining static. In the past she built security rules based on IP addresses and CIDR blocks. In the cloud she needs consider basing security rules on DNS names.

Alice deployed a Squid proxy to control access to Yum repositories and Amazon S3. Squid can be used to allow access to all of Amazon S3 or only specific buckets. It can also be used to direct traffic to follow a different path based on policy.

Alice was able to host her application at AWS and leverage her company's existing security infrastructure. In addition, she built a highly available solution using Amazon Route 53 and Auto Scaling.

Friday, February 6, 2015

Discovering Windows Version on EC2 Instances

Windows Server 2003 end of life is less than six months away. As I start to think about upgrading, I was looking for an easy way to identify what version of Windows is running on each EC2 instance.  I would like to do this without having to log into each instance.

One solution is to use the System log.  If the instance has the EC2 Config service running on it it will report the OS version (along with a few key driver versions to the console).  You can access the System Log from the console by right clicking on an instance and choosing "View System Log".  For example, the output below is from a Windows 2003 R2 instance I just launched.  Notice the OSVersion on line three.

2015/02/06 11:50:30Z: AMI Origin Version: 2015.01.14
2015/02/06 11:50:30Z: AMI Origin Name: Windows_Server-2003-R2_SP2-English-64Bit-Base
2015/02/06 11:50:30Z: OsVersion: 5.2
2015/02/06 11:50:30Z: OsServicePack: Service Pack 2
2015/02/06 11:50:30Z: OsProductName: Microsoft Windows Server 2003 R2
2015/02/06 11:50:30Z: OsBuildLabEx: NotFound
2015/02/06 11:50:30Z: Language: en-US
2015/02/06 11:50:30Z: EC2 Agent: Ec2Config service v2.3.313.0
2015/02/06 11:50:30Z: EC2 Agent: Ec2Config service fileversion v2.3.313
2015/02/06 11:50:32Z: Driver: Citrix PV Ethernet Adapter v5.9.960.49119
2015/02/06 11:50:32Z: Driver: Citrix PV SCSI Host Adapter v6.0.2.56921
2015/02/06 11:50:33Z: Message: Waiting for meta-data accessibility...
2015/02/06 11:50:34Z: Message: Meta-data is now available.
2015/02/06 11:50:47Z: AMI-ID: ami-529be93a
2015/02/06 11:50:47Z: Instance-ID: i-01bcc4f0
2015/02/06 11:50:47Z: Ec2SetPassword: Enabled
2015/02/06 11:50:48Z: RDPCERTIFICATE-SUBJECTNAME: O=Amazon.com, OU=EC2, CN=i-01bcc4f0
2015/02/06 11:50:48Z: RDPCERTIFICATE-THUMBPRINT: 965A5E6BF3DD3A31C66B4BE78D2B41735A10B540
2015/02/06 11:50:50Z: Username: Administrator
2015/02/06 11:50:50Z: Password: 
2015/02/06 11:50:50Z: Message: Windows is Ready to use

I created a script that will query the system log (also called the console) from every Windows instance in every region using PowerShell.  It then applies a regular expression to parse the OS version number.

Get-EC2Region | % {
    $Region = $_.RegionName

    (Get-EC2Instance -Region $Region).Instances | Where-Object {$_.Platform -eq "Windows"} | % {
        $Instance = $_
        #Get the console output for each instance
        $Console = (Get-EC2ConsoleOutput -Region $Region -InstanceId $Instance.InstanceId).Output
        $Console = [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($Console));

        #Parse the console output looking for OS and Driver versions.  Note that different EC2Config versions use different formats.
        $OSVersion1 = If($Console -match "OS: Microsoft Windows NT (?[\d.]*)"){ $Matches['OSLongVersion'] };
        $OSVersion2 = If($Console -match "OsVersion: (?[\d.]*)"){ $Matches['OSShortVersion'] };
        $OSVersion = if($OSVersion1 -ne $null) { $OSVersion1 } Else { $OSVersion2 }
        $NICDriver1 = If($Console -match "Driver: AWS PV Network Device v(?[\d.]*)"){ $Matches['NICDriver'] };
        $NICDriver2 = If($Console -match "Driver: Citrix PV Ethernet Adapter v(?[\d.]*)"){ $Matches['NICDriver'] };
        $NICDriver = if($NICDriver1 -ne $null) { $NICDriver1 } Else { $NICDriver2 } 
        $HBADriver1 = If($Console -match "Driver: AWS PV Storage Host Adapter v(?[\d.]*)"){ $Matches['HBADriver'] }
        $HBADriver2 = If($Console -match "Driver: Citrix PV SCSI Host Adapter v(?[\d.]*)"){ $Matches['HBADriver'] }
        $HBADriver = if($HBADriver1 -ne $null) { $HBADriver1 } Else { $HBADriver2 } 

        #Add a new row to the output
        New-Object PSObject -Property @{
            Region = $Region
            InstanceId = $Instance.InstanceId
            InstanceName = ($Instance.Tags | Where-Object {$_.Key -eq "Name"}).Value
            ImageId = $Instance.ImageId
            OSVersion = $OSVersion
            NICDriver = $NICDriver
            HBADriver = $HBADriver
        }
  
    }
 } | Format-Table -AutoSize -Property Region, InstanceName, InstanceId, ImageId, OSVersion, NICDriver, HBADriver                

You can see the sample output below.  The last instance is Windows 2003 indicated by the version number 5.2.  You can find a list of version numbers on Microsoft's Web Site.

Region    InstanceName       InstanceId ImageId      OSVersion  NICDriver     HBADriver  
------    ------------       ---------- -------      ---------  ---------     ---------  
us-east-1 SERVER001          i-583480b4 ami-e49a0b8c 6.2        5.9.960.49119 6.0.2.56921
us-east-1 SERVER002          i-da8c5236 ami-c4fe9aac 6.3        7.2.4.1       7.2.4.1    
us-east-1 SERVER003          i-b316c842 ami-c4fe9aac 6.3        7.2.4.1       7.2.4.1    
us-east-1 SERVER004          i-bee1715f ami-7614ac1e 6.1.7601   5.9.960.49119 6.0.2.56921
us-east-1 SERVER005          i-01bcc8f0 ami-529be93a 5.2        5.9.960.49119 6.0.2.56921
Now a few warnings.  First, you must have EC2 Config installed in the instance to output information to the console.  Second, depending on the version of EC2 config the output format is different.  I have discovered two versions in my testing, but there may be others.  Third, I have noticed that you cannot always distinguish the R2 version of an OS.

Blogger is messing with the script a bit.  You can download a copy here.  Just rename it from .txt to .ps1.

Wednesday, January 28, 2015

Cross-Account IAM Roles in Windows PowerShell

NOTE: This article is reposted from the AWS .Net Blog.  Please post comments here.

As a company’s adoption of Amazon Web Services (AWS) grows, most customers adopt a multi-account strategy. Some customers choose to create an account for each application, while others create an account for each business unit or environment (development, testing, production). Whatever the strategy, there is often a use case that requires access to multiple accounts at once. This post examines cross-account access and the AssumeRole API, known as Use-STSRole in Windows PowerShell.

A role consists of a set of permissions that grant access to actions or resources in AWS. An application uses a role by calling the AssumeRole API function. The function returns a set of temporary credentials that the application can use in subsequent function calls. Cross-account roles allow an application in one account to assume a role (and act on resources) in another account.

One common example of cross-account access is maintaining a configuration management database (CMDB). Most large enterprise customers have a requirement that all servers, including EC2 instances, must be tracked in the CMDB. Example Corp., shown in Figure 1, has a Payer account and three linked accounts: Development, Testing, and Production.

Figure 1: Multiple Accounts Owned by a Single Customer

Note that linked accounts are not required to use cross-account roles, but they are often used together. You can use cross-account roles to access accounts that are not part of a consolidated billing relationship or between accounts owned by different companies. See the user guide to learn more about linked accounts and consolidated billing.

Scenario

Bob, a Windows administrator at Example Corp., is tasked with maintaining an inventory of all the instances in each account. Specifically, he needs to send a list of all EC2 instances in all accounts to the CMDB team each night. He plans to create a Windows PowerShell script to do this.

Bob could create an IAM user in each account and hard-code the credentials in the script. Though this would be simple, hard-coding credentials is not the most secure solution. The AWS best practice is to Use IAM roles. Bob is familiar with IAM roles for Amazon EC2 and wants to learn more about cross-account roles.

Bob plans to script the process shown in Figure 2. The CMDB script will run on an EC2 instance using the CMDBApplication role. For each account, the script will call Use-STSRole to retrieve a set of temporary credentials for the CMDBDiscovery role. The script will then iterate over each region and call Get-EC2Instance using the CMDBDiscovery credentials to access the appropriate account and list all of its instances.

Figure 2: CMDB Application and Supporting IAM Roles

Creating IAM Roles

Bob begins to build his solution by creating the IAM roles shown in Figure 3. The Windows PowerShell script will run on a new EC2 instance in the Payer account. Bob creates a CMDBApplication role in the Payer account. This role in used by the EC2Instance allowing the script to run without requiring IAM user credentials. In addition, Bob will create a CMDBDiscovery role in every account. The CMDBDiscovery role has permission to list (or “Discover”) the instances in that account.

Figure 3: CMDB Application and Supporting IAM Roles

Notice that Bob has created two roles in the Payer account: CMDBApplication and CMDBDiscovery. You may be asking why he needs a cross-account role in the same account as the application. Creating the CMDBDiscovery role in every account makes the code easier to write because all accounts are treated the same. Bob can treat the Payer account just like any of the other accounts without a special code branch.

Bob first creates the Amazon EC2 role, CMDBApplication, in the Payer account. This role will be used by the EC2 instance that runs the Windows PowerShell script. Bob signs in to the AWS Management Console for the Payer account and follows the instructions to create a new IAM Role for Amazon EC2 with the following custom policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["sts:AssumeRole"],
      "Resource": ["*"]
    }
  ]
}

Policy 1: Policy Definition for the CMDBApplication IAM Role

The CMDBApplication role grants a single permission, sts:AssumeRole, which allows the application to call the AssumeRole API to get temporary credentials for another account. Notice that Bob is following the best practice of Least Privilege and has assigned only one permission to the application.

Next, Bob creates a cross-account role called CMDBDiscovery in each of the accounts, including the Payer account. This role will be used to list the EC2 instances in that account. Bob signs in to the console for each account and follows the instructions to create a new IAM role for cross-account access. In the wizard, Bob supplies the account ID of the Payer account (111111111111 in our example) and specifies the following custom policy.

Note that when creating the role, there are two options. One provides access between accounts you own, and the other provides access from a third-party account. Third-party account roles include an external ID, which is outside the scope of this article. Bob chooses the first option because his company owns both accounts.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["ec2:DescribeInstances"],
      "Resource": ["*"]
    }
  ]
}

Policy 2: Policy Definition for the CMDBDiscovery IAM Role

Again, this policy follows the best practice of least privilege and assigns a single permission,ec2:DescribeInstances, which allows the caller to list the EC2 instances in the account.

Creating the CMDB Script

With the IAM roles created, Bob next launches a new EC2 instance in the Payer account. This instance will use the CMDBApplication role. When the instance is ready, Bob signs in and creates a Windows PowerShell script that will list the instances in each account and region.

The first part of the script, shown in Listing 1, lists the instances in a given region and account. Notice that in addition to the account number and region, the function expects a set of credentials. These credentials represent the CMDBDiscovery role and will be retrieved from the AssumeRole API in the second part of the script.

 
Function ListInstances {
    Param($Credentials, $Account, $Region)
          
    #List all instances in the region
    (Get-EC2Instance -Credentials $Credentials -Region $Region).Instances | % {
        If($Instance = $_) {
  
            #If there are instances in this region return the desired attributes
            New-Object PSObject -Property @{
                Account = $Account
                Region = $Region
                InstanceId = $Instance.InstanceId
                Name = ($Instance.Tags | Where-Object {$_.Key -eq 'Name'}).Value
            }
        }
    }
}

Listing 1: Windows PowerShell Function to List EC2 Instances

The magic happens in the second part of the script, shown in Listing 2. We know that the script is running on the new EC2 instance using the CMDBApplication. Remember that the only thing this role can do is call the AssumeRole API. Therefore, we should expect to see a call to AssumeRole. The Windows PowerShell cmdlet that implements AssumeRole is Use-STSRole.

#List of accounts to check
$Accounts = @(111111111111, 222222222222, 333333333333, 444444444444)
  
#Iterate over each account
$Accounts | % {
    $Account = $_
    $RoleArn = "arn:aws:iam::${Account}:role/CMDBDiscovery"
  
    #Request temporary credentials for each account and create a credential object
    $Response = (Use-STSRole -Region $Region -RoleArn $RoleArn -RoleSessionName 'CMDB').Credentials
    $Credentials = New-AWSCredentials -AccessKey $Response.AccessKeyId -SecretKey $Response.SecretAccessKey -SessionToken $Response.SessionToken
  
    #Iterate over all regions and list instances
    Get-AWSRegion | % {
        ListInstances -Credentials $Credentials -Account $Account -Region $_.Region
    }
  
} | ConvertTo-Csv

Listing 2: Windows PowerShell Script That Calls AssumeRole

Use-STSRole will retrieve temporary credentials for the IAM role specified in the ARN parameter. The ARN uses the following format, where "ROLE_NAME" is the role you created in "TARGET_ACCOUNT_NUMBER" (e.g. CMDBDiscovery).

arn:aws:iam::TARGET_ACCOUNT_NUMBER:role/ROLE_NAME

Use-STSRole will return an AccessKey, a SecretKey, and a SessionToken that can be used to access the account specified in the role ARN. The script uses this information to create a new credential object, which it passes to ListInstances. ListInstances uses the credential object to list EC2 instances in the account specified in the role ARN.

That’s all there is to it. Bob creates a scheduled task that executes this script each night and sends the results to the CMDB team. When the company adds additional accounts, Bob simply adds the CMDBDiscovery role to the new account and updates the account list in his script.

Summary

Cross-account roles are a valuable tool for large customers with multiple accounts. Roles provide temporary credentials that a user or application in one account can use to access resources in another account. These temporary credentials do not need to be stored or rotated, resulting in a secure and maintainable architecture.