Connecting to RDS SQL Server from a .NET 5 Application on Linux

AWS Directory Services allows you to join AWS resources to Microsoft Active Directory. This includes Amazon Relational Database Service (RDS), Amazon FSx, Amazon Workspaces, Amazon Appstream 2.0, Amazon Connect, Amazon QuickSight, Amazon WorkDocs, Amazon WorkMail, and of course Amazon Elastic Compute Cloud (EC2) Windows instances. In addition, AWS recently announced the ability to Seamlessly Domain Join Linux EC2 Instances.

As I modernize .NET applications by moving to .NET 5 and Linux, I can continue to leverage Active Directory for credential management. Seamless domain join for EC2 Linux greatly simplifies the undifferentiated heavy lifting of configuring these architectures. This post will explore how to connect to RDS for SQL Server from a .NET 5 application running on EC2 Linux, using domain credentials.

Posts

Building Linux Docker Containers on EC2 Windows

In the post, I will show you how to build a Linux container in Visual Studio running on a EC2 Windows Instance.

The AWS Toolkit for Visual Studio allows you to deploy your project to Elastic Container Service (ECS) Fargate and recently as a container image to AWS Lambda among other options. In both of these cases, you must build a Linux container from Visual Studio or the dotnet command line. If you are working in Visual Studio on Windows, Docker Desktop uses a Linux container running in Hyper-V to build the container. Unfortunately, Hyper-V does not run on EC2 Instances (though it will run on EC2 bare metal).

Posts

Bulk Importing EC2 Instances

I have been testing a a preview of a new PowerShell command, Import-EC2Instance, that will be added to the AWS PowerShell API next week. The new command allows you to import a VM from VMware or Hyper-V. I covered this in my book, but at the time the functionality was not available in PowerShell and I had to use the Java API.

While the new command will upload and convert your VM, you can also do the upload and convert independently. This left me wondering if I could use the AWS Import/Export Service to ship a an external drive full of VMDK files and skip the upload process. After some testing, it turns out you can. Depending on the number of VMs you plan to migrate and the speed of your internet connection, this may be a great alternative.

Let me clarify that I am speaking of two similarly named services here. EC2 Import is used to convert a VMDK (or VHD) into an EC2 Instance. AWS Import allows you to ship large amounts of data using removable media.

Normally, the EC2 Import process works like this. First, the PowerShell module breaks up the VMDK into 10MB chucks and uploads it to an S3 bucket. Next, it generates a manifest file that describes how to put the pieces back together, and uploads that to S3. Then, it calls the ec2-import-instance REST API passing a reference to the manifest. Finally, the import service uses the Manifest to reassemble the VMDK file and convert it into an EC2 instance.

The large file is broken into chunks to make the upload easier and allow it recover from a connection error (retrying a part rather than the entire file.) With the AWS Import/Service there is no need to break up the file. Note that S3 supports objects up to 5TB and EC2 volumes can only be 1TB. So there is no reason not to upload the VMDK as a single file.

So, all we need to do is create the manifest file and call the E2 Import API passing a reference to the manifest file. If you have ever looked at one of these manifest files, they can look really daunting. But, with only a single part, it's actually really simple. Note that all of the URLs are pre-signed so the Import Service can access your VMDK file without granting IAM permissions to the import service.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54


$Bucket = 'MyBucket'
$VolumeKey = 'WIN2012/Volume.vmdk'
$VolumeSize = 20

#We need to know how big the file is to create the Manifest
$FileSize = (Get-S3Object -BucketName $Bucket -Key $VolumeKey).Size
$ByteRangeEnd = $FileSize - 1 #Byte Range is zero based

#Let's create a few pre-signed URLs for the import engine
$ManifestKey = "$VolumeKey.Manifest.xml"
$SelfDestructURL = Get-S3PreSignedURL -BucketName $Bucket -Key $ManifestKey -Expires (Get-Date).AddDays(7) -Protocol HTTPS -Verb GET
$HeadURL = Get-S3PreSignedURL -BucketName $Bucket -Key $VolumeKey -Expires (Get-Date).AddDays(7) -Protocol HTTPS -Verb HEAD
$GetURL = Get-S3PreSignedURL -BucketName $Bucket -Key $VolumeKey -Expires (Get-Date).AddDays(7) -Protocol HTTPS -Verb GET
$DeleteURL = Get-S3PreSignedURL -BucketName $Bucket -Key $VolumeKey -Expires (Get-Date).AddDays(7) -Protocol HTTPS -Verb DELETE

#The URLs need to be HTML encoded to write into the XML document
Add-Type -AssemblyName System.Web
$SelfDestructURL = [System.Web.HttpUtility]::HtmlEncode($SelfDestructURL)
$HeadURL = [System.Web.HttpUtility]::HtmlEncode($HeadURL)
$GetURL = [System.Web.HttpUtility]::HtmlEncode($GetURL)
$DeleteURL = [System.Web.HttpUtility]::HtmlEncode($DeleteURL)

#Create a Manifest file with one enourmous part
$Manifest = @"
<manifest>
    <version>2010-11-15</version>
    <file-format>VMDK</file-format>
    <importer>
        <name>ec2-upload-disk-image</name>
        <version>1.0.0</version>
        <release>2010-11-15</release>
    </importer>
    <self-destruct-url>$SelfDestructURL</self-destruct-url>
    <import>
        <size>$FileSize</size>
        <volume-size>20</volume-size>
        <parts count="1">
            <part index="0">
                <byte-range end="$ByteRangeEnd" start="0">
                <key>$VolumeKey</key>
                <head-url>$HeadURL</head-url>
                <get-url>$GetURL</get-url>
                <delete-url>$DeleteURL</delete-url>
            </byte-range></part>
        </parts>
    </import>
</manifest>
"@

#Upload the manifest file to S3
Write-S3Object -BucketName $Bucket -Content $Manifest -Key $ManifestKey

#Kick off an import task using the new manifest file
$Task = Import-EC2Instance -BucketName $Bucket -ManifestFileKey $ManifestKey -InstanceType m1.large -Architecture x86_64 -Platform Windows

Obviously there is room for improvement here. You could import directly to a VPC, support Linux instances, or use the Import-Ec2Volume command to import additional (non-boot) volumes. Hopefully this is good starting point.

Note that prerequisites for the EC2 Import still apply. For example, you must convert the VMDK files to an OVF before shipping.

Posts

Writing to the EC2 Console

I have been building a bunch of Windows AMIs for EC2 recently. If the instance fails to build it can be a real bear to diagnose issues. You don't have access to the console to watch what's happening. It would be great if I could log to the EC2 Console (also called the System Log on the web site) so I knew what was happening. So I hacked the EC2Config Service to see how it was writing to the console.

The EC2 Console, it turns out, is listening to Serial Port COM1. So if want to write a message to the log, all you have to do is write to COM1. Of course the EC2 Config Service already has COM1 open, so we have to close it first. Here is a quick sample.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


Stop-Service Ec2Config
$Port = New-Object System.IO.Ports.SerialPort
$Port.PortName = "COM1"
$Port.BaudRate = 0x1c200
$Port.Parity = [System.IO.Ports.Parity]::None
$Port.DataBits = 8
$Port.StopBits = [System.IO.Ports.StopBits]::One
$Port.Open()
$Port.WriteLine("This was written directly to the serial port");
$Port.Close()

You can also use a helper class that ships with EC2 Config Service called ConsoleLibrary. This implementation is thread-safe, adds the date and time, and takes care of all the serial port configuration details. Of course you still need to close the EC2 Config Service before running this code.

1
2
3
4
5
6
7
8


#Stop the EC2 Config Service
Stop-Service Ec2Config
#Ensure the log file exists or you will get an error
New-Item -ItemType File -Force -Path 'C:\Windows\system32\WindowsPowerShell\v1.0\Logs\Ec2ConfigLog.txt'
#Load the Ec2Config Library
[System.Reflection.Assembly]::LoadFrom("C:\Program Files\Amazon\Ec2ConfigService\Ec2ConfigLibrary.dll")
#Write to the Console
[ConsoleLibrary.ConsoleLibrary]::Instance().WriteToConsole("This was written using the Ec2ConfigLibrary", $true)

As you can see below, me messages appear mixed in with the standard console messages, but note that the Console is only updated during boot. If you write to the log after boot the messages will not appear until the next reboot.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


2014/07/19 18:18:50Z: AMI Origin Version: 2014.07.10
2014/07/19 18:18:50Z: AMI Origin Name: Windows_Server-2012-R2_RTM-English-64Bit-Base
2014/07/19 18:18:50Z: OS: Microsoft Windows NT 6.2.9200.0
2014/07/19 18:18:50Z: Language: en-US
2014/07/19 18:18:50Z: EC2 Agent: Ec2Config service v2.2.5.0
2014/07/19 18:18:50Z: Driver: AWS PV Network Device v7.2.0.0
2014/07/19 18:18:50Z: Driver: AWS PV Storage Host Adapter v7.2.0.0
2014/07/19 18:18:51Z: Message: Waiting for meta-data accessibility...
2014/07/19 18:18:51Z: Message: Meta-data is now available.
2014/07/19 18:18:51Z: AMI-ID: ami-9ade1df2
2014/07/19 18:18:51Z: Instance-ID: i-05132d2e
2014/07/19 18:18:51Z: Ec2SetPassword: Disabled
2014/07/19 18:18:51Z: RDPCERTIFICATE-SUBJECTNAME: WIN-JCK73T6NRFU
2014/07/19 18:18:51Z: RDPCERTIFICATE-THUMBPRINT: 68D80A1B0567E60D0BD2C6A8068D7E1D55ED3DDC
2014/07/19 18:18:53Z: Message: Windows is Ready to use
<span style="background-color: yellow;">This was written directly to the serial port
2014/07/19 19:30:54Z: This was written using the Ec2ConfigLibrary</span>
2014/07/19 19:32:57Z: AMI Origin Version: 2014.07.10
2014/07/19 19:32:57Z: AMI Origin Name: Windows_Server-2012-R2_RTM-English-64Bit-Base
2014/07/19 19:32:57Z: OS: Microsoft Windows NT 6.2.9200.0
2014/07/19 19:32:57Z: Language: en-US
2014/07/19 19:32:57Z: EC2 Agent: Ec2Config service v2.2.5.0
2014/07/19 19:32:58Z: Driver: AWS PV Network Device v7.2.0.0
2014/07/19 19:32:58Z: Driver: AWS PV Storage Host Adapter v7.2.0.0
2014/07/19 19:32:58Z: Message: Waiting for meta-data accessibility...
2014/07/19 19:32:59Z: Message: Meta-data is now available.
2014/07/19 19:32:59Z: AMI-ID: ami-9ade1df2
2014/07/19 19:32:59Z: Instance-ID: i-05132d2e
2014/07/19 19:32:59Z: Ec2SetPassword: Disabled
2014/07/19 19:32:59Z: RDPCERTIFICATE-SUBJECTNAME: WIN-JCK73T6NRFU
2014/07/19 19:32:59Z: RDPCERTIFICATE-THUMBPRINT: 68D80A1B0567E60D0BD2C6A8068D7E1D55ED3DDC
2014/07/19 19:33:00Z: Message: Windows is Ready to use