Part 2 is here
In the previous parts we worked to setup our basic AWS infrastructure including SNS, S3, and Role. We also added the appropriate permissions for the flow of events to work unimpeded. Next, we setup a typical CI/CD flow using Azure DevOps so that as we make changes to code and infrastructure our application is changed appropriately; this fulfills the main tenants of GitOps and Infrastructure as Code (IaC). In this part, we will actually develop the code for our Lambdas and test that our various other resources are set up correctly.
Define the DynamoDB Table
In keeping with our main theme, we will deploy a DynamoDB table using Cloud Formation. Here is the YAML we will use:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| AWSTemplateFormatVersion: 2010-09-09 | |
| Description: "Creates infrastructure for Thumnbail Creator" | |
| Resources: | |
| ImageTable: | |
| Type: AWS::DynamoDB::Table | |
| Properties: | |
| AttributeDefinitions: | |
| – AttributeName: "Id" | |
| AttributeType: "S" | |
| KeySchema: | |
| – AttributeName: "Id" | |
| KeyType: HASH | |
| ProvisionedThroughput: | |
| ReadCapacityUnits: 5 | |
| WriteCapacityUnits: 5 | |
| TableName: "ImageDataTable2" |
Something to note here is, I chose to NOT use a parameter to define the name of the table. You definitely could but, that then speaks to deployment considerations since you generally do not want your table names disappearing. Also, with Dynamo the naming is localized to your Amazon account so you dont have to worry about extra-account conflicts.
What is DynamoDB?
DyanamoDB is a NoSQL database available in both normal and global variants from Amazon. It is ideal for handling use cases where data is unstructured and/or coming in a high volumes where enforcing consistency found in RDBMS databases is not a primary concern.In our case, the data that we store will be from Amazon Rekognition Label Detection which will be consistently different and thus makes sense to store in a NoSQL fashion.
The way Dynamo works is it expect SOME structure (in this case we guarantee there will ALWAYS be an Id column) provided which serves as the tables index. There is a great amount of flexibility in how primary and secondary keys are defined along with sort indexes within those key structures.
Create Thumbnail Function
Our first Lambda will respond to an image being added to our “raw” bucket and create a copy of that image with its dimensions reduced (thumbnail). Originally, when I did this I used System.Drawing but was met with a libgdiplus error; the error happens because System.Drawing is built on gdiplus which is not installed, by default, into Ubuntu Docker images. Rather than attempting to get this work I did some research and found the SixLabors imaging library that was featured at re:Invent. (link: https://sixlabors.com/projects/imagesharp)
One of the other interesting bits to this is, when Lambda receives an event from SNS the format is a bit different from when its comes from S3 directly. For that I create a mapping classset that can be used with Newtonsoft JSON.net.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| using System.Collections.Generic; | |
| namespace AnalyzeImageFunction | |
| { | |
| public class SnsRecord | |
| { | |
| public ICollection<SnsS3Model> Records { get; set; } | |
| } | |
| public class SnsS3Model | |
| { | |
| public S3Model S3 { get; set; } | |
| } | |
| public class S3Model | |
| { | |
| public S3BucketModel Bucket { get; set; } | |
| public S3ObjectModel Object { get; set; } | |
| } | |
| public class S3BucketModel | |
| { | |
| public string Name { get; set; } | |
| } | |
| public class S3ObjectModel | |
| { | |
| public string Key { get; set; } | |
| } | |
| } | |
| // usage | |
| var snsData = JsonConvert.DeserializeObject<SnsRecord>(evnt.Records?[0].Sns.Message); | |
| var s3Data = snsData.Records.ElementAt(0).S3; |
Here is the core logic which does the resizing – outside of this it is all about reading the Stream from S3 and writing it to our Thumbnail S3 bucket.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Stream GetResizedStream(Stream stream, decimal scalingFactor, string mimeType) | |
| { | |
| using (Image<Rgba32> image = Image.Load(stream)) | |
| { | |
| var resizeOptions = new ResizeOptions | |
| { | |
| Size = new SixLabors.Primitives.Size | |
| { | |
| Width = Convert.ToInt32(image.Width * scalingFactor), | |
| Height = Convert.ToInt32(image.Height * scalingFactor) | |
| }, | |
| Mode = ResizeMode.Stretch | |
| }; | |
| image.Mutate(x => x.Resize(resizeOptions)); | |
| var memoryStream = new MemoryStream(); | |
| image.Save(memoryStream, mimeType.AsEncoder()); | |
| return memoryStream; | |
| } | |
| } |
Analyze Image Function
In addition to creating the thumbnail image we also want to run the image through Amazon Rekognition (Computer Vision) and use the Detect Labels to gather data about the image. This data will then be written to our DynamoDB table.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| public async Task ExecuteAsync(SNSEvent evnt, ILambdaContext context) | |
| { | |
| var snsData = JsonConvert.DeserializeObject<SnsRecord>(evnt.Records?[0].Sns.Message); | |
| var s3Data = snsData.Records.ElementAt(0).S3; | |
| using (var client = new AmazonRekognitionClient(RegionEndpoint.USEast1)) | |
| { | |
| var detectedLabels = await DetectLabelsAsync(client, | |
| s3Data.Bucket.Name, | |
| s3Data.Object.Key); | |
| await WriteToDynamoTableAsync(s3Data.Object.Key, detectedLabels); | |
| context.Logger.LogLine("Operation Complete"); | |
| } | |
| } | |
| async Task<IList<Label>> DetectLabelsAsync(AmazonRekognitionClient client, string bucketName, string keyName) | |
| { | |
| var labelsRequest = new DetectLabelsRequest | |
| { | |
| Image = new Image | |
| { | |
| S3Object = new S3Object() | |
| { | |
| Name = keyName, | |
| Bucket = bucketName | |
| } | |
| }, | |
| MaxLabels = 10, | |
| MinConfidence = 80f | |
| }; | |
| var response = await client.DetectLabelsAsync(labelsRequest); | |
| return response.Labels; | |
| } | |
| async Task WriteToDynamoTableAsync(string imageName, ICollection<Label> labelList) | |
| { | |
| var clientConfig = new AmazonDynamoDBConfig | |
| { | |
| RegionEndpoint = RegionEndpoint.USEast1 | |
| }; | |
| var itemDataDictionary = new Dictionary<string, AttributeValue> | |
| { | |
| { "Id", new AttributeValue { S = Guid.NewGuid().ToString() }}, | |
| { "ImageName", new AttributeValue { S = imageName } } | |
| }; | |
| foreach (var label in labelList) | |
| { | |
| itemDataDictionary.Add(label.Name, new AttributeValue { N = label.Confidence.ToString() }); | |
| } | |
| using (var client = new AmazonDynamoDBClient(clientConfig)) | |
| { | |
| var request = new PutItemRequest | |
| { | |
| TableName = "ImageDataTable2", | |
| Item = itemDataDictionary | |
| }; | |
| await client.PutItemAsync(request); | |
| } | |
| } |
In Dynamo each row is unstructured and can have a different schema – each column for that document is represented by a key on the provided ItemDataDictionary (as shown above).
As always for reference here is the complete source: https://github.com/xximjasonxx/ThumbnailCreator/tree/release/version1
Part 4 is here
Random Blurbs and Things of this Nature 
3 thoughts on “Serverless Proxy Pattern: Part 3”