Part 2 is here
In the previous parts we worked to setup our basic AWS infrastructure including SNS, S3, and Role. We also added the appropriate permissions for the flow of events to work unimpeded. Next, we setup a typical CI/CD flow using Azure DevOps so that as we make changes to code and infrastructure our application is changed appropriately; this fulfills the main tenants of GitOps and Infrastructure as Code (IaC). In this part, we will actually develop the code for our Lambdas and test that our various other resources are set up correctly.
Define the DynamoDB Table
In keeping with our main theme, we will deploy a DynamoDB table using Cloud Formation. Here is the YAML we will use:
Something to note here is, I chose to NOT use a parameter to define the name of the table. You definitely could but, that then speaks to deployment considerations since you generally do not want your table names disappearing. Also, with Dynamo the naming is localized to your Amazon account so you dont have to worry about extra-account conflicts.
What is DynamoDB?
DyanamoDB is a NoSQL database available in both normal and global variants from Amazon. It is ideal for handling use cases where data is unstructured and/or coming in a high volumes where enforcing consistency found in RDBMS databases is not a primary concern.
In our case, the data that we store will be from Amazon Rekognition Label Detection which will be consistently different and thus makes sense to store in a NoSQL fashion.
The way Dynamo works is it expect SOME structure (in this case we guarantee there will ALWAYS be an Id column) provided which serves as the tables index. There is a great amount of flexibility in how primary and secondary keys are defined along with sort indexes within those key structures.
Create Thumbnail Function
Our first Lambda will respond to an image being added to our “raw” bucket and create a copy of that image with its dimensions reduced (thumbnail). Originally, when I did this I used System.Drawing but was met with a libgdiplus error; the error happens because System.Drawing is built on gdiplus which is not installed, by default, into Ubuntu Docker images. Rather than attempting to get this work I did some research and found the SixLabors imaging library that was featured at re:Invent. (link: https://sixlabors.com/projects/imagesharp)
One of the other interesting bits to this is, when Lambda receives an event from SNS the format is a bit different from when its comes from S3 directly. For that I create a mapping classset that can be used with Newtonsoft JSON.net.
Here is the core logic which does the resizing – outside of this it is all about reading the Stream from S3 and writing it to our Thumbnail S3 bucket.
Analyze Image Function
In addition to creating the thumbnail image we also want to run the image through Amazon Rekognition (Computer Vision) and use the Detect Labels to gather data about the image. This data will then be written to our DynamoDB table.
In Dynamo each row is unstructured and can have a different schema – each column for that document is represented by a key on the provided ItemDataDictionary (as shown above).
As always for reference here is the complete source: https://github.com/xximjasonxx/ThumbnailCreator/tree/release/version1