Today, I am going to introduce something I have been experimenting with over the last few weeks in my spare time. I really appreciate the benefits of serverless but, I have always felt there was so much more to it than simple Lambda functions and the like. So I wanted to explore how I could create entire systems that featured little to no code at all and still support complex functionality.

This is will be the first part in a multi-part series where I use CloudFormation to build a Thumbnail Creator / Image Analyzer app in AWS that leverages this new approach to using Serverless.

The API Gateway

One of the central pieces of this pattern, particularly for web apps is API Gateway (API Management on Azure) and its ability to integrate with backend services such as S3 and DynamoDB (Storage and Cosmos on Azure) allowing for pass through calls to these services to handle common operations.

In our example, we will proxy both S3 and DynamoDB with API Gateway to support both the reading of image data stored in DynamoDB and the storing of raw images in an S3 bucket. All created using Cloud Formation so that it can be stood up again and again as needed.

DevOps

To make deployments easier, I leverage CI/CD services via Azure DevOps, as it provides a superior experience to the CI/CD tooling offered by the AWS platform. Here I utilize the YAML based pipeline syntax for Builds and build the two Lambda functions in tandem and publish my Cloud Formation YAML template. A Release pipeline uploads my code artifacts to S3 (best place to source the Lambda binaries) and Create/Updates the stack represented by my Cloud Formation template.

Getting Started: Create our Role

In AWS, Role’s play a vital role in ensuring security for applications interacting with AWS services. Amazon recommends using Roles over credentials since it eliminates the need to keep passwords floating around and are, generally, more flexibile. Here is the starting point for the role we will use throughout:

As you can see, this Role features A LOT of policies. There is a solid case to be made that we would be better off splitting this role into smaller roles so we lessen the amount of damage that can be done if an attacker were to somehow gain access to a service with this role.

On the flip side, one of the advantages to serverless is a decreased attack surface for attackers in general. As a general rule, the less of “my” stuff in the wild, the less chance there is for an attack – I trust Amazon (and Microsoft) more with security than I do myself.

The other issue with this role definition is it is very open – for example:

• The role is given access to ALL permissions for S3, XRay, Lambda, Logs, and CloudWatch for ANY resource

This is very bad since it means anyone can use this role to look at (or access) anything. Obviously, when we build applications we want to constrain them to only the things that they care about.  As a rule, we should find ourselves rarely, if ever, using *.

Be that as it may, I am starting this way to remove permission concerns from our plate as we develop this application. Towards the end, we will come back and update this permissions to only what we need and only on the resources that are part of our application. It is called out here as a warning to not use this in a production system.

Create the buckets

For our application we will need two buckets: One to handle raw images and one to handle the generated thumbnails of those images. Here is the YAML template for our bucket creation:

This being Cloud Formation, we will of course allowing the calling process to dictate what names we should use for our buckets. Keep in mind, if you decide to run this, bucket names MUST be unique globally so, you might have to get creative.

For the most part I assume this template is pretty self-explanatory, We are creating resources named RawBucket and ThumbnailBucket each of type AWS::S3::Bucket. We use the value passed in via the RawBucketName and ThumbnailBucketName parameters. I will point out that the names really are for display purposes, the resource names are the main block for configuration and what you will reference throughout the template (RawBucket and ThumbnailBucket in this case).

Where this might get a bit fuzzy is with the NotificationConfiguration section under RawBucket. If you are not aware, AWS allows you to configure notifications from S3 buckets when certain events happen. This was really the birth place of Serverless, responding quickly to these internal events. By taking this approach, we can build very complex systems without needing to write a lot of code ourselves, we just plug services together.

S3 Buckets support a number of Notification types including Lambda, Topic, and Queue each of which has valid use cases. One limitations to keep in mind with S3 Notifications is THEY ARE DELIVERED ONCE. This means, if you want to do fan out actions, you MUST use something that can do that for you – SNS is used most often for this case (TopicConfiguration). This is what I will be using since I will want to perform Image Analysis AND Thumbnail Creation via Lambda when a new Object is PUT into the raw bucket.
Docs: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-notificationconfig.html

Looking at the above source you can see we reference the Topic (SNS) ImageUploadTopic and we send the notification ONLY for s3:ObjectCreated events.

Create the SNS Topic

Here is the YAML for the SNS Topic:

Here we define the subscriptions that will denote which services SNS will notify and how when a message is received.

We have not created these Lambda functions yet, we will do so in Part 2.

If you want to skip ahead here is the complete source I am using:
https://github.com/xximjasonxx/ThumbnailCreator/tree/release/version1

Part 2 here