Recently, I completed the execution of the New Hire Bootcamp for West Monroe with a focus on Cloud. The presentation contained elements from both Azure and AWS but my focus was primarily on AWS. The principal goal was to expose our incoming new hire class to the technologies that they will be using as they are assigned to project at West Monroe – most of them are straight out of college.

Cloud is a difficult platform as most people will attest, its main value is making the complex scenarios (like elastic scaling and event driven architectures) easier to implement and maintain; mainly by leveraging pre-built components which are designed to scale and take advantage of the existing infrastructure more so than a custom built component. Our goal within this bootcamp was, over the course of 4hrs to have them implement an event driven event processing system. It went well and I thought many of the explanations and examples can have a wider appeal.

Part 1: The Lambda

Lambda functions represent Amazon’s approach to “serverless” architectures. “serverless” is, in my view, the next evolution in hosting when we fully break away from the concept of a “server” and related plumbing and view Cloud as merely hosting code and handling all of the scaling for us. While I do not personally think we are to the point where we should abandon nginx, IIS, or Apache I do believe Lambda (and the paradigm it is a part of) opens up immense possibilities when considered in a wider cloud infrastructure.

The biggest one here is supporting event driven architectures. Where previously, you would have to write a good amount of code to support something like a CQRS implementation or queue polling now you can simply write a function to listen for an event raised within your cloud infrastructure. In the bootcamp we created a function that fired when an object was created in a specific bucket in S3.

In doing this, we are able to have our Lambda make calls to Rekognition, which is Amazon’s Machine Vision offering. We can then store the results in our DynamoDb table which holds the metadata for the image when it was initially uploaded.

The code for calling Rekognition is easy and looks like this:

const rekognition: AWS.Rekognition = new AWS.Rekognition({
    region: "us-east-1"
});

function detectLabels(bucketName: string, keyName: string): Promise<any> {
    return new Promise<any>((resolve, reject) => {
        const params = {
            Image: {
                S3Object: {
                    Bucket: bucketName,
                    Name: keyName
                }
            },
            MaxLabels: 123,
            MinConfidence: 70
        };

        rekognition.detectLabels(params, (err, data) => {
            if (err) {
                reject(err);
            }
            else {
                resolve(data);
            }
        });
    });
}

function findFaces(bucketName: string, keyName: string): Promise<any> {
    return new Promise<any>((resolve, reject) => {
        const params = {
            Image: {
                S3Object: {
                    Bucket: bucketName,
                    Name: keyName
                }
            }
        };

        rekognition.detectFaces(params, (err, data) => {
            if (err) {
                reject(err);
            }
            else {
                resolve(data);
            }
        });
    });
}

One of the major prerequisites here is the installation, locally, of the AWS CLI and the running of aws configure which allows you to add the access key information associated with your logon – it also keeps sensitive key information out of your code; use an AWS role for your Lambda to give the Lambda access to the needed resources (Dynamo and Rekognition in this case).

Once we make the call we need to update Dynamo. Because Dynamo is a document based database, we can support free style JSON and add and remove columns as needed. Here we will look up the item to see if it exists and then run an update. The update code looks like this:

const params = {
   TableName: "wmp-nhbc-bootcamp-images",
   Key: { "keyName": keyName },
   UpdateExpression: "set labels=:l, faces=:f",
   ExpressionAttributeValues: {
        ":l": resultData[LABELS_RESULT_INDEX],
        ":f": resultData[FACES_RESULT_INDEX]
   },
   ReturnValues: "NONE"
};

const client: AWS.DynamoDB.DocumentClient = new AWS.DynamoDB.DocumentClient({
    region: "us-east-1"
});
client.update(params, (err, data) => {
if (err) {
     reject(err);
}
else {
     resolve(true);
}

We are simply finding the result and updating the fields; those fields get created if they do not already exist.

What makes this so powerful is that AWS will scale the Lambda as much as needed to keep up with demand. Pricing is very cheap where the first million requests are free, with each subsequent batch of million costing $0.20 per million.

Part 2: The Beanstalk

Elastic Beanstalk is Amazon container service for deployments; not to be confused with their container repository. I say container because it allows you to upload code to a container and have it scale the cluster for you.

For this, there is no code to show but its important, as before, that your servers be deployed with a role that can access the servers they need. In this case, as this is the API it needs to access both Dynamo (to write the metadata) and S3 (to store the image). Probably the most complex part was increasing the max message size for the servers (to support the file upload). This had to be done through .ebextensions which allow you to run code as part of the container code to configure the servers. Here is what we wrote:

---
files:
  "/etc/nginx/conf.d/proxy.conf":
    mode: "000755"
    owner: root
    group: root
    content: |
      client_max_body_size 20M;

Honestly, the hardest part of this was getting gulp-zip to include the hidden folders within the archive. This ended up being the gulp task for this:

const gulp = require('gulp');
const shell = require('shelljs');
const copy = require('gulp-copy');
const archiver = require('gulp-archiver');

gulp.task('prepare', function() {
    shell.exec('rm -rf package');
});

gulp.task('archive-build', function() {
    shell.exec('tsc --outDir package --sourceMap false --module "commonjs" --target "es6"');
});

gulp.task('file-copy', function() {
    return gulp.src([
        './package.json',
        '.ebextensions/**/*.*'
    ], { dot: true })
    .pipe(copy('./package'));
});

gulp.task('create-archive-folder', [ 'prepare', 'archive-build', 'file-copy' ]);

gulp.task('archive', [ 'create-archive-folder' ], function() {
    return gulp.src('./package/**/*.*', { dot: true })
    .pipe(archiver('server.zip'))
    .pipe(gulp.dest('./'));
});

Note the dot: true, it is required to get the process to pick up the hidden files and folders. We are using TypeScript here as the transpiler. With this in place we could move on to the front end written using Angular 2.

Part 3: Finding Faces

Really, the app is fairly simple and supports the ability to upload images, view a list of the images, and drill into a specific one. One cool thing I did add was some code to draw boxes around the faces found by the detectFaces call in Rekognition. To do this, I ended up having to draw the image to a element and then draw boxes using the available commands. This logic looks like this:

@ViewChild('imageOverlay') overlay;

buildFaceBoxes(faces: any[]): void {
  let canvas = this.overlay.nativeElement;
  let context = canvas.getContext('2d');
  let source = new Image();

  source.onload = (ev) => {
  this.adjustCanvasDims(source.naturalWidth, source.naturalHeight);
    context.drawImage(source, 0, 0, source.naturalWidth, source.naturalHeight);

    const imageWidth: number = source.naturalWidth;
    const imageHeight: number = source.naturalHeight;

    for (let x: number = 0; x<faces.length; x++) {
      const face = faces[x];
      const leftX = imageWidth * face.BoundingBox.Left;
      const topY = imageHeight * face.BoundingBox.Top;
      const rightX = (imageWidth * face.BoundingBox.Left)
        + (imageWidth * face.BoundingBox.Width);
      const bottomY = (imageHeight * face.BoundingBox.Top)
        + (imageHeight * face.BoundingBox.Height);

      this.buildFaceBox(context, leftX, topY, rightX, bottomY);
    }
  };

  source.src = this.getS3Path();
}

buildFaceBox(context: CanvasRenderingContext2D, leftX: number,
  topY: number, rightX: number, bottomY: number): void {

  context.beginPath();
  context.strokeStyle = 'blue';
  context.lineWidth = 5;
  context.moveTo(leftX, topY);

  // draw box top
  context.lineTo(rightX, topY);
  context.stroke();

  // draw box right
  context.moveTo(rightX, topY)
  context.lineTo(rightX, bottomY);
  context.stroke();

  // draw box bottom
  context.moveTo(rightX, bottomY);
  context.lineTo(leftX, bottomY);
  context.stroke();

  // draw box left
  context.moveTo(leftX, bottomY);
  context.lineTo(leftX, topY);
  context.stroke();
}

Once you get it, its pretty easy. And it even works for multiple faces.

So, I am pleased that our attendees got through this as well as they did, this is not easy. It was a great learning experience for both myself and them.

My next goal is to recreate this application using Azure.