StarCraft Unit API

It has been a long time since I last created a post here. It is not a question of desire, more of time. In the past two months, I served as Best Man at my youngest brother’s wedding on September 30 and then, two weeks later, I married my girlfriend Woo of four years. We only just got back from the honeymoon. I must say, I was thankful my many years in consulting taught me how to organize and plan; I ended up doing the lionshare of the wedding planning and, I will say, using Agile and Scrum to plan it made it a snap.

These events did force me to forgo speaking for the last 6 months of the year, mercifully in a way as West Monroe has kept me impressively busy. But now, with all of this behind me I can finally turn my attention back to speaking and community involvement. To that end, I will be returning to Codemash in January to speak on React, Redux, and Redux Observables (our team has been using this extensively in our current project).

To that end, I have been wanting to create a new source of data for my future talks and so I decided on cataloging the various StarCraft units. My hope is, in addition to serving as a data source, I might be able to use it also practice Machine Learning to calculate new build orders.

Anyway, to build this API I decided to take a new tactic and leverage Azure Functions with HTTP Triggers and the “new” Azure CosmosDB (the successor to DocumentDb). I thought I would walk through things here:

Creating the CosmosDb

Setting up the backend database was very easy. I simply searched in the Azure portal for Cosmos and followed the steps for setup. I wont get into throughput settings or anything like that as I dont see this being used that heavily.

Create the Azure Function to Create

This ended up being the hardest part, mainly because my Azure CLI tools were out of date and it caused a weird bug when running locally – the request would always come through as a GET – which sucks if you are expecting POST and looking for BODY content. Once I upgraded the problem went away. Just an FYI.

So, Visual Studio tooling has come a LONG way in this aspect, its super easy now to create these Azure functions locally, test them, and seamlessly deploy them. I recommend creating the solution after the project as a whole and using the the “projects” to partition off the various pieces of your API.

In my case, I went with StarcraftApi for my solution name and created AdminApi which will hold admin functions ( in this example we will create a unit ). You can also create class library projects to share logic between the various APIs – hint you will want to make sure these class libraries using the same .NET Standard setting as the Azure function project (AdminApi).

SolutionExplorer

I try to isolate a single Azure function for each file here so, CreateUnit for this example. The goal here is to take take the contents of the incoming BODY and insert the Json into my CosmosDb. You will remember that Cosmos is a NoSQL database so there is not defined schema you need to follow.

Ok, so if you actually look at one of these functions there is a lot to take in, especially in the method signature portion.

signature

  • FunctionName – this is for Azure and discovery – it gives the “name” for this function, since Run isnt very descriptive
  • HttpTrigger – indicates how the requested is given to the function. In this case, via a Http POST request matching the route api/Unit (case insensitive)
    • Admin here indicates that the _master key must be passed to this function to authenticate usage

Once you have these in place you can upload the code to Azure and it can be executed. You can also run it locally though, be aware, the local server does NOT seem to check for Admin creds; I think that is intentional.

Inserting Data

When you create your CosmosDb you will be given a connection string. Cosmos fronts a variety of different NoSQL Database technologies, for my example I am using Mongo, so I will have a Mongo connection string and use the Mongo .NET libraries to connect (MongoDB.Driver v2.3.0 – v2.4.x seems to have a known bug where it wont connect properly).

So, the weird thing here is, even though we have a CosmosDB we do not actually have a database. I mean, easy to create you can click +Add Collection and it will prompt you for the database at which point you can do “Create New”.

cosmos

Collections are where the data will actually live and collections live in databases. Like I said, not hard just very weird when you think about it. But it is a similar paradigm from SQL Azure, where you had to create the server first and then database; just the naming is weird here.

Full sample that I used is here: https://stackoverflow.com/questions/29327856/how-to-insert-data-into-a-mongodb-collection-using-the-c-sharp-2-0-driver

Happy Coding. Hit me in the comments if you have any questions.

Advertisements

Building an Event Driven Arch with AWS

Recently, I completed the execution of the New Hire Bootcamp for West Monroe with a focus on Cloud. The presentation contained elements from both Azure and AWS but my focus was primarily on AWS. The principal goal was to expose our incoming new hire class to the technologies that they will be using as they are assigned to project at West Monroe – most of them are straight out of college.

Cloud is a difficult platform as most people will attest, its main value is making the complex scenarios (like elastic scaling and event driven architectures) easier to implement and maintain; mainly by leveraging pre-built components which are designed to scale and take advantage of the existing infrastructure more so than a custom built component. Our goal within this bootcamp was, over the course of 4hrs to have them implement an event driven event processing system. It went well and I thought many of the explanations and examples can have a wider appeal.

Part 1: The Lambda

Lambda functions represent Amazon’s approach to “serverless” architectures. “serverless” is, in my view, the next evolution in hosting when we fully break away from the concept of a “server” and related plumbing and view Cloud as merely hosting code and handling all of the scaling for us. While I do not personally think we are to the point where we should abandon nginx, IIS, or Apache I do believe Lambda (and the paradigm it is a part of) opens up immense possibilities when considered in a wider cloud infrastructure.

The biggest one here is supporting event driven architectures. Where previously, you would have to write a good amount of code to support something like a CQRS implementation or queue polling now you can simply write a function to listen for an event raised within your cloud infrastructure. In the bootcamp we created a function that fired when an object was created in a specific bucket in S3.

In doing this, we are able to have our Lambda make calls to Rekognition, which is Amazon’s Machine Vision offering. We can then store the results in our DynamoDb table which holds the metadata for the image when it was initially uploaded.

The code for calling Rekognition is easy and looks like this:

const rekognition: AWS.Rekognition = new AWS.Rekognition({
    region: "us-east-1"
});

function detectLabels(bucketName: string, keyName: string): Promise<any> {
    return new Promise<any>((resolve, reject) => {
        const params = {
            Image: {
                S3Object: {
                    Bucket: bucketName,
                    Name: keyName
                }
            },
            MaxLabels: 123,
            MinConfidence: 70
        };

        rekognition.detectLabels(params, (err, data) => {
            if (err) {
                reject(err);
            }
            else {
                resolve(data);
            }
        });
    });
}

function findFaces(bucketName: string, keyName: string): Promise<any> {
    return new Promise<any>((resolve, reject) => {
        const params = {
            Image: {
                S3Object: {
                    Bucket: bucketName,
                    Name: keyName
                }
            }
        };

        rekognition.detectFaces(params, (err, data) => {
            if (err) {
                reject(err);
            }
            else {
                resolve(data);
            }
        });
    });
}

One of the major prerequisites here is the installation, locally, of the AWS CLI and the running of aws configure which allows you to add the access key information associated with your logon – it also keeps sensitive key information out of your code; use an AWS role for your Lambda to give the Lambda access to the needed resources (Dynamo and Rekognition in this case).

Once we make the call we need to update Dynamo. Because Dynamo is a document based database, we can support free style JSON and add and remove columns as needed. Here we will look up the item to see if it exists and then run an update. The update code looks like this:

const params = {
   TableName: "wmp-nhbc-bootcamp-images",
   Key: { "keyName": keyName },
   UpdateExpression: "set labels=:l, faces=:f",
   ExpressionAttributeValues: {
        ":l": resultData[LABELS_RESULT_INDEX],
        ":f": resultData[FACES_RESULT_INDEX]
   },
   ReturnValues: "NONE"
};

const client: AWS.DynamoDB.DocumentClient = new AWS.DynamoDB.DocumentClient({
    region: "us-east-1"
});
client.update(params, (err, data) => {
if (err) {
     reject(err);
}
else {
     resolve(true);
}

We are simply finding the result and updating the fields; those fields get created if they do not already exist.

What makes this so powerful is that AWS will scale the Lambda as much as needed to keep up with demand. Pricing is very cheap where the first million requests are free, with each subsequent batch of million costing $0.20 per million.

Part 2: The Beanstalk

Elastic Beanstalk is Amazon container service for deployments; not to be confused with their container repository. I say container because it allows you to upload code to a container and have it scale the cluster for you.

For this, there is no code to show but its important, as before, that your servers be deployed with a role that can access the servers they need. In this case, as this is the API it needs to access both Dynamo (to write the metadata) and S3 (to store the image). Probably the most complex part was increasing the max message size for the servers (to support the file upload). This had to be done through .ebextensions which allow you to run code as part of the container code to configure the servers. Here is what we wrote:

---
files:
  "/etc/nginx/conf.d/proxy.conf":
    mode: "000755"
    owner: root
    group: root
    content: |
      client_max_body_size 20M;

Honestly, the hardest part of this was getting gulp-zip to include the hidden folders within the archive. This ended up being the gulp task for this:

const gulp = require('gulp');
const shell = require('shelljs');
const copy = require('gulp-copy');
const archiver = require('gulp-archiver');

gulp.task('prepare', function() {
    shell.exec('rm -rf package');
});

gulp.task('archive-build', function() {
    shell.exec('tsc --outDir package --sourceMap false --module "commonjs" --target "es6"');
});

gulp.task('file-copy', function() {
    return gulp.src([
        './package.json',
        '.ebextensions/**/*.*'
    ], { dot: true })
    .pipe(copy('./package'));
});

gulp.task('create-archive-folder', [ 'prepare', 'archive-build', 'file-copy' ]);

gulp.task('archive', [ 'create-archive-folder' ], function() {
    return gulp.src('./package/**/*.*', { dot: true })
    .pipe(archiver('server.zip'))
    .pipe(gulp.dest('./'));
});

Note the dot: true, it is required to get the process to pick up the hidden files and folders. We are using TypeScript here as the transpiler. With this in place we could move on to the front end written using Angular 2.

Part 3: Finding Faces

Really, the app is fairly simple and supports the ability to upload images, view a list of the images, and drill into a specific one. One cool thing I did add was some code to draw boxes around the faces found by the detectFaces call in Rekognition. To do this, I ended up having to draw the image to a element and then draw boxes using the available commands. This logic looks like this:

@ViewChild('imageOverlay') overlay;

buildFaceBoxes(faces: any[]): void {
  let canvas = this.overlay.nativeElement;
  let context = canvas.getContext('2d');
  let source = new Image();

  source.onload = (ev) => {
  this.adjustCanvasDims(source.naturalWidth, source.naturalHeight);
    context.drawImage(source, 0, 0, source.naturalWidth, source.naturalHeight);

    const imageWidth: number = source.naturalWidth;
    const imageHeight: number = source.naturalHeight;

    for (let x: number = 0; x<faces.length; x++) {
      const face = faces[x];
      const leftX = imageWidth * face.BoundingBox.Left;
      const topY = imageHeight * face.BoundingBox.Top;
      const rightX = (imageWidth * face.BoundingBox.Left)
        + (imageWidth * face.BoundingBox.Width);
      const bottomY = (imageHeight * face.BoundingBox.Top)
        + (imageHeight * face.BoundingBox.Height);

      this.buildFaceBox(context, leftX, topY, rightX, bottomY);
    }
  };

  source.src = this.getS3Path();
}

buildFaceBox(context: CanvasRenderingContext2D, leftX: number,
  topY: number, rightX: number, bottomY: number): void {

  context.beginPath();
  context.strokeStyle = 'blue';
  context.lineWidth = 5;
  context.moveTo(leftX, topY);

  // draw box top
  context.lineTo(rightX, topY);
  context.stroke();

  // draw box right
  context.moveTo(rightX, topY)
  context.lineTo(rightX, bottomY);
  context.stroke();

  // draw box bottom
  context.moveTo(rightX, bottomY);
  context.lineTo(leftX, bottomY);
  context.stroke();

  // draw box left
  context.moveTo(leftX, bottomY);
  context.lineTo(leftX, topY);
  context.stroke();
}

Once you get it, its pretty easy. And it even works for multiple faces.

So, I am pleased that our attendees got through this as well as they did, this is not easy. It was a great learning experience for both myself and them.

My next goal is to recreate this application using Azure.

Cloud Bootcamp at West Monroe

One of the great things I love about working at West Monroe is the spirit of mentorship and camaraderie that are central to our culture. While there are numerous examples, perhaps my up and coming favorite is our New Hire onboarding process. While we have, for many years, made it a priority to put new hires through our Consulting 101 class so they gain an understanding of the world of consulting, we took it to a whole new level last year when, as opposed to hiring an outside firm, we asked internal leaders to develop curriculum to onboard new hires in the various technologies and methodologies that are in use at West Monroe.

 

Last year, the inaugural year,  I helped lead the mobile portion of this training. Our main focus was iOS with Xamarin as this is where we see most of our client work. The result was impressive, two of those in the class were able to quickly roll on to a crucial Xamarin project and ultimately contributed to a rousing success (one is now leading his own project while the Senior is traveling in China, while the other has become our leading expert on Android).

As I have moved away from mobile to focus on my roots of web and backend development I was asked to put together a new curriculum this year, one focused on cloud computing. This is due to the immense success of the previous year which has seen the time for this training increase to 10 full working days – Cloud will have its own 4hr block.

This desire to mentor and cultivate our young developers into the future leaders is part of what makes working at West Monroe so rewarding. Now starting my 4th year at the firm I have taken great pride in seeing so many of the young consultants I have worked with and mentor now taking leading roles on their projects and continuing to improve.

Toolbar Navigation in Xamarin Forms

It always amazes me when things that, as a framework developer, I would consider obvious are overlooked completely. Part of the promise of Forms is the ability to create a single definition and get a native feel as it gets applied to different platforms. Part of achieving this native feel is following the idiomatic standards of the platform. Today we are going to talk about one area of Forms that drives me nuts: ToolbarItems.

On iOS it is a COMMON use case that you have a ‘Cancel’ button to the left of the page title and some primary action to the right. For whatever reason, even though we are almost to 3.0 for Forms, the team has STILL not adopted this standard. Instead, we get a very cludgy and ugly system where the system puts ALL buttons to the right and, for overflow, creates a hideous toolbar beneath the title that I have never seen in a single iOS app.

Last night, while working through an app I am making I had to fight with this shortcoming and I think it makes sense to detail the approach I came to, based heavily on https://timeyoutake.it/2016/01/02/creating-a-left-toolbaritem-in-xamarin-forms/

Define the Toolbar Items

<ContentPage.ToolbarItems>
   <ToolbarItem Text="Save" Priority="0" Command="{Binding SaveCommand}" />
   <ToolbarItem Text="Cancel" Priority="1" Command="{Binding CancelCommand}" />
</ContentPage.ToolbarItems>

Here we are setting up. Priority denotes the order items are displayed on the right side. In our case, we need to use this so we can denote Order. Now, yes, I know there is an actual Order property on ToolbarItem and it would be great to use it. Sadly, we cant. Due to the design of ToolbarItem, Order overrides Priority thus, you end up with only a single menu item in the top and thus the approach we are going to use wont work. For now, assume 0 means Right, and 1 means Left – leave off Order. You cannot use it with the approach I am going to show.

The Custom Page

Forms being Forms, we are going to need a CustomRenderer to handle this special functionality. For this case I like to create a custom type to target with the renderer however, you dont have to do it; its a personal preference that I take.

<?xml version="1.0" encoding="UTF-8"?>
<controls:SubContentPage xmlns="http://xamarin.com/schemas/2014/forms"     xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"     xmlns:controls="clr-namespace:GiftList.Controls"     x:Class="GiftList.Pages.ManageListItemPage"     BackgroundColor="{StaticResource BlueBackground}"     Title="{Binding PageTitle, Mode=OneWay}">
    <AbsoluteLayout HorizontalOptions="Fill" VerticalOptions="Fill">
    </AbsoluteLayout>

    <ContentPage.ToolbarItems>
        <ToolbarItem Text="Save" Priority="0" />
	<ToolbarItem Text="Cancel" Priority="1" />
    </ContentPage.ToolbarItems>
</controls:SubContentPage>

This is more or less a marker, nothing truly special is going on, I honestly could use the actual PageRenderer and it could be done the same.

Now the magic – the custom renderer (iOS only)

Since this idiom only exists on iOS we only need to write the custom renderer for iOS, Android can continue to use the default. This is a bit complex, so we will take it in chunks. Important: this is all happening in ViewWillAppear, do not use OnElementChanged – it will not work as you expect, you have to take this action AFTER Forms has rendered your view.

var navigationItem = NavigationController.TopViewController.NavigationItem;
var leftSide = new List<UIBarButtonItem>();
var rightSide = new List<UIBarButtonItem>();
var element = (ContentPage)Element;

This is all base assignment with a lot of coming from the link posted above. As you can see I am explicitly casting the associated Element to ContentPage, highlighting that I did not have to create the custom type.

The general goal here is to look at the items after the fact and reorganize them into Left and Right sides. Here is the code that does that, for loop is recommended due to the strange indexing that Xamarin does under the hood with ToolbarItems.

for (int i = 0; i < element.ToolbarItems.Count; i++)
{
    var offset = element.ToolbarItems.Count - 1;
    var item = element.ToolbarItems[offset - i];
    if (item.Priority == 1)
    {
        UIBarButtonItem barItem = navigationItem.RightBarButtonItems[i];
	barItem.Style = UIBarButtonItemStyle.Plain;
	leftSide.Add(barItem);
    }
    else
    {
        UIBarButtonItem barItem = navigationItem.RightBarButtonItems[i];
	barItem.Style = UIBarButtonItemStyle.Done;
	rightSide.Add(barItem);
    }
}

navigationItem.SetLeftBarButtonItems(leftSide.ToArray(), false);
navigationItem.SetRightBarButtonItems(rightSide.ToArray(), false);

Put simply, if the ToolbarItem has a Priority of 1 we assume it to be secondary and thus we place it to the left of the title using the native SetLeftBarButtonItems call. Notice that these calls allow us to place multiple buttons on each side – please dont do this, Apple wants the top bar kept clean to prevent clutter and confusion. If you have more options you can use an actual Toolbar within Apple.

So that’s it. That should work and you should get items on the Left and Right sides of the Title in iOS. Be careful modifying this thing for Android – I recommend heavy use of the <OnPlatform> tag.

Rant

Why this is not part of the main platform is beyond me. The general reasoning is that Android doesnt have the concept of a left icon. Putting aside the fact that it does, surely the Forms team has the capability to tailor this on a per platform basis. Maybe in 3.0. It just frustrates me to no end because this is something that platform should do, and there is little excuse, at this point, why it does not.

Getting Started with Flow and VSCode

Normally when I do JavaScript development I leverage TypeScript, its been my goto for a few years now. But for a new project at work we are deciding to forgo Typescript and leverage pure JavaScript. As we are going to focus on using ES6 this is not a problem as much of the Typescript syntax is now available in ES6 (thanks to Babel we can transpile to ES5). But one thing that is not available is typing which sucks cause, having a type checker can really help developers avoid errors.

After much discussion we decided this is a great opportunity to play with Flow, which is a static type checker for JavaScript. The rest of this article will focus on getting flow running and VSCode validating your JS with type safety.

Let’s add Babel & Flow

Babel makes everything so nice, allows me to use the ES6 syntax in Node (v7.x has most of it but there are a few things). For Flow, it is recommended you install the following packages as dev dependencies:

babel-cli babel-preset-flow flow-remove-types

The last one can be confusing since their install docs do not call it out explicitly. In addition, you will want to add:

babel-preset-env flow-bin

Probably the trickiest thing for me was wrapping my head around flow-bin since it was not immediately clear. It basically allows you to have Flow locally in the project, which is the recommended approach over a global install.

Now, we need to update the package.json so that we can build using babel which will remove our type annotations. The command is simply:

babel src/ -d dist/

I am assuming here that all source files are in a directory called src and the transpiled final version will go into dist.

The final thing is to create the .babelrc file with the follow contents:

{
    "presets": ["flow", "env"]
}

These presets will strip out type annotations (since they are not valid JavaScript) and transpile our ES6 code into ES5 which is recognized by Node.

Integrating Flow

The absolute first thing you need to do is run the initialization script to create the default config file (.flowconfig) recognized by Flow.

yarn run flow init

You can leave this file be for the most part, wont be modifying it here.

Here is the code I wrote with flow:

/* @flow */

export default class Calc {
    static add(num1: number, num2: number): number {
        return num1 + num2;
    };

    static sub(num1: number, num2: number): number {
        return num1 - num2;
    };

    static mult(num1: number, num2: number): number {
        return num1 * num2;
    };

    static div(num1: number, num2: number): number {
        return num1 / num2;
    };
}

Its very simple and effectively creates a library of mathematical functions. Next, use this script as your index.js, I have included a couple type errors for flow to catch.

/* @flow */

import calc from './lib/calc';

console.log(calc.add("5", 6));
console.log(calc.sub(5, 6));
console.log(calc.mult(30, "20"));
console.log(calc.div(40, 20));

To run the follow command use yarn (or npm):

yarn run flow

You will need to invoke yarn (or npm) since, I assume, you used flow-bin package which means your Flow is packaged inside your project.

Against the code above, flow should pick out the two cases where a string was passed and the error is similar to what you would receive in a language like C# if you made the same error.

Add Some Scripts

The thing, as I said, about Flow code is its not JavaScript and so if you run it straight it will fail every time. The Flow preset from Babel stripes out the type annotations when building to leave valid JavaScript. Since this is the case, we should define an NPM script that we can run each time. Open package.json and add the following line to your scripts section.

“build”: “flow && babel src/ -d dist”

This command does two things:

1) it tells Flow to check the files with the @flow annotation for errors – if it finds any the process will fail.

2) it runs the code through Babel to both strip out type annotations (for all files under src/) and transpiles to ES5.

To execute this simply do

yarn build

or

npm run build

If you then examine the files under dist/ you will see a stark difference from the code you wrote; this is the power of Babel.

Let’s Configure VSCode

Ok, so at this point we can run everything from the command line, we get type checking and its all good, right? Well no. VSCode is probably going nuts because its still trying to parse the code in your .js files like its pure Javascript. We need to make it aware of Flow. Turns out, that is pretty easy.

Click the “Extensions” icon on the left hand navigation column and search for ‘Flow’. One of the items that will come up is Flow Language Support (below).

screen1

After installing you will notice the errors are still present. But why? Well, its because VSCode (correctly) assumes you are still working with pure JavaScript, we need to disable this. But because Flow is not a global part of our environment (and shouldnt be) we dont want to change settings for the editor, we want to do it for our workspace.

To do this, you need a settings.json file. Best way to create that is to indicate you want to override the default language settings.  First, click your current language from the lower right portion of the application status bar (below).

screen1.5

This will bring up a menu where you need to select to configure the language based settings for JavaScript (might also want to do this for JSX if you are using React and any other derivative JavaScript extensions).

screen2

This bring you into the settings.json file editor (it is a special file after all). From the right hand side of the top bar select ‘Workspace Settings‘.

screen3

You are now in your local settings.json file. Here is the absolute minimum that I recommend to support Flow.

screen4

flow.useNPMPackagedFlow just tells the extension to look in node_modules for the flow binary.

Basically, we are disable the internal JavaScript validator in favor of the Flow one. I do have a sidebar on this below since I believe you need to be aware of what this means. But regardless, after doing this, Visual Studio should cease seeing Flow type annotations as JavaScript errors. Congrats. That is it.

My Sidebar

I generally recommend that developers exclude .vscode from source control. However, there is an obvious problem in this case – without the settings.json file code will appear broken in VSCode for developers that dont have the right settings.

This goes one step further when everyone is using different editors (in our case we expect VSCode, Sublime, and WebStorm to be in play). Its not a huge deal really, you just need to make sure you communicate the usage of this library; I would really hate writing the type annotations just to get red squiggles all over my code, even though its valid.

So the point here is communicate and make sure that everyone can use Flow effectively. Its a great tool, but not something a single developer can use without the rest of the team, I feel, at least not easily or naturally.

Codestock and a New Focus

This past weekend Codestock was held in Knoxville, TN. I made my return after about 3yrs absent to present my Intro to iOS with Xamarin talk. The focus of this talk, much like when I gave it in Indianapolis for Indy.Code() was to highlight 3 fundamental competencies in iOS (View Controllers, Storyboards/Segues, and AutoLayout) and explain them as a Xamarin developer.

Overall, I was pleased with how the talk went and got great engagement from the audience about half of whom were mobile developers. Great end to the speaking year.

For those of you who do not know, I am getting married to the love of my life and my girlfriend of almost 4yrs later this year. As part of the planning, I decided I would not speak beyond August so, after being turned down for BeerCityCode and ThatConference I decided that Codestock will be my last public speaking engagement outside Chicago for 2017 as I focus on the wedding. But it goes beyond that.

Recently, I have decided to shift my focus at West Monroe Partners and return to my roots as a backend developer. It’s not that I dislike Xamarin or that I see mobile as a dead end, I just like to change it up every few years – a big reason why I stay in consulting. With that being the case, I look forward to returning to the speaking circuit in 2018 with a focus on AWS, Azure, Docker, Nodejs, C# and the other myraid of tools that are used on the backend.

Thank you to everyone who came to my talks this year and thank  you to the various conferences that allowed me to speak to their awesome attendees. Every time I get accepted it is always an honor.

See you in 2018

AWS DevOps with .NET – Part 1

I have found that DevOps is a crucial part of any development team as it takes a lot of the operational burden out of the hands of developers allowing them to focus on writing code. Additionally, it can make sure that certain testing and verification is done each time, every time, consistently, which improves the overall QA cycle.

We had a project at West Monroe about a year ago that had us deploying .NET code to Amazon AWS; as a side note the client maintained control and access to Amazon, we did not have access to this. During this process, we continued to use Azure as the main deployment engine. Thus, we had to perform communication with Amazon using Powershell scripts. It was rather painful.

As it happens, Amazon has their own suite of DevOps tools for their platform:

  • CodeCommit – A Git Repository for your projects
  • CodeBuild – A build engine for your code
  • CodeDeploy – Deploy your code to EC2 instances
  • CodePipeline – Provide end to end CI/CD for your application

So I decided I wanted to see how their products stacked up so moving forward I, and West Monroe developers, could use Amazon’s tools if the deployment target was going to be Amazon. For this first part, I will only be covering CodeCommit and CodeBuild because I have yet to get the CodeDeploy process to work.

Before I get started I would like to make a few things clear:

  1. CodeBuild does NOT support .NET Standard, but as .NET Core can run on Linux we can build .NET Core apps, but not full .NET apps, at least yet.
  2. Amazon requires a host of YML files be present in your application for these processes to work. While the documentation does mention them, I feel it could be a lot clearer.

So, lets get started.

Creating your Git Repository

Each developer on your team should have their own local account to log into the AWS console with appropriate rights set. If they are going to be developing code ensure they have the AWSCodeCommitPowerUser policy. Note: It is also acceptable to use the AWSCodeCommitFullAccess policy however, I think only one user should have this policy as normal developers should not need full access, just the ability to interact with the repository. Note: you will need to have full access to actually CREATE the repo, or you can use the root account.

Next, ensure each user has the IAMSelfManagedServiceSpecificCredentials and IAMReadonlyAccess policies. The former allows them to update the credentials for specific services (CodeCommit in this case) while the IAMReadonlyAccess policy enables them actually access IAM in their AWS console. Here is a screenshot of what my group CoreWebUser looks like:

codecommit_policies

I have also included AWSCodeBuildDeveloperAccess because, later on, I will want this account I am using to be able to start builds using AWS CodeBuild, but more on that later.

Each user who will be a developer needs to generate their Git credentials (that is what the two policies above enable). To do this, each user must login and go into IAM select their user from the Users table.

There is a tab on the subsequent page called Security credentials, the Git credential section is all the way at the bottom. Per Amazon policy, the password is going to be a lot of gibberish for security, make sure to save these credentials somewhere.

Let’s create the repository

Now, lets create that repository – head over to Services -> Developer Tools -> CodeCommit. Once you create the repository Amazon will prompt to cover the steps that we covered above. The important thing here is the clone URL which is your Git remote (or where you will push and pull remote code from).

And thats pretty much it. Poking around on my own SampleApp I have been pretty impressed, especially with the Visualizer which duplicates the branching that I have seen in both SourceTree and GitKraken on the web. A deeper look at this product is outside our scope. This is the easy step, next we move on to the moderately hard step of building the code.

For my code base, I used the default .NET Core web template by running this command

dotnet new web

I then made a simple modification to the hard coded response string. I then pushed this to CodeCommit on the master branch.

Got the Code, let’s build it

As of this writing, AWS does NOT support .NET builds meaning, you will NOT be able to select a Windows machine. This being the case ANY scenario involving .NET Standard is out. What we CAN do is leverage .NET Core which runs anywhere, including Docker and Linux which CodeBuild can support.

With any remote build process the first thing I always tell people to do is UNDERSTAND your own build process and make sure it works locally. Because if you dont understand or it doesnt build locally its MUCH harder to get it to build remotely.

To that end, I suggest building your local version first so you better understand this process. So let’s build it, run this command:

dotnet build

If you run this, you will notice the application does not build (or if it does you somehow ran the next step :)).  We need to run a restore to bring in the packages to support our app. Run this command then rerun the build command.

dotnet restore

The reason I had you do this was so the build sequence was understood because we have to create a YAML file to explain this to Amazon. Amazon’s tools rely heavily on these sorts of files. Create a new file called builldspec.yml. It has to be this name and it HAS to be at the root of your application, no exceptions. Here is an example of mine:

version: 0.1

phases:
  pre_build:
    commands:
      - dotnet restore
  build:
    commands:
      - dotnet build
artifacts:
  files:
    - bin/Debug/netcoreapp1.1/*
  discard-paths: yes

A full specification of this file is located here: http://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html

I cannot stress enough the importance of the format of this file. Follow the spec EXACTLY and use tabs where you see indents. The errors provided if you dont do this is very cryptic and speaks to “number of containers”. Follow the format.

You can see how why I explained the restore then build for .NET core apps. This spec file will ensure Amazon runs those those same commands. Now push this into your remote repository.

Setting up the Build

As with CodeCommit you will need to under the policies attached to the users to access this. Also, like with CodeCommit you have policies for general use, read only, and admin.

  • AWSCodeBuildAdminAccess – This policy should be given to one user. And you will need this policy to create the build project.
  • AWSCodeBuildDeveloperAccess – This policy should be given to most of your developers, unless there is a reason you want some to not be able to start builds, in which case they should either have no policy for CodeBuild or they should have the AWSCodeBuildReadOnlyAccess policy.

Once you have this set, we can  head over to Services -> Developer Tools -> CodeBuild in the AWS Console and start the real fun. First step is easy, create the project (make sure you are using either the root user or the user with AWSCodeBuildAdminAccess).

Now, let’s go section by section:

codebuild_1

This section is pretty self explanatory. Project name is required, description is not. Fill it in and move on.


codebuild_2

For this section start off my selecting AWS CodeCommit in the Source Provider drop down. You will notice there are other options as well. We wont cover them here.

Once you select the CodeCommit source AWS will display a second dropdown with the Repositories available, pick yours.

This section is complete, let’s move on.


This next section is where the fun part is. When I started this process I roughly knew what Docker was, now I have a better idea, still a bit hazy but noticing something on the .NET Core Setup page helped this along. Here is what my section looks like:

codebuild_3

The key here is the Custom Image ID which we specify as the Docker image that Microsoft maintains with the latest version of .NET Core. Remember what we said earlier, .NET Standard is NOT supported here because it MUST be run on a Window machine (Mono not withstanding). .NET Core is designed to run ANYWHERE and since Docker only really support Linux images its ideal for this situation.

The path microsoft/dotnet:latest refers to the Microsoft Docker container and the ‘dotnet’ image there. You can use Amazon ECR which is a Docker esque service that provides containerized instances, but I found this much easier 🙂

Notice the Build specification setting. We could have also specified a sort of build script using the second option but I prefer the YAML approach since not understanding would pigeon-hole you when your build process becomes complex.

Enough with this section, lets move on.


When you run a build on any platform you generate output, in the land of DevOps we refer to this output as Artifacts, this next section helps CodeBuild understand how to handle your artifacts. Here is my configuration:

codebuild_4.png

In this example, I am using S3 as the output location. You will need to create the bucket that I have listed. The one interesting point here is the Artifacts name. Now here, I gave the explicit dev.zip which is intentional as I intend to create a Zip file as the final artifact. However, this feature must be set in the Advanced Settings (shown next) and the name will be literal. That means, you need to supply the extension, even if you tell CodeBuild you want it to Zip the output.

Moving on.


Next we need to define the service account that will actually run the build. What is important here is that the role that is used (and Amazon can auto-generate it if the user has the permissions) have the rights to access the OTHER Amazon services as needed (S3, in this case, if you choose to put your artifacts there). Here is what mine looks like:

codebuild_5

Once you complete this setup, you should go into IAM and ensure this role has the ability to access S3, otherwise the build will fail.

Moving On.


I mentioned previously that you need to tell CodeBuild that it should Zip the artifacts the build process outputs. Expand Show advanced settings.

codebuild_6

The important thing here, as indicated by the arrow is the Artifacts packaging option which will tell CodeBuild to zip, thereby matching the extension we provided previously.

That’s it. Now we can start a Build


Here is the output from my build:

codebuild_7

If we go to S3 we can access our bucket and even download the Zip file and verify its contents.

Where do we go from here?

So, we have our application building and yeah, its a pain in the ass and extremely limiting that we cant use this with .NET Standard but its something and I do tend to think that .NET Core is the way of the future, especially for web apps given the flexibility it allows in deployment. My next post, hopefully sometime this week, will focus on using CodeDeploy and CodePipeline to complete the end to end process. For the moment, I am still fighting with them.

Overall, I have found Amazon to be acceptable but, the reality is if you are going to use .NET you really should have a profound reason for not using Azure. The integration with Microsoft’s tools far exceeds anything you will get with Amazon. Plus, we havent even covered the appsec.yml file which is required, I think, to use CodeDeploy. This contrasts heavily with Azure where much this is taken care of for you. Part of that is Microsoft coverage of its own language and Amazon being language agnostic more or less and not favoring one platform over the other.

I look forward to completing my understanding of this process, if for no other reason than to continue to enhance my understanding of the AWS platform and its various features.