Part 1 – Planning and Getting Started
Part 2 –  Upload File

One of the key pieces when utilizing cloud architecture, be it for a Microservice or a more traditional monolithic approach, is selecting ready made components that can alleviate the need for custom code. For serverless this need is vitally important as going the serverless approach inherently implies a heavier reliance on theses components.

For our application, we are able to upload images and place them in our blob store. For the next step we would like the following to happen:

• Upon being added to blob storage, an event is fired which triggers an Azure Function
• The Azure Function will execute and take the new image and run it through Project Oxford (Microsoft Cognitive Services) and use Computer Vision to gather information about the image
• Update the existing record in Mongo to have this data

BlobTrigger

As with our HTTP handling Azure Functions, we rely on Triggers to automatically create the link between our backend Cloud components and our Azure Functions. This is one of the huge advantages of serverless, it is to be able to easily listen for events that happen within the cloud infrastructure.

In our case, we want to invoke our Azure Function when a new image is added to our blob storage, so for this we will use the BlobTrigger. Here is an example of it in action:

[FunctionName("ImageAddedTrigger")]
public static async Task Run([BlobTrigger("images/{name}", Connection =
   "StorageConnectionString")]Stream imageStream, string name, TraceWriter log)
{
}

As with the HTTP Azure Function, we use the FunctionName attribute to specify the display name our function will use in the Azure portal.

To get this class you will need to add the WindowsAzure.Storage Nuget package, which will give you this trigger.

As with the HTTP Trigger, the “thing” that is invoking the function is passed as the first parameter, which in this case will be the stream of the newly added blob. As a note, there is a restriction on the type the first parameter can be when using the BlobTrigger. Here is a list: https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/azure-functions/functions-bindings-storage-blob.md#trigger—usage

Within the BlobTrigger the first parameter is a “route” to the new blob. So, if we dissect this, images is our container within the blob storage and the {name} is the name of the new image within the blob storage. The cool thing is, we can bind this as a parameter in the function, hence the second method parameter called name.

Finally, you will notice the named parameter here Connection. This can either be the full connection string to your blob storage which has your container (images in this case) or it can be a name in your Application Settings which represents the connection string. The later here is preferred as it allows things to be more secure and easier to deploy to different environments.

Specifying the Connection

Locally, we can use the local.settings.json file as such:

On Azure, as we said, this is something you would want to specify in your Application Settings so it can be environment specific. The Values properties are surfaced in the application.

Executing Against Cognitive Services

So, with BlobTrigger we get a reference to our new blob as it is added, now we want to do something with it. This next section is pretty standard, it involves including Cognitive Services within our application and call the AnalyzeImageAsync which will run our image data through the Computer Vision API. For reference, here is the code that I used:

log.Info(string.Format("Target Id: {0}", name));
IVisionServiceClient client = new VisionServiceClient(VisionApiKey, VisionApiUrl);
var features = new VisualFeature[] {
    VisualFeature.Adult,
    VisualFeature.Categories,
    VisualFeature.Color,
    VisualFeature.ImageType,
    VisualFeature.Tags
};

var result = await client.AnalyzeImageAsync(imageStream, features);
log.Info("Analysis Complete");

var image = await DataHelper.GetImage(name);
log.Info(string.Format("Image is null: {0}", image == null));
log.Info(string.Format("Image Id: {0}", image.Id));

if (image != null)
{
    // add return data to our image object<span id="mce_SELREST_start" style="overflow: hidden; line-height: 0;"></span>

    if (!(await DataHelper.UpdateImage(image.Id, image)))
    {
        log.Error(string.Format("Failed to Analyze Image: {0}", name));
    }
    else
    {
        log.Info("Update Complete");
    }
}

I am not going to go into how to get the API key, just use this link: https://azure.microsoft.com/en-us/try/cognitive-services/

To get access to these classes for Computer Vision you will need to add the Nuget package Microsoft.ProjectOxford.Vision.

Limitations of the BlobTrigger

So, BlobTrigger is not a real time operation. As stated by Microsoft on their GitHub:

NOTE: The WebJobs SDK scans log files to watch for new or changed blobs. This process is not real-time; a function might not get triggered until several minutes or longer after the blob is created. In addition, storage logs are created on a “best efforts” basis; there is no guarantee that all events will be captured. Under some conditions, logs might be missed. If the speed and reliability limitations of blob triggers are not acceptable for your application, the recommended method is to create a queue message when you create the blob, and use the QueueTrigger attribute instead of the BlobTrigger attribute on the function that processes the blob.

https://github.com/Azure/azure-webjobs-sdk/wiki/Blobs#-how-to-trigger-a-function-when-a-blob-is-created-or-updated

What this means is, you have to be careful with using BlobTrigger because if you have a lot activity you might not get a quick enough response, so the recommendation here is to use QueueTrigger. Queue storage is a fine solution but, I am a huge fan of ServiceBus which also supports queues. So, instead of diving into QueueTrigger I want to talk about ServiceBusTrigger which I think is a better solution.

Create the ServiceBus Queue

First we need to create the queue we will be listening to, to do that we need to go back to the portal and click Add, search for Service Bus.

You can take all of the defaults with the create options.

ServiceBus is essentially Microsoft’s version of SNS and SQS (if you are familiar with AWS), but it essentially supports all forms of Pub/Sub, absolutely vital in Microservice so the various services can communicate as state changes occur.

At the top of the screen we can select to Add a Queue. Give the queue a name (any name is fine), just something you will be referencing a bit later.

Once the queue finishes deploying you can access it and select the Shared access policies. Here you can create the policy that permits access to the queue. I generally have a sender and a listener policy. No matter how you do it, you need to make sure you have something that has the rights to read from the queue and write to it.

Once you have created the policy you can select it to get the Connection String; you will need this later so dont navigate away. Ok, lets code.

ServiceBusTrigger

The ServiceBusTrigger is not in the standard SDK Nuget package as the BlobTrigger and HttpTrigger are, for this you will need the Microsoft.Azure.WebJobs.ServiceBus. Now, as we did with the BlobTrigger we need to ensure we can specify the connection string we want the trigger to monitor.

https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus#trigger—attributes

We can do this, similar to above, by specifying the Connection string in our local.settings.json file, just remember to specify the same value in your Azure Function Application Settings. Here is an example of our backend trigger updated to use ServiceBusTrigger.

As you can see, its roughly the same (apologies for using the image, WordPress failed to parse the code properly). The first parameter is the name of the queue we are listening to and is accessible by the given Connection string.

There is one thing I want to point before we move on, it has to do with Cognitive Services. I dont know whether its a bug or not but, when you are dealing ServiceBus your queue entries are simple messages or primitives. In this case, I am writing the name of the new image to the queue and this trigger will read that name and then download the appropriate blob from Storage.

For whatever reason, this doesnt work as you expect. Let me show you what I ended up with:

var url = string.Format("https://<blobname>.blob.core.windows.net/images/{0}", name);
var result = await client.AnalyzeImageAsync(url, features);

I actually wanted to read the byte data out of the blob storage based on the name, then I figured I would be able to pass that data into a MemoryStream and pass the stream to the AnalyzeImage and that would be the end of it. Not so, it crashes with no error message when passed. So, I noticed I can also pass a Url to AnalyzeImage so I just create the direct Url to my blob. Granted if you are wanting to keep the images private this doesnt work as well. Just something to note if you decide to copy this example.

The rest of the code here is the same as above where we read back the result and then update the entry inside Mongo.

Writing to the Queue

The final change that has to be made is when the user uploads an image we want to write our message into the queue, in addition to saving the image to Blob storage. This code is very easy and requires the WindowsAzure.ServiceBus Nuget package.

public static async Task<bool> AddToQueue(string imageName)
{
    var client = QueueClient.CreateFromConnectionString(SenderConnectionString);
    await client.SendAsync(new BrokeredMessage(imageName));

    return true;
}

Pretty straightforward. I am simply sending over the name of the image that was added, remember the name is the ObjectId that was returned from the Mongo create operation.

QueueTrigger

I didnt cover it here but there is such a thing as QueueStorage which is effectively a queue using our Storage account. This works similar to the ServiceBus but, as I said above, I really view this as a legacy piece and I think ServiceBus is the future. Nevertheless, it remains an option when dealing with a scenario where BlobTrigger does not work fast enough

Conclusion

Ok, so if everything is working you have a system that can take an uploaded image and send it off for processing to Cognitive Services. This is what we call “deferred processing” and is very common in high volume systems; systems where there is just not the ability to process things in real time. This “deferred processing” model is in widespread use at places like Facebook, Twitter, and others though much more complicated than our example. This even underpins popular design patterns like CQRS (Command Query Responsibility Segregation).

In short, the reason I like this model and, ultimately, Azure Functions (or the serverless model more specifically) is it allow us to take advantage of what is already there and not have to write things ourselves. We could write the pieces of this architecture that monitor and process but why? Microsoft and Amazon have already done so and support level of scalability that we likely cannot anticipate.

In our next section, we will create the GetAll endpoint and start talking about the API layer using Azure API Management.

See Part 1: Getting Started

For the first part of this application we are going to create an Azure Function which allows upload of a file. As this file is uploaded its binary data will be stored in Azure Blob Storage while other information will be stored in a MongoDB document. The later will also receive the Cognitive Service Analysis Result data.

I like to look at each project as a “microservice”, that is related functions which together fulfill a logic unit of functionality. Later, we will discuss how to connect these together using Azure API Management.

Before we perform any code, let us think back to our document and the overall flow we are going for. We know that the user will, via HTTP, upload a file which we want to save in Azure Storage and create a record for in Mongo.

Preparing Our Components

Logging into the Azure Portal, we will want to, optionally, create a Resource Group to hold everything in; this is my convention as I find it makes finding things easier as well as cleaning up when you need to tear things down.

I wont walk you through how to do these things but here is a high level summary of what you will need:

• A Storage Account with a single container – I called my “images”
• A CosmosDB using the Mongo API. Really you could use whatever API you want but, the rest of the examples will be using MongoDB.Driver for the Nuget package

For the CosmosDB you dont actually have to do anything more than create it as the MongoDB.Driver code will actually handle creating the database and collection for you.

What you will want to do is copy and hold onto the connection strings for both of these resources:

• For Storage: Access the newly created Storage Account. In the left navigation bar select AccessKeys, the Connection String for the Storage Account is located here
• For Cosmos: Access the newly created Cosmos Database. In the left hand navigation, select Connection String. You will want the Primary Connection String

Let’s write the code

So far I have tried creating Azure Functions with both Visual Studio and Visual Studio Code and for this case I have found Visual Studio to be the better tool, especially with the Azure Functions and Web Jobs Tools extension (download). This will give you access to the Azure Functions project type.

If we use Create New Project in Visual Studio, so long as you have the above update you should be able to make this selection:

To help out, the extension provides an interface to select from the most common Azure Function types; by no means is this exhaustive but its a great starting point. Here is what each type is:

• Empty – An empty project allowing you to create things from scratch
• Http Trigger – An Azure Function that responds to a given path (we will be using this)
• Queue Trigger – An Azure Function that monitors a Service Bus Queue
• Timer Trigger – An Azure Function that fires every so often

For this we are going to use an HTTP Trigger since the function we are creating will service HTTP requests. For Access Rights just choose Anonymous, I will not be covering how authentication works with Azure Functions in this series.

Here is what the starting template for HTTP will look like:

public static class Function1
{
    [FunctionName("Function1")]
    public static async Task Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)]HttpRequestMessage req, TraceWriter log)
    {
        log.Info("C# HTTP trigger function processed a request.");

        // parse query parameter
        string name = req.GetQueryNameValuePairs()
            .FirstOrDefault(q => string.Compare(q.Key, "name", true) == 0)
            .Value;

        // Get request body
        dynamic data = await req.Content.ReadAsAsync();

        // Set name to query string or body data
        name = name ?? data?.name;

        return name == null
            ? req.CreateResponse(HttpStatusCode.BadRequest, "Please pass a name on the query string or in the request body")
            : req.CreateResponse(HttpStatusCode.OK, "Hello " + name);
    }
}

The notable aspects of this code are:

• FunctionName – This should parallel the name of the file (doesnt have to), but this is the name you will see when the function is listed in Azure
• HttpTrigger – This is an attribute that informs the underlying Azure workings to map the first parameter to the incoming HttpRequestMessage. You can provide additional helpers to extract data out of the Url as well as provide for Route matching, the same as in WebAPI.

Let’s talk a bit more about HttpTrigger. You will see this sort of pattern through Azure Functions. These “bindings” allow us take the incoming “thing” and cast it to what we need. Depending on the type of binding (HttpTrigger in this case) you can bind it to various things.

Additionally, the HttpTrigger supports verb filtering so the above would only allow POST and GET calls. The named parameter Route allows you to specify the route that the Azure Function will look for to handle; these are the same concepts from WebAPI. This will also be used with BlobTrigger later on.

Finally, there are no limitations around return result. Azure will only look for a method called Run, you can return whatever you want. The default is an HttpResponseMessage but I have used IList and other complex and primitive types.

Here is the code for upload, you will note that I am also using HttpResponseMessage.

[FunctionName("UploadImage")]
public static async Task Run([HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "upload")]HttpRequestMessage req, TraceWriter log)
{
    var provider = new MultipartMemoryStreamProvider();
    await req.Content.ReadAsMultipartAsync(provider);
    var file = provider.Contents.First();
    var fileInfo = file.Headers.ContentDisposition;
    var fileData = await file.ReadAsByteArrayAsync();

    var newImage = new Image()
    {
        FileName = fileInfo.FileName,
        Size = fileData.LongLength,
        Status = ImageStatus.Processing
    };

    var imageName = await DataHelper.CreateImageRecord(newImage);
    if (!(await StorageHelper.SaveToBlobStorage(imageName, fileData)))
        return new HttpResponseMessage(HttpStatusCode.InternalServerError);

    return new HttpResponseMessage(HttpStatusCode.Created)
    {
       Content = new StringContent(imageName) };
    };

This code is pretty straightforward though, I admit, learning how to read the image data without using the Drawing API was challenging.

Basically, we are reading the contents of the response message as multipart/form-data. We can some basic information about the image from the processed form-data. We then use our helper method to create a record in Mongo thereby generating the unique ID identifying the document. We then use this unique ID as the name for the image in blob storage.

Here is the code for creating the Mongo Document:

public static async Task CreateImageRecord(Image image)
{
    var settings = MongoClientSettings.FromUrl(new MongoUrl(MongoConnectionString));
    var mongoClient = new MongoClient(settings);
    var database = mongoClient.GetDatabase("imageProcessor");
    var collection = database.GetCollection("images");
    await collection.InsertOneAsync(image);

    return image.Id;
}

And the code for adding the image to Azure Storage:

public static async Task SaveToBlobStorage(string blobName, byte[] data)
{
    CloudStorageAccount storageAccount = CloudStorageAccount.Parse(StorageConnectionString);
    CloudBlobClient client = storageAccount.CreateCloudBlobClient();
    CloudBlobContainer container = client.GetContainerReference("images");

    var blob = container.GetBlockBlobReference(blobName);
    await blob.UploadFromByteArrayAsync(data, 0, data.Length);

    return true;
}

To complete this code the following Nuget packages will need to be installed

• MongoDB.Driver (latest)
• Newtonsoft.Json (v9.0.1)
• WindowsAzure.Storage (latest)

I had to use a different version of Newtonsoft because of differences when adding it to a .NET Standard Class Library.

Finishing Up

So, in this walkthrough we created our first service with our first endpoint. Now, we have a way to upload files to our storage. Our next step will be to write the code which, upon adding the file to blob storage, kicks off another Azure Function which runs the image data through Microsoft Cognitive Services Computer Vision API. We want to collect that data and update our existing Image record with the findings and then show that to the user. We still have a ways to go, but we made good strides.

Reference: https://drive.google.com/open?id=1lVo_woIZAAiiGyDECKvuKL97SfibwIja – this is the Image.cs class file I created with supports both serialization to and from Mongo via Bson and via the web using Json.

Today I will conclude this series as we dive into the final bit of our example and feature the setup of one of my favorite ways to structure my data access layer.

Part 3: Redux Observables

One of the great things about Redux is how well it organizes your state and how easy it can be to follow state changes. The uni-directional flow works well for this, but where it falls flat is when it comes to asynchronous operations, namely calls to get and manipulate data from a backend server.

This is not a new problem. Sagas, Thunks, and other approaches have been made to solve this problem. I cannot say that Redux Observables are the best, but certainly I have finding more and more uses for Reactive approaches in the code I am writing so, I welcome the ability to use RxJS in Redux.

The first step in this process is to update Redux to support this approach. See, Redux expects an actual object to be returned from a reducer which is fine for state changes, but not realistic for async operations. We need to change the middleware to support other types being returned; for this we need custom middleware, which we get from the redux-observable NPM package.

Most of the setup work for the custom middleware happens in the configureStore method which we created during the Redux portion of this series. Here is a shot of the updated configureStore method

It is not at all unusual for this method, as your application grows in size and complexity to gain complexity as well. In this case, we have brought in the applyMiddleware method from redux.

We use this new method to apply the result of createEpicMiddleware which comes from the redux-observable package. The parameter to this call is a listing of our “epics”.

An epic is a new concept that redux-observable introduces. For reference, here is a look at the Redux flow with these Epics included.

I like to think of epics as “Observable aware Reducers” mainly because they sit at the same level and have a similar flow. That being said, I do not look at epics as devices for updating state in most cases, instead I look at them as more specialized aspects of the system. Here is an example of the epic I use to get a list of Todo items from my sample application:

What is happening here is actually straightforward, however the methods of RxJS can make things a bbit hard to understand at first. Essentially, our call above which passed in rootEpic allowed Redux to pass emitted actions into our Epics. You recall that, in Redux, every action is passed to every reducer; which is why every reducer must have an exist case. Using combineReducers we can mash all of these reducers into one giant one. Similarly the call above with rootEpic is doing the same thing.

Unlike Reducers however, Epics do not need to have an exit case defined. They can safely ignore an action if it does not pertain to it. In this case, we use switchMap to ensure the any pre-existing occurrences of the operation are cancelled to make way for the new. Full docs: https://www.learnrxjs.io/operators/transformation/switchmap.html

The rule here is that we always return the core object of RxJS: Observable. Observabbles are, in many ways, similar to Promises. However, one major difference is that Observables can be thought of as being alive and always listening where Promises exist for their operation alone. This difference enables Observables to very easily carry out advanced scenarios without adding a lot of of extra work.

For the above, if fetchItems was called more than once, only one call would ever be in play. This is important because, the Observable returned once the call does complete sends off an action to a Reducer to add the fetched items into state. As a general rule, on our teams we do not use Epics to carry out changes to state, though it is possible we find that having this separation makes things a bit easier.

To call into an epic, you simply raise the action as you would normally via the dispatcher.

Here we call loadItems in componentWillMount (essentially when the App loads). This will raise the FETCH action that caused things to happen.

A more advanced scenario

Ok, so now you have the general idea, let’s look at something a bit more complex: forkJoin (https://www.learnrxjs.io/operators/combination/forkjoin.html).

In our example, we allow the user to create new Todo items and update existing ones. When the user is ready they can hit our sync button which saves all of the changed data to the server. This is an obvious scenario where “we want to do a bunch of discrete things and then, when they are all done, we want to do a finishing action”. This sort of thing before Promises was absolutely brutal.

Since we are using Obbservables we can do this without Promises but we will use a similar structure. For us, forkJoin is analogous to Promise.all.

In this code we do some very basic filtering to find new and existing items which have changed. We want to call two separate endpoints for these two things. Another strategy would have been to send everything up and let the server figure it out; but that is less fun. And this is even easier to do in C#.

The important thing to understand is that our methods createItem and updateItem both return observables (they update the local state to reset dirty tracking flags and, for new items, hydrate the Id field to override the temp Id given).

Here we use mergeMap (https://www.learnrxjs.io/operators/transformation/mergemap.html) to allow the inner Observables to complete and update their state as that action is not important to the action of indicating the sync is complete. For reference, here is the code for createItem.

You can see that we use map (https://www.learnrxjs.io/operators/transformation/map.html) here which is crucial so the observable that is returned can work with forkJoin, we dont want to wait for any internal completion at this level.

So what will happen is when post is called, it will return an Observable and that is immediately returned (along with all others). Internally, when the call does complete it will return our action result; map will then wrap this in an observable.

Ok, so this inner observable will be striped out of the outer by mergeMap (along with all others) and will be added to an array of Observables within another one using concat, in addition to two others (syncComplete and snackbarItemUpdate).

So that is crazy complicated. Try to remember that the parameter passed into mergeMap is the array of completed observables (completed in the sense that the web call finished) which contain state changes that need to be applied in addition to actions which hide a busy indicator and show a snackbar.

This is all compressed into a single observable (via concat) and returned to the middleware. The middleware will then dispatch each internal (which it expects to resolve to an object) action. This will then be checked by other epics and your set of reducers. In our case, the actions will perform state changes before finally signalling to dismiss the busy indicator and show our snackbar.

I realize that my explanation there was probably very hard to follow, also I am no RxJS expert. However, it does enable some very cool scenarios out of the box, and I like it because I believe there many advantages offered over Promises.

Let’s bring it home

So that concludes the series. I am actually giving a presentation based on this material, most recently at Codemash in Sandusky. I really do believe that Observables offers some solid advantages over what I have seen of Thunks and Sagas bbut, as always, evaluate your needs first and foremost.

Here is the code for the full app used throughout this demo: https://gitlab.com/xximjasonxx/todo-app

Examine the various remote branches to see starting points that you can use to see how well you understand the setup for the various parts.

Cheers.

One of the great things I love about working at West Monroe is the spirit of mentorship and camaraderie that are central to our culture. While there are numerous examples, perhaps my up and coming favorite is our New Hire onboarding process. While we have, for many years, made it a priority to put new hires through our Consulting 101 class so they gain an understanding of the world of consulting, we took it to a whole new level last year when, as opposed to hiring an outside firm, we asked internal leaders to develop curriculum to onboard new hires in the various technologies and methodologies that are in use at West Monroe.

Last year, the inaugural year,  I helped lead the mobile portion of this training. Our main focus was iOS with Xamarin as this is where we see most of our client work. The result was impressive, two of those in the class were able to quickly roll on to a crucial Xamarin project and ultimately contributed to a rousing success (one is now leading his own project while the Senior is traveling in China, while the other has become our leading expert on Android).

As I have moved away from mobile to focus on my roots of web and backend development I was asked to put together a new curriculum this year, one focused on cloud computing. This is due to the immense success of the previous year which has seen the time for this training increase to 10 full working days – Cloud will have its own 4hr block.

This desire to mentor and cultivate our young developers into the future leaders is part of what makes working at West Monroe so rewarding. Now starting my 4th year at the firm I have taken great pride in seeing so many of the young consultants I have worked with and mentor now taking leading roles on their projects and continuing to improve.