Serverless Microservice: Upload File

See Part 1: Getting Started

For the first part of this application we are going to create an Azure Function which allows upload of a file. As this file is uploaded its binary data will be stored in Azure Blob Storage while other information will be stored in a MongoDB document. The later will also receive the Cognitive Service Analysis Result data.

I like to look at each project as a “microservice”, that is related functions which together fulfill a logic unit of functionality. Later, we will discuss how to connect these together using Azure API Management.

Before we perform any code, let us think back to our document and the overall flow we are going for. We know that the user will, via HTTP, upload a file which we want to save in Azure Storage and create a record for in Mongo.

Preparing Our Components

Logging into the Azure Portal, we will want to, optionally, create a Resource Group to hold everything in; this is my convention as I find it makes finding things easier as well as cleaning up when you need to tear things down.

I wont walk you through how to do these things but here is a high level summary of what you will need:

  • A Storage Account with a single container – I called my “images”
  • A CosmosDB using the Mongo API. Really you could use whatever API you want but, the rest of the examples will be using MongoDB.Driver for the Nuget package

For the CosmosDB you dont actually have to do anything more than create it as the MongoDB.Driver code will actually handle creating the database and collection for you.

What you will want to do is copy and hold onto the connection strings for both of these resources:

  • For Storage: Access the newly created Storage Account. In the left navigation bar select AccessKeys, the Connection String for the Storage Account is located here
  • For Cosmos: Access the newly created Cosmos Database. In the left hand navigation, select Connection String. You will want the Primary Connection String

Let’s write the code

So far I have tried creating Azure Functions with both Visual Studio and Visual Studio Code and for this case I have found Visual Studio to be the better tool, especially with the Azure Functions and Web Jobs Tools extension (download). This will give you access to the Azure Functions project type.

If we use Create New Project in Visual Studio, so long as you have the above update you should be able to make this selection:

microservice2

To help out, the extension provides an interface to select from the most common Azure Function types; by no means is this exhaustive but its a great starting point. Here is what each type is:

  • Empty – An empty project allowing you to create things from scratch
  • Http Trigger – An Azure Function that responds to a given path (we will be using this)
  • Queue Trigger – An Azure Function that monitors a Service Bus Queue
  • Timer Trigger – An Azure Function that fires every so often

For this we are going to use an HTTP Trigger since the function we are creating will service HTTP requests. For Access Rights just choose Anonymous, I will not be covering how authentication works with Azure Functions in this series.

Here is what the starting template for HTTP will look like:

public static class Function1
{
    [FunctionName("Function1")]
    public static async Task Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)]HttpRequestMessage req, TraceWriter log)
    {
        log.Info("C# HTTP trigger function processed a request.");

        // parse query parameter
        string name = req.GetQueryNameValuePairs()
            .FirstOrDefault(q => string.Compare(q.Key, "name", true) == 0)
            .Value;

        // Get request body
        dynamic data = await req.Content.ReadAsAsync();

        // Set name to query string or body data
        name = name ?? data?.name;

        return name == null
            ? req.CreateResponse(HttpStatusCode.BadRequest, "Please pass a name on the query string or in the request body")
            : req.CreateResponse(HttpStatusCode.OK, "Hello " + name);
    }
}

The notable aspects of this code are:

  • FunctionName – This should parallel the name of the file (doesnt have to), but this is the name you will see when the function is listed in Azure
  • HttpTrigger – This is an attribute that informs the underlying Azure workings to map the first parameter to the incoming HttpRequestMessageYou can provide additional helpers to extract data out of the Url as well as provide for Route matching, the same as in WebAPI.

Let’s talk a bit more about HttpTrigger. You will see this sort of pattern through Azure Functions. These “bindings” allow us take the incoming “thing” and cast it to what we need. Depending on the type of binding (HttpTrigger in this case) you can bind it to various things.

Additionally, the HttpTrigger supports verb filtering so the above would only allow POST and GET calls. The named parameter Route allows you to specify the route that the Azure Function will look for to handle; these are the same concepts from WebAPI. This will also be used with BlobTrigger later on.

Finally, there are no limitations around return result. Azure will only look for a method called Run, you can return whatever you want. The default is an HttpResponseMessage but I have used IList and other complex and primitive types.

Here is the code for upload, you will note that I am also using HttpResponseMessage.

[FunctionName("UploadImage")]
public static async Task Run([HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "upload")]HttpRequestMessage req, TraceWriter log)
{
    var provider = new MultipartMemoryStreamProvider();
    await req.Content.ReadAsMultipartAsync(provider);
    var file = provider.Contents.First();
    var fileInfo = file.Headers.ContentDisposition;
    var fileData = await file.ReadAsByteArrayAsync();

    var newImage = new Image()
    {
        FileName = fileInfo.FileName,
        Size = fileData.LongLength,
        Status = ImageStatus.Processing
    };

    var imageName = await DataHelper.CreateImageRecord(newImage);
    if (!(await StorageHelper.SaveToBlobStorage(imageName, fileData)))
        return new HttpResponseMessage(HttpStatusCode.InternalServerError);

    return new HttpResponseMessage(HttpStatusCode.Created)
    {
       Content = new StringContent(imageName) };
    };

This code is pretty straightforward though, I admit, learning how to read the image data without using the Drawing API was challenging.

Basically, we are reading the contents of the response message as multipart/form-data. We can some basic information about the image from the processed form-data. We then use our helper method to create a record in Mongo thereby generating the unique ID identifying the document. We then use this unique ID as the name for the image in blob storage.

Here is the code for creating the Mongo Document:

public static async Task CreateImageRecord(Image image)
{
    var settings = MongoClientSettings.FromUrl(new MongoUrl(MongoConnectionString));
    var mongoClient = new MongoClient(settings);
    var database = mongoClient.GetDatabase("imageProcessor");
    var collection = database.GetCollection("images");
    await collection.InsertOneAsync(image);

    return image.Id;
}

And the code for adding the image to Azure Storage:

public static async Task SaveToBlobStorage(string blobName, byte[] data)
{
    CloudStorageAccount storageAccount = CloudStorageAccount.Parse(StorageConnectionString);
    CloudBlobClient client = storageAccount.CreateCloudBlobClient();
    CloudBlobContainer container = client.GetContainerReference("images");

    var blob = container.GetBlockBlobReference(blobName);
    await blob.UploadFromByteArrayAsync(data, 0, data.Length);

    return true;
}

To complete this code the following Nuget packages will need to be installed

  • MongoDB.Driver (latest)
  • Newtonsoft.Json (v9.0.1)
  • WindowsAzure.Storage (latest)

I had to use a different version of Newtonsoft because of differences when adding it to a .NET Standard Class Library.

Finishing Up

So, in this walkthrough we created our first service with our first endpoint. Now, we have a way to upload files to our storage. Our next step will be to write the code which, upon adding the file to blob storage, kicks off another Azure Function which runs the image data through Microsoft Cognitive Services Computer Vision API. We want to collect that data and update our existing Image record with the findings and then show that to the user. We still have a ways to go, but we made good strides.

Reference: https://drive.google.com/open?id=1lVo_woIZAAiiGyDECKvuKL97SfibwIja – this is the Image.cs class file I created with supports both serialization to and from Mongo via Bson and via the web using Json.

 

Advertisements

4 thoughts on “Serverless Microservice: Upload File

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s