Building an Real Time Event Processor – Part 3

See Part 2 here

Up to this point we have not really stored any data for our real time application, outside of the intrinsic storage we get with Event Hub and the temporary storage that occurs with a Service Bus queue. Now we get into storing data and alerting users to its presence so they can update.

Understanding the nature of stored data in real time data applications

It is a bit of an oxymoron but, there is no such thing as real time data. What we really mean for term is “continuously producing data within intervals” to give the appearance of real time. Another aspect to this is that our data is often “pushed” to clients so that visualizations can be appropriately updated.

In this sense, it is best to approach this as you are storing a lot of data and on top of that data you are creating aggregations which are stored and used for updates. It is not tenable for front end systems to constantly query an ever larger set of data, it simply gets slow. We need to localize our efforts and use cumulative or window based approaches.

As an example, if your organization wanted to know how much product had been sold for a day. You would keep a cumulative total using an aggregating window rather than query a larger data set for the answer. The later approach would work but would slow down with time. You still want to store the raw data as there are benefits to using large data sets to see trends but, in the moment, you only need a very small piece of the data.

Now that we understand that, let’s talk about how this application deals with its data.

Save our aggregate data

So, when the Stream Analytics function runs its query it selects the data INTO an output. In this case, the result is set to an Azure Function that creates a Document in CosmosDB with the data.

For a Function App, Stream Analytics calls the Azure function via Http thus, this is set as a HttpTrigger using the POST method and expecting the data in the body.

Since the Stream Analytics job expects a successful HttpResponse we must reserve the output for this purpose, we cannot return the new CosmosDb document. Instead, we set an Output binding which allows us to create the document as an out parameter.

We could choose to use a POCO (Plain Old C# Object) to represent what is going into the Document but, given the simplicity of this example it is not necessary.

The result here is as the function receives data from Stream Analytics it is automatically written to CosmosDB. This will also invoke the Change Feed which our next function will listen for. But, before we talk about that I want to ask the question of whether using CosmosDB for this is correct.

Choosing the right database

The short answer is kind of, but not really. I often advise clients to stay away from CosmosDB Core SQL offering due to the pricing. You have to remember that CosmosDb is aimed at high volume and multi-region scenarios. You should only use it if you have such a requirement. The one exception is table storage which is a Cosmos branded service replacing the old Azure Table Storage that predates Cosmos.

In short,  due to its price and target use cases, for most organizations, it does not make a lot of sense. I decided to use it here because it is aimed for high availability so it can take in data much faster than a traditional RDBMS – plus the data being generated is nearly guaranteed to be consistent due to what it is.

The other thing to consider, especially with Real Time applications is whether there is a need to store aggregated data from a long time ago. Remember, we do still store the raw data in Azure Blob storage (which is much cheaper than Cosmos). We have to ask the question, does knowing the aggregate for 1 week ago really provide business value? Would it be cheaper to accept the cost of querying it from our historical records. The answer here is usually that it is better to query.

The point here is, real time aggregations like this often work best with a time series database, such as Redis which can be configured to prune old data automatically. In fact, Redis would be the go to for this sort of operation in a production scenario. But, as of this writing, there is NO trigger and bindings for Redis service so, I would not have been able to stay to my goal of writing no plumbing logic for this exercise. If this were a production situation, Redis would be highly considered far more so than Cosmos.

I also think it means I need to write such a binding for other to use.

Notify Users

Modern NoSQL databases, almost without exception, expose a feed which indicates when a change occurred to the data. This is fundamental for building event streams around database which are the building block for real time applications as well as Event Driven Architectures, Event Sourced Architectures, and other distributed data systems.

You can think of this, in some ways, as the modern version of the Trigger concept from RDBMS except, rather than living in the database, these triggers are at the application level and can be used to tie components together in a decoupled fashion.

Here is our code which listens to the CosmosDB change feed and sends the new document created to users via the Azure SignalR service:

The use of the CosmosDBTrigger is what hooks into the change feed for the target database and collection where our data is going to be added. Additionally, CosmosDB requires a lease collection which tracks who is consuming the change feed for that document.

Our next parameter allow us to send documents to the Azure SignalR service for a particular Hub. I wont go into explaining SignalR and what Hubs are you can learn more about that here: https://docs.microsoft.com/en-us/aspnet/signalr/overview/getting-started/introduction-to-signalr#connections-and-hubs

Our binding looks for the AzureSignalRConnectionString to know which SignalR service to connect to. Since this is the Azure SignalR service is it imperative that our service is configured for serverless mode under Settings. Update the CORS setting for SignalR service to be * or whatever is appropriate. Not doing this will cause problems when we attempt to connect.

Above, you notice that we defined a negotiate function with a Output binding of type SignalRConnectionInfo. This is the method that will be initially contacted by our clients to get our information, it beats having to store the access key somewhere where its widely available – using negotiate the key is downloaded and used internally to the client (though with Developer Tools you can still pick it out). It is the serverless setting in the SignalR service which drives this behavior.

Our client looks like this:

Note the .withUrl call that specifies the base URL to our Azure Function. The first call made will be this URL appended with /negotiate. After that call completes, SignalR will attempt to open the socket.

Remember to configure CORS in your Azure Function app as well – its under Platform Features.

The final bit to call out is the newFirstLetterData event. SignalR works by “calling” a JavaScript function from the backend (or that is how it makes it seem). In our code we add instances of SignalRMessage to an instance of IAsyncCollector – one of the parameters TargetName is set to this value indicating what method should be called when that message lands on a client.

To ensure I did not miss anything with this step (it took me the longest to figure out) here is a tutorial that covers this topic: https://docs.microsoft.com/en-us/azure/azure-signalr/signalr-quickstart-azure-functions-javascript

Up Next

So that completes the overall pipeline technically. My earnest hope is your is working as well as mine. For our final discussion I want to discuss areas we can improve and what the key takeaways from this exercise were.

Building an Real Time Event Processor – Part 2

Part 1: Building a Real Time Event Processor – Part 1

Welcome to Part 2 of this series on building a real time pipeline using Azure Services and Azure Functions. In the first part, we talked through the services we will use and laid out our goals for this project. Chief among them was to ensure we leveraged Azure Function bindings to their fullest to eliminate the need to write boilerplate connect code for various Azure services. By doing this we allow the underlying service to handle this for us.

At the end of Part 1 we had create our Timer function which was responsible for routinely running and sending data into our pipeline by way of a Service Bus Queue. Now, we will write the code to dequeue our data and send it to Azure Blob Storage and an Azure Event Hub.

Dequeue Our Data

The data we stored in the queue takes the following format:

Remember, when messages are stored in Service Bus they are stored as UTF8 byte arrays to reduce space. The trigger will convert this to a string or if the destination type is a complex object, it will attempt to deserialize it to that type. Here is the code for our function:

In my opinion, when you see code like this you are using Azure Functions to their fullest extend. Our function consists of a loop and that is it – and truthfully we could probably use some LinqFu to remove even that.

What this method does is a few things:

  • First, the trigger for the Service Bus upconverts the byte array into a standard string. But because the destination type is not a primitive (NamesGenerationRecord) the trigger will attempt to deserialize the string into that object.
  • Next, we define an Output binding for an Event Hub which forces the destination type of IAsyncCollector<T> to capture event string that need to be set to our Event Hub. And because it would not make sense to send only one event, it is a collection which will send asynchronously as we add additional messages
  • Finally, we define the return as going to a Blob. This return will write our raw Json contents to a Blob entry with the {id} name. More on this is a second as its really neat but not well documented.

We get all of that functionality with mostly attribution. That is what I call super cool. Before we move on, let’s talk about the return of this function and how it works.

Understanding Bindings

So, what gets stored in the Service Bus is a JSON string that looks like this:

When the ServiceBusTrigger delivers this string to the function it notes that its not delivering a string but rather an instance of NamesGenerationRecord which has this definition:

Crucial here is that we create a property called Id. It is this property value that will be bound to the {id} in our return template:

[return: Blob(“raw-names/{id}.txt”, FileAccess.Write, Connection = “AzureWebJobsStorage”)]

Initially, I thought this would view the string as JSON and create a JObject from it and look for an JToken with the name id but, when I tried that I received errors around the runtime being unable to find the id field to bind to. Using a custom object solved this problem.

I found this very interesting and it gives enormous possibilities. Given the way Microsoft tends to operate its likely the many of the functions support this for both Output and Return bindings.

The end result here is for each item dequeued from our Service Bus Queue we will create a blob in the raw-names container with a name of a Guid ensuring a high degree of uniqueness.

Preparing the Event Hub

Simply storing this data is not all that interesting although this is a great way to build a Data Lake that can serve well for running queries of a historical nature. The value we want is to analyze our data in real time.  There are services that could handle this, with vary degrees of scalability and success:

  • Azure Event Hub
  • Azure Event Grid
  • Azure Service Bus
  • Azure Blob Storage

Some of these options may not make sense to you and that is fine. I said they COULD work for this scenario but, its unlikely using something like the change feed of an Azure Blob Storage would make a lot of sense but it could work.

Among these options, the BEST service, in my view, for our scenario is Azure Event Hub due to being cost effective and highly scalable with minimal latency. It also enforces ordering within partitions and can hold vast amounts of data for days at a time.

Further, it also hooks in well with Azure Stream Analytics Jobs which are vital to real time processing pipelines.

To setup and Event Hub, follow the instructions here: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create

Here is a visual of what the communication between the Azure Function and Event Hub looks like – I am assuming 5 names per queue message which translates to 5 events sent to Event Hub.

Queueing

For this visualization I created a view inside the Event Hub as messages are being sent along. In this situation EACH execution of the Dequeue Processor Function will send 5 envelopes to Event Hub. These will get processed in order as they all live within the same partition.

Real Time Aggregation with Stream Analytics Job

I cannot count how many times I have come across code bases where developers have chosen to use an Azure Function or some other code based implementation to implement aggregation. While this works it is far from ideal. The Stream Analytics Jobs are the recommended approach both from the standpoint of scale and ease of use – better to write a SQL query to perform aggregation using many machines than attempt to rely on the Azure Function runtime.

Here is an example of the query I used for this exercise:

This query executes in three phases.

First, it reads from our Input which in this case is NamesData an alias for the Event Hub which our Azure Function was pouring data into in the previous section.  This looks at each name entry and grabs the first character (base 1) from each Name in the data set. This subquery results are named FirstLetters.

Next, the subquery results are used as the source for another subquery, LetterCounts. This query takes the first data and counts how many times each letter appears in the set. Key to this query is not just the grouping logic but the TumblingWindow. Let me explain.

Remember that you are not querying against a standard database of data with these queries. Your data set is consistently moving, considerably so in high volume cases. For this reason, you MUST restrict what data your query considers using windows. Stream Analytics supports 4 of these windows:

  • Tumbling
  • Session
  • Hopping
  • Sliding

Depending on how fluid you want your data to be generated you would pick the appropriate window. They are described here: https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-window-functions.

For our example, we use Tumbling which segments the event stream into a distinct windows and ensures events do not cross into other segments. In the case of the query above, we get the events broken into 30s intervals with each event belonging to one and only one segment. There are use cases for wanting events to belong to multiple segments, as is the case with Sliding Windows.

The final bit of this query select the results of LetterCounts into the Output SaveGroupdData which fronts an Azure Function to take our aggregated data to the next stage.

Up Next

In this post, we continued to lessons of Part 1 and dequeue our raw data, parsed it and sent the raw events into our Event Hub. Within the Event Hub our data is stored (temporarily for x days) and it can be read by various processes in real time. One of the best ways to perform this reading is using Azure Stream Analytics jobs which enable SQL like querying of data in the stream using windows to execute data segmentation. The result was a sequence of writes to our output Azure Function (we will discuss in Part 3) with the aggregated data results from the query.

In the next segment we will store our Aggregate results and notify connected clients who may update their UIs display our real time data.

One Note

One of the most popular outputs for Stream Analytics jobs is Power BI. This tool can take in the data and build these auto updating charts to represent data within the stream. I was not able to use Power BI for this but it is a common output for use cases like this.

Building an Real Time Event Processor – Part 1

So, I decided recently to undertake a challenge to build an end to end real time processing pipeline using Azure Functions. They key here would be to only use Outbinding bindings, triggers, and return bindings to communicate with other Azure services. In this way, it lessens the amount of code needed and reduces overall chances for bugs.

In addition, it gives insight into potential ways of building real time data processing clients, a scenario becoming increasingly common in business. Further, they are a challenge and require intense communication and collaboration with various persons to ensure the data is partitioned and chunked appropriately. Throw in the cloud dimension where our setup needs to be cost effective as well as functional and its quite the challenge.

Let’s get started.

Understanding Bindings

One of the most useful feature in Azure Functions is bindings whereby we can encapsulate functionality that would normally necessitate boilerplate code for a simple attribute on a parameter. This allows the function to communicate with many things and the management of that communication to be handled by Azure.

Perhaps the most popular and well known binding is HttpTrigger whereby a Azure Function is triggered when a Http request is made that matches a specific pattern. This trigger outputs itself as HttpRequest instance which contains all of the information about the incoming request. Other triggers function in similar ways – I even have a post about writing a custom binding.

Regardless, when using Azure Functions you should aim to heavily use bindings as much as possible. It will save you from writing bad or incorrect service connection code.

Enough of that, let’s talk shop.

The Architecture of our Pipeline

Selection of services is critical when designing a real time pipeline, each case will be different and have different limitations. Recently, I helped a client understand the difference between Azure IoT Hubs and Event Hubs and what scenario is geared for. In that case explaining that IoT hubs allow for an enormous amount of concurrent connections (sensible given the use case) where Event Hubs allow a fair number but much less than IoT. This limitation often informs other architecture decisions you would make.

For our sample, I choose the following services:

  • Azure Functions – should not come as a surprise. They are the backbone and connectors of the event pipeline. As a rule of thumb, which building an Azure Function be mindful of its responsibilities. They are not designed to be large and there are certainly better services for performing heavy amounts of processing
  • Event Hub – Event Hubs are designed to handle large amounts of data and maintain order within partitions and minimal latency. It is able to send its incoming data blocks to multiple services for processing. If Capture is enabled, Event Hub can store each event it receives in an associated blob storage. The blob storage could serve as a simple Data Lake over time
  • Stream Analytics Job – This service enables queries to be run over the data in Event Hub. It is commonly used to aggregate windows of data and send results to other services for storage and further processing. These windows exists because, in a real time application, the data set is never fixed.
  • CosmosDb (Core) – CosmosDB is Microsoft’s global NoSQL database solution used to support various providers for global and multi-region scenarios. Due to it being NoSQL it favors the AP portion of the CAP theorem (https://en.wikipedia.org/wiki/CAP_theorem) and this is well suited to a situation where it will receive data frequently and in a controlled manner where consistency is not a major concern
  • Azure SignalR Service – SignalR enables transmission of new aggregate data sets from Stream Analytics to connected users. This could power charts or graphs in real time to allow for more visibility of per minute operations

Now that we know the services here is a visual of how they are connected:

EventFlow

A few things of note that we take advantage of in this pipeline:

  • CosmosDb supports change feeds which an Azure Function can monitor and trigger itself when a change is detected. This enables the so called streaming architecture that underpins this sort of pipeline. It lets the entire system respond when something in data changes
  • At the Dequeue stage we use a combination of return and output binding to send our data to two places. Further, we use a binding to parameterize out Blob return so we can set the name of the blob created (we will explore in code later in this series). The use of Output bindings with Azure functions enables the function to initiate more than one service as an output OR, in the case of Event Hub, notify the service multiple times through the course of one iteratiom

Now that we understand our flow, let’s look at our first Azure Function – a Timer Trigger.

Phase 1: Generate our Data

Data plays a huge role in a real time processing pipeline. In my smaller example I have service which generates random Western style names. My goal is to capture these names and send them to the queue. The size of this data can vary since I may get back 4 names or 10. Here is the code for the function:

This code uses a ServiceBus return string – this will cause whatever string we return from this function to be added to a Service Bus queue named newnames-queue in its raw JSON form (this is why we use the Linq to Json sytle from Json.Net library). The binding will take care of encoding the string into a byte array to minimize transport and storage size while it resides in the queue.

In the attribution for this function, you can see we specify the queue name as well as the connection string (without the Entity Path) to the Service Bus. The Configuration setting ServiceBusConnection is set for the Azure Function app and the attribute will handle resolving it.

Up next

This brings us to the end of Part 1 of this series. Our next part will show how we dequeue this message in a way that maintain strong typing and send it to both blob storage and Event Hub in one fell swoop.

Part 2

Part 3

 

Getting Started with Kafka and .NET Core on Kubernetes

Real time streaming is at the hard of  many modern business critical systems. The ability for data to be constantly streamed can give organizations new insights into their data and the real time nature means these trends can be seen in real time, creating possible value streams for organizations.

It being so popular there are many options available from cloud hosted to self hosted. One of the big ones is Apache Kafka which enables this sort of “pubsub on steroids” that enables it to scale its data injest and streaming capabilities to fit the needs of almost any organization. In this post, I want to walk through a basic setup of Apache Kafka using Bitnami’s Helm Chart.

Prepare your cluster

Truth be told, I did this using an Azure Kubernetes Service cluster that I spin up on occasion that has three large VMs backing it. I have found using things like Kubernetes for Docker, minikube, and others that you run into resource limitations that make it hard to deploy. For that reason, either give your local cluster an immense amount of resources or using a cloud managed one. I recommend Azure simply because by default the AKS cluster is backed by a scale set that you can Deallocate as needed – saves immensely on cost

Bitnami Helm Chart

I love Helm because it enables quick deployment of supporting software that would otherwise take a lot of reading and learning and tweaking. Instead, you can use Helm to execute this: https://github.com/bitnami/charts/tree/master/bitnami/kafka

The instructions above are actually written to target Helm2, latest is Helm3. For the most part its similar enough, though I love the removal of Tiller in Helm3. Syntax of the command is a little different – here is what I used:

helm install kafka-release –namespace kube-kafka bitnami/kafka

This creates a release (an instance of deployment in Helm terminology) called kafka-release places the deployed components in a Kubernetes namespace called kube-kafka and deploys resources based on the bitnami/kafka Helm chart – if you look at the above link there are many ways to override how things are deployed via this chart

After running the command, Helm will start deploying resources which then have to spin up. In addition, it will layout some instructions for how you can play with the Kafka cluster on your own, using a temporary pod. But before you can do this, the deployment must enter the Ready state. Best to run the following command to know when things are ready:

kubectl get pods -n kube-kafka –watch

This will watch the Pods (which are the things that have to spin up due to container creation) and when both are ready you can safely assume Kafka is ready.

Kafka oversimplified

So, I am still learning the architecture internally of Kafka. I know that it is basically a message broker system that enables a multicast eventing to various topics that can have multiple subscribers. But I honestly need to do more to learn its internals – suffice to say for this exercise, you send a message to a topic in Kafka and all listeners for that topic will receive the message.

The term Producer and Consumer will be used throughout the remainder of this post. Producer sends data to the cluster nodes, and Consumers receive that data.

Create the Producer

Our producer will be rudimentary and over simplified but just to get the idea of the sort of structure these types of applications take. Remember, this is not Production level code so, copy and paste at your own risk.

So to start you need to add the Confluent.Kafka NuGet package (current version as of this writing is 1.4.0).

Next, create a config type and set the BootstrapServers – this is the server your code will contact to setup the message broker and send messages to based on where that broker ends up (not sure how all of that works yet). Suffice to say, when you finished running your Helm install this is the Service DNS name you are given – it follows the standard convention used by Kubernetes to name services.

For our example we cycle over all of the int values available to .NET (all the way up to int.MaxValue) so we can keep our producer going for a long time, if need be. For each iteration our code simply writes a message to the broker indicating the current iteration number.

We use the ProduceAsync method to send this message to Kafka – we use a try/catch here to catch any sending errors. Everything is written out to STDOUT via Console.WriteLine.

One of the key arguments to ProduceAsync is the name of the topic to associate our message to. This is what our consumers will listen to so a single message sent to this topic can be fanned out to as many consumers as are listening. This is the power of this type of architecture as it allows for event based processing with a high degree of decoupling. This allows different parts of our application to simply respond to the event rather than being part of a longer chain of functions.

Build the Consumer

As with the Producer, the first step here is to add the Confluent.Kafka NuGet package

As with the Producer, our first step is to add the Confluent.Kafka Nuget package so we can use the built-in types to communicate with Kafka.

You can see with the Consumer that we subscribe to our topic (increment-topic in this case). I was a bit surprised this was using the built in .NET eventing model since I think that would make more sense. Instead, we have to create a busy loop that attempts to consume each time and checks if it gets anything.

From there we just bring the message we received.  You notice that our BootstrapServers value is the same as it was in the Producer, should be – we are writing and reading from the same Broker.

The GroupId, I do not know what this is – need to read up on it but I set it all the same.

Testing our Example

Our goal is to run against Kafka hosted in Kubernetes and thus we will want both our Producer and Consumer there as well. You could use kubectl port-forward to expose the Kafka port locally but, I didnt have much luck with that as the code would immediately try to call a Kubernetes generated service name which was not exposed. I might tinker with this some more. My main goal with this exercise is to see this working in Kubernetes.

The best way to do this, or at least the simplest is to create a vanilla pod with the consumer and producer executing as separate containers. Truthfully, you would never want to use a vanilla PodSpec in Production (usually you want it in a Deployment or ReplicaSet) since doing so would not maintain high availability and resiliency (if the Pod dies, everything stops, it doesnt get recreated). Nevertheless, for this simple example I will be creating a multi-container pod – we can check the log files to ensure things are working.

Here is the PodSpec:

I will assume that you know how to build and push Docker images to a repository like Docker Hub (or Azure Container Registry in this case).

Next, we apply this spec file to our Kubernetes cluster

kubectl apply -f podspec.yaml

And we can use the following to wait until our pod is running (we should see 2/2 when ready)

kubectl get pods -n kube-kafka

Next let’s get a sample of the logs from our Producer to ensure things are connecting our messages are going out. We can run the following command (using the PodName from the PodSpec above):

kubectl logs kafka-example -n kube-kafka producer

If things are working you should see a lot of message indicating a message send. If you see errors, double check that you typed everything right and that, in particular, your BootstrapServer value is correct (you can use kubectl to query the services in the deployed namespace to ensure you have the right name)

Assuming that is working, we can perform a similar command to see the logs for the Consumer:

kubectl logs kafka-example -n kube-kafka consumer

Once again, if all things are working you should see messages being read and displayed in the console.

Congratulations!! You have just completed your first end to end Kafka example (maybe its not your first but congrats all the same).

Why is this important?

I talked about real time applications at the onset – these are comprised, generally, of event streaming whereby as data enters the system it causes events to be generated (often times the events are the data) which can be sent all over the system. It can create records in a database, updated metric counters, flip switches, and many other things – this is actually the basis behind IoT applications, streaming a tremendous amount of telemetry data and using systems like Kakfa to analyze and process that data in real time.

I love these types of applications because they enable so many unique business scenarios and really cool real time charts. In my next post, I hope to take my Kafka knowledge further and do something more useful.

Creating a File Upload with Kubernetes and Azure

Kubernetes makes managing services at scale very easy by creating abstractions around the common pain points of application management. One of the core values is to treat things as stateless since everything is designed in a way to be highly reliable and failure resistant.

This stateless nature can befuddle developers who want to create services that persist state, databases are the most common manifestation of this need. Putting aside the ongoing debate on whether databases should be located in a cluster when managed providers exist, I wanted to instead focus on the aspect of disk storage.

It is a well establish fact that storing user data within a container or within a Pod in Kubernetes is simply not acceptable and presents too many challenges even outside potential for data loss.  Thankfully, each of the managed providers enables a way to map volumes from their storage services into Kubernetes allowing the cluster to save data to a persistent, scalable, and resilient storage medium. In this post I will walk through how I accomplished this with Azure Kubernetes Service.

Part 1: The Application

I am going to deploy a simple Web API .NET Core application with a single endpoint to accept an arbitrary file uploaded. Here is what this endpoint looks like:

All we are doing here is reading in the stream from the request and then saving it to a directory as defined in our configuration. Locally, this will be driven by our appsettings.json file. DotNet Core will automatically ensure that Environment variables are also added as the program starts – these will overwrite values with the same name coming from the JSON files (this will be very important to us).

We can now create our mostly standard Dockerfile – below:

Do you see a slight difference? In the Dockerfile I created an environment variable to overwrite the value in appSettings.json (/image_write in this case). This now gives me a way to mount external resources to this location in the container, very important when we get into Kubernetes.

Build this image and push it to a registry your cluster has access to.

Part 2: Setup and mount the Azure File Share

Our next step involves creating an Azure file share and enabling our cluster to communicate with it thus allowing us to mount it when Pods are deployed/brought online.

Microsoft actually does a very good job explaining this here: https://docs.microsoft.com/en-us/azure/aks/azure-files-volume

By following these instructions you end up with a new Azure Storage Account that contains a file share. We store the connection string for this file share in an Kubernetes secret (in the same namespace our stuff is going to get deployed to (I call mine file-upload).

Here is the deployment spec I used to deploy these Pods with the appropriate mounting:

So you can see in the container spec section, we mount our savepath volume to our defined path. We then define this volume as coming from Azure in our volumes section. The rest of the definition is as we would expect.

From here you would need to enable external access to the Pods, you have three options:

  • Service of type NodePort and then call the appropriate IP with /file using POST – refer to the endpoint definition for the parameter name.
  • Service of type LoadBalancer – instructions same as above
  • Use of Ingress controller to split the connection at the Kubernetes level

Lessons Learned

This was a pretty neat exercise and I was impressed at just how easy it was to set this up. Having our data be stored on a managed provider means we can apply Kubernetes to more scenarios and get more value – since the managed cloud providers just have more resources.

 

 

Heading to New Signature

I decided to leave my post at West Monroe after nearly 6 years in their service. My new position will be a DevOps Consultant at New Signature – I will be tasked with helping clients adopt a Cloud first and DevOps centric mentality. I am absolutely ecstatic for my new role.

In saying that, leaving WMP is bittersweet. I am excited for the new challenge but, I am also cognizant of just what a great place to work it is. I owe them for helping me recover my career after I went through some hard times before that. I rarely told this story but, after working at Centare and having things not pan out at all how I thought, I questioned staying in consulting. Combined with a lot of personal pain I was in a rough spot when they offered me a Senior Consultant position in 2014.

Along with their offer I had a mobile development company make an offer as well so, it really came down to what I wanted. Luckily my girlfriend, now wife, helped me see clearly and I decided to take the plunge into WMP, a company I thought was going to be a challenge to work for due to the perceived corporate nature. But I did not find this “corporate culture” instead I found a home. I found great people that were smart and pushed me as I pushed them to be the best we could be.

Over the next 6 years I went through what I can only describe as a career renaissance. Thanks to WMP I regained my vigor in the community and achieved the status of Microsoft MVP (if only for a few years) and I reignited my passion for backend and DevOps, something that I know champion heavily and led me to finding the job at New Signature. But more than anything, the people I met are something I can never forget and was what made leaving so hard.

As I turn to the future, I am nervous. I will be taking my family to Atlanta to build a better life for my son and finally put down roots. This will be a challenge but, I have already been greeted by wonderful people at New Signature and I am very excited for the challenges this new post will offer me.

Big thanks to everyone at West Monroe and I look forward to the day our paths cross again.

Setting up a Microservices Example on AKS

Kubernetes is a platform that abstracts the litany of operational tasks for applications into a more automative fashion and enables application needs declarations via YAML files. Its ideal for Microservice deployments. In this post, I will walk through creating a simple deployment using Azure AKS, Microsoft managed Kubernetes offering.

Create the Cluster

In your Azure Portal (you can do this from the az command line as well) search for kubernetes and select ‘Kubernetes Service’. Creating the cluster is very easy, just follow the steps.

  • Take all of the defaults (you can adjust the number of nodes, but I will show you how to cut cost for this)
  • You want to be using VM Scale Sets (this is a group of VMs that comprise the nodes in your cluster)
  • Make sure RBAC is enabled in the Authentication section of the setup
  • Change the HTTP application routing flag to Yes
  • It is up to you if you want to link your service into App Insights

Full tutorial here: https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough

Cluster creation takes time. Go grab a coffee or a pop tart.

Once complete you will notice several new Resource Groups have been created. The one you specified contains the Kubernetes services itself, I consider this the main resource group that I will deploy other services into – the others are for supporting the networking needed by the Kubernetes service.

I want to draw you attention to the resource group that starts with MC (or at least mine does, it will have the region you deployed to). Within this resource group you will find a VM scale set. Assuming you are using this cluster for development, you shut off the VMs within this scale set to save on cost. Just a word to the wise.

To see the Cluster in action, proxy the dashboard: https://docs.microsoft.com/en-us/azure/aks/kubernetes-dashboard

Install and Configure kubectl

This post is not an intro and setup of Kubernetes per se so I assume that you already have the kubectl tool installed locally if not: https://kubernetes.io/docs/tasks/tools/install-kubectl

Without going to deep into it, kubectl connects to a Kubernetes cluster via a context. You can actually see the current context with this command:

kubectl config current-context

This will show you which Kubernetes cluster your kubectl instance is currently configured to communicate with. You can use the command line to see all available contexts or read the ~/.kube/config file (Linux) to see everything.

For AKS, you will need to update kubectl to point at your new Kubernetes service as the context. This is very easy.

az aks get-credentials -n <your service name> -g <your resource group name>

Executing this command will create the context information locally and set your default context to your AKS cluster.

If you dont have the Azure Command Line tools, I highly recommend downloading them. (https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest).

Deploy our Microservices

Our example will have three microservices – all of which are simple and contrived to be used to play with our use cases. The code is here: https://github.com/xximjasonxx/MicroserviceExample

Kubernetes runs everything as containers so, before we can start talking about our services we need a place to store the Docker images so Kubernetes can pull them. You can use Docker Hub, I will use Azure Container Registry, Azure’s Container Registry service, it has very nice integration with the Kubernetes service.

You can create the Registry by searching for container in the Azure search bar and selecting ‘Container Registry’. Follow the steps to create it, I recommend storing it in the same Resource Group that your Kubernetes service exists in, you will see why in a moment. Full tutorial: https://docs.microsoft.com/en-us/azure/container-registry/container-registry-get-started-portal

Once this is created we need to attach it to our Kubernetes service so images can be pulled when requested by our Kubernetes YAML spec files. This process is very easy, and documented here: https://docs.microsoft.com/en-us/azure/aks/cluster-container-registry-integration

We are now ready to actually deploy our microservices as Docker containers running on Kubernetes.

Names API and Movies API

Each of these APIs are structured the same and serve as the source of data for our main service (user-api) which we will talk about next. Assuming you are using the cloned source, you can run the following commands to push these APIs into the ACR:

docker build -t <acr url>/names-api:v1 .
az acr login –name <acr name>
docker push <acr yrl>/names-api:v1

The commands are the same for movies-api. Notice the call to az acr login which grants the command line access to the ACR for pushing – normally this would all be done by a CI process like Azure DevOps.

Once the images is in the ACR (you can check via Repositories under the Registry in the Azure Portal) you are ready to have Kubernetes call for it. This, again, takes an az aks command line call. Details are here: https://docs.microsoft.com/en-us/azure/aks/cluster-container-registry-integration

As a personal convention I store my Kubernetes related specs in a folder called k8s this enables me to run all of the files using the following command:

kubectl apply -f k8s/

For this example, I am only using a single spec file that defines the following:

  • A namespace for our resources
  • A deployment which ensures at least three pods are always active for each of the two APIs
  • A service that handles routing to the various pods being used by our service
  • An Ingress that enables cleaner pathing for the services via URL pattern matching

If you are not familiar with these resources and their uses, I would recommend reviewing the Kubernetes documentation here: https://kubernetes.io/docs/home/

If you head back to your Kubernetes dashboard the namespaces should appear in your dropdown list (left side). Selecting this will bring up the Overview for the namespace. Everything should be green or Creating (yellow).

Once complete, you can go back into Azure and access the same Resource Group that contains your VM scale set, look for the Public IP Address address.  Here are two URLs you  can use to see the data coming out of these services:

http://<your public IP>/movie – returns all source movies
http://<your public IP>name – returns all source people

The URL pathing here is defined by the Ingress resources – you can learn more about Ingress resources here: https://kubernetes.io/docs/concepts/services-networking/ingress. Ingress is one of the most important tools you have in your Kubernetes toolbox, especially when building microservice applications.

User API

The User API service is our main service and will call the other two services we just deployed. Because it will call them it needs to know the URL, but I do not want to hard code this, I want it to be something I can inject. Kubernetes offers ConfigMap for just this purpose.  Here is the YAML I defined for my ConfigMap:

ConfigMaps are key value pairs under a common name, server-hostnames. Then, we can access our values via their respective keys.

How we get these values into our API happens via the Pods which are provisioned for our Deployment. Here is that YAML:

Note the env section of the YAML. We can load our ConfigMap values into environment variables which are then accessible from within the containers. Here is an example of reading it (C#):

As with the other two services you can run a kubectl apply command against the k8s directory to have all of this created for you. Of note though, if you change namespaces or service names you will need to update the ConfigMap values.

Once deployed you can access our main endpoint /user off the public Url as before. This will randomly build the Person list with a set of favorite movies.

Follow up

So, this was, as I said, a simple example of deploying microservices to Azure AKS. This is but the first step in this process and up next is handling concepts like retry, circuit breaking, and service isolation (where I define what services can talk to). Honestly, this is best handled through a tool like Isito.

I hope to not show more of that in the future.

Dynamic Routing with Nginx Ingress in Minikube

So, this is something I decided to set my mind to understanding how I can use Ingress as a sort of API Gateway in Kubernetes. Ingress is the main means of enabling applications to access a variety of services hosted within a Kubernestes cluster and its underpins many of the more sophisticated deployments you will come across.

For my exercise I am going to use minikube to avoid the $8,000 bill Amazon was gracious enough to forgive last year 🙂 In addition, for the underlying service, I am using a .NET Core WebAPI Hosted via OpenFaaS (howto).

Understanding OpenFaaS

Without going to deep into how to set this up (I provided the link above) I created a single Controller called calc that has actions for various mathematical operations (add, substract, multiple, and divide). Each of these actions can be called via the following URL structure:

<of-gateway url>:8080/function/openfaas-calc-api.openfaas-fn/calc/<op_name>

Note: open-faas-calc-api is the name of the API in OpenFaaS as I named it, yours will likely differ

The goal of our Ingress is, via the IP returned by minikube ip we want to simplify the URI structure to the following:

<minikube ip>/calc/<op_name>

Within our Ingress definition we will rewrite this request to match the URL structure shown above.

Create a basic Ingress

Let’s start with the basics first, here is the configuration that is a good starting point for doing this:

You can find the schema for an Ingress definition in the Kubernetes documentation here. Ingress is a standard component in Kubernetes that is implemented by a vendor (minikube support Nginx out of the box, other vendors include Envoy, Kong, Treafik, and others).

If you run a kubectl apply on this file the following commands will work

<minikube ip>/function/openfaas-calc-api.openfaas-fn/calc/<op_name>

However, this is not what we want. To achieve the rewrite of our URL we need to use annotations to configure NGINX specifically – we actually used the ingress.class annotation above.

Annotate

NGINX Ingress Controller contains a large number of supported annotations, documented here. For our purposes we are interested in two of them:

  • rewrite-target
  • use-regex

Here is what our updated configuration file looks like:

You can see the minusha we need to pass for OpenFaaS calls has been moved to our rewrite-target. The rewrite-target is the URL that will ultimately be passed to the backend service matched via path (and host if supposed).

What is interesting here is we have given a Regex pattern to the path value meaning, our rule will apply for ANY URL that has /calc/<anything> as a format. The (.*) is a Regex capture group enabling us to extract the value. We can have as many as we like and they get numbered $1, $2, $3, and so on.

In our case, we are only matching one thing – the operation name. When it is found, we use $1 to update our rewrite-target. The result is the correct underlying URL that our service is expecting.

We can now call our service with the following URL and have it respond:

<minikube ip>/calc/<op_name>

Thus we have achieved what we were after.

Additional Thoughts

Ingress is an extremely powerful concept within Kubernetes and it enables a wide array of functionality often seen with PaaS services such as API Gateway (Amazon) and API Management (Azure). Without a doubt it is a piece of the overall landscape developers will want to be well versed in to ensure they can create simple and consistent URL to enable REST, gRPC, and other style of services for external access.

Authenticating JWT Tokens with Azure Functions

So recently, I decided to work on creating some HTTP exposed Azure Functions to return data if a JWT token was valid and various 4xx response codes otherwise. Needless to say, I did not expect it to be as hard as it turned out to be, I would say that Microsoft has work to do to enables support of full blown APIs with Azure Functions provided they are not held behind an API Management gateway service; this may be what is intended.

How did I create my token?

So, I used JwtSecurityToken in the Microsoft.IdentityModel.Tokense Nuget package with a Symmetric Security Key to generate a signed signature. This was pretty easy – here is my token generation code:

For our purposes we want to be able to decode the token to get some non confidential information (the username) so we can do some lookup for user related information – we could also choose to use the UserId as well here if we so desired (in fact we should if the user can change their username)

Decrypting the Token

Here is my code for decrypting the token above via a Read service I wrote as a common method for other Microservices:

The important thing here is that we use the same Issuer, Audience, and Key as during the encryption process. Validate will use these values and check our token – there are a variety of exceptions that can come out of this operation so you will want to the call to be ready to catch them for the various error cases: <Docs>

Ok so that all is actually pretty easy, now lets get into the hard part. Our goal is, when our Azure Function is called we want to receive the parsed result from the JWT token so we can centralize this logic and use it across many functions.

Normally, the way you would do this is to create a Filter that checks the request and, if valid, passes the value to some sort of base class that holds our function. Often this requires DI since we are injecting our Read Service into the Filter. We support this in normal Web API with ServiceFilter. Unfortunately, Microsoft currently does not support this, or any approach that I could find for Azure Functions. So what do we do.

Introducing Extensions

So, the Function runtime does support custom extensions which can act, in a way, like filters do in .NET Core (Azure Functions do actually support Filters, they are just new and arent as feature rich as their MVC/WebAPI counterparts).

Using an extension we can make our Azure Function call look like the following:

Do you see it? UserToken is our custom extension. Its job is to look at the incoming request and grab the token, decode it, and pass an object with various bits of claim data. Be careful with what you pass, you do not want sensitive data in your claims data since anyone can head over to jwt.io and decode it and see your claims.

First Step: Create a Value Provider

Extensions are a means to call custom bindings. Bindings seek to provide a value. Azure Function host provides the IValueProvider that we need to implement to create our Value Provider. This class will perform the operation relevant to our custom binding. Below are the two pieces of this class that are relevant: Constructor and GetValueAsync

As I mentioned earlier, the Validate method (called by ReadToken) can throw a litany of exceptions depending on problems with the token. Ultimately, the value returned from here is what our Azure Function receives.

The reason I choose to include the constructor was to begin to illustrate how the ReadTokenService is hydrated – you will find that DI is rather limited at this level and requires some odd hacks to get it to work. We will get into it as we unwrap this.

Ok good, this is your value provider, now we need to create the binding which calls this.

Part 2: Create our Binding to call the Value Provider

The binding is the layer between the Extension and the Value Provider. It immediately receives a binding context that gives it information about the incoming request so we can extract information – this is where we get the raw headers that contain our token from. Here we implement the IBinding interface. Here is my constructor, ToParameterDescriptor, and BindingAsync(BindingContext context):

So, the first thing to unpack here is the constructor – technically there is NO DI support within extensions (for some reason). How I got around this was I passed IServiceProvider which is our DI Container that I can extract dependencies from – this is what I do via the Service Locator Pattern: we extract both our configuration facade and the service to read our token.

Where this comes into play is when we create our ValueProvider – we pass the service to read the token into the constructor as we create it.

The remaining code in BindAsync is our logic for extracting the raw token from the Auth header (if it is present) and passing it, again via the constructor, to our Value Provider.

As for the ParameterDescriptor, I dont really know what this is doing or what it is used for, it doesnt seem to have an impact, positive or negative, on this use case.

Ok, so no we have create a Binding which calls our Value Provider to carry out the operation. We use Service Locator pattern on the DI container to extract the dependencies that we need. Our next step is to create the Binding Provider

Part 4: Create the Binding Provider

Our extension calls a specific binding provider to get the binding to carry out the intended operation for the extension. This is driven using the IBindingProvider interface and implementing the TryCreateAsync method. For our example, this class is very tiny, I show it in its entirety below:

Again, you can see I pass IServiceProvider into this method via the constructor and then pass it to our binding which we described in the previous step.  I am sure you can see where this is going 🙂

Part 5: Create the Extension Provider

We have finally arrived at the extension provider. This is where we register our Extension with the runtime so it can be used within our code. This implements the IExtensionConfigProvider to support the Initialize method:

And this is where we can get our IServiceProvider reference that we pass into all of those layers. In truth, since Azure Function do NOT support DI we manually build our container and pass it into our lower levels.

The catch to this approach though is, you do NOT want to write your dependency registration twice. To that end, I wrote an extension method called RegisterDependencies so I wouldnt need to have duplicate code. Additionally, I had to manually register the IConfiguration facade (this is done for you in normal startup flow).

The final block here is adding the binding rule for our parameter level attribute so that, when the runtime sees the attribute it knows to invoke the create method on our binding provider. Here is the code for our attribute:

The one thing extra here is the user of the Binding attribute to denote the attribute represents a binding that will be used in Azure Function.

Part 6: Change our Starting Host

So, if you have ever worked with Azure Functions v2.0 you are recommended to use the FunctionsStartup class. Supporting this is a means to register extensions in a declarative way, a way that I could not get to work though I suspect involves the steps listed here. Regardless, IFunctionsHostBuilder (the type passed to the Configure method when using the interface) does NOT have a way to register extensions from code. So what to do?

Well, it turns out you can change IFunctionsHostBuilder with IWebJobsStartup which is the old way of doing this and that will provide a method to register the extension provider – shown below:

Again, note the call to RegisterDependencies which unifies our registration so we do not have duplicate code. I have yet to notice any irregularities with using the web jobs approach vs the Function host yet, please comment if you see anything.

Part 7: Handling the Result of our Token Parse

So, Azure Functions do offer the FunctionExceptionFilterAttribute base class which enables a hook to respond to uncaught exceptions raised by functions. Unfortuantely, this hook does not enable you to alter the response so, even if you catch the relevant exception the response code is already written – it seems to be more for logging that handling.

So, the best I could come up with is each function has to be aware of how to intepert a failed parse response. Here is my complete function that shows this:

You can see that we introduced an enum called TokenState that I pass from the underlying Value Provider. This is not an ideal approach since it means each developer writing a function must be aware of the various auth scenarios that can occur for their function. This leads to duplication and create error proned code. But, it is the best that I can find, this far.

Closing

So, honestly, disappointed in Microsoft. I feel like the Azure Functions are really designed to be used behind an API Management gateway to alleviate some of the checks but, the DI maturity is abhorid. I really do hope this is more the case of me missing something than this being the actual state, especially given the rising importance of serverless in modern architectures.

I know I showed a lot of code and I hope you had good takeways from this. Please leave comments and I will do my best to answer any questions. Or I would love to know if I missed something with the way to do this that makes it easier.

Using Terraform to Deploy Event Grid Subscriptions for Function Apps

I recently set a goal for myself to create a microservice style application example whereby I would enable individual services to listen for event coming through Azure Event Grid. As a caveat to this, I wanted to build everything up, and thus have everything managed, via Terraform scripts. This proved very challenging and I wanted to take time to discuss my final approach.

Terraform, no datasource for Event Grid Topics

So, annoyingly, Terraform does NOT contain a datasource for Event Grid topics, meaning in order to reference the properties of a target topic you need to either store the values in a vault or something similar, or grab the outputs from creation and pass them around as parameters; I choose to do the later, for now.

Capturing the Relevant values from Topic Creation

As part of my environment setup process, I defined the following script in Terraform HCL to create the relevant topic for the environment; this does mean I will have a single Topic for each environment which often sufficient for most use cases

Key to this is the outputs. I am using Azure DevOps to execute the build pipeline. In ADO, you are able to give task names (Output Variables) which then allows you to reference task level variables for that task from other tasks. I use this bash script to extract the topic_id from the above:

If you were following along in Azure DevOps, the EnvTerraform is the custom name I gave via Output Variables for the Apply Terraform operation.  I am using the jq command line tool to parse the JSON that comes out of this output file.

Finally, we can use an echo command to ask the ADO runtime to set our variable. This variable is scoped, privately, to the task. We use the isOutput parameter to indicate that it should be visible outside the task. And finally we give it the value we wish to set.

The importance of this will become clear soon.

Create the Event Subscription

Event Grid Topics contain subscription which contain the routing criteria for message to various endpoints. Event Grid supports a wide range of endpoints and is, in my view, one of the most useful PaaS components offered by the Azure platform. For our case, we want to route our events to a WebHook which will invoke an Azure Function marked with the EventGridTrigger attribute, available via the relevant Nuget package.

Before we get started there is a problem we must solve for. When subscriptions are created, Azure will send a pulse to the webhook endpoint to ensure it is valid. This endpoint just needs to return a 2xx status code. One issue, in order to even get to our method, we need to get passed the enforced Function App authentication. Thus, we need to pass our MasterKey in the subscription to enable Event Grid to actually call our function.

It turns out this is no small task. The only way to get this value is to use the Azure CLI and then, again, expose the value as a Task level variable.

Here is the script I threw inside an Azure CLI task in Azure DevOps:

Some notes here:

  • First execution gets us the subscriptionId for the active subscription being used by the Azure CLI task
  • Next we need to get the appName – as I want my script to be fairly generalized I am undertaking an approach where I pass the created function app hostname into the script and parse out the name of the function app from the Url using sed
  • Next, build the resource Id for the Azure Function app – you can also get this from the output of a function app resource or datasource – I choose not to do it this way as a matter of preference and convenience
  • Next, using the Management API for the Function App, we ask for a list of all keys and again use jq to grab the lone masterKey
  • Finally, using the echo approach to create our output variable (named funcAppMasterKey) so we can use it later and we will

In terms of the actual Terraform script to create the Event Subscription, it looks like this:

One massive tip here, if you specify topic_id for scope, do NOT specify topic_name. My thought is TF concatenates these values under the hood but I was never able to get it to work that way.

For the webhook, we follow a standard format and specify the exact name of our trigger. This is roughly the same Url structure you would use to invoke the trigger method locally for testing.

Finally, notice the use of masterKey at the end of the webhook Url. This is passed in as a parameter variable based on the value we discovered in the Azure CLI task.

Running this

For my purposes, I elected to approach solving this by breaking apart my TF script into two parts: one which creates all of the normal Azure resources, including the Function App and then a second that specifies the subscriptions I wish to create – there are certainly other ways to approach this.

By splitting things apart I was able to perform my Azure CLI lookup in an orderly fashion.