I recently set a goal for myself to create a microservice style application example whereby I would enable individual services to listen for event coming through Azure Event Grid. As a caveat to this, I wanted to build everything up, and thus have everything managed, via Terraform scripts. This proved very challenging and I wanted to take time to discuss my final approach.
Terraform, no datasource for Event Grid Topics
So, annoyingly, Terraform does NOT contain a datasource for Event Grid topics, meaning in order to reference the properties of a target topic you need to either store the values in a vault or something similar, or grab the outputs from creation and pass them around as parameters; I choose to do the later, for now.
Capturing the Relevant values from Topic Creation
As part of my environment setup process, I defined the following script in Terraform HCL to create the relevant topic for the environment; this does mean I will have a single Topic for each environment which often sufficient for most use cases
Key to this is the outputs. I am using Azure DevOps to execute the build pipeline. In ADO, you are able to give task names (Output Variables) which then allows you to reference task level variables for that task from other tasks. I use this bash script to extract the topic_id from the above:
If you were following along in Azure DevOps, the EnvTerraform is the custom name I gave via Output Variables for the Apply Terraform operation. I am using the jq command line tool to parse the JSON that comes out of this output file.
Finally, we can use an echo command to ask the ADO runtime to set our variable. This variable is scoped, privately, to the task. We use the isOutput parameter to indicate that it should be visible outside the task. And finally we give it the value we wish to set.
The importance of this will become clear soon.
Create the Event Subscription
Event Grid Topics contain subscription which contain the routing criteria for message to various endpoints. Event Grid supports a wide range of endpoints and is, in my view, one of the most useful PaaS components offered by the Azure platform. For our case, we want to route our events to a WebHook which will invoke an Azure Function marked with the EventGridTrigger attribute, available via the relevant Nuget package.
Before we get started there is a problem we must solve for. When subscriptions are created, Azure will send a pulse to the webhook endpoint to ensure it is valid. This endpoint just needs to return a 2xx status code. One issue, in order to even get to our method, we need to get passed the enforced Function App authentication. Thus, we need to pass our MasterKey in the subscription to enable Event Grid to actually call our function.
It turns out this is no small task. The only way to get this value is to use the Azure CLI and then, again, expose the value as a Task level variable.
Here is the script I threw inside an Azure CLI task in Azure DevOps:
Some notes here:
- First execution gets us the subscriptionId for the active subscription being used by the Azure CLI task
- Next we need to get the appName – as I want my script to be fairly generalized I am undertaking an approach where I pass the created function app hostname into the script and parse out the name of the function app from the Url using sed
- Next, build the resource Id for the Azure Function app – you can also get this from the output of a function app resource or datasource – I choose not to do it this way as a matter of preference and convenience
- Next, using the Management API for the Function App, we ask for a list of all keys and again use jq to grab the lone masterKey
- Finally, using the echo approach to create our output variable (named funcAppMasterKey) so we can use it later and we will
In terms of the actual Terraform script to create the Event Subscription, it looks like this:
One massive tip here, if you specify topic_id for scope, do NOT specify topic_name. My thought is TF concatenates these values under the hood but I was never able to get it to work that way.
For the webhook, we follow a standard format and specify the exact name of our trigger. This is roughly the same Url structure you would use to invoke the trigger method locally for testing.
Finally, notice the use of masterKey at the end of the webhook Url. This is passed in as a parameter variable based on the value we discovered in the Azure CLI task.
For my purposes, I elected to approach solving this by breaking apart my TF script into two parts: one which creates all of the normal Azure resources, including the Function App and then a second that specifies the subscriptions I wish to create – there are certainly other ways to approach this.
By splitting things apart I was able to perform my Azure CLI lookup in an orderly fashion.