Consider using Service Bus Queues instead of Azure Storage Queues when using Application Insights

Previously, I have discussed things that weren’t so clear to me when implementing Application Insights distributed tracing. There was another thing that I came across but thought it was worthy of a post on its’s own.

Simply put, it is that Azure Storage Queues cannot effectively pass on telemetry information. Look at Service Bus instead.

With Azure Functions, we use queue-based triggers to run non-immediate tasks within or in-between services. These queues are all Azure Storage Queues, and they were chosen as that because they did everything we needed, and we can move to a different queuing or messaging tech when the need arose. That day came when we needed to use distributed tracing across services to allow us to see all aspects of our processes from start to finish.

To help explain the below, I have created the following repository to help demonstrate what I mean. It is an Azure Function that takes in a name, searches the Star Wars API for anyone with that name, and places the details of each person as a message on a queue; there is an additional function to pick up that queue message and hit the same API to get details of the films that character has been in.

With regard to distributed tracing, any HTTP request will have additional correlation headers added; specifically a Request-Id giving the operation id of the call, and Request-Context which holds (at least) the AppId of the calling Application Insights instance. This is all pretty straight forward for a HTTP request, but queue messaging is a bit different.

Although Azure Storage Queues use HTTP to send on a message, that HTTP call is sent to Azure storage, and not your service, and all correlation information is lost. The best you can do is send up the Request-Id and Request-Context as part of the message, but by the time the message is dequeued and read, it is possible multiple pieces of telemetry have been sent with the wrong correlation information.

This is what I see when I call my function, searching for the name “Skywalker”, three characters are returned (Luke, Anakin, Shmi), and then the character details are sent off individually to an Azure Storage Queue:

And then a new unrelated operation for Luke:

And then a new unrelated operation from Anakin:

And then a new unrelated operation for Shmi:

Service Bus Queues are different to storage queues as these puppies are an implementation of the Advanced Message Queuing Protocol and if implemented using message handlers, the correlation will work just fine. Fortunately from what I can tell, the Application Insights SDK uses message handlers on .NET Core projects and you don’t need to do anything more than install the nuget package.

The result, without any additional coding effort, is the ability to trace through all queues in one session:

Additional things to know with regard to Azure Functions

I said above that you need to install the Application Insights SDK, but you may wonder why I said this when Azure Functions automatically logs to AppInsights through the ILogger already injected. The reason for this is the logging provided by Azure Functions works well for simple logging, but if you want to do custom properties or metrics, or (this is the kicker) accept correlation information needed for distributed trace logging, you will need the full SDK.

When adding the SDK to Azure Functions, the Microsoft Docs explain how to use dependency injection to use the SDK. That’s all well and good until you get warnings the Application Insights SDK version 2.11 has moved away from using the TelemetryConfiguration.Active singleton, which is needed as part of the injection process. I am not sure if the AppInsights team never really considered Azure Functions when making this change, or if the Azure Functions team has not updated their docs. I have left my example at 2.10, as I figuring it out was something for a later date.

In conclusion

We have discussed how Storage Queues are pretty useless if you want to implement Distributed Tracing, a brief detail on how distributed tracing works, and what to do and expect when implementing Service Bus Queues.

Although I have spoken purely of Service Bus, I believe Azure Event Hubs will give the same result.