Forcing an Azure Function to process Queue Storage messages one at a time

Last updated on September 3, 2023

By default, Azure Functions process Azure Storage queue messages in parallel. This is normally great as it tremendously decreases the time required for processing all of the items. However, some occasions might require you to process the messages one by one.

As I needed to do this for a customer case which I’ll explain later, I discovered that it was not as straight forward as I had thought. Or rather, no one internet resource would detail all the steps I was required to take. Hence, I decided to write this blog post that covers everything, and hopefully, you will find it useful if you are attempting to achieve 1-by-1 queue processing in your functions.

A bit of background

I have a customer who has automation in place for inviting guest users in large batches. After the invitation, the guests are added to security groups which grant them access to specific resources. Due to “reasons”, these security groups are mail-enabled security groups which causes one problem: we can’t add the users to the groups via Microsoft Graph because the Add member operation only supports Office 365 groups and regular Azure AD security groups.

Luckily, we can connect to Exchange Online in PowerShell and add the users to mail-enabled security groups with the Add-DistributionGroupMember commandlet. So, what I did was the following:

1. Created a new Azure Storage queue.
2. Made the primary guest invitation logic send the user ID and the group ID to the queue to wait to be processed.
3. Created a new PowerShell Azure Function with a queue trigger bound to the created queue.

The Exchange Online concurrency problem

When the PowerShell script in my Azure Function processed one message at a time, it worked perfectly. However, by default, Azure Functions process queue messages in parallel, and that’s when we started to run into problems.

The Exchange Online sessions have a concurrency limit. You can have a maximum of 10 connections open at a time. In practice, this means that the script works fine only if the queue receives and the Azure function processes a maximum of 10 messages at any one time. However, I did mention large batches, right? And by large batches, I mean there can be hundreds of messages hitting the queue in a very short period of time. When that happens, the Azure Function attempts to process a much larger number of them at once, which leads us to get a bunch of these exceptions:

Fail to create a runspace because you have exceeded the maximum number of connections allowed : 10 for the policy party : MaxConcurrency. Please close existing runspace and try again.

When that happens many times, the messages end up in the poison queue and will no longer be retried. By the way, there is a great tool called Azure Storage Explorer that allows you to move large amounts of queue messages from the poison queue to the processing queue with a single click of a button. I highly recommend it, and I have no idea why the functionality is not available in Azure Portal.

Parallelism is great when you need a bunch of things processed quickly. However, in this case, I’m more interested in reliability than speed. So, how can we force the Azure Function to process the queue messages one at a time?

Forcing the Azure Function to process a single queue message at a time

We need to tweak a couple of things to make our Azure Function process queue messages one at a time. You can do both of these configurations via Azure Portal.

host.json

In your Azure Function, go to App files, and add the below queues configuration to the host.json file (note that you might also have other settings in the file, so adjust your changes accordingly):

Setting the batchSize setting to 1 makes the function fetch only one message from the queue at a time, and
newBatchThreshold defines our function to check the queue for new messages only when the number of processed messages drops down to this value (0 = no messages are being processed any longer). Together, these settings ensure that a single Azure Function instance only processes one queue message at a time.

At other times, maxPollingInterval is the maximum amount of time we allow our function to “procrastinate” before it starts processing new queue messages. As I mentioned before, I’m not in a rush here, so I can allow the function to mentally prepare for 30 seconds. It can start processing the messages sooner, too, though, if it feels like it.

The last two settings are related to errors and reprocessing. visibilityTimeout essentially defines the delay between retries, and maxDequeueCount sets the maximum number of times the function tries to process the queue message before moving it to the poison queue (including the first run). As I mentioned before, if that happens, you can move the messages from the poison queue to the processing queue by using the Azure Storage Explorer desktop application.

Application settings

Then, still in the Azure Function, go to Configuration, and add the following application settings:

The WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT ensures that there can only ever be one Azure Function instance up, and FUNCTIONS_WORKER_PROCESS_COUNT limits the instance so that it can only execute one worker process at a time.

Ensuring the same behavior in Durable Functions

Normally, Azure Functions running on the Consumption plan are limited to 10 minutes of runtime. When you estimate you might be running into that cap, you start looking into making your function durable.

Azure Durable Functions have a fan-out-fan-in pattern which is normally used for processing items in parallel. To execute a certain logic for an N number of items, you most likely want to utilize this pattern.

But what if you don’t want the items to be processed in parallel but rather one-by-one? You might want to consider doing this if you are, e.g., calling Microsoft Graph in your activity function and experiencing a lot of throttling due to all the different threads bombarding the same endpoint at once.

Behind the scenes, Durable Functions also utilize Azure Storage queues in their processing, and hence the activity functions behave in a very similar manner to regular queue triggered functions. However, configuring them to process the queue items one by one happens slightly differently.

First, you need to configure the application settings the same way as above. Then, to ensure there can only ever be one activity function (and one orchestrator) in execution at any one time, you add the following to the function host.json. Again, remember that these configurations affect all the functions in the Function App. If you need to have different behaviour for different functions, you need to split them into separate Function Apps.

Afterword

That’s it! Ensuring queue items are processed one by one is quite a simple thing to do once you know how to do it. Heh, isn’t that always the case? 🙂

If you have any questions or comments regarding this topic, feel free to write them down below. And if enjoyed reading this article or found it useful, you might want to consider the following:

• Share this article with your friends!
• Add my blog to your RSS feed, e.g., on Feedly
• Sign up for my Insider (news)letter. I write a recap of things that are going on semi-regularly, typically once per month.
• Follow me on your favorite social media platform(s). I post a bit different content on each one of them.
  • Twitter (most active, tech related, occasional fun tidbits)
  • LinkedIn (primarily for the Finnish audience)
  • Github (code! when I have time…)
  • YouTube (still a bit work in progress, but feel free to sub if you want to get immediately notified when I do post videos)

Other than that, I wish you happy times in your Azure development adventures, and I’ll write something interesting for you again soon enough! Ciao!

Laura