Category: Tips & Tricks

Being vacation time and therefore not too much action, but with me still working, I decided to have a look at Azure Alerts to see if that would be something we could take advantage of in our daily DevOps routine.

I decided to set up an alert on average response times from one of our API:s being greater than X, and setting X to a low value (40 milliseconds) just to see what would happen.

Then I started to wait. And I waited. And I waited. Nothing.

I set it to an even lower value, this time to 20 milliseconds.
And I waited. Still nothing.

Surly there must be something wrong with Azure Alerts, right?
Nope. Looking closer at the GUI I noticed that even when the metric was in milliseconds, the threshold value was expecting seconds…
Now that’s a gotcha for ya!

If you look closely in the bottom right corner, it clearly states “seconds” and not milliseconds. I wish I had read that a couple of hours sooner than I did.

After that it worked just as I thought.

In a post way back here I wrote about temporarily stopping all Azure WebJobs within an Azure Web App. Single continuous WebJobs can be stopped manually one by one in the Azure Portal, but what about stopping a triggered WebJob?

After doing some research, and with that I mean stopping one of the triggered WebJobs in our test environment, I came to the conclusion that using the setting “WEBJOBS_DISABLE_SCHEDULE” and set the value to 1 does the trick without any fuzz.

Earlier today I was debugging and chasing down a strange behavior in an ServiceBus triggered WebJob in Azure. Looking at the logs I could tell that the WebJob was triggered when a new message arrived, but after that it seemed like it was idle for quite some time before starting to process the message. Since it took (quite) some time the WebJob lost the lock on the message, and it went back into the queue for a second go at processing the same thing.

I initially thought it was due to some kind of intermittent issue with the ServiceBus, that it might be serving the contents of the message slowly and that made the WebJob look like it was idling before the lock expired.

Since I couldn’t track the issue down all the way, I introduced a AutoRenewTimeout policy on an arbitrary number of minutes and started to follow the messages more closely in the logs. What I found was really interesting: The WebJob triggered on a couple of messages that came in at the same time, but after that it started to process the messages one by one. That means that the last couple of messages had their locks lost because they were claimed way before the actual processing began.

Going back to the code I noticed that the WebJob had the Singleton attribute which guarantees that only one instance of the WebJobs runs at the same time. For some, still unknown, reason it honors this and only runs one instance, but at the same time also honors the MaxConcurrentCalls value. So what was happening was that the WebJob first claimed a lock on 16 messages, and them began processing them one by one. No wonder the lock had expired on the last couple of messages…

When the singleton attribute was introduced in the code it was probably the MaxConcurrentCalls value that they were looking for.

So, a word of caution: The singleton attribute, used by itself without other configuration updates, might give you some weird results that aren’t actually what you’re after.

After a couple of failures due to Azure WebJobs not stopping when I thought they were going to, I figured out the only fail safe (?) option that will stop both Triggered and Continuous WebJobs. Not just that: It’s dead easy!

You can try to stop the App Service itself, but for some reason that does not apply to the WebJobs, and God only knows why. Stopping an individual Continuous WebJob can be done from the Azure Portal, but what about stopping multiple WebJobs including a Triggered one? I’ve even experienced that deleting a WebJob didn’t stop it, I’ve actually seen that multiple times.

To stop all WebJobs, both Triggered and Continuous, add WEBJOBS_STOPPED with the value 1 via “Configuration” / “Application settings” in the Azure Portal. As far as I’m concerned this has worked beautifully every time I’ve tried it.