Azure WebJob SDK Intro

I have some fellow colleagues who are extremely talented but they don’t have much working experience with Azure. So each time they are assigned to a job where they need to interact with the azure for data,  they kind of struggle. For example, few days earlier one of them is assigned to a job where he have to resize all the images. Writing the code which can resize the image is not that hard. He did that part in fair amount of time. But he find it bit difficult to interact with the storage. To help from such situation Azure blessed us with Webjob SDK.

Azure Webjob SDK is all about model binding in the cloud. Making it easier to talk to storage ! This is helpful and can speed up work because now I don’t have to write stuffs to establish connectivity with azure storage. Instead, I can concentrate on the core business logic part and let Webjob SDK to handle the connectivity issue for me. The best part of Webjob SDK is that it offers stuffs out of the box like triggering. So now I can do something more meaningful like do some task upon the arrival of new files or information on queue or so.

The first step is to install the nuget package for Azure Webjobs. Adding package from nuget is damn easy, I open up my nuget console and hit the following command :

Azure_WebJob_SDK_01

Now, I need to initialize an object which will do all the heavy lifting for me, things like triggering, binding, invocation of function and so on. I can do that by initiating JobHost object. So, here is my entry point, canonical main function:

I press Ctrl+Shift+B to build the solution and everything seems fine. Then I have tried to run this solution. Boom! Now, whats that? Its a run time error! But I have not done anything except initializing the JobHost object. Whats causing the error?

When I look into the error message it says –
“Microsoft Azure WebJobs SDK Dashboard connection string is missing or empty. The Microsoft Azure Storage account connection string can be set in the following ways:

  1. Set the connection string named ‘AzureWebJobsDashboard’ in the connectionStrings section of the .config file in the following format <add name=”AzureWebJobsDashboard” connectionString=”DefaultEndpointsProtocol=http|https; AccountName=NAME;AccountKey=KEY” />, or
  2.  Set the environment variable named ‘AzureWebJobsDashboard’, or
  3.  Set corresponding property of JobHostConfiguration.”

The error message is helpful as it also indicates how to resolve the issue. But wait a minute.  I don’t have any plan to use storage yet, so why its asking me to set storage credential? Plus from my  Azure Webjobs deployment experience, I know Azure storage is not required for Webjobs ! So, why it is issuing this error now?

Well, even though I don’t use storage entities, I still need to configure it because WebJobs’ SDK use some blobs and other storage entities for its own tracking and operation.

Note: 

  • AzureWebJobsStorage connection string is required.
  • AzureWebJobsDashboard connection string is optional.  you only need it if you want to be able to use the Dashboard UI for logs, etc.

Luckily, I already have an storage account which I am going to use at this point. If you don’t have one you can create storage account easily. Make sure the storage is placed the same region as the host website.

Now that I have provided the required storage information I will no longer face the existing runtime error. However, I know the template still need one more tweaking before it finally runs without an error. To find out the error lets re-run our simple program.

Azure_WebJob_SDK_02 - Copy

As you can see the exception saying “can’t be invoked from Azure WebJobs SDK. Is it missing Azue WebJobs SDK attributes?” Resolving this issue is quite easy. I just have to Use public on the main program class.

I took all the moving parts and placed them on the right spot and now everything seems workable. Now, I can focus on our canonical main function –

A JobHost is an execution container for your WebJob which knows what all functions you have in your program. The  RunAndBlock() method will cause the JobHost to listen for any triggers on any functions that you define in this Host.

Before I move any further, PLEASE NOTE I am using WebJob 1.0.1. If you are using earlier version the code snippet that I am showing might not work for you!

The following code snippet has two methods.

  • ProcessText() and
  • CopyBlobToBlob()

The method ProcessText() will be triggered when a text file will be found in the blob under the container named “input-dir” with a extension matches to “.txt”.  After that it will write the file with the same name in “output-dir”.

Note : If you don’t have a container with the name “output-dir” in your blob then this method will create a container by itself.

Azure_WebJob_SDK_05

CopyBlobToBlob() method works like the same way ProcessText() does. But since here I am pulling images so instead of TextReader, I am using Stream. The attribute BlogTrigger[] gets fired when a new blob content appears in the blob container. The Blob[] attribute write new blob contents to specified blob container.

Wrapping Up :

Azure Webjob SDK simplifies the communication part between your app and the cloud storage. So developers can focus on their main business logic and and they can leverage their app with the powerful binding features that Webjob SDK offers. They can simple pass proper attribute in their method parameter to connect to Azure Storage (notably blobs, tables, and queues) and SDK will handle everything.

Reference :

Advertisements

Azure Webjob ebook is on the way !

Your_Voice_Counts_01

I have been writing tech articles on the blog for quite some time now, and what motivates me to write regularly is YOU. Each time I receive a query or a request over mail, I feel special, because I feel somehow this small attempt of mine to help you guys is paying off.

Lately, many of you have mailed me to say that you guys are enjoying the Azure Webjob Series and asked me whether I will be able to provide a pdf version of the series to make it easier for you to read. You have also asked my plan regarding ‘Azure Webjobs‘  series and what are topics I am planning to cover in my future posts.

well, I just want to say –

“Your Voice Counts !”

Azure Webjob ebook is on the way !

THOUGH I am not giving any particular release date just yet, I can promise you that azure webjob ebook will be available very soon. Till then you can visit my blog for new posts on this series.

Regarding the roadmap I have planned for azure webjobs, here it is –

Get ready for Global Azure Bootcamp 2016

GlobalAzureBootCampLogo2016

After the success of the first Global Azure Bootcamp in Bangladesh, I Shahriar Hossain, on behalf of Microsoft Azure Bangladesh user group is happy to announce the probable date and venue of Global Azure Bootcamp 2016, Bangladesh. This year Global Azure Bootcamp will be held on April 2, 2016. Like previous year the event is likely to be held at Newscred, Dhaka, Bangladesh.

GABC_GroupJust to give you a quick recap, Last year we received satisfactory response from the IT professionals and university students. 120 enthusiasts had participated in the event. The event had received praises from different corners of the country and had drawn media attention as well. Many had expressed the need for such events in the future as these interactive events introduces IT professionals and specially the startup businesses and enterprises with the power of Azure. Along with Bangladesh, Global Azure Bootcamp 2015 was held at 183 other locations in 64 countries on the same day. The details of Global Azure Bootcamp 2015 (Bangladesh) can be found here.

Global Azure Bootcamp 2015 was so much successful and had received so many positive feedbacks that this year we are aiming higher and we are hoping to see much more participation and much more enthusiasm than last year.

So all the Azure enthusiasts out there in Bangladesh, its time to get excited for the Global Azure Bootcamp 2016.

Webjob deployment experience with Visual Studio

When it comes about IDE, I bet nothing could give such fluent experience like Visual Studio. In this post I will boost up our existing Azure Webjobs Deployment experience with the power of Visual Studio. But for that we will requires Visual Studio 2013 or above with Azure SDK version 2.4 minimum.

In VS when I go to create New Project by clicking File>New Project, I see numbers of template are available under Cloud section. But I am particularly interested in the template named “Microsoft Azure WebJob” .

AzureWebjobs_02_01

Select the template and create a project with your preferred name. Looking at the project structure (more specifically Program.cs file) you might be thinking this template is identical with our regular console application template. Yes, its almost identical. To find out the hidden treasure this template offers, right click the project and you will see “Publish as Azure WebJob…” button in the context menu.

AzureWebjobs_02_02

Clicking on “Publish as Azure WebJob…” button will open up a new dialog box from where you can easily configure your webjob settings. The setting include webjob name, its running mode and so on.

AzureWebjobs_02_03

Once you are done with the configure click the OK button to proceed. This will take you to the Publish Web dialog box where you will basically hook your webjobs with a web app. Clicking the Publish button will publish your webjobs.

NOTE :
” Use this template for an independent WebJob ! i.e. when you want to run a WebJob in a web app by itself, with no web application attached with it.”

Independent WebJob scenario comes into play when you might want to to scale your WebJob resources independently of your web application resources.

Where the Webjob file actually exists?

While its true I can relay on portal and Visual Studio to deploy webjobs, there might come situation when we will need the physical path of your webjobs. A WebJob is stored under the following directory in your site :

site\wwwroot\App_Data\jobs{job type}{job name}

{job type} -> could be either continuous (job that is always running)  or triggered job that starts from an external trigger (on demand / scheduler).
{job name} -> Your WebJob’s name.

For each scheduled WebJob VS create an Azure Scheduler job collection (if one doesn’t exist already). The naming convention for

  • The scheduler job collection is named WebJobs-{regionname}.
    {regionname} -> The region where the web app is hosted in.
  • The scheduler job is named {webappname}-{job name} .
    {job name} -> Your WebJob’s name.

Note: A WebJobs enabled project has the following items added to it:

  1. The Microsoft.Web.WebJobs.Publish NuGet package.
  2. A webjob-publish-settings.json file that contains deployment and scheduler settings.

Wrapping Up :

Pay close attention on the following fact :

  • If your Azure Web Sites site is running on multiple instances and you create a new continuous WebJob, the WebJob will run on all instances by default! On its contrary, triggered WebJob will run on just one instance selected at random !
  • If you want your continuous WebJob to run on only one instance, then you have tweak the Setting.json file with the { “is_singleton”: true } .
  • If you set { “is_singleton”: truefor triggered/schedule job then strange behavior may occur!

If a schedule occurrence is missed or fails by any reason, the runtime will try to get back on schedule by running the function again. Find details here in source code.

Reference :

Azure Webjobs deployment

Intro to Azure Webjobs covers what it is and why you should actually start using it for your job like batch processing, cpu intensive task etc. In this blog post I will just hit the surface of azure webjobs. So no big deal, this will be an easy to follow step by step guideline so that you can see how easy to get started with azure webjobs.

But before I start, there are few concepts/terms I need you to be aware of. Webjobs could be invoked in two different ways.

  1. Triggered jobs – When you want to do some task based on some event trigger or want to run on a schedule time those jobs are triggered jobs.
  2. Continuous jobs – Continuous jobs basically run a while loop.

Redeploy, Recycle any time :

Azure Webjobs handle the redeployment process based on job types. For triggered jobs redeployment will not affect instantly, that means the currently running WebJob will not affected by it. The next run will use the new binaries. On its contrary, for continuous job redeployment will abort the current running process and restart with the new binaries. Now you might be wondering how it is actually possible to modify or delete binaries  in webjobs while its already in use. Well, thats an interesting question. Under the hood, WebJob copies the binaries to a temporary directory before it actually start running. That is how it always ensure that your file never gets locked down in the process of redeployment.

Enough talk. Let me jump into our first simple demo to see how things actually wired up in webjobs.

I have just created a simple console application with only one line of code that prints the sentence Hello Webjobs! in the console and then I fly to the debug folder under bin directory which shows something like this –

AzureWebjobs_02_01

Now, my target is to zip the entire contents of this folder. This is the zip file that I am going to upload to azure webjobs. In my case the zip contains an executable file named HelloWebjobs which prints a statement in the console.

Then I head over to azure portal and click AzureWebjobs_02_02 from the left side panel which will show me a list of web apps running on my account. I click the web app(in my case its called webapplication39758) with which I want to integrate my webjob task. Then I scroll down the Settings plate unless I found the WebJobs option. Clicking this will open up a new blade. Click the AzureWebjobs_02_02+ button and this will open up a new blade.

AzureWebjobs_02_03

I provide a Name for my webjob , also configure the way it should run (i.e. On Demand or Continuous) and finally upload the zip file that I have created little earlier. Once I am done with all these settings I click the Create button to complete the process.

Once the webjob gets created successfully it will showed up in the WebJobs blade.

AzureWebjobs_02_04

Since it is a On Demand type job so I need to manually run it from the portal. I can do so by right clicking on the job and from the context menu I select Run.

AzureWebjobs_02_05

My notification icon got green saying the job ran successfully. I can always check the log file for further investigation. When I clicked the log  url it shows me an overview that states which executable file it tried to run and list of recent job status.  I can see if any of my webjob ever failed by having a look on the status segment.

AzureWebjobs_02_07

If for any reason I need to dig deeper I can do that easily. I can click any of the job listed here on the above dashboard and this will show me in depth information.

AzureWebjobs_02_06

This is pretty much everything!

Before wrapping this tutorial I want to emphasize one last thing. Be careful while creating the zip. While your zip can contain one to many files, you need to make sure it has a entry point which will initiate the webjob task. Recall from Intro to Azure Webjobs you can write scripts on different language. Which of your script/executable file will be considered as your entry point shall be automatically detected by the following logic

  • First look for a file called run.{supported extension} (first one found wins).
  • If not found look for any file with a supported extension.
  • If not found, this is not a runnable WebJob

Wrapping Up:

In azure you can setup your webjobs in varies way, this is just one of the process that I have shown here. My simple console app only print a sentence, but that is only for demonstration purpose. You should instead run your CPU intensive work. If you ever come across in a situation where you want some other type of file to be executed which is currently not supported in webjobs then for such situation the work around that you can follow is to create a run.cmd file and create your executor command there (powershell -Command run.ps).

Intro to Azure Webjobs

Azure Webjobs is a topic that I have been planning to write for a long time, even before I took “Azure Webjobs and Some Advanced Scenario” session on Northern University Bangladesh(NUB) but never actually got time to write it down here on blog.  I guess today is the day when finally I will write stuffs on Azure Webjobs. This is the first part of Azure Webjobs series.

Almost all the enterprise application that I have worked with so far has one thing in common i.e. they all need to do some processing. This processing might required to execute in the background process or it might required  to run on scheduled intervals. For say, one of my web app requirement could be squeezing the image size while another web app requirement could be deleting the log files at specific time interval. As you can see the processing differs based on the business logic.  But that is not the point I want you to focus. Instead I would like to shed on the fact that all the web app has a common requirement and that is they need to perform some operation.  The common and easiest solution I could come up with such scenario is to build a simple command line app and then schedule it on windows scheduler. This would definitely works but I don’t want my client to run a server just for this simple purpose!  This is where Azure Webjobs come in to action.

So, What is Azure Webjobs?

In simple words, this is just a scheduler! But it runs as PaaS that means you don’t have to sweat the underlying infrastructure any more, you just call it and handover your task, azure will handle the rest for you. If you ask me what could be the selling tagline for this feature.  I would say “This is damn easy to implement and it comes with so many features and extensions that worth it to implement in any enterprise grade web apps.” It was codenamed SimpleBatch.

AzureWebjobs_basicArchitecture_01

Following is a list of task for which Azure Webjobs is the perfect candidate :

  • Image processing or other CPU-intensive work.
  • Queue processing.
  • RSS aggregation.
  • File maintenance, such as aggregating or cleaning up log files.
  • Other long-running tasks that you want to run in a background thread, such as sending emails.

But why do I use Azure Webjobs?

If you are like me, probably you won’t start using it unless you find real value in it. Let me tell you awesomeness of azure webjobs and there by help you decide if you really wants to check this out.

  • Its Free : Why buy a cow if you can get milk for free? Thats the point. Azure Webjobs comes in no cost. Plus using Azure Webjobs means you don’t have to take the headache to manage the underlying VM.
  • Automatic Queue triggered : This is something I like most about Azure Webjobs. It automatically triggered each time some new stuffs appears.  Following is a snapshot that shows how easily we can configure queue trigger system.
    QueueProcessing_01
  • Resilient to errors : The word resilient mean something that spring back into its original shape after bending.  While executing azure webjobs if a particular messages (its content) causes a function to fail then instead of deleting the queue message its picked up again and tries five times to perform the task, if it still can’t process the task then the message moved to poison queue.
    Ressilent
  • Work in parallel & Auto Scale out : It can pick and process maximum 16 message from a queue at a time ! Plus it scales out with your web app automatically.  Obviously you may not like this default behavior, me neither.
  • Unlimited in number :  Inside an web app you can fire up as much webjobs you want. But bear in mind even though webjobs does not cost you money but it uses your web apps resource. So, its good practice to keep the number limited.
  • Auto Deserialize : I already told you the awesomeness of azure webjobs i.e. automatic queue triggered feature. Now, here is another one. It can sense the response and does auto deserialization 🙂
  • Easy tracking from Portal : Portal experience for webjobs is superb. You can easily create and check the status of your webjobs. In fact if you want you can check the full stack trace and download it for further use.

Now that you are impressed, you might be wondering what sorts of expertise you need to have.

To be honest, You really don’t have to learn anything new! Azure webjobs support a large number of executable and scripts. If you can write code on any of the following you will be good to go with webjobs.

  • .exe – .NET assemblies compiled with the WebJobs SDK
  • .cmd, .bat, .exe (using windows cmd)
  • .sh (using bash)
  • .php (using php)
  • .py (using python)
  • .js (using node)
  • .ps1 (using powershell)
  • .jar (using java)
  • .fsx (using the F# fsi.exe interactive compiler)
  • project.json (using Dnx)

Wrapping Up:

We often run into situation when we want to take something simple and do it n times. When such situation arises i.e. you need to do cronjob/batch processing, you should consider implementing WebJobs as it makes developing, running and scaling easier. In the upcoming tutorial I will cover Webjobs in details.

Reference :

.Net wrapper for MailChimp v3.0

mailchimp_logo_2

A lot of you probably have already used MailChimp to market your product on the internet. I have used it several times for my client and portal experience is really a hassle free. For those of you who don’t know what mailchimp is let me simply state it in one line –

“MailChimp is a simple email marketing software (EMS) which gives you a number of easy options for designing, sending and saving templates of your emails.”

In the mid of 2015, MailChimp have officially released version 3.0 API and they start to deprecate their earlier version. Since they were deprecating their earlier version the only stable .Net wrapper for v2.0 also start losing its light. I followed danesparza on his github and he was planning to work for v3.0 but did not mention any date. I thought to give a pull request but then I realize mailchimp v2.0 is a weird hybrid of RESTful and RPC, almost all of their API interaction has changed in v3.0 . So, its better to design the whole wrapper from scratch. I also came across to their official blog where Pete Holiday says a good api should not need an sdk which clearly gave hint that they are not going to offer any wrapper/sdk for any language!

So, I Did one for .Net  😀

Its not fully done yet, but most of the features that you might need in your day to day business is already implemented. The best part is, it is open source in github here, so you can have a look on the code base and send me a pull request any time you want 🙂

You can fetch campaign analytic report, can manage your Template,  can create and send campaign, can automate the workflow. Pretty much all the features are already baked in with this wrapper, check the Read Me file  for details.

How to use it :

It’s dead simple. Fire up an console application and open the Package manager console and type:

mailchimp

Open the config file and add the following line.

<add key="MailChimpApiKey" value="--Your_MailChimp_APIKey--" />

Once you are done with this settings, you are good to use the entire wrapper 🙂

One question that you might be wondering with which class you are going to interact. Well, all class are prefixed with the word MailChimp. If you are using Visual Studio you can take the advantage of IntelliSense.

MailChimp_Intellisence

For example, if you want to work with Report then you should probably want to instantiate MailChimpReports class.

Following code demonstrate how you can Create a campaign, upload raw email template and send mail to its subscriber :

Disclaimer :

I have tried my best to make it a bug free, have checked the entire solution against test data and it passed all the time. But that really doesn’t ensure its fully bug free, this is because the data that I used to check the process was created by me! I have asked the mailchimp support team to provide test data/test account for this purpose and they said they don’t have anything like that! Anyway, great hanselman once said A Bug Report is a Gift.  So it is! If you find any bug feel free to create an issue. Please be specific about what’s not working.

Thats pretty much everything I could write about MailChimp.Api.Net. Buzz me in the comment section if you have any feedback.

 

Azure App Service Poster

Azure Websites is a Platform-as-a-Service (PaaS) that enables you to build, deploy and scale enterprise-grade web Apps in seconds. Since, its a PaaS that means you can literally focus on your application code, and let Azure take care of the infrastructure to scale and securely run it for you.

 Following is a diagram that shows the overall system design of Azure Websites, more specifically Azure App Service.

Poster_AppService_01

You can download this poster from here.