Azure DevOps Hidden Gems #6 – Use the Manual Intervention Task to Pause a Stage of the Release Pipeline

Posted by Graham Smith on September 17, 2019No Comments (click here to comment)

I've been working with Azure DevOps quite a lot recently (having used its predecessors for many years) and I'm constantly amazed by features I never knew existed or which I vaguely knew existed but hadn't fully appreciated. In this blog post series I'm attempting to shine a light on some of these hidden gems for the benefit of others. The full list of posts is here and if you have any suggestions for other posts please leave a comment!

If you've created a release pipeline in Azure DevOps you probably know that there is rich functionality for approvals and gates to control a deployment between stages of the pipeline. Approvals are as you would imagine: a requirement for one or more people to approve either that a release stage is allowed to proceed or that a release stage has completed successfully. Gates are slightly different. From the docs: "Gates allow you to configure automated calls to external services, where the results are used to approve or reject a deployment. You can use gates to ensure that the release meets a wide range or criteria, without requiring user intervention.":

That's all well and good for controlling the deployment between stages of a pipeline. But what if you need to control the flow within a stage of a pipeline?

A colleague and I had this requirement recently when designing a release pipeline to manage the creation and updating of resources in Azure using Terraform, the cross-platform infrastructure as code technology from HashiCorp. One of the useful features of Terrafrom is the ability to call a command that will work out which resources are going to be created, destroyed or updated for a given Terraform configuration. A bit like the -WhatIf parameter in PowerShell if you are familiar with that. The problem in our scenario is that we need to halt the pipeline part way through a stage so someone can look at what Terraform is about to do and decide if it makes sense, and abort the stage if it doesn't. There is a solution of course, and it's the Manual Intervention task.

The task is simplicity itself to use, although it does need to run in the context of an Agentless job. Don't forget to add this or you will search in vain for the Manual Intervention task:

With the Manual Intervention task added to the Agentless job it's just a matter of setting a few properties:

When the pipeline is running it halts at the Manual Intervention task and waits for an intervention to either Resume or Reject the release:

If you weren't aware of this task you might be tempted to split a stage in to two stages to handle this scenario. Whilst this would probably work to my mind it's messy and inelegant and you should certainly check out the Manual Intervention task first.

Hope this helps!

Cheers -- Graham

Azure DevOps Hidden Gems #5 – Only Download Artifacts Needed for Stages of a Release

Posted by Graham Smith on July 24, 2019No Comments (click here to comment)

I've been working with Azure DevOps quite a lot recently (having used its predecessors for many years) and I'm constantly amazed by features I never knew existed or which I vaguely knew existed but hadn't fully appreciated. In this blog post series I'm attempting to shine a light on some of these hidden gems for the benefit of others. The full list of posts is here and if you have any suggestions for other posts please leave a comment!

If you are as impatient as I am then builds and releases can never finish quickly enough, and consequently I am always delighted to find a potential optimisation. My jaw dropped as I read about this one—how come I don't remember ever having previously read about it or even seen it?

The optimisation relates to classic releases (ie ones comprising visual tasks), although there is an equivalent for YAML releases. By default all the artefacts of a build are downloaded for a classic release, but what if you don't need everything? Then your release is probably taking longer than necessary! The good news is as of Sprint 131 we've been able to select just the artefacts that are needed for each stage of the pipeline.

To achieve this, open a classic release for editing. Under Tasks select the first stage and click on Agent job (or whatever the Run on agent is called):

Now in the right-hand pane scroll down to Artifact download and click the down arrow to show all the artifacts from the build. Simply deselect the ones that aren't required:

The specifics above aren't really important, but for completeness this is my QA stage where I want to deploy a website and then run automated acceptance tests. I don't need anything else. The great thing though is that you can now repeat this for other stages. In my example the next stage is PRD where I'm only deploying the website to the live environment. I don't need the acceptance tests, so I can deselect them. Great!

You can find the official documentation here, which also links to the equivalent way to do this in YAML pipelines.

Hope this helps!

Cheers -- Graham

Azure DevOps Hidden Gems #4 – Understand Build Agents by Installing One Locally on Your Development Machine

Posted by Graham Smith on July 12, 2019No Comments (click here to comment)

I've been working with Azure DevOps quite a lot recently (having used its predecessors for many years) and I'm constantly amazed by features I never knew existed or which I vaguely knew existed but hadn't fully appreciated. In this blog post series I'm attempting to shine a light on some of these hidden gems for the benefit of others. The full list of posts is here and if you have any suggestions for other posts please leave a comment!

If you've ever examined the logs generated by the agents in Azure Pipelines that do all the actual work you might have puzzled over what exactly is going on behind the scenes as your code is built and deployed. I know I have! We can see from build and release tasks that there are variables such as $(Build.ArtifactStagingDirectory) and (System.DefaultWorkingDirectory) that point to folders where things are happening and that it's all taking place in a folder hierarchy with seemingly cryptic folder names such as D:\a\1\s orD:\a\r1\a. But what exactly is happening in all these different folders?

If you are using Microsoft-hosted agents then they are pretty much black boxes and there is no way to peel back the covers and see what's going on. You can though output a list of all the variables and their values—see below. If you are using Self-hosted agents and you have appropriate permissions to remote to the server then you might have better luck in being able to see what's going on but if your server is a critical part of your build and release process or maybe it's a headless server or perhaps the agent is running in a docker container, then maybe it's not a good idea to go poking around or perhaps there are extra hurdles that you don't want to contend with.

A simple answer to this is to install an agent on your local machine. You can then play around to your heart's content safe in the knowledge that you have full visibility of what's happening and that you won't break a critical system. The process is pretty straightforward as follows:

  1. Create a dedicated Agent Pool in Azure DevOps at Organization Settings > Pipelines > Agent Pools > New agent pool.
  2. From the same location download the agent for your OS.
  3. Create a folder (such as c:\build-agent if you are on Windows) and unzip the contents of the agent download to this folder.
  4. Follow the instructions for configuring the agent which are available for Linux, macOS and Windows. Don't forget to choose the Agent Pool you created earlier and run the agent as a service as recommended.

Those steps are all that's required to get an agent up-and-running. Next up is to start using the agent to build and perhaps release an application. Chances are that you have a test project handy but if not it's quick to create one and get it configured for build—I usually create an ASP.NET Core application. I won't go through that process here except to say that whether you use something that already exists or you create something from scratch obviously you need to configure your build (and release) to use the Agent Pool you created earlier.

You will also need the tools that are used to build (and release) your application installed locally on your workstation. In the .NET world, if you have Visual Studio installed then you've probably got everything you need for a simple demo application. However if you are using any specialist tools such as Selenium for automated tests then there will be more to do. Exactly what is obviously tool specific, but you can get an idea from the Microsoft-hosted agents. For example, if you need chromedriver.exe then by looking at the Details tab of one of the hosted agents you can see that the path of chromedriver.exe is set by an environment variable called ChromeWebDriver:

In this case all you need to do is create a folder, copy chromedriver.exe to the folder and create a system environment variable to point to the folder. (You might have to reboot for the new variable to be recognised.)

With build (and perhaps release) configured you can now poke around in the folder structure of your agent to see exactly what is happening and where. A great diagnostic tip for any build or release is to output all the environment variables and their values to the logs. On Windows simply create a command line task and have it execute cmd /k set. On Linux use printenv | sort with a Bash script. I use this technique as a standard component of builds and releases and if you are using Microsoft-hosted agents printenv | sort works universally as presumably on Windows agents there is some sort of PowerShell alias at work.

Hope this helps!

Cheers -- Graham

Azure DevOps Hidden Gems #3 – Pull Request Validation Builds AND Releases

Posted by Graham Smith on July 4, 2019No Comments (click here to comment)

I've been working with Azure DevOps quite a lot recently (having used its predecessors for many years) and I'm constantly amazed by features I never knew existed or which I vaguely knew existed but hadn't fully appreciated. In this blog post series I'm attempting to shine a light on some of these hidden gems for the benefit of others. The full list of posts is here and if you have any suggestions for other posts please leave a comment!

If you are using git in Azure Repos you can protect a branch (master for example) with a branch policy that forces any changes to master to come in via a pull request to merge code from another branch. Branch policies have a fantastic wealth of options, and whilst they are definitely a gem I don't think they are exactly hidden:

One of the options available from Protect this branch is the ability to run a validation build against an ad hoc merge of the source and destination branches. This allows the proposed merge to be subjected to unit tests and anything else you might have in place to help with code quality. Typically you'll want to use the build that is normally run as part of the deployment pipeline, but of course not all tasks will need to run—there's probably no point in deploying artifacts for a validation build for example. This is where my previous tip comes in to play—the ability to run tasks conditionally according to custom conditions.

The ability to have a proposed merge validated by a build is great, but there's more! It's also possible to extend this concept to one or more stages of the release pipeline. For example, if the first stage of your release pipeline is configured to run automated acceptance tests you can have these run against the proposed merge following a successful validation build. Brilliant!

You can find the instructions for configuring validation releases here and a great walkthrough of how to configure an end-to-end scenario by Microsoft's Olivier Léger here. I've used the validation build and release feature and I love it, so do give it a try if it's a fit for your scenario.

Hope this helps!

Cheers -- Graham

Versioning .NET Core Assemblies in Azure DevOps isn’t Straightforward (and Probably Won’t be in Other CI/CD Tools Either)

Posted by Graham Smith on June 26, 2019No Comments (click here to comment)

As part of ongoing work to enhance an existing Azure DevOps CI/CD pipeline that builds and deploys an ASP.NET Core application I thought I'd spend a pleasant 5 minutes versioning the .NET Core assemblies with the pipeline's build number. A couple of hours and 20+ test builds later...

Out of the box, creating a new build in Azure Pipelines using the ASP.NET Core template in the classic editor results in five tasks of which four are concerned with dotnet commands:

A quick look at the documentation for dotnet build and then this awesome blog post that explains the dizzying array of options and it's pretty clear that adding /p:Version=$(Build.BuildNumber) as a command line parameter to dotnet build should suffice as a good starting point. Except it didn't, with File version and Product version stubbornly remaining at their default values:

I established that /p:Version= works fine from a command line, so what's going on? After a bit of research and testing I discovered that unless you tell it otherwise dotnet publish (and dotnet test for that matter) compiles the application before doing its thing of publishing files to a folder. The way the Azure Pipelines tasks are configured means that dotnet publish is effectively cancelling out the effect of dotnet build. (And since dotnet test also cancels out out the effect of dotnet build leaves me wondering what is the point of including dotnet build in the first place?) As part of this research I also discovered that build, test and publish also do a restore unless told otherwise, again making me wonder what the point of the Restore task is? So out of the box then it seems like the four .NET Core tasks are resulting in lots of duplication and for someone like me the cause of head-scratching as to why assembly versioning doesn't work.

So based on a few hours of testing here is what I think the arguments of the different tasks need to be (for visual tasks or as YAML) to avoid duplication and implement assembly versioning.

Firstly, if you want to include an implicit Restore task:

  • build = --configuration $(BuildConfiguration) --no-restore /p:Version=$(Build.BuildNumber)
  • test = --configuration $(BuildConfiguration) --no-build
  • publish = --configuration $(BuildConfiguration) --output $(Build.ArtifactStagingdirectory) --no-build

Secondly, if you want to omit an explicit Restore task:

  • test = --configuration $(BuildConfiguration)
  • publish = --configuration $(BuildConfiguration) --output $(Build.ArtifactStagingdirectory) /p:Version=$(Build.BuildNumber)

In the first version build creates the binaries which are then used by test and publish, with the --no-build switch implicitly setting the --no-restore flag. I haven't tested it but that presumably means that --configuration $(BuildConfiguration) for test and publish is redundant.

Update A friend and former colleague Tweeted that --configuration is still needed for test and publish:

In the second version test and publish both create their own sets of binaries. (Is that the right thing to do from a purist CI/CD perspective? Maybe, maybe not.)

I did my testing on a Microsoft-hosted build agent and whilst it felt like both options above were quicker than the default settings I can't be certain without rigorous testing on a self-hosted agent with no other load. Either way though, it feels good to have optimised the tasks and I finally got assembly versioning working. Are there other optimisations? Have I missed something? Please leave a comment!

Cheers -- Graham

Azure DevOps Hidden Gems #2 – Run Build or Release Tasks According to Custom Conditions

Posted by Graham Smith on June 24, 2019No Comments (click here to comment)

I've been working with Azure DevOps quite a lot recently (having used its predecessors for many years) and I'm constantly amazed by features I never knew existed or which I vaguely knew existed but hadn't fully appreciated. In this blog post series I'm attempting to shine a light on some of these hidden gems for the benefit of others. The full list of posts is here and if you have any suggestions for other posts please leave a comment!

Imagine this scenario: you have a code branch on which you want to run an all-singing, all-dancing build packed full of tasks, and another branch where you only want to run a subset of those tasks. Cloning the build and stripping out unwanted tasks to create a second build is the answer, right? Not necessarily! It turns out that most tasks can be set to run conditionally, according to criteria that you specify.

To configure this feature (I'm illustrating using visual tasks but there is a YAML equivalent) open the task and head over to Control Options. For Run this task select Custom conditions and then enter your conditions in Custom condition:

In the build task example above the task will only run if the build is succeeding and the build is running against the master branch. For any other branches it will be skipped.

To understand the full capabilities of this fantastic feature you should take a look at the Conditions overview page and then the Expressions page which has a full guide to the conditions syntax. I'll blog soon about a specific scenario where this feature is just exactly what is needed to avoid creating a second build and the potential maintenance issue that causes.

Hope this helps!

Cheers -- Graham

Azure DevOps Hidden Gems #1 – Use Secure Files in a Build or Release Pipeline

Posted by Graham Smith on June 19, 2019No Comments (click here to comment)

I've been working with Azure DevOps quite a lot recently (having used its predecessors for many years) and I'm constantly amazed by features I never knew existed or which I vaguely knew existed but hadn't fully appreciated. In this blog post series I'm attempting to shine a light on some of these hidden gems for the benefit of others. The full list of posts is here and if you have any suggestions for other posts please leave a comment!

If you've created a Build or Release pipeline in Azure DevOps you've probably used the Variables feature to store either plain text or secret variables that can be passed in to the build or pipeline:

This works well for plain text, but what if you have more complicated requirements, such as secrets contained in a file that can't simply be copied as plain text in to a standard variable? Sure, there are solutions external to Azure DevOps that you could use (Azure Key Vault for example) but you could end up using a sledgehammer to crack a nut. No matter though, as Azure DevOps provides a solution through Secure Files. You can find this by navigating to Pipelines > Library and then clicking the Secure Files tab:

In the screenshot above I've used + Secure file to upload a file called config (which in this particular case is a file that contains credentials for connecting to an Azure Kubernetes Service cluster). Secure files are made available in the build or pipeline through the use of the Download Secure File task, which places the file in the $(Agent.TempDirectory) directory of the Azure Pipelines Agent. The file can then be used on a command line where a parameter is expecting a file, for example:

This is obviously a very specific example (an incomplete extract of a Bash script that is using kubectl to create secrets on a Kubernetes cluster) but hopefully you get the idea of how secret files can be used. Once the build or release has completed the file gets deleted—a good thing on a self-hosted agent although Microsoft-hosted agents are destroyed anyway after use.

Hope this helps!

Cheers -- Graham

A Better Way of Deploying a Dockerized Application to Azure Kubernetes Service Using Azure Pipelines

Posted by Graham Smith on January 21, 2019No Comments (click here to comment)

Throughout 2018 I wrote a mini blog post series aimed at providing specific and detailed guidance on how to create a CI/CD pipeline using VSTS/Azure DevOps to deploy a dockerized ASP.NET Core application to Azure Kubernetes Service (AKS):

Whilst the resulting solution works I wasn't entirely happy with several aspects and I've spent a great deal of time thinking and tinkering to come up with something better. In this blog post I explain what I wasn't happy with and how my new solution addresses most of my concerns. You don't necessarily need to read the posts above as I'm going to provide some context, but it will probably make things much clearer if you are planning to implement any of my suggestions.

The sample application I've been using to deploy to Kubernetes consists of the following components:

  • ASP.NET Core web application, that sends messages to a
  • NATS message queue service, which pushes messages to a
  • .NET Core message queue handler application, which saves messages to an
  • Azure SQL database

Apart from the database all the components run as docker containers. The container images are built in in an Azure Pipelines build pipeline and images pushed to an Azure Container Registry (ACR). An Azure Pipelines release pipeline then deploys the necessary services and deployments to AKS which causes the images to be pulled from ACR and instantiated as containers inside pods. My release pipeline consists of two environments: dat (developer automated test where automated acceptance tests might take place) and prd (production). That's just arbitrary of course and in a live scenario the pipeline can have whatever environments are needed.

My sample application is called MegaStore and you can find the code on GitHub here. In the rest of this post I explain my areas of concern and how I addressed them.

Azure Pipelines Tasks

Whilst there is no doubt that Azure Pipelines Tasks are great for quickly building a pipeline and definitely make it easier for those less familiar with the technology behind a task to get started, I now see some tasks as more of a curse than a blessing. I've particularly taken issue with tasks that manipulate a command line application (such as docker or kubectl) and which results in the task becoming something of a Swiss Army Knife task. Why have I taken issue? There are several reasons, some specific to the Swiss Army Knife variety and some of tasks in general:

  • There is often a need to set mandatory fields in ‘Swiss Army Knife' tasks even though those parameters will not be used by the chosen sub-command. Where there are multiple instances of the same task in use this becomes very tedious and is a potential maintenance problem when something changes. (Yes, I know tasks can be cloned but this doesn't make me any happier.)
  • Tasks by their nature only allow you to do what they have been coded to do and you can sometimes find yourself in a blind alley. For example, at the time of writing the only way I know of updating an existing Kubernetes ConfigMap without deleting it first and re-creating it is with a piped command, for example:

    Running a command such as this isn't possible with the current Deploy to Kubernetes Azure DevOps task, which is very limiting.
  • Speaking of command lines, my next issue is that tasks abstract you from what is actually going on behind the scenes. For simple tasks such as copying files this might be fine, however I've become frustrated at the way tasks such as Docker or Deploy to Kubernetes ‘hide' what they are doing, and the way that makes fine-tuning that little bit harder. Additionally, for me it's also a lost learning opportunity—a missed chance to learn the full syntax of a command because the task is constructing it on your behalf.
  • Another big issue is that tasks such as Docker or Deploy to Kubernetes offer nothing in the way of code usability, and break the DRY principle in multiple dimensions (ie there is scope for repetition within an environment and also across environments). To illustrate, the release pipeline in my 2018 mini blog series consisted of no fewer than 30 Deploy to Kubernetes tasks across two environments, resulting in a great deal of repetition.
  • Finally, the use of tasks in the current version of Azure Pipelines releases means that you don't have your ‘code' under proper version control. I know there are changes coming that will help to address this, and whilst they will be welcome I think there is an opportunity to do better.

So what's my solution to all this? Very simply, get rid of multiple Swiss Army Knife tasks and implement Bash scripts running from a single Bash task. I started off by using the Inline script feature of Bash tasks but this didn't help with getting code in to version control and I also quickly realised that there were big code reusability opportunities to be had across environments by using File Path scripts. By using Bash scripts stored in the repo I solved all the issues mentioned above and in the case of the release portion of the pipeline I reduced the number of tasks from 15 in each environment to two! What follows are the techniques I used to achieve this for the Docker builds and Kubernetes deployments.

Converting Docker builds to use a Bash script was reasonably straightforward so I'll start by discussing the first problem I encountered when converting Deploy to Kubernetes tasks to Bash scripts, which was how to authenticate to Kubernetes. Tasks rely on the creation of a Kubernetes service connection (Project Settings > Service connections) and I'd been using the Kubeconfig version which involves pasting in the contents of the Kubeconfig file that gets created (if you run the appropriate command) when you set up an AKS cluster:

By tracing the logging output of the Deploy to Kubernetes tasks I could see what was happening: a Kubeconfig file was being saved to disk and referenced in a kubectl command using the --kubeconfig parameter that points to the file on disk. I could successfully pass the file in from an Artifact as a proof of concept but how to store the Kubeconfig contents securely and create the file dynamically? The obvious choice was a secret variable however that didn't work because it destroyed the Kubeconfig formatting which is important in the re-hydrated file on disk. After a lot of fiddling I finally turned to LoECDA who are super-responsive via Twitter, and very quickly the suggestion came back to try using Secure files (Pipelines > Library > Secure files). This worked perfectly: a file is first uploaded to the Secure files area and this is then available for use using the Download Secure File task. The file is downloaded in to a temporary folder which can be referenced as the $AGENT_TEMPDIRECTORY variable in a Bash script. Great!

Next up was sorting out the practicalities of using Bash scripts in Bash tasks. I created a deployment (dep) folder in the repo to hold the scripts and then arranged for this folder to be available as an Artifact created directly from the GitHub repo:

I used VS Code to create the Bash files however in order for the file to be executed as a Bash script it needs its permissions setting to make it executable (chmod +x). This needs to be done from a Linux environment and there are several possibilities for achieving this including Windows Subsystem for Linux if you are on Windows 10. I chose to go with Azure Cloud Shell, which can be configured to run either a Bash or a PowerShell command line in the cloud! Once that was configured it was a case of cloning my repo, navigating to the dep folder and running chmod +x some-filename-sh. There's no GUI in Azure Cloud Shell so it does involve using git commands to push the changes back to GitHub. If this is new to you then git add *git commit -m "Commit message" and git push origin master are what you need. To authenticate you'll likely need to use a personal access token unless you go to the bother of setting up SSH. It gets to be a bit of a pain having to enter credentials every time you want to push to GitHub however the git config credential.helper store command will save credentials across Azure Cloud Shell sessions to make life easier.

Finding out what commands needed to be executed in the Bash scripts required a bit of detective work, and involved a combination of understanding what the task was attempting to accomplish and then looking at the build or release logs to see the actual output. With the basic command figured out this exercise offered the opportunity to do a bit of fine tuning. For example, I'd been tagging my docker images with the latest tag but it turns out that this isn't a great idea for release pipelines. By writing the actual command myself I was able to get exactly what I wanted.

I describe how I organised the Bash scripts to move away from a monolithic pipeline below. In this section I want to describe the tips and tricks I used to actually write the Bash scripts. Generally, the scripts make heavy use of variables to make them applicable to all release environments, however there are some essential things to know:

  • Variables created as part of Azure DevOps pipelines can be used as variables (ie passed in to a script) however with the exception of secrets they are also created as environment variables which are available directly in scripts. This means that a variable created as MyVariable is available as $MYVARIABLE directly in a Bash script (in Bash scripts the variable is really a constant which convention dictates should be in upper case and any periods need converting to underscores to ensure valid syntax).
  • Variables created as part of Azure DevOps pipelines can have the same name as long as they are scoped to a different environment. So you can have two variables called MyVariable with different values for each environment and simply refer to $MYVARIABLE in the Bash script, ie no need to pass $MYVARIABLE in as a parameter to the script for different environments.
  • As mentioned above, secrets are not created as environment variables and must be passed in to a script via the Arguments field, and in the script a variable is declared to accept the incoming parameter. Important: as of the time of writing a secret needs to be passed in to the Argument field as $(MYSECRET) ie with parentheses around the actual parameter name. If you omit the parentheses the secret is not passed in. A non-secret parameter doesn't require parentheses and I have queried whether this is a a bug here.
  • Later in this post I explain how I break up a monolithic pipeline in to multiple pipelines, which results in the same variables being needed in different pipelines. By using Variable Groups I was able to avoid repeated variable declarations and manage many variables from just one location.
  • In addition to variables that are created manually, built-in variables are also available as environment variables in the script. The ones I've used are $AGENT_TEMPDIRECTORY to define the download location of the Kubeconfig file from the Secure files area, $RELEASE_ENVIRONMENTNAME to refer to the environment (ie dat or prd) and also $BUILD_BUILDNUMBER used to tag docker images with a unique build number in the build process and then to refer to them by their unique name in the release. However, there are many built-in variables available to use—see here for details but remember that for use in Bash scripts you should change text to uppercase and must replace periods with an underscore.

I'm not a Bash scripting expert and I'm sure my scripts would be considered very rudimentary. The great thing though is that you can do whatever you like now the code is a script. Possibilities might include adding error handling or refactoring further using functions. There's potential to really go to town here.

Monolithic Pipeline

At the time of writing this article in early 2019 there aren't that many blog post examples of implementing a CI/CD pipeline to deploy an application to Kubernetes. Furthermore, the posts that do exist tend, not unreasonably, to use a simplistic application scenario to illustrate the concepts. Typically, this involves deploying the whole application as part of a single pipeline, and indeed this is the route I took with my 2018 blog post mini series. However, it became quickly apparent to me that this is an unsatisfactory arrangement for two main reasons:

  • Just one change to one of the application components would cause all the components of the application to be redeployed (or more correctly the parts of the application that have their docker images built by the pipeline).
  • A change to the Kubernetes configuration would also trigger a redeployment of all of the application components. Sometimes this is necessary but often it's not.

These issues arise because the trigger for the build component of the pipeline is set as the root of the GitHub repo, so if anything changes in the repo a build is triggered. Clearly not an optimal situation.

My solution to this problem is to divide the monolithic pipeline in to multiple pipelines that correspond to the individual components of the overall application. Then with a bit of refactoring of the codebase it's possible to use a very nifty feature of Azure Pipelines that allows a build to be triggered from one or more specific folders (or files for that matter) in the repo, ie a much more granular solution.

One complication that I had to cater for is that the pipeline isn't just building docker images and marshalling them in to the Kubernetes cluster: additionally, the pipeline is configuring Kubernetes elements such as Namespaces, Secrets and ConfigMaps.

Through the use of Bash scripts as described above the number of tasks needed is drastically reduced: just one Bash task for the builds and two tasks for releases (a Download Secure File task to copy the kubeconfig file to disk and a Bash task to host the bash script). All scripts are Namespace/environment aware.

In terms of Azure Pipelines build and release pipelines my current CI/CD solution is as follows:


This is a release that is not associated with a build and its sole purpose is to configure a Kubernetes Namespace in preparation for the deployment of the application. As such, this component is only intended to be run to either initialise a new Kubernetes cluster or (rarely) if one of the configuration items needs to change (in which case elements of the application will likely have to be redeployed for the configuration to be built in to the appropriate pods).

The configuration handled by megastore.init.release is as follows:

  • Creation of a Namespace for a corresponding Azure Pipelines environment.
  • Creation (or update) of ACR credentials (as a specialised Secret) that allow Deployments to pull docker images from ACR.
  • Creation (or update) of the message queue URL as a ConfigMap.
  • Creation (or update) of the Application Insights instrumentation key as a ConfigMap.

This configuration is handled by


This is another release that is not associated with a build, and in this case the requirement is to deploy the NATS message queue service. The absence of a build is due to the docker image being pulled from Docker Hub. The downside of not having a build associated with the release is that if any of the NATS configuration changes the release needs to be triggered manually. I see this as an infrequent requirement though. The message queue service doesn't have any dependencies on any other part of the application and so is the first component to be deployed following the initial Kubernetes configuration.

The configuration handled by megastore.message-queue.release is as follows:

  • Deployment of the Kubernetes Service for the message queue.
  • Deployment of the Kubernetes Deployment for the message queue.

This configuration is handled by and megastore.savesalehandler.release

This build and linked release are responsible for deploying a new version of the .NET Core message queue handler application which receives message from the message queue and saves them to an Azure SQL database. The docker image is built and uploaded to ACR using this generic Bash script. This in turn triggers the megastore.savesalehandler.release which deals with the following configuration:

  • Creation (or update) of the database connection string as a Secret.
  • Deployment of the Kubernetes Deployment for the message queue handler component.
  • Update the image for the Deployment to the latest version using the unique tag for the build that triggered the release.

This configuration is handled by The build is triggered through the Azure Pipelines Path filters feature:

Using the Path filters feature ensures that the build will only be triggered for continuous integration if a file in the specified folder is changed. and megastore.web.release

This build and linked release are responsible for deploying a new version of the ASP.NET Core web application which sends messages to the message queue service. As with the message queue handler, the docker image is built and uploaded to ACR using this generic Bash script. The build triggers the megastore.web.release which deals with the following configuration:

  • Creation (or update) of the ASPNETCORE_ENVIRONMENT environment variable as a ConfigMap.
  • Deployment of the Kubernetes Deployment for the web component.
  • Deployment of the Kubernetes Service for the web component.
  • Update the image for the Deployment to the latest version using the unique tag for the build that triggered the release.

This configuration is handled by and once again the build is triggered through the Azure Pipelines Path filters feature:

As before, using the Path filters feature ensures that the build will only be triggered for continuous integration if a file in the specified folder is changed.

And Finally...

In breaking down a monolithic pipeline in to multiple pipelines I exposed the problem of what to do with the shared helper library of functions that is use both by the megastore.web and megastore.savesalehandler components, because if this code changes one or sometimes both components will need redeploying. I think the answer is that helper libraries like these do not belong in the Visual Studio solution and instead should be developed separately and distributed and referenced as NuGet packages.

One of my aspirations is to get as much pipeline configuration in the GitHub repo as possible and you might well ask why I'm not using yaml files. Apart from the fact that I just haven't had time to look at this in detail yet, at the time of writing it's only a partial solution as it's only available for the build portion of the pipeline. This will change hopefully later this year when the release portion of the pipeline is supported, and at that point I'll make the switch.

That's it for now! Whether you are deploying to AKS or somewhere else I hope this post has provided you with ideas to supercharge your Azure DevOps pipelines.

Cheers -- Graham

Deploy a Dockerized ASP.NET Core Application to Azure Kubernetes Service Using a VSTS CI/CD Pipeline: Part 4

Posted by Graham Smith on September 11, 2018No Comments (click here to comment)

In this blog post series I'm working my way through the process of deploying and running an ASP.NET Core application on Microsoft's hosted Kubernetes environment. These are the links to the full series of posts to date:

In this post I take a look at application monitoring and health. There are several options for this however since I'm pretty much all-in with the Microsoft stack in this blog series I'll stick with the Microsoft offering which is Azure Application Insights. This posts builds on previous posts, particularly Part 3, so please do consider working through at least Part 3 before this one.

In this post, I continue to use my MegaStore sample application which has been upgraded to .NET Core 2.1, in particular with reference to the csproj file. This is important because it affects the way Application Insights is configured in the ASP.NET Core web component. See here and here for more details. All my code is in my GitHub repo and you can find the starting code here and the finished code here.

Understanding the Application Insights Big Picture

Whilst it's very easy to get started with Application Insights, configuring it for an application with multiple components which gets deployed to a continuous delivery pipeline consisting of multiple environments running under Kubernetes requires a little planning and a little effort to get everything configured in a satisfactory way. As of the time of writing this isn't helped by the presence of Application Insights documentation on both and (ASP.NET Core | Kubernetes) which sometimes feels like it's conflicting, although it's nothing that good old fashioned detective work can't sort out.

The high-level requirements to get everything working are as follows:

  1. A mechanism is needed to separate out telemetry data from the different environments of the continuous delivery pipeline. Application Insights sends telemetry to a ‘bucket' termed an Application Insights Resource which is identified by a unique instrumentation key. Separation of telemetry data is therefore achieved by creating an individual Application Insights Resource, each for the development environment and the different environments of the delivery pipeline.
  2. Each component of the application that will send telemetry to an Application Insights Resource needs configuring so that it can be supplied with the instrumentation key for the Application Insights Resource for the environment the application is running in. This is a coding issue and there are lots of ways to solve it, however in the MegaStore sample application this is achieved through a helper class in the MegaStore.Helper library that receives the instrumentation key as an environment variable.
  3. The MegaStore.Web and MegaStore.SaveSaleHandler components need configuring for both the core and Kubernetes elements of Application Insights and a mechanism to send the telemetry back to Azure with the actual name of the component rather than a name that Application Insights has chosen.
  4. Each environment needs configuring to create an instrumentation key environment variable for the Application Insights Resource that has been created for that environment. In development this is achieved through hard-coding the instrumentation key in docker-compose.override.yaml. In the deployment pipeline it's achieved through a VSTS task that creates a Kubernetes config map that is picked up by the Kubernetes deployment configuration.

That's the big picture—let's get stuck in to the details.

Creating Application Insights Resources for Different Environments

In the Azure portal follow these slightly outdated instructions (Application Insights is currently found in Developer Tools) to create three Application Insights Resources for the three environments: DEV, DAT and PRD. I chose to put them in one resource group and ended up with this:

For reference there is a dedicated Separating telemetry from Development, Test, and Production page in the official Application Insights documentation set.

Configure MegaStore to Receive an Instrumentation Key from an Environment Variable

As explained above this is a specific implementation detail of the MegaStore sample application, which contains an Env class in MegaStore.Helper to capture environment variables. The amended class is as follows:

Obviously this class relies on an external mechanism creating an environment variable named APP_INSIGHTS_INSTRUMENTATION_KEY. Consumers of this class can reference MegaStore.Helper and call Env.AppInsightsInstrumentationKey to return the key.

Configure MegaStore.Web for Application Insights

If you've upgraded an ASP.NET Core web application to 2.1 or later as detailed earlier then the core of Application Insights is already ‘installed' via the inclusion of the Microsoft.AspNetCore.All meta package so there is nothing to do. You will need to add Microsoft.ApplicationInsights.Kubernetes via NuGet—at the time of writing it was in beta (1.0.0-beta9) so you'll need to make sure you have told NuGet to include prereleases.

In order to enable Application Insights amend BuildWebHost in Program.cs as follows:

Note the way that the instrumentation key is passed in via Env.AppInsightsInstrumentationKey from MegaStore.Helper as mentioned above.

Telemetry relating to Kubernetes is enabled in ConfgureServices in Startup.cs as follows:

Note also that a CloudRoleTelemetryInitializer class is being initialised. This facilitates the setting of a custom RoleName for the component, and requires a class to be added as follows:

Note here that we are setting the RoleName to MegaStore.Web. Finally, we need to ensure that all web pages return telemetry. This is achieved by adding the following code to the end of _ViewImports.cshtml:

and then by adding the following code to the end of the <head> element in _Layout.cshtml:

Configure MegaStore.SaveSaleHandler for Application Insights

I'll start this section with a warning because at the time of writing the latest versions of Microsoft.ApplicationInsights and Microsoft.ApplicationInsights.Kubernetes didn't play nicely together and resulted in dependency errors. Additionally the latest version of Microsoft.ApplicationInsights.Kubernetes was missing the KubernetesModule.EnableKubernetes class described in the documentation for making Kubernetes work with Application Insights. The Kubernetes bits are still in beta though so it's only fair to expect turbulence. The good news is that with a bit of experimentation I got everything working by installing NuGet packages Microsoft.ApplicationInsights (2.4.0) and Microsoft.ApplicationInsights.Kubernetes (1.0.0-beta3). If you try this long after publication date things will have moved on but this combination works with this initialisation code in Program.cs:

Please do note that this a completely stripped down Program class to just show how Application Insights and the Kubennetes extension is configured. Note again that this component uses the CloudRoleTelemetryInitializer class shown above, this time with the RoleName set to MegaStore.SaveSaleHandler. What I don't show here in any detail is that you can add lots of client.Track* calls to generate rich telemetry to help you understand what your application is doing. The code on my GitHub repo has details.

Configure the Development Environment to Create an Instrumentation Key Environment Variable

This is a simple matter of editing docker-compose.override.yaml with the new APP_INSIGHTS_INSTRUMENTATION_KEY environment variable and the instrumentation key for the corresponding Application Insights Resource:

Make sure you don't just copy the code above as the actual key needs to come from the Application Insights Resource you created for the DEV environment, which you can find as follows:

Configure the VSTS Deployment Pipeline to Create Instrumentation Key Environment Variables

The first step is to amend the two Kubernetes deployment files (megastore-web-deployment.yaml and megastore-savesalehandler-deployment.yaml) with details of the new environment variable in the respective env sections:

Now in VSTS:

  1. Create variables called DatAppInsightsInstrumentationKey and PrdAppInsightsInstrumentationKey scoped to their respective environments and populate the variables with the respective instrumentation keys.
  2. In the task lists for the DAT and PRD environments clone the Delete ASPNETCORE_ENVIRONMENT config map and Create ASPNETCORE_ENVIRONMENT config map tasks and amend them to work with the new APP_INSIGHTS_INSTRUMENTATION_KEY environment variable configured in the *.deployment.yaml files.

Generate Traffic to the MegaStore Web Frontend

Now the real fun can begin! Commit all the code changes to trigger a build and deploy. The clumsy way I'm having to delete an environment variable and then recreate it (to cater for a changed variable name) will mean that the release will be amber in each environment for a couple of releases but will hopefully eventually go green. In order to generate some interesting telemetry we need to put one of the environments under load as follows:

  1. Find the public IP address of MegaStore.Web in the PRD environment by running kubectl get services --namespace=prd:
  2. Create a PowerShell (ps1) file with the following code (using your own IP address of course):
  3. Run the script (in Windows PowerShell ISE for example) and as long as the output is white you know that traffic is getting to the website.

Now head over to the Azure portal and navigate to the Application Insights Resource that was created for the PRD environment earlier and under Investigate click on Search and then Click here (to see all data in the last 24 hours):

You should see something like this:

Hurrah! We have telemetry! However the icing on the cake comes when you click on an individual entry (a trace for example) and see the Kubernetes details that are being returned with the basic trace properties:

Until Next Time

It's taken my quite a lot of research and experimentation to get to this point so that's it for now! In this post I showed you how to get started monitoring your Dockerized .NET Core 2.1 applications running in AKS using Application Insights. The emphasis has been very much on getting started though as Application Insights is a big beast and I've only scratched the surface in this post. Do bear in mind that some of the NuGets used in this post are in beta and some pain is to be expected.

As I publish this blog post VSTS has had a name change to Azure DevOps so that's the title of this series having to change again!