Archives for Azure DevOps Hidden Gems

Azure DevOps Hidden Gems #4 – Understand Build Agents by Installing One Locally on Your Development Machine

Posted by Graham Smith on July 12, 2019No Comments (click here to comment)

I've been working with Azure DevOps quite a lot recently (having used its predecessors for many years) and I'm constantly amazed by features I never knew existed or which I vaguely knew existed but hadn't fully appreciated. In this blog post series I'm attempting to shine a light on some of these hidden gems for the benefit of others. The full list of posts is here and if you have any suggestions for other posts please leave a comment!

If you've ever examined the logs generated by the agents in Azure Pipelines that do all the actual work you might have puzzled over what exactly is going on behind the scenes as your code is built and deployed. I know I have! We can see from build and release tasks that there are variables such as $(Build.ArtifactStagingDirectory) and (System.DefaultWorkingDirectory) that point to folders where things are happening and that it's all taking place in a folder hierarchy with seemingly cryptic folder names such as D:\a\1\s orD:\a\r1\a. But what exactly is happening in all these different folders?

If you are using Microsoft-hosted agents then they are pretty much black boxes and there is no way to peel back the covers and see what's going on. You can though output a list of all the variables and their values—see below. If you are using Self-hosted agents and you have appropriate permissions to remote to the server then you might have better luck in being able to see what's going on but if your server is a critical part of your build and release process or maybe it's a headless server or perhaps the agent is running in a docker container, then maybe it's not a good idea to go poking around or perhaps there are extra hurdles that you don't want to contend with.

A simple answer to this is to install an agent on your local machine. You can then play around to your heart's content safe in the knowledge that you have full visibility of what's happening and that you won't break a critical system. The process is pretty straightforward as follows:

  1. Create a dedicated Agent Pool in Azure DevOps at Organization Settings > Pipelines > Agent Pools > New agent pool.
  2. From the same location download the agent for your OS.
  3. Create a folder (such as c:\build-agent if you are on Windows) and unzip the contents of the agent download to this folder.
  4. Follow the instructions for configuring the agent which are available for Linux, macOS and Windows. Don't forget to choose the Agent Pool you created earlier and run the agent as a service as recommended.

Those steps are all that's required to get an agent up-and-running. Next up is to start using the agent to build and perhaps release an application. Chances are that you have a test project handy but if not it's quick to create one and get it configured for build—I usually create an ASP.NET Core application. I won't go through that process here except to say that whether you use something that already exists or you create something from scratch obviously you need to configure your build (and release) to use the Agent Pool you created earlier.

You will also need the tools that are used to build (and release) your application installed locally on your workstation. In the .NET world, if you have Visual Studio installed then you've probably got everything you need for a simple demo application. However if you are using any specialist tools such as Selenium for automated tests then there will be more to do. Exactly what is obviously tool specific, but you can get an idea from the Microsoft-hosted agents. For example, if you need chromedriver.exe then by looking at the Details tab of one of the hosted agents you can see that the path of chromedriver.exe is set by an environment variable called ChromeWebDriver:

In this case all you need to do is create a folder, copy chromedriver.exe to the folder and create a system environment variable to point to the folder. (You might have to reboot for the new variable to be recognised.)

With build (and perhaps release) configured you can now poke around in the folder structure of your agent to see exactly what is happening and where. A great diagnostic tip for any build or release is to output all the environment variables and their values to the logs. On Windows simply create a command line task and have it execute cmd /k set. On Linux use printenv | sort with a Bash script. I use this technique as a standard component of builds and releases and if you are using Microsoft-hosted agents printenv | sort works universally as presumably on Windows agents there is some sort of PowerShell alias at work.

Hope this helps!

Cheers -- Graham

Azure DevOps Hidden Gems #3 – Pull Request Validation Builds AND Releases

Posted by Graham Smith on July 4, 2019No Comments (click here to comment)

I've been working with Azure DevOps quite a lot recently (having used its predecessors for many years) and I'm constantly amazed by features I never knew existed or which I vaguely knew existed but hadn't fully appreciated. In this blog post series I'm attempting to shine a light on some of these hidden gems for the benefit of others. The full list of posts is here and if you have any suggestions for other posts please leave a comment!

If you are using git in Azure Repos you can protect a branch (master for example) with a branch policy that forces any changes to master to come in via a pull request to merge code from another branch. Branch policies have a fantastic wealth of options, and whilst they are definitely a gem I don't think they are exactly hidden:

One of the options available from Protect this branch is the ability to run a validation build against an ad hoc merge of the source and destination branches. This allows the proposed merge to be subjected to unit tests and anything else you might have in place to help with code quality. Typically you'll want to use the build that is normally run as part of the deployment pipeline, but of course not all tasks will need to run—there's probably no point in deploying artifacts for a validation build for example. This is where my previous tip comes in to play—the ability to run tasks conditionally according to custom conditions.

The ability to have a proposed merge validated by a build is great, but there's more! It's also possible to extend this concept to one or more stages of the release pipeline. For example, if the first stage of your release pipeline is configured to run automated acceptance tests you can have these run against the proposed merge following a successful validation build. Brilliant!

You can find the instructions for configuring validation releases here and a great walkthrough of how to configure an end-to-end scenario by Microsoft's Olivier Léger here. I've used the validation build and release feature and I love it, so do give it a try if it's a fit for your scenario.

Hope this helps!

Cheers -- Graham

Azure DevOps Hidden Gems #2 – Run Build or Release Tasks According to Custom Conditions

Posted by Graham Smith on June 24, 2019No Comments (click here to comment)

I've been working with Azure DevOps quite a lot recently (having used its predecessors for many years) and I'm constantly amazed by features I never knew existed or which I vaguely knew existed but hadn't fully appreciated. In this blog post series I'm attempting to shine a light on some of these hidden gems for the benefit of others. The full list of posts is here and if you have any suggestions for other posts please leave a comment!

Imagine this scenario: you have a code branch on which you want to run an all-singing, all-dancing build packed full of tasks, and another branch where you only want to run a subset of those tasks. Cloning the build and stripping out unwanted tasks to create a second build is the answer, right? Not necessarily! It turns out that most tasks can be set to run conditionally, according to criteria that you specify.

To configure this feature (I'm illustrating using visual tasks but there is a YAML equivalent) open the task and head over to Control Options. For Run this task select Custom conditions and then enter your conditions in Custom condition:

In the build task example above the task will only run if the build is succeeding and the build is running against the master branch. For any other branches it will be skipped.

To understand the full capabilities of this fantastic feature you should take a look at the Conditions overview page and then the Expressions page which has a full guide to the conditions syntax. I'll blog soon about a specific scenario where this feature is just exactly what is needed to avoid creating a second build and the potential maintenance issue that causes.

Hope this helps!

Cheers -- Graham

Azure DevOps Hidden Gems #1 – Use Secure Files in a Build or Release Pipeline

Posted by Graham Smith on June 19, 2019No Comments (click here to comment)

I've been working with Azure DevOps quite a lot recently (having used its predecessors for many years) and I'm constantly amazed by features I never knew existed or which I vaguely knew existed but hadn't fully appreciated. In this blog post series I'm attempting to shine a light on some of these hidden gems for the benefit of others. The full list of posts is here and if you have any suggestions for other posts please leave a comment!

If you've created a Build or Release pipeline in Azure DevOps you've probably used the Variables feature to store either plain text or secret variables that can be passed in to the build or pipeline:

This works well for plain text, but what if you have more complicated requirements, such as secrets contained in a file that can't simply be copied as plain text in to a standard variable? Sure, there are solutions external to Azure DevOps that you could use (Azure Key Vault for example) but you could end up using a sledgehammer to crack a nut. No matter though, as Azure DevOps provides a solution through Secure Files. You can find this by navigating to Pipelines > Library and then clicking the Secure Files tab:

In the screenshot above I've used + Secure file to upload a file called config (which in this particular case is a file that contains credentials for connecting to an Azure Kubernetes Service cluster). Secure files are made available in the build or pipeline through the use of the Download Secure File task, which places the file in the $(Agent.TempDirectory) directory of the Azure Pipelines Agent. The file can then be used on a command line where a parameter is expecting a file, for example:

This is obviously a very specific example (an incomplete extract of a Bash script that is using kubectl to create secrets on a Kubernetes cluster) but hopefully you get the idea of how secret files can be used. Once the build or release has completed the file gets deleted—a good thing on a self-hosted agent although Microsoft-hosted agents are destroyed anyway after use.

Hope this helps!

Cheers -- Graham