Deploy a Dockerized Application to Azure Kubernetes Service using Azure YAML Pipelines 3 – Terraform Deployment Pipeline

Posted by Graham Smith on April 7, 2020No Comments (click here to comment)

This is the third post in a series where I'm taking a fresh look at how to deploy a dockerized application to Azure Kubernetes Service (AKS) using Azure Pipelines after having previously blogged about this in 2018. The list of posts in this series is as follows:

  1. Getting Started
  2. Terraform Development Experience
  3. Terraform Deployment Pipeline (this post)
  4. Running a Dockerized Application Locally
  5. Application Deployment Pipelines
  6. Telemetry and Diagnostics

In this post I take a look at how to create infrastructure in Azure using Terraform in a deployment pipeline using Azure Pipelines. If you want to follow along you can clone / fork my repo here, and if you haven't already done so please take a look at the first post to understand the background, what this series hopes to cover and the tools mentioned in this post. I'm not covering Azure Pipelines basics here and if this is of interest take a look at this video and or this series of videos. I'm also assuming familiarity with Azure DevOps.

There's quite a few moving parts to configure to move from command-line Terraform to running it in Azure Pipelines so here's the high-level list of activities:

  • Create a Variable Group in Azure Pipelines as a central place to store variables and secrets that can be used across multiple pipelines.
  • Configure a self-hosted build agent to run on a local Windows machine to aid troubleshooting.
  • Create storage in Azure to act as a backend for Terraform state.
  • Generate credentials for deployment to Azure.
  • Create variables in the variable group to support the Terraform resources that need variable values.
  • Configure and run an Azure Pipeline from the megastore-iac.yml file in the repo.

Create a Variable Group in Azure Pipelines

In your Azure DevOps project (mine is called megastore-az) navigate to Pipelines > Library > Variable Groups and create a new variable group called megastore. Ensure that Allow access to all pipelines is set to on. Add a variable named project_name and give it a meaningful value that is also likely to be globally unique and doesn't contain any punctuation and click Save:

Configure a Self-Hosted Agent to Run Locally

While a Microsoft-hosted windows-latest agent will certainly be quite satisfactory for running Terraform pipeline jobs they can be a little bit slow and there is no way to peek in and see what's happening in the file system which can be a nuisance if you are trying to troubleshoot a problem. Additionally, because a brand new instance of an agent is created for each new request they mask the issue of files hanging around from previous jobs. This can catch you out if you move from a Microsoft-hosted agent to a self-hosted agent but is something that you will certainly catch and fix if you start with a self-hosted agent. The instructions for configuring a self-host agent can be found here. The usual scenario is that you are going to install the agent on a server but the agent works perfectly well on a local Windows 10 machine as long as all the required dependencies are installed. The high-level installation steps are as follows:

  1. Create a new Pool in Azure DevOps called Local at Organization Settings > Pipelines > Agent Pools > Add pool.
  2. On your Windows machine create a folder such as C:\agents\windows.
  3. Download the agent and unzip the contents.
  4. Copy the contents of the containing folder to C:\agents\windows, ie this folder will contain two folders and two *.cmd files.
  5. From a command prompt run .\config.cmd.
  6. You will need to supply your Azure DevOps server URL and previously created PAT.
  7. Use windows-10 as the agent name and for this local instance I recommend not running as a service or at startup.
  8. The agent can be started by running .\run.cmd at a command prompt after which you should see something this:
  9. After the agent has finished running a pipeline job you can examine the files in C:\agents\windows\_work to understand what happened and assist with troubleshooting any issues.

Create Backend Storage in Azure

The Azure backend storage can be created by applying the Terraform configuration in the backend folder that is part of the repo. The configuration outputs three key/value pairs which are required by Terraform and which should be added as variables to the megastore variable group. The backend_storage_access_key should be set as a secret with the padlock:

Generate Credentials for Deployment to Azure

There are several pieces of information required by Terraform which can be obtained as follows (assumes you are logged in to Azure via the Azure CLI—run az login if not):

  1. Run az account list --output table which will return a list of Azure accounts and corresponding subscription Ids.
  2. Run az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/SubscriptionId", substituting SubscriptionId for the appropriate Id from step 1.
  3. From the resulting output create four new variables in the megastore variables group as follows:
    1. azure_subscription_id = SubscriptionId from step 1
    2. azure_client_id = appId value from the result of step 2
    3. azure_tenant_id = tenant value from the result of step 2
    4. azure_client_secret = password value from the result of step 2, which should set as a secret with the padlock
  4. Remember to save the variable group after entering the new values.

Create Terraform Variable Values in the megastore Variable Group

In the previous post where we ran Terraform from the command-line we supplied variable values via dev.tfvars, a file that isn't committed to version control and is only available for local use. These variable values need creating in the megastore variable group as follows, obviously substituting in the appropriate values:

  • aks_client_id = "service principal id for the AKs cluster"
  • aks_client_secret = "service principal secret for the AKs cluster"
  • asql_administrator_login_name = "Azure SQL admin name"
  • asql_administrator_login_password = "Azure SQL admin password"
  • asql_local_client_ip_address = "local ip address for your client workstation"

Remember to save the variable group after entering the new values.

Configure an Azure Pipeline

The pipeline folder in the repo contains megastore-iac.yml which contains all the instructions needed to automate the deployment of the Terraform resources in an Azure Pipeline. The pipeline is configured in Azure DevOps as follows:

  1. From Pipelines > Pipelines click New pipeline.
  2. In Connect choose GitHub and authenticate if required.
  3. In Select, find your repo, possibly by selecting to show All repositories.
  4. In Configure choose Existing Azure Pipelines YAML file and in Path select /pipeline/megastore-iac.yml and click Continue.
  5. From the Run dropdown select Save.
  6. At the Run Pipeline screen use the vertical ellipsis to show its menu and then select Rename/move:
  7. Rename the pipeline to megastore-iac and click Save.
  8. Now click Run pipeline > Run.
  9. If the self-hosted agent isn't running then from a command prompt navigate to the agent folder and run .\run.cmd.
  10. Hopefully watch with joy as the megastore Azure infrastructure is created through the pipeline.
Analysis of the YAML File

So what exactly is the YAML file doing? Here's an explanation for some of the schema syntax with reference to a specific pipeline run and the actual folders on disk for that run (the number shown will vary between runs but otherwise everything else should be the same):

  • name: applies a custom build number
  • variables: specifies a reference to the megastore variable group
  • pool: specifies a reference to the local agent pool and specifically to the agent we created called windows-10
  • jobs/job/workspace: ensures that the agent working folders are cleared down before a new job starts
  • script/'output environemt variables': dumps all the environment variables to the log for diagnostic purposes
  • publish/'publish iac artefact': takes the contents of the git checkout at C:\agents\windows\_work\3\s\iac and packages them in to an artifact called iac.
  • download/'download iac artefact': downloads the iac artifact to C:\agents\windows\_work\3\iac.
  • powershell/'create file with azurerm backend configuration': we need to tell Terraform to use Azure for the backend through a configuration. This configuration can't be present when working locally so instead it's created dynamically through PowerShell with some formatting commands to make the YAML structurally correct.
  • script/'terraform init': initialises Terraform in C:\agents\windows\_work\3\iac using Azure as the backend through credentials supplied on the command line from the megastore variable group.
  • script/'terraform plan and apply': performs a plan and than an apply on the configurations in C:\agents\windows\_work\3\iac using the credentials and variables passed in on the command line from the megastore variable group.

Final Thoughts

Although this seems like a lot of configuration—and it probably is—the ability to use pipelines as code feels like a significant step forward compared with GUI tasks. Although at first the YAML can seem confusing once you start working with it you soon get used to it and I now much prefer it to GUI tasks.

One question which I'm still undecided about is where to place some of the variables needed by the pipeline. I've used a variable group exclusively as it feels better for all variables to be in one place, and for variables used across different pipelines this is definitely where they should be. However, variables that are only used by one pipeline could live with the pipeline itself, as this is a fully supported feature (editing the pipeline in the browser lights up the Variables button where variables for that pipeline can be added). However having variables scattered everywhere could be confusing, hence my uncertainty. Let me know in the comments if you have a view!

That's it for now. Next time we look at running the sample application locally using Visual Studio and Docker Desktop.

Cheers -- Graham

Deploy a Dockerized Application to Azure Kubernetes Service using Azure YAML Pipelines 2 – Terraform Development Experience

Posted by Graham Smith on April 7, 2020No Comments (click here to comment)

This is the second post in a series where I'm taking a fresh look at how to deploy a dockerized application to Azure Kubernetes Service (AKS) using Azure Pipelines after having previously blogged about this in 2018. The list of posts in this series is as follows:

  1. Getting Started
  2. Terraform Development Experience (this post)
  3. Terraform Deployment Pipeline
  4. Running a Dockerized Application Locally
  5. Application Deployment Pipelines
  6. Telemetry and Diagnostics

In this post I take a look at how to create infrastructure in Azure using Terraform at the command line. If you want to follow along you can clone or fork my repo here, and if you haven't already done so please take a look at the first post to understand the background, what this series hopes to cover and the tools mentioned in this post. I'm not covering Terraform basics here and if you need this take a look at this tutorial.

Working With Terraform Files in VS Code

As with most code I write, I like to distinguish between what's sometimes called the develop inner loop and the deployment pipeline. The developer inner loop is where code is written and quickly tested for fast feedback, and the deployment pipeline is where code is committed to version control and then (usually) built and deployed and subjected to a variety of tests in different environments or stages to ensure appropriate quality.

Working with infrastructure as code (IaC) against a cloud platform is obviously different from developing an application that can run completely locally, but with Terraform it's reasonably straightforward to create a productive local development experience.

Assuming you've forked my repo and cloned the fork to a suitable location on your Windows machine, open the repo's root folder in VS Code. You will probably want to install the following extensions if you haven't already:

The .gitignore file in the root of the repo contains most of the recommended settings for Terraform plus one of my own:

The following files in the iac folder are of specific interest to my way of working locally with Terraform:

  • variables.tf: Here I declare variables here but don't provide default values.
  • terraform.tfvars: Here I provide values for all variables that are common to working both locally and in the deployment pipeline, and which aren't secrets.
  • dev.tfvars: Here I provide values for all variables that are specific to working locally or which are secrets. Crucially this file is omitted from being committed to version control, and the values supplied by dev.tfvars locally are supplied in a different way in the deployment pipeline. Obviously you won't have this file and instead I've added dev.txt as a proxy for what your copy of dev.tfvars should contain.
  • versions.tf: Here I specify the minimum versions of Terraform itself and the Azure Provider.

The other files in the iac folder should be familiar to anyone who has used Terraform and consist of configurations for the following Azure resources:

With all of the configurations I've taken a minimalist approach, partly to keep things simple and partly to keep Azure costs down for anyone who is looking to eek out free credits.

Running Terraform Commands in VS Code

What's nice about using VS Code for Terraform development is the integrated terminal. For fairly recent installations of VS Code a new terminal (Ctrl+Shift+') will create one of the PowerShell variety at the rood of the repo. Navigate to the iac folder (ie cd iac) and create dev.tfvars based on dev.txt, obviously supplying your own values. Next run terraform init.

As expected a set of new files is created to support the local Terraform backend, however these are a distraction in the VS Code Explorer. We can fix this, and clean the Explorer up a bit more as well:

  1. Access the settings editor via File > Preferences > Settings.
  2. Ensuring you have the User tab selected, in Search settings search for files:exclude.
  3. Click Add Pattern to add a glob pattern.
  4. Suggested patterns include:
    1. **/.terraform
    2. **/*.tfstate*
    3. **/.vscode
    4. **/LICENSE

To be able to deploy the Terraform configurations to Azure we need to be logged in via the Azure CLI:

  1. At the command prompt run az login and follow the browser instructions to log in.
  2. If you have access to more than one Azure subscription examine the output that is returned to check that the required subscription is set as the default.
  3. If necessary run az account set --subscription "subscription_id" to set the appropriate subscription.

You should now be able to plan or apply the configurations however there is a twist because we are using a custom tfvars file in conjunction with terraform.tfvars (which is automatically included by convention). So the correct commands to run are terraform plan -var-file="dev.tfvars" or terraform apply -var-file="dev.tfvars", remembering that these are specifically for local use only as dev.tfvars will not be available in the deployment pipeline and we'll be supplying the variable values in a different way.

That's it for this post. Next time we look at deploying the Terraform configurations in an Azure Pipeline.

Cheers -- Graham

Deploy a Dockerized Application to Azure Kubernetes Service using Azure YAML Pipelines 1 – Getting Started

Posted by Graham Smith on April 7, 2020No Comments (click here to comment)

In 2018 I wrote a series of blog posts about deploying a dockerized ASP.NET Core application to Azure Kubernetes Service (AKS) and finished up with this post where for various reasons I abandoned the Deploy to Kubernetes GUI tasks used by what was then VSTS and instead made use of refactored Bash scripts to deploy Kubernetes resources.

In the 2018 series of posts I didn't start out with infrastructure as code (IaC) and also since then a lot has changed with the tooling and the technology so in my next few posts I'm going to revisit this topic to see how things look in 2020. The blog series at the moment is looking like this:

  1. Getting Started (this post)
  2. Terraform Development Experience
  3. Terraform Deployment Pipeline
  4. Running a Dockerized Application Locally
  5. Application Deployment Pipelines
  6. Telemetry and Diagnostics

As with my previous 2018 series of posts I'm not suggesting that the ideas I'm presenting are the best and only way to do things. Rather, the intention is that the concepts offer a potential learning opportunity and a stepping stone to figuring out how you might approach this in a real-world scenario. Even if you don't need to use any of this in production I think there's a great deal of fun and satisfaction to be had from gluing all of the bits together.

The Big Picture

The dockerized application that I'll be deploying to AKS consists of the following components:

  • An ASP.NET Core web application, that sends messages to a
  • NATS message queue service, which stores messages to be retrieved by a
  • .NET Core message queue handler application, which saves messages to an
  • Azure SQL Database

The lifecycle of this application and the infrastructure it runs on is as follows:

  • All Azure resources are managed by Terraform using Azure Pipelines. These include a Container Registry, an AKS Cluster, an Azure SQL Database server and databases and Application Insights instances.
  • An AKS cluster is configured with two namespaces called qa and prd which form a basic CI/CD pipeline.
  • An Azure SQL Database server is configured with three databases called dev, qa and prd.
  • Application components (except the Azure SQL Database) run locally in a dev environment using docker-compose. Messages are saved to the dev Azure SQL Database.
  • Deployments of application components (except the Azure SQL Database) are managed separately using dedicated Azure Pipelines. The Container Registry is used to store tagged images and new images are first pushed to the qa and then to the prd namespaces on the AKS cluster.
  • Telemetry and diagnostics are collected by three separate Application Insights instances, one each for the three (dev, qa and prd) environments.

The overall aim of this series is to show how the big pieces of the jigsaw fit together and I'm intentionally not covering any of the lower-level details commonly associated with CI/CD pipelines such as testing. Maybe some other time!

What You Can Learn by Following This Blog Series

Some of the technologies I'm using in this blog series are vast in scope and I can only hope to scratch the surface. However this is a list of some of the things that you can learn about if you follow along with the series:

  • The great range of tools we now have that support running Linux on Windows via WSL 2.
  • An example of the Terraform developer inner loop experience and how to extend that to running Terraform in a deployment pipeline using Azure Pipelines.
  • Assistance with debugging Azure Pipelines by running self-hosted agents (both Windows and Linux flavours) on a Windows 10 machine.
  • Creating Azure Pipelines as pipeline as code using YAML files, including the use of templates to aid reusability and deployment jobs to target an environment.
  • How to avoid using Swiss Army Knife-style Azure Pipelines tasks and instead use native commands tuned exactly to a situation's requirements.
  • How to segment telemetry and diagnostics for each stage of the CI/CD pipeline using separate Application Insights resources.

Tools You Will Need / Want

There is a long list of tools needed for this series and getting everything installed and configured is quite an exercise. However you may have some of this already and it can also be great fun getting the newer stuff working. Some of the tools can be installed with Chocolatey and it's definitely worth checking this out if you haven't already. Generally, I've listed the tools in the order you will need them so you don't need to install everything before working through the next couple of posts in the series. Everything in the list should be installed in Windows 10. There are some tools that need installing in the Ubuntu distro but I cover that in the relevant post.

That's it for this post. Next time we start working with Terraform at the command line.

Cheers -- Graham

Create an Azure DevOps Services Self-Hosted Agent in Azure Using Terraform, Cloud-init—and Azure DevOps Pipelines!

Posted by Graham Smith on November 14, 2018No Comments (click here to comment)

I recently started a new job with the awesome DevOpsGroup. We're hiring and there's options for remote workers (like me), so if anything from our vacancies page looks interesting feel free to make contact on LinkedIn.

One of the reasons for mentioning this is that the new job brings a new Azure subscription and the need to recreate all of the infrastructure for my blog series on what I will now have to rename to Deploy a Dockerized ASP.NET Core Application to Azure Kubernetes Service Using an Azure DevOps CI/CD Pipeline. For this new Azure subscription I've promised myself that anything that is reasonably long-lived will be created through infrastructure as code. I've done a bit of this in the past but I haven't been consistent and I certainly haven't developed any memory muscle. As someone who is passionate about automation it's something I need to rectify.

I've used ARM templates in the past but found the JSON a bit fiddly, however in the DevOpsGroup office and Slack channels there are a lot of smart people talking about Terraform. When combined with Brendan Burns' post on Expanding the HashiCorp partnership and HashiCorp's post about being at Microsoft Ignite well, there's clearly something going on between Microsoft and HashiCorp and learning Terraform feels like a safe bet.

For my first outing with Terraform I decided to create an Azure DevOps Services (neé VSTS, hereafter ADOS) self-hosted agent capable of running Docker multi-stage builds. Why a self-hosted agent rather than a Microsoft-hosted agent? The main reason is speed: a self-hosted agent is there when you need it and doesn't need the spin-up time of a Microsoft-hosted agent, plus a self-hosted agent can cache Docker images for re-use as opposed to a Microsoft-hosted agent which needs to download images every time they are used. It's not a perfect solution for anyone using Azure credits as you don't want the VM hosting the agent running all the time (and burning credit) but there are ways to help with that, albeit with a twist at the moment.

List of Requirements

In the spirit of learning Terraform as well as creating a self-hosted agent I came up with the following requirements:

  • Use Terraform to create a Linux VM that would run Docker, that in turn would run a VSTS [sic] agent as a Docker container rather than having to install the agent and associated dependencies.
  • Create as many self-hosted agent VMs as required and tear them down again afterwards. (I'm never going to need this but it's a great infrastructure as code learning exercise.)
  • Consistent names for all the Azure infrastructure items that get created.
  • Terraform code refactored to avoid DRY problems as far as possible and practicable.
  • All bespoke settings supplied as variables to make the code easily reusable by anyone else.
  • Terraform code built and deployed from source control (GitHub) using an ADOS Pipeline.
  • Mechanism to deploy ‘development' self-hosted agents when working locally and only allow deploying ‘production' self-hosted agents from the ADOS Pipeline by checking in code.

This blog post aims to explain how I achieved all that but first I want to describe my Terraform journey since going from zero to my final solution in one step probably won't make much sense.

My Terraform Journey

I used these resources to get going with Terraform and I recommend you do the same with whatever is available to you:

It turns out that there were quite a few decisions to be made in implementing this solution and lots of parts that didn't work as expected so I've saved the discussion of all that to the end to keep the configuration steps cleaner.

Configuring Pre-requisites

There are a few things to configure before you can start working with my Terraform solution:

  • Install Azure CLI and Terraform on your local machine.
  • If you are using Visual Studio Code as your editor ensure you have the Terraform extension by Mikael Olenfalk installed.
  • If required, create an SSH key pair following these instructions if you need them. The goal is to have a public key file at ~\.ssh\id_rsa.pub.
  • To allow Terraform to access Azure create a password-based Azure AD service principal and make a note of the appId and password that are returned.
  • Create a new Agent Pool in your ADOS subscription—I called mine Self-Hosted Ubuntu.
  • Create a PAT in your ADOS subscription limiting it to Agent Pools (read, manage) scope.
  • Fork my repository on GitHub and clone to your local machine.

Examining the Terraform Solution

Let's take a look at the key contents of the GitHub repo, which have separate folders for the backend storage on Azure and the actual self-hosted agent.

backend-storage folder
  • main.tf—This is a simple configuration which creates an Azure storage account and container to hold the Terraform state. The names of the storage account and container are used in the configuration of the self-hosted agent.
  • variables.tf—I've pulled the key variables out in to a separate file to make it easier for anyone to know what to change for their own implementation.
ados-self-hosted-agent-ubuntu folder
  • main.tf—I've chosen to keep the configuration in one file for simplicity of writing this blog post, however you'll frequently see the different resources split out in to separate files. If you've followed this tutorial then most of what's in the configuration should be straightforward, however you will see that I've made extensive use of variables. I like to see all of my Azure infrastructure with consistent names so I pass in a base_name value on which the names of all resources are based. What is different from what you will have seen in the tutorial is that some resources have a count = "${var.node_count}" property, which tells Terraform to create as many copies of that particular resource as value of the node_count variable. For those resources the name must be unique of course and you'll see a slightly altered syntax of the name property, for example "${var.base_name}-vm-${count.index}" for the VM name. This creates those types of resources with an incrementing numerical suffix.
  • variables.tf—I've made heavy use of variables both to make it easier for others to use the repo and also to separate out what I think should and shouldn't be tracked via version control. You'll see that variables are grouped in to those that I think should be tracked by version control for traceability purposes and those which are very specific to an Azure subscription and which shouldn't (or in the case of secrets mustn't) be tracked by version control. In the tracked group the variables are set in variables.tf however the non-tracked group have their variables set in a separate (ignored by git) file locally and via Pipeline Variables in ADOS.
  • .gitignore—note that I've added cloud-init.txt to the standard file for Terraform as cloud-init.txt contains secrets.

Configuring Backend Storage

In order to use Azure as a central location to save Terraform state you will need to complete the following:

  1. In your favourite text editor edit \backend-storage\variables.tf supplying suitable values for the variables. Note that storage_account needs to be globally unique.
  2. Open a command prompt, login to the Azure CLI and and navigate to the backend-storage folder.
  3. Run terraform init to initialise the directory.
  4. Run terraform apply to have Terraform generate an execution plan. Type yes to have the plan executed.

Configuring and Running the Terraform Solution Locally

To get the solution working from your local machine to create development self-hosted agents running in Azure you will need to complete the following:

  1. In your favourite text editor edit \ados-self-hosted-agent-ubuntu\variables.tf supplying suitable values for the variables that are designated to be tracked and changed via version control.
  2. Add a terraform.tfvars file to the root of \ados-self-hosted-agent-ubuntu and copy the following code in, replacing the placeholder text with your actual values where required (double quotes are required):
  3. Add a cloud-init.txt file to the root of \ados-self-hosted-agent-ubuntu and copy the following code in, replacing the placeholder text with your actual values where required (angle brackets must be omitted and the readability back slashes in the docker run command must also be removed):
  4. In a secure text file somewhere construct a terraform init command as follows, replacing the placeholder text with your actual values where required (angle brackets must be omitted):
  5. Open a PowerShell console window and navigate to the root of \ados-self-hosted-agent-ubuntu. Copy, paste and then run the terraform init command above to initialise the directory.
  6. Run terraform apply to have Terraform generate an execution plan. Type yes to have the plan executed.

It takes some time for everything to run but after a while you should see ADOS self-hosted agents appear in your Agent Pool. Yay!

Configuring and Running the Terraform Solution in an Azure DevOps Pipeline

To get the solution working in an Azure DevOps Pipeline to create production self-hosted agents running in Azure you will need to complete the following:

  1. Optionally create a new team project. I'm planning on doing more of this so I created a project called terraform-azure.
  2. Create a new Build Pipeline called ados-self-hosted-agent-ubuntu-build and in the Where is your code? window click on Use the visual designer.
  3. Connect up to your forked GitHub repository and then elect to start with an empty job.
  4. For the Agent pool select Hosted Ubuntu 1604.
  5. Create a Bash task called terraform upgrade and paste in the following as an inline script:
  6. Create a Copy Files task called copy files to staging directory and configure Source Directory to ados-self-hosted-agent-ubuntu and Contents to show *.tf and Target Folder to be $(Build.ArtifactStagingDirectory).
  7. Create a PowerShell task called create cloud-init.txt, set the Working Directory to $(Build.ArtifactStagingDirectory) and paste in the following as an inline script:
  8. Create a Bash task called terraform init, set the Working Directory to $(Build.ArtifactStagingDirectory) and paste in the following as an inline script:
  9. Create a Bash task called terraform plan, set the Working Directory to $(Build.ArtifactStagingDirectory) and paste in the following as an inline script:
  10. Create a Bash task called terraform apply, set the Working Directory to $(Build.ArtifactStagingDirectory) and paste in the following as an inline script:
  11. Create a Publish Build Artifacts task called publish artifact and publish the $(Build.ArtifactStagingDirectory) to an artifact called terraform.
  12.  Create the following variables, supplying your own values as required (variables with spaces should be surrounded by single quotes) and ensuring secrets are added as such:
  13. Queue a new build and after a while a while you should see ADOS self-hosted agents appear in your Agent Pool.
  14. Finally, from the Triggers tab select Enable continuous integration.

Discussion

One of the great things I found about Terraform is that it's reasonably easy to get a basic VM running in Azure by following tutorials. However the tutorials are, understandably, lacking when it comes to addressing issues such as repeated code and I spent a long time working through my configurations addressing repeated code and extracting variables. One fun aspect was amending the Terraform resources so that a node_count variable can be set with the desired number of VMs to create. Pleasingly, Terraform seems really smart about this: you can start with two VMs (labelled 0 and 1), scale to four VMs (0, 1, 2 and 3) and then scale back to one VM which will remove 1, 2 and 3 to leave the VM labelled 0.

Do be aware that some of the tutorials don't cover all of the properties of resources and it's worth looking at the documentation for those fine details you want to implement. For example I wanted to be able to SSH in to VMs via a FQDN rather than IP address which meant setting the domain_name_label property of the azurerm_public_ip resource. In another example the tutorial didn't specify that the azurerm_virtual_machine should delete_os_disk_on_termination, the absence of which was a problem when implementing the scaling VMs up and down capability. It was all in the documentation though.

An issue that I had to tackle early on was the mechanism for configuring the internals of VMs once they were up-and-running. Terraform has a remote-exec provisioner which looks like it should do the job but I spent a long time trying to make it work only to have the various commands consistently terminate with errors. An alternative is cloud-init. This worked flawlessly however the cloud-init file has to contain an ADOS secret and I couldn't find a way to pass the secret in as a parameter. My solution was to exclude a local cloud-init.txt file from version control so the secret didn't get committed and then to dynamically build the file as part of the build pipeline.

Another decision I had to make was around workflow, ie how the solution is used in practice. I've kept it simple but have tried to put mechanisms in place for those that need something more sophisticated. For experimenting and getting things working I'm just working locally, although for consistency I'm storing Terraform state in Azure. Where required all resources are labelled with a ‘dev' element somewhere in their name. For ‘production' self-hosted agents I'm using an ADOS build pipeline, with all required resources having a ‘prd' element in their name. Note that there is a Terraform workspaces feature which might work better in a more sophisticated scenario. I've chosen to keep things simple by deploying straight from a build however I do output a Terraform plan as an arttifact so if you wanted you could pick this up and apply it in a release pipeline. I've also not implemented any linting or testing of the Terraform configuration as part of the build but that's all possible of course. You'll also notice that I haven't implemented the build pipeline with the YAML code feature. It was part of my plans but I ran in to some issues which I couldn't fix in the time available.

It's worth noting a few features of the build pipeline. I did look at tasks in the ADOS marketplace and while there are a few I'm increasingly becoming dissatisfied with how visual tasks abstract away what's really happening and I'm more in favour of constructing commands by hand.

  • terraform upgrade—In order to run Terraform commands in an ADOS pipeline you need to have access to an agent that has Terraform installed. As luck would have it the Microsoft-hosted agent Hosted Ubuntu 1604 fits the bill. Well almost! I ran in to an issue early on before I started using separate blobs in Azure to hold state for dev and prd where Terraform wasn't happy that I was developing on a newer version of Terraform than was running on the Hosted Ubuntu 1604 agent. It was a case of downgrading to an older version of Terraform locally or upgrading the agent. I chose the latter which is perfectly possible and very quick. There's obviously a maintenance issue with keeping versions in sync but I created a variable in the ADOS pipeline to help. As it turns out with the final solution using separate state blobs in Azure this is less of a problem however I've chosen to keep the upgrade task in, if for no other reason than to illustrate what's possible.
  • terraform init—Although I've seen the init command have its variables passed to it through as environment variables at the end of this demo I couldn't get the command to work this way and had to pass the variables through as parameters.
  • terraform plan—The environment variables trick does work with the plan command. Note that the command outputs a file called tfplan which gets saved as part of the artifact. I use the file immediately in the next task (terraform apply) but a release pipeline could equally use the file after (for example) someone had inspected what the plan was going to actually do using terraform show.

Finally, there's one killer feature that seems not to have been implemented yet and that's Auto-shutdown of VMs. I find this essential to ensure that Azure credits don't get burned, so if you are like me and need to preserve credits do take appropriate precautions until the feature is implemented. Happy Terraforming!

Cheers -- Graham