Archives for Continuous Delivery with TFS

Continuous Delivery with TFS: Track Technical Debt with SonarQube

Posted by Graham Smith on June 18, 20154 Comments (click here to comment)

So far in this blog post series on building continuous delivery pipelines with the TFS ecosystem the focus on baking quality in to the application has centred mainly on static code analysis, unit tests and automated web tests. But just because you have no broken static code analysis rules and all your various types of tests are green isn't a guarantee that there aren't problems lurking in your codebase. On the contrary, you may well be unwittingly accumulating technical debt.  And unless you go looking for it chances are you won't find out that you have a technical debt problem until it starts to cause you major problems.

For some years now the go-to tool for analysing technical debt has been SonarQube (formerly Sonar). However SonarQube hails from the open source world and it hasn't exactly been a seamless fit in to the C# and TFS world. All that changed around the time of Build 2015 with the announcement that Microsoft had joined forces with the makers of SonarQube to start to address the situation. The video from Build 2015 which tells the story is well worth watching. To coincide with this announcement the ALM Rangers published a SonarQube installation guide aimed at TFS users. I used this guide to assist me in writing this blog post to see how SonarQube can be set up to work with our continuous delivery pipeline. It's worth noting that the guide mentions that it's possible to use integrated security with the jTDS driver that SonarQube uses to connect to SQL Server but I struggled for several hours before throwing in the towel. Please share in the comments if you have had success in doing this. Another difference between the guide and my setup is that the guide uses the all-in-one Brian Keller VM whereas I'm installing on distributed VMs.

Create New SonarQube Items

SonarQube requires a running Java instance and needs quite a bit of horsepower so the recommendation is to run it on a dedicated server. You'll need to create the following:

  • A new domain service account -- I created ALM\SONARQUBE.
  • A new VM running in your cloud service -- I called mine ALMSONARQUBE. As always in a demo Azure environment there is a desire to preserve Azure credits so I created mine as a basic A4 running Windows Server 2012 R2. Ensure the server is joined to your domain and that you add ALM\SONARQUBE to the Local Administrators group.
Install SonarQube

The following steps should be performed on ALMSONARQUBE :

  1. Download and install a Java SE Runtime Environment appropriate to your VMs OS. There are myriad download options and it can be confusing to the untrained Java eye but on the index page look out for the JRE download button:
  2. Download and unblock the latest version of SonarQube from the downloads page. There isn't a separate download for Windows -- the zip contains files that allow SonorQube to run on a variety of operating systems. Unzip the download to a temp location and copy the bin, conf and other folders to an installation folder. I chose to create C:\SonarQube\Main as the root for the bin, conf and other folders however this is slightly at odds with the ALM guide where they have a version folder under the main SonarQube folder. As this is my first time installing SonarQube I'm not sure how upgrades are handled but my guess is that everything apart from the conf folder can be overwritten with a new version.
  3. At this point you can run C:\SonarQube\Main\bin\windows-x86-64\StartSonar.bat (you may have to shift right-click and Run as administrator) to start SonarQube against its internal database and browse to http://localhost:9000 on ALMSONARQUBE to confirm that the basic installation is working. To stop SonarQube simply close the command window opened by StartSonar.bat.
Confirm SQL Server Connectivity

If you are intending to connect to a remote instance of SQL Server I highly recommend confirming connectivity from ALMSONARQUBE as the next step:

  1. On the ALMSONARQUBE machine create a new text file somewhere handy and rename the extension to udl.
  2. Open this Data Link Properties file and you will be presented with the ability to make a connection to SQL Server via a dialog that will be familiar to most developers. Enter connection details that you know work and use Test Connection to confirm connectivity.
  3. Possible remedies if you don't have connectivity are:
    1. The domain firewall is on for either or both machines. Consider turning it off as I do in my demo environment or opening up port 1433 for SQL Sever.
    2. SQL Sever has not been configured for the TCP/IP protocol. Open Sql Server Configuration Manager [sic] and from SQL Server Network Configuration > Protocols for MSSQLSERVER enable the TCP/IP protocol. Note that you'll need to restart the SQL Server service.
Create a SonarQube Database

Carry out the following steps to create and configure a database:

  1. Create a new blank SQL Server database on 2008 or 2012 -- I created SonarQube. I created my database on the same instance of SQL Server that runs TFS and Release Management. That's fine in a demo environment but in a production environment where you may be using the complimentary SQL Server licence for running TFS that may cause a licensing issue.
  2. SonarQube needs the database collation to be case-sensitive (CS) and accent-sensitive (AS). You can actually set this when you create the database but if it needs doing afterwards right-click the database in SSMS and choose Properties. On the Options page change the collation to SQL_Latin1_General_CP1_CS_AS.
  3. Still in SSMS, create a new SQL Server login from Security > Logins, ensuring that the Default language is English. Under User Mapping grant the login access to the SonarQube database and grant access to the db_owner role.
  4. On ALMSONARQUBE navigate to C:\SonarQube\Main and open from the conf folder in Notepad or similar. Make the follwoing changes:
    1. Find and uncomment sonar.jdbc.username and sonar.jdbc.password and supply the credentials created in the step above.
    2. Find the Microsoft SQLServer 2008/2012 section and uncomment sonar.jdbc.url. Amend the connection string so it includes the name of the database server and the database. The final result should be something like sonar.jdbc.url=jdbc:jtds:sqlserver://ALMTFSADMIN/SonarQube;SelectMethod=Cursor.
  5. Now test connectivity by running SonarStart.bat and confirming that the database schema has been created and that browsing to http://localhost:9000 is still successful.
Run SonarQube as a Service

The next piece of the installation is to configure SonarQube as a Windows service:

  1. Run C:\SonarQube\Main\bin\windows-x86-64\InstallNTService.bat (you may have to shift right-click and Run as administrator) to install the service.
  2. Run services.msc and find the SonarQube service. Open its Properties and from the Log On tab change the service to log on as the ALM\SONARQUBE domain account.
  3. Again test all is working as expected by browsing to http://localhost:9000.
Configure for C#

With a working SonarQube instance the next piece of the jigsaw is to enable it to work with C#:

  1. Head over to the C# plugin page and download and unblock the latest sonar-csharp-plugin-X.Y.jar.
  2. Copy the sonar-csharp-plugin-X.Y.jar to C:\SonarQube\Main\extensions\plugins and restart the SonarQube service.
  3. Log in to the SonarQube portal (http://localhost:9000 or http://ALMSONARQUBE:9000 if on a remote machine) as Administrator -- the default credentials are admin and admin.
  4. Navigate to SettingsSystem > Update Center and verify that the C# plugin is installed:
Configure the SonarQube Server as a Build Agent

In order to integrate with TFS a couple of SonarQube components we haven't installed yet need access to a TFS build agent. The approach I've taken here is to have the build agent running on the actual SonarQube server itself. This keeps everything together and ensures that your build agents that might be servicing checkins are not bogged down with other tasks. From ALMSONARQUBE:

  1. Run Team Foundation Server Setup (typically by mounting the iso and running tfs_server.exe) and perform the install.
  2. At the Team Foundation Server Configuration Center dialog chose Configure Team Foundation Build Service > Start Wizard.
  3. Use the usual dialogs to connect to the appropriate Team Project Collection and then at the Build Services tab choose the Scale out build services option to add more build agents to the existing build controller on the TFS administration server.
  4. In the Settings tab supply the details of the domain service account that should be used to run the build agents.
  5. Install Visual Studio 2013.4 as it's the easiest way to get all the required bits on the build server.
  6. From within Visual Studio navigate to Tools > Extensions and Updates and then from the Updates tab update Microsoft SQL Server Update for database tooling.
  7. Update nuget.exe by opening an Administrative command prompt at C:\Program Files\Microsoft Team Foundation Server 12.0\Tools and running nuget.exe update -self.
  8. Finally, clone an existing Contoso University build definition that is based on the TfvcTemplate.12.xaml template, or create and configure a new build definition for Contoso University. I called mine ContosoUniversity_SonarQube. Queue a new build based on this template and make sure that the build is successful. You'll want to fix any glitches at this stage before proceeding.
Install the SonarQube Runner Component.

The SonarQube Runner is the is recommended as the default launcher to analyse a project with SonarQube. Installation to ALMSONARQUBE is as follows:

  1. Create a Runner folder in C:\SonarQube.
  2. Download and unlock the latest version of from the downloads page.
  3. Unzip the contents of to C:\SonarQube\Runner so that the bin, conf and lib folders are in the root.
  4. Edit C:\SonarQube\Runner\conf\ by uncommenting and amending as required the following values:
    2. sonar.jdbc.url=jdbc:jtds:sqlserver://ALMTFSADMIN/SonarQube;SelectMethod=Cursor
    3. sonar.jdbc.username=SonarQube
    4. sonar.jdbc.password=$PasswordForSonarQube$
  5. Create a new system variable called SONAR_RUNNER_HOME with the value C:\SonarQube\Runner.
  6. Amend the Path system variable adding in C:\SonarQube\Runner\bin.
  7. Restart the build service -- the Team Foundation Server Administration Console is just one place you can to do this.
Integration with Team Build

In order to call the SonarQube runner from a TFS build definition a component called Sonar.MSBuild.Runner has been developed. This needs installing on ALMSONARQUBE is as follows:

  1. Create an MSBuild folder in C:\SonarQube.
  2. Download and unlock the latest version of from the C# downloads page.
  3. Unzip the contents of to C:\SonarQube\MSBuild so that the files are in the root.
  4. Copy SonarQube.Integration.ImportBefore.targets to C:\Program Files (x86)\MSBuild\12.0\Microsoft.Common.Targets\ImportBefore. (This folder structure may have been created as part of the Visual Studio installation. If not you will need to create it manually.)
  5. The build definition cloned/created earlier (ContosoUniversity_SonarQube) should be amended as follows:
    1. Process > 2.Build > 5. Advanced > Pre-build script arguments = /key:ContosoUniversity /name:ContosoUniversity /version:1.0
    2. Process > 2.Build > 5. Advanced > Pre-build script path = C:\SonarQube\MSBuild\SonarQube.MSBuild.Runner.exe
    3. Process > 3. Test > 2. Advanced > Post-test script path = C:\SonarQube\MSBuild\SonarQube.MSBuild.Runner.exe
  6. Configure the build definition for unit test results as per this blog post. Note though that Test assembly file specification should be set to **\*unittest*.dll;**\*unittest*.appx to avoid the automated web tests being classed as unit tests.
Show Time

With all the configuration complete it's time to queue a new build. If all is well you should see that the build report contains a SonarQube Analysis Summary section:


Clicking on the Analysis results link in the build report should take you to the dashboard for the newly created ContosoUniversity project in SonarQube:


This project was created courtesy of the Pre-build script arguments in the build definition (/key:ContosoUniversity /name:ContosoUniversity /version:1.0). If for some reason you prefer to create the project yourself the instructions are here. Do note that the dashboard is reporting 100% unit tests coverage only because my Contoso University sample action uses quick and dirty unit tests for demo purposes.

And Finally...

Between the ALM Rangers guide and the installation walkthrough above I hope you will find getting started with SonarQube and TFS reasonably straightforward. Do be aware that I found the ALM Ranger's guide to be a little confusing in places. There is the issue of integrated security with SQL Server that I never managed to crack and then a strange reference on page 22 about not being needed after integrating with team build which had me scratching my head since how else can the components know how to connect to the SonarQube portal and database? It's early days though and I'm sure the documentation will improve and mature with time.

Performing the installation is just the start of the journey of course. There is a lot to explore, so do take time to work through the resources at

Cheers -- Graham

Continuous Delivery with TFS: Enable Test Impact Analysis

Posted by Graham Smith on May 28, 2015No Comments (click here to comment)

Test Impact Analysis is a feature that first appeared with Visual Studio / Microsoft Test Manager 2010 and provides for the ability to recommend tests that should be re-run in response to changes that have been made at the code level. It's a very useful feature but it does need some configuration before it can be used. In this post in my series on continuous delivery with TFS we look at the steps that need to be taken to enable TIA in our development pipeline.

Setting the Scene

The scenario I'm working with is where a new nightly build of the sample ASP.NET application has been deployed to the DAT stage and all of the automated Selenium web tests have passed. This now leaves the build ready to deploy in to the DQA stage so that any manual tests (including manual tests that have been automated to run from MTM) can be run from a browser on a client workstation. With TIA configured there are then at least two places to check for any tests that are recommended for running again.

Whilst working through the configuration for TIA I discovered that TIA doesn't seem to work in a multi-tenant web server, which is what I've set up for this blog post series to keep the number of VMs to a minimum. More correctly, I suspect that TIA doesn't work where there is a separate application pool for each website in a multi-tenant web server. I haven't investigated this thoroughly but it's something to bear in mind if you are trying to get TIA working and something I'll address in my future blog post series on continuous delivery with TFS 2015. The MSDN guidance for setting up TIA is here but it's slightly at odds with the latest version of MTM and in any case doesn't have all the details.

Create a Lab Center Environment and a Test Settings Configuration

As good a starting point as any for TIA configuration is to create a new environment in the MTM Lab Center. I covered this here so I won't go through all the steps again, however the environment needs to contain the web server that hosts the DQA stage's web site and it should be configured for the Web Server role.

Now move over to Test Settings and create a new entry -- I called mine Manual Test Run. Choose Manual for the What type of tests do you want to run? question and then in the Roles page select Web Server to join the Local role which is pre-selected and mandatory. In the Data and Diagnostics page the Local role needs to be configured for ASP.NET Client Proxy for IntelliTrace and Test Impact as a minimum and the Web Server role needs to be configured for Test Impact as a minimum. Additionally, after selecting Test Impact click on Configure at the far right. In the dialog's Advanced tab ensure Collect data from ASP.NET applications running on Internet Information Services is checked.


Configure the Test Plan

Either use an existing test plan or create a new one and in Testing Center > Plan > Contents add a new suite called Instructor. Create a new Test Case in Instructor called Can Navigate and then add the following steps:

  1. Launch IE
  2. Type URL and hit enter
  3. Click on Instructors

Now from Testing Center > Test > Run Tests run the Can Navigate test:


The aim of running the test this first time is to record each step so the whole test can be replayed on future runs. This sort of automation isn't to be confused with deep automation using tools such as Selenium or CodedUI, and is instead more akin to recording macros in Microsoft Office applications. Nevertheless, the technique is very powerful due to the repeatability it offers and is also a big time saver. The process of recording the steps is a little fiddly and I recommend you follow the MSDN documentation here. The main point to remember is to mark each step as passed after successful completion so that MTM correctly associates the action with the step. Hopefully the build steps are obvious, the aim being simply to display the list of instructors.

The final configuration step for the test plan is to configure Run Settings from Testing Center > Plan:

  1. Manual runs > Test settings = Manual Test Run (created above)
  2. Manual runs > Test environment = ALMWEB01 (created above, your environment name may differ)
  3. Builds > Filter for builds = ContosoUniversity_Main_Nightly (Ready for Deployment) (or whatever build you are using)
  4. Builds > Build in use = Latest build you have marked with the Build Quality of Ready for Deployment. (Build quality is arbitrary -- just needs to be consistent.)


Web Server Configuration

In order for TIA data to be collected on the web server there are several configuration steps to be completed:

  1. When environments are created by MTM Lab Center they are configured to use the Network Service account. We need to use the dedicated ALM\TFSTEST domain account (or whatever you have called yours) so on the web server run the Test Agent Configuration Tool and make the change. Additionally the TFSTEST account needs to be in the Local Administrators group on the web server.
  2. The domain account (ALM\CU-DQA) that is used as the identity for the application pool (CU-DQA) for the DQA website needs to have a local profile. You can either log on to the server as ALM\CU-DQA or use the runas /user:domain\name /profile cmd.exe command, supplying the appropriate credentials.
  3. The CU-DQA application pool needs to have the Load User Profile property set to True.
Web Client Configuration

The web client machine (I'm using Windows 8.1) needs to be running MTM and also has to have the Microsoft Test Agent installed. This should be configured with the ALM\TFSTEST account (which should be in the Local Administrators group) and be registered to the test controller.

Putting it All Together

With the configuration out of the way it's now time to generate TIA data. The Can Navigate test needs one successful run on the web client against the currently selected build in order to generate a baseline. As noted above it seems that TIA doesn't play nicely with multiple application pools and I found I needed to stop the CU-DAT and CU-PRD application pools before running the test.

  1. In MTM navigate to Testing Center > Test > Run Tests and run Can Navigate as earlier.
  2. The Test Runner will fire up and you should click Start Test.
  3. The Test Runner will display the steps of the test and the VCR style controls. From the Play dropdown choose Play all:
  4. When the test has successfully completed use the Mark test case result dropdown to mark the test as a Pass:
  5. A dialog will pop up advising that impact data is being collected and when it closes you should Save and Close the test.

Back at Testing Center > Test > Run Tests, in order to check that TIA data was generated click on View results with Can Navigate selected. In the attachments section you should see a file ending in testimpact.xml:


We now need to create a change to the method that displays the list of instructors. Typically this change will arise as part of fixing a bug however it will be sufficient here to fake the change in order to get TIA to work. To achieve this open up the Contoso University demo app and navigate to ContosoUniversity.Web > Controllers > InstructorController > Index. Make any non-breaking change -- I changed the OrderBy of the query that returns the instructors.

Check the code in to version control and then start a new build. If the build is successful you can confirm that Can Navigate is now flagged as an impacted test. Firstly you should see this in the build report either from Visual Studio or Team Web Access:


(Whilst you are examining the build report mark the Build Quality as Ready for Deployment, but note that you would typically do this after confirming a successful DAT stage). Secondly, in MTM navigate to Testing Center > Track > Recommended Tests. Change the Build in use to the build that has just passed and then change Previous build to compare to the build that was in use at the time of creating the baseline. A dialog should pop up advising that there may be tests that need to be re-run. After dismissing the dialog Can Navigate should be listed under Recommended tests:


Test Case Closed

As is often the case with continuous delivery pipeline work it seems like there is a great deal of configuration required to get a feature working and TIA is no exception. One valuable lesson is that whilst a multi-tenant web application configuration certainly saves on the number of VMs required for a demo environment it does cause problems and should almost certainly be avoided for an on-premises installation. I'll definitely be using separate web servers when I refresh my demo setup for TFS 2015. And when Windows Nano Server becomes available we won't be thinking twice about trying to save on the number of running VMs. Exciting times ahead...

Cheers -- Graham

Continuous Delivery with TFS: Configure Application Insights

Posted by Graham Smith on May 17, 20155 Comments (click here to comment)

If you get to the stage where you are deploying your application on a very frequent basis and you are relying on automated tests for the bulk of your quality assurance then a mechanism to alert you when things go wrong becomes crucial. You should have something in place anyway of course but in practice I suspect that application monitoring is either frequently overlooked or remains stubbornly on the to-do list.

A successful continuous delivery pipeline implementation shouldn't rely on the telephone or email as the alerting system and in this post in my blog series on implementing continuous delivery with TFS we look at how to integrate relevant parts of Microsoft's Application Insights (AI) tooling in to the pipeline. If you need to get up to speed with its capabilities I have a Getting Started blog post here. As a quick refresher AI is a suite of components that integrate with your application and servers and which sends telemetry to the Azure Portal. As a bonus, not only do you get details of diagnostic issues but also rich analytics on how you application is being used.

Big Picture

AI isn't just one component and in fact there are at least three main ways in which AI can be configured to provide diagnostic and analytic information:

  • Adding the Application Insights SDK to your application.
  • Installing Status Monitor on an IIS server.
  • Creating Web Tests that monitor the availability of an HTTP endpoint available on the public Internet.

One key point to appreciate with AI and continuous delivery pipelines is that unless you do something about it AI will put the data it collects from the different stages of your pipeline in one ‘bucket' and you won't easily be able to differentiate what came from where. Happily there is a way to address this as we'll see below. Before starting to configure AI there are some common preparatory steps that need to be addressed so let's start with those.


If you have been following along with this series of blog posts you will be aware that so far we have only created DAT and DQA stages of the pipeline. Although not strictly necessary I created a PRD stage of the pipeline to represent production: if nothing else it's handy for demonstrations where your audience may expect to see the pipeline endpoint. I won't detail all the configuration steps here as they are all covered by previous blog posts however the whole exercise only took a few minutes. As things stand none of these stages exposes our sample web application to the public Internet however this is necessary for the creation of Web Tests. We can fix this in the Azure portal by adding an HTTP endpoint to the VM that runs IIS:


Our sample application is now available using a URL that begins with the cloud service name and includes the website name, for example Be aware that this technique probably falls foul of all manner of security best practices however given that my VMs are only on for a few hours each week and it's a pure demo environment it's one I'm happy to live with.

The second item of groundwork is to create the containers that will hold the AI data for each stage of the pipeline. You will need to use the new Azure portal for this at First of all a disclaimer. There are several techniques at our disposal for segregating AI data as discussed in this blog post by Victor Mushkatin, and the comments of this post are worth reading as well since there are some strong opinions. I tried the tagging method but couldn't get it to work properly and as Victor says in the post this feature is at the early stages of development. In his post Victor creates a new Azure Resource for each pipeline stage however that seemed overly complicated for a demo environment. Instead I opted to create multiple Application Insights Resources in one Azure Resource group. As an aside, resource groups are fairly new to Azure and for any new Azure deployment they should be carefully considered as part of the planning process. For existing deployments you will find that your cloud service is listed as a resource group (containing your VMs) and I chose to use this as the group to contain the Application Insights Resources. Creating new AI resources is very straightforward. Start with the New button and then choose Developer Services > Application Insights. You'll need to provide a name and then use the arrow selectors to choose Application Type and Resource Group:


I created the following resource groups which represent the stages of my pipeline: CU-DEV, CU-DAT, CU-DQA and CU-PRD. What differentiates these groups is their instrumentation keys (often abbreviated to ikey). You'll need to retrieve the ikeys for each group and the way to do that in the new portal is via Browse > Filter By > Application Insights > $ResourceGroup$ > Settings > Properties where you will see the Instrumentation Key selector.

Add the Application Insights SDK to ContosoUniversity

We can now turn our attention to adding the Application Insights SDK to our Contoso University web application:

  1. Right-click your web project (ContosoUniversity.Web) within your Visual Studio solution and choose Add Application Insights Telemetry.
  2. The Add Application Insights to Project dialog opens and invites you to sign in to Azure:
  3. The first few times I tried to connect to Azure I got errors about not being able to find an endpoint but persistence paid off. I eventually arrived at a dialog that allowed me to choose my MSDN subscription via the Use different account link:
  4. Having already created my AI resources I used Configure settings to choose the CU-DEV resource:
  5. Back in the Add Application Insights to Project dialog click on the Add Application Insights to Project link to have Visual Studio perform all the necessary configuration.

At this stage we can run the application and click around to generate telemetry. If you are in Debug mode you can see this in the Output window. After a minute or two you should also see the telemetry start to appear in the Azure portal (Browse > Filter By > Application Insights > CU-DEV).

Configure AI in Contoso University for Pipeline Stages

As things stand deploying Contoso University to other stages of the pipeline will cause telemetry for that stage to be added to the CU-DEV AI resource group. To remedy this carry out the following steps:

  1. Add an iKey attribute to the appSettings section of Web.config:
  2. Add a transform to Web.Release.config that consists of a token (__IKEY__) that can be used by Release Management:
  3. Add the following code to Application_Start in Global.asax.cs:
  4. As part of the AI installation Views.Shared._Layout.cshtml is altered with some JavaScript that adds the iKey to each page. This isn't dynamic and the JavaScript instrumentationKey line needs altering to make it dynamic as follows:
  5. Remove or comment out the InstrumentationKey section in ApplicationInsights.config.
  6. In the Release Management client at Configure Apps > Components edit the ContosoUniversity\Deploy Web Site component by adding an IKEY variable to Configuration Variables.
  7. Still in the Release Management client open the Contoso University\DAT>DQA>PRD release template from Configure Apps > Agent-based Release Templates and edit each stage supplying the iKey value for that stage (see above for how to get this) to the newly added IKEY configuration variable.

After completing these steps you should be able to deploy your application to each stage of the pipeline and see that the Web.config of each stage has the correct iKey. Spinning up the website for that stage and clicking around in it should cause telemetry to be sent to the respective AI resource.

Install Status Monitor on the IIS server

The procedure is quite straightforward as follows:

  1. On your IIS server (ALMWEB01 if you are following the blog series) download and run the Status Monitor installation package from here.
  2. With the installation complete you'll need to sign in to your Microsoft Account after which you'll be presented with a configuration panel where the CU-DAT, CU-DQA and CU-PRD websites should have been discovered. The control panel lets you specify a separate AI resource for each website after which you'll need to restart IIS:
  3. In order to ensure that the domain accounts that the websites are running under have sufficient permissions to collect data make sure that they have been added to the Performance Monitor Users Windows local group.

With this configuration complete you should click around in the websites to confirm that telemetry is being sent to the Azure portal.

Creating Web Tests to monitor HTTP Availability

The configuration for Web Tests takes place in the new Azure portal at There are two types of test -- URL ping and a more involved Multi-step test. I'm just describing the former here as follows:

  1. In the new portal navigate to the AI resource you want to create tests for and choose the Availability tile:
  2. This opens the Web Tests pane where you choose Add web test:
  3. In the Create test pane supply a name and a URL and then use the arrow on Test Locations to choose locations to test from:

After clicking Create you should start to see data being generated within a few seconds.

In Conclusion

AI is clearly a very sophisticated solution for providing rich telemetry about your application and the web server hosting and I'm exited about the possibilities it offers. I did encounter a few hurdles in getting it to work though. Initial connection to the Azure portal when trying to integrate the SDK with Contoso University was the first problem and this caused quite a bit of messing around as each failed installation had to be undone. I then found that with AI added to Contoso University the build on my TFS server failed every time. I'm using automatic package restore and I could clearly see what's happening: every AI NuGet package was being restored correctly with the exception of Microsoft.ApplicationInsights and this was quite rightly causing the build to fail. Locally on my development machine the package restore worked flawlessly. The answer turned out to be an outdated nuget.exe on my build server. The fix is to open an Administrative command prompt at C:\Program Files\Microsoft Team Foundation Server 12.0\Tools and run nuget.exe update -self. Instant fix! This isn't AI's fault of course, although it is a mystery why one of the AI NuGets brought this problem to light.

Cheers -- Graham

Continuous Delivery with TFS: Behind the Scenes of the RM Deployment Agent

Posted by Graham Smith on March 8, 20152 Comments (click here to comment)

As with many aspects of technology understanding how something works behind the scenes can be a real boon when it comes to troubleshooting. In this post in my blog series on implementing continuous delivery with TFS we take a look at the Release Management for Visual Studio Deployment Agent, and specifically how it does its thing. Bear in mind that I don't have any inside knowledge about the Deployment Agent and this post is just based on my own experience and observations.

Basic Hook-Up

The first step in eliminating easy errors with the Deployment Agent is to ensure that it is installed correctly and can communicate with the RM Server. The key question is whether your servers are part of a domain. If they are then the easiest way to configure RM is to create a domain account (RMDEPLOYER for example) and add this to the Manage Users section of the RM Client as a Service User. On target nodes add this domain account to the Administrators group and then install the Deployment Agent specifying the domain RMDEPLOYER as the service account. See this post for a bit more detail. If your servers are not part of a domain you will need to use shadow accounts which are simply local accounts that all have the same name and password. The only difference is that you add the different shadow accounts of the different nodes to the Manage Users section of the RM Client as a Service User -- make sure you use the machine name as well as the account name ie the Windows Account field should be $MyNodeName$\RMDEPLOYER.

The test that all is working is to see the servers listed as Ready in the RM Client under Configure Paths > Servers. Something I have observed in my demo environment is that when deployment nodes boot up before my all-in-one TFS machine they don't seem to communicate. When that happens I use a PowerShell script to remotely restart the service (eg Start-AzureVM -ServiceName $cloudservicename -Name $SERVERNAME).

In a production environment your circumstances could be wildly different from a clean demo setup. I can't cover all the scenarios here but if you are experiencing problems and you have a complicated environment then this post could be a good troubleshooting starting point.

Package Deployment Mechanism

When the agent is installed and running it polls the RM server for packages to deploy on a schedule that can be set in the RM client under Administration > Settings > Deployer Settings > Look for packages to deploy every. On my installation it was set at 28 seconds but if time is critical you may want to shorten that.

When the agent detects that it has a package to deploy or actions to perform it copies the necessary components over to C:\Users\RMDEPLOYER\AppData\Local\Temp\RM\T\RM (where RMDEPLOYER is the name of the account the Deployment Agent is running under which might be different in your setup). There are at least two types of folder that get created:

  • Deployer Tools. This contains any tools and their dependencies that are needed to perform tasks. These could be executables, PowerShell scripts and so on. They are organised in a folder structure that relates to their Id and version number in the RM server database. For example in my database XCopy Deployer (irxcopy.cmd) has Id = 12 Version = 2 in dbo.DeployerTool and is thus deployed to C:\Users\RMDEPLOYER\AppData\Local\Temp\RM\T\RM\DeployerTools\12\2.
  • Action or Component. These folders correspond to the actions that will take place on that node. The names are the same as the names in the Release Management client Deployment Log (from Releases > Releases). A sub folder (whose name includes the date and time and other more mysterious numbers) contains the tool, the files it will deploy or otherwise work with and a file called IR_ProcessAutoOutput.log which is the one displayed when clicking the View Log link in the Deployment Log:

Component folders warrant a little bit more analysis. What exactly gets deployed to the timestamped sub-folder is dependant on how the component is configured under Configure Apps > Components, specifically the Build Drop Location. If this is configured simply with a backslash (\) then all of the drop folder is deployed.  This can be further refined by specifying a specific folder, in which case the contents of that folder get deployed. For example the Contoso University\Deploy Web Site component specifies \_PublishedWebsites\ContosoUniversity.Web as the Build Drop Location folder which means that just the website files are deployed.

It's perhaps worth noting here that there are two mechanisms for the Deployment Agent to pull in files: UNC or HTTP(S). This is configured on a per-server basis in Configure Paths > Servers > Deployment Agent. UNC is much quicker than HTTP(S) but the latter method is the only choice if your node doesn't have access to the UNC path.

A final aspect to touch on is that over time the node would get choked with old deployments if no action were taken, and to guard against this the Deployment Agent runs a cleanup process on a schedule specified in Administration > Settings > Deployer Settings. This is something to look at if disk space is tight.

Debugging Package Deployment

Having described how package management works -- at least at a high level -- what are the troubleshooting options when a component is failing to deploy or run correctly? These are the logs that are available on a target node:

  • IR_ProcessAutoOutput.log -- saved to the action or component folder as above.
  • DeploymentAgent.log -- cumulative log saved to C:\Users\RMDEPLOYER\AppData\Local\Temp\Microsoft\ReleaseManagement\12.0\Logs.
  • $GUID$DeploymentAgent.log -- instance log saved to C:\Users\RMDEPLOYER\AppData\Local\Temp\Microsoft\ReleaseManagement\12.0\Logs. Not sure of the value of these since I've never seen them contain anything.

If between them these logs don't provide sufficient information for troubleshooting your problem you can increase the level of detail -- this post has the details but don't forget to restart the Microsoft Deployment Agent service. Additionally, if you have SMTP set up and working you will also receive a Deployment Failed! notification email. This can be particularly useful because it invariably contains the command line that failed. This leads on to a useful debugging technique where you rerun the failing command yourself. For example if the command was running a PowerShell script simply open the PowerShell console, switch to the correct folder and paste in the command from the email. Chances are that you will get a much more informative error message this way.

Final Thoughts

I know from personal experience that debugging RM components can be a frustrating experience. Typically it's a daft mistake with a parameter or something similar but sorting this type of problem out can really eat time. Do you have any tips for debugging components? Are there other error logs that I haven't mentioned? Please do share your experiences and findings using the comments.

Cheers -- Graham

Continuous Delivery with TFS: Automatically Versioning Assemblies as Part of The Build

Posted by Graham Smith on February 27, 201514 Comments (click here to comment)

In a previous post in this series on implementing continuous delivery with TFS we looked at how some simple tweaks to the build process can help with the goal of baking quality in. This post continues in the vein of making improvements to the pipeline by addressing the issue of assembly versioning. What issue is that, I hear some of you asking? It's the situation where your Visual Studio solution contains many projects (maybe dozens) and you want all the projects to have same assembly versioning, ie the details you would traditionally set in AssemblyInfo.cs. A Google search will reveal several ways to accomplish this but most techniques involve some maintenance when a new project is added. In this post I explain how to make a publicly available low maintenance solution work with the Release Management build process template. I should point out that this issue won't affect everyone, and if you or your business don't care about this issue then do feel free to ignore. It is quite interesting though as it involves editing a build process template.

TFSVersioning on CodePlex

If assembly versioning is important to you and you use TFS there is a good chance you've seen the TFSVersioning solution available on CodePlex. It's a very nice piece of work that versions all of your solution's assemblies as part of the build process. If a new project is added it automatically gets included, so it's a low maintenance solution. There are essentially two ways to use TFSVersioning -- with the build process template that the project provides or with your own process template. This latter technique is a little involved as it requires editing your build template, but it's the technique we need to use here since we would like to use the ReleaseTfvcTemplate.12.xml build process template that ships with Release Management 2013.4. It turns out that editing this template is quite a job and I'm indebted to my good friend, colleague and TFS guru Bharath Sundaresan for figuring out all of the complicated details. An additional required hurdle is that the project hasn't been updated for TFS 2013 but fortunately it's not a lot of work to remedy this. The TFSVersioning deployment pack is available from the Downloads page and it has some great documentation which I recommend reading before you begin.

Update TFSVersioning for TFS2013

The core component that we need to update for TFS 2013 is TfsBuild.Versioning.Activities.dll. To accomplish this follow these steps:

  1. Download the latest source code for TFSVersioning from the Source Code page and unzip to somewhere convenient.
  2. Navigate to the latest version under Prod and open BuildVersioning.sln. Remove the TfsBuild.Versioning.Activities.Tests and TfsBuild.Versioning.Activities.Tests.Stubs projects as we don't need to amend them for what we are doing here.
  3. Expand the References node of the TfsBuild.Versioning.Activities project and notice that the Microsoft.TeamFoundation.* references are marked as missing:
  4. Remove these references and replace them with the 2013 versions from C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\ReferenceAssemblies\v2.0.
  5. You shoud now be able to build a Release version of TfsBuild.Versioning.Activities.dll.

Once you have succesfully updated the project for TFS 2013 it's probably a good idea to make sure that a basic installation of TFSVersioning works. Follow these steps to verify this:

  1. Download and unzip the latest TFSVersioning deployment pack -- currently 2.0.1. Copy VersioningBuildTemplate20.xaml from the pack to the ContosoUniversity BuildProcessTemplates folder and check in to version control.
  2. Under ContosoUniversity create a new folder called CustomActivityStorage and copy over the new version of TfsBuild.Versioning.Activities.dll. Check in to version control.
  3. From Team Explorer in Visual Studio navigate to Builds > Actions > Manage Build Controllers.
  4. In Manage Build Controllers dialog choose Properties and in Manage Controller Properties set Version control path to custom assemblies to $/ContosoUniversity/CustomActivityStorage.
  5. Now create a test build definition, replacing the standard release template with VersioningBuildTemplate20.xaml and setting all required properties including the drop folder.
  6. Whilst editing the build definition set any properties under the Build Versioning and Build Versioning (Optional) sections as you wish. Refer to the documentation for TFSVersioning for details.
  7. Queue a manual build of the test build definition. Observe in the drops folder that the ContosoUniversity.* binaries all have the same File version.
Update the ReleaseTfvcTemplate.12 Release Template with the TFSVersioning Custom Activity

This process broadly follows the Harder Installation but More Instructive section of the TfsVersioning User and Development Guide however modifying ReleaseTfvcTemplate.12.xaml requires several more steps. Partly this is because TfsBuild.Versioning.Activities.dll contains more functionality that isn't referred to in the documentation and partly because ReleaseTfvcTemplate.12.xaml is missing some activities that (reading between the lines) were present in the template that was used by the TFSVersioning project.  In the instructions below I assume a degree of familiarity with editing release templates. If you need guidance take a look here for just one example of how to get started. You should be aware that there are two ways to edit templates: through the XAML designer and through notepad. The former is less prone to error but slow and the latter is much faster but with the distinct possibility of a copy and paste error. The technique I describe below also sets you up for relatively easy debugging of the process template since there is a good chance of not getting everything right first time.

  1. Install the updated TfsBuild.Versioning.Activities.dll to the Global Assembly Cache by opening a Visual Studio command prompt (from C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\Shortcuts if your shortcuts are missing in Windows 8.1) and issuing a command similar to gacutil -i "C:\Users\Graham\Downloads\tfsversioning-103318\Prod\V\Source\TfsBuild.Versioning.Activities\bin\Release\TfsBuild.Versioning.Activities.dll". 
  2. Copy C:\Program Files (x86)\Microsoft Visual Studio 12.0\Release Management\Client\bin\ReleaseTfvcTemplate.12.xaml to the ContosoUniversity BuildProcessTemplates folder and check in to version control. Chances are you already have a template with the same name so you'll probably want to change the name to ReleaseTfvcTemplate.12.Versioning.xaml or similar. Once checked in open this template in Visual Studio so it displays in the XAML editor.
  3. Set up the Visual Studio Toolbox to work with TfsBuild.Versioning.Activities.dll in a custom tab. You can reference the version in CustomActivityStorage. Note that you only need to add the VersionAssemblyInfoFiles item.
  4. Drag the VersionAssemblyInfoFiles activity from the toolbox to the workflow as the first item under Compile, Test and Publish. Feel free to give the activity a custom name. If you examine the properties of the activity you will see all the InArguments that need to be married up with either Variables or Arguments that do not yet exist in the process template:
  5. The arguments can be created as per the TfsVersioning User and Development Guide instructions using the Arguments editor but a faster way is to open the template in Notepad, copy the following values and append them to the <x:Members> section.
  6. The next step is to add the metatdata items that allow each of the above arguments to be set. Again, it's possible to use the Metadata editor but the faster Notepad way is to copy the following values and append them to the <mtbw:ProcessParameterMetadataCollection> section.
  7. Back in the XAML editor navigate to the Arguments editor and supply default values for some arguments as follows:
    1. AssemblyFileVersionPattern = "1.0.J.B"
    2. AssemblyInfoFilePattern = "AssemblyInfo.*"
    3. AssemblyVersionPattern = ""
    4. BuildNumberPrefix = 0
    5. DoCheckinAssemblyInfoFiles = False
    6. ForceCreateVersion  = False
    7. UseVersionSeedFile = False
    8. VersionSeedFilePath= "TfsVersion\VersionSeed.xml"
  8. Navigate to the Variables editor and create the following variables (you may need to Browse for Types to get some of the variable types):
    1. Name = BuildAgent; Variable Type = Microsoft.TeamFoundation.Build.Client.iBuildAgent; Scope = Compile, Test and Publish
    2. Name = BuildDetail; Variable Type = Microsoft.TeamFoundation.Build.Client.iBuildDetail; Scope = Compile, Test and Publish
    3. Name = BuildDirectory; Variable Type = String; Scope = Compile, Test and Publish
    4. Name = Workspace, Variable Type = Microsoft.TeamFoundation.VersionControl.Client.Workspace; Scope = Compile, Test and Publish
  9. From Toolbox > Team Foundation Build Activities add the following activities to the top of  Compile, Test and Publish so they appear in the order listed:
    1. Activity = GetBuildAgent; Result = BuildAgent
    2. Activity = GetBuildDetail; Result = BuildDetail
    3. Activity = GetWorkspace; Name = String.Format("{0}_{1}_{2}", BuildDetail.BuildDefinition.Id, Microsoft.TeamFoundation.LinkingUtilities.DecodeUri(BuildAgent.Uri.AbsoluteUri).ToolSpecificId, BuildAgent.ServiceHost.Name); Result = Workspace
  10. Return to the properties of the VersionAssemblyInfoFiles activity and use the ellipsis at the end of each row to replace Enter a VB expression with the correct value. The result should be as follows:
  11. As a final step in this section save all the changes and check them in to version control.
Testing the Updated ReleaseTfvcTemplate.12 Release Template

At long last we are in a position to test the new template. The easiest way is to edit the test build definition created above and replace VersioningBuildTemplate20.xaml with our updated ReleaseTfvcTemplate.12.xaml version. Set any properties as required and queue a new build. With luck you will have a successful build and a set of uniformly versioned assemblies!

If you are having difficulty in implementing the steps above the debugging process is reasonably straightforward. Once the build template has been added to the test build definition you can make changes to the template, save them and then check them in to version control. Simply queue a new build to check your changes.

The final piece of the jigsaw when everything is working is to edit ContosoUniversity_Main_Nightly to use the new version of the template. And to enjoy a well-deserved drink.

Cheers -- Graham

Continuous Delivery with TFS: Save to a Folder for Stages You Can’t Yet Deploy to

Posted by Graham Smith on February 16, 2015No Comments (click here to comment)

In previous posts in this blog series on continuous delivery with TFS our activities have been geared up to deploying the sample application to target servers -- or nodes as they are sometimes referred to. But what happens when for some reason you have an environment that's just not ready to be a target for automated deployment? Maybe the business is just not ready to go that far right now. Maybe there is some technical hurdle you have yet to overcome. On the other hand you have already experienced how easy it is to manage the configuration of your application with Release Management and how it can bring config order where once there was chaos.

If you find yourself in this position a possible interim solution is to use Release Management to create what I call Release Ready Builds™ of your application. These are builds which have the correct configuration for the environment they are destined for but which are saved to a staging disk location rather than being actually deployed to target servers. Deployment of these builds is still likely to be a manual process but that's probably what you are doing already and at least the configuration and any other jiggery-pokery can be taken care of by Release Management.

Create a New Stage

In order to illustrate this technique I'm going to add a new stage to the deployment pipeline called PRD. This will represent the production environment that for whatever reason I'm unable to deploy to using Release Management automation. Carry out the following in Release Management:

  1. Add a stage type called PRD from Administration > Manage Pick Lists > Stage Type.
  2. Create a new Agent-based environment called Contoso University\PRD from Configure Paths > Environments. I linked my web server (ALMWEB01) for convenience but in a non demo context you would probably want to use something more permanent such as a build server. This would of course entail installing the Deployment Agent on that server.
  3. Add the new environment to the Contoso University\DAT>DQA release path from Configure Paths > Agent-based Release paths. The stage should be fully automated with email notifications turned off and this is probably a good time to change the name of the release path to Contoso University\DAT>DQA>PRD.
  4. Whilst in the newly renamed Contoso University\DAT>DQA>PRD release path help to speed up the debugging process by making the DQA stage fully automated, removing any Approvers from the Approval Step and turning off email notifications.
  5. Open the Contoso University\DAT>DQA release template from Configure Apps > Agent-based Release Templates and in Properties change the name to Contoso University\DAT>DQA>PRD.
  6. On the web server (ALMWEB01 in my case) create a folder called C:\ReleaseReadyBuilds.
Configure New Components

With the PRD stage added we now need to create new components that will deploy to a folder structure. For the web application it's not really all that different because we're using XCopy anyway as the deployment method so it's just a case of specifying a new location. The database side of things is more interesting because we need to create scripts rather than run actions against a database. On top of all this we need a mechanism to ensure that each release is placed in a unique folder. To achieve this carry out the following steps in Release Management Configure Apps > Components:

  1. Create a new component called Contoso University\Script DACPAC and configure as follows:
    1. Source > Builds with application (= selected) > Path to package = \
    2.  Deployment:
      1. Tool = DACPAC Database Deployer
      2. Arguments = /Action:Script /SourceFile:"__FileName__"  /TargetServerName:"__ServerName__" /TargetDatabaseName:"__DatabaseName__" /OutputPath:"__OutputPath__"
      3. Parameters > OutputPath > Description = The location for the DACPAC script file
  2. Create a new component called Contoso University\Script Login & User SQL Script and configure as follows:
    1. Source > Builds with application (= selected) > Path to package = \Scripts
    2. Deployment:
      1. Tool = Windows Common IO
      2. Arguments = -File ./ManageWindowsIO.ps1 -Action __Action__ -FileFolderName "__FileFolderName__" -DestinationName "__DestinationName__"
      3. Parameters > DestinationName > Description = Location to copy the file to
    3. Configuration Variables:
      1. Variable Replacement Mode = Before Installation
      2. File Extension Filter = *.sql
      3. Parameter #1 = LOGIN_OR_USER | Standard | Name of login or user to create
      4. Parameter #2 = DB_NAME | Standard | Database to set security for
  3. Create a new component called Contoso University\RRB and configure as follows:
    1. Source > Builds with application (= selected) > Path to package = \
    2. Deployment:
      1. Tool = Windows Common IO
      2. Arguments = -File ./ManageWindowsIO.ps1 -Action __Action__ -FileFolderName "__FileFolderName__" -DestinationName "__DestinationName__"
      3. Parameters > DestinationName > Description = Destination name
Configure the PRD Stage

Next we need to configure the PRD stage. To achieve this carry out the following steps in the Contoso University\DAT>DQA>PRD agent-based release template:

  1. In order to speed up the debugging process visit the DAT and DQA stages and click the top left hand icon of every action or component to toggle the skip state:
  2. In the PRD stage drag the ALMWEB01 server to the Deployment Sequence.
  3. Drag the Contoso University\Deploy Web Site to ALMWEB01 and set the parameters as follows:
    1. Installation Path  = C:\ReleaseReadyBuilds\PRD\Web
  4. Navigate to Toolbox > Windows OS and drag a Create Folder action below Contoso University\Deploy Web Site and set the FolderName to C:\ReleaseReadyBuilds\PRD\Database.
  5. Navigate to Toolbox > Components and right-click the Components node. Add Contoso University\Script DACPAC.
  6. Drag the Contoso University\Script DACPAC component below the Create Folder action and set the parameters as follows:
    1. FileName = ContosoUniversity.Database.dacpac
    2. ServerName = ALMSQL01
    3. DatabaseName = CU-PRD
    4. OutputPath = C:\ReleaseReadyBuilds\PRD\Database\DACPAC.sql
  7. Navigate to Toolbox > Components and right-click the Components node. Add Contoso University\Script Login & User SQL Script.
  8. Drag the Contoso University\Script Login & User SQL Script component below the Contoso University\Script DACPAC component and set the parameters as follows:
    1. Action = move
    2. FileFolderName = Create login and database user.sql
    3. DestinationName = C:\ReleaseReadyBuilds\PRD\Database\Create login and database user.sql
    5. DB_NAME = CU-PRD
  9. Navigate to Toolbox > Components and right-click the Components node. Add Contoso University\RRB.
  10. Drag the Contoso University\RRB component below the Contoso University\Script Login & User SQL Script component and set the parameters as follows:
    1. Action = create
    2. FileFolderName = C:\ReleaseReadyBuilds\$(BuildNumber)
  11. Drag a second Contoso University\RRB component below the first one and set the parameters as follows:
    1. Action = move
    2. FileFolderName = C:\ReleaseReadyBuilds\PRD
    3. DestinationName = C:\ReleaseReadyBuilds\$(BuildNumber)\PRD
Check the Stage Works

Navigate to Releases > Releases and manually create a release based on the latest build. Because the previous stages are in skip mode the PRD stage should finish almost immediately and you should very quickly be able to verify everything is working. Essentially you should be checking for a folder similar to C:\ReleaseReadyBuilds\ContosoUniversity_Main_Nightly_20150214.8 that contains a PRD folder with Web and Database folders containing our deployable artefacts. The magic that achieves this is the $(BuildNumber) configuration variable that is one of several that are only available in components. If that's all working then you should revisit the DAT and DQA stages and toggle all the component and actions so that they are not skipped, but leave DQA in automated mode. Now run a couple of complete builds from the nightly build definition in Visual Studio, confirming that each PRD build is placed in its own separate folder. Finally, when that's working you can revisit the DQA stage and return it to its manual status.

By way of finishing off I want to stress that you should be doing everything possible to use the full automation capabilities of Release Management to deploy your application along the whole delivery pipeline. But when you really do have a problem with a particular stage hopefully the technique I've illustrated here will tide you over.

Cheers -- Graham

Continuous Delivery with TFS: Promoting a Release to the DQA Stage

Posted by Graham Smith on February 14, 2015No Comments (click here to comment)

In a previous post in this series on implementing continuous delivery with TFS we arrived at the point of being able to run automated web tests on a build of the application deployed to the DAT environment. Any build that passes this stage is a candidate to be promoted to the next stage in the pipeline, and in my demo environment this is DQA (development quality assurance). The scenario I had in mind when building the demo environment was that it would be used by a Scrum team delivering features from a backlog of Product Backlog Items. Part of this development effort would include some manual testing, either of features that are impossible or impractical to test automatically ("text is a certain colour" for example), or for features where the effort of writing an automated test wouldn't make sense. In my demo environment the DQA stage is where this manual testing activity would take place.

Typically, not every build that passes the DAT stage will be promoted to the DQA stage as there may be insufficient change to warrant manual inspection. Additionally, if someone is in the middle of running some manual tests in DQA they probably don't want to find that the build they are testing has suddenly been replaced by a new one. Consequently the deployment of builds that do need to be promoted to DQA will need to be triggered manually. This leads on to questions such as who should trigger a deployment to DQA, how can they decide which build to promote if there are choices to be made and which tools should be used? (Note that I'm assuming here that any build that passes the DAT stage is a candidate for promotion to DQA. It's perfectly possible though to build in an approval step at the end of the DAT stage to require someone to explicitly indicate that a build can be promoted to DQA.)

Revisiting the Release Management Approvals Workflow

In order to understand the options that are available for answering the questions I have just posed a good starting point is to revisit the Release Management approvals workflow for the DQA stage by opening Release Management and navigating to Configure Paths > Agent-based Release Paths and opening the Contoso University\DAT>DQA release path. In a previous post we configured the DQA stage so that each stage is manual and all activity is governed by members of the Quality Assurance security group:


I am well aware that The Scrum Guide "recognizes no titles for Development Team members other than Developer" and I'm not going to go in to the ins and outs of which team members should be allowed to govern the deployment of releases. Suffice it to say though that Release Management provides plenty of options for whatever scenario you are working with, and each step can be either a named individual or a security group consisting of one of more individuals.

Looking at the DQA stage image in closer detail there are some subtle details (highlighted in red) that are worth mentioning. Firstly, it's possible to toggle email notifications for each step by clicking the email icon, which changes to a brighter image when notifications are turned on. Secondly, it's possible to have multiple Approvers for the Approval Step. Note that this means that all Approvers need to approve before the build can become a candidate for the next stage. (A further stage hasn't been defined in my demo environment so the Approval Step concept doesn't make complete sense here, but you get the idea.)

Managing the Governance of the Release Process

Having looked at the options for setting up who should be allowed to govern the release to DQA process we can now turn to the actual process of managing it. Although the Release Management client allows for the management of the approvals process (Releases > My Approval Requests) there is another tool called Release Explorer (actually a web page) which is dedicated to this purpose. This simplifies things somewhat since you don't need to install the client on lots of PCs and you don't need to have lots of users needing to get to grips with the client user interface. The Release Explorer URL will be something like http://almtfsadmin:1000/ReleaseManagement -- obviously if your server name is different then so will the URL be. The first time you open Release Explorer can be a bit of a shock -- it's ugly and of limited functionality to say the least. The Release Management User Guide has instructions on how to use what functionality is present.

With DQA configured as above and with all email notifications turned on triggering a build of ContosoUniversity_Main_Nightly will result in the following process once the DAT stage is complete:

  1. Anyone in the Acceptance StepApprover role receives an Deployment Acceptance Required! email.
  2. Release Explorer lists the release as waiting for Acceptance. If you have multiple releases listed and you have a reasonably descriptive release template name chances are that you'll need to hover over the release name to show the date and time as a popup (highlighted in red below):
  3. Still in Release Explorer the release can be selected and clicking the Approve button displays a dialog for confirming the actual deployment to DQA with the ability to add a comment.
  4. At the start of the deployment anyone in the Deployment StepOwner role receives a Deployment in Progress email.
  5. At the end of the deployment anyone in the Validation Step > Validator role receives a Deployment Validation Required! email.
  6. Release Explorer lists the release as waiting for Validation. A person in the Validator role would presumably navigate to the application and make sure it is working as expected or in a more sophisticated setup start a series of automated validation smoke tests.
  7. Still in Release Explorer the release can be selected and clicking the Approve button displays a dialog for confirming the successful deployment to DQA with the ability to add a comment.
  8. Anyone in the Approval Step > Approver role receives a Deployment Approval Required! email.
  9. Release Explorer lists the release as waiting for Approval. All the people in the Approver role now need to assure themselves by whatever means that, in the words of the email, the "deployed application meets the minimum quality requirements needed to complete this stage".
  10. Still in Release Explorer the release can be selected and clicking the Approve button displays a dialog for confirming the successful release to DQA with the ability to add a comment.
  11. In Release Management under Releases > Releases the release now has the status Released.

The full process can involve a lot of emails and a lot of individual steps. It might be overkill for every stage of your deployment but it's good to know that there is a comprehensive set of options should you need them.

Which Release to Promote to DQA?

Unfortunately the workflow around determining which release to actually promote to DQA is somewhat clunky and I can only hope that this will be improved in a future version of Release Management. Of course if you just want the latest and greatest then it's easy to work that out from Release Explorer as it will be the first release in the list. Another scenario though is where the development team are coding features against PBI Task work items and fixing bugs against Bug work items and checking code in against those work items as appropriate. In this case someone who is responsible for manual testing may want to choose a specific release that contains certain features or fixed bugs. Each build report has a list of Associated work items and either Visual Studio or Team Web Access can be used to inspect build reports but unfortunately there is nothing in Release Explorer. Of course each build report only lists what has been associated since the last build so there may be some effort required to find the build that contains all the desired work items.

Regrettably once the desired build has been identified in Visual Studio or Team Web Access there doesn't appear to be an exact way to match it with a release listed in Release Explorer since Release Explorer identifies a release by date and time and the build report uses date and build number for that day. Of course if only one release can become a candidate for promotion each day (because you run DAT overnight for example) then disambiguation will be easy but in a more fast-paced environment that may not be the case.

That completes our examination of the workflow for promoting a release to the DQA environment. If you have any ideas for fine tuning it do let me know via the comments.

Cheers -- Graham

Continuous Delivery with TFS: Making Sense of the DSC Feature in Release Management

Posted by Graham Smith on February 8, 20154 Comments (click here to comment)

When I first started listing the draft titles of blog posts for my series on implementing continuous delivery with TFS naturally the vNext / Agent-less / PowerShell DSC feature of Release Management that shipped with 2013.3 was on my list. And why not? Surely this was the successor to the agent-based way of doing things? Out with the old and in with the new...

Naturally I'd looked in to PowerShell DSC and knew that it was touted as a make-it-so technology for configuring Windows servers: rather than using an imperative script to install and configure components one uses a declarative approach that describes what a server should look like and PowerShell DSC goes off and does one's bidding, so to speak. It wasn't immediately obvious how the new vNext features of Release Management would relate to the delivery pipeline I was building in Azure but I trusted that time would tell. Well time has now told and as far as my research is concerned I can't see that the vNext features have any part to play. Deploying a DACPAC? Running automated tests via Microsoft Test Manager? The vNext features appear to be irrelevant.

Interestingly I'm not the only one who has come to the conclusion that vNext is not a must-do replacement for agent-based deployments. Both Colin Dembovsky and Donovan Brown have recently blogged on similar lines. So what is the point of the vNext features in Release Management? Clearly if you want to ensure that your environment is configured correctly before you deploy your components then a vNext release template might be the way to go. But most organisations are probably (or should be) thinking about automating the configuration of their servers at a higher, more global level, not just when it comes to triggering an actual deployment. Certainly at the time of writing this post I think I'm right in saying that if you want to use Release Management with Visual Studio Online you have to use a vNext release template, but this just feels like Microsoft haven't implemented using agent-based release templates yet.

Although I'm planning to cover PowerShell DSC in a different blog post series as far as this series is concerned I'm not going to complicate things by covering the vNext way of implementing releases as it feels like it won't add much value and will be entering a world of unnecessary rework and pain. Disagree? Sound off in the comments...

Cheers -- Graham

Continuous Delivery with TFS: Running Automated Web Tests with MTM

Posted by Graham Smith on February 3, 20152 Comments (click here to comment)

In this instalment of my series on implementing continuous delivery with TFS we pick up where we left off in the previous post and add the automated web tests we created to Microsoft Test Manager. We then look at how to schedule these tests for automatic execution through the deployment pipeline. Exciting stuff so lets get started...

Configure a Test Controller

Our starting point is to configure our TFS admin machine as a test controller. (You could create a dedicated machine of course but in our demo environment it's an unnecessary overhead.)

  1. Create a new domain account called TFSTEST and add to the Administrators group on your ALMTFSADMIN machine.
  2. Download the Agents for Microsoft Visual Studio 2013. See here for 2013.4 releases.
  3. Mount the Agents for Microsoft Visual Studio 2013 iso and from the TestController folder install the test controller on ALMTFSADMIN configuring it to run with the TFSTEST account and for the appropriate Team Project Collection:
  4. Click Apply settings to ensure that the configuration steps were successful.
Configure a Web Client Test Machine

The big picture here is that we will deploy the Contoso University application to the DAT (Development Automated Test) environment and then run the automated tests against DAT. We won't be using our development machine and instead will make use of a dedicated web client test machine which should be configured as follows:

  1. Create an new Azure VM from a Windows 8.1 Enterprise (x64) image called ALMCLIENTWIN81. (Note that Windows desktop editions are only available to MSDN subscribers).
  2. Add the RMDEPLOYER and TFSTEST to the Administrators group.
  3. Install the Release Management Deployment Agent and ensure it is in the Ready status in the Release Management client. (See here for details.)
  4. Install Microsoft Test Manager 2013.
  5. Mount the Agents for Microsoft Visual Studio 2013 iso (see above) and from the TestAgent folder install the test agent configuring it to run with the TFSTEST account and to register with the test controller installed above:
  6. Click Apply settings to ensure that the configuration steps were successful.
  7. Install FireFox.
Configure Microsoft Test Manager

As usual I'm assuming you have a degree of familiarity with the tooling, in this case Microsoft Test Manager 2013. If not then see my getting started with Microsoft Test Manager blog post here. You will need to have installed MTM on to your development machine and successfully connected to your ContosoUniversity Team Project. Now carry out the following steps.

  1. Open MTM and connect to Lab Center. Navigate to Controllers and ensure that ALMTFSADMIN is registered as the Test Controller and that the Test Agent on ALMCLIENTWIN81 is in the Ready status:
  2. Navigate to Lab and create a new Standard environment called ALMCLIENTWIN81. From the Machines page add ALMCLIENTWIN81 and give it the Web Client role and specify credentials which are in the Administrators group on ALMCLIENTWIN81. From the Verification page click Verify to initiate the verification procedure. If all verifications pass you should end up with a new environment in the ready state:
  3. Navigate to Test Settings and create a new entry called Automated Test Run and choose Automated for the What type of tests do you want to run? question. From the Roles page choose the Web Client role which should match the environment created above. From the Data and Diagnostics page choose everything except Screen and Voice Recorder and feel free to configure each option as you see fit. Click on Finish to save the settings and close MTM.
  4. Open MTM and connect to Testing Center. If this is the first time you have connected you will need to Add a Test Plan:
  5. Add a test plan called Regression and then choose Select plan to open it at the Contents tab. Add a new suite called Department by right-clicking the Regression node and choosing New suite:
    microsoft-test-manager-new suite
  6. Now over in the right-hand pane create two new test cases called Can_Navigate_To_Departments and  Can_Create_Departmemt:
  7. Note the IDs of these new test cases and switch to Visual Studio, opening the ContosoUniversity solution if it isn't already open. From Team Explorer > Work Items search for each of the two new test cases and in turn open them to carry out the next step for both test cases.
  8. Click on the Associated Automation tab and then choose the ellipsis at the right of the Automated test name field. This brings up the Choose Test dialog from where you can select the required test:
  9. Make sure to save the test case work items after associating the automated tests.
  10. Back in Microsoft Test Manager navigate to the Run Settings tab of the test plan. Under Automated runs set Test settings = Automated Test Run and Test environmentALMCLIENTWIN81 -- the items we created above. Make sure you Save and Close.
Configure Release Management

With our automated web tests incorporated in to an MTM Test Plan the next step is to configure Release Management to run them as part of the DAT stage of the deployment pipeline. Carry out the following steps from within Release Management:

  1. From Configure Paths > Servers add ALMCLIENTWIN81 and ensure it is in the Ready status.
  2. From Configure Paths > Environments add ALMCLIENTWIN81 via the Link Existing button.
  3. From Configure Apps > Components create a new component called Contoso University\Run MTM Auto Tests and configure as follows:
    1. Source > Builds with application (= selected) > Path to package = \
    2. Deployment > Tool = MTM Automated Tests Manager
    3. Configuration Variables > Variable Replacement Mode = Only in Command
  4. From Configure Apps > Agent-based Release Templates open Contoso University\DAT>DQA and edit the Deployment Sequence for DAT as follows:
    1. From Toolbox > Servers drag ALMCLIENTWIN81 to the Deployment Sequence below ALMSQL01.
    2. From Toolbox > Components right click the node and add Contoso University\Run MTM Auto Tests.
    3. Now drag Contoso University\Run MTM Auto Tests to ALMCLIENTWIN81 and configure the compnent as follows:
      1. TestRun Title = Department Tests
      2. PlanId = 28 (or whatever your Plan ID is as it is likely to be different)
      3. SuiteId = 29 (or whatever your Suite ID is as it is likely to be different)
      4. ConfigId = 1 (or whatever your Configuration ID is as it is likely to be different)
      5. TfsCollection = http://almtfsadmin:8080/tfs/PrmCollection (or whatever your Team Collection URL is as it is likely to be different)
      6. TeamProject = ContosoUniversity
      7. TestEnvironment = ALMCLIENTWIN81
      8. Build Directory = $(PackageLocation)
Some Final Configuration

In order for automated tests to run I've found that as a minimum the Deployment Agent account (RMDEPLOYER) needs to be in the Project Collection Administrators group.  It may be possible to fine tune this requirement but it's a level of trial and error I haven't had time to perform. This permission can be granted on your TFS admin machine from Team Foundation Server Administration Console > $(TfsAdminMachineName) > Application Tier > Team Project Collections > $(TeamProjetName) > General > Group Membership > [$(TeamProjetName)]\Project Collection Administrators > Properties.

If your Visual Studio solution contains a mix of unit test and automated test projects and the build definition is configured to run unit tests you are going to want to ensure the automated tests are excluded as they will fail and cause the build to report it only partially succeeded. There are various ways to do this but the technique I've used is to modify all build definitions to change Process > Build Process Parameters > 3. Test > 1. Automated tests > 1. Test source > Test sources spec to unittest where it was previously just test. This obviously ties in with my project naming convention of ContosoUniversity.Web.UnitTests and ContosoUniversity.Web.AutoTests.

Start a Deployment

From Visual Studio manually queue a new build from ContosoUniversity_Main_Nightly. If everything is in place the build should succeed and you can open Microsoft Test Manager to check the results. Navigate to Testing Center > Test > Analyze Test Runs. You should see your test run listed and double-clicking it will open up the fruits of all our endeavours so far, the test run results:



If getting to this stage felt like it required a huge amount of detailed configuration you are probably right. There are a great many moving parts to get working correctly and any future simplification of the process by Microsoft in the in the future will be welcomed.

Although we've reached a major milestone on this continuous delivery journey there is still plenty more to talk about so do watch out for the next installation.

Cheers -- Graham

Continuous Delivery with TFS: Creating Automated Web Tests with Selenium

Posted by Graham Smith on January 26, 201510 Comments (click here to comment)

In this blog post (which is part of my series on on implementing continuous delivery with TFS) we look at creating automated web tests with Selenium. Although the need for manual testing will probably never go away any organisation that wants to deploy software on a frequent basis is going to have to automate some if not most of the application testing effort if quality has any hope of being maintained. Clearly, if you want to release your application once a week but it takes two weeks to properly manually test your application you have a big problem, and automation is probably the only sensible answer.

Let's be clear that test automation is a mammoth subject and not only involves automating the driving of the application but also data setup and teardown and service virtualisation ie stubbing interfaces to external systems. Then there is managing your tests between sprints or projects -- when you have 500 or so this is a non-trivial task. I can't possibly cover the subject properly in a couple of posts and instead my aim is to show you how to begin to create a framework for writing automated tests and crucially how to make them run in the TFS ecosystem. I'll be using Selenium since it's used by John Sonmez in his superb Pluralsight course Creating an Automated Testing Framework With Selenium. I thoroughly recommend this course to anyone who is embarking on an automated testing project, and some of the code I use comes directly from John's course (with his kind permission). If you are using a different automation technology such as CodedUI similar principles will apply. As with many other things in software development there is a quick and dirty way to write automated tests (by recording them which makes them very brittle -- think recorded Excel macros) and there is the professional approach where you put the time and effort in to make a much more robust solution. I'll be using the latter technique although I will only scratch the surface of what's possible.

Create the Framework

Our simple automation solution will be in two parts, the first of which is a fluent API framework that models the web application's web pages. To get the ball rolling create a new class library project in the ContosoUniversity solution called ContosoUniversity.Web.SeFramework and complete these steps:

  1. Via NuGet install Selenium WebDriver and Selenium WebDriver Support Classes.
  2. Remove Class1.cs and add a class called Driver.cs. Ensure Driver.cs contains the following code:
  3. Open the properties of ContosoUniversity.Web and from the Web tab find the IIS Express port number. Replace the four zeros in the above code with the port number.
  4. Add a new class called DepartmentsPage.cs and ensure it contains the following code:
  5. Add a new class called NewDepartmentPage.cs and ensure it contains the following code:
Create the Tests

To keep everything in layers we'll create our tests in a new Unit Test project. We need to use the MSTest framework because (jumping ahead here) Microsoft Test Manager only works with MSTest at the moment. So, add a new Unit Test project called ContosoUniversity.Web.AutoTests and complete these steps:

  1. Create a project reference to ContosoUniversity.Web.SeFramework.
  2. Add a new class called ContosoTest.cs and ensure it contains the following code:
  3. Add a new class called DepartmentTests.cs and ensure it contains the following code:
  4. Build the solution so the tests are found in Test Explorer. Select the Can_Cerate_Department and Can_Navigate_To_Departments tests then add them to a new playlist by right-clicking and selecting Add to Playlist. Save the playlist in the root of ContosoUniversity.Web.AutoTests.
Running the Tests

In order to run the tests using the code above you'll need to install FireFox. You can't just run the tests though because you need to take steps to ensure there is a web server to handle the requests. The technique is to run the solution so the application starts and IIS Express is running (the icon should appear in the System Tray). Then from the Debug menu in Visual Studio select Detach All. This will close the browser that is running the application but will leave IIS Express running. You can now run the tests (make sure you are running in a Debug configuration) and watch Selenium do all the hard work!

What Have we Done Here?

I want to reiterate that this post isn't meant to be a tutorial on how to write automated tests and you should regard the code as an example of how to get up-and-running quickly rather than being of production-ready quality. Although this isn't a tutorial some explanation of what's going on is in order.

In the ContosoUniversity.Web.SeFramework project the Driver class is essentially the link to FireFox and also has some methods (not used here) to help with tricky web pages that take time to do their thing. The DepartmentsPage has a method to navigate to the page that lists all of the departments and a helper property and method for use in Assert statements. The interesting class is NewDepartmentPage since this illustrates the technique for writing code that allows the use of a fluent syntax for writing tests.

In the ContosoUniversity.Web.AutoTests project the ContosoTest class is a base class that contains common functionality (starting and stopping the browser) for actual test classes to inherit. DepartmentTests inherits ContosoTest and uses the fluent API to cause the browser to navigate around the application doing things that we want to test. It's all very basic but hopefully enough to illustrate the huge potential of this type of testing.

With our automated web tests created we now need to configure the infrastructure and tooling to automate their execution according to a schedule. Watch out for the next instalment when we'll be doing just that.

Cheers -- Graham