Continuous Delivery with TFS / VSTS – Enhancing a CI Build to Help Bake Quality In

Posted by Graham Smith on February 16, 20164 Comments (click here to comment)

In the previous instalment of this blog post series on Continuous Delivery with TFS / VSTS we created a basic CI build. In this post we enhance the CI build with further configurations that can help bake quality in to the application. Just a reminder that I’m using TFS to create my CI build as it’s the lowest common denominator. If you are using VSTS you can obviously follow along but do note that screenshots might vary slightly.

Set Branch Policies

Although it's only marginally related to build this is probably a good point to set branch policies for the master branch of the ContosoUniversity repository. In the Web Portal for the team project click on the cog icon at the far right of the blue banner bar:

web-portal-control-panel-icon

This will open up the Control panel at the team project administration page. Navigate to the Version Control tab and in the Repositories pane navigate down to master. In the right pane select Branch Policies:

web-portal-control-panel-branch-policies

The branch policies window contains configuration settings that block poor code from polluting the code base. The big change is that the workflow now changes from being able to commit to the master branch directly to having to use pull requests to make commits. This is a great way of enforcing code reviews and I have more detail on the workflow here. In the screenshot above I've selected all the options, including selecting the ContosoUniveristy.CI build to be run when a pull request is created. This blocks any pull requests that would cause the build to fail. The other options are self explanatory, although enforcing a linked work item can be a nuisance when you are testing. If you are testing on your own make sure you Allow users to approve their own changes otherwise this will cause you a problem.

Testing Times

The Contoso University sample application contains MSTest unit tests and we want these to be run after the build to provide early feedback of any failing tests. This is a achieved by adding a new build step. On the Build tab in the Web Portal navigate to the ContosoUniversity.CI build and place it in edit mode. Click on Add build step and from the Add Tasks window filter on Test and choose Visual Studio Test.

For our simple scenario there are only three settings that need addressing:

  • Test Assembly -- we only want unit tests to run and ContosoUniversity contains other tests so changing the default setting to **\*UnitTests*.dll;-:**\obj\** fixes this.
  • Platform -- here we use the $(BuildPlatform) variable defined in the build task.
  • Configuration -- here we use the $(BuildConfiguration) variable defined in the build task.

web-portal-visual-studio-test-unit-test-configuration

With the changes saved queue the build and observe the build report advising that the tests were run and passed:

web-portal-build-build-succeeded-with-unit-tests-passing

Code Coverage

In the above screenshot you'll notice that there is no code coverage data available. This can be fixed by going back to the Visual Studio Test task and checking the Code Coverage Enabled box. Queueing a new build now gives us that data:

web-portal-build-build-succeeded-with-code-coverage-enabled

Of slight concern is that the code coverage reported from the build (2.92%) was marginally higher than that reported by analysing code coverage in Visual Studio (2.89%). Whilst the values are the same for all practical purposes the results suggest that there is something odd going on here that warrants further investigation.

Code Analysis

A further feedback item which is desirable to have in the build report is the results of code analysis. (As a reminder, we configured this in Visual Studio in this post so that the results are available after building locally.) Displaying code analysis results in the build report is straightforward for XAML builds as this is an out-of-the-box setting -- see here. I haven't found this to be quite so easy with the new build system.There's no setting as with XAML builds but that shouldn't be a problem since it's just an MSBuild argument. It feels like the correct argument should be /p:RunCodeAnalysis=Always (as this shouldn't care how code analysis is configured in Visual Studio) however in my testing I couldn't get this to work with any combination of the Visual Studio Build / MSBuild task and release / debug configurations. Next argument I tried was /p:RunCodeAnalysis=True. This worked with either Visual Studio Build or MSBuild task but to get it to work in a release configuration you will need to ensure that code analysis has been enabled for the release configuration in Visual Studio (and the change has been committed!). The biggest issue though was that I never managed to get more than 10 code analysis rules displayed in the build report when there were 85 reported in the build output. Perhaps I'm missing something here -- if you can shed any light on this please let me know!

Don't Ignore the Feedback!

Finally, it may sound obvious but there's little point in configuring the build report to contain feedback on the quality of the build if nobody is looking at the reports and is doing something to remedy problems. However you do it this needs to be part of your team's daily routine!

Cheers -- Graham

  • Ryan

    I have implemented within my company’s local TFS CI last year almost everything in this article
    – MSTest: I had to custom the assemblies naming convention to only execute in-house code not third party
    – Code Coverage: for C# and JavaScript and fail the build check-in when it goes below the custom threshold
    – Code Analysis: I custom the Code Analysis to be more stringent
    – What I also did was to create a weekly schedule build to run Code Metrics and a few others to generate a report. Identify when we need to refactor code.

    • Graham Smith

      Thanks very much for sharing your success story Ryan! Sounds like you are really making use of all the tooling.

      Cheers – Graham

  • Alan

    As an alternative to the /p:RunCodeAnalysis msbuild flag you can drop in this code quality build task I created based on Resharper code inspections, will break the build when code quality metrics are violated

    https://marketplace.visualstudio.com/items?itemName=alanwales.resharper-code-analysis

  • Graham Smith

    Hi Alan

    Many thanks for the link to your task – looks very useful indeed!

    Cheers – Graham