Continuous Delivery with TFS / VSTS – Automated Acceptance Tests with SpecFlow and Selenium Part 2

Posted by Graham Smith on July 25, 2016One Comment (click here to comment)

In part-one of this two-part mini series I covered how to get acceptance tests written using Selenium working as part of the deployment pipeline. In that post the focus was on configuring the moving parts needed to get some existing acceptance tests up-and-running with the new Release Management tooling in TFS or VSTS. In this post I make good on my promise to explain how to use SpecFlow and Selenium together to write business readable web tests as opposed to tests that probably only make sense to a developer.

If you haven't used SpecFlow before then I highly recommend taking the time to understand what it can do. The SpecFlow website has a good getting started guide here however the best tutorial I have found is Jason Roberts' Automated Business Readable Web Tests with Selenium and SpecFlow Pluralsight course. Pluralsight is a paid-for service of course but if you don't have a subscription then you might consider taking up the offer of the free trial just to watch Jason's course.

As I started to integrate SpecFlow in to my existing Contoso University sample application for this post I realised that the way I had originally written the Selenium-based page object model using a fluent API didn't work well with SpecFlow. Consequently I re-wrote the code to be more in line with the style used in Jason's Pluralsight course. The versions are on GitHub -- you can find the ‘before' code here and the ‘after' code here. The instructions that follow are written from the perspective of someone updating the ‘before' code version.

Install SpecFlow Components

To support SpecFlow development, components need to be installed at two levels. With the Contoso University sample application open in Visual Studio (actually not necessary for the first item):

  • At the Visual Studio application level the SpecFlow for Visual Studio 2015 extension should be installed.
  • At the Visual Studio solution level the ContosoUniversity.Web.AutoTests project needs to have the SpecFlow NuGet package installed.

You may also find if using MSTest that the specFlow section of App.config in ContosoUniversity.Web.AutoTests needs to have an <unitTestProvider name="MsTest" /> element added.

Update the Page Object Model

In order to see all the changes I made to update my page object model to a style that worked well with SpecFlow please examine the ‘after' code here. To illustrate the style of my updated model, I created CreateDepartmentPage class in ContosoUniversity.Web.SeFramework with the following code:

The key difference is that rather than being a fluent API the model now consists of separate properties that more easily map to SpecFlow statements.

Add a Basic SpecFlow Test

To illustrate some of the power of SpecFlow we'll first add a basic test and then make some improvements to it. The test should be added to ContosoUniversity.Web.AutoTests -- if you are using my ‘before' code you'll want to delete the existing C# class files that contain the tests written for the earlier page object model.

  • Right-click ContosoUniversity.Web.AutoTests and choose Add > New Item. Select SpecFlow Feature File and call it Department.feature.
  • Replace the template text in Department.feature with the following:
  • Right-click Department.feature in the code editor and choose Generate Step Definitions which will generate the following dialog:
  • By default this will create a DepartmentSteps.cs file that you should save in ContosoUniversity.Web.AutoTests.
  • DepartmentSteps.cs now needs to be fleshed-out with code that refers back to the page object model. The complete class is as follows:

If you take a moment to examine the code you'll see the following features:

  • The presence of methods with the BeforeScenario and AfterScenario attributes to initialise the test and clean up afterwards.
  • Since we specified a value for Budget in Department.feature a method step with a (poorly named) parameter was created for reusability.
  • Although we specified a name for the Administrator the step method wasn't parameterised.

As things stand though this test is complete and you should see a NewDepartmentCreatedSuccessfully test in Test Explorer which when run (don't forget IIS Express needs to be running) should turn green.

Refining the SpecFlow Test

We can make some improvements to DepartmentSteps.cs as follows:

  • The GivenIEnterABudgetOf method can have its parameter renamed to budget.
  • The GivenIEnterAnAdministratorWithNameOfKapoor method can be parameterised by changing as follows:

In the preceding change note the change to both the attribute and the method name.

Updating the Build Definition

In order to start integrating SpecFlow tests in to the continuous delivery pipeline the first step is to update the build definition, specifically the AcceptanceTests artifact that was created in the previous post which needs amending to include TechTalk.SpecFlow.dll as a new item of the Contents property. A successful build should result in this dll appearing in the Artifacts Explorer window for the AcceptanceTests artifact:


Update the Test Plan with a new Test Case

If you are running your tests using the test assembly method then you should find that they just run without and further amendment. If on the other hand you are using the test plan method then you will need to remove the test cases based on the old Selenium tests and add a new test case (called New Department Created Successfully to match the scenario name) and edit it in Visual Studio to make it automated.

And Finally

Do be aware that I've only really scratched the surface in terms of what SpecFlow can do and there's plenty more functionality for you to explore. Whilst it's not really the subject of this post it's worth pointing out that when deciding to adopt acceptance tests as part of your continuous delivery pipeline it's worth doing so in a considered way. If you don't it's all too easy to wake up one day to realise you have hundreds of tests which take may hours to run and which require a significant amount of time to maintain. To this end do have a listen to Dave Farley's QCon talk on Acceptance Testing for Continuous Delivery.

Cheers -- Graham

Continuous Delivery with TFS / VSTS – Automated Acceptance Tests with SpecFlow and Selenium Part 1

Posted by Graham Smith on June 15, 20167 Comments (click here to comment)

In the previous post in this series we covered using Release Management to deploy PowerShell DSC scripts to target nodes that both configured the nodes for web and database roles and then deployed our sample application. With this done we are now ready to do useful work with our deployment pipeline, and the big task for many teams is going to be running automated acceptance tests to check that previously developed functionality still works as expected as an application undergoes further changes.

I covered how to create a page object model framework for running Selenium web tests in my previous blog series on continuous delivery here. The good news is that nothing much has changed and the code still runs fine, so to learn about how to create a framework please refer to this post. However one thing I didn't cover in the previous series was how to use SpecFlow and Selenium together to write business readable web tests and that's something I'll address in this series. Specifically, in this post I'll cover getting acceptance tests working as part of the deployment pipeline and in the next post I'll show how to integrate SpecFlow.

What We're Hoping to Achieve

The acceptance tests are written using Selenium which is able to automate ‘driving' a web browser to navigate to pages, fill in forms, click on submit buttons and so on. Whilst these tests are created on and thus able to run on developer workstations the typical scenario is that the number of tests quickly mounts making it impractical to run them locally. In any case running them locally is of limited use since what we really want to know is if checked-in code changes from team members have broken any tests.

The solution is to run the tests in an environment that is part of the deployment pipeline. In this blog series I call that the DAT (development automated test) environment, which is the first stage of the pipeline after the build process. As I've explained previously in this blog series, the DAT environment should be configured in such a way as to minimise the possibility of tests failing due to factors other than code issues. I solve this for web applications by having the database, web site and test browser all running on the same node.

Make Sure the Tests Work Locally

Before attempting to get automated tests running in the deployment pipeline it's a good idea to confirm that the tests are running locally. The steps for doing this (in my case using Visual Studio 2015 Update 2 on a workstation with FireFox already installed) are as follows:

  1. If you don't already have a working Contoso University sample application available:
    1. Download the code that accompanies this post from my GitHub site here.
    2. Unblock and unzip the solution to a convenient location and then build it to restore NuGet packages.
    3. In ContosoUniversity.Database open ContosoUniversity.publish.xml and then click on Publish to create the ContosoUniversity database in LocalDB.
  2. Run ContosoUniversity.Web (and in so doing confirm that Contoso University is working) and then leaving the application running in the browser switch back to Visual Studio and from the Debug menu choose Detatch All. This leaves IIS Express running which FireFox needs to be able to navigate to any of the application's URLs.
  3. From the Test menu navigate to Playlist > Open Playlist File and open AutoWebTests.playlist which lives under ContosoUniversity.Web.AutoTests.
  4. In Test Explorer two tests (Can_Navigate_To_Departments and Can_Create_Department) should now appear and these can be run in the usual way. FireFox should open and run each test which will hopefully turn green.

Edit the Build to Create an Acceptance Tests Artifact

The first step to getting tests running as part of the deployment pipeline is to edit the build to create an artifact containing all the files needed to run the tests on a target node. This is achieved by editing the ContosoUniversity.Rel build definition and adding a Copy Publish Artifact task. This should be configured as follows:

  • Copy Root = $(build.stagingDirectory)
  • Contents =
    • ContosoUniversity.Web.AutoTests.*
    • ContosoUniversity.Web.SeFramework.*
    • Microsoft.VisualStudio.QualityTools.UnitTestFramework.*
    • WebDriver.*
  • Artifact Name = AcceptanceTests
  • Artifact Type = Server

After queuing a successful build the AcceptanceTests artifact should appear on the build's Artifacts tab:


Edit the Release to Deploy the AcceptanceTests Artifact

Next up is copying the AcceptanceTests artifact to a target node -- in my case a server called PRM-DAT-AIO. This is no different from the previous post where we copied database and website artifacts and is a case of adding a Windows Machine File Copy task to the DAT environment of the ContosoUniversity release and configuring it appropriately:


Deploy a Test Agent

The good news for those of us working in the VSTS and TFS 2015 worlds is that test controllers are a thing of the past because Agents for Microsoft Visual Studio 2015 handle communicating with VSTS or TFS 2015 directly. The agent needs to be deployed to the target node and this is handled by adding a Visual Studio Test Agent Deployment task to the DAT environment. The configuration of this task is very straightforward (see here) however you will probably want to create a dedicated domain service account for the agent service to run under. The process is slightly different between VSTS and TFS 2015 Update 2.1 in that in VSTS the machine details can be entered directly in the task whereas in TFS there is a requirement to create a Test Machine Group.

Running Tests -- Test Assembly Method

In order to actually run the acceptance tests we need to add a Run Functional Tests task to the DAT pipeline directly after the Visual Studio Test Agent Deployment task. Examining this task reveals two ways to select the tests to be run -- Test Assembly or Test Plan. Test Assembly is the most straightforward and needs very little configuration:

  • Test Machine Group (TFS) or Machines (VSTS) = Group name or $(TargetNode-DAT-AIO)
  • Test Drop Location = $(TargetNodeTempFolder)\AcceptanceTests
  • Test Selection = Test Assembly
  • Test Assembly = **\*test*.dll
  • Test Run Title = Acceptance Tests

As you will see though there are many more options that can be configured -- see the help page here for details.

Before you create a build to test these setting out you will need to make sure that the node where the tests are to be run from is specified in Driver.cs which lives in ContosoUniversity.Web.SeFramework. You will also need to ensure that FireFox is installed on this node. I've been struggling to reliably automate the installation of FireFox which turned out to be just as well because I was trying to automate the installation of the latest version from the Mozilla site. This turns out to be a bad thing because the latest version at time of writing (47.0) doesn't work with the latest (at time of writing) version of Selenium (2.53.0). Automation installation efforts for FireFox therefore need to centre around installing a Selenium-compatible version which makes things easier since the installer can be pre-downloaded to a known location. I ran out of time and installed FireFox 46.1 (compatible with Selenium 2.53.0) manually but this is something I'll revisit. Disabling automatic updates in FireFox is also essential to ensure you don't get out of sync with Selenum.

When you finally get your tests running you can see the results form the web portal by navigating to Test > Runs. You should hopefully see something similar to this:


Running Tests -- Test Plan Method

The first question you might ask about the Test Plan method is why bother if the Test Assembly method works? Of course, if the Test Assembly method gives you what you need then feel free to stick with that. However you might need to use the Test Plan method if a test plan already exists and you want to continue using it. Another reason is the possibility of more flexibility in choosing which tests to run. For example, you might organise your tests in to logical areas using static suites and then use query-based suites to choose subsets of tests, perhaps with the use of tags.

To use the Test Plan method, in the web portal navigate to Test > Test Plan and then:

  1. Use the green cross to create a new test plan called Acceptance Tests.
  2. Use the down arrow next to Acceptance Tests to create a New static suite called Department:
  3. Within the Department suite use the green cross to create two new test cases called Can_Navigate_To_Departments and Can_Create_Department (no other configuration necessary):
  4. Making a note of the test case IDs, switch to Visual Studio and in Team Explorer > Work Items search for each test case in turn to open it for editing.
  5. For each test case, click on Associated Automation (screenshot below is VSTS and looks slightly different from TFS) and then click on the ellipsis to bring up the Choose Test dialogue where you can choose the correct test for the test case:
  6. With everything saved switch back to the web portal Release hub and edit the Run Functional Tests task as follows:
    1. Test Selection = Test Plan
    2. Test Plan = Acceptance Tests
    3. Test Suite =Acceptance Tests\Department

With the configuration complete trigger a new release and if everything has worked you should be able to navigate to Test > Runs and see output similar to the Test Assembly method.

That's it for now. In the next post in this series I'll look at adding SpecFlow in to the mix to make the acceptance tests business readable.

Cheers -- Graham

Continuous Delivery with TFS: Creating Automated Web Tests with Selenium

Posted by Graham Smith on January 26, 201510 Comments (click here to comment)

In this blog post (which is part of my series on on implementing continuous delivery with TFS) we look at creating automated web tests with Selenium. Although the need for manual testing will probably never go away any organisation that wants to deploy software on a frequent basis is going to have to automate some if not most of the application testing effort if quality has any hope of being maintained. Clearly, if you want to release your application once a week but it takes two weeks to properly manually test your application you have a big problem, and automation is probably the only sensible answer.

Let's be clear that test automation is a mammoth subject and not only involves automating the driving of the application but also data setup and teardown and service virtualisation ie stubbing interfaces to external systems. Then there is managing your tests between sprints or projects -- when you have 500 or so this is a non-trivial task. I can't possibly cover the subject properly in a couple of posts and instead my aim is to show you how to begin to create a framework for writing automated tests and crucially how to make them run in the TFS ecosystem. I'll be using Selenium since it's used by John Sonmez in his superb Pluralsight course Creating an Automated Testing Framework With Selenium. I thoroughly recommend this course to anyone who is embarking on an automated testing project, and some of the code I use comes directly from John's course (with his kind permission). If you are using a different automation technology such as CodedUI similar principles will apply. As with many other things in software development there is a quick and dirty way to write automated tests (by recording them which makes them very brittle -- think recorded Excel macros) and there is the professional approach where you put the time and effort in to make a much more robust solution. I'll be using the latter technique although I will only scratch the surface of what's possible.

Create the Framework

Our simple automation solution will be in two parts, the first of which is a fluent API framework that models the web application's web pages. To get the ball rolling create a new class library project in the ContosoUniversity solution called ContosoUniversity.Web.SeFramework and complete these steps:

  1. Via NuGet install Selenium WebDriver and Selenium WebDriver Support Classes.
  2. Remove Class1.cs and add a class called Driver.cs. Ensure Driver.cs contains the following code:
  3. Open the properties of ContosoUniversity.Web and from the Web tab find the IIS Express port number. Replace the four zeros in the above code with the port number.
  4. Add a new class called DepartmentsPage.cs and ensure it contains the following code:
  5. Add a new class called NewDepartmentPage.cs and ensure it contains the following code:
Create the Tests

To keep everything in layers we'll create our tests in a new Unit Test project. We need to use the MSTest framework because (jumping ahead here) Microsoft Test Manager only works with MSTest at the moment. So, add a new Unit Test project called ContosoUniversity.Web.AutoTests and complete these steps:

  1. Create a project reference to ContosoUniversity.Web.SeFramework.
  2. Add a new class called ContosoTest.cs and ensure it contains the following code:
  3. Add a new class called DepartmentTests.cs and ensure it contains the following code:
  4. Build the solution so the tests are found in Test Explorer. Select the Can_Cerate_Department and Can_Navigate_To_Departments tests then add them to a new playlist by right-clicking and selecting Add to Playlist. Save the playlist in the root of ContosoUniversity.Web.AutoTests.
Running the Tests

In order to run the tests using the code above you'll need to install FireFox. You can't just run the tests though because you need to take steps to ensure there is a web server to handle the requests. The technique is to run the solution so the application starts and IIS Express is running (the icon should appear in the System Tray). Then from the Debug menu in Visual Studio select Detach All. This will close the browser that is running the application but will leave IIS Express running. You can now run the tests (make sure you are running in a Debug configuration) and watch Selenium do all the hard work!

What Have we Done Here?

I want to reiterate that this post isn't meant to be a tutorial on how to write automated tests and you should regard the code as an example of how to get up-and-running quickly rather than being of production-ready quality. Although this isn't a tutorial some explanation of what's going on is in order.

In the ContosoUniversity.Web.SeFramework project the Driver class is essentially the link to FireFox and also has some methods (not used here) to help with tricky web pages that take time to do their thing. The DepartmentsPage has a method to navigate to the page that lists all of the departments and a helper property and method for use in Assert statements. The interesting class is NewDepartmentPage since this illustrates the technique for writing code that allows the use of a fluent syntax for writing tests.

In the ContosoUniversity.Web.AutoTests project the ContosoTest class is a base class that contains common functionality (starting and stopping the browser) for actual test classes to inherit. DepartmentTests inherits ContosoTest and uses the fluent API to cause the browser to navigate around the application doing things that we want to test. It's all very basic but hopefully enough to illustrate the huge potential of this type of testing.

With our automated web tests created we now need to configure the infrastructure and tooling to automate their execution according to a schedule. Watch out for the next instalment when we'll be doing just that.

Cheers -- Graham