Here at cmgroup we have been working on some HTML prototyping. Whilst doing this we have been working on improving our automated testing. This blog entry outlines our strategy and how we have achieved it.
Our requirements for the automated testing were to cover:
- Multi browser / platform testing of HTML output
- Write the test scripts in a language suitable for product management
- Leverage the testing framework provided by VS2010
The following technology was chosen. MSTest for obvious reasons; the requirement was there to use it. We chose Specflow as it is a very nice BDD framework using Gherkin scripts. I have posted examples of its use previously on this blog and it meets the 2nd requirement above. Thirdly we chose to use selenium as it provides some very good dotnet bindings and provides us with the ability to test cross browser and platform.
Our test setup is shown in the following diagram:
(Lower Right Node should be Vista x64)
Firstly we have selenium standalone running on two nodes, one Vista and one Windows 7, so we can test on different browsers and versions. We are using the current beta version of Selenium2, which currently has some stability problems. There is a push to fix a whole load of bugs this week in selenium and we are hoping beta2 is released shortly.
We are also running against real browsers using the Chrome Driver, Firefox Driver and Internet Explorer Driver, so these two nodes have to have a desktop available, i.e. A user logged in at all times, which is a only a minor pain.
The unit tests are then run within MSTest on the build controller, executing the Specflow tests, which in turn drive the .NET version of Selenium RC, which executes operations on the remote instances of selenium server.
Writing the Tests
A typical SpecFlow scenario would be written something like the following:
Scenario Outline: Display Next Prev Buttons Given I am using <browser> on <seleniumhost> And A course exists When I launch the course Then The course is launched in my default browser And The next and previous buttons are visible in the browser Examples: | browser | seleniumhost | description | | ie | 192.168.10.126 | IE8 on Win 7 64bit | | ie | 192.168.10.127 | IE7 on Vista 64bit | | firefox | 192.168.10.126 | FF3.6 on Win 7 64bit | | chrome | 192.168.10.126 | Chrome 8 on Win 7 64bit |
This sets up the tests to execute against the listed hosts, using the specifed browsers. This scenario would be run 4 times against with a test on IE7 on Vista.
We are presuming that the next beta of selenium2 and the final version are going to solve our stability problems. We shall then have a great test environment for BDD, able to test our apps using the MSTest framework and BDD and believe this will give us a greater confidence in our apps meeting the User Acceptance Criteria our Product Owners are providing.
For more information on selenium and specflow, here are some links:
If you’ve read my post on the value of automated UI testing, you’ll know that I value it over Unit Testing. That’s not to say Unit Testing isn’t important, just that there is greater business value in Automated UI testing, if you can get it to work for you.
In this post I’m going to describe how SpecFlow could be used to achieve this with WPF applications. I’m not going to pretend that my small example here is going to solve problems of applying this to large projects, but hopefully it will provide a start point, a trigger to further investigation and establishing some best practices, appropriate for the product being tested.
Firstly then, a quick description of my Demo App. If you’ve read some of my other posts you will be familiar with the groundbreaking WPF media player. It allows user to select a folder, display the media in that folder and play that media:
Unit Tests in VS
Unit Testing is now an integrated part of Visual Studio and has been for a number of years. Everyone reading this is no doubt very familiar with this, so I won’t venture to explain it. Suffice it to say, that ideally we want to do UI testing in a testign framework that is well supported in Visual Studio. SpecFlow supports both MSTest as well as nUnit.
Introduction to Gherkin
Specifying User Acceptance Criteria for stories is always in danger of being ambiguous. Capturing them in a standardised notation often helps make thes acceptance criteria more explicit. Whilst at Nokia, our sprint planning sessions involved the PO describing the story and then the team defining UACs as Gherkin scenarios.
Gherkin is a simple notation based on scenarios that can be written using Given – When – Then clauses. Given being the system state, When being a user action and Then being the outcome. Gherkin is ‘Business Readable’, so can be fully understood by the PO and the business.
Introduction to UI Automation
Microsoft have helped matters in testing UI written in WPF through their UI Automation API. This API enables us to drive WPF applications and check content of controls. Again, I am not going into too much detail as there is more than enough elsewhere and to be honest I have only recently started looking at the UI Automation API. The reason I bring it up here is that we need a way of pulling Unit Tests with MS Test, UI Automation to drive the UI and Gherkin to allow a simple notation for defining Scenarios and ultimately User Acceptance Criteria. Enter SpecFlow….
Introduction to SpecFlow
Lifted straight from their website: “SpecFlow aims at bridging the communication gap between domain experts and developers by binding business readable behavior specifications to the underlying implementation.”
Specflow is designed for .NET, supports MS Test and integrates with Visual Studio. On initial inspection, this looked like what I had been looking for. I shall now go through an example of writing tests using SpecFlow for the application described above. There is obviously plenty more information on the various product web sites so I would advise going there rather than copying my example. This example is here to fire the imagination, if that is what you are after.
Add a feature
We are starting from an empty Test Project in Visual Studio and adding a feature is as simple as selecting the “SpecFlow Feature File” on “Add New Item”. The dialog also includes Event Definition files and Step Definition files, which we will use later.
This adds a new .feature file which contains a basic feature and scenario. I edited this to add a couple of scenarios which should be self explanatory. After all, that is the purpose of Gherkin:
Here you can see the Scenarios as Give, When, Then steps. Once this file has been added the tests will build and run as inconclusive. Each scenario forms a test case. SpecFlow does some magic in generating the code for the feature file. What must be done now is the writing of any setup / tear down code and methods for each step.
Add Event Defs
An event definition file can be added which has methods for executing code at various stages during the running of scenarios and tests. In our case we just want to kick off the application on each feature test and kill it after:
Add Steps for feature
The bulk of the coding must be done in the Steps Code. Taking the example of the first scenario we want to write 3 steps for the given / when / then clauses in the gherkin:
As you can see, SpecFlow uses attributes to match steps to methods, really very simple. Most of this code is the neccessary boilerplate for UI Automation, which leads us on to how we might deal with this in a real application
The first thing I noticed, which was really to do with the UI Automation, was the amount of boilerplate that was needed. In a real application, this would get significantly more complex than this exmaple, very quickly. Ideally each View would have a UI Automation wrapper written for it that would encapsulate the UI Automation code, allowing the test developer to call single methods to perform opearions on the UI and to check values in controls. I am guessing that at this point it would be worth investigating F# to do this, as it seems that a more functional language may help in this. Any comments on that will be gratefully recieved.