How to run Silverlight automated tests on TFS build server?
After many hours trying to find a complete solution to execute some Silverlight automated tests on a TFS 2010 build server without any success, I was forced to try to build my own solution. Thanks to my colleague Mathieu Szablowski, a Microsoft ALM expert here at Pyxis, we were able to craft a solution by combining StatLight (an open source tool used to run Silverlight tests) with a custom activity in Team Build 2010.
Configuring Silverlight test execution
With StatLight, it is possible to run automated tests from a Silverlight Test Project (created from the Silverlight Toolkit project template) and write the results to an XML file. The general principle here is to configure your build to run the Silverlight tests and have the build react if something is wrong.
The first step is to download and install both the Silverlight Toolkit and StatLight on your build server.
Once this is done, open Team Explorer on the build server and follow the steps below:
1. In the BuildProcessTemplate folder of your TFS project, make a copy of the DefaultTemplate.xaml template file.
2. Rename the copied template to a name like SilverlightDefaultTemplate.xaml and open it to edit the workflow.
3. Go to Process -> Sequence -> Run On Agent -> Try Compile, test, and … -> Sequence and add a sequence named Silverlight Test Run just after the “If a compilation exception” box.
4. From the toolbox, add an InvokeProcess TFS activity in the sequence. Open the properties and enter the following name: Invoke StatLight.
5. Still in the properties, enter the path of your StatLight installation on the build server in the FileName property. Example: “C:Program Files(x86)StatLightStatLight.exe.
6. In the “Silverlight Test Run” sequence box, create a TestResultFile variable that will be used to indicate where to write and read the XML report file generated by StatLight. You can assign the following value: BinariesDirectory + “TestResult.xml.
7. You must use the “-x” parameter to indicate to StatLight what Silverlight test file (.xap) to run and the “-r” parameter to specify where to write the test result report. In the arguments property of your “Invoke Statlight” activity, enter the following line: “-x=” “” + BinariesDirectory + “SilverlightTestProject.xap”"-r =” “” + TestResultFile.
8. To ensure that your build service will be able to interact with the StatLight program, you must configure your build service to run in interactive mode. Open the TFS administration console, select Build Configuration, click Properties on your BuildService and select Run build service as: Interactive Process.
Your Run Silverlight Test workflow box should almost look like this:
Note: To see your custom activities in the workflow (xaml template file) without any errors, you must open your XAML template file in a project that reference your custom activie’s dll.
Now you can run a build and verify that Silverlight tests were run by checking that the XML report has been posted at the location specified by the “-r” parameter in the argument properties of the “Invoke StatLight” activity.
Creating custom activities
The next step is to create and configure custom activities in Team Build. The steps to accomplish this are well detailed in the following blog: http://www.ewaldhofman.nl/post/2010/04/29/Customize-Team-Build-2010-e28093-Part-4-Create-your-own-activity.aspx
For our specific need, our custom activities will call a class that will be responsible for parsing the XML file generated by StatLight and analyzing the test results to provide a list of all the tests that failed.
Then, you can use this class within the custom activities to retrieve the list of tests that failed. From this list, in the custom activities, just run the following code to fail the build:
Or you can run the code below to add a warning according to the desired behavior:
When you have completed your custom activity, you have to follow the configuration steps (step 11 to 25) mentioned in the Ewald Hofman blog’s mentionned above to complete the solution. You need to insert your custom activities immediately after the “Invoke StatLight” activity, instead of the “Get the Build” activity mentioned in the Hofman blog.
Here is an exemple of the result in Activity Log (detailed build information) with one failed test:
The same example with a failed test in the Summary view:
Until Microsoft provides a built-in alternative, this solution will enable our team nSemble to develop fully tested Silverlight applications using Agile engineering practices and continuous integration required to implement working software iteration after iteration.
