How to run Silverlight automated tests on TFS build server?

After many hours trying to find a complete solution to execute some Silverlight automated tests on a TFS 2010 build server without any success, I was forced to try to build my own solution. Thanks to my colleague Mathieu Szablowski, a Microsoft ALM expert here at Pyxis, we were able to craft a solution by combining StatLight (an open source tool used to run Silverlight tests) with a custom activity in Team Build 2010.

Configuring Silverlight test execution

With StatLight, it is possible to run automated tests from a Silverlight Test Project (created from the Silverlight Toolkit project template) and write the results to an XML file. The general principle here is to configure your build to run the Silverlight tests and have the build react if something is wrong.

The first step is to download and install both the Silverlight Toolkit and StatLight on your build server.

Once this is done, open Team Explorer on the build server and follow the steps below:

1. In the BuildProcessTemplate folder of your TFS project, make a copy of the DefaultTemplate.xaml template file.

2. Rename the copied template to a name like SilverlightDefaultTemplate.xaml and open it to edit the workflow.

3. Go to Process -> Sequence -> Run On Agent -> Try Compile, test, and … -> Sequence and add a sequence named Silverlight Test Run just after the “If a compilation exception” box.

4. From the toolbox, add an InvokeProcess TFS activity in the sequence. Open the properties and enter the following name: Invoke StatLight.

5. Still in the properties, enter the path of your StatLight installation on the build server in the FileName property. Example: “C:Program Files(x86)StatLightStatLight.exe.

6. In the “Silverlight Test Run” sequence box, create a TestResultFile variable that will be used to indicate where to write and read the XML report file generated by StatLight. You can assign the following value: BinariesDirectory + “TestResult.xml.

7. You must use the “-x” parameter to indicate to StatLight what Silverlight test file (.xap) to run and the “-r” parameter to specify where to write the test result report. In the arguments property of your “Invoke Statlight” activity, enter the following line: “-x=” “” + BinariesDirectory + “SilverlightTestProject.xap”"-r =” “” + TestResultFile.

8. To ensure that your build service will be able to interact with the StatLight program, you must configure your build service to run in interactive mode. Open the TFS administration console, select Build Configuration, click Properties on your BuildService and select Run build service as: Interactive Process.

Your Run Silverlight Test workflow box should almost look like this:

Note: To see your custom activities in the workflow (xaml template file) without any errors, you must open your XAML template file in a project that reference your custom activie’s dll.

Now you can run a build and verify that Silverlight tests were run by checking that the XML report has been posted at the location specified by the “-r” parameter in the argument properties of the “Invoke StatLight” activity.

Creating custom activities

The next step is to create and configure custom activities in Team Build. The steps to accomplish this are well detailed in the following blog:  http://www.ewaldhofman.nl/post/2010/04/29/Customize-Team-Build-2010-e28093-Part-4-Create-your-own-activity.aspx

For our specific need, our custom activities will call a class that will be responsible for parsing the XML file generated by StatLight and analyzing the test results to provide a list of all the tests that failed.

Then, you can use this class within the custom activities to retrieve the list of tests that failed. From this list, in the custom activities, just run the following code to fail the build:

Or you can run the code below to add a warning according to the desired behavior:

When you have completed your custom activity, you have to follow the configuration steps (step 11 to 25) mentioned in the Ewald Hofman blog’s mentionned above to complete the solution. You need to insert your custom activities immediately after the “Invoke StatLight” activity, instead of the “Get the Build” activity mentioned in the Hofman blog.

Here is an exemple of the result in Activity Log (detailed build information) with one failed test:

The same example with a failed test in the Summary view:

Until Microsoft provides a built-in alternative, this solution will enable our team nSemble to develop fully tested Silverlight applications using Agile engineering practices and continuous integration required to implement working software iteration after iteration.

email

à propos de pyxis

L’Agilité guide nos pratiques depuis plus de 10 ans. Pour nous, les approches Agiles permettent de livrer rapidement et fréquemment des logiciels de qualité. Pionniers de l'Agilité au Québec, nous tirons profit chaque jour des avantages découlant des méthodes Agiles.
voir mon profil »

  • Pingback: Tweets that mention How to run Silverlight automated tests on TFS build server? | Pyxis blog -- Topsy.com

  • http://staxmanade.blogspot.com Jason Jarrett

    FYI: In the most recent commits of StatLight (at around the 2011-01-11) date https://github.com/staxmanade/StatLight/commits/master You can see that StatLight now support the MSTest GenericTest report xml file. This would allow you to possibly exclude the parsing step…

    You can read a bit on the following thread. http://statlight.codeplex.com/Thread/View.aspx?ThreadId=241274

    Hope this helps.

  • http://colinbowern.com Colin Bowern

    Going the generic test format route (documented at http://www.nielshebling.de/?p=167) is equally as fragile. The only upside is getting visibility of the test results in the TFS database. It feels like we’re hacking together a solution with duct tape.

  • xyz25

    Excellent post. How can I give multiple xap files as arguments for the statlight through the custom activity created as mentioned in Ewald Hofman’s blog to run have statlight run tests present in multiple silverlight test projects of my solution? I want to generate this expression programmatically. Unfortunately the expression editor does not allow to insert a script to generate the argument list. Creating a loop in the build workflow to iterate over the list of xaps is also not an option because that will result in multiple test result files. I need only 1. How did you guys manage this problem? Any pointers regarding this would be helpful.

  • Sam Piper

    This is a very useful article, can you post the code for the custom activity that plugs the results file into the build process? It would be very helpful.

  • acid

    Is this possible to do in tfs 2008? We recently upgraded the service on the build server to be able to build silverlight 4 and .net 4.0 projects.

  • Pingback: Custom Build Activities