In the previous articles when we built a package, we generally tested it visually. We run it and saw if it responds to our interaction with the UI as we expected. We always unchecked the testing options in the VSPackage wizard.
It is not because I hate testing or just simply do not like it! I think testing VSPackages is a topic that requires at least one dedicated article. To be honest, when I wrote the code for the VsxTools sample library in Part #10, I really felt that testing was missing. I wrote a few hundred lines of code; how could I be sure it worked as I expected?
There are many great books and essays about testing. In this article I do not intend to write a new one. I am a fan of test-driven development but do not take myself into account as a testing guru. Here I try to focus especially for a few basic questions of VSPackage testing.
I believe in the power of automated tests and want to show you that automated testing is available for VSPackages. I do not want to treat deeply the methodology of VSPackage testing, I’d rather show you the toolbox you can use to create the set of tests that make you confident your package works as it is expected.
So, I suppose you have the basic knowledge about unit testing and how to use Visual Studio to write and run unit tests. If you feel you miss the basics, please visit MSDN and I am sure you will find the necessary information about this topic.
Visual Studio 2005 and 2008 Team System has specialized editions for testers. Fortunately, when using Visual Studio 2008 Professional you will find the basic testing functionality built in so having a license for Professional edition is perfect for our testing purposes.
Creating a package with tests
As we did so many times before, we are going to create a new package, but this time we ask the VSPackage wizard to generate test projects for us. So, let’s create a new package called SimpleTesting. When the wizard starts, select C# as the package language. Please, fill the basic VSPackage information as in the following picture:
On the next page select the Menu command and Tool Window options:
When the wizard asks, set the menu command options as in the following picture:
Set the tool window options:
And as a difference from our previously created packages, not set both test project options in the last wizard steps.
This time VSPackage wizard adds three new projects to out solution. The first — called SimpleTesting — is the package itself. The two test projects are SimpleTesting_IntegrationTestProject and SimpleTesting_UnitTestProject. Each test project contains a few files with pre-generated test cases:
Selecting the Test|Windows|Test View menu function you can see the test cases:
Fixing an “issue”
At this point we have everything ready to run our tests. You can start all tests in the current solution with the Test|Run|All Tests in Solution menu function or just simply pressing the Ctrl+R and A keys. While tests run a new instance of Visual Studio (of course, the Experimental Hive) starts and an “invisible agent” interacts with it. This is what out test cases do! However when the test case evaluation is over you will recognize that one test case called ShowToolWindowNegativeTest fails. Is something wrong with our package?
This test is about to emulate that a tool window cannot be created and in this case it cannot be shown up. This is a negative test that must raise an exception. The test case signs it expects this with the attribute attached to the test method:
[TestMethod()]
[ExpectedException(typeof(TargetInvocationException),
"Did not throw expected exption when windowframe object was null")]
public void ShowToolwindowNegativeTest() {...}
Unfortunately, the test case behaves as if it had not had the ExpectedException attribute.
There is an “issue” — I mean bug — in the VSPackage wizard: it adds a wrong reference assembly to the SimpleTesting_UnitTestProject. Take a look at the properties of the Microsoft.VisualStudio.QualityTools.UnitTestFramework referenced assembly and you will find it 8.0.0.0. Actually this is the assembly belonging to the VS 2005 testing toolset. From this point the solution is easy: remove this referenced assembly and add the correct version (9.0.0.0) with the Add Reference function. The SimpleTesting_IntegrationTestProject does not have this issue.
Now, when you run tests again, all test cases should conclude with success.
Adding a missing TestMethod attribute
In the SimpleTesting_UnitTestsProject there is an “orphan” WindowPropertyTest unit test case where the TestMethod attribute is missing. It can be found in the MyToolWindow.cs file. The missing attribute does not cause any problem; it simply does not run the test case. To have the test case run, if it is there, put the TestMethod attribute manually:
[TestMethod()]
public void WindowPropertyTest() {...}
When running all tests WindowPropertyTest also must run successfully.
Issue with CodePlex and Team Explorer integration
I had another issue that caused me very embarrassing hours. I have all the source code on CodePlex (Team Foundation Server is behind). When I am connected to TFS (and I am nearly always connected), the Run All Tests in Solution function seems to freeze my Visual Studio that is not responding for a few minutes. After 5-9 minutes the tests start to run. If I go off-line everything works. When I go on-line again my tests work OK. When I restart Visual Studio with on-line connection to TFS, long waiting starts again.
If you do not have this kind of issue you can skip this part since I do not want to bore you...
I had several projects on CodePlex with intensive unit tests and I did not have this problem. I experienced it only with test projects related to VSPackages. I recognized that while I am waiting for my tests start, there is an intensive traffic on my wireless network adapter. After a few analyses with a network monitor I discovered that during this wait-time my network adapter downloaded about 38 MBytes of information from the CodePlex side (by IP address). I am not a networking guy so at this point I finished any further examination. Right now my only workaround is to set my project off-line during unit tests...
I will contact the VSX team about this problem, I hope they can help. Till this time I appreciate your comments and suggestions about this issue.
Hosting tests in the VS IDE
When running tests, test cases always assume a certain minimal context in order they really can check what they are created for. For example, if our test case transforms a file, we need a context where the source file is provided and we have a placeholder for the output. If we want to check the business logic of our application using a data access layer, we must create a context where we can communicate with the data access layer (or with some object that seems to be the data access layer component).
When Visual Studio runs a unit test, it does it in a separate process. Test cases “feel” if they are enclosed into an executable file and perceive the same context that their runner process. When we create simple unit tests those generally run in the VSTestHost.exe process. There are many reasons for that separation; one of the most important is the separation of tests from the VS IDE process. If the test process goes down due to some serious issue (e.g. Stack Overflow Exception since an infinite recursion), it does not kill the VS IDE, even VS IDE can recognize and report the issue.
What should be this context if we test VSPackages? It depends on, what we test. If we test the internal algorithms, helper classes, simple services within our unit test cases we could run it as in case of any “traditional” test cases, even the VSTestHost.exe is a good context, but also NUnit, CSUnit and so on.
However when we want to test our package from user interaction or VS integration perspective – for example to check if our package creates the expected menu item, etc. — we need another context that emulates or represents the VS IDE itself. There is a well-known testing pattern called mocking that actually “emulates” the context our test cases need. There are thousands of pages written about mocking. All most known test tools including NUnit and the Visual Studio itself supports mocking in a very good level. There are so-called mocking frameworks that are independent from test tools (of course depend on platform).
What if we could run our VSPackage tests within the VS IDE (devenv.exe) itself? That could be a perfect environment for packages that are hosted in the production environment within devenv.exe. Actually, test cases generated by the VSPackage wizard do run in a devenv.exe hosting process! When you run the tests actually a new Visual Studio 2008 Experimental Hive instance is started to host the assembly representing the test cases. This is what I wrote down a few paragraphs ago as an “invisible agent interacts with our package”.
Selecting VS IDE as test host
The debugging support in Visual Studio — by the way, this is also extensible through VSX — makes it possible to host tests in any process through a so-called Test Adapter. The topic of this article (and even for the series) is quite far away to treat that, but if you are interested about how it can be done, look for the VSIDEHostAdapter sample in the VS SDK 2008.
We can tell Visual Studio to use devenv.exe (with the experimental hive) to run our tests. We have several ways to do it.
After VSPackage wizard generated our test projects, it also generate two solution item files with the .testrunconfig files. When opening any of these files, in its editor you will find a Hosts tab where the Host Type is set to VS IDE:
VS IDE has Test Adapter registered with Visual Studio 2008 and allows specifying other test context parameters: the registry hive that VS should use as a test host. If you would set Host Type to Default, the test case would run in its default host type.
You can set the host type of each unit test decorating the unit test method with the HostType attribute as in the following example:
[TestMethod()]
[HostType("VS IDE")]
public void LaunchCommand() { ... }
This code extract comes from the MenuItemTest.cs file in the IntegrationTestProject. Note that the attribute value is set to “VS IDE”. When you set the host type in the .testrunconfig files to Default and the HostType attribute of the test case to “VS IDE”, the case will run in VS IDE, since the HostType attribute overrides the .testrunconfig settings.
We can make a small experiment about how these settings work. Set the Host Type in both .testrunconfig file to default and run all unit tests. There are 2 failed tests from the pre-generated 15: CPPWinformsApplication and VBWinformsApplication. What happened?
The other 13 tests run successfully since either they did not require VS IDE or they had the [HostType(“VS IDE”)] attribute. Try and put these attributes to the failed test case methods. Now, when running the tests again, all of them conclude with success. Please do not forget to set back the original state in the .testrunconfig files: Host Type should be set to VS IDE and Visual Studio Registry Hive set to 9.0 (RANU).
If you have your own tests that require VS IDE as test host, I suggest you the following best practice:
Group your tests requiring VS IDE separately from those that do not (by assemblies, namespaces, folder, files or whatever makes this grouping clear for you).
Set the Host Type for VS IDE for those assemblies that have test cases requiring it.
Mark the corresponding test methods with the [HostType(“VS IDE”)] attribute so that it can be seen explicitly.
Diving into the test projects
To use a polite expression to name the state of VS SDK 2008 package testing support documentation I would use the word “poor”. I think this is an area where the VSX Team should work a lot. It is worth to dive into the test projects created by the VSPackage wizard, since you can obtain a lot of information out form that code and use it instead of the missing ones. VSPackage wizard generated two test projects. We go into each of them and look what kind of test cases they implement and what are the takeaways from those implementations.
The UnitTestProject
The SimpleTesting_UnitTestProject uses mocking to run tests in an emulated context. Any VS Shell services and all of their method calls are emulated through mocking, so our package “feels” itself as if it were hosted in the VS IDE!
This project contains the following pre-generated test cases:
| Test case (method) |
Description |
| CreateInstance |
Checks if the package creation (calling the default constructor) does not raise an exception.
|
| IsIVsPackage |
Checks if the package implements the IVsPackage interface.
|
| SetSite |
Checks if the package can be correctly sited and unsited.
|
| InitializeMenuCommand |
Checks if the simple menu command we created in our package can be added to the VS Shell menu.
|
| MenuItemCallback |
Checks if the callback method of our simple menu command on the Tools menu (that shows up a simple message box in the production environment) works correctly.
|
| MyToolWindowConstructorTest |
Checks that our tool window can be constructed and the user control representing its UI is also created.
|
| WindowPropertyTest |
Checks, if the tool window’s UI can be accessed through the overridden Window property of the tool window class.
|
| ValidateToolWindowShown |
Checks, if the ShowToolWindow method is able to display the tool window.
|
| ShowToolwindowNegativeTest |
Checks, that a tool window that cannot be created by the shell cannot be displayed.
|
This unit tests are very simple and check only simple functions. If you want to understand what is going behind the scenes when a unit test runs, you had better start with understand how VSPackage mocking works. Treating all details exceeds the borders of this article, but I give you a few hints that help you to start with.
The tests projects have a reference to the Microsoft.VSSDK.UnitTestLibrary assembly that contains a few helper classes for unit tests and mocking. With VS 2008 SDK V1.0 you get not only the binary assembly but also the source code of this assembly under the VisualStudioIntegration\Common\Source\CSharp\UnitTest folder of the VS SDK. As I analyzed the source I had the feeling that is just a startup state of this assembly, since it contains only a few types. I do not know what will be the future of this assembly. I can imagine it will be further developed or changed totally to something else. Right now the source is a good base to understand how mocking is implemented.
The SimpleTesting_UnitTestProject contains a few files (three) having the Mock suffix in their names. These are the definitions of mocking classes that emulate a very simple context for our unit tests.
To help you understand how these tests work, let me explain you one of them: the ValidateToolWindowShown test case. This method checks if the ShowToolWindow private method in our package class works correctly when called by the shell. Just to recall what ShowToolWindow does, here is the complete source of this method:
private void ShowToolWindow(object sender, EventArgs e)
{
ToolWindowPane window = this.FindToolWindow(typeof(MyToolWindow), 0, true);
if ((null == window) || (null == window.Frame))
{
throw new NotSupportedException(Resources.CanNotCreateWindow);
}
IVsWindowFrame windowFrame = (IVsWindowFrame)window.Frame;
Microsoft.VisualStudio.ErrorHandler.ThrowOnFailure(windowFrame.Show());
}
If you do not remember how it works, please read Part #4. One important detail is that the FindToolWindow method uses the SVsUIShell service’s CreateToolWindow method inside to physically create our tool window. So, let’s see how this method is checked by the test case:
1 [TestMethod()]
2 public void ValidateToolWindowShown()
3 {
4 IVsPackage package = new SimpleTestingPackage() as IVsPackage;
5 OleServiceProvider serviceProvider =
6 OleServiceProvider.CreateOleServiceProviderWithBasicServices();
7 BaseMock uiShellService =
8 UIShellServiceMock.GetUiShellInstanceCreateToolWin();
9 serviceProvider.AddService(typeof(SVsUIShell), uiShellService, false);
10 Assert.AreEqual(0, package.SetSite(serviceProvider),
11 "SetSite did not return S_OK");
12 MethodInfo method = typeof(SimpleTestingPackage).GetMethod("ShowToolWindow",
13 BindingFlags.NonPublic | BindingFlags.Instance);
14 object result = method.Invoke(package, new object[] { null, null });
15 }
This test case generates two mock instances (I highlighted them in the code). The OleServiceProvider type here is coming from the Microsoft.VsSDK.UnitTestLibrary namespace and is a mock for the OleServiceProvider type in the Microsoft.VisualStudio.Shell namespace. In line 5 and 6 the code creates the mock instance. Line 7 and 8 create a mock object for the SVsUIShell service through the UIShellServiceMock type that is declared also in the unit test project. Line 9 adds the mocked SVsUIShell instance to the list of available services. The SVsUIShell mock enters to the stage when the ShowToolWindow method (this is the method we want to test) calls FindToolWindow:
private void ShowToolWindow(object sender, EventArgs e)
{
ToolWindowPane window = this.FindToolWindow(typeof(MyToolWindow), 0, true);
...
}
The real test begins with line 10 where our package gets sited into the mocked context. Lines 12-14 call our package’s private ShowToolWindow method. If this call does not raise an exception, our test is taken into account as successful.
To get you closer to the mock implementation in unit tests, let’s look inside the UIShellServiceMock class in the project. There are two classes with the same name in different namespaces; we look for the one in the MyToolWindowTest folder. The list below is not the full list of the source code, but it is enough to explain the important details:
using System;
using Microsoft.VisualStudio;
using Microsoft.VisualStudio.Shell.Interop;
using Microsoft.VsSDK.UnitTestLibrary;
namespace UnitTestProject.MyToolWindowTest
{
static class UIShellServiceMock
{
private static GenericMockFactory uiShellFactory;
internal static BaseMock GetUiShellInstance()
{
if (uiShellFactory == null)
{
uiShellFactory = new GenericMockFactory("UiShell",
new Type[] { typeof(IVsUIShell), typeof(IVsUIShellOpenDocument) });
}
BaseMock uiShell = uiShellFactory.GetInstance();
return uiShell;
}
internal static BaseMock GetUiShellInstanceCreateToolWin()
{
BaseMock uiShell = GetUiShellInstance();
string name = string.Format("{0}.{1}", typeof(IVsUIShell).FullName,
"CreateToolWindow");
uiShell.AddMethodCallback(name,
new EventHandler<CallbackArgs>(CreateToolWindowCallBack));
return uiShell;
}
private static void CreateToolWindowCallBack(object caller,
CallbackArgs arguments)
{
arguments.ReturnValue = VSConstants.S_OK;
IVsWindowFrame frame = WindowFrameMock.GetBaseFrame();
arguments.SetParameter(9, frame);
}
...
}
}
The static uiShellFactory’s type, GenericMockFactory class is responsible to create mock objects. Behind the scenes it generates dynamic assemblies and dynamic types. The GetInstance method can be used to obtain a mock instance.
The test case we analyzing above uses the GetUiShellInstanceCreateToolWin method to obtain a mock instance. In its body it uses the AddMethodCallback to register a mock method (in this case a mock for CreateToolWindow). When we call CreateToolWindow through the mock object, it actually calls CreateToolWindowCallBack that emulates the successful tool window creation and sets the output arguments and return value accordingly.
I think it worth looking into other test cases. You will gain a lot of information about how VS Shell and its services work.
The IntegrationTestProject
The SimpleTesting_IntegrationTestProject contains real integration tests where the test cases run within a Visual Studio 2008 Experimental Hive process. If you have a look into the source files into this project you will find not only simple test cases, but a few files that contain useful information. I suggest having a look at them, I am sure you will find interesting details:
| File |
Description |
| NativeMethods.cs |
A few definitions to access Win32 API methods in user32.dll and kernel32.dll through P/Invoke.
|
| Utils.cs |
Helper method definitions that are used by the integration tests. For those — just like me — beginning VSX programming it has a lot of value: you can find simple “how-to” answers in this file. Instead of diving into details, let me tell you a few method names. These really do what you think of (and there are many more): CreateEmptySolution, ForceSaveSolution, AddNewItemFromVsTemplate, SaveDocument!
|
| DialogBoxPurger.cs |
This is a very nice class that can be used to close dialog boxes that come up during VS Shell calls. The implementation of this class is a few hundred of lines, but its use is really simple and “spectacular”.
|
The IntegrationTestProject contains the following pre-generated test cases:
| Test case (method) |
Description |
| PackageLoadTest |
Checks if the package can be successfully loaded (and sited) into the VS IDE.
|
| LaunchCommand |
Checks if the simple menu command (in the Tools menu) generated by the VSPackage wizard works correctly: displays and closes a message box. This is an example to peek the DialogBoxPurger class at work.
|
| ShowToolWindow |
Checks, if the menu command displaying the simple tool window works as expected.
|
CPPWinformsApplication
WinformsApplication
VBWinformsApplication |
This test cases check if a C++, C# and VB.NET Winforms application can be successfully created while our package is loaded into the VS IDE. This test is to check if our package does not cause any unexpected side-effect.
|
| CreateEmptySolution |
Checks if an empty solution can be created while our package is loaded and it does not cause unexpected side-effects.
|
If you decide to look how these methods work, I suggest you to start with CreateEmptySolution and follow with the Winforms tests. Each test case embodies the tests into a UIThreadInvoker envelope like this:
[TestMethod][HostType("VS IDE")]
public void CreateEmptySolution()
{
UIThreadInvoker.Invoke((ThreadInvoker)delegate()
{
...
});
}
This class executes the code represented by the delegate in the Invoke method’s argument on the current UI thread. The delegate bodies represent methods emulating user interactions.
I suggest not just have a look into the test cases but also to debug into them.
Where we are?
In this article we created test cases with the VSPackage wizard and took a dive into the generated test methods. The generated code has a few “issues”; I showed how to fix them.
We got two test projects “free of charge”:
_UnitTestProject: Test cases to make some basic “health tests” about our package. These tests use mocking to emulate that the package is loaded into within Visual Studio.
_IntegrationTestProject: Test cases that check if our package loaded into the VS IDE works as expected without any side-effects. These set of tests really run within the Visual Studio 2008 Experimental Hive.
Right now there is only a few documented information about what is the preferred pattern to test packages developed with VS 2008 SDK. It seems Microsoft started to develop tool libraries for package testing purposes, but these libraries are far away from being ready. Till that time we can write our own tests where the code generated by the VSPackage wizard can be used as a good starting point.
Posted
Feb 07 2008, 06:46 PM
by
inovak