Running unit tests from visual studio

[partnerinflight]

Hey guys:

So I'm evaluating using MemProfiler for my group. One of the features I was very excited about is the API, and being able to run unit tests that would do memory leak detection.

There are two main scenarios that I wanted to have happen:

1. I write my unit test and run from inside VisualStudio. I'm ok with using either NUnit or MsTest (I'm running under ReSharper that has test runners for both), but I definitely want to be able to start the test from inside VisualStudio. (think TDD)

2. I run my unit tests from within an automated environment.

I've looked around on the forums but can't seem to find a way to accomplish either of these scenarios cleanly.

The first one seems downright impossible, as VisualStudio starts mstest automatically, and ReSharper's unit test runners are not configurable.

The best scenario for me would've been if the first call to MemProfiler.FastSnapShot automatically kicked off memprofiler's process and attached to current process. It would've been not quite as good but almost, especially since MemProfiler supports attaching to a process, to be able to Process.Start memprofiler, specifying an Attach parameter, but I find no such parameter in MemProfiler's command line options.

Am I missing something? Because if not, it's fairly frustrating.

The second scenario has a similar problem -- I have to run all of my unit tests under MemProfiler if I want any of them to utilize the MemProfiler API. That may not be acceptable for my group.

I hope I'm just missing some configuration option. Otherwise the API is pretty useless to me.

Thanks!

Update:

So I've been trying to manually kick off mstest.exe /noisolation or nunit running under profiler. I'm profiling a WPF app, and the scenario I am profiling actually involves popping up a WPF window.

My results: NUnit simply crashes when I try to run my unit test. (Doesn't crash if not running under profiler.)

MSTest throws a very whacky exception:

Test method MemoryLeakUnitTests.Window2Test.Window2ConstructorTest threw exception: System.TypeLoadException: Could not load type 'Invalid_Token.0x01000043' from assembly 'MemoryLeakUnitTests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'..


Bottom line: I'm unable to figure out how to use the API to unit test WPF code. Help please.

[Andreas Suurkuusk]

The memory assertions API is designed to run tests within an automated environment, but using a test runner like ReSharper within Visual Studio to run tests containing memory assertions is currently not supported. I understand that this is annoying and we're looking into solutions for this.

The problem using memory assertions when unit testing is that the test code (e.g. NUnit or mstest) must run under the profiler. Attaching a profiler is currently not supported by the runtime, so the test runner must be started by the profiler. Attaching a profiler will be supported in .NET 4.0, so future versions of the profiler will most likely support the possibility to attach a profiler for memory assertions using the API. (The attach to feature in .NET Memory Profiler only analysis the memory of the process, it's not attaching a "real" profiler).

What we do when running automated tests is that we assign a category (e.g. "MemoryTest") to the tests that should be run under the profiler and then we run the tests twice. One time not using the profiler, which runs all the tests that don't perform any memory assertions, and one time under the profiler, which runs all the tests marked with the "MemoryTest" category.

For TDD, you can for instance run the NUnit GUI under the profiler. It's not as convenient as running within Visual Studio, but if you add a Tool entry in Visual Studio that start NUnit under the profiler, it's hopefully not too bad.

The problems you had running mstest and NUnit under the profiler worried me. Are you using declarative assertions? When I tried the declarative assertions myself I also noticed the same problem. This is of course a high priority issue and we have been working on a solution during the day. We did find an error in our profiler, which we have now fixed. However, even though it was an error in our profiler, we have not seen this issue before. Maybe a .NET runtime update has something to do with it.

There is a new build of the profiler available at: http://memprofiler.com/MemProfilerInstaller3_1_345.msi (or http://memprofiler.com/MemProfilerInsta ... -64bit.msi for the 64-bit version).


Can you download this build and see if it works better? We want to make an official release as soon as possible.

[partnerinflight]

Hi Andreas:

So it' working a bit better, but still not quite right.

Three issues:

1. If I run nUnit under profiler I cannot then rebuild my unit test assembly while nUnit is running. I'm told the .pdb file is in use so the assembly can't be copied. nUnit without profiler doesn't exhibit this issue. As you can imagine this makes TDD impossible -- I can't start/stop the profiler every time I want to change my unit test.

2. nUnit cannot exit when running under profiler -- it just freezes when I hit the close button.

3. I'm getting unexpected behavior with the unit test itself. When ran under unit the test passes, (I expect it to fail), but I do get a MemProfiler popup saying there's been a leak detected. How come the test passes then?

Here's the body of my test:

/// <summary>
///A test for Window2 Constructor
///</summary>
[TestMethod()]
[NoNewInstances("TestWpfApp.Window2")]
public void Window2ConstructorTest()
{
Assert.IsTrue(MemProfiler.IsProfiling);
UnitTestUtilities.RunDispatcherUntil(() => window.IsLoaded);
UnitTestUtilities.RunDispatcher(10000);
}

Am I just doing something wrong here?

Finally, did you guys make any fixes for msTest? Were you able to reproduce my issue?


Thanks!

[partnerinflight]

Andreas, any progress on the three issues I mentioned above? Were you able to repro them? Also would it be possible to role these changes into 3.5?

Btw, I was thinking more about the profiler issue. I wonder if it's possible to do something clever with the API structured in such a way as to automatically launch an external process (under profiler), do the unit test, and then report back?

Basically the (perspective) customer ask is to make it easier to use MemProfiler in unit test (both automated and TDD) scenarios.

Thanks!

[Andreas Suurkuusk]

I'm sorry for the delay. I have been working on this issue full time since your last post, but there is still some problems related to unloaded AppDomains, nUnit, and WPF. I will need to look into this further and get back to you again, but in the meantime I have provided some comments to your questions.

1. To make source information available, the profiler needs to load the .pdb. In v3.1 it would be fairly easy to only read the debug information while collecting a snapshot and then release the .pdb. However, the new stack reducer in v3.5 will need source information when running the profiler. We will investigate ways of not keeping the .pdb file open. Currently, you can avoid locking the .pdb by selecting to "Never" load debug symbols in the session options.

2. Previously you mentioned that you opened a WPF window. How is it opened? During my testing I noticed that unless I use a message loop to show a window (using Application.Run instead of just Window.ShowDialog or something similar), the AppDomain cannot be unloaded. One Thread in the AppDomain is stuck on a wait for a native event related to the graphics driver, which causes AppDomain unload to fail. This occurs even if not running under the profiler, and each time the test is reloaded, another AppDomain gets stuck. The stuck AppDomains causes problems when exiting the process when running under the profiler. You can use the "Stop" command to force nUnit to exit.

3. When you use declarative assertions, a failed memory assertion will not automatically be forwarded to the unit testing framework (the profiler has no information on where to find the unit assertion methods). This can be solved by using the MemAssertion.AssertionFailed event. E.g.

Code: Select all

[TestFixture]
public class TestClass
{
   [TestFixtureSetUp]
   public void FixtureSetup()
   {
      MemAssertion.AssertionFailed += new EventHandler( MemAssertion_AssertionFailed );
   }

   [TestFixtureTearDown]
   public void FixtureTearDown()
   {
      MemAssertion.AssertionFailed -= new EventHandler( MemAssertion_AssertionFailed );
   }

   /// 
   /// Handles a failed memory assertion by failing a unit test assertion.
   /// 
   void MemAssertion_AssertionFailed( object sender, EventArgs e )
   {
      Assert.Fail( "Memory assertion has failed." );
   }

   /// 
   /// A test that uses declarative assertions.
   /// 
   [Test]
   public void SomeTest()
   {
      MethodWithDeclarativeAssertion();
   }

   [NoNewInstances(...)]
   void MethodWithDeclarativeAssertion()
   {
      // ...
   }
}

I have not yet had the time to test the new build with MSTest, but the fix for NUnit should work with MStest as well.

The changes we have made will be included into v3.5, but v3.5 is currently not suitable for a public release. We will soon announce a beta of the new version.

I don't fully understand in what way you want to launch the external process for unit testing. Can you provide some additional explanation? Note that we will make an "external API" available, that will give you much more control over the profiler compared to the command line. For some information about this, see the forum post at http://memprofiler.com/forum/viewtopic.php?t=1317

[partnerinflight]

Great, Andreas! Waiting for the next build.

It's definitely necessary to avoid keeping the pdb open... can you maybe copy it into a cache or something? As I've said, TDD is impossible if you have to keep closing/reopening nUnit. I'll try the approach you suggested though.

2. Here's the code for the unit test I used:

Code: Select all

 /// <summary>
        ///A test for Window2 Constructor
        ///</summary>
        [TestMethod()]
        //[NoNewInstances("TestWpfApp.Window2")]
        public void Window2ConstructorTest()
        {
            Assert.IsTrue(MemProfiler.IsProfiling);
            MemProfiler.FastSnapShot();
            DoRunLeakTest();
            Assert.IsTrue(MemAssertion.NoInstances(typeof (Window2)), "Expected no leaks, found a leak!");
        }

        private void DoRunLeakTest()
        {
            UnitTestUtilities.RunDispatcherUntil(() => window.IsLoaded);
            Window w2 = window.LaunchMemoryLeakWindow(0, "test");
            UnitTestUtilities.RunDispatcherUntil(() => w2.IsLoaded);
            w2.Close();
        }

Here's the RunDispatcherUntil function:

Code: Select all

  /// <summary>
        /// This method pumps the events in the main thread queue
        /// So that you can make any operation onto the UI take effect right away
        /// And continues to do so until the callback returns true, or a timeout occurs
        /// </summary>
        /// <param name="callback">Callback to test against for ending the loop</param>
        public static void RunDispatcherUntil(RunDispatcherUntilCallback callback)
        {
            DispatcherFrame currentFrame = new DispatcherFrame();
            double totalMilliseconds = 0;
            new DispatcherTimer(TimeSpan.FromMilliseconds(100), DispatcherPriority.Normal,
                                delegate(object sender, EventArgs args)
                                {
                                    DispatcherTimer exitTimer = (DispatcherTimer)sender;
                                    totalMilliseconds += exitTimer.Interval.TotalMilliseconds;
                                    if (callback())
                                    {
                                        exitTimer.Stop();
                                        currentFrame.Continue = false;
                                    }
                                    else if (totalMilliseconds > 10000.0)
                                    {
                                        Assert.Fail("Timeout Occured in RunDispatcherUntil");
                                    }
                                },
                                Dispatcher.CurrentDispatcher);

            Dispatcher.PushFrame(currentFrame);
        }

What I meant by the external process is something along the following lines:

1. User calls MemProfiler.StartProcess(Action delegate) with the delegate being a pointer to their unit test.
2. MemProfiler launches a brand new process under the profiler/mstest/nunit, somehow (compilation, DLR, not sure...) passing it the delegate the user provided.
3. The new process runs the delegate and then reports back the results somehow (IPC, etc).

I'm not sure that's realistic, but might be worth investigating. Of course if the AttachToThread function winds up getting implemented, as is mentioned on the other thread you pointed to, then a separate process would be unnecessary.

Thanks!

[partnerinflight]

Any news on this? Any new builds?

[Andreas Suurkuusk]

Unfortunately it seems like there is a bug in the .NET Runtime (and the .NET profiling API). If an AppDomain that has created a WPF instance (e.g. a System.Windows.Window) gets unloaded, an access violation may occur, causing the process to dead-lock. I have contacted Microsoft about this, but I have not yet received a reply. As soon as we get some more information (and hopefully a workaround), we will make a new build of .NET Memory Profiler. I will get back to you as soon as I have some news about this.

[Andreas Suurkuusk]

We have now created a new build of .NET Memory Profiler 3.1. It can be downloaded from http://memprofiler.com/MemProfilerInstaller3_1_348.msi (or http://memprofiler.com/MemProfilerInsta ... -64bit.msi for the 64-bit version). Using this build it should work better to do TDD, e.g. you should be able to recompile your project while it's loaded into a testrunner. However, to do this you would still have to make sure that the symbols are not loaded (by selecting "Never" as the load symbols option). This is something we will try to improve in the next version.

Unfortunately, this new build will probably not work for you. Microsoft has confirmed that there's a bug in the .NET framework. When unloading an AppDomain that has created instances of certain classes (e.g. System.Windows.Window) while profiling, the profiled process will crash (the crash will cause the process to dead-lock). This problem seems to have been introduced in .NET Framework 3.5 SP1 and, AFAIK, there's nothing we can do about it. Hopefully this will be solved in the next maintenance release of .NET.

[partnerinflight]

Thanks, Andreas! We wound up getting 20 licenses for our team. I haven't tested the new build yet, but will as soon as I get some time.

I will follow up with the appropriate .NET people to make sure this bug gets addressed.

Thanks again.

[Andreas Suurkuusk]

Great, I hope the profiler will be useful for your team. Unfortunately it seems like Microsoft will not fix the bug for .NET Framework 3.5 SP1. A fix will be included in .NET 4.0, however (hopefully in beta 2).