Pex v0.10.40408.0: NUnit, MbUnit, xUnit.Net, Visual Basic.NET, F# and more…

Download
the new version!

Today we’ve released a new release of Pex on
DevLabs and on our academic downloads. This highlights of this release are: NUnit, MbUnit and xUnit.net support
out of the box, writing parameterized
unit tests
in VisualBasic.NET and F#, better Code
Contracts
support
. As always, if we encourage you to send us feedback,
bugs, stories on our forums at http://social.msdn.microsoft.com/Forums/en-US/pex/threads/ .

NUnit, MbUnit and xUnit.net supported
out of the box

Pex now supports MSTest, NUnit, MbUnit and xUnit.net out of the box. Pex will automatically
detect which framework you are using by inspecting the assembly reference list, and
automatically save the generated tests decorated with the correct attributes for that
framework.

image

The default test framework can also be specified through the global options (Tools
–> Options –> Pex –> enter the test framework name in TestFramework).

image

 

Writing Parameterized Unit Tests in VisualBasic.NET

While the Pex white box analysis engine works at the MSIL level, Pex only emits C#
code for now. In previous releases, this limitation made it impossible to use Pex
parameterized unit tests from non-C# code. In this release, we have worked around
this problem by automatically saving the generated tests in a ‘satellite’ C# project.

Let’s see this with an example. The screenshot below shows a single VisualBasic.NET
test project with a Pex parameterized unit test:

clip_image002

We can right-click in the HelloTest.Hello method and select “Run Pex Explorations”:

clip_image004

At this point, Pex will start exploring the test in the background as usual. This
is where the new support comes in: When a generated test comes back to Visual Studio,
Pex will save it in a separate C# project automatically (after asking you where to
drop the new project):

clip_image006

The generated tests are now ready to be run just as any other unit tests!

Writing Parameterized Unit Tests from F#

Similarly to VisualBasic.NET, we’ve made improvements in our infrastructure to enable
writing parameterized unit tests in F#. Let’s see this with a familiar
example. We have a single F# library that has xUnit.net unit tests and reference Microsoft.Pex.Framework
(project Library2 below). In that project, we add a parameterized unit test (hello_test):

clip_image008

We can right-click on the test method name and Pex will start the
exploration of that test in the background. Because of the limitations of the F# project
system, you absolutely need to right-click on the method
name
in F# if you want contextual test selection to work. Because the project
is already referencing xunit.dll, Pex will also automatically detect that you are
using xUnit.net and use that framework. When the first test case comes back to VisualStudio,
Pex saves it in a separate C# project:

clip_image010

The tests are saved in the generated test project and ready to be run by your favorite
test runner!

PexObserve: Observing values, Asserting values

We’ve completely re-factored the way values can be logged on the table or saved as
assertions in the generated tests. The following example shows various ways to log
and assert values:

image

In the Observe method, we use the return value and out parameter output to automatically
log and assert those values. Additionally, we add “view input” on the fly to the parameter
table through the ValueForViewing method, and we add “check input” to be asserted
through the ValueAtEndOfTest method. After running Pex, we get the following results:

image

As expected, input, ‘view input’, output and result show up in the parameter table.

image

In the generated test, we see assertions for the return value, out parameters and
other values passed through the ValueAtEndOfTest method.

Code Contracts : Reproducible
generated tests

When Pex generates a unit test that relied on a runtime contract, Pex also adds a
check to the unit test which validates that the contracts have been injected into
the code by the contracts rewriter. If the code is not rewritten when re-executing
the unit test, it is marked as inconclusive. You will appreciate this behavior when
you run your unit tests both in Release and in Debug builds, which usually differ
in how contracts get injected.

clip_image012

Code Contracts:  Automatic filtering of the contract violations

When Pex generates a test that violates a Code Contract pre-condition (i.e. Contract.Requires),
there are basically two scenarios: the precondition was on top of the stack and should
be considered as an expected exception; or it is a nested exception and should be
considered as a bug. Pex provides a default exception filtering that implements this
behavior.

Stubs: simplified
syntax

We’ve considerably simplified the syntax of stubs by removing the ‘this’ parameter
from the stub delegate definition. Let’s illustrate this with a test that stubs the
‘ReadAllText’ method of a fictitious ‘IFileSystem’ interface. 

Stubs: generic
methods

The Stubs framework
now supports stubbing generic methods by providing particular instantiations of that
method. In the following example, the generic Bar<T> method is stubbed for the
particular Bar<int> instantiation:

image

Stubs and Pex: Pex will choose the stubs behavior by default

We provide a new custom attribute, PexChooseAsStubFallbackBehaviorAttribute, that
hooks Pex choices to the Stub fallback behavior. To illustrate what this means, let’s
modify slightly the example above by removing the stub of ReadAllText:

image

If this test was to be run without the PexChooseAsStubFallbackBehavior attribute,
it would throw a StubNotImplementedException. However, with the PexChooseAsStubFallbackBehavior
attribute, the fallback behavior calls into PexChoose to ask Pex for a new string.
In this example in particular, on each call to ReadAllText, Pex will generate a new
string for the result. You can see this string as a new parameter to the parameterized
unit test. Therefore, when we run this test under Pex, we see different behavior happening,
including the “hello world” file:

image

Note that all the necessary attributes are added at the assembly level by the Pex
Wizard.

Miscellanous bug fixes and improvements

  • [fixed] Dialogs do not render correctly under high DPI
  • When a generic parameterized unit tests does not have any generic argument instantiations,
    Pex makes a guess for you.
  • When a test parameter is an interface or an abstract class, Pex now searches the known
    assemblies for implementations and concrete classes. In particular, that means that
    Pex will often automatically use the automatically generated Stubs implementations
    for interfaces or abstract classes.
  • Static parameterized unit tests are supported (if static tests are supported by your
    test framework)
  • Better solving of decimal and floating point constraints. We will report on the details
    later.

Breaking Changes

  • The PexFactoryClassAttribute is no longer needed and has been removed. Now, Pex will
    pick up object factory methods marked with the PexFactoryMethodAttribute from any static
    class
    in the test project containing the parameterized unit tests. If the generated
    tests are stored in a separate project, that project is not searched.
  • The PexStore API has been renamed to PexObserve.
  • Pex is compatible with Code Contracts versions strictly newer than v1.1.20309.13.
    Unfortunately, v1.1.20309.13 is the currently available version of Code Contracts.
    The Code Contracts team is planning on a release soon.

 

Happy Pexing!