State-based vs Interaction-based testing in Dynamics CRM

When writing Unit Tests one can focus on two different aspects of the code under test. We can call these two types of tests ‘State-based’ and ‘Interaction-based’. The first of these concentrates on the end result achieved by the code under test – what was the end result of the code being run, whereas the second focuses on how that result was achieved.

This distinction is sometimes called ‘white-box vs black-box’ testing. With white-box testing (interaction-based) we can ‘see inside’ the code as it is running, and observe the internals of the code under test. With black-box (state-based) testing, the workings of the machinery are hidden from us; all we can see are the inputs and outputs.

White Box (Interaction-based) testing Black Box (State-Based) testing
Images Designed by Freepik

When writing unit tests for CRM plugins or workflows, until recently the only available method was to use a mocking framework, which restricts you to writing white-box testing. Mocking frameworks focus on interactions – they work by creating a mocked implementation of an interface (such as the IOrganisationService) and as part of your test you then have to write code specifying what should happen when a particular call to the mocked interface took place (ie, “if a call to the RetrieveMultiple method is called with these parameters, return this entity collection”.) Your Assertions take the form of “A call should have been made to the Update method, with these parameters, so many times”. From this we can infer that the code is working correctly, but we cannot actually verify the state of the CRM database at the end of the test.

FakeXRMEasy gives us the ability to do state-based testing for Dynamics CRM code, because it not only mocks the Organisation Service, it also creates an in-memory CRM database with which you can interact during your tests. All calls to the Organisation Service made during your tests (Creates, Updated, Retrieves, Associates, etc) behave in exactly the same way that they would against a real instance of CRM and you can verify your code by testing the state of the in-memory database using these same methods.

I’ve recently started using FakeXRMEasy for all my CRM Unit testing because I believe that writing State-based tests results in tests which are more maintainable, and this is a critical factor when building up a large library of tests for a project over a period of time.

State-based tests are more maintainable for the following reasons:

You can change the implementation of your code without invalidating your tests

Imagine that we have a plugin which is called when a record is updated, and which in turn updates a number of other related records in a parent-child relationship to the record being updated. Your initial implementation might use the Update method of the Organisation Service, and your test would contain an assertion that the Update method was called with certain parameters, a certain number of times (once for each child record).

// Assert that an update was called n times on an entity whose name is "newChildEntityName"
Service.AssertWasCalled(s => s.Update(Arg<Entity>.Matches(
p => p.LogicalName == "newChildEntityName")), o => o.Repeat.Times(numberOfChildRecords));

If you were later to decide that, for performance reasons, you wanted to change your code so that it used the ExecuteMultiple request instead of executing many update statements in a row, this would invalidate your test, even though your code would still be acheiving exactly the functional result you required.

Using state-based testing instead, we don’t have to know the details of how the code works in order to write the test – ie don’t have to know whether a query is using FetchXML or a Query Expression, and we don’t have to specify that ‘this type of query executed with these parameters will return this result set’. We just set up the faked CRM context at the start of the test, and verify the state of the records we are interested in at the end, using standard CRM API calls.

Here is an example in LINQPad of the most simple test I could write to demonstrate that you really only need to know the CRM API in order to use FakeXRMEasy:

Black box tests are more readable

State based tests are more easily understood and easier to write. You don’t have to master the sometimes convoluted syntax associated with mocking frameworks, and your tests will be using the same methods as the code you are writing your production code in. If you are a CRM developer, your tests will just use the already familiar methods of the CRM API (create, update, retrieve, etc).

Readability of tests is important because it makes it easier to understand what the test is actually testing, and so easier to maintain in the long run. It also makes it easier for a developer who is new to the project to understand the code and its tests.

Black box tests are more “BDD”

Black box testing lends itself to a more BDD (behaviour driven development) approach. Tests which make assertions about the expected state of the application are easier to turn into BDD style feature files than those which test the inner workings of the application, and are easier to discuss with business analysts, testers and end users.

No need to amend production code

With most mocking frameworks, when dealing with sealed classes such as RetrieveAttributeResponse, it is not possible to set the returned AttributeMetadata property as it is read-only, so it is necessary to write a wrapper class and to amend production code to use the wrapper code as described in this article by Lucas Alexander. Whilst this approach does work, it is preferable not to have to amend production code in order to make it to be testable in this way, and it adds another source of possible confusion for developers coming across this code for the first time. With FakeXRMEasy this is not necessary. Calls to RetrieveAttributeRequest, RetrieveEntityResponse and other similar methods can be achieved without any modification to production code.

When should we carry out White Box testing?

It is still possible to perform Interaction-based testing with FakeXRMEasy and there are times when you might still want to do this. Interaction based tests are useful if you want to test that your implementation is working as expected. A couple of examples of this are:

If you have conditional logic in your code which (say) switches between using a Retrieve and a RetrieveMultiple under certain conditions, your test could assert that the correct method had been called.

If you notice a bug such that a call to the API is being called more times than expected, you could introduce an assertion to check that (say) a query is only being carried out once.

Here’s the same test from earlier, this time written as an interaction test:

This LINQPAD script contains the above tests, with both the state and interaction based assertions present.

This article by the Jordi Montaña, creator of FakeXRMEasy goes into the differences between using a normal mocking framework and FakeXRMEasy in much more depth.

I hope I’ve managed to convey some of the many advantages that I feel using FetchXRMEasy brings to unit testing in CRM.

Posted in Dynamics CRM, FakeXRMEasy, Testing | Tagged , , | Leave a comment

Leave a Reply