Apr 25 2009
.Net Mocking Frameworks – Capability Comparison
I have a couple years of experience of TDD under my belt, but it's only recently that I've felt like I am a relatively decent practitioner of it. I attribute this to forcing myself to take the plunge into mocking, and the knowledge of patterns and loosely-coupled design that I've gained from it.
You see, I work on a pretty large and complex ASP.Net webforms product, and tests were introduced late into the development cycle of the initial release. We favored integration testing with real data sources over actual unit tests. While I did write some tests, I knew that our product was not very testable by design.
Recently we put together a public facing API for programming against the product, that we were able to build from scratch. This was a natural opening to apply test driven practices and start building unit tests from the get-go. Due to the service oriented nature of the data that the product consumes, I soon found myself realizing that I needed mocks in a big way. I gritted my teeth and dove head-first into Rhino.mocks.
Several weeks down the road, it was obvious that Rhino just isn't right for our environment. The learning curve is too steep to win quick success with all our developers (and therefore by extension, our project managers). I began looking for another framework.
This led me on a frustrating research mission to find the differences between frameworks without actually trying them all. Due to a lack of in-depth comparisons of the actual capabilities of the various mocking frameworks, I've ended up putting together this chart. I'm hoping it helps some people. My interest is very high in this arena, so I will be keeping it up to date.
(Just for interest's sake -- we ended up choosing Moq, due to the ease of the API. The philosophy of Moq jives perfectly with our relatively fast-paced and efficient development environment. We don't care about purism -- we care about getting it done.)
.Net Mocking Framework Comparison
Other Notes
Rhino
|
Moq
|
NMock2
|
TypeMock.Net
|
The above observations were gleaned from many web pages, documentation repositories and blog posts, and reflect my limited understanding of each framework. There may be errors. If you think I'm wrong, or you can think of other capability aspects that I've overlooked, please let me know!
7 responses so far
Nice comparison! Thanks for this post.
Some additions to NMock2:
- "confusing product version": sad but true, we continue to develop NMock2 (with the 2 in the name). NMock2 is based on NMock version 2 (2 in the version number). Unfortunately, we cannot convince the "old" NMock team to link their pages to our version. The current version can be found at http://sourceforge.net/projects/nmock2.
- support for events: you mark it as partial support. What do you miss?
- class mocks: this is a new feature of our current release candidate, only.
- "callback on mocked method": this is support with actions (e.g. DelegateAction) in NMock2.
- "magic strings": it is true that that makes refactoring a bit more complex. However with the right toolset (e.g. ReSharper) we don't have problems with that. And from a pure TDD point of view, you should change your unit test first
these magic strings have advantages and disadvantages: the disadvantage is clear, no compile time check, no intellisense support; the advantage is the flexibility we get: our syntax is not limited to what you can express in C#, what results in the easiest syntax to read (compared to the other frameworks). And if you do TDD then you read your unit tests more often than you write them.
Other things to compare:
- documentation, although we are very bad regarding this
- understandability of error messages: here we are very good, I think
- performance: not crucial but important, too
But, as long as developers do use a mocking framework (which ever) it's a good sign
Happy mocking
Urs
Urs -
Thanks for the reply. I marked the event support as partial because all I could find on events with NMock2 was some posts that showed workarounds for raising events. Same with callbacks. It might have been NMock I was looking at. Of course now that you point it out, I found in the documentation how to do it, so looks like no problems there. I'll update the chart asap.
When you say NMock2 now supports mocking classes, do you mean non-virtual members of a class as well?
Documentation was something I considered adding, but it's a little subjective. I think lots of people feel that Rhino is well documented, where I (being a relative noob) don't agree with that
Understandability of error messages may be something I add to the chart though, as this is often brought up as a failure of Rhino, which seems like a valid criticism.
[...] .Net Mocking Frameworks – Capability Comparison A nice comparison of popular .NET mocking frameworks. Good information to have. [...]
Hi Morgan,
Great work on the comparison. By the way did you see Andrew Kazyryevitch comparison? (http://code.google.com/p/mocking-frameworks-compare/)?
Just to update your table, we support callbacks now. And if you'e like to see the comparison we did, email me at Gilz at typemock dot com.
Thanks.
Gil Zilberfeld
Typemock.
Thanks Gil. I hadn't seen A.K.'s project but I'm glad you posted it. Looks extremely useful.
I'll update the chart.
[...] a bit of research I decided on using Rhino Mocks. The decision was primarily made from examining this comparison and my past experiences with Ayende. I have previously used Fluent NHibernate from Ayende, and was [...]
[...] Moq and some are commercial such as TypeMock . Here are two links that compare few of them http://www.phpvs.net/2009/04/25/net-mocking-frameworks-capability-comparison/ and http://codevanced.net/post/Mocking-frameworks-comparison.aspx [...]