At my last client I got the unusual chance to try three Java mocking frameworks within the same project. The project had started to use EasyMock as the project team felt that Mockito didn’t really have any decent documentation (it does but it’s in the Mockito codebase, not on the Google Code wiki). However as a ThoughtWorks team was working next door there was a strong push to use that.
My personal preference is still for JMock so I also selfishly pushed that into the project to round out the selection. With all three there; we were ready for a Mock Off!
The first distinctive thing is that EasyMock and Mockito can mock concrete classes not just interfaces. The second thing that all three have very different methods of constructing and verifying mocks. EasyMock is record-replay, JMock constructs expectations that are automatically verified by a JUnit runner when the test stops executing, Mockito uses a TestSpy where you can verify what happened to the mock whenever you want.
The record-replay paradigm lead to EasyMock being discarded early on. It has two kinds of problems as far as I was concerned. Firstly you have the feeling of “inside out” where the test is copying the internals of the class under test. Secondly I was confused several times as to what state my mock was in and having to switch the mocks to replay mode felt noisy, particularly where you were setting multiple collaborators in the test (do you switch them to replay once their recording is done or do you record all mocks then switch all of the them to replay?).
Mockito’s easy mocking of concrete classes made it a valuable addition to the toolkit and it does deliver the promised noise free setup and verificiation. However its use of global state was frustrating, particularly in that you can create a compiling use of the API that fails at runtime. If you call verify without a following method then you get an error about illegal use of the framework. Although this is meant to happen in the test that follows the test with the illegal construction, in practice this is hideous pain when the test suite is a non-trivial size. It also meant that a test was appearing to pass when actually nothing was being verified and the error appeared in another pair’s test (the one that implemented the next test) making them think that something was wrong with their code.
Mockito also uses a lot of static imports (which I do have a weaknesses for) but it also means that you have to remember the entry points into the framework.
JMock by comparision to Mockito feels quite verbose, the price for having all that IDE support and a discoverable API is that you have to have a Mockery or two in all your classes and you are defining all your Expectations for the Mockery. There’s no denying that even with the IDE autocomplete you’re doing a lot more typing with JMock. From a lazy programmer point of view you are going to go with Mockito every time.
And that is pretty much what happened in the project. Which I think is a shame. Now in explaining this I am going to go into a bit of ThoughtWorks testing geekery so if you are lazy feel free to go off and use Mockito because the pain I’m talking about will happen in the future not now.
I feel that Mockito is a framework for Test Driven Development and JMock is a framework for Test Driven Design. A lot of times you want the former: tight-deadline work and brownfield work. You want to verify the work you are doing but often design is getting pushed to the sidelines or you actually lack the ability to change the design as you might want. Mockito is great for that.
However Mockito doesn’t let the code speak to you. It takes care of so much of the detail that you often lose feedback on your code. One thing that making you type out your Expectations in JMock does is make you think, really think, about the way your code is collaborating. I think you are much more likely to design your code with JMock because it offers lots of opportunities to refactor. For example if two services are constantly being mocked together then maybe they contain services that are so closely related they should be unified, logically or physically. You don’t see that opportunity for change with Mockito.
By using both JMock and Mockito I was able to see that Mockito did tend to make us a bit lazy and our classes tended to grow fatter in terms of functionality and collaborators because there was no penalty for doing so. I was also concerned that sometimes Mockito’s ability to mock concrete classes meant that sometimes we mocked objects that should have been real or stub implementations.
Mockito is a powerful framework and I suspect that Java developers will take to it as soon as they encounter it. However for genuine Test Driven Design I think Mockito suffocates the code and requires developers with a lot more discipline and design-nous, almost certainly not the people that wanted an easy to use framework in the first place.
Echo One 
I started to work with JMock but I choose Mockito for any reasons.
For me, Mockito is more easy to use. In JMock I always need to declare ‘mockery.checking’ and ‘mockery.assertIsSatisfied()’. JMock is more verbose. Another thing, interface is good, but if a old project doen’t work with interface I don’t need to create one and change the design. To mock concrete class is more easy in Mockito.
Can I give a example.
In JMock:
import org.jmock.Expectations;
import org.jmock.Mockery;
private AnyService servicesMock;
private MyClass myClass;
//setup
servicesMock = mockery.mock(AnyService.class);
myClass.setAnyService(servicesMock);
// test method
mockery.checking(new Expectations() {
{
one(servicesMock).validateNumber(any(Integer.class));
one(servicesMock).getMagicCalc();
will(returnValue(123));
}
});
myClass.showCalc();
// tearDown
mockery.assertIsSatisfied();
In Mockito:
import static org.mockito.Mockito.*;
@Mock
private AnyService servicesMock;
private MyClass myClass;
//setup
myClass.setAnyService(servicesMock);
// test method
with(servicesMock.getMagicCalc()).thenReturn(123) // Fluent Interface
myClass.showCalc();
verify(servicesMock).validateNumber(anyInt());
So, Mockito is more simple, less code, more fluent, and expetarions is more easy to understand. However, BDD with Cucumber + Rspec + JRuby for Java Tests is exciting.
Cheers,
Ricardo Almeida
I think your examples confirm a lot of what I said in the post. It also provides a good example of how Mockito allows you to mock too much. A Calculator or Validator, for me, are classic things that should be tested with the actual object, not a mock. Also if I look at your JMock expectations I have a much better idea of what your code should be doing than in the Mockito example (although I suspect you mean that getMagicCalc can be called any number of times).
As a minor point if you @RunWith(JMock.class) you don’t need to expressly verify your expectations. That would be the more common idiom for JMock.
The only thing I really disagree with is that Mockito has the benefit of being a fluent interface. I agree it is, but JMock is kind of the daddy of the fluent interface don’t you think? Also you have multiple static start points for Mockito whereas most of JMock flows from that mockery you have to create (although you still have to memorise the expectation syntax).
I think Mockito will be more popular with developers (see my post), my concern is that it will lead to poorly designed code as it hides the complexity building up in the codebase.
Nice post, and I think “Mockito is a framework for Test Driven Development and JMock is a framework for Test Driven Design” hits the spot.
I talked to Sczepan when he was in London and it seemed to me that he’d built a good solution to a problem the team shouldn’t have had, but that was above his pay grade. He’s certainly challenged us to think about our usability.
In the end, we’re all suffering from the inadequacies of Java and if we lived in a world that had adopted, say, Fortress, we’d all waste less time.
Hi
I don’t understand the issue regarding mocking concrete classes–JMock can do this as well (I do this all the time), see http://www.jmock.org/mocking-classes.html. Am I missing something?
Regards
Patrick
I felt that Mockito has it right by allowing mocking of concrete classes by default. It feels a lot more elegant than the setImpostoriser method on Mockery (compared to a lot of stuff in JMock I’m not sure the naming really makes it clear what is happening).
If you are going to mock a mix of concrete and interface classes I would probably be tempted to use Mockito because the result is going to look more consistent.
I think that Mockito and JMock use the same strategy (and libraries) of generating proxies behind the scenes to achieve this so the question is more how you expose the ability via the API.
@Rrees We put a little grit in the works on purpose, to really highlight the fact that there are classes being mocked — although perhaps it could have been named better. We’ve seen so much code where class mocking is out of control, leaving the code and tests tightly bound and brittle.
There’s also an interesting performance issue with class mocking for large test suites so that some of the other frameworks have had to introduce caching.
@Steve I think it that same difference between development and design. If I am designing my software with tests then I deserve (and need) that grit.
But when I am developing a system and perhaps I can’t change the design for political or collective responsibility reasons then all I see is JMock throwing grit in my way and I am going to prefer to use Mockito.
I think it is better to be lenient and then allow increasing levels of strictness. The more in control of circumstances I am, the more discipline I want to impose on myself. The same way that testing suites evolve in complexity and completeness if they aren’t created from the outset.
It’s probably a topic worthy of another post but I can’t personally think of a valid reason for mocking a concrete class except where it is imposed on you by someone else’s design decisions. I’m trying to think of an example an I’m drawing a blank at the moment.
@rrees You’re probably right, although, I like to make a point that the cleanup starts /now/, although I think there are still issues of expressiveness.
Mockito was written to address a different problem (brought about by the development practices of a (ahem) certain consultancy) from JMock — and I prefer it to EasyMock (sorry, Tammo)
S.
Hi,
JMock advocates tend to say that it gives better feedback about the quality of the code – noisiness points out design issues. In my opinion mocking framework is not enough to teach good design. I’ve seen some really ugly codebases and somehow developers failed to hear JMock’s feedback. Therefore I believe that in order to keep your code clean you need to complement the tool with coaching. I committed a blog post on related subject here: http://monkeyisland.pl/2008/11/03/iwish
I totally respect you prefer JMock – use whatever technology you like to keep your code clean & well designed. I’m the mockito guy and I’m biased just like you 🙂
Too me, test driven development and test driven design is the same thing. However, if I have to use JMock to execute TD-Design then I’m sticking with TD-Development 🙂
I’m wondering how much time did your team actually spend with JMock? Was it equal with the time you used Mockito? Reading your blog post I can sense a bit of resentment your team weren’t given a chance to use JMock widely and long enough. I hope in your next project you will have an opportunity to use only JMock and it will fulfill your hopes.
>team felt that Mockito didn’t really have any decent documentation (it does but it’s in the Mockito codebase, not on the Google Code wiki).
I had no idea that it makes any difference how documentation content is presented – after all it’s available on the web page. The point of having all documentation in javadoc is that you have consistent content on the web page and in the code. Also it is available straight from your IDE even if you’re offline. I’m almost offended that you call it indecent but again I respect & appreciate your feedback.
>However its use of global state was frustrating, particularly in that you can create a compiling use of the API that fails at runtime.
Global state is just the trade off of the syntax. Mockito chose this solution, JMock chose Mockery instance. You clearly prefer the latter and it’s fine. Both approaches have pros & cons.
Bear in mind that in *all* mocking frameworks you can create compiling use of the API that fails at runtime. In trunk version of Mockito that will be released soon I added some stack trace voodoo that removes problems you mention in your post.
Sorry for the length of this response :). Last last thing is an almost personal message to Steve: when I demoed you Mockito at XP Tuesday long time ago I made a mistake to show you examples on ugly code. Believe me or not: Mockito was was built to address readability, maintainability and fluency of tests. It would be cool if you didn’t broadcast that you completely cracked the motivations behind Mockito because frankly… you didn’t!
Hey Szczepan,
To clarify one of your complaints: the team prior to my arrival had failed to find the documentation to Mockito. I helped them find it and included the clarification in the post in case other people were struggling. I did not say that Mockito does not have any decent documentation. I was saying that a team from outside the Agile community chose EasyMock over Mockito because they were expecting to find the documentation in the Google Code wiki. That’s just feedback.
My huge concern with Mockito is that what it really does is generate those same ugly codebases as you mention above only quicker and more quietly while giving the developers a false sense of security. You don’t see a difference between design and development but I do and what worries me about Mockito is that it loses signal as well as “noise”.
No framework can force “good design” on people but if we want to have tests as specification then we need to clearly capture the intent of the interaction. Looking generally at frameworks like EasyBDD, RSpec and JDave you can see how test frameworks are becoming part of the design specification of the codebase.
I would also like to remind you that you are the one who is personally bound to in a mocking framework. I am happy to use the best framework available. I have pointed out to Steve things that I think Mockito does right compared to JMock. The post highlights my concern with code written with Mockito and the attitudes in engenders. I have used JMock on projects and now Mockito (and briefly EasyMock). I am not by any margin a JMock advocate seething with resentment because I couldn’t use “my” framework. I felt it was interesting to have all three frameworks in project and the post is based around that experience. I would like it if you actually did respect that rather than just saying that you do.
Pingback: An interesting blog post about Mockito (and other mocking frameworks) « Dahlia Bock
>I would like it if you actually did respect that rather than just saying that you do.
Although it’s not clear from the tone of my replay but I really do, I promise! And I shut up just now. Take care Robert and keep on blogging!
Hi there,
As a enterprise java developer i find Jmock is well and good for testing java, but if i want to write my test case in groovy (to test java clasess) i found Jmock tough to use becuase of it’s heavy usage of inner classes (ie.. context.cheking ( new Expectations(){……..}} );.. which groovy doesn’t support 😦 anonymous inner classes.. could any one guide me work work with groovy test script + jmock
@amuthan I would expect jmock to be tricky to use with groovy, since much of its visible layer is there in a vain attempt to build a DSL. That said, the real substance of the implementation is just below the DSL layer and it wouldn’t be very hard to have Groovy drive that layer instead. Then the DSL would be in Groovy, which would probably work much better.