Agile2009 – Jour 1 – Integration Tests Are A Scam – J. B. Rainsberger
I went to see J.B. Rainsberger’s “Integration tests are a scam” today at Agile 2009.
A good portion of the presentation focused on the sheer numbers of tests required to tests the combined complexity of many classes integrated together. His argument being that the gap between the actual coverage of integration tests and adequate coverage is largely unmeasurable, but makes for such uncertainty in test coverage as to make integration tests mostly meaningless. In fact worse, they lead to a false sense of security.
His solution to this is one of separating dependencies (notably services) with interfaces (see the dependency inversion principle) and using these interfaces to:
- test the client object using mocks
- test the server object via its interface to ensure contract (interface) compliance
This will allow a great deal of confidence in the successful integration of components because their interactions have been validate sufficiently with incurring the cost of integration tests. In fact he recommends putting integration tests aside, only bringing them in to find specific bugs, but throw them away once the bug is solved and unit tested.
Another important advantage of this approach is the increased test execution speed. This goes without saying.
Past experimentation with this led me to the conclusion that some of the pain you feel when using mocks to test each class in isolation is tied directly to the design of the code being tested or mocked. Consequently the cure for this pain is a modeling one. Breaking down objects into smaller ones with fewer dependencies helps to reduce the complexity of the individual components (see the single responsibility principle).
Applications go from small model/large objects to large model/small objects.
This poses a new problem for teams. Breaking objects down into smaller ones can be a challenge, especially for developers less comfortable object modeling concepts. This often leads to a clutter of very poorly named objects that are unnavigable. Great attention must be put into ensuring navigable namespaces and classes.
The presentation did not state that acceptance, stress and other types of tests should be dismissed, but ultimately that interaction testing with integration tests was unjustifiably costly for its poor ROI.
All in all a well presented, if a controversial subject. I agree that favoring unit and interaction tests is a great way to develop with code confidence. I’m not sure I’m ready to throw away integration testing, but maybe I’ll give it a try for a bit. Who knows, maybe I’ll like it.
What do you think?
