Don’t you love to crank code? As developers, we love to refactor; we love to be productive. As managers, how can we let the coding cowboys on our team be free, while maintaining code quality? At Twitter, we’ve been experimenting with feature testing. There are at least seven teams doing some form of feature testing of their services. Some of these teams have even discarded most of their single class tests, resulting in a test suite that’s 95% feature tests. It may be hard to believe, but throwing away single class tests can actually make development teams faster.
Let’s back up a bit and first define some terms:
Single class test - A test verifying methods of a single class. Any dependencies external to the class are ignored or mocked out. Note that some single class tests also qualify as feature tests in a few cases, depending on the scope of the “feature” under test.
Feature test (#featuretests) - A test verifying a service or library as the customer would use it, but within a single process. A few examples:
Integration test - One of a few definitions is possible (thus we’ll avoid this term)
Unit test - A test verifying a “unit” of functionality. What qualifies as a unit test is ambiguous, since the size of a unit can vary, or could even mean any test defined in JUnit. We will avoid this term as well.
Having established the terminology, we should be clear that this concept is not new. It was alluded to here ten years ago. Today, feature tests, sometimes referred to as integration tests, are being used on multiple projects. More novel, however, is the decision to delete single class tests on a project and use all or mostly feature tests.
Although creating your first feature test on a service takes a bit of time and can require complex setup, benefits typically outweigh the cost. Let’s look at some of the advantages.
Tests as documentation of use cases - Developers can easily step through the entire service, and working through the feature tests gives new developers good understanding of the feature. The end purpose of feature tests is generally much clearer than individual unit tests.
We have encountered many concerns about feature testing over the years, but we’ve found that most of those arguing against it have not fully embraced it on a project. Below are some common concerns we’ve heard:
Let’s start out with an example however of a webapp written using webpieces.
The design of this small project is a query for tweets that hits a remote service TweetSearchService to retrieve tweet ids, and then hits HydratorService to retrieve the actual tweet data. The design looks like:
The code to setup our test suite is fairly terse (excluding the reusable setup code called from this method):
This exemplifies a well-designed feature test framework, in that minimal code is required to
If you look closely, there is also an AppOverridesModule which allows you to override remote endpoints with mocks, using Guice:
Now, let’s move on to a typical test case, which is similarly terse:
Feature tests generally follow the same basic template:
Setting up the first feature test is a bit of work, but many have been shocked at how little test code you have to write once your framework is in place. You really get a big bang for your buck in feature testing. This one test covers all five classes shown in the design diagram above. Using the single-class test strategy, you’d have to write five tests to cover those same classes.
The full code can be found here.
In feature testing, there are some pretty interesting and fun design scenarios that come up. For instance, how do you write a feature test if SearchController above is using an ExecutorService? How do you test code that uses Thread.sleep? How do you deal with code that has a while(true) loop? Dealing with time (e.g. System.currentTimeMillis()) can be tricky in feature testing, as well as getting timer tasks to fire appropriately in the feature test (e.g., how do you make a timer task run now, even though it was scheduled to go off tomorrow in the production code?).
There are a few patterns to help with these advanced cases, which are beyond the scope of this article, but we are available to help with specific scenarios -- feel free to reach out!
As with anything in software engineering, feature testing is not a silver bullet, but we are ecstatic about our development speed these days. Refactoring is much more likely to happen on our team, as we do not have the burden of worrying about rewriting tests and blowing away our safety net. As our safety nets grow, we gain the ability to be coding cowboys/cowgirls that have a good safety net to catch our mistakes. Hit us up on Twitter at @deansoldtweets or @frogbuddha42 and feel free to use the Twitter hashtag #featuretests.
Although this blog article is too brief to make a complete case, we hope we’ve given you a compelling reason to consider feature testing.
Did someone say … cookies?