Bounding Unicorns

Practical Test Driven Development

I had a conversation at one point with a startup founder who said he did test driven development 100% of the time. All code without exception had to have tests written for it first.

Despite writing many tests, being a vocal advocate of test coverage and effective tests, I often times do not do test driven development. I usually write tests for bugs before developing the fixes, and I often write tests when writing code similar to existing code in an application, but during initial development I often write the code first and the tests later. Let's examine the reasons why.

Cost vs benefit

Assuming we are talking about commercial software, the ultimate purpose of writing tests is deliver better software. A company makes money by selling a product or a service, and test coverage is only beneficial to the extent that improves the product or service being offered and provides more of the improvement than it costs.

Tests, like code, take time to write. The time taken to write tests is meant to pay off in the long term as bugs in the program are easier to find and subsequently fix, or said bugs are found by developers while the code is written rather than by users. But the law of diminishing returns applies: the more tests there are, the less benefit additional tests offer. At some point the benefits do not offset the time spent writing the tests.

Overlapping test types

For example, most modern web applications have controller unit tests and integration tests. Most functionality in controllers can be covered by both, on different levels. Creating both sets of tests for every feature point is unnecessary and wasteful, though examining the entire program and the test suite is generally needed to arrive at this conclusion.

Testing the same code many times

Another example is authentication: it is possible to write tests for each action/url in a web application checking that access to said action is either permitted or denied for each of the possible user types. In an application with complex access control, this might make sense. In an application where all users are the same and a user must be logged in to do anything other than view the login page checking each action is likely to be a waste of time.

Implementation changes

Depending on how a test is written, it may be more or less coupled to the implementation. Lower level (unit) tests are generally more tightly coupled to the code they are testing, and integration tests - despite being loosely couped to the particular implementation in terms of logic - often depend on the UI implementation to find and interact with the application's controls.

Creating tests while the implementation is in flux runs the risk of the tests being outdated by the time the implementation is complete. The faster a company is moving, the higher this risk. This is why prototypes generally are not developed test first - a prototype evolves so rapidly that tests written before the code are likely to be useless by the time the code is completed, frequently even the architecture undergoes enough change that the way the tests were written no longer applies.

Common pitfalls and lack thereof

Suppose bug-free programs could be written. What would the benefit of tests be for such programs?

There are classes of errors that some programmers make, and some do not. A common example is accidental assignment of zero (x = 0) instead of comparison with zero (x == 0). Some programming languages go to various lengths to make the two impossibe to confuse, such as by prohibiting using assignments as expressions or using := syntax. But some programmers, myself included, have no issue with typing the appropriate number of equal signs when needed.

Another example is SQL injection. In theory a program can be vulnerable to SQL injection at any point it takes user input. In practice if the programmer does not use constructs that have the potential to pass user input to the database unsanitized, there is no reason to test specifically for SQL injection vulnerabilities. If there is code where user input is specifically handled in a way that deviates from the common case, that and only that code needs to be tested for SQL injection vulnerabiities.

Tooling

It is notable that TDD proponents are typically using a framework that makes writing tests easy - like Ruby on Rails. What if instead they were tasked to, say, develop a site based on a PHP content management system that was extremely large and complicated yet had no test coverage? They would have had to develop their own test framework and write tests for the CMS before they could start the project they were actually tasked to do! Depending on the complexity of the project, the customer may have been happy to manually check that "it works" rather than wait months and spend lots of money building test infrastructure.

One Offs

Many projects, especially the larger ones, have a need for one-off code. It can be a report in a web application or a data import of some description. Should this code be developed in a TDD fashion when it is only going to be used once? I imagine many projects would reject such code from being in the project proper, but this means this code is unavailable the next time someone needs to make a similar-but-different report or data import.

One might argue that if the code is to be reused, it should be tested and developed "properly". And here we come to what I consider the crux of the issue: iterating on the software vs iterating on the process used to build it. Forcing TDD without exception is enforcing a rigid process on the developers. An agile process allows the freedom to begin with a prototype, validate that it is reusable by using it again, then ensure its quality is on par with standards required for production code.

Allowing process variability, in my mind, is a sign of high level engineering.