Saturday, May 5, 2007

What really drives development?

There is a lot of buzz in the software community about Test-Driven Development. I like to boil TDD down to this:
  1. Write tests first
  2. The tests are the de jure standard
  3. Correct software passes the tests
This is actually a pretty good engineering approach to software. It enforces a well-documented and agreed-upon standard, and provides clear, measurable standards for success. In fact, if the tests are written correctly, they will also verify the range of operation for varying input parameters.

I should expand a bit on number 2: the de jure standard, as opposed to de facto, is the agreed-upon, explicit standard. A de facto standard is one which may not be fully codified in writing, or one which is adopted as a matter of use and not planning. Stating that tests are the de jure standard does not imply that they supplant a written software spec; rather, they should simply mirror the software spec (or the spec should match the tests). In the end, every point in the spec should have a test written for it.

At any rate, this is a pretty good systems-level approach to software engineering. I was somewhat disappointed, then, to read a piece by Ravi Mohan called Learning From Sudoku Solvers. Ravi basically points to five failed attempts by Ron Jeffries to write a Sudoku solver using Agile (Test-Drive Development) practices, and one, far more successful, attempt by Peter Norvig. The tagline from reddit was Test Driven Development versus Thought Driven Development.

My disappointment is basically that nobody along the way identified the problem with using Test-Driven Development here: not Ron, not Ravi, nor the Reddit submitter.

The problem lies in shades of gray in software engineering: an algorithm is more or less atomic, systems are atoms, and tests operate on one atom at a time. The problem with the test-driven approach to solving the Sudoku puzzles is that it tries to break the algorithm apart, and in doing so, makes several incomplete, fragmented almost-atoms. The only tests that should be used with the Sudoku algorithm are to input an unsolved puzzle and see if the right thing comes out. (NB: the "right thing" may not be one solution; some puzzles have multiple solutions. Of course, this has to be considered in the test.)

If it is applied correctly, Test-Driven Development will guide a programmer to a well-factored solution. It it not, however, autopilot for good programming any more than Caps Lock is cruise control for awesome. A lot of care has to be taken to ensure that tests are applied at the correct level.