Who Tests the Tests?

Q: Who tests the tests?
A: The whole team.

1. Story Tests or Customer Tests test the fully-integrated application. Those tests are developed in close collaboration with the Customer and his/her helpers (for example, Business Analysts, Professional Testers).

2. Programmer Tests, which come from doing Test Driven Development, are tested by seeing them fail (for the right reasons) before making them pass. When pair programming, two sets of eyes see those tests and the code that makes them pass.

The Programmer and Story tests are also run again many times during development. And incremental/iterative development (ala XP, or Scrum, or any of the other Agile methods) requires refactoring as the team adds new features. Refactoring will change the code, but the tests should continue to pass.

The tests continue to pass despite refactoring because the behavior of the tested code should stay the same until both the tests and code-under-test are deliberately changed. If the tests fail during refactoring, either the tests are too close to the code, or the refactoring wasn't done entirely correctly. In either case, at least one or two programmers are going to look at those tests and the associated code again. So the tests are getting another set of eyes reviewing them.

Sometimes I have written a programmer test, which I expect to fail, and it passes unexpectedly. I take that seriously. Is the test not good enough? Is the code that I planned to change already doing what I want? Good thing I have a pair to help me answer those questions.