Who Tests the Tests?

Q: Who tests the tests?
A: The whole team.

1. Story Tests or Customer Tests test the fully-integrated application. Those tests are developed in close collaboration with the Customer and his/her helpers (for example, Business Analysts, Professional Testers).

2. Programmer Tests, which come from doing Test Driven Development, are tested by seeing them fail (for the right reasons) before making them pass. When pair programming, two sets of eyes see those tests and the code that makes them pass.

The Programmer and Story tests are also run again many times during development. And incremental/iterative development (ala XP, or Scrum, or any of the other Agile methods) requires refactoring as the team adds new features. Refactoring will change the code, but the tests should continue to pass.

The tests continue to pass despite refactoring because the behavior of the tested code should stay the same until both the tests and code-under-test are deliberately changed. If the tests fail during refactoring, either the tests are too close to the code, or the refactoring wasn't done entirely correctly. In either case, at least one or two programmers are going to look at those tests and the associated code again. So the tests are getting another set of eyes reviewing them.

Sometimes I have written a programmer test, which I expect to fail, and it passes unexpectedly. I take that seriously. Is the test not good enough? Is the code that I planned to change already doing what I want? Good thing I have a pair to help me answer those questions.

Peer Discussion Improves Student Performance

Science Daily reports that peer discussion using "clickers" helps students learn in a way that simple lecturing does not.

Clickers are simple audience response devices, similar to a TV remote control, that allow students to record their answers to thought-provoking, multiple-choice questions in class. After students answer a question individually, the instructor often asks them to discuss the question and then vote again before revealing the answer. After discussion, they usually do better on the question - but why?

The important point is that none of the students were told what the right answer was," said Su. "Even when students in a discussion group all got the initial answer wrong, after talking to each other they were able to figure out the correct response, to learn. That was unexpected, and I think that's dramatic.

Industrial Logic's Agile eLearning allows students to add comments or questions on every page, and even comment or question specific parts of a quiz. You can see a sample of this in the new videos in the Welcome Album. (See "Community and Feedback" in the track "Find Your Way: Start Here")

William Wake described the essence of an automated test as Arrange, Act, Assert. I've added "Erase", to account for the clean-up that some tests have to do. You might ask why "Erase" added to "Arrange, Act, Assert" and not some word starting with "A". I think starting with "eh?" is close enough. :-)

"The thing about elves is they've got no ... begins with m," Granny snapped her fingers irritably."

"Manners?"

"Hah! Right, but no."

"Muscle? Mucus? Mystery?"

"No. No. No. Means like ... seein' the other person's point of view."

Verence tried to see the world from a Granny Weatherwax perspective, and suspicion dawned.

"Empathy?"

"Right. None at all. Even a hunter, a good hunter, can feel for the quarry. That's what makes 'em a good hunter. Elves aren't like that. They're cruel for fun [...]"

—Terry Pratchett, Lords and Ladies

In C++, with certain test frameworks, a test might be specified in a manner something like the following in C++.

TEST(TestBlurImageFilter)
{
    // Arrange
    string outfileName = NewTempFileName("TestImageFilter");
    Image* sourceImage = new Image("lena.png");

    // Act
    ImageFilter* filter = new BlurImageFilter();
    filter->ProcessToFile(sourceImage, outfileName);

    // Assert
    AssertImagesEqual("expected_lena_blurred.png", outfileName);

    // Erase
    DeleteTempFile(outfileName);
    delete filter;
    delete sourceImage;
}

Note: generally you don't want to deal with files in unit tests; working in-memory would be much faster. Also, if this is one of those frameworks that throws an exception, or otherwise aborts the test if an assertion fails, then the "Erase" portion of the test won't get executed if AssertImagesEqual failed. Let's assume that's not a problem for the moment.

Let's imagine that you then write a another test like so:

TEST(TestUnblurImageFilter)
{
    // Arrange
    string outfileName = NewTempFileName("TestImageFilter");
    Image* sourceImage = new Image("lena.png");

    // Act
    ImageFilter* filter = new UnblurImageFilter();
    filter->ProcessToFile(sourceImage, outfileName);

    // Assert
    AssertImagesEqual("expected_lena_unblurred.png", outfileName);

    // Erase
    DeleteTempFile(outfileName);
    delete filter;
    delete sourceImage;
}

Now you've got duplicated "Arrange" and "Erase" sections. And duplicated logic in tests can be just as bad it would be in production code. Fortunately, most test frameworks already have support for extracting "Arrange" and "Erase" to methods in a "test fixture". The above code could be refactored to something like the following:

class ImageFilterTests : public TestFixture
{
public:
    ImageFilterTests()
        : sourceImage(NULL), filter(NULL)
    {
    }

    string outfileName;
    Image* sourceImage;
    ImageFilter* filter;

    virtual void SetUp()
    {
        // Arrange
        outfileName = NewTempFileName("TestImageFilter");
        sourceImage = new Image("lena.png");
    }
    virtual void TearDown()
    {
        // Erase
        DeleteTempFile(outfileName);
        delete filter;
        delete sourceImage;
    }
};

TEST_F(ImageFilterTests, TestBlurImageFilter)
{
    // Act
    filter = new BlurImageFilter();
    filter->ProcessToFile(sourceImage, outfileName);

    // Assert
    AssertImagesEqual("expected_lena_blurred.png", outfileName);
}

TEST_F(ImageFilterTests, TestUnblurImageFilter)
{
    // Act
    filter = new UnblurImageFilter();
    filter->ProcessToFile(sourceImage, outfileName);

    // Assert
    AssertImagesEqual("expected_lena_unblurred.png", outfileName);
}

Not only has this eliminated the duplicated logic, most unit test frameworks will also guarantee running the TearDown method even if the test fails, so you don't have to write your own try/catch blocks or other contortions for exception-safe "erase".

You'll see that I also added a constructor to insure that the pointer variables have valid NULL values so we don't delete garbage pointers if the Image or Filter objects were not allocated successfully. (You should consider using boost::shared_ptr and/or boost::scoped_ptr if you're dealing with object pointers in C++ code and tests, by the way.)

In those C++ test frameworks where the test-fixture creation and deletion is done just before and after executing the test, the SetUp and TearDown methods can (almost always) be replaced with a constructor and destructor instead. Using that and boost::scoped_ptr to insure exception-safe object deletion would allow us to write the following code:

class ImageFilterTests : public TestFixture
{
public:
    ImageFilterTests()
        : outfileName(NewTempFileName("TestImageFilter")),
          sourceImage(new Image("lena.png"))
    {
        // Arrange
    }

    virtual ~ImageFilterTests()
    {
        // Erase
        DeleteTempFile(outfileName);
    }

    string outfileName;
    boost::scoped_ptr sourceImage;
    boost::scoped_ptr filter;
};

TEST_F(ImageFilterTests, TestBlurImageFilter)
{
    // Act
    filter.reset(new BlurImageFilter());
    filter->ProcessToFile(sourceImage.get(), outfileName);

    // Assert
    AssertImagesEqual("expected_lena_blurred.png", outfileName);
}

TEST_F(ImageFilterTests, TestUnblurImageFilter)
{
    // Act
    filter.reset(new UnblurImageFilter());
    filter->ProcessToFile(sourceImage.get(), outfileName);

    // Assert
    AssertImagesEqual("expected_lena_unblurred.png", outfileName);
}

C++ Mocking Framework

Google released Google Test for C++ earlier this year, and just recently released Google C++ Mocking Framework.

These are extensively documented... check them out!

What a Tangled Web We Weave, When Don't Have Effective Tracking

This is a re-post of a blog entry I wrote in 2003...

Steve Norrie points to a recent online article by Jerry Weinberg published on CrossTalk, the Journal of Defense Software Engineering here: Destroying Communication and Control in Software Development.

I'd like to mention a few of the Extreme Programming (XP) solutions to some of the problems mentioned in this article. Be aware that XP is a light-weight process, relying on people rather than technology to do the right thing. (A savvy person can undermine technology anyway.)

Requirements. One of the first areas that communication can be destroyed is by not doing requirements well... not involving the customer, thinking that requirements are a waste of time, and so on. Extreme Programming recommends involving the customer, or a qualified representative of the customer, throughout the entire project. And in addition to talking about the requirements often and in detail, XP requires writing the requirements down in an executable form - automated acceptance tests.

The configuration management system (CMS). The CMS tracks requirements, design, code, test data, test results, user documentation, etc. Jerry notes that information in the CMS can be undermined by failing to keep it up to date, restricting read-access from people who should have access, removing data or failing to put data into it. Extreme Programming doesn't require any specific CMS system (software or otherwise) for this tracking, but does suggest using the simplest thing that actually works. For code, tests, and test data, I recommend using CVS or some other source-code-management system. For acceptance test results, many XP teams record those on a white board or poster board, updated them weekly or daily, charting them over the course of the project. XP does strongly recommend using index cards (story cards) for tracking requirements during initial and weekly planning, and also strongly recommends documenting the relationship between the automatic acceptance tests and the story cards. The person playing the Customer role, as well as the whole team, is responsible for keeping track of the stories. Many XP teams also record the stories in web-accessible ways, such as a wiki.

In regards to a bug-tracking database, some XP teams track bugs the same way as stories - index cards and acceptance tests. You might think that this won't work, but consider that several XP teams have reported that their bug rate after implementing XP dropped from hundreds per six months to around one bug per month. It helps that bugs are not usually recorded until after a story is finished, and in XP, a story is not finished until it is passing its acceptance tests - this requires conversation between the tester and coder as soon as problems are noticed during the implementation and testing of the story. When a bug is recorded, it probably indicates that an acceptance test that was passing has started failing.

Weinberg recommends that you "set and enforce a policy of complete and open information at all times." Agile processes like XP need accurate information daily. Many projects keep tracking information on poster boards and white boards, visible not only to all team members, but anyone in management who walks by. This is the "Project Progress Poster" concept that Weinberg recommends in Quality Software Management: Anticipating Change. Since in my company, few people in management walk by, we keep tracking information in our wiki web pages.

Quality Assurance. Weinberg recommends "Prevent these abuses by having quality assurance report to the highest levels of management, and not to project management." In XP, QA testers are delegates of the person playing the Customer role -- the same person who defines the requirements. QA testers should not only be implementing and running the automated acceptance tests, but also running stress tests and manual testing of the product's user interface.

Weinberg reports that testing often comes too late in the project to be useful in effective risk management. (There's a phrase in XP circles: "Doctor, it hurts when I do this..."). Don't wait until it is too late. XP requires testing to start in the first iteration of the project -- the first week. This and other XP practices enables effective risk management.

The XP solutions noted here require that project management be willing to face reality at all times. One of the quickest routes to failing with XP is to not do XP. If project management destroys information, hides it, degrades it, or inserts misleading inforation, intentionally or not, it is going to be very difficult to have a successful project, no matter what the methodology used.

Science Daily Snips

Hopes for cleaner, safer energy:

Quicker, Easier Way To Make Coal Cleaner "Construction of new coal-fired power plants in the United States is in danger of coming to a standstill, partly due to the high cost of the requirement — whether existing or anticipated — to capture all emissions of carbon dioxide [...] Instead of capturing all of its CO2 emissions, plants could capture a significant fraction of those emissions with less costly changes in plant design and operation [...]"

Making Gases More Transportable "Chemists at the University of Liverpool have developed a way of converting methane gas into a powder form [...] made out of a mixture of silica and water which can soak up large quantities of methane molecules [...] might be easily transported or used as a vehicle fuel."

Agile Methods Help Programmers Shine

Now and then someone says something like "Agile methods won't turn a mediocre programming into a star. That irritates me, because bad non-agile methods can make a very good programmer produce mediocre results. Without taking into account their environment, no one will know how good that "mediocre" programmer can actually be.

A star NASCAR driver probably won't win the race, if his car has a flat tire and his support team can't or won't replace it. It doesn't matter how good the driver is, if the car is bad.

One simple idea - that the "traditional" way of developing an application is by implementing one complete layer at a time (whether bottom-up or top-down) - dooms a project to deliver no working features until late in the development cycle. When feedback finally arrives about the features (wrong requirements and/or wrong implementations), there is often not enough time to fix all the problems, and so the developers look bad. The testers also look bad if they don't have enough time to test, or enough time for fixes to be tested.

Turn that idea around - develop one vertical slice of an application at a time, so each feature can be tested as soon as possible - allows the customers and testers to provide feedback early in the development cycle. That can make a difference in how good the final product can be. Instead of projects running late or shipping with bugs, the project has known good features, can ship when enough features have been done, and the developers look so much better. People might say the project has star programmers.

Many programmers and testers have been trained in that "traditional" style. It's assumed in many books, schools, and corporate cultures. I call it "Unconscious Waterfall" when the style is just assumed. It takes a very remarkable developer to do good work in this environment.

In some shops where they do something they call "Waterfall" and they make it work, it turns out that they verify that many features are working correctly early in the project - there are many places where requirements, tests, and code are examined, verified, validated. Those kinds of waterfall projects are pretty rare, from what I see.

Adopt pets from the Humane Society

I adopted two cats last weekend. A five-month old female and a 3-year old female. Beautiful, curious, energetic, affectionate. I'll post pictures later.

Check the web pages of your local humane society. They probably have cats, dogs, rabbits and birds. If you can, give some of them a loving home.

When a student is ready...

When a student is ready, a teacher will appear. On an agile team, everyone can be a student and teacher. One team member may be more knowledgeable about the unit testing framework, but another may know more about object-oriented design. One team member may be more skilled with the IDE, and another knows the in-and-outs of the mocking framework.

Pair programming is an excellent time for these kinds of knowledge transfer. Code-reviews (particularly those conducted asynchronously over email) are a low-bandwidth medium for teaching and learning. It can happen, but teaching and learning has a lot more room to occur with in-person interactions.

The converse: "When a student is not ready, a teacher will not appear." also happens. People who reject pair programming without giving it a serious try are sometimes afraid to expose their fallibility. I didn't expect to enjoy pair programming, and when I first gave it a try, I was making typos and exposing my fallibility right from the start. And then the person I was pairing with was just as clumsy and fallible. As we continued to pair, we learned not only how fallible we are, but also how smart we are. We get to know we each as human beings. We find out that instead of being a waste of time, we make progress faster than we could if we were programming alone.

It takes courage to pair program. Do it for a while, and you'll find it also provides more safety than programming alone does.

Refactoring for Testable C++ at OOPSLA

I will be co-presenting a workshop on Refactoring Legacy C++ Code to Testable C++ with Joshua Kerievsky at OOPSLA 2008. If you deal with C++ code and want to make it testable, this tutorial is for you.

This tutorial for intermediate-level C++ programmers is based on my own pragmatic experience working with legacy C++, as well as the experience of other Industrial Logic coaches. It is part lecture with slides and video, and part practice on C++ code that we provide.

I gave this workshop with Gil Broza at the Agile 2008 conference, and the class members really appreciated digging into the labs -- to the point that many of them choose to work on the code and tests in one of the labs instead of taking a break.

If you are working with hard-to-test legacy C++ code, and want to get it under test so you can safely refactor it, this tutorial will present several hard-to-test examples of C++ code and show how it can be refactored into testable code. The legacy code problems that may experience have several possible refactorings to get them under test, and we will cover the pros and cons of the most common refactorings and problems.

Looking at Good Object Oriented Design

Looking at good Object Oriented design is like watching an effective manager (or CEO) who knows how to delegate the proper responsibilities to his or her team(s).

jBehave's Golden Rules

I like these "Golden Rules" for contributers to jBehave:

Here are some rules that we follow when developing JBehave:

• Example Driven: Only create code to solve a realistic (and preferably real) problem.
• Behaviour Driven: Only create classes once you can describe and automate examples of their behaviour.
• No Misbehaving: Automated examples must work before being checked into the repository, and existing tests must not be broken.

They have a new release, by the way. With "plain text" scenarios. Check it out.