Wednesday, July 28, 2021

Mass Testing

The Association for Software Testing is crowd-sourcing a book, Navigating the World as a Context-Driven Tester, which aims to provide responses to common questions and statements about testing from a context-driven perspective.

It's being edited by Lee Hawkins who is posing questions on TwitterLinkedInSlack, and the AST mailing list and then collating the replies, focusing on practice over theory.

I've decided to contribute by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be.

Perhaps you'd like to join me?


"Do more test cases mean better test coverage?"

Simply, no. Less simply, depending on the assumptions you care to make, perhaps.

The terms test case and test coverage are loaded, so let's talk about a somewhat analogous problem:

Does more bulletproof glass mean the Popemobile is better protected?

I find it helpful to turn this kind of yes/no question into an exploration: 

Under what circumstances could more bulletproof glass mean the Popemobile is better protected?

This helps me to think of challenges, caveats, and clarifications. To give a few examples:

  • Better protected than what?
  • Better protected from what?
  • Better protected, judged how?
  • Is there any existing bulletproof glass?
  • Where is the existing bulletproof glass?
  • How protective is the existing glass?
  • ... against what kinds of projectiles?
  • ... propelled how?
  • What other kinds of bulletproof glass are available and how protective are they?
  • Where would it be possible to put additional bulletproof glass if we had some?
  • ... and where is the Popemobile susceptible to damage from bullets?
  • ... and which areas at risk are not covered?
  • What threats are we trying to protect the Pope from?
  • Would any amount of bulletproof glass protect the Pope from those threats?
  • Are there any ways other than glass to mitigate the risks of those threats?

Then there are other questions, ones that matter in general because we don't have infinite resources. For example:

  • Does the Pope need better protection?
  • If so, do we need to change the Popemobile to achieve it?
  • At what cost?
  • At what opportunity cost?

So, could I think of circumstances in which more test cases mean better test coverage? Yes. Do those circumstances hold generally? Not a chance in Hell.
Image: The Independent

No comments:

Post a Comment