Sweet Fifteen

What is the right number of tests? Which tester hasn't been asked that question many times in one form or another? When will the testing be done? Can you test to make sure this works? How much effort would it be to test that? Can you show that performance has improved? We need to shorten the run time of the automated tests, can you remove some? How many test cases are passing?

What is the right number of tests? According to Matalan, I found out over the Christmas holiday, the sweet spot appears to be fifteen:

We check our garments at least 15 times to ensure they meet your expectations on quality. 

Fifteen. It'd be easy to scoff, wouldn't it? Testing is never done, testing can never be complete, testing doesn't ensure anything, testing is brainwork, testing is an art, I tell you, it's an art!

Now, don't get me wrong: I love the theory, the philosophy, the abstract. I can be as up my own backside about testing as the next person. (And I am. I cite this blog as evidence.) But I also recognise that we work in a world where rubber, and risk, is constantly hitting the road. We are at the sharp end. The decisions we make under the constraints we have at any given time can matter. We also sometimes need to be able to provide genuine answers to questions like the ones at the top, when they're asked genuinely.

So I don't scoff (these days). I take the jarring statements and questions as an opportunity for a thought experiment. For example: what might be meant by the claim that Matalan are making? What real-world conditions could motivate the need to make such a claim? What kind of evidence could be used to back the claim up, were it ever challenged, and to what extent does that matter?

Which is why, while stalking round the store as my daughters hunted for new jumpers and a DVD to while away a couple of in-law hours that afternoon (after a family vote we ended up with The Pirates! In an Adventure with Scientists!) I found myself asking questions like these:

• is the claim about every type of garment, regardless of its complexity? A sock gets the same attention as a three-piece suit?
• is this a claim about some garments, some types of garment, every instance of a garment?
• what are "our garments"? Those made by Matalan, those sold by Matalan, something else?
• is it the same fifteen tests every time?
• what even is a test of a garment? Are all tests equal? At all stages of manufacture, delivery, display, ...?
• whose expectations are being satisfied?
• who is likely to read this poster, on the outside of a store in a small out-of-town estate?
• where else is the claim being made?
• how is satisfaction being judged?
• what is meant by quality? And how is it measured?
• is the poster addressing a business need? Maybe potential customers are put off by perceptions of low quality?
• is fifteen a marketing number based on data? Maybe in focus groups, people feel more confident with fifteen than fourteen or sixteen?
• is fifteen, or perhaps the wording or phrasing, based on psychological research? Is the advert tuned to achieve its aim?
• could a plausible number really be as low as fifteen, surely hundreds of checks are made during design, prototyping, trials, ...?
• is this advert itself part of some A/B test? Are others seeing a different claim elsewhere?

Yes, yes, you say, very smart and all that, but what exactly does this kind of blathering achieve?

Fair point. For me, it serves as a reminder to stay humble, and also to think outside of the rails on which my thoughts might naturally run. There could be a bunch of reasons, a stack of context, assumptions, data, and belief behind a statement. Just because it doesn't fit you or me, given what we know, in the milliseconds it can take to form a reaction, doesn't mean it doesn't have a justification. There can be both sport and learning in taking the time to consider that.

I don't always remember, of course, and when I catch myself failing I think of Bob Marshall's definition of an idiot:

Anyone who is just trying to meet their needs, in the best way they know how, where their way makes little or no sense to us.

which I interpret as a call for "empathy first."

So, if you managed to endure what has turned out to be essentially a stream of consciousness this far down the page, and are right now wondering why on earth I bothered, let me just say there's at least fifteen reasons ...