Saturday, December 11, 2021

Testing is not the Goal


The Association for Software Testing is crowd-sourcing a book, Navigating the World as a Context-Driven Tester, which aims to provide responses to common questions and statements about testing from a context-driven perspective.

It's being edited by Lee Hawkins who is posing questions on TwitterLinkedInSlack, and the AST mailing list and then collating the replies, focusing on practice over theory.

I've decided to contribute by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be.

Perhaps you'd like to join me?


“Testing is a bottleneck”

I'd really hope it isn't, but let's see if we can work out what's happening and how we could improve the situation.

First, what are you seeing that makes you think testing is a bottleneck? When I've heard that complaint in the past it's usually been because stakeholders wanted "finished" features in the field immediately and perceived that "testing" was preventing that.

In a world where testing occurs after development and before deployment there are, crudely, three possibilities:

Product is ready but no testing is happening. One cause of this scenario is lack of people available to do testing. Another is that testing can't start before some other dependency is resolved, perhaps access to an environment. A third is that some testing has happened and it found issues deemed significant enough for fixing. These could be scheduling, resourcing, and quality issues rather than evidence of a bottleneck.

Product is ready and testing is happening and it's the right testing. If the right testing is happening then we're good. We can move swiftly on to ...

Product is ready and testing is happening but it's the wrong testing. If we've already identified that the testing is wrong then some kind of assessment must have happened and a difference between what we want and what's being done has been observed.

Assuming we can trust that analysis, let's reassess and change what we're doing! Maybe the risk profile has shifted and we can do less, or different, or even no testing at all. Testing itself is not the goal.

Maybe the time available has changed and it's not possible to add more. Of course, this doesn't necessarily mean that testing is taking too long. It could be that product was released for testing late. That does happen sometimes, I hear.

Alternatively, it's possible that the right testing is being done to only some of the features that are ready for testing: there is more stuff to test than resources to test it. This, of all the potential scenarios we've discussed, is closest to the classic bottleneck: a component that is limiting the throughput of a system.

Fortunately, approaches such as the Theory of Constraints provide ways to analyse and remediate just this kind of situation. Possible actions include staggering delivery, increasing capacity, or removing the need for the resource that is overwhelmed.

So, is testing the bottleneck? Well, it might be, but not deploying on the date a stakeholder would prefer isn't enough evidence to make that claim convincingly.


No comments:

Post a Comment