Testing For Me

I spoke at UKSTAR 2018 this week, an eight-minute talk in a Storytelling track. This post is a prettied-up version of the notes I made for it along with some of the slides. The full slide deck is in the Media page.

My story is called The Anatomy of a Definition of Testing. It's not a suspense story though, so I'll give you the definition right up front:

Testing is the pursuit of relevant incongruity.

That is, for me, testing is the pursuit of relevant incongruity. But how did I get there?

Well, the journey started with Explore It! by Elisabeth Hendrickson, a great book about exploratory testing which has, near the beginning, this definition:

Tested = Checked + Explored

It's finessed a little by

Neither checking nor exploring is sufficient on its own

and the idea that testing is to

... interact with the software or system, observe its actual behavior, and compare that to your expectations.

Interestingly, the definition doesn't really play a significant part in the rest of the book, but as I was reading I kept on returning to it, asking myself questions like these:

Eventually I thought I'd ask Elisabeth whether she could help me out (she said she would!) and we exchanged a few emails, and that helped me to clarify my thoughts. But during the conversation, I began to wonder was this process, of thinking about testing, itself testing? I mean, I was doing things that were consistent with things I do when testing:

But was I really testing? By Elisabeth's definition, I wasn't sure that I could say I was. But it felt a lot like testing. So I went looking for other definitions and found loads!

And I recognise aspects of all of those in my testing, but none of them capture all that testing is for me. Reflecting further, I remembered a talk that Rikard Edgren gave at EuroSTAR 2015 where he said this beautiful thing:

Testing is simple: you understand what is important and then you test it.

Adam Knight has talked and written about Fractal Exploratory testing, and describes it like this:

as each flaw ... is discovered ... [a] mini exploration will result in a more targeted testing exploration around this feature area

To me, they're both talking about how investigation of one area leads to investigation in a subset of that area, and in a subset of that area. A kind of traversal through the testing space where the actions that are performed at any level are similar. And I recognise that too, but it's not all that testing is. For me.

I tried to draw my testing on a feature. It looked like this.

Sometimes multiple activities feed into another. Sometimes one activity feeds into multiple others. Activities can run in parallel, overlap, be serial. A single activity can have multiple intended or accidental outcomes ...

I tried to draw it again. It looked like this:

A vision of an attempt to somehow keep a lid on the ambiguity, and the unknowns, and the complexity in order to be able to get on and test.

A colleague pointed me at a drawing by John Bach of his testing.

That scribble on the left is not necessarily confusion and chaos, but cycling, cross-checking, confirming, until a course of action that seems propitious can be identified and followed out to the right with occasions when exploration goes deep down. And, naturally, I recognise that in my testing too. But it isn't all that testing is for me.

So, without a definition but with a lot of thoughts about what I wanted from my definition I tried to list factors of testing and then come up with a definition that covers them all.

And I thought about it for a good long time. I mean, really long. And eventually out popped my definition:

Testing is the pursuit of relevant incongruity.

But that's a bit of a mouthful. Let's unpack it a little:

Incongruity: Oxford Dictionaries define this as "not in harmony or keeping with the surroundings". I interpret lack of harmony as a potential problem and lack of keeping as an actual problem, and those two interpretations are interesting and useful in testing.

Pursuit: Again, there are two senses that capture important aspects of testing for me. You can pursue something that you don't know is there and that you may never find, like a dream. Or you can pursue the solution to a problem that you know you have, that's right in front of you.

Why not some other verb? For me, an investigation has identified something to investigate, finding requires something to be found and I want to be able to say I tested even if I found no issues, exploration can work but I don't want my definition to be thought of as a definition of exploratory testing, much as I love it.

Relevant: if this work doesn't matter to anyone, why are we doing it? Whoever that is can help us to understand whether any incongruities we identify are valuable to them, relevant to the project.

So that's my definition:

Testing is the pursuit of relevant incongruity.

Notice that it says nothing about particular techniques or methods, or products, or systems. It exists, deliberately, in a space where it must be applied in context, at a time, for a purpose.

But what do I do with it?

Well, any time I'm working I can ask myself whether what I'm doing is contributing the pursuit of relevant incongruity. If it is, I'm testing — I still have the question of whether I'm testing the right things, at the right time but that's a different problem for another story.

If I'm not in pursuit of relevant incongruity I can ask whether I should be, and why. Sometimes it's legit to take off your tester hat and do something else, project housekeeping for example, because it needs doing, because you've got the skills, because you can be most efficient, because it's your turn, or whatever. But sometimes it can provoke me into thinking that I'm not doing what I should be.

Which is great, and I buzzed along very nicely using it. And then I heard Michael Bolton say this on the Quality Remarks podcast:

The goal of testing is identifying problems that matter

And I thought "Hello! That feels pretty familiar!" Although "problems" loses the subtlety of "incongruity", and "finding" I already said I have some reservations about, and note that he's talking about the "goal" of testing, not testing itself. But still, there's a similar sentiment here, and look how much snappier it is than mine!

So I asked him about it, and he said "many things stay in the mind more easily when they can be expressed concisely and snappily."

Which is true to my experience, and also very useful, because it emphasises his different need. He's a teacher, and he wants to share his description and have his pupils remember it. And that's OK, it keeps me humble: I shouldn't impose my view of testing on others, and other views can be just as valid.

And so that's my definition:

Testing is the pursuit of relevant incongruity.

It's my definition to help me do what I need to do in my context. It's for me. But I hope it was of some interest to you...
Image: Jit Gosai (via Twitter)