Skip to main content

Testing For Me




I spoke at UKSTAR 2018 this week, an eight-minute talk in a Storytelling track. This post is a prettied-up version of the notes I made for it along with some of the slides. The full slide deck is in the Media page.

My story is called The Anatomy of a Definition of Testing. It's not a suspense story though, so I'll give you the definition right up front:
Testing is the pursuit of relevant incongruity.
That is, for me, testing is the pursuit of relevant incongruity. But how did I get there?

Well, the journey started with Explore It! by Elisabeth Hendrickson, a great book about exploratory testing which has, near the beginning, this definition:
Tested = Checked + Explored
It's finessed a little by
Neither checking nor exploring is sufficient on its own
and the idea that testing is to
... interact with the software or system, observe its actual behavior, and compare that to your expectations.
Interestingly, the definition doesn't really play a significant part in the rest of the book, but as I was reading I kept on returning to it, asking myself questions like these:


Eventually I thought I'd ask Elisabeth whether she could help me out (she said she would!) and we exchanged a few emails, and that helped me to clarify my thoughts. But during the conversation, I began to wonder was this process, of thinking about testing, itself testing? I mean, I was doing things that were consistent with things I do when testing:


But was I really testing? By Elisabeth's definition, I wasn't sure that I could say I was. But it felt a lot like testing. So I went looking for other definitions and found loads!


And I recognise aspects of all of those in my testing, but none of them capture all that testing is for me. Reflecting further, I remembered a talk that Rikard Edgren gave at EuroSTAR 2015 where he said this beautiful thing:
Testing is simple: you understand what is important and then you test it.
Adam Knight has talked and written about Fractal Exploratory testing, and describes it like this:
as each flaw ... is discovered ... [a] mini exploration will result in a more targeted testing exploration around this feature area
To me, they're both talking about how investigation of one area leads to investigation in a subset of that area, and in a subset of that area. A kind of traversal through the testing space where the actions that are performed at any level are similar. And I recognise that too, but it's not all that testing is. For me.

I tried to draw my testing on a feature. It looked like this.


Sometimes multiple activities feed into another. Sometimes one activity feeds into multiple others. Activities can run in parallel, overlap, be serial. A single activity can have multiple intended or accidental outcomes ...

I tried to draw it again. It looked like this:


A vision of an attempt to somehow keep a lid on the ambiguity, and the unknowns, and the complexity in order to be able to get on and test.

A colleague pointed me at a drawing by John Bach of his testing.


That scribble on the left is not necessarily confusion and chaos, but cycling, cross-checking, confirming, until a course of action that seems propitious can be identified and followed out to the right with occasions when exploration goes deep down. And, naturally, I recognise that in my testing too. But it isn't all that testing is for me.

So, without a definition but with a lot of thoughts about what I wanted from my definition I tried to list factors of testing and then come up with a definition that covers them all.


And I thought about it for a good long time. I mean, really long. And eventually out popped my definition:
Testing is the pursuit of relevant incongruity.
But that's a bit of a mouthful. Let's unpack it a little:


Incongruity: Oxford Dictionaries define this as "not in harmony or keeping with the surroundings". I interpret lack of harmony as a potential problem and lack of keeping as an actual problem, and those two interpretations are interesting and useful in testing.


Pursuit: Again, there are two senses that capture important aspects of testing for me. You can pursue something that you don't know is there and that you may never find, like a dream. Or you can pursue the solution to a problem that you know you have, that's right in front of you.

Why not some other verb? For me, an investigation has identified something to investigate, finding requires something to be found and I want to be able to say I tested even if I found no issues, exploration can work but I don't want my definition to be thought of as a definition of exploratory testing, much as I love it.


Relevant: if this work doesn't matter to anyone, why are we doing it? Whoever that is can help us to understand whether any incongruities we identify are valuable to them, relevant to the project.

So that's my definition:
Testing is the pursuit of relevant incongruity.
Notice that it says nothing about particular techniques or methods, or products, or systems. It exists, deliberately, in a space where it must be applied in context, at a time, for a purpose.

But what do I do with it?

Well, any time I'm working I can ask myself whether what I'm doing is contributing the pursuit of relevant incongruity. If it is, I'm testing — I still have the question of whether I'm testing the right things, at the right time but that's a different problem for another story.

If I'm not in pursuit of relevant incongruity I can ask whether I should be, and why. Sometimes it's legit to take off your tester hat and do something else, project housekeeping for example, because it needs doing, because you've got the skills, because you can be most efficient, because it's your turn, or whatever. But sometimes it can provoke me into thinking that I'm not doing what I should be.

Which is great, and I buzzed along very nicely using it. And then I heard Michael Bolton say this on the Quality Remarks podcast:
The goal of testing is identifying problems that matter
And I thought "Hello! That feels pretty familiar!" Although "problems" loses the subtlety of "incongruity", and "finding" I already said I have some reservations about, and note that he's talking about the "goal" of testing, not testing itself. But still, there's a similar sentiment here, and look how much snappier it is than mine!

So I asked him about it, and he said "many things stay in the mind more easily when they can be expressed concisely and snappily."

Which is true to my experience, and also very useful, because it emphasises his different need. He's a teacher, and he wants to share his description and have his pupils remember it. And that's OK, it keeps me humble: I shouldn't impose my view of testing on others, and other views can be just as valid.

And so that's my definition:
Testing is the pursuit of relevant incongruity.
It's my definition to help me do what I need to do in my context. It's for me. But I hope it was of some interest to you...
Image: Jit Gosai (via Twitter)

Comments

Popular posts from this blog

Can Code, Can't Code, Is Useful

The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "If testers can’t code, they’re of no use to us" My first reaction is to wonder what you expect from your testers. I am immediately interested in your working context and the way

Meet Me Halfway?

  The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "Stop answering my questions with questions." Sure, I can do that. In return, please stop asking me questions so open to interpretation that any answer would be almost meaningless and certa

Testing (AI) is Testing

Last November I gave a talk, Random Exploration of a Chatbot API , at the BCS Testing, Diversity, AI Conference .  It was a nice surprise afterwards to be offered a book from their catalogue and I chose Artificial Intelligence and Software Testing by Rex Black, James Davenport, Joanna Olszewska, Jeremias Rößler, Adam Leon Smith, and Jonathon Wright.  This week, on a couple of train journeys around East Anglia, I read it and made sketchnotes. As someone not deeply into this field, but who has been experimenting with AI as a testing tool at work, I found the landscape view provided by the book interesting, particularly the lists: of challenges in testing AI, of approaches to testing AI, and of quality aspects to consider when evaluating AI.  Despite the hype around the area right now there's much that any competent tester will be familiar with, and skills that translate directly. Where there's likely to be novelty is in the technology, and the technical domain, and the effect of

Testers are Gate-Crashers

  The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "Testers are the gatekeepers of quality" Instinctively I don't like the sound of that, but I wonder what you mean by it. Perhaps one or more of these? Testers set the quality sta

Postman Curlections

My team has been building a new service over the last few months. Until recently all the data it needs has been ingested at startup and our focus has been on the logic that processes the data, architecture, and infrastructure. This week we introduced a couple of new endpoints that enable the creation (through an HTTP POST) and update (PUT) of the fundamental data type (we call it a definition ) that the service operates on. I picked up the task of smoke testing the first implementations. I started out by asking the system under test to show me what it can do by using Postman to submit requests and inspecting the results. It was the kinds of things you'd imagine, including: submit some definitions (of various structure, size, intent, name, identifiers, etc) resubmit the same definitions (identical, sharing keys, with variations, etc) retrieve the submitted definitions (using whatever endpoints exist to show some view of them) compare definitions I submitted fro

Build Quality

  The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "When the build is green, the product is of sufficient quality to release" An interesting take, and one I wouldn't agree with in general. That surprises you? Well, ho

Make, Fix, and Test

A few weeks ago, in A Good Tester is All Over the Place , Joep Schuurkes described a model of testing work based on three axes: do testing yourself or support testing by others be embedded in a team or be part of a separate team do your job or improve the system It resonated with me and the other testers I shared it with at work, and it resurfaced in my mind while I was reflecting on some of the tasks I've picked up recently and what they have involved, at least in the way I've chosen to address them. Here's three examples: Documentation Generation We have an internal tool that generates documentation in Confluence by extracting and combining images and text from a handful of sources. Although useful, it ran very slowly or not at all so one of the developers performed major surgery on it. Up to that point, I had never taken much interest in the tool and I could have safely ignored this piece of work too because it would have been tested by

Am I Wrong?

I happened across Exploratory Testing: Why Is It Not Ideal for Agile Projects? by Vitaly Prus this week and I was triggered. But why? I took a few minutes to think that through. Partly, I guess, I feel directly challenged. I work on an agile project (by the definition in the article) and I would say that I use exclusively exploratory testing. Naturally, I like to think I'm doing a good job. Am I wrong? After calming down, and re-reading the article a couple of times, I don't think so. 😸 From the start, even the title makes me tense. The ideal solution is a perfect solution, the best solution. My context-driven instincts are reluctant to accept the premise, and I wonder what the author thinks is an ideal solution for an agile project, or any project. I notice also that I slid so easily from "an approach is not ideal" into "I am not doing a good job" and, in retrospect, that makes me smile. It doesn't do any harm to be reminded that your cognitive bias

Test Now

The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "When is the best time to test?" Twenty posts in , I hope you're not expecting an answer without nuance? You are? Well, I'll do my best. For me, the best time to test is when there

Vanilla Flavour Testing

I have been pairing with a new developer colleague recently. In our last session he asked me "is this normal testing?" saying that he'd never seen anything like it anywhere else that he'd worked. We finished the task we were on and then chatted about his question for a few minutes. This is a short summary of what I said. I would describe myself as context-driven . I don't take the same approach to testing every time, except in a meta way. I try to understand the important questions, who they are important to, and what the constraints on the work are. With that knowledge I look for productive, pragmatic, ways to explore whatever we're looking at to uncover valuable information or find a way to move on. I write test notes as I work in a format that I have found to be useful to me, colleagues, and stakeholders. For me, the notes should clearly state the mission and give a tl;dr summary of the findings and I like them to be public while I'm working not just w