The Anatomy of a Definition of Testing

At CEWT 3 I offered a definition of testing up for discussion. This is it:

Testing is the pursuit of actual or potential incongruity

As I said there, I was trying to capture something of the openness, the expansiveness of what testing is for me: there is no specific technique; it is not limited to the software; it doesn't have to be linear; there don't need to be requirements or expectations; the same actions can contribute to multiple paths of investigation at the same time; it can apply at many levels and those levels can be distinct or overlapping in space and time.
 
And these are a selection of the comments and questions that it prompted before, during and after the event, loosely grouped:

Helicopter view

• it is sufficiently open that people could buy into it, and read into it, particularly non-testers.
• it's accurate and to the point.
• it has the feel of Weinberg's definition of a problem. 
• it sounds profound but I'm not sure whether there is any depth.
• it seems very close to the regular notion of targeting information/unknowns.

Coverage

• can not testing be part of this idea of testing?
• how does the notion of tacit testing (from CEWT 3 discussion) fit in?
• Kaner talks about balancing freedom and responsibility in testing. Is that covered here?
• the definition doesn't talk about risk.

Practical utility

• it couldn't be used to help someone new to testing decide what to do when testing.
• I could imagine putting this onto a sticky and trying to align my actions with it.

Definitional

• what do you mean by pursuit? 
• incongruity is too complex a word.
• what other words could replace testing in the definition and it still hold?
• when I see or I wonder about whether it's exclusive (in the Boolean sense).

In this post I'm going to talk about just the words. I spent a deal of time choosing my words - and that in itself is a warning sign. If I have to graft to find words whose senses are subtly tuned to achieve just the interpretation that I want, then I worry that others will easily have a different interpretation.

And, despite this being a definition of testing for me, it's interesting to observe how often I appeal to my feelings and desires in the description below. Could the degree of personal investment compromise the possibility of it having general appeal or utility, I wonder.

"pursuit"

Other definitions use words like search, explore, evaluate, investigate, find out, ... I was particularly keen to find a verb that captured two aspects of testing for me: finding out what is there, and digging into what has been found.

What I like about pursuit is that it permits (at least to me) both, and additionally conveys a sense of chasing something which might be elusive, itinerant, latent or otherwise hard to find. Oxford Dictionaries has these definitions, amongst others of pursue:

• follow or chase (someone or something)
• continue to investigate or explore (an idea or argument)

These map onto my two needs in ways that other verbs don't:

• search: feels more about the former and less about the latter.
• investigate: feels more natual when there's a thing to investigate.
• explore: could possibly do duty for me (and it's popular in testing definitions) but exploratory testing can be perceived as cutting out other kinds of testing and I don't want that interpretation.
• evaluate: needs data; pursuit can gather data.
• find out: feels like it has finality in it. To reflect the fact that testing is unlikely to be complete I'd want to say something like "Testing is the attempt to find out about actual or potential incongruity"

"incongruity"

As one of the criticisms of my definition points out, this word is not part of most people's standard lexicon. Oxford Dictionaries says that it means this:

 Not in harmony or keeping with the surroundings or other aspects of something.

I like it because it permits nuance in the degree to which something needs to be out of place: it could be completely wrong, or just feel a bit odd in its context. But the price I pay for the nuance is the lack of common currency. On balance I accepted this in order to keep the definition short.

"actual or potential"

I felt unhappy with a definition that didn't include this, such as:

Testing is the pursuit of incongruity

because I wanted testing's possibilities to include suggesting that there might be a problem. If the definition of incongruity I am using permitted possible disharmony then I'd have been happier with this shorter variant.

I have subsequently realised that I am, to some extent, reflecting a testing/checking distinction here too: a check with fixed expectations can find actual incongruity while testing could in addition find potential incongruity.

However, the entire definition is, for me, in the context of the relative rule - so any incongruities of any kind are tied to a context, person, time - and also the need to govern the actions in the pursuit by some notions of what is important to the people who are important to whatever is being tested.

But, even given that, I still find it hard to accept the definition without potential. Perhaps because it flags the lack of certainty inherent in much testing.
Image: https://flic.kr/p/6Hkgyy

Edit: Olekssii Burdin wrote his own definition of testing after reading this, and Harnessed Tester offers his here too.