Means Testing

I've found it interesting to read the recent flurry of thought on the testing/checking question, thoughts referring back to Testing and Checking Refined, by James Bach and Michael Bolton, in which the following definitions for the terms are offered:

• Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modelling, observation, inference, etc.
• Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

The definitions are accompanied by a glossary which attempts to clarify some of the other terms used. These chaps really do sweat the semantics but if I could be so presumptuous as to gloss the distinction I might say: testing is an activity which requires thought; checking is simply a rigid comparison of expectation to observation.

There has been much misunderstanding about the relationship between the two terms, not least that they are often seen in opposition to one another while at the same time testing is said to include checking, as in Why the testing/checking debate is so messy – a fruit salad analogy,  where Joep Schuurkes laments:

What we’re left with are only two concepts, "testing" and "checking", and the non-checking part of testing is gone.

In Exploring, Testing, Checking, and the mental model, Patrick Prill proposes that

When a human is performing a check, she is able to evaluate many assertions, that often are not encoded in the explicit check.

For me, by the definitions above, this means that the human is not simply performing a check. The check is narrow while the human perspective can be broad (and deep, and lateral, ...) which, for Bolton and Bach, as I interpret it, means that this is testing. I think that this is also what Anders Dinsen is getting at in Why the dichotomy of testing versus checking is the core of our craft:

As a tester, I carry out checks when I test, but when I do, the checks I am doing are elements in the testing and the whole activity is testing, not checking.

One thing that I find missing from the conversation that I've seen and heard is the notion of intent. I'm wondering whether it's useful to think about some action as a check or test depending on the way that the result is expected to be, or actually, used in the context of the person who is doing the interpretation.

Here's an example: in Migrate Idea I talked about some scripts that I designed to help me to gain confidence in the migration and upgrade of a service across multiple servers. The scripts consisted of a series of API requests followed by assertions, each of which had a very narrow scope. Essentially they were looking to see that some particular question gave particular answers against the pre- and post-migrated service on a particular server.

At face value, they appear to be checks, and indeed I used them like that when evaluating the migration server-by-server. After putting in the (testing) effort to craft the code, I expended little effort interpreting the results beyond accepting the result they gave (all clear vs some problem).

However, by aggregating the results across servers, and across runs, I found additional value, and this is something that I would happily call testing by the definitions at the top. So perhaps these scripts are not intrinsically a check or a test; they simply gather data.

Could it be that what is done with that data can help to classify the actions as (part of) a check or test? At different times, I conjecture that the same instance of some action can be checking, testing, both or neither.  If I don't exercise thought, that (at best) means checking. If I do, that means testing.
Image: https://flic.kr/p/5w6L2a