Step on It

In recent times I've spoken and written about how much fun and how productive it's been to build random walkers to help me to test services I've been working on: Walking the Talk webinar, A Model Student, and Navigate, Survey, and Explore.

The walkers are clients which use dice rolls to make decisions as they navigate paths through a service, asserting things about the state as they go.

Traditional unit tests tend to be extremely specific. They imagine the system in a particular state with a hard-coded input and an expected output. 

In my walkers, the assertions are relatively generic, for example:

• a service response payload conforms to a schema
• the value of a field representing progress will not decrease across a series of interactions
  
• some values in service responses must be in a particular relationship to others, or to the input.
  

I doubt this is a technically-correct use of the term but I've been thinking of these things as invariants, statements that are true in broad, defined, contexts, or perhaps even across the whole system under test.

The approach has a strong link with property-based testing which generalises unit tests by specifying properties of the inputs (e.g. integers) and the consequent outputs (e.g. a naturally-sorted list). 

From the Hypothesis library documentation, the testing "works by generating arbitrary data matching [the] specification and checking that [the output] guarantee [...] holds." 

It's also in a tradition of model-based testing where a model (informally, a kind of flowchart describing the system under test) is traversed and correctness checks are made along the way. As Harry Robinson puts it in Finite State Model-Based Testing on a Shoestring:

1. Create a finite state model of an application.
2. Generate sequences of test actions from the model.
3. Execute the test actions against the application.
4. Determine if the application worked right.
5. Find bugs.
   

Believe me, this stuff is exciting and powerful but in a dry and abstract description like I've given can sound extremely, well, dry and abstract.

I gave a quick demo and presentation about a walker at work last week. I wanted something that visualised why it was exciting and powerful and practical and how it complements what we already do.

This is what I came up with:

The background light green blob is a minefield. It represents the space of possible states of our service. In it, somewhere, are mines, or bugs. We don't know all of their locations and, in any case, as we develop the software the locations change.

One way, reductive for sure, to think about testing is that it seeks to find mines by stamping on the minefield. An interesting aspect of that game (safety considerations aside!) is to find valuable places and productive ways to stamp.

Unit tests are the tiny circles. They stamp repeatedly on the same bit of the minefield.

The unit test might be in that place because, for example, we care to check that a specific input gives a specific output, or because a bug was found there in the past, or that it's considered to be a representative place to check for mines in the surrounding area (the mid-green clouds). 

We might call these surrounding areas equivalence classes.

Exploratory testing, particularly when driven by risk and the current context, will regularly march around other parts of the field, pursuing paths that look promising tactically and strategically, and sometimes overlapping the coverage of existing automation.

By default, the walker doesn't care about trying to focus on areas with previous bugs or areas of risk or equivalence classes or anything else. It just puts on its heaviest boots and runs around the field. (I like to think it's also waving its hands in the air, screaming with pure pleasure, and grinning like a maniac.)

These approaches are complementary and, can be combined. 

I use walkers as a tool in my exploratory testing. I either hack or configure them to bias to particular areas, or navigation tactics, or to collecting particular kinds of data that I can explore later. 

I can take assertions from unit tests and generalise them for the walkers, and I can take issues found by the walker and add unit tests for them, or change existing unit tests to better reflect a new understanding of the service.

I'm pleased to say that the analogy with the minefield was helpful but, much like the walkers themselves, it's not my idea nor novel. Looking back, I think I first came across it in either Rapid Software Testing or possibly Black Box Software Testing courses over ten years ago. 

A little cursory searching turns up Cem Kaner talking about it in 2002's Paradigms of
Black Box Software Testing but with the possibility that the idea goes back at least to 1994:

But both he and James Bach in Reasons to Repeat Tests credit Brian Marick as the originator. Bach mentions being inspired by a talk of Marick's called Classic Testing Mistakes I didn't find the talk, but the paper of the same title is itself a classic.

Why does this matter? Well, I'm thinking this stuff in theory and practice has been around for ages but is little known or used in our industry. How can we speed up its adoption?
Image: https://flic.kr/p/5TCvmi

Edit: Paul Hankin noted on Twitter: "Fuzzing is an analogous testing idea -- and the cleverest fuzzers understand the code they are testing and try to find paths that exercise so-far-untested parts of the code (rather than just try random inputs)."

Edit 2: I contacted both Brian Marick and James Bach. Brian doesn't recall originating the minefield analogy; James feels certain that Brian mentioned it in his talk. 

Edit 3: I found a reference on Marick's web site to the Classic Testing Mistakes talk being delivered at STAR 97.