The Tester as Engineer?

Much of Definition of The Engineering Method by Billy Vaughn Koen chimes with what I've come to believe about testing. 

In part, I think, this is because thinkers who have influenced my thinking were themselves influenced by Koen's thoughts. In part, also, it's because some of my self-learned experience of testing is prefigured in the article. In part, finally, it's because I'm reading it through biased lenses, wanting to find positive analogy to something I care about. 

I recognise that this last one is dangerous. As Richard Feynman said in Surely You're Joking, Mr. Feynman!: "I could find a way of making up an analog with any subject ... I don’t consider such analogs meaningful.” 

This is a short series of posts which will take an aspect of Definition of The Engineering Method that I found interesting and explore why, taking care not to over-analogise.

• Getting Myself Koen
• Sota so Good
• Meta is Better
• The Tester as Engineer?

--00--

In this series so far I've pulled out a couple of Koen's key concepts for attention: sotas and the Rule of Engineering. I find them both aesthetically pleasing and with practical applications. However, they are cast explicitly for engineers and I'm a tester. I wonder whether, by Koen's intention, they'd apply to me? Are testers engineers? Does testing overlap with engineering? If so, where? If not, why not?

The definition of an engineering problem and its derived definition of an engineer might help to judge the answer (p. 42-3):

If you desire change; if this change is to be the best available; if the situation is complex and poorly understood; and if the solution is constrained by limited resources, then you too are in the presence of an engineering problem ... If you cause this change by using the heuristics that you think represent the best available, then you too are an engineer ... the engineer is defined by a heuristic — all engineering is heuristic.

Let's take each of the criteria in turn:

• change: it's the remit of testers to cause change, in the information state of a project if not directly in any deliverable.
• best available: in Koen's world, "best" is conditional on context and the participants. It doesn't mean objectively maximal. So I intepret this as doing the perceived most important things in an attempt to uncover the most important information.
• complex and poorly understood: looked at from an appropriate level of granularity, pretty much everything is complex and contains unknowns.
• limited resources: there was never a project where the manager said "take all the time you like testing this, I don't care when it ships".
• use heuristics: I would like to think that testers (consciously) use heuristics in their work.

I'm uncomfortable aligning testing and engineering by this route. If I was prepared to say that an activity is only testing when a problem is complex and poorly understood then I could define testers as people who take on complex and poorly understood problems. Unfortunately, I don't agree with the premise: I think it's possible to test something that is not complex, and I think it's possible to test something that's well understood (to whatever degree is relevant in context). In those circumstances, though, I'd suggest that the chances of provoking a change in the information state is likely to be reduced.

Is Koen saying that engineers can't work on trivial things? Or perhaps that they are not doing engineering when they do?

There's a long-running debate in the testing world about whether testing is a role or a job title. I've mused on it myself over the years and concluded that activities we might agree are testing are not the sole remit of people we might call testers. From #GoTesting:

To get to the desired (bigger picture) quality involves asking the (bigger picture) questions; that is, testing the customer's assumptions, testing the scope of the intended solution - you can think of many others - and indeed testing the need for any (small picture) testing, on this project, at this time.

Whether this is done by someone designated as a tester or not, it is done by a human and, as Rands said this week, I believe these are humans you want in the building. #GoTesting

You can play this the other way too: not everything someone with the role title tester does is necessarily what we might call testing.

I spent some time wondering what to make of this paragraph (p. 51):

We have noted that the engineer is different from other people ... The engineer is more inclined to give an answer when asked and to attempt to solve problems that are [non-trivial, but seem practically possible] ... The engineer is also generally optimistic ... and willing to contribute to a small part of a large project as a team member and receive only anonymous glory.

Although he's careful to caveat most of these attributes ("more inclined", "generally") I am allergic to all-encompassing assertions. With respect to testing, I wrote about it in You Shouldn't be a Tester If ...:

A belief that you should conform to a list of context-free statements about what a tester must be would concern me. I'd ask whether you really have testerly tendencies if you prefer that idea to a pragmatic attitude, to doing the best thing you can think of, for the task in hand, under the constraints that exist at that point.

This, to me, is closely allied with Koen's idea of what engineering is and only serves to enhance the dissonance I feel with his assertions about what an engineer is.

Koen does make role comparisons in his article, in particular the engineer and the scientist. He is not keen on the idea of engineering as applied science, apparently wanting instead to regard science as a tool within engineering (p. 63):

Misunderstanding the art of engineering, [some people] become mesmerised by the admittedly extensive use made of science by engineers and ... identify [science] with engineering [but] the engineer recognizes both science and its use as heuristics.

Tellingly, I don't recall him permitting scientists to use what he might call engineering methods. To me, it is simply not the case that all science proceeds by induction, hypothesis generation, and comparison to some natural state of affairs.

There's a sweet definition of a heuristic "in its purest form" (p. 48) that I thought might be relevant:

it does not guarantee an answer, it competes with other possible values, it reduces the effort needed to obtain a satisfactory answer to a problem and it depends on time and context for its choice.

Scientists conduct thought experiments. What are they if not heuristic by this definition? In fact, what are any experiments if not heuristic — hundreds of factors in any experimental setup and methodology could, unknowingly, invalidate the result. One the points of pride for committed scientists is that their findings, though valuable for a time, are likely to be shown wrong in some respect by a later scientist.

Koen also compares engineering with systems thinking and notes the crucial role of feedback (p. 56):

The success or failure of the engineer's effort is fed back to modify the heuristics in the engineer's sota

This seems natural to how I want to view testing. I like the idea of sotas and I really like the idea of overlapping and shared sotas in a given environment. On a project, for example, as we learn more about how the system under development behaves we modify our expectations of it and the way we engage with it. But we also take actions that we desire will provoke other changes. The sotas evolve based on feedback.

A few years ago I had a deep and wide-ranging landslide rush of a conversation with Anders Dinsen that we documented in What We Found Not Looking For Bugs. In trying to characterise what testing does in the abstract, I wrote:

• Some testing, t, has been performed
• Before t there was an information state i
• After t there is an information state j
• It is never the case that i is equal to j (or, perhaps, if i is equal to j then t was not testing)
• It is not the case that only t can provide a change from i to j. For example, other simultaneous work on the system under test may contribute to a shared information state.
• The aim of testing is that j is a better state than i for the relevant people to use as the basis for decision making

... I might propose [an information state is] something like a set of assertions about the state of the world that is relevant to the system under test, with associated confidence scores. I might argue that much of it is tacitly understood by the participants in testing and the consumption of test results. I might argue that there is the potential for different participants to have different views of it - it is a model, after all. I might argue that it is part of the dialogue between the participants to get a mutual understanding of the parts of j that are important to any decisions.

Casting around for non-heuristic definitions of engineers to contrast his ideas with, Koen explores the possibility of there being a recipe, a set of steps which, if followed, will lead to good engineering. He concludes (p. 62):

... more candid authors admit that engineers cannot simply work their way down a list of steps but must circulate freely within the proposed plan — iterating, backtracking and skipping stages almost at random. Soon structure degenerates into a set of heuristics badly in need of other heuristics to tell what to do when.

Again, this feels like what I do when I'm testing. I wrote about it in Testing All The Way Down, and Other Directions:

It's not uncommon to view testing as a recursive activity ... I feel like I follow that pattern while I'm testing. But ... testing can be done across, and around, and inside and outside, and above and below, and at meta levels of a system ... Sometimes multiple activities feed into another. Sometimes one activity feeds into multiple others. Activities can run in parallel, overlap, be serial. A single activity can have multiple intended or accidental outcomes, ... all the way down, and the other directions.

So, are testers engineers? Frankly I find myself bothered when Koen talks about engineers as a group, and about what they are like and not like. I have the same problem making generalisations about testers or pretty much any other set of people defined by a variable in common. I can't in good faith say that (all) testers are engineers.

But I don't think that matters. There's so much to like and exploit in what Koen writes about engineering methodology. I can see many parallels with the way that I like to think about testing, and the context in which testing tasks place.

But, and it's a big but, I find that also with science: the scientific method and the notion of mandated science are are useful tool and a useful lens through which to view my day job. And I also find it with design, and software development, and editing, and detective work, and ...

Again, I don't think that "but" matters. I  accept that the engineering method is heuristic and I can say that it's a tool I can, do, and will use in my testing.