The Tester as Engineer?

Much of Definition of The Engineering Method by Billy Vaughn Koen chimes with what I've come to believe about testing. 

In part, I think, this is because thinkers who have influenced my thinking were themselves influenced by Koen's thoughts. In part, also, it's because some of my self-learned experience of testing is prefigured in the article. In part, finally, it's because I'm reading it through biased lenses, wanting to find positive analogy to something I care about. 

I recognise that this last one is dangerous. As Richard Feynman said in Surely You're Joking, Mr. Feynman!: "I could find a way of making up an analog with any subject ... I don’t consider such analogs meaningful.” 

This is a short series of posts which will take an aspect of Definition of The Engineering Method that I found interesting and explore why, taking care not to over-analogise.

• Getting Myself Koen
• Sota so Good
• Meta is Better
• The Tester as Engineer?

--00--

In this series so far I've pulled out a couple of Koen's key concepts for attention: sotas and the Rule of Engineering. I find them both aesthetically pleasing and with practical applications. However, they are cast explicitly for engineers and I'm a tester. I wonder whether, by Koen's intention, they'd apply to me? Are testers engineers? Does testing overlap with engineering? If so, where? If not, why not?

The definition of an engineering problem and its derived definition of an engineer might help to judge the answer (p. 42-3):

If you desire change; if this change is to be the best available; if the situation is complex and poorly understood; and if the solution is constrained by limited resources, then you too are in the presence of an engineering problem ... If you cause this change by using the heuristics that you think represent the best available, then you too are an engineer ... the engineer is defined by a heuristic — all engineering is heuristic.

Let's take each of the criteria in turn:

• change: it's the remit of testers to cause change, in the information state of a project if not directly in any deliverable.
• best available: in Koen's world, "best" is conditional on context and the participants. It doesn't mean objectively maximal. So I intepret this as doing the perceived most important things in an attempt to uncover the most important information.
• complex and poorly understood: looked at from an appropriate level of granularity, pretty much everything is complex and contains unknowns.
• limited resources: there was never a project where the manager said "take all the time you like testing this, I don't care when it ships".
• use heuristics: I would like to think that testers (consciously) use heuristics in their work.

I'm uncomfortable aligning testing and engineering by this route. If I was prepared to say that an activity is only testing when a problem is complex and poorly understood then I could define testers as people who take on complex and poorly understood problems. Unfortunately, I don't agree with the premise: I think it's possible to test something that is not complex, and I think it's possible to test something that's well understood (to whatever degree is relevant in context). In those circumstances, though, I'd suggest that the chances of provoking a change in the information state is likely to be reduced.

Is Koen saying that engineers can't work on trivial things? Or perhaps that they are not doing engineering when they do?

There's a long-running debate in the testing world about whether testing is a role or a job title. I've mused on it myself over the years and concluded that activities we might agree are testing are not the sole remit of people we might call testers. From #GoTesting:

To get to the desired (bigger picture) quality involves asking the (bigger picture) questions; that is, testing the customer's assumptions, testing the scope of the intended solution - you can think of many others - and indeed testing the need for any (small picture) testing, on this project, at this time.

Whether this is done by someone designated as a tester or not, it is done by a human and, as Rands said this week, I believe these are humans you want in the building. #GoTesting

You can play this the other way too: not everything someone with the role title tester does is necessarily what we might call testing.

I spent some time wondering what to make of this paragraph (p. 51):

We have noted that the engineer is different from other people ... The engineer is more inclined to give an answer when asked and to attempt to solve problems that are [non-trivial, but seem practically possible] ... The engineer is also generally optimistic ... and willing to contribute to a small part of a large project as a team member and receive only anonymous glory.

Although he's careful to caveat most of these attributes ("more inclined", "generally") I am allergic to all-encompassing assertions. With respect to testing, I wrote about it in You Shouldn't be a Tester If ...:

A belief that you should conform to a list of context-free statements about what a tester must be would concern me. I'd ask whether you really have testerly tendencies if you prefer that idea to a pragmatic attitude, to doing the best thing you can think of, for the task in hand, under the constraints that exist at that point.

This, to me, is closely allied with Koen's idea of what engineering is and only serves to enhance the dissonance I feel with his assertions about what an engineer is.

Koen does make role comparisons in his article, in particular the engineer and the scientist. He is not keen on the idea of engineering as applied science, apparently wanting instead to regard science as a tool within engineering (p. 63):

Misunderstanding the art of engineering, [some people] become mesmerised by the admittedly extensive use made of science by engineers and ... identify [science] with engineering [but] the engineer recognizes both science and its use as heuristics.

Tellingly, I don't recall him permitting scientists to use what he might call engineering methods. To me, it is simply not the case that all science proceeds by induction, hypothesis generation, and comparison to some natural state of affairs.

There's a sweet definition of a heuristic "in its purest form" (p. 48) that I thought might be relevant:

it does not guarantee an answer, it competes with other possible values, it reduces the effort needed to obtain a satisfactory answer to a problem and it depends on time and context for its choice.

Scientists conduct thought experiments. What are they if not heuristic by this definition? In fact, what are any experiments if not heuristic — hundreds of factors in any experimental setup and methodology could, unknowingly, invalidate the result. One the points of pride for committed scientists is that their findings, though valuable for a time, are likely to be shown wrong in some respect by a later scientist.

Koen also compares engineering with systems thinking and notes the crucial role of feedback (p. 56):

The success or failure of the engineer's effort is fed back to modify the heuristics in the engineer's sota

This seems natural to how I want to view testing. I like the idea of sotas and I really like the idea of overlapping and shared sotas in a given environment. On a project, for example, as we learn more about how the system under development behaves we modify our expectations of it and the way we engage with it. But we also take actions that we desire will provoke other changes. The sotas evolve based on feedback.

A few years ago I had a deep and wide-ranging landslide rush of a conversation with Anders Dinsen that we documented in What We Found Not Looking For Bugs. In trying to characterise what testing does in the abstract, I wrote:

• Some testing, t, has been performed
• Before t there was an information state i
• After t there is an information state j
• It is never the case that i is equal to j (or, perhaps, if i is equal to j then t was not testing)
• It is not the case that only t can provide a change from i to j. For example, other simultaneous work on the system under test may contribute to a shared information state.
• The aim of testing is that j is a better state than i for the relevant people to use as the basis for decision making

... I might propose [an information state is] something like a set of assertions about the state of the world that is relevant to the system under test, with associated confidence scores. I might argue that much of it is tacitly understood by the participants in testing and the consumption of test results. I might argue that there is the potential for different participants to have different views of it - it is a model, after all. I might argue that it is part of the dialogue between the participants to get a mutual understanding of the parts of j that are important to any decisions.

Casting around for non-heuristic definitions of engineers to contrast his ideas with, Koen explores the possibility of there being a recipe, a set of steps which, if followed, will lead to good engineering. He concludes (p. 62):

... more candid authors admit that engineers cannot simply work their way down a list of steps but must circulate freely within the proposed plan — iterating, backtracking and skipping stages almost at random. Soon structure degenerates into a set of heuristics badly in need of other heuristics to tell what to do when.

Again, this feels like what I do when I'm testing. I wrote about it in Testing All The Way Down, and Other Directions:

It's not uncommon to view testing as a recursive activity ... I feel like I follow that pattern while I'm testing. But ... testing can be done across, and around, and inside and outside, and above and below, and at meta levels of a system ... Sometimes multiple activities feed into another. Sometimes one activity feeds into multiple others. Activities can run in parallel, overlap, be serial. A single activity can have multiple intended or accidental outcomes, ... all the way down, and the other directions.

So, are testers engineers? Frankly I find myself bothered when Koen talks about engineers as a group, and about what they are like and not like. I have the same problem making generalisations about testers or pretty much any other set of people defined by a variable in common. I can't in good faith say that (all) testers are engineers.

But I don't think that matters. There's so much to like and exploit in what Koen writes about engineering methodology. I can see many parallels with the way that I like to think about testing, and the context in which testing tasks place.

But, and it's a big but, I find that also with science: the scientific method and the notion of mandated science are are useful tool and a useful lens through which to view my day job. And I also find it with design, and software development, and editing, and detective work, and ...

Again, I don't think that "but" matters. I  accept that the engineering method is heuristic and I can say that it's a tool I can, do, and will use in my testing.

Meta is Better

Much of Definition of The Engineering Method by Billy Vaughn Koen chimes with what I've come to believe about testing. 

In part, I think, this is because thinkers who have influenced my thinking were themselves influenced by Koen's thoughts. In part, also, it's because some of my self-learned experience of testing is prefigured in the article. In part, finally, it's because I'm reading it through biased lenses, wanting to find positive analogy to something I care about. 

I recognise that this last one is dangerous. As Richard Feynman said in Surely You're Joking, Mr. Feynman!: "I could find a way of making up an analog with any subject ... I don’t consider such analogs meaningful.” 

This is a short series of posts which will take an aspect of Definition of The Engineering Method that I found interesting and explore why, taking care not to over-analogise.

• Getting Myself Koen
• Sota so Good
• Meta is Better
• The Tester as Engineer?

--00--

It is Koen's contention that engineering is applied heuristics. He takes the term sota to be the set of heuristics known by an individual or a group at a specific time (see Sota so Good) and expects that an outcome will be motivated by heuristics from the sotas of the parties involved in the problem definition and solution.

Having determined that, he looks for a general rule for engineering (p. 41):

Since every specific implementation of the engineering method is completely defined by the heuristic it uses, [the quest for a rule to implement the engineering method] is reduced to finding a heuristic that will tell the individual engineer what to do and when to do it.

For those of you worrying quietly at the back, by this point Koen has acknowledged that heuristics are fallible rules of thumb. However, I'm worrying with you when wondering quite what it means for an implementation to be defined by a heuristic. My  current interpretation is something like this: "choosing and applying a relevant engineering heuristic is an instance of the engineering method (for Koen)".

I am categorically not worrying about Where he goes next, though. Just look at this (p. 42):

My Rule of Engineering is in every instance to choose the heuristic to use from what my personal sota takes to be the sota representing the best engineering practice at the time I am required to choose.

BOOM!

I covered "best practice" in Sota so Good so I'll ignore it here and move on to what I love about this rule:

• it reminds the engineer to check their personal biases
• it admits time (and so broader context) to the equation
• it shows us that we can use heuristics to choose a suitable heuristic

Koen stretches the concept further by saying that the intersection of the sotas of all engineers across all times will contain only one heuristic (p. 42):

The Rule of Engineering is: Do what you think represents best practice at the time you must decide, and only this rule must be present.

Worriers, if you're concerned that this has a whiff of the No True Scotsman about it, then so am I: is the heuristic defined by being in the sota of all engineers, or is the set of engineers defined by their having this heuristic in their sota?

I'm prepared to put this quibble to one side, though. I find Koen's formulation beautiful. A heuristic which helps us decide which approach to take is a useful step back from any situation and implicitly admits context into its interpretation and application.

When defining testing for myself I strove to achieve that kind of generality, to pack layers of nuance into a simple principle, to provide tools for decision-making at the point of use. This is what I came up with:

Testing the is the pursuit of relevant incongruity.

I like it and find it valuable, but still have concerns that the terms I've used hinder easy interpretation by others. I have always admired how Jerry Weinberg managed turn that kind of trick so well. These three lines have been part of my daily life for years:

• A problem is the difference between what is perceived and what is desired.
• Quality is value to some person.
• Things are the way they are because they got that way.

I think Koen's Rule of Engineering might be joining them.

Sota so Good

Much of Definition of The Engineering Method by Billy Vaughn Koen chimes with what I've come to believe about testing. 

In part, I think, this is because thinkers who have influenced my thinking were themselves influenced by Koen's thoughts. In part, also, it's because some of my self-learned experience of testing is prefigured in the article. In part, finally, it's because I'm reading it through biased lenses, wanting to find positive analogy to something I care about. 

I recognise that this last one is dangerous. As Richard Feynman said in Surely You're Joking, Mr. Feynman!: "I could find a way of making up an analog with any subject ... I don’t consider such analogs meaningful.” 

This is a short series of posts which will take an aspect of Definition of The Engineering Method that I found interesting and explore why, taking care not to over-analogise.

• Getting Myself Koen
• Sota so Good
• Meta is Better
• The Tester as Engineer?

--00--

Much of Definition of The Engineering Method by Billy Vaughn Koen chimes with what I've come to believe about testing. 

In part, I think, this is because thinkers who have influenced my thinking were themselves influenced by Koen's thoughts. In part, also, it's because some of my self-learned experience of testing is prefigured in the article. In part, finally, it's because I'm reading it through biased lenses, wanting to find positive analogy to something I care about. 

I recognise that this last one is dangerous. As Richard Feynman said in Surely You're Joking, Mr. Feynman!: "I could find a way of making up an analog with any subject ... I don’t consider such analogs meaningful.” 

This is the first in a short series of posts which will take an aspect of Definition of The Engineering Method that I found interesting and explore why, taking care not to over-analogise.

• 
  

--00--

The leaping off point for Definition of the Engineering Method is the scientific method. From Wikipedia:

The scientific method is an empirical method of acquiring knowledge that has characterized the development of science since at least the 17th century. It involves careful observation, applying rigorous skepticism about what is observed, given that cognitive assumptions can distort how one interprets the observation. It involves formulating hypotheses, via induction, based on such observations; experimental and measurement-based testing of deductions drawn from the hypotheses; and refinement (or elimination) of the hypotheses based on the experimental findings. 

Specifically Koen observes that, while the scientific method has been much studied, he does not believe that engineering is simply "applied science" and there is little literature on what he calls the engineering method. Yet engineering has been, is, and will continue to be critical to the way our world is shaped and the way we live in it. Shouldn't we care about how we engineer our environment and ourselves?

(Answer: yes.)

As Koen develops his ideas on what the engineering method could be, he co-opts a term, state of the art (abbreviated as sota), which he takes to be the set of heuristics known by an individual or a group at a specific time. For any pair of engineers, it is likely that their sotas will intersect at some foundational points and then differ where their experience has caused them to specialise on tasks. A given engineer's sota will change through her career, as she finds some heuristics more or less valuable.

This concept is interesting because it allows him to visualise, with simple set diagrams, how problems fall into or outside of the sotas of individuals and groups. This feels neat and I found it easy to nod along to until I read this (p. 31):

The [sota] of the engineering profession defines the best possible engineering solution. This overall solution represents best engineering practice and is the most reasonable practical standard against which to judge the individual engineer. It us a relative standard instead of an absolute one, and like all sotas it changes in time

As a card-carrying context-driven tester, I flinched. Best practice? Really?

Fortunately, in Koen's context, "best" has a specific interpretation (p. 12):

A fundamental characteristic of an engineering solution is that it is the best available from the point of view of a specific engineer. If this engineer knew the absolute good, he would do that good. Failing that, he calculates his best based on his subjective estimate of an informed society's perception of the good.

Sotas allow Koen to represent an engineering sota distinct from a society's sota, and speculate on how this difference can lead to solutions to the wrong problem, or the wrong problem being put forward to be solved. Where sota intersection is smaller, there is likely to be less shared understanding and more chance of problem and solution being incompatible in some important way.

In this consideration of the relationships between the expert and the public, there are echoes of Harry Collins' work on Science Studies. Take this passage (p. 39):

Some argue that we are witnessing a shrinking of [joint efforts between the general population and engineers] as society disciplines the engineering profession because of disagreement over past solutions. No longer is it sufficient for an engineer to assert that a mass transportation system or nuclear reactor is needed and safe ... What is most urgently needed is research to determine the minimum overlap necessary for a non engineer to be technologically literate.

I was also struck by similarities between sotas and the thoughts on the propagation of theory (a term we were using more broadly than its conventional meaning) around communities that Sneha Bhat and I presented in Transforming Theory and Practice:

... we perceive two broad types of theory: behavioural (what the system does) and practical (how to exercise the system).  In a typical project the first of these is likely to get reported back to the team in general, but the latter is likely to remain local with either this tester or her peers when they share information that helps them to do their job. 

This flow of theory aligns well with the idea that the sotas of different individuals and different can be populated differently, and perhaps even helps to explain how that can be the case for members of the same team with different areas of expertise.

An angle we explored, but I don't think Koen does, is the idea that practitioner knowledge may be tacit, again per Harry Collins. In Transforming Theory and Practice we wrote:

An expert practitioner may add enormous value to a project by virtue of being able to divine the causes of problems, find workarounds for them, and gauge the potential impacts on important scenarios. But they may find it hard to explain how they arrived at their conclusions ... This kind of tacit knowledge is learned over time, and with experience ...

Could tacit knowledge be part of an individual's sota? Is it the case that all engineering decisions are thought to be justifiable in terms of some aspect of a sota? Could the "art" aspect of sota permit undefined intuition? I like these kinds of questions.

But how are sotas relevant to testing? It's relevant to me in my testing because, as part of an engineering team solving problems for people who are generally not engineers, I want to do these kinds of things:

• recognise that the customer perspective is not the team's perspective
• recognise that my personal perspective is not the same as my colleagues' perspective
• look for the aspects of all relevant perspectives that are most relevant to the problem
• look for ways to communicate with all parties such that the right information is available to make informed choices about aspects of the problem that matter

The sota, and Koen's Venn-style diagrams, give me some language for, and visualisation of, that space.

Getting Myself Koen

Much of Definition of The Engineering Method by Billy Vaughn Koen chimes with what I've come to believe about testing. 

In part, I think, this is because thinkers who have influenced my thinking were themselves influenced by Koen's thoughts. In part, also, it's because some of my self-learned experience of testing is prefigured in the article. In part, finally, it's because I'm reading it through biased lenses, wanting to find positive analogy to something I care about. 

I recognise that this last one is dangerous. As Richard Feynman said in Surely You're Joking, Mr. Feynman!: "I could find a way of making up an analog with any subject ... I don’t consider such analogs meaningful.” 

This is a short series of posts which will take an aspect of Definition of The Engineering Method that I found interesting and explore why, taking care not to over-analogise.

• Getting Myself Koen
• Sota so Good
• Meta is Better
• The Tester as Engineer?

--00--

So I finally got a copy of Definition of the Engineering Method by Billy Vaughn Koen. It's probably only ten years since I first came across it in the AST's Black Box Software Testing, or perhaps Rapid Software Testing, or maybe both.

I'll review it later but right now I want to note something that I love. No, two things.

First, Koen on the engineering method's reliance on heuristics contrasted with the strictly logical underpinning of science (p. 14, 20-2):

... if the system you want to change is complex and poorly understood; if the change you will accept must be the best available; and if it is constrained by limited resources, then you are in the presence of an engineering problem

[...]

Unlike scientific laws, heuristics have never taken kindly to the harness of conventional logic systems ... Science is based on conflict, criticism or critical thought ... a new scientific theory, say B', replaces an old one, B, after a series of confrontations in which it is able to show that — as an approximation to reality — it is either broader in scope or simpler in form.

[...]

One heuristic does not replace another by confrontation but by doing a better job in a given context. Both the engineer and Michelangelo "criticise by creation, not by finding fault."

Second, listening to a recent episode of In Our Time on Paul Dirac, who won a Nobel prize for his theories of quantum mechanics, where one of the experts on Dirac said:

He could do something that mathematicians considered not quite right. There was something called the Dirac Delta Function that he introduced [which is] not a function even though he called it one and mathematicians worried about it for years ... I think this is where the engineering side comes in; if it worked and did what he wanted it to do he didn't worry whether it had ... the blessing of pure mathematicians.

So he had this fantastic mix of having exquisite mathematical knowledge and the pragmatism of saying "if it's not quite there but it works, I'll run with it."

Did I say two things? I meant three things. Koen's paper is amazing; the synchronicity of  hearing the ideas applied to Dirac while reading Koen is thrilling; and the connection to my long-term interest in combining theory and practice really gets me going.
Image: Amazon

Edit: I began reviewing Definition of the Engineering Method in Sota so Good.

Time Share

In knowledge work it's usual for ideas to need to be shared. In software development a common direction of travel for concepts is from one set of people with a vision for a feature to another set of people who will bring it to life. Sounds simple, right?

But it isn't simple and many ways exist to try to address that lack of simplicity, including thick wodges of functional specification and layers of sign-offs through to terse sticky notes, a chat, and iteration. Whatever the approach there's one constant: a commitment of time from the idea holders.

An anti-pattern common in our industry is that these individuals often also have emerging and urgent tasks which trump the merely important ones ... such as the sharing of ideas. Despite our best intentions we've seen it at our place too, and it impacts the quality of the thing that we can build in the time available to us.

Naturally, this kind of problem can be tackled at all sorts of levels. At one level, I was interested to find something with the potential for immediate effect, at low cost, and that I had sufficient clout to influence without requiring systemic change or permissions.

This is how my thinking went: first, while the people with the notion of what to build won't know everything up front, they usually have a good idea of their core wants and pieces of scope they'd be prepared to trade.

Second, while they may be time-poor, they can be persuaded to spare time for a short meeting with the promise of an outcome which doesn't require preparation by them.

Third, while lack of time might be problematic, deliberately constrained time can be a helpful forcing function.

So I asked if we could try experimental meetings on nominated candidate projects where scope was unclear, with these inspirations:

• Three Amigos: a small-group conversation with folks bringing varied perspectives
• Example Mapping: a goal of generating constraints, examples, and questions
• Essentials: a focus on core ideas, not specific implementations

The format we've arrived at is a 30-minute facilitated meeting. The time box is an enticement to attend and a motivation for staying on topic. Conversation and facilitation reduces up-front cost to attendees. The facilitator keeps things at a scope level, encourages participation, and records the findings on a whiteboard which is set up this way:

• at the top of the board, write an agreed one-line description of the project
• on the left, enumerate constraints; things that are in or out of scope
• in the centre, generate use cases for constraints
• on the right, record questions or requests for data

At the end of the meeting, the board (and the non-shallow shared understandings) serve as a starting point for a wider kick-off and an initial envelope for the project scope. It's not perfect and it sometimes fails spectacularly — usually when it becomes apparent that we don't have consensus on what the project is about. The fact that we've continued to do them is evidence that the relatively small investment is perceived to return sufficient value to justify it.

Some reflection: Although I started this experiment around 18 months ago I'm writing these notes in the context of the DEWT 9 peer conference on Testability that I attended a a couple of weeks ago.

At DEWT we talked about how arguing for change in terms of business value is more likely to succeed than talking about testability itself.

When I presented my DEWT notes to my team at Linguamatics it was suggested that many testability factors are also developability factors and I think that holds true here. The gains that we've got from this experiment are across the team.  And that's an idea worth sharing!
https://flic.kr/p/HTSP

Testability? Shhh!

Last weekend I was talking Testability at DEWT 9. Across the two days I accumulated nodes on the mind map above from the presentations, the facilitated discussions, and the informal conversations. As you can see it's a bit dense (!) and I'll perhaps try to rationalise or reorganise it later on. For now, though, this post is a brain dump based on the map and a few other notes. 

--00--

Testability is the ease with which the product permits itself to be tested (its intrinsic testability)

• ... but also factors outside of the product that enable or threaten its testing (so extrinsic)
• ... meaning that people, knowledge, opportunity, time, environments, and so on can be part of testability.

Desired outcomes of increasing testability might include

• easier, more efficient, testing and learning
• better feedback, risk identification and mitigation
• building  trust and respect across project teams

The term testability can be unhelpful to non-testers

• ... and also to testers (!)
• ... and so consider casting testability conversations in terms of outcomes
• ... and business value.

Actions that might be taken with the intention of increasing testability include

• changing the product (e.g. logging, control functions)
• collecting information (e.g. stakeholder requirements, user desires)
• applying tooling (e.g. for deployment, log analysis)
• acquiring expertise (e.g. for the customer domain, your own product range)
• obtaining more time (e.g. by moving deadlines, cutting low priority tasks)

Situations in which testability might be decreased include

• side-effecting an attempt to increase testability (e.g. introduce bugs, waste time on useless tools)
• losing motivation (e.g. because of poor working conditions, ill health)
• being asked to test to an unreasonable standard (e.g. to great depth in little time, in unhelpful circumstances)
• recognising that the existing test strategy misses important risks (e.g. there are previously unknown dependencies)

Blockers or challenges to testability might include

• only talking to other testers about it
• an inability to persuade others why it would be valuable
• bad testing
• previous failures

When requests for testability are denied, consider

• getting creative
• going under the radar
• finding allies
• the main mission

It might be appropriate to sacrifice testability when

• adding it would risk the health, safety, or commitment of the team, product, or company
• trading one kind of testability against another (e.g. adding dependencies vs increasing coverage)
• no-one would use the information that it would bring
• another term will be more acceptable and help to achieve the same goal
• another action for the same or lower costs will achieve the same goal
• a business argument cannot be made for it (this point may summarise all of the above)

Intrinsic changes for testability are features

• ... and should be reviewed alongside other requested product changes
• ... in whatever process is appropriate in context (of the product and the proposed change)
• ... by whoever is empowered to make those kinds of decisions.

Extrinsic changes for testability are less easily typed

• ... but should still be reviewed by appropriate people
• ... to an appropriate level
• ... in relevant process for the context.

Unsurprisingly, there are some things that I want to think about more ...

If attributes of testers contribute to testability then it's possible to increase/decrease testability by changing the tester. Logically I am comfortable with that but emotionally I find the language tough to accept and would like to find a different way to express it

I found intrinsic and extrinsic testability a useful distinction but too coarse because, for instance, I haven't distinguished factors that influence testability and factors that require testability.

Although it was easy to agree on intrinsic testability, there was less agreement on the existence or extent of extrinsic testability and no clear boundary on extrinsic testability for those who are prepared to accept it. I'm not sure that matters for practical purposes but definitional questions interest me.

There was consensus that we should sshhh! ourselves on testability and instead describe testability issues in terms of their impact business value. Unfortunately, characterising business value is not itself an easy task.

--00--

The participants at DEWT 9 were: Adina Moldovan, Andrei Contan, Ard Kramer, Bart Knaack, Beren van Daele, Elizabeth Zagroba, Huib Schoots, James Thomas, Jean-Paul Varwijk, Jeroen Schutter, Jeroen van Seeter, Joep Schuurkes, Joris Meerts, Maaike Brinkhof, Peter Schrijver, Philip Hoeben, Ruud Cox, Zeger van Hese.

Material from DEWT is jointly owned (but not necessarily agreed with) by all of the participants. Any mistakes or distortions in the representation of it here are mine.

Thank you to the Association for Software Testing for helping to fund the event through their grant programme.

Other material referenced or posted during the conference:

• James Bach's Heuristics of Software Testability
• Maria Kedemo and Ben Kelly's Dimensions of Testability
• Ash Winter and Rob Meaney's 10 Ps of Testability
• Rob Meaney's CODS 
• #DEWT9 on Twitter

Failure, Am I?

It's nothing personal I'm sure but at this week's Cambridge Agile Exchange Ben Mancini told me that I'm a failure. Ouch!

Failure, he says, is "the state or condition of not meeting a desirable or intended objective ... may be viewed as the opposite of success" and his hypothesis is that failure provides more learning than success but that we talk more about success than failure.

In his talk, Ben showed examples of people who have succeeded despite repeated failure, talked about the negative effects of failure, ways to reframe perceived failure, and the benefits of pushing through failure's cloud to its silver lining. Along the way he shared some of his own failures and invited us to share personal faux pas with our neighbours in the audience then look for the learnings in them.

One of the reasons I attend meetups is to be provoked into thought so, in no particular order ...

In my time I've certainly done things that I'd have preferred not to, and some things that others would have preferred I hadn't, and some things that were indisputably not what was required or expected by anyone. Perhaps I'm over-sensitive, but I wonder whether that makes me a failure, or just someone who has, on occasion, failed? If I'm going to call something a failure, I find that intuitively I want to judge actions rather than people.

Ben's definition of failure comes from Wikipedia. At first flush it seems reasonable but read further down the Wikipedia page and you'll find nuance that again accords with my own instinct. As with so many things, failure is subject to the relative rule: failure is failure to somebody at some time. The same event might be viewed differently by different people or by an individual in the moment and then later.

It's easy to say "failure is a better teacher than success" but historically I've been sceptical about it, at least stated as baldly as that: is it really better every time, for any person, for all kinds of learning? I changed my position slightly after reading Matthew Syed's excellent book, Black Box Thinking. I think learning from an action — positive or negative — requires reflection about (for example) what was done, what happened, the relationship between the two, other things that were done contemporaneously and other things that might have happened.

A failure might provoke more of that kind of reflection, for sure. As Richard Cook writes in his paper How Complex Systems Fail:

    ... all practitioner actions are actually gambles, that is, acts that take place in the face of uncertain outcomes. The degree of uncertainty may change from moment to moment. That practitioner actions are gambles appears clear after accidents; in general, post hoc analysis regards these gambles as poor ones. But the converse: that successful outcomes are also the result of gambles; is not widely appreciated.

For me, the key here is post-hoc. The same kind of learning might be taken from positive events if we reviewed them. To reflect on Ben's hypothesis: do we really talk more about success than failure? Which we, measured how?

I find Cook compelling on sociotechnical modes of failure, see e.g. Everybody Flirts, Fail Over, and Read it and Weep, and his system-wide perspective prompts another interesting question: whose learning are we interested in? The individual or the system?

In the talk, James Dyson was used as an example of someone who had failed many times before succeeding. His quest to create a bagless vacuum cleaner is well-documented and for me is an example of one context in which I'm comfortable to say that failure (on some specific interpretation) is indisputably a learning experience.

Dyson created thousands of incrementally different prototypes, iterating his way to one that had all of the functionality that he needed. Was each attempt a failure? Or was each attempt a step towards his desired outcome, a quantum of success? Setting up actions as experiments means that getting a result at all is the goal. Generate-and-test is a legitimate strategy.

Related, which parent hasn't allowed an action to proceed, confident that it will not work, because a bruised knee or a burnt finger or a low mark in the test appears to be the way that the offspring wants to learn a particular lesson. Advice could have taught it, but a painful experience can help too. How should we view this kind of event? From the child's perspective, in the moment, it's a painful failure. From the parent's perspective, in the moment, it's vindication, success. Who is right? For how long?

Most parents would try to set up those kinds of outcomes in a safe way. Assuming that learning from failure does have high value, I wonder whether it is diminished by happening in a safe environment? Might the scale of learning increase with jeopardy? But some failure could be terminal: should we learn to walk the tightrope by crossing Niagara Falls?

As a manager I've used the safe space tactic although I try to be open and explicit about it. Perhaps I'll set it up as "I would be delighted if you showed me I was wrong" or "If this works for you, then I'll have learned something too." I think of this as a way of making the effort into an experiment.

Some jobs are apparently set up for failure: a salesperson might expect to sell once in 100 opportunities. Is that 99 failures per success? I've heard it cast in this way: each rejection means one less before the sale. There is no fear of failure with this philosophy and, while those of us who cringe at the idea of working in sales might find it hard to believe, that kind of approach can be learned.

I wonder how to control the learning that comes from failure. It's well-known that machine learning approaches which rely on being "taught" by trying, failing, and being corrected  can pick up on unexpected aspects of their training material. Is there an analogue for human learning? Ben listed a bunch of authors in his talk, people who'd tried, tried, and tried again to be published despite numerous rejections. What was their learning? To be resilient? To sell themselves well? To find their tribe? To get better at writing?

Could it be that some of those people learned nothing through failure to convince an editor that they had a story worth telling? Could they, for example, simply be already resilient folk with large egos that needed satisfying? What about survivorship bias? Where are all the people who failed as many times but didn't ultimately get published? What was their learning? Was it greater than those who were published? How is that even measured?

My goal in this post was to spend a limited time to work through the right-now thoughts that were spurred by Ben's talk. I think I achieved that. Having seen them, you might decide that all I have is shallow, half-cooked, or just plain nonsense. If so, have I failed? If you liked the notes, have I succeeded? Could either outcome make me a success or a failure? To who? On what basis?