The Value of Testing is ...

I was at Cambridge Agile Exchange's first Unconference the other night and I pitched a 15-minute discussion with the title The value of testing is ...:

I'm a tester and I'm interested in the value that others perceive testing brings to product development in their contexts. Where does testing add value? Where does it add most value? When does it add value? What kinds of testing activities deliver the right kind of value at the right kinds of costs? Do you need testers in order to test? 

The participants in the session included product owners, scrum masters, developers, coaches, an ex-test manager, and a colleague from Linguamatics who has just switched to testing from linguistics. (And I'm still hiring.)

The mission I had in mind was to gather data, to solicit and record views, and to not push some kind of militant testing agenda. I think that worked well, and if I had to pull one observation out it'd be how quickly the conversation turned to threats to the value of testing. 

Here's the mind map we created together:

Notes: This map has some groupings that I didn't manage on the fly, a couple of things added from memory, and a little reconstruction of the stuff I couldn't read on my photos. Any errors are mine.

Thank you to all the participants and CAE for the opportunity. It was fun.

Image: Dani Oliver

Be a Which-What-Who?

Yesterday's Team Eating (the Linguamatics brown bag lunch thing I organise) was called A Linguist's Idioticon. In it, one of my colleagues gave an overview of some of the linguistic terminology used in the natural language processing components of our core product, I2E.

In a section on wh-words, Rudyard Kiping's Six Honest Serving Men were listed:

I keep six honest serving-men
(They taught me all I knew);
Their names are What and Why and When
And How and Where and Who.

Aside from any interesting language properties they might have, you might have seen these words presented as perspectives from which to view a situation alongside techniques like Edward de Bono's Six Thinking Hats.

In passing, the presentation noted that which and whose are rarely included when people are asked to list wh-words (and, as a further aside, and the talk had many of those, neither are whom or whither — but they are pretty archaic these days).

It occurred to me that when using the wh-word lenses, there may be some value in considering which and whose too. They differ from the others because they require a noun to go with them; you can't simply ask "which?" but instead need to nominate a referent, "which thing?"

How might  this be useful? Imagine that you are reviewing a feature proposal. You ask "who?" as a way to find potential stakeholders and come up with, say, end users and system administrators. You could then additionally list factors relevant to the project and ask about those, for example: "whose budgets?", "whose judgements?", or "whose priorities?" and find other people in a targeted way.

So I'm speculating: could this kind of tweak systematically broaden the scope of an analysis, but in specific and relevant directions?

And you're thinking: what?
Image: Dr. Seuss

Looking at Observability

The Test team book club is reading Guide: Achieving Observability from Honeycomb, a high-level white paper outlining what observability of a system means, why you might want it, and factors relevant to achieving and getting value from it.

It's not a particularly technical piece but it's sketched out to sufficient depth that our conversations have compared the content of the guide to the approaches taken in some of our internal projects, the problems they present, and our current solutions to them.

While I enjoy that practical stuff a great deal, I also enjoy chewing over the semantics of the terminology and making connections between domains. Here's a couple of first thoughts in that area.

The guide distinguishes between monitoring and observability.

• monitoring: "Monitoring .. will tell you when something you know about but haven't fixed yet happens again" and "... you are unable to answer any questions you didn’t predict in advance. Monitoring ... discard[s] all the context of your events".
• observability: "Observability is all about answering questions about your system using data", "rapidly iterate through hypothesis after hypothesis" and "strengthen the human element, the curiosity element, the ability to make connections."

I don't find that there's a particularly bright line between monitoring and observability: both record data for subsequent analysis and whether the system is (appropriately) observable depends on whether the data recorded is sufficient to answer the questions that need to be asked of it. I think this maps interestingly to conversations around checking and testing and the intent of the data gathering and later questions against that data.

Increasing observability will generally result in more data being collected. A couple of dimensions are important here:

• high cardinality: refers to keys which can take many values. User, request, or session identifiers in a busy web service might be a good example. Being able to slice out data according to these kinds of variables allows analysis to take place at appropriate resolutions for given questions. 
• richness: refers to the variety of data collected or, to put it another way, the amount of context that's associated with any "event" that the system records.

This relates nicely to a quote I pulled from Edward Tufte's Visualising Evidence last year, and chimes very much with my own experience:

When assessing evidence, it is helpful to see a full data matrix, all observations for all variables, those private numbers from which the public displays are constructed. No telling what will turn up. (p. 45)

This is one of the reasons that, at a small scale, I think Excel's pivot tables are such a valuable tool for quick, convenient, exploratory data analysis, and why one day I'll write a post about that.
Image: https://flic.kr/p/24JYSq

The Process is Personal is Political

This year I've read three books that follow projects from the perspective of individual contributors, managers, and the business. They are The Goal, The Phoenix Project, and The Soul of a New Machine.

The Goal is perhaps the most well-known. It's a pedagogical novel in which a manufacturing plant manager is given three months to turn his failing plant around. He stumbles across a mentor who, with well-chosen questions and challenges, exposes the fallacies of the traditional production models being followed and suggests new ways to analyse the business. The philosophy at the heart of this is the Theory of Constraints, where constraints can be anything that gets in the way of the business goal, and the aim, as described by a couple of its key players is "not so much to reduce costs but to increase throughput". (p. 298, Kindle)

The Phoenix Project is also fictional but this time set in the world of IT, Ops, Dev, and Test. Again, the main protagonist is under pressure to change outcomes in a complex system, again finds a mentor and, again, The Theory of Constraints is central. In fact, the authors freely admit that the book is "an homage to The Goal, hoping to show that the same principles ... could be used to improve technology work." (p. 341)

The Soul of a New Machine sees journalist Tracy Kidder given access to a product team at the Data General Corporation during the development of the MV/8000 computer (codenamed Eagle) in the late 1970s. It predates the other two books, and has no obvious aim beyond the telling of a good story through the varied lenses of a selection of its protagonists.

For those who've been around implementation, deployment, and maintenance projects of any size or complexity over any length of time there will be many moments of empathy, sympathy, and antipathy in these three works. As a tester, I particularly enjoyed reading about debugging the prototype Eagle: "Veres ... tells Holberger [that they] ran 921 passes [of the test suite] last night, with only 30 failures. And Holberger makes a face. In this context, 921 is a vast number. It means that any given instruction in the diagnostic program may have been executed millions of times. Against 921 passes, 30 failures is a very small number. It tells them the machine is failing only once in a great while — and that's bad news ..." (p. 194)

There's a bigger picture here, though. Crudely, the first two books are about processes and their underpinnings while the third is about people and their interactions. They are symbiotic, they interact intimately: process is made by people, people follow imposed process and are the instigators of emergent process. Understanding both people and process is crucial in the workplace.

People and process are both also subject to politics, and understanding that is important too. In the Phoenix Project, as improvements are attempted in one group, turf wars and ass-covering activity break out around the place. In The Soul of a New Machine, the Eagle project can only exist because of the experienced under-the-radar manoeuvrings of the group manager, Tom West. "We're building what I thought we could get away with" he says early on. (p. 31)

I'd recommend all three of these books. Why? Well, the recognisable episodes and personalities are great fun in a Big Chief I-Spy kind of way, but the higher value for me came from the opportunity to use someone else's eyes to view them. And then, naturally, to reflect on that and try to apply it to my own contexts.

The editions I read:

• The Soul of a New Machine, Tracy Kidder (Avon) 1981
• The Goal, Eli Goldblatt 30th Anniversary Edition (North River Press) 2014, on Kindle
• The Phoenix Project 5th Anniversary Edition, Gene Kim, Kevin Behr, George Spafford (IT Revolution) 2018

Thanks to the Dev manager for the loan of The Soul of a New Machine.

Images: Amazon and AbeBooks.

A Doug Report

A couple of years ago our book club at work read On Testing Non-testable Programs. At the time I idly wondered whether I could use it to make an ontology of oracles but didn't pursue the idea beyond finding that Doug Hoffman had already provided a classification scheme back in 1998: A Taxonomy for Test Oracles

The other week he presented a webinar, The Often Overlooked Test Oracle, which built on some of that earlier work, outlining a range of oracle types with pros, cons, and other notes.

I like theory like this that can guide my practice and I'm still practicing my sketchnotes so I tried to capture the set of oracles he covered in a sketch. I'll just leave it here.

Bear Testing

I got my dad a Bear Grylls book, How to Stay Alive: The Ultimate Survival Guide for Any Situation, last Christmas. He keeps it in the toilet — in case of emergencies, presumably.

Flicking through the bright orange volume at the weekend, a section on being lost caught my eye. Having spent a proportion of the last four weeks struggling to understand a performance testing experiment, I am currently very familiar with that sense of bearings going AWOL.

Grylls is largely concerned with physical endurance in extreme environments. Clearly, the stakes are somewhat different for me, at my desk, wrestling with several databases, a few servers, a set of VMs and their hypervisors, numerous configuration options, and the code I've written to co-ordinate, log, and analyse the various actions and interactions.

Yet, still, Grylls' advice says something relevant to me:

Millions of people get lost every year ... Their critical faculties grow impaired and they become less able to make smart choices ...  When that happens, it's almost always because they've made one simple mistake: they didn't stop as soon as they realized they were lost.

It's natural human instinct to keep going. We don't like going backwards ... We talk ourselves into thinking that we're going the right way ... Instead we need to be rigorous about not fooling ourselves. We need to swallow our pride.

• Stop: don't make a bad situation worse by pushing blindly on and getting more lost.
• Think: your brain is your best survival tool, so control it and use it to think logically.
• Observe: if you have a map, look for big, obvious features that you can't mistake for anything else in order to orientate yourself. 
• Plan: have a definite strategy which will force you to think things through clearly and, crucially, keep your morale up. 

We've all sometimes dug deep pits for ourselves and then kept digging, mislaid our sense of perspective, amped up our sense of pride, broken out of time boxes, continued on with one more desperate, unfounded, experiment in the vain hope it'll miraculously clear things up.

Likewise, we all know that we should take a step back, zoom out, defocus. I like the bluntness here though: we should simply STOP. (If I'm honest, I also like the recursivity of the S in STOP being stop.)

So I'm already nodding in recognition, and smiling, when he closes with this beauty: "Nothing is more dispiriting than not knowing what you're doing or where you are going." Quite.

Diploadocus Testing

Eric Proegler was in town last week and I asked him if he'd be up for re-running his recent TestBash talk, Continuous Performance Testing, at our office.

Good news! he was, and he told us the story of how a lumbering testing dinosaur who watched the agile asteroid crashing into his world could become a nimble testing ninja, slotting valuable and targeted performance tests confidently into modern continuous integration.

Bad news! I continued my adventures in sketchnoting and they're not the greatest.