Testing By Exploring

This week I listened to a Ministry of Testing podcast on exploratory testing.  After the intros, as the panelists  attempted in turn to give their definitions of exploratory testing, I realised that I didn't have one for myself.

It's not that I don't know or haven't seen definitions. Here's one I've heard a few times, from back in 2008, in A Tutorial in Exploratory Testing by Cem Kaner: 

Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.

It's not that I think The Exploratory Elders like Kaner got it right back in the day and their words should be carved on monumental stones and polished on Sundays, or that I'm not interested any more. I was reading Maaret Pyhäjärvi's post, Exploratory Testing the Verb and the Noun, just this morning.

It's not that I think definitions are just playing semantics either. Spending the time to understand what I mean when I say something can be enlightening, as well as respectful to the people I choose to say it to. Further, I use definitions as hands-on, practical tools. This is one of my favourites, due to Jerry Weinberg:

A problem is a difference between things as desired and things as perceived.

Exploration is intimately wrapped up in the way I want to test and it was intriguing to find that I hadn't given some explicit thought to the term exploratory testing before. After a little reflection, I think the answer is that I made my definition of testing accommodate it.

Testing is the pursuit of relevant incongruity.

I'll use whatever approach I think is most suitable given the context to implement my testing. Predominantly that is exploration and I don't think  exploration in software is materially different to exploration in general. Here's a couple of snapshots of the term from the Oxford English Dictionary:

The action of examining thoroughly; investigation, scrutiny, study; an instance of this.

The action of travelling to or around an uncharted or unknown area for the purposes of discovery and gathering information; (later also) the action or activity of going to or around an unfamiliar place in order to learn about it. Also as a count noun: an instance of this; an expedition for the purpose of discovery.

When I think of explorers my mind's eye conjures visions of Victorian men with large mutton chops hacking their way through the jungle, planting flags on icy  mountain summits, and sailing merrily into the areas of the map labelled Here Be Monsters.

And that's how many people seem to see exploratory testing too. Perhaps not the facial hair, but certainly the low-tech blood-sweat-and-tears manual labour.

Unsurprisingly, that's not my perspective. What I see are people striving to achieve a goal with intent and agency, while fighting against a range of constraints and in the face of considerable uncertainties.

They use tactics such as surveying, scouting, and time-boxing. They will take measurements, observations, and samples. They will use tools in the manner intended and creatively in other ways when the need arises. 

They will exploit their surroundings by moving left, right, forward, and back, but also up trees or onto high ground to get a different view or down into the ground to understand the terrain. They will build on existing knowledge, frame experiments to create new knowledge, and collect material as they go for later consumption and absorption into their growing body of knowledge.

They will frequently pause to take stock and decide what the next target is in the wider context of their overall mission. From the crow's nest they might see a landmass and decide to try to circumnavigate it, land on it, or ignore it, whichever will serve their mission of finding a lost island, or making contact with native tribes, or locating the Northwest Passage.

They will react to what they observe, and to what the environment throws at them. They will try to square their observations with data they have picked up from elsewhere, and they'll assiduously document as they go, for their own benefit, for their sponsors' interest, and for those who will follow on behind them.

I'll grant you that, yes, those whiskery chaps in unsuitable clothing seem to be doing things the hard way. But that's because they lived 150 years ago and the cutting edge of their times looks blunt today. You can bet your best stovepipe hat that if they'd had access to Land Rovers, satellites, or global internet connectivity they'd have used them in the cause of their mission.

Familiar? It feels so to me, and I am comfortable that I don't have a definition of exploratory testing. Unless you count this:

Exploratory testing is testing by exploring.

Image: Michael Palin (from Erebus, a book about Victorian explorers my dad lent me and I'm reading at the moment)

Testing Stories

 

Testing Stories is a new ebook written by testers about their testing:

Software testing professionals from around the globe have volunteered to each share a story related to software testing, with the aim of inspiring others from their experiences.

The team have decided to donate all royalties sold through leanpub to Open Sourcing Mental Illness, as we collectively value their mission statement and work in mental health in the tech and open source communities.

This is a fun project and, more importantly, a good cause. I was happy to contribute along with Dave Symonds, Richard Hill, Sandhya Krishnan, Feroz Ally, Venkat Ramakrishnan, Louise Harney, Haroon Sheikh, Haroon Sheikh, Patricio Miner, Bruce Hughes, Felipe Knorr Kuhn, Louise Woodhams, Mike Harris, Aaron Flynn, Ady Stokes, Kim Cote, Peter Johnson, Ben Dowen, Claire Reckless, Laveena Ramchandani, Andrea Jensen, Steven Devonport, Sapan Parikh, Beth Marshall, Prashant Hegde, Johanna South, Suman Bala, Michael "Fritz" Fritzius, Lisa Crispin, John McGee, NITHIN S.S, and Gemma Ashfield.

Exploratory Tooling

Last week I started a new job. The team I've joined owns a back-end service and, along with all the usual onboarding process, inevitable IT hassles, and necessary context-gathering, one of my goals for my first week was to get a local instance of it running and explore the API.

Which I did.

Getting the service running was mostly about ensuring the right tools and dependencies were available on my machine. Fortunately the team has wiki checklists for that stuff, and my colleagues were extremely helpful when something was missing, out of date, or needed an extra configuration tweak.

Starting to explore the service was boosted by having ReDoc for the endpoints and a Postman collection of example requests against them. I was able to send requests, inspect responses, compare both to the doc, and then make adjustments to see what effects they had.

If that's testing of any kind, it's probably what I call pathetic testing:

There's this mental image I sometimes have: I'm exploring the product by running my fingers over it gently. Just the lightest of touches. Nothing that should cause any stress. In fact, I might hardly be using it for any real work at all. It's not testing yet really; not even sympathetic testing although you might call it pathetic testing because of itself it's unlikely to find issues.

One of the functions offered by the service is a low-latency search endpoint which enables autocompletion on the client side. You know the kind of thing; as the user types, suggestions appear for them to choose from. 

The doc for this is fine at a high level. I was interested to understand the behaviour at a lower level but found Postman (with my level of expertise) required too many actions between requests and made comparison across requests difficult. The feedback loop was too long for me.

So I wrote a tool. And if that sounds impressive don't be fooled: I was standing on the shoulders of giants.

What does my tool do? It's a shell script that runs curl to search for a term with a set of parameters, pipes the response through jq to parse the hits out of the JSON payload, and then uses sed to remove quoting. Here's the bare bones of it:

#!/bin/zsh

p1=$1
p2=$2
term=$3

URL='http://localhost/search?query='$term'&p1='$p1'&p2='$p2

curl --location $URL | jq '.results | @csv' | sed 's/[\"\\]//g'

A series of runs might look like this, if I'm exploring consistency of results as I vary the search term starting with "te":

$ search x y te
team,term,test,teams,terms,tests

$ search x y ter
term,terms

$ search x y term
term,terms

$ search x y terms
terms

$ search x y termst

$ search x y terms
terms

$ search x y term
term,terms

Or perhaps like this if I'm exploring the parameters:

$ search x y test
test,tested,testing

$ search A y test
test,Test,tested,Tested,testing,Testing

$ search A B test
test,Test,tested

There is nothing clever going on here technically but I get a major benefit practically: I have abstracted away everything that I don't need so that my next test can be composed and run with minimal friction. I can quickly cycle through variations and compare this and previous experiments easily. Feedback loop tightened.

Actually, when I said "I have abstracted away everything that I don't need" what I really meant was "I have a very specific mission here, which is to look at how search terms and parameters affect search results. Because I'm on that mission, I choose not to view all of the other data returned by the server on each of my requests. I may miss something interesting by doing that but I accept the trade-off".

That aside, there are numerous things that I could do with this tool now that I have it. For example:

• Write a script with a list of search terms in it and call each of them in turn, collect the results and write them to a file that I could analyse in, say, Excel. 
• Point it at a production server as well, and compare my test environment to production in every call.
• Launch hundreds of these searches at the same time from another script as a simple-minded stress test.

Or I could just throw it away because it has served its purpose: facilitating familiarisation with a feature of our service at low cost and high speed, initiating source code inspection and conversations, and along the way helping me to find a few inconsistencies that I can feed back to the team.
Image: https://flic.kr/p/8pCp4W
Highlighting: https://pinetools.com/syntax-highlighter

Unit Testing? Grow Up!

Yesterday, I attended Unit Testing for Grown ups, a webinar with Gil Zilberfeld. Despite the title, it wasn't a tech-heavy tour through testing strategies, test doubles, or testability but instead a series of business arguments in favour of testing, primarily at the unit level. 

If I had to sum it up in a couple of bullets, it'd be something like this:

• developers, your work does not start at the first line of production code and end when you push
  
• managers, your teams can probably work smarter

And if I had a few sentences, it'd go like this:

The goal of a software development group is to solve problems by writing code that "works" and is maintainable. 

It is usually the case that as a codebase expands unintended side-effects occur more frequently and the costs of integration, testing, and bug fixing grow.

How to reduce these costs? Test first, and mostly at the unit level.

But the tests have to be good tests so developers should train their creative skills for thinking about possible problems, and then provide coverage for the right set of them. 

This will be easier if the code is testable. Which means that testability needs to be part of the design of the system. 

Test-Driven Development could be a great way to address all of the points above but it can be hard to get your head around the cost-benefit payoff (as a developer and manager), hard to begin, and then also hard to maintain the discipline.

Bug Advocacy

This month I've been taking the Bug Advocacy course at the Association for Software Testing. It's been ten years since I took the introductory Foundations course, the first in the Black Box Software Testing series, and with this degree of hindsight I can see how fundamental that was in how I like to test.

I've done plenty of learning in the decade since I started testing, so much of the material in Bug Advocacy is not new to me. That doesn't detract from the value of the course. I've taken the opportunity to refresh my memory, and to look at how the other students interpret the same material and how they go about the practical exercises, and compare that to my own approach.

I love that these courses are run with small cohorts, emphasise practice to reinforce theory but also to ask questions of it, and require that students review each other's work as an aid to learning.

Each week there are exercises that have the students interact with each other and software. We then write reports and reviews which themselves are reviewed and reported on. If that sounds convoluted, or even meta, don't be fooled: on a bug reporting course in particular, the concentration on data gathering, organisation, and dissemination is incredibly rewarding.

Tester credibility and influence is a key element in the course, emphasised repeatedly. If we, and the information we provide, and the actions we take, have the respect of those we work with then we are more likely to be able to help decision-makers make the right kinds of decisions, including which bugs to schedule for fix, and when.

  The course materials can be found at testingeducation.org.

The Spec, But Why?

 

I'm in the middle of BBST Bug Advocacy at the Association for  Software Testing right now.  As you might imagine, on a course with that name there's been plenty of talk about what we mean by terms like bug, value, and quality. One of the great things about the four-week course is the mix of book work and application, so we students are repeatedly challenged with situations in which the learning can be practically applied.

I have a lot of time for both Seth Godin and Shane Parrish so I'd have been listening carefully to Seth's appearance on the Knowledge Project podcast anyway but, given the context I'm in, the passage I've transcribed below stood out. It's about how the concept of quality is concretised as conformance to spec, and how that in turn directly drives physical actions. It starts at around 1:04:45:

There's lots to be said about the spec. First lets talk about Edwards Deming and what spec and quality mean. Quality is not luxury, quality is not expensive, quality is not that you love it, quality is just one thing: it meets spec.

So if I look under an electron microscope at any part of a Lexus, which is by any measure the highest quality car there is, it's filled with defects. But they're not defects that matter, because they're defects that are within spec.

So, we begin by understanding what is the spec of the work we're going to do? If it meets spec, not only is it quality but it is good enough, and good enough is not a slur. Good enough is a definition: it met spec.

So once it's good enough we ship the work. If you're not happy with that, change your spec. 

But let's be really clear about what the spec is. That what it meant for a Lexus to be good enough when they first came out was it had to be a standard deviation better than a Mercedes. That was their definition.

If someone's gonna say "no, no, we can't ship this Lexus because it's not perfect!" the product manager should say "no, no, no, it was never supposed to be perfect, it never can be perfect, it simply met spec."

The hard work was in defining the spec, a spec that will get you to the next step.

What I like about this perspective on quality (and it's similar to Crosby's view) is that it emphasises understanding what you're trying to create, to what standard, for who. 

What I particularly like about the way it was described here is those last few words: "get you to the next step". That other stuff is the what, but this is the why.
Image: https://flic.kr/p/5ZziiY

Secret Agency

I've listened to every episode of Gene Kim's Idealcast, a podcast about "the important ideas changing how organizations compete and win." If that terse statement sounds desert dry to you, then think again, the show is a wide open ocean of practical experience and considered theory.

I particularly enjoyed the one with Elisabeth Hendrickson whose playbook has chapters on management, software development, people development and more, along with the testing chops of Explore It! that she's particularly known for in our field.

Gene's presentation style is that of a knowledgeable friend, making space for his guests to lay out their perspective on a topic, and giving each insight a big "yes, and". He injects verbal sidebars into the podcast from time to time, pausing the interview to zoom in on a point that was made in passing and direct the listener to references that will give background, or talking about how a specific example made the concept click for him, or highlighting a connection to thoughts aired in another of the podcasts.

So, when he tweeted a stream of notes quoting Elisabeth, and pointing to her "FABULOUS DOES20 talk ... Influence > Authority and Other Principles of Leadership" I was ready to put down what I was doing and watch it. 

Which I did, and it is fabulous, and if I can boil the message down into a couple of sentences, it would be something like this: regardless of whether you are in a position of hierarchical power in your organisation your ability to be a change agent depends on your actions; you do have agency even if you don't realise it yet. To help to effect change you should act to demonstrate the value of the change,  praise behaviours that move the change along, visualise the motivation for the change, and amplify the voices that have evidence for that motivation.