Skip to main content

Who Cares?


Should the public care about software testing? That's the question that the Association for Software Testing and the BCS Software Testing Specialist Group asked at our first joint peer conference on 22nd November 2020. The remit was:

It’s our contention that most people don’t think very much about software development and software testing, despite software being deeply embedded in almost all aspects of our lives.

When there’s a publicised issue such as a national bank failing to process customer orders for several days, a government IT project overrunning for years and then being canned, or a self-driving car causing a fatal accident, then society might take notice for a while.

However, even when that happens it’s rare for the complexities and risks associated with the creation, integration, and maintenance of software systems to be front and centre in the discussion, and testing almost never makes it to the agenda at all.

In this workshop we aim to explore why that is, and what testers, testing organisations, software development companies, and governments could do to persuade the public that software testing is worth understanding and caring about.

As with so many events in these troubled times, we shifted an in-person experience online and were concerned that the buzz that comes from being in the same room as other people who are both knowledgeable and care deeply about the topic would be lost. 

I'll blog about the way we set it up later, but I'm pleased to report that the buzz was still there. I'll also blog about the breadth and depth of the conversations we had around the presentations but here I'm just going to summarise what the speakers said.

--00--

We kicked things off with Fiona Charles using Qantas flight QF72 as an example of how well-intentioned automation, complexity, and fail-safe mechanisms can result in at best unwanted behaviour and, at worst, disastrous outcomes. In that incident, faulty sensors caused on-board systems to misinterpret the plane's attitude as dangerous and take unnecessary action to correct it. Flight envelope constraints prevented the pilot from overriding the manoeuvre, leaving him a helpless spectator as passengers were tossed violently around the cabin.

Fiona posed the question "what level of control should we keep for ourselves?" and suggested that we have to build products that keep people at the centre of operations, facilitating and enhancing skilled human decision-making and enabling the human operator to work in whatever way is best for them, not fight against a system built for its own convenience.

Many modern systems (particularly those labelled AI) rely heavily on data, but data is not knowledge, and it comes with biases. As an industry and a scociety we are increasingly recognising this, but we tend not to recognise that the way the data is collected is biased, the architecture of the systems is biased, the models underlying the design are biased, the decision processes are biased, and, in fact, anything humans create is likely biased in some way.  

To help ourselves to see that bias and counteract it, Fiona said, we need to take seriously such questions as what could possibly go wrong, what possible outcomes there could be, and how does our product solve well-known concerns such as security or accessibility? Oh yes, and we should look to tune our bullshit detectors.

Harmful outcomes also concern Amit Wertheimer. In his presentation he took the position that software testing as an activity in its own right is just one way to help to create working products that do no harm. He described a separate team of testers distinct from developers as a crutch, providing a way for the builders of a product to disengage from the need for checking their work.

For Amit, in general, developers should be reviewing what they create and should be ready to "feel the pain" of it being found wanting by its users. That's not to say that the critical thinking skills that we testers like to claim have no value, but rather that other people, directly involved in development, can provide them too. He makes an exception for highly-specialised fields where domain knowledge may be instrumental in understanding the desired and observed behaviour of a product.

The public should not care about these details, though, he says. The public should care whether their product works and does no damage, not whether testers were involved in its production, or how.

Huib Schoots offered this perspective too, and also the opposite: the public should not care about software testing because that's the responsibility of the producers, yet the public should care about software testing because they need to be able to trust products, particularly in safety-critical situations.

He asked us to consider what happens when there's some kind of significant software failure. People tweet about it for a while, it blows up in the news perhaps, and then we all forget about it. Where is the forum in which the public can ask questions and request improvements? How can regulation and the law evolve with rapidly-changing technology?

He called for efforts to make an environment in which the public and software producers can get together to discuss how people's needs and concerns can be met, but then also warned about the risks of these being overrun by fringe views and conspiracy theories and noise.

Eric Proegler had no qualms about coming down on one side of the question: the public must care about software testing because of the increasing dependency of our lives and lifestyles on software.

Testers also have something to think about, he says. With current technology, it's easier than ever to create an "AI solution" and push it out to innumerable devices at a terrifying pace with huge numbers of poorly-understood and dynamic dependencies.

Software testing is under siege, in Eric's view.  One way forward is to find ways to incentivise better testing. Weak testing is motivated by short-termism, by getting to market, and by meeting quarterly targets. The business need trumps the societal need most times and Eric challenged us to wonder how we can change that.

The BCS are trying to change that, Adam Leon Smith claims, by engaging with government, the media, the public, and industry to highlight failures in IT and understand how they can be reduced both in frequency and impact. "We shouldn't hinder innovation" shouldn't be a popular opinion, in his opinion.

Regulation and certification is one avenue that can be explored. While this might seem like an anathema to some, Adam noted that software testing in some parts of some industries is already regulated, and cited the UK gaming machine testing strategy regulation as an example.

Acknowledging that testing is context-dependent, Adam speculated that there is scope for wider regulation of this kind. He would begin with high-risk scenarios where context could be restricted sufficiently that a non-generic standard for testing could plausibly be created. The high-level results of this kind of work could be communicated to consumers using a simple labelling scheme, similar to that used on food packaging to summarise the "healthiness" of the contents.

Take a step back, Janet Gregory urged us. We need to think about risks while producing our products and then find a way to communicate the risks to our users. A labelling scheme of the kind Adam suggested could certainly help, but Janet would like to see something like industry-wide checklists for things to consider on purchase, similar to the paperwork packaged with medicines that list potential side-effects.

She also asked for much tighter controls on the way in which products are advertised and cited self-driving cars as an example. For those kinds of products, adverts might claim "this model is an industry standard for safety" but consumers need to be aware that this doesn't mean getting in, pressing the take-me-to-mother's-house button and then going to sleep.

It's not enough to simply make this information available, of course. It needs to be made available in format that consumers can and, crucially, want to engage with. Techniques such as visualisations and analogy could be employed here. Bodies like AST and BCS can definitely be part of a dialogue around that.

Lalit Bhamare rounded things off for us with a call to action for consumers themselves. He talked about clean software — parallel to clean air — and said that unless consumers demand it, producers are unlikely to sacrifice convenience and profit to provide it.

What is clean software? Features that Lalit mentioned included reliability, quality, customer service, and a strong consideration of a product's impact on the world and not just the bottom line. Companies today largely take the public's acceptance of low-quality software for granted.

Testing is a crucial part of Lalit's idea. The public should care about software testing because testing helps them to have confidence that the software is clean, and if there's a public conversation about testing then manufacturers will be forced to pay attention to it.

The material created at the conference is jointly owned by the participants: Lalitkumar Bhamare, Fiona Charles, Janet Gregory, Paul Holland, Nicola Martin, Eric Proegler, Huib Schoots, Adam Leon Smith, James Thomas, and Amit Wertheimer.
Image: Rare Records

Comments

  1. I've gotten into the habit of circulating news stories around my company where it seems likely that something bad that happened might well have had a testing (or rather, lack of testing) dimension, so as to provoke discussion (or at least, thought).

    And if I personally come across something In Real Life where I think that might have happened, I will write to a CEO to say "I struggled with your app/product/website. I applied the methods I apply in my Day Job as a software tester, and I think you need to go back and have some human beings look at this." Usually this happens where a product works 100% fine but only if you know the happy path to get the result, and that happy path isn't obvious to an Ordinary User. This often suggests to me that the developers only employed automated testing and never let a human being loose on the product before it was released. I rarely get much of a response, but I live in hope.

    ReplyDelete
  2. It's a credit to you that you take the time and effort to do those things.

    Do you have thoughts on how our industry, or software testing specifically, should change to make it less likely that users have those kinds of experiences?

    ReplyDelete

Post a Comment

Popular posts from this blog

Can Code, Can't Code, Is Useful

The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "If testers can’t code, they’re of no use to us" My first reaction is to wonder what you expect from your testers. I am immediately interested in your working context and the way

Meet Me Halfway?

  The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "Stop answering my questions with questions." Sure, I can do that. In return, please stop asking me questions so open to interpretation that any answer would be almost meaningless and certa

Not Strictly for the Birds

  One of my chores takes me outside early in the morning and, if I time it right, I get to hear a charming chorus of birdsong from the trees in the gardens down our road, a relaxing layered soundscape of tuneful calls, chatter, and chirrupping. Interestingly, although I can tell from the number and variety of trills that there must be a large number of birds around, they are tricky to spot. I have found that by staring loosely at something, such as the silhouette of a tree's crown against the slowly brightening sky, I see more birds out of the corner of my eye than if I scan to look for them. The reason seems to be that my peripheral vision picks up movement against the wider background that direct inspection can miss. An optometrist I am not, but I do find myself staring at data a great deal, seeking relationships, patterns, or gaps. I idly wondered whether, if I filled my visual field with data, I might be able to exploit my peripheral vision in that quest. I have a wide monito

Testing (AI) is Testing

Last November I gave a talk, Random Exploration of a Chatbot API , at the BCS Testing, Diversity, AI Conference .  It was a nice surprise afterwards to be offered a book from their catalogue and I chose Artificial Intelligence and Software Testing by Rex Black, James Davenport, Joanna Olszewska, Jeremias Rößler, Adam Leon Smith, and Jonathon Wright.  This week, on a couple of train journeys around East Anglia, I read it and made sketchnotes. As someone not deeply into this field, but who has been experimenting with AI as a testing tool at work, I found the landscape view provided by the book interesting, particularly the lists: of challenges in testing AI, of approaches to testing AI, and of quality aspects to consider when evaluating AI.  Despite the hype around the area right now there's much that any competent tester will be familiar with, and skills that translate directly. Where there's likely to be novelty is in the technology, and the technical domain, and the effect of

Postman Curlections

My team has been building a new service over the last few months. Until recently all the data it needs has been ingested at startup and our focus has been on the logic that processes the data, architecture, and infrastructure. This week we introduced a couple of new endpoints that enable the creation (through an HTTP POST) and update (PUT) of the fundamental data type (we call it a definition ) that the service operates on. I picked up the task of smoke testing the first implementations. I started out by asking the system under test to show me what it can do by using Postman to submit requests and inspecting the results. It was the kinds of things you'd imagine, including: submit some definitions (of various structure, size, intent, name, identifiers, etc) resubmit the same definitions (identical, sharing keys, with variations, etc) retrieve the submitted definitions (using whatever endpoints exist to show some view of them) compare definitions I submitted fro

Testers are Gate-Crashers

  The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "Testers are the gatekeepers of quality" Instinctively I don't like the sound of that, but I wonder what you mean by it. Perhaps one or more of these? Testers set the quality sta

Vanilla Flavour Testing

I have been pairing with a new developer colleague recently. In our last session he asked me "is this normal testing?" saying that he'd never seen anything like it anywhere else that he'd worked. We finished the task we were on and then chatted about his question for a few minutes. This is a short summary of what I said. I would describe myself as context-driven . I don't take the same approach to testing every time, except in a meta way. I try to understand the important questions, who they are important to, and what the constraints on the work are. With that knowledge I look for productive, pragmatic, ways to explore whatever we're looking at to uncover valuable information or find a way to move on. I write test notes as I work in a format that I have found to be useful to me, colleagues, and stakeholders. For me, the notes should clearly state the mission and give a tl;dr summary of the findings and I like them to be public while I'm working not just w

Build Quality

  The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "When the build is green, the product is of sufficient quality to release" An interesting take, and one I wouldn't agree with in general. That surprises you? Well, ho

Make, Fix, and Test

A few weeks ago, in A Good Tester is All Over the Place , Joep Schuurkes described a model of testing work based on three axes: do testing yourself or support testing by others be embedded in a team or be part of a separate team do your job or improve the system It resonated with me and the other testers I shared it with at work, and it resurfaced in my mind while I was reflecting on some of the tasks I've picked up recently and what they have involved, at least in the way I've chosen to address them. Here's three examples: Documentation Generation We have an internal tool that generates documentation in Confluence by extracting and combining images and text from a handful of sources. Although useful, it ran very slowly or not at all so one of the developers performed major surgery on it. Up to that point, I had never taken much interest in the tool and I could have safely ignored this piece of work too because it would have been tested by

The Best Laid Test Plans

The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "What's the best format for a test plan?" I'll side-step the conversation about what a test plan is and just say that the format you should use is one that works for you, your coll