Skip to main content

Testing and Semantics


The other day I got tagged on a Twitter thread started by Wicked Witch of the Test about people with a background in linguistics who’ve ended up in testing. That prompted me to think about the language concepts I've found valuable in my day job, then I started listing them, and then realised how many of them I've mentioned here over the years.  

This post is one of an occasional series collecting some of those thoughts. 

--00--

In this series so far we've looked at words and syntax. In both cases we've found that natural language is an imprecise medium for communication.

  • We might know the same words and grammar as others
  • ... but they will have their own idea about what they mean
  • ... and even where we agree there is ambguity
  • ... and all of us, the world, and the language are evolving
  • ... all the time.

Today we'll add semantics which, in a pleasing twist, is itself ambiguous, meaning the study of meaning and also some specific meaning.

Sounds very formal, you might say. Where and when could semantics be relevant to testers in their day-to-day work? 

Well, erm, just about everywhere and all the time!

On a recent project my team wanted to provide input validation rules for clients of a service to allow them to check user input without a round-trip to the server. Initially this seemed simple: we'd have a list of rules with names like "minimum," "maximum," "minimumLength," "maximumLength," and "allowableCharacters."

But it doesn't take long to get to a point where we were wondering how these constraints might apply to different data types, how to make behaviour consistent across them, and how to combine multiple rules on the same input. 

These are questions of semantics, of meaning, and for non-trivial applications they can quickly get tricky. We would have got it very wrong (for us and our users) if the semantics of our constraint language were hard to understand or, worse, ambiguous.

So we paused and looked around for existing frameworks we could borrow from, such as constraint validation in HTML 5. Artificial languages such as these, which are machine-parsable, tend to be smaller and less prone to ambiguity than natural languages, from necessity and through the careful work of the designers.

But semantics isn't confined to formal languages. Your whole product has semantics for its users, whether you intend it and consciously craft it or not. And it can be different for different users.

For example, in a UI we'd generally like a consistent experience in which the semantics of a button with a cross in it (say "close") can be combined with the semantics of other components such as a window or dialog box to mean "close the window" or "close the dialog". If that's not the case, where does it leave a user in their ability and confidence to navigate the software easily?

Outside of the user experience, a shared understanding of the terms a team is using can help to avoid unnecessary friction and increase information sharing and collaboration. However, arguing over nuances rather than getting on and doing can also block everything else. 

Context plays a part in where I'll put myself on that particular spectrum. Sometimes a project needs a terminological grounding to stop stakeholders talking past one another (as Iain McCowatt found while establishing testing principles across teams at a large corporation) on other occasions only sucking-it-and-seeing will help a team to get a feel for the problems at hand (as my team at Linguamatics did when trying to work out team values and mission.)
 
What I do favour in general, though, is being explicit about what any definitions are when we have them, or that we're agreeing to proceed without them for the moment when we don't.

If you've been around testing for a while you've probably seen "it's just semantics" and "it's not just semantics" tossed casually into conversations. For example, Dan Ashby in reply to Casey Rosenthal, on a Twitter thread:

I don't think it's just semantics here. It seems to be the meaning behind the words that's causing problems. Not the words itself. There's an entire craft relating to exploratory testing that seems to be missing from your radar.

A lot of time and effort from many people has gone into defining what testing is or isn't. I've done it myself and there are numerous different versions. So what does testing mean?

The concept of a namespace is interesting here. You may be familiar with the idea already from package structures in computer languages, where the reuse of names with different semantics is enabled by adding a label to indicate the version being used. For example: dan.exploratory-testing means something different to casey.exploratory-testing. We only need to use the differentiating labels when there is a chance of ambiguity.

While this can certainly solve one kind of problem, it also risks confusion or even us-and-them opposition between groups who favour particular namespaces, or perhaps between those in a namespace and those who are not even aware that there is a namespace.

Semantics is not just a linguistic construct. It has potential social impacts too.

In Linguistics, compositionality is the idea that the meaning of a sentence or phrase is built from the meanings of its parts guided by the syntax. Take a sentence like "James tests software", we might break this down into simple syntax like this:

[S [N James] [VP [V tests] [N software] ] ]

Where a sentence (S) is made up of a noun (N) and a verb phrase (VP), and VP is itself broken down into a verb (V) and another noun.

We first build the meaning of VP. From our understanding of "tests" and "software" we can have an idea what it means to be testing software rather than testing saliva samples, or electrical circuits, or widgets coming off a production line.

Next we combine the meaning of "tests software" with "James" to understand that it's a specific person that is being asserted to test software rather than some other person, or a team, or Selenium.

This initially seems appealing and logical. From a testing perspective there are helpful analogies to be taken from it, such as the example of a cancel button applying in the same way to windows and dialog boxes from earlier.

But there are less helpful analogies too. Who hasn't had a conversation about the risks of relying on the testing of subsystems for testing the whole system at some point in their career? I had one last week.

In fact, the principle of compositionality doesn't hold in language either, or at least not straightforwardly for some cases. Take sarcasm, where building the meaning of a sentence by composing the meanings of individual words is likely to miss other cues that give the intended interpretation. Often the meaning is the complete opposite of "what was said."

In philosophy, the context principle recognises that it can be dangerous to look for the meaning of some term outside of the context in which it is being used. Context-driven testing wraps that insight into two of its own principles:

  • The value of any practice depends on its context.
  • There are good practices in context, but there are no best practices.

Without some universal understanding of a thing, an understanding that applies regardless of any context, we can't claim that any actions on it will have particular value.
Image: Wikimedia

Comments

Popular posts from this blog

Can Code, Can't Code, Is Useful

The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "If testers can’t code, they’re of no use to us" My first reaction is to wonder what you expect from your testers. I am immediately interested in your working context and the way

Meet Me Halfway?

  The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "Stop answering my questions with questions." Sure, I can do that. In return, please stop asking me questions so open to interpretation that any answer would be almost meaningless and certa

Not Strictly for the Birds

  One of my chores takes me outside early in the morning and, if I time it right, I get to hear a charming chorus of birdsong from the trees in the gardens down our road, a relaxing layered soundscape of tuneful calls, chatter, and chirrupping. Interestingly, although I can tell from the number and variety of trills that there must be a large number of birds around, they are tricky to spot. I have found that by staring loosely at something, such as the silhouette of a tree's crown against the slowly brightening sky, I see more birds out of the corner of my eye than if I scan to look for them. The reason seems to be that my peripheral vision picks up movement against the wider background that direct inspection can miss. An optometrist I am not, but I do find myself staring at data a great deal, seeking relationships, patterns, or gaps. I idly wondered whether, if I filled my visual field with data, I might be able to exploit my peripheral vision in that quest. I have a wide monito

Testing (AI) is Testing

Last November I gave a talk, Random Exploration of a Chatbot API , at the BCS Testing, Diversity, AI Conference .  It was a nice surprise afterwards to be offered a book from their catalogue and I chose Artificial Intelligence and Software Testing by Rex Black, James Davenport, Joanna Olszewska, Jeremias Rößler, Adam Leon Smith, and Jonathon Wright.  This week, on a couple of train journeys around East Anglia, I read it and made sketchnotes. As someone not deeply into this field, but who has been experimenting with AI as a testing tool at work, I found the landscape view provided by the book interesting, particularly the lists: of challenges in testing AI, of approaches to testing AI, and of quality aspects to consider when evaluating AI.  Despite the hype around the area right now there's much that any competent tester will be familiar with, and skills that translate directly. Where there's likely to be novelty is in the technology, and the technical domain, and the effect of

Postman Curlections

My team has been building a new service over the last few months. Until recently all the data it needs has been ingested at startup and our focus has been on the logic that processes the data, architecture, and infrastructure. This week we introduced a couple of new endpoints that enable the creation (through an HTTP POST) and update (PUT) of the fundamental data type (we call it a definition ) that the service operates on. I picked up the task of smoke testing the first implementations. I started out by asking the system under test to show me what it can do by using Postman to submit requests and inspecting the results. It was the kinds of things you'd imagine, including: submit some definitions (of various structure, size, intent, name, identifiers, etc) resubmit the same definitions (identical, sharing keys, with variations, etc) retrieve the submitted definitions (using whatever endpoints exist to show some view of them) compare definitions I submitted fro

Testers are Gate-Crashers

  The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "Testers are the gatekeepers of quality" Instinctively I don't like the sound of that, but I wonder what you mean by it. Perhaps one or more of these? Testers set the quality sta

Vanilla Flavour Testing

I have been pairing with a new developer colleague recently. In our last session he asked me "is this normal testing?" saying that he'd never seen anything like it anywhere else that he'd worked. We finished the task we were on and then chatted about his question for a few minutes. This is a short summary of what I said. I would describe myself as context-driven . I don't take the same approach to testing every time, except in a meta way. I try to understand the important questions, who they are important to, and what the constraints on the work are. With that knowledge I look for productive, pragmatic, ways to explore whatever we're looking at to uncover valuable information or find a way to move on. I write test notes as I work in a format that I have found to be useful to me, colleagues, and stakeholders. For me, the notes should clearly state the mission and give a tl;dr summary of the findings and I like them to be public while I'm working not just w

Build Quality

  The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "When the build is green, the product is of sufficient quality to release" An interesting take, and one I wouldn't agree with in general. That surprises you? Well, ho

Make, Fix, and Test

A few weeks ago, in A Good Tester is All Over the Place , Joep Schuurkes described a model of testing work based on three axes: do testing yourself or support testing by others be embedded in a team or be part of a separate team do your job or improve the system It resonated with me and the other testers I shared it with at work, and it resurfaced in my mind while I was reflecting on some of the tasks I've picked up recently and what they have involved, at least in the way I've chosen to address them. Here's three examples: Documentation Generation We have an internal tool that generates documentation in Confluence by extracting and combining images and text from a handful of sources. Although useful, it ran very slowly or not at all so one of the developers performed major surgery on it. Up to that point, I had never taken much interest in the tool and I could have safely ignored this piece of work too because it would have been tested by

The Best Laid Test Plans

The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "What's the best format for a test plan?" I'll side-step the conversation about what a test plan is and just say that the format you should use is one that works for you, your coll