Decision By Precision?

I2E, the flagship product at Linguamatics, is a text mining engine and so sits in the broad space of search tools such as grep, Ctrl-F, and even Google. In that world, evaluating "how good" or "how relevant", or the "correctness" of a set of search results is interesting for a number of reasons, including:

it may be hard to define what those terms mean, in general cases.
it may be possible to calculate some kind of metric on well-understood, small, data sets but less so at scale.
it may be possible to calculate some kind of metric for simple searches, but less so for complex ones.
on different occasions the person searching may have different intent and needs from the same search.

But today we'll concentrate on two standard metrics that can be easily defined and which have agreed definitions: precision (roughly "how useful the search results are") and recall (roughly "how complete the results are").

Imagine we want to test our search engine. We have a set of documents and we will search them for the single word "testing". The image below, from Wikipedia, shows how we could calculate the metrics.

There's a lot of information in there, let's unpack some of it:

The square represents the documents.
The solid grey circles are occurrences of the word "testing".
The clear grey circles are occurrences of other words.
The central black circle is the set of results from the search.
The term positive means that a word is in the results.
The term negative means that a word is not in the results.
The term true means that a word is classified correctly.
The term false means that a word is classified incorrectly.

Let's overlay some numbers to make it clear. We inspect the documents (outside of the SUT) and find that there are 1000 words of which 100 are occurrences of the word "testing" (these are the solid grey circles.)

We run our search using the SUT and get back 50 results (the central black circle). We inspect those results and find that 35 are the word "testing" (the true positives) and 15 are something else (the false positives - asserted to be correct, but in fact incorrect).

The pictographs at the bottom of the image give us the formulae we need: precision comes only from the set of results we can see, and in this case is 35/50 or 70%. Recall requires knowledge of the whole set of documents, and for us is 35/100 or 35%.

A striking difference but which is better? Can one be better? These things are metrics, so can they be gamed?

Well, if the search simply returned every word in the documents its recall would be 100/100, or 100%, but precision would be very low at 100/1000, or 10%, because precision takes the negative content in the search results into account.

So can you get 100% precision? You certainly can: have the search return only those results with an extremely high confidence of being correct. Imagine only one result is returned, and it's a good one, then precision is 1/1 or 100%. Sadly, recall in this case is 1/100 or 1%.

Which is very interesting, really, but what does it have to do with testing?

Good question; it's background for a rough and ready analogy that squirted out of a conversation at work this week, illustrating the appealing trap of simple confirmatory testing. Imagine that you run your system under test with nominated input, inspect what comes out, and check that against some acceptance criteria. Everything in the output meets the criteria. Brilliant! Job done? Or precision 100%?

Images: https://flic.kr/p/sqtWUT, Wikipedia

Comments

Meet Me Halfway?

The Association for Software Testing is crowd-sourcing a book, Navigating the World as a Context-Driven Tester , which aims to provide responses to common questions and statements about testing from a context-driven perspective . It's being edited by Lee Hawkins who is posing questions on Twitter , LinkedIn , Mastodon , Slack , and the AST mailing list and then collating the replies, focusing on practice over theory. I've decided to contribute by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me? --00-- "Stop answering my questions with questions." Sure, I can do that. In return, please stop asking me questions so open to interpretation that any answ...

The Best Programmer Dan Knows

I was pairing with my friend Vernon at work last week, on a tool I've been developing. He was smiling broadly as I talked him through what I'd done because we've been here before. The tool facilitates a task that's time-consuming, inefficient, error-prone, tiresome, and important to get right. Vern knows that those kinds of factors trigger me to change or build something, and that's why he was struggling not to laugh out loud. He held himself together and asked a bunch of sensible questions about the need, the desired outcome, and the approach I'd taken. Then he mentioned a talk by Daniel Terhorst-North, called The Best Programmer I Know, and said that much of it paralleled what he sees me doing. It was my turn to laugh then, because I am not a good programmer, and I thought he knew that already. What I do accept, though, is that I am focussed on the value that programs can give, and getting some of that value as early as possible. He sent me a link to the ta...

Can Code, Can't Code, Is Useful

The Association for Software Testing is crowd-sourcing a book, Navigating the World as a Context-Driven Tester , which aims to provide responses to common questions and statements about testing from a context-driven perspective . It's being edited by Lee Hawkins who is posing questions on Twitter , LinkedIn , Mastodon , Slack , and the AST mailing list and then collating the replies, focusing on practice over theory. I've decided to contribute by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me? --00-- "If testers can’t code, they’re of no use to us" My first reaction is to wonder what you expect from your testers. I am immediately interested ...

Beginning Sketchnoting

In September 2017 I attended Ian Johnson 's visual note-taking workshop at DDD East Anglia . For the rest of the day I made sketchnotes, including during Karo Stoltzenburg 's talk on exploratory testing for developers (sketch below), and since then I've been doing it on a regular basis. Karo recently asked whether I'd do a Team Eating (the Linguamatics brown bag lunch thing) on sketchnoting. I did, and this post captures some of what I said. Beginning sketchnoting, then. There's two sides to that: I still regard myself as a beginner at it, and today I'll give you some encouragement and some tips based on my experience, to begin sketchnoting for yourselves. I spend an enormous amount of time in situations where I find it helpful to take notes: testing, talking to colleagues about a problem, reading, 1-1 meetings, project meetings, workshops, conferences, and, and, and, and I could go on. I've long been interested in the approaches I've evol...

Don't Know? Find Out!

In What We Know We Don't Know , Hillel Wayne crisply summarises a handful of research findings about software development, describes how the research is carried out and reviewed and how he explores it, and contrasts those evidence-based results with the pronouncements of charismatic thought leaders. He also notes how and why this kind of research is hard in the software world. I won't pull much from the talk because I want to encourage you to watch it. Go on, it's reasonably short, it's comprehensible for me at 1.25x, and you can skip the section on Domain-Driven Design (the talk was at DDD Europe) if that's not your bag. Let me just give the same example that he opens with: research shows that most code reviews focus more on the first file presented to reviewers rather than the most important file in the eye of the developer. What we should learn: flag the starting and other critical files to receive more productive reviews. You never even thought about that possi...

How do I Test AI?

Recently a few people have asked me how I test AI. I'm happy to share my experiences, but I frame the question more broadly, perhaps something like this: what kinds of things do I consider when testing systems with artificial intelligence components . I freestyled liberally the first time I answered but when the question came up again I thought I'd write a few bullets to help me remember key things. This post is the latest iteration of that list. Caveats: I'm not an expert; what you see below is a reminder of things to pick up on during conversations so it's quite minimal; it's also messy; it's absolutely not a guide or a set of best practices; each point should be applied in context; the categories are very rough; it's certainly not complete. Also note that I work with teams who really know what they're doing on the domain, tech, and medical safety fronts and some of the things listed here are things they'd typically do some or all of. Testing ...

Express, Listen, and Field

Last weekend I participated in the LLandegfan Exploratory Workshop on Testing (LLEWT) 2024, a peer conference in a small parish hall on Anglesey, north Wales. The topic was communication and I shared my sketchnotes and a mind map from the day a few days ago. This post summarises my experience report. Express, Listen, and Field Just about the most hands-on, practical, and valuable training I have ever done was on assertiveness with a local Cambridge coach, Laura Dain . In it she introduced Express, Listen, and Field (ELF), distilled from her experience across many years in the women’s movement, business, and academia. ELF: say your key message clearly and calmly, actively listen to the response, and then focus only on what is relevant to your needs. I blogged a little about it back in 2017 and I've been using it ever since. Assertiveness In a previous role, I was the manager of a test team and organised training for the whole ...

Software Sisyphus

The Association for Software Testing is crowd-sourcing a book, Navigating the World as a Context-Driven Tester , which aims to provide responses to common questions and statements about testing from a context-driven perspective . It's being edited by Lee Hawkins who is posing questions on Twitter , LinkedIn , Mastodon , Slack , and the AST mailing list and then collating the replies, focusing on practice over theory. I've decided to contribute by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me? --00-- "How can I possibly test 'all the stuff' every iteration?" Whoa! There's a lot to unpack there, so let me break it down a little: who is suggesting that "al...

Not a Happy Place

A few months ago I stopped having therapy because I felt I had stabilised myself enough to navigate life without it. For the time being, anyway. I'm sure the counselling helped me but I couldn't tell you how and I've chosen not to look deeply into it. For someone who is usually pretty analytical this is perhaps an interesting decision but I knew that I didn't want to be second-guessing my counsellor, Sue, or mentally cross-referencing stuff that I'd researched while we were talking. And talk was what we mostly did, with Sue suggesting hardly any specific tools for me to try. One that she did recommend was finding a happy place to visualise, somewhere that I could be out of the moment for a moment to calm disruptive thoughts. (Something like this .) Surprisingly, I found that I couldn't conjure anywhere up inside my head. That's when I realised that I've always had difficulty seeing with my mind's eye but never called it out. If I try to imagine ev...

Why Question?

Questions are a powerful testing tool and, like any tool, can be used in different ways in different scenarios with different motivations and different results. A significant part of my role is generating questions and I will generally have a lot of them. I will rarely ask them all, though, and I've put a lot of time and effort into learning to be comfortable with that. A couple of examples: I was in a meeting this week where the technical conversation was too deep for me to give a perspective from a position of knowledge. I could have disengaged, but I didn't. Instead, I asked occasional questions, not wanting to derail the discussion or disrupt the flow. Some were detail questions, to help grow my understanding. Some were scoping questions, to help understand motivations. The one that really landed, however, was about the focus of the meeting. Although I couldn't contribute at a low level, I understood enough to suspect that we were not discussing the key problem tha...

Hiccupps

Search This Blog