Hiccupps

  When I opened Blogger the other morning I saw that the Hiccupps all-time page view counter had ticked up to exactly 500000. My initial feeling was pleasure. Ooh! A milestone! Half a million views! Get me ! But that was fleeting.  As with many metrics, today's value is almost the least engaging aspect. What's more interesting is what the number represents, what it is desired that it should represent, and how closely those two concepts align on axes that matter. Also interesting: who wanted this metric (and there may be multiple interested parties) and whether they still want it, what assumptions are being made about it, what values (or milestones) matter, what it is being plugged into, and what decisions are made using it. That's not all, of course, because one sample is just one sample. More interesting is where it sits in context to others, if and how it is trending, and whether it correlates with relevant related metrics over the same period. I could go on. I don't ...

The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn ,   Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "What is the best testing metric?" Now there's a question that has occupied many minds for many long hours. Don't worry, though, because this will...

  The Association for Software Testing is crowd-sourcing a book, Navigating the World as a Context-Driven Tester , which aims to provide responses to common questions and statements about testing from a context-driven perspective . It's being edited by Lee Hawkins who is posing questions on Twitter ,  LinkedIn ,  Slack , and the AST mailing list and then collating the replies, focusing on practice over theory. I've decided to contribute by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "We need some productivity metrics from testers" OK. I'd like to help you meet your need if I can but to do that I'll need to ask a few questions. Let's start with these: Who needs the metrics? Is there a particular pr...

The Association for Software Testing is crowd-sourcing a book, Navigating the World as a Context-Driven Tester , which aims to provide responses to common questions and statements about testing from a context-driven perspective . It's being edited by Lee Hawkins who is posing questions on Twitter ,  LinkedIn ,  Slack , and the AST mailing list and then collating the replies, focusing on practice over theory. I've decided to contribute by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "What percentage of our test cases are automated?" There's a lot wrapped up in that question, particularly when it's a metric for monitoring the state of testing. It's not the first time I've been asked either. In my...

This morning I was delighted to see the long-awaited return of the Cambridge Tester Meetup in the form of an online Lean Coffee!  Here's a few aggregated notes from the conversation in the group I was in. Onboarding of testers in Covid land From the company's side: Looking for tips for getting testers up and running remotely. Structured introduction plan, inluding people, tools, and relevant docs. Encouraging the new team member to complement that with finding their own path. Remember that it's hard being new and remote. Make sure that time to learn and to use the product is available. Try to find some task that can make them feel productive, an early win. Have a buddy with priority and time to answer questions. Set expectations on both sides. From the onboarder's side: Be prepared to ask questions. Self-motivation is really important. Set some goals for yourself (to get that achievement hit). Be prepared for it to be harder. Be brave (even if you don't feel it). T...

A friend lent me two books on metrics. One was so thumpingly wrong I could not finish it. One was so joyfully right I had to read it end-to-end, despite the author's recommendation not to. Dave Nicolette will be pleased to know that his book, Software Development Metrics , was the second of the two. Why so right? Because immediately I felt like I was in the hands of a pragmatist, a practitioner, and a personable guide. By page 8 he's chattily differentiated measures and metrics, identified a bunch of dimensions for projects, and then categorised metrics too. Along the way, he drops this piece of wisdom (p. 5): The sort of development process you're using will influence your choice of metrics. Some metrics depend on the work being done in a certain way. A common problem is that people believe they're using a given process, when in fact they're working according to a conflicting set of assumptions. If you apply metrics that depend on the process being done cor...

I2E, the flagship product at Linguamatics , is a text mining engine and so sits in the broad space of search tools such as grep , Ctrl-F , and even Google. In that world, evaluating "how good" or "how relevant", or the "correctness" of a set of search results is interesting for a number of reasons, including: it may be hard to define what those terms mean, in general cases. it may be possible to calculate some kind of metric on well-understood, small, data sets but less so at scale.  it may be possible to calculate some kind of metric for simple searches, but less so for complex ones. on different occasions the person searching may have different intent and needs from the same search. But today we'll concentrate on two standard metrics that can be easily defined and which have agreed definitions: precision (roughly "how useful the search results are")  and recall (roughly "how complete the results are"). Imagine we wan...

Four lightning talks at the Cambridge Tester meetup at Linguamatics last night, four topics apparently unrelated to one another, four slices through testing. Yet, as I write up my notes this morning I wonder whether there's a common thread ... Samuel Lewis showed us the money. Or, at least, where the money is held before being dispensed through those holes in the wall. He included some fascinating background information about ATMs (and a scary security story) but the thrust of his talk was the risks and corresponding mitigation strategies in a massive project to migrate the ATMs for a big bank to a new application layer and OS (more scariness: many are still running Windows XP). Much of the approach involved audit trails of various kinds, with customer and other stakeholders sharing their road maps and getting a view of the test planning and strategy in return. I enjoyed that the customer themselves was considered a risk (because they had a reputation for changing their mi...

Several times in recent months I've found myself being asked how many test cases there are in automated suites at Linguamatics . On each occasion I have had to admit that I don't know and, in fact, I'm not particularly motivated to calculate that  particular metric . Depending on how the conversations went, I've asked in return what my questioner thinks we can understand about the quality, coverage, relevance, value and so on of testing by knowing the number of cases. (And let's leave vocabulary aside for now.) Are you a counter? Try this scenario: you wrote sum , a program that adds two numbers, and have asked me to test it... Sure! I can test that for you. To do it, I'll write a test case generator. Here it is: #!/bin/bash echo "#!/bin/bash" for i in `seq 1 $1 ` do s= $(( $i + $i )) echo "r= \` ./sum $i $i \` ; if [ \$ r -eq $s ]; then echo \" OK \" ; else echo \" NO \" ; fi" done A s...

To the context-driven tester there are no best practices, merely practices to be applied in contexts where they are appropriate on missions to which they contribute. Context-driven  is differentiated from context-aware and other similar-sounding terms by virtue of the total freedom it gives to (and requires of) the tester to approach each situation afresh, driving the choice of practice from the context and not vice versa. That's not to say that expertise and experience can't play a part - we'd hope that knowledge of the range of practices that could be applied will mean a more productive selection - merely that the organisation, strategy, reporting and so on of the project is considered part of the project and not a predetermined factor. All options being open, and context being the ultimate arbiter of the value of an activity to a project, it's interesting to wonder whether there is anything that is indisputably never appropriate. Perhaps burning our test mat...