Hiccupps

Maaret Pyhäjärvi gave a talk, Exploratory Testing Reimagined , last week.  There's a  blog post  to go with it, so I won't say much on top of what's in my sketchnotes above, just pull out a few reflections. I don't have the data or wide range of experience reports that Maaret does, but my own anecdotal results from using AI tools for testing align closely with what she says in the talk. I have created skills to assist me, and my colleagues, in our testing work. For example, I used the contents of this blog and five years of test notes in my current role to distil a conversational workflow, facilitated by Claude, which runs through risk identification, test idea generation, and test mission creation. All four skills (the facilitation is its own skill) are backed by a fifth, on general testing theory and practice, and a sixth about how I like to interact with the AI: which notably includes that I want it to be concise, evidence-backed, well-structured, and critical. I st...

There was a time when testing was all about the mnemonics . Well, we had no AI back then so thinking about our human craft and how to share what we had learned using our human intelligence with other engaged humans for later recall in their human heads seemed important. But this post isn't about dumping on AI. It isn't about mnemonics either even if WTTWTH does look like something that'd fit well into that ancient world.  No, this post is just a snappier version of what I said to my team this week when I was walking through some testing I'd done the day before. It was concerned with a change to a particular turn in the dialog system we're building where multiple variables are in play, some populated by an external call to an LLM service.  I wanted to make the point that the bulk of the testing work was done in the research I did and the spreadsheet I made, not in the interaction with our product. That spreadsheet was the result of me looking in our service's cod...

The Wikipedia page on infinite loops in programming describes them as "a sequence of instructions that, as written, will continue endlessly, unless an external intervention occurs." One common example might be a while loop whose exit condition is never met, and needs to be aborted by a human pressing Control-C . With that concept in mind, we can make an easy analogy to software development where the same kinds of events happen over and over and over until our product becomes irrelevant, or uneconomic, or our organisation closes down, and the loop is exited.  Inside the loop, the world in which our product exists will change, the market in that world will change, the requirements on our product in the market will change, the product itself changes as our teams add features, or fix bugs, or update libraries, or run on new platforms, and so on.  So, for their lifetimes, our products inhabit an infinite loop of change and, if the...

Maaret Pyhäjärvi  posted the quote above on LinkedIn a few weeks ago. It speaks strongly to me so I asked Maaret if she'd written more (because she's written a lot ) on this specific point. She hasn't, and the sentence keeps coming back into my head and I'd like to understand why, so I thought I'd try to write down what I take from it. I think it's easy to skim read as some kind of definition of exploratory testing but that would be a mistake in my eyes.  Testing by Exploring  summarises how I felt last time I went into the definition in any depth and, for me, Maaret's quote is concerned with the why  but says nothing of the what or how . But let's say we have a shared definition of exploratory testing, would I make this statement this baldly generally? No, I probably would not. Why? First, it's written in very personal terms (" my time", "the best testing I could") and, second, as a  contex...

Why wouldn't you just give up? The system under test has poor testability which means that the testing you'd like to do will take longer and the resolution of your findings will be lower than you'd like. So do you give up, battle through, wait for someone to add what you need ... or change the product yourself ? There are potential risks to that last option, of course, because (duh!) you're changing the product. But there are potential wins, too, because you're getting the data you want earlier and can give richer feedback to your stakeholders. I took that route this week.  The service I'm looking at is essentially a pipeline of steps, each of which calls out to a third-party service and post-processes the result, set up like this: result1 = step1.run(input) result2 = step2.run(result1) result3 = step3.run(result2) I wanted the response time for a large number of requests against its API, which I can get easily from the client ...

I'm reading Rick Rubin's The Creative Act: A Way of Being . It's spiritual without being religious, simultaneously vague and specific, and unerringly positive about the power and ubiquity of creativity.  We artists — and we are all artists he says — can boost our creativity by being open and welcoming to knowledge and experiences and layering them with past knowledge and experiences to create new knowledge and experiences.  If that sounds a little New Age to you, well it does to me too, yet also fits with how I think about how I work. This is in part due to that vagueness, in part due to the human tendency to pattern-match, and in part because it's true. I'm only about a quarter of the way through the book but already I am making connections to things that I think and that I have thought in the past. For example, in some ways it resembles essay-format Oblique Strategy cards and I wrote about the potential value of them to testers 12 years ago. This week I found the...

On Friday, in the weekly ensemble I have with some of the medical practitioners at work, I suggested we take a look at the first challenge from The Testing Map .  We all spend a lot of time testing although the others focus more on the domain side than the pure software side. That focus was reflected in the kinds of checks they tended towards: input strings that have obvious human semantics but which, for a seasoned software tester, would probably fall into a single equivalence class. While exploring some those strings we stumbled into security testing with the input "I don't know" because, due to the apostrophe, the testing challenge credited us with an attempted SQL injection. From there we were able to talk about script and HTML injection, and that slid into opening the developer tools in our browser and poking around the source, network traffic, cookies, and so on. The consensus when debriefing was that coverage can be improved when looki...

"How did you work that out so quickly?" my friend asked earlier this week when I sent him a handful of additional ways to view the problem we'd been pairing on.   "Stack Overflow and lots of small, fast, cheap experiments," I replied. And that's often true. It might not be Stack Overflow, but it's usually breaking a problem down, iterating on coherent chunks of it, composing the pieces into larger chunks, and building back to the original problem. The approach is reliable for me, but asking if it will be quick is like asking how long the proverbial string is . Sometimes it takes longer and I need to iterate on the approach itself, finding other "seams" along which to break the problem apart into units that can be addressed independently.  Sometimes I just give up and wait for the next opportunity to learn . Whether this is the right thing depends on all sorts of factors including whether I need an answer, whether I need the answer,...

  I often review and collaborate on unit tests at work. One of the patterns I see a lot is this: there are a handful of tests, each about a page long the tests share a lot of functionality, copy-pasted the test data is a complex object, created inside the test the test data varies little from test to test. In Kotlin-ish pseudocode, each unit test might look something like this: @Test fun `test input against response for endpoint` () { setupMocks() setupTestContext() ... val input = Object(a, OtherObject(b, c), AnotherObject(d)) ... val response = someHttpCall(endPoint, method, headers, createBodyFromInput(input) ) ... val expected = Object(w, OtherObject(x, y), AnotherObject (z)) val output = Object(process(response.getField()), otherProcess(response.getOtherField()), response.getLastField()) assertEquals(expected, output) } ... While these tests are generally functional, and I rarely have reason to doubt that ...

My team has been building a new service over the last few months. Until recently all the data it needs has been ingested at startup and our focus has been on the logic that processes the data, architecture, and infrastructure. This week we introduced a couple of new endpoints that enable the creation (through an HTTP POST) and update (PUT) of the fundamental data type (we call it a definition ) that the service operates on. I picked up the task of smoke testing the first implementations. I started out by asking the system under test to show me what it can do by using Postman to submit requests and inspecting the results. It was the kinds of things you'd imagine, including: submit some definitions (of various structure, size, intent, name, identifiers, etc) resubmit the same definitions (identical, sharing keys, with variations, etc) retrieve the submitted definitions (using whatever endpoints exist to show some view of them) compare definitions I submitted fro...

  Last month I started thinking about slicing , my instinctive approach to looking for perspectives on a problem, an opinion, an observation, or anything else. This time around I've got an example to talk about. On Fridays I ensemble with a group of medical quality engineers and medical knowledge engineers. We learn from each other, about testing and about the domain. On and off recently we've looked at a project of theirs which aims to understand better what work they do, how they do it, and why it's that way, and then write it up for internal and external consumption. In one early session, with a wider group in the company, there was an extremely open and exciting conversation about what should be covered in this effort.  It was the kind of discussion that greenfield projects often have, before scope is nailed down, where the world seems ripe with possibility, no difficulties have been identified, and there is no talk of who will taking respons...

I happened across Exploratory Testing: Why Is It Not Ideal for Agile Projects? by Vitaly Prus this week and I was triggered. But why? I took a few minutes to think that through. Partly, I guess, I feel directly challenged. I work on an agile project (by the definition in the article) and I would say that I use exclusively exploratory testing. Naturally, I like to think I'm doing a good job. Am I wrong? After calming down, and re-reading the article a couple of times, I don't think so. 😸 From the start, even the title makes me tense. The ideal solution is a perfect solution, the best solution. My context-driven instincts are reluctant to accept the premise, and I wonder what the author thinks is an ideal solution for an agile project, or any project. I notice also that I slid so easily from "an approach is not ideal" into "I am not doing a good job" and, in retrospect, that makes me smile. It doesn't do any harm to be reminded that your cognitive bias...

  Kent Beck, in More What, Less How , identified an unknown. But how will he move towards knowing? By exploring in 3X: Explore/Expand/Extract. The rules of exploration are that it’s a numbers game. More experiments, more value. Try a little bit of a lot of things. One of those experiments will have dramatically more impact than anything to date. Follow up that. This speaks to me.  On Friday I picked up the latest build of a new service with a partially-implemented API. The ticket that produced it was for some internal plumbing and, after various smoke tests and other poking around, followed by a demo of my approach and findings to the team, and some fixes, I could have stopped looking at it. But I thought enough functionality might be enabled that I could begin to navigate the API a little so I tried hand-rolling requests in Postman. After some false starts and a couple of workarounds (for those unfinished pieces of the API) I was able to make the bare bones of a user journey....