Hiccupps

You know when you suddenly discover that you have an internal model of a chunk of the world that's been guiding your thinking and behaviour but you never realised?  I love that. The most recent instance for me was a couple of weeks ago when asked what problems I have with test data at work. That's when it became obvious that I have three broad categories of test data, and want something different from each of them. Of course I do! Duh. And why didn't I realise this before? Not sure. Perhaps I've never framed that kind of high-level abstract question, Maybe more often I'm interested in test data for the actual problem in front of me? Whatever, here's my initial draft of what I think I think... Data for specific logic. I use this in traditional regression tests that need a static expected result: unit tests, for instance. I see a lot of examples that have clearly grabbed runtime application data that's the right shape and b...

How do you test a product with distributed logic and non-deterministic components? Well, it depends on the product and when the question is asked. This post is one answer, for now. A service I work on exposes an API for internal and external clients and calls multiple other services. It orchestrates a reasonably complex journey for a user, with their inputs and the responses of the services determining each step.  Our customers can configure the service to provide a variant of the standard journey for their users. This gives them choices on the available steps, the order of the steps, and, importantly, some of the logic for taking particular steps. This configuration lives in a different repository to the service, not owned by us, and it can be changed and deployed independently from the service. As you can imagine, we test our service in multiple ways at different granularities using the usual tooling, including unit tests fo...

In Looking Good, Testers!  I made the case that testing is about looking and that delegating the looking to someone or something  else involves a risk calculation: do you both understand the task the same way, is the third party likely to carry out the task in the way you requested, how easily can you check any results they give back, could they complement you in some way, how will you keep your knowledge and skills up to date if you don't do this task any more, how much does this task matter, how much do you care about any of that, ...? I didn't say anything there about selecting what to look at or for  ... and I'm not going to talk about it here either because it's essentially the whole of testing. I will note one thing, though: to find things other people don't find,  look where other people don't look .  What I do want to cover here is looking at things that, arguably, you don't need to.  My team was recently asked to add a new layer to a service we ow...

There was a time when testing was all about the mnemonics . Well, we had no AI back then so thinking about our human craft and how to share what we had learned using our human intelligence with other engaged humans for later recall in their human heads seemed important. But this post isn't about dumping on AI. It isn't about mnemonics either even if WTTWTH does look like something that'd fit well into that ancient world.  No, this post is just a snappier version of what I said to my team this week when I was walking through some testing I'd done the day before. It was concerned with a change to a particular turn in the dialog system we're building where multiple variables are in play, some populated by an external call to an LLM service.  I wanted to make the point that the bulk of the testing work was done in the research I did and the spreadsheet I made, not in the interaction with our product. That spreadsheet was the result of me looking in our service's cod...

The Wikipedia page on infinite loops in programming describes them as "a sequence of instructions that, as written, will continue endlessly, unless an external intervention occurs." One common example might be a while loop whose exit condition is never met, and needs to be aborted by a human pressing Control-C . With that concept in mind, we can make an easy analogy to software development where the same kinds of events happen over and over and over until our product becomes irrelevant, or uneconomic, or our organisation closes down, and the loop is exited.  Inside the loop, the world in which our product exists will change, the market in that world will change, the requirements on our product in the market will change, the product itself changes as our teams add features, or fix bugs, or update libraries, or run on new platforms, and so on.  So, for their lifetimes, our products inhabit an infinite loop of change and, if the...

In Your job is to deliver code you have proven to work  Simon Willison writes: As software engineers we ... need to deliver code that works — and we need to include proof that it works as well.  He is coming at this from the perspective of LLM-assisted coding, but most of what he says applies in general. I think this is a reasonable consise summary of his requirements for developers: Manual happy paths: get the system into an initial state, exercise the code, check that it has the desired effect on the state. Manual edge cases: no advice given, just a note that skill here is a sign of a senior engineer.  Automated tests: should demonstrate the change like Manual happy paths  but also fail if the change is reverted.  He notes that, even though LLM tooling can write automated tests, it's humans who are accountable for the code and it's on us to "include evidence that it works as it should." Coincidentally, just the week before I read his post I told one of my...

Maaret Pyhäjärvi  posted the quote above on LinkedIn a few weeks ago. It speaks strongly to me so I asked Maaret if she'd written more (because she's written a lot ) on this specific point. She hasn't, and the sentence keeps coming back into my head and I'd like to understand why, so I thought I'd try to write down what I take from it. I think it's easy to skim read as some kind of definition of exploratory testing but that would be a mistake in my eyes.  Testing by Exploring  summarises how I felt last time I went into the definition in any depth and, for me, Maaret's quote is concerned with the why  but says nothing of the what or how . But let's say we have a shared definition of exploratory testing, would I make this statement this baldly generally? No, I probably would not. Why? First, it's written in very personal terms (" my time", "the best testing I could") and, second, as a  contex...

I'm been working on an application that will orchestrate data from multiple services. As the developers add clients for those services, they have been writing integration tests and, naturally, many of them use mocked data. Mostly the data consists of non-trivial JSON response bodies full of domain-specific terminology captured during interactions with the other services. Consequently, many of our tests reflect this complexity and domain-specificity by asserting on its data structures and particular terms. This is functionally fine, but problematic for readability because test intent can be hidden in a mass of incomprehensible word salad. Again, this is usually fine for the author when writing the test because the intent is front of mind but it's problematic for other readers, including the author later. I have been vocal about this drawback and today one of my colleagues asked me to summarise my prescription for it. Without thinking I said this, and I ...

We are generally not the target users of the software products we work on.  That's not to say we never use our applications or that we have no interest in those who do, but mostly we rely on feedback from elsewhere to tell us whether needs are being met.  Sure, as testers we'll interact with the software and maybe even consider ourselves to be a proxy for users. Yes, we'll probably have people in product and business roles translating, or inventing, customer requirements for us. And, yes, perhaps we'll even dogfood the stuff we build sometimes, to some extent. I've lost count of the number of times I've asked developers whether they ran the software after their changes and been told that the tests pass. I've lost count of the number of colleagues I've had who work on a service and have little or no idea what clients it has and what kinds of tasks their users are trying to complete or why. I've lost count of the number of times I've been in argume...

  My friend Mirek wrote an interesting post recently: Where Rust fits for me . In it, he made a hierarchy of the programming languages he reaches for on a regular basis and why he picks a particular one for any specific task. I'd summarise it crudely like this: Shell: only for very basic setup. Anything with non-trivial logic belongs somewhere else. Python: his go-to but with caveats about the expected lifetime of the code: if it needs to persist without regular attention then Python won't do. Rust: Whatever doesn't fit in the other two categories.  I enjoy seeing people be thoughtful about their tools and reflect on the way that they work. Understanding what you do and why is a helpful first step on improving what you do and why, should you want to. And I always want to. So I thought I'd attempt the same kind of analysis and started listing languages that I use regularly: Groovy : I use Groovy exclusively in Jenkins pipelines. It's useful to have a little familiar...

  Recently a few people have asked me how I test AI. I'm happy to share my experiences, but I frame the question more broadly, perhaps something like this: what kinds of things do I consider when testing systems with artificial intelligence components .  I freestyled liberally the first time I answered but when the question came up again I thought I'd write a few bullets to help me remember key things. This post is the latest iteration of that list. Caveats: I'm not an expert; what you see below is a reminder of things to pick up on during conversations so it's quite minimal; it's also messy; it's absolutely not a guide or a set of best practices; each point should be applied in context; the categories are very rough; it's certainly not complete.  Also note that I work with teams who really know what they're doing on the domain, tech, and medical safety fronts and some of the things listed here are things they'd typically do some or all of. Testing ...

A couple of months ago, in Can You Hack It? , I wrote about how I increased the testability of a service by changing it in a way that allowed me to simulate the behaviour of one of its dependencies. With that in place I could force specific code paths to be followed and so explore different scenarios easily. That was sufficient for a quick and dirty experiment but, because I was changing code, each round was slower than I'd have liked as I had to edit, compile, run, and then test. --00-- When the next opportunity to work in that area came up, on a different service, I looked for an improvement to my test approach. I realised that I could remove the compile-run step by having some configuration that would specify the response from the external service. So I taught our product to look for the URL of a downstream service in an environment variable every time it wanted to call it. This gave me very precise control of the outgoing requests which I pointed at a local mock server called ...