Hiccupps

How do you test a product with distributed logic and non-deterministic components? Well, it depends on the product and when the question is asked. This post is one answer, for now. A service I work on exposes an API for internal and external clients and calls multiple other services. It orchestrates a reasonably complex journey for a user, with their inputs and the responses of the services determining each step.  Our customers can configure the service to provide a variant of the standard journey for their users. This gives them choices on the available steps, the order of the steps, and, importantly, some of the logic for taking particular steps. This configuration lives in a different repository to the service, not owned by us, and it can be changed and deployed independently from the service. As you can imagine, we test our service in multiple ways at different granularities using the usual tooling, including unit tests fo...

  Testing is inherently about looking because, put simply:  If you don't look, you're not likely to find. The interesting challenge is to look in the right way at the right place at the right time. This is what motivates any kind of intentional testing: how can we put ourselves in a strong position to look for, see, and — crucially —  recognise  the things that matter to the people that matter, when they matter?   --00-- Tools can help with this mission: using tools we can look more deeply, more broadly, for more complex patterns, for harder-to-spot traces, faster, more often, more efficiently, and so on.  There's a trade-off, naturally. As the tool takes us further from the material being worked on we must either trust it more or check its results more thoroughly ... to the extent that we care about the results. At a crude level, think of it as a spectrum. At one end we might have a knife. It's a tool I can use ...

There was a time when testing was all about the mnemonics . Well, we had no AI back then so thinking about our human craft and how to share what we had learned using our human intelligence with other engaged humans for later recall in their human heads seemed important. But this post isn't about dumping on AI. It isn't about mnemonics either even if WTTWTH does look like something that'd fit well into that ancient world.  No, this post is just a snappier version of what I said to my team this week when I was walking through some testing I'd done the day before. It was concerned with a change to a particular turn in the dialog system we're building where multiple variables are in play, some populated by an external call to an LLM service.  I wanted to make the point that the bulk of the testing work was done in the research I did and the spreadsheet I made, not in the interaction with our product. That spreadsheet was the result of me looking in our service's cod...

The Wikipedia page on infinite loops in programming describes them as "a sequence of instructions that, as written, will continue endlessly, unless an external intervention occurs." One common example might be a while loop whose exit condition is never met, and needs to be aborted by a human pressing Control-C . With that concept in mind, we can make an easy analogy to software development where the same kinds of events happen over and over and over until our product becomes irrelevant, or uneconomic, or our organisation closes down, and the loop is exited.  Inside the loop, the world in which our product exists will change, the market in that world will change, the requirements on our product in the market will change, the product itself changes as our teams add features, or fix bugs, or update libraries, or run on new platforms, and so on.  So, for their lifetimes, our products inhabit an infinite loop of change and, if the...

I'm been working on an application that will orchestrate data from multiple services. As the developers add clients for those services, they have been writing integration tests and, naturally, many of them use mocked data. Mostly the data consists of non-trivial JSON response bodies full of domain-specific terminology captured during interactions with the other services. Consequently, many of our tests reflect this complexity and domain-specificity by asserting on its data structures and particular terms. This is functionally fine, but problematic for readability because test intent can be hidden in a mass of incomprehensible word salad. Again, this is usually fine for the author when writing the test because the intent is front of mind but it's problematic for other readers, including the author later. I have been vocal about this drawback and today one of my colleagues asked me to summarise my prescription for it. Without thinking I said this, and I ...

  My friend Mirek wrote an interesting post recently: Where Rust fits for me . In it, he made a hierarchy of the programming languages he reaches for on a regular basis and why he picks a particular one for any specific task. I'd summarise it crudely like this: Shell: only for very basic setup. Anything with non-trivial logic belongs somewhere else. Python: his go-to but with caveats about the expected lifetime of the code: if it needs to persist without regular attention then Python won't do. Rust: Whatever doesn't fit in the other two categories.  I enjoy seeing people be thoughtful about their tools and reflect on the way that they work. Understanding what you do and why is a helpful first step on improving what you do and why, should you want to. And I always want to. So I thought I'd attempt the same kind of analysis and started listing languages that I use regularly: Groovy : I use Groovy exclusively in Jenkins pipelines. It's useful to have a little familiar...

In a recent episode of the Vernon Richard show , testing's dynamic duo were inspired by Valentine's Day to talk about their love for our craft. On the topic of tools, Richard sings the praises of Selenium, but Vernon takes a different tack, instead talking about tool creation ... and me!  I worked at the same company as Vern for a couple of years and we'd find time to pair on something most weeks even though our responsibilities didn't overlap at all. Sometimes he'd bring a problem, sometimes it'd be show-and-tell about what we were working on, occasionally it was about people and process, but what really lit him up was when we looked at code together, particularly if we built something. I was out walking and listening to podcasts when the episode popped up in my feed. Honestly, it was a strange feeling to hear myself being talked about, even though Vern gave me a heads-up that he'd done it just after it was recorded. What he said is nothing that he hasn...

Why wouldn't you just give up? The system under test has poor testability which means that the testing you'd like to do will take longer and the resolution of your findings will be lower than you'd like. So do you give up, battle through, wait for someone to add what you need ... or change the product yourself ? There are potential risks to that last option, of course, because (duh!) you're changing the product. But there are potential wins, too, because you're getting the data you want earlier and can give richer feedback to your stakeholders. I took that route this week.  The service I'm looking at is essentially a pipeline of steps, each of which calls out to a third-party service and post-processes the result, set up like this: result1 = step1.run(input) result2 = step2.run(result1) result3 = step3.run(result2) I wanted the response time for a large number of requests against its API, which I can get easily from the client ...

  I often review and collaborate on unit tests at work. One of the patterns I see a lot is this: there are a handful of tests, each about a page long the tests share a lot of functionality, copy-pasted the test data is a complex object, created inside the test the test data varies little from test to test. In Kotlin-ish pseudocode, each unit test might look something like this: @Test fun `test input against response for endpoint` () { setupMocks() setupTestContext() ... val input = Object(a, OtherObject(b, c), AnotherObject(d)) ... val response = someHttpCall(endPoint, method, headers, createBodyFromInput(input) ) ... val expected = Object(w, OtherObject(x, y), AnotherObject (z)) val output = Object(process(response.getField()), otherProcess(response.getOtherField()), response.getLastField()) assertEquals(expected, output) } ... While these tests are generally functional, and I rarely have reason to doubt that ...

The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "If testers can’t code, they’re of no use to us" My first reaction is to wonder what you expect from your testers. I am immediately interested ...

My team has been building a new service over the last few months. Until recently all the data it needs has been ingested at startup and our focus has been on the logic that processes the data, architecture, and infrastructure. This week we introduced a couple of new endpoints that enable the creation (through an HTTP POST) and update (PUT) of the fundamental data type (we call it a definition ) that the service operates on. I picked up the task of smoke testing the first implementations. I started out by asking the system under test to show me what it can do by using Postman to submit requests and inspecting the results. It was the kinds of things you'd imagine, including: submit some definitions (of various structure, size, intent, name, identifiers, etc) resubmit the same definitions (identical, sharing keys, with variations, etc) retrieve the submitted definitions (using whatever endpoints exist to show some view of them) compare definitions I submitted fro...