Hiccupps

How do you test a product with distributed logic and non-deterministic components? Well, it depends on the product and when the question is asked. This post is one answer, for now. A service I work on exposes an API for internal and external clients and calls multiple other services. It orchestrates a reasonably complex journey for a user, with their inputs and the responses of the services determining each step.  Our customers can configure the service to provide a variant of the standard journey for their users. This gives them choices on the available steps, the order of the steps, and, importantly, some of the logic for taking particular steps. This configuration lives in a different repository to the service, not owned by us, and it can be changed and deployed independently from the service. As you can imagine, we test our service in multiple ways at different granularities using the usual tooling, including unit tests fo...

  Testing is inherently about looking because, put simply:  If you don't look, you're not likely to find. The interesting challenge is to look in the right way at the right place at the right time. This is what motivates any kind of intentional testing: how can we put ourselves in a strong position to look for, see, and — crucially —  recognise  the things that matter to the people that matter, when they matter?   --00-- Tools can help with this mission: using tools we can look more deeply, more broadly, for more complex patterns, for harder-to-spot traces, faster, more often, more efficiently, and so on.  There's a trade-off, naturally. As the tool takes us further from the material being worked on we must either trust it more or check its results more thoroughly ... to the extent that we care about the results. At a crude level, think of it as a spectrum. At one end we might have a knife. It's a tool I can use ...

I'm continuing to be inspired by Patrick Prill's series of posts on the way AI is impacting personal, societal, and economic systems and today that leads me to share another short analogy. See also Hedging Your Bets  and A SpotifAI Model? Into the Groove by Jonathan Scott recounts the history of recorded music. You probably already have some background awareness of a timeline something like this: wax cylinders were the early commercial format, overtaken later by 78 rpm shellac discs which in turn preceded vinyl 33s and 45s, before CDs and now streaming. What you may not be aware of is the explosion of formats, players, converters, novelty features, quality innovations, pricing strategies, lawsuits, and mergers that lead ultimately to standardisation and consolidation.  Formats: the medium is one obvious difference between formats but there are others including the needle, the type of amplification, the speed of rotation, whether the needle moves left/right or up/down in the ...

My friend Patrick Prill continues to dissect the growth, use, and risks of AI in our industry at the rate of an article or two a day, each one teasing out a particular thread from the very tangled issue ball. ( Here's yesterday's.) I recommend reading them for a thoughtful and detailed analysis and reflection on today's software and social ecosystems and the ways that AI is reshaping them.  They certainly make me think and, as with last week's Hedging Your Bets on AI , I find it helpful to capture those thoughts as analogies, so here's another. A very long time ago I wrote a music fanzine: A5, photocopied, 50p or a quid, sold by a scruffy long-hair in a German army jacket out of a battered record bag in the back rooms of pubs between bands. I remember well when domestic studio and CD-R technology became affordable because the volume of submissions for review increased dramatically ... and the overall quality likewise decreased. The technolog...

My friend and colleague Patrick Prill has been writing articles about AI in our industry at a prodigious rate over the last couple of weeks. (Here's  the second of yesterday's pieces.) I recommend reading them for a thoughtful and detailed analysis and reflection on today's software and social ecosystems and the ways that AI is reshaping them. I can't compete with the breadth, depth, or frequency of his output so, although this post is on the same topic, I offer only a short analogy.  Step back in time 60, 70 or 80 years and wander through England's pleasant pastures. You will see lanes and fields bordered with hedgerows maintained by skilled workers, living and breathing boundaries, functional for keeping stock in and intruders out but also a component of the local ecological web. Hand-laid hedges are habitats. Birds nest in them, rodents and small mammals use them like a road network to travel safely around the landscape, larger animals use them as windbreaks or f...

In 2024 I wrote a post, An AI Red Light , about a creative project I did with my daughter using both AI and "traditional" tooling. This was my conclusion: I think we were pragmatic in our choice of tools. Where our vision required control (video) we used a tool that gave us that control. Where we were happy to cede some control in exchange for output at a quality we could not produce ourselves (vocals) we used a different tool.  I didn't set out to do some kind of AI experiment. Instead I set out to have some fun with my daughter and found tools that enabled us to produce something, some art, that we're proud of and I'm happy is our work. For sure, we were sometimes more creators and sometimes more curators, but at no point were we ever mere spectators.  Two years down the road I have access to, and use, LLMs in my work. With them, I have been able to make things that I would not have done without them such as the user interface on my model-based dialog walker .  ...

In Your job is to deliver code you have proven to work  Simon Willison writes: As software engineers we ... need to deliver code that works — and we need to include proof that it works as well.  He is coming at this from the perspective of LLM-assisted coding, but most of what he says applies in general. I think this is a reasonable consise summary of his requirements for developers: Manual happy paths: get the system into an initial state, exercise the code, check that it has the desired effect on the state. Manual edge cases: no advice given, just a note that skill here is a sign of a senior engineer.  Automated tests: should demonstrate the change like Manual happy paths  but also fail if the change is reverted.  He notes that, even though LLM tooling can write automated tests, it's humans who are accountable for the code and it's on us to "include evidence that it works as it should." Coincidentally, just the week before I read his post I told one of my...

We are generally not the target users of the software products we work on.  That's not to say we never use our applications or that we have no interest in those who do, but mostly we rely on feedback from elsewhere to tell us whether needs are being met.  Sure, as testers we'll interact with the software and maybe even consider ourselves to be a proxy for users. Yes, we'll probably have people in product and business roles translating, or inventing, customer requirements for us. And, yes, perhaps we'll even dogfood the stuff we build sometimes, to some extent. I've lost count of the number of times I've asked developers whether they ran the software after their changes and been told that the tests pass. I've lost count of the number of colleagues I've had who work on a service and have little or no idea what clients it has and what kinds of tasks their users are trying to complete or why. I've lost count of the number of times I've been in argume...

  Recently a few people have asked me how I test AI. I'm happy to share my experiences, but I frame the question more broadly, perhaps something like this: what kinds of things do I consider when testing systems with artificial intelligence components .  I freestyled liberally the first time I answered but when the question came up again I thought I'd write a few bullets to help me remember key things. This post is the latest iteration of that list. Caveats: I'm not an expert; what you see below is a reminder of things to pick up on during conversations so it's quite minimal; it's also messy; it's absolutely not a guide or a set of best practices; each point should be applied in context; the categories are very rough; it's certainly not complete.  Also note that I work with teams who really know what they're doing on the domain, tech, and medical safety fronts and some of the things listed here are things they'd typically do some or all of. Testing ...

For over a year the roadworks near our house have been a riot of signage, inspiring me and my daughter to make silly songs using their words for lyrics as we walked to school.  Then she got the idea that we should make a "proper" song. So I downloaded n-Track and we hacked together a techno instrumental over a few evenings. Unfortunately, recording decent quality vocals at home without much equipment is non-trvial and then real life intervened anyway so the project stopped. A few months later I came aross Suno , an AI song generator, and had a lightbulb moment. Suno exposes a prompt for musical style and a prompt for lyrics so we had it make a song, When Red Light Shows , based on the signs. I think it was my daughter that suggested we should make a video too, so we did, with a non-AI tool, CapCut . The lyrics came from signs, the music came from AI, and we made the images yet I feel that we were creative across all of those areas. Reflecting on our experience, ...

The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00--  "Why don’t we replace the testers with AI?" We have a good relationship so I feel safe telling you that my instinctive reaction, as a member of the T...

  One of my chores takes me outside early in the morning and, if I time it right, I get to hear a charming chorus of birdsong from the trees in the gardens down our road, a relaxing layered soundscape of tuneful calls, chatter, and chirrupping. Interestingly, although I can tell from the number and variety of trills that there must be a large number of birds around, they are tricky to spot. I have found that by staring loosely at something, such as the silhouette of a tree's crown against the slowly brightening sky, I see more birds out of the corner of my eye than if I scan to look for them. The reason seems to be that my peripheral vision picks up movement against the wider background that direct inspection can miss. An optometrist I am not, but I do find myself staring at data a great deal, seeking relationships, patterns, or gaps. I idly wondered whether, if I filled my visual field with data, I might be able to exploit my peripheral vision in that quest. I have a wide monito...

I'm definitely no expert in Large Language Models but expertise is not required, given all the publicity, to know that variation in the responses from the technology is expected.  To illustrate that, I asked ChatGPT  "In one sentence, why do LLM responses vary on the same prompt?" and got three different, although clearly related and overlapping, answers: LLM responses vary on the same prompt due to the model's stochastic nature, diverse training data, and sensitivity to slight changes in input phrasing.  LLM responses vary on the same prompt due to the model's probabilistic nature, employing diverse sampling techniques and leveraging its extensive training data to generate a range of contextually plausible and grammatically coherent outputs. LLM responses vary on the same prompt due to the model's inherent stochasticity and diverse training data, leading to dif...