Hiccupps

  My friend Mirek wrote an interesting post recently: Where Rust fits for me . In it, he made a hierarchy of the programming languages he reaches for on a regular basis and why he picks a particular one for any specific task. I'd summarise it crudely like this: Shell: only for very basic setup. Anything with non-trivial logic belongs somewhere else. Python: his go-to but with caveats about the expected lifetime of the code: if it needs to persist without regular attention then Python won't do. Rust: Whatever doesn't fit in the other two categories.  I enjoy seeing people be thoughtful about their tools and reflect on the way that they work. Understanding what you do and why is a helpful first step on improving what you do and why, should you want to. And I always want to. So I thought I'd attempt the same kind of analysis and started listing languages that I use regularly: Groovy : I use Groovy exclusively in Jenkins pipelines. It's useful to have a little familiar...

A couple of months ago, in Can You Hack It? , I wrote about how I increased the testability of a service by changing it in a way that allowed me to simulate the behaviour of one of its dependencies. With that in place I could force specific code paths to be followed and so explore different scenarios easily. That was sufficient for a quick and dirty experiment but, because I was changing code, each round was slower than I'd have liked as I had to edit, compile, run, and then test. --00-- When the next opportunity to work in that area came up, on a different service, I looked for an improvement to my test approach. I realised that I could remove the compile-run step by having some configuration that would specify the response from the external service. So I taught our product to look for the URL of a downstream service in an environment variable every time it wanted to call it. This gave me very precise control of the outgoing requests which I pointed at a local mock server called ...

Why wouldn't you just give up? The system under test has poor testability which means that the testing you'd like to do will take longer and the resolution of your findings will be lower than you'd like. So do you give up, battle through, wait for someone to add what you need ... or change the product yourself ? There are potential risks to that last option, of course, because (duh!) you're changing the product. But there are potential wins, too, because you're getting the data you want earlier and can give richer feedback to your stakeholders. I took that route this week.  The service I'm looking at is essentially a pipeline of steps, each of which calls out to a third-party service and post-processes the result, set up like this: result1 = step1.run(input) result2 = step2.run(result1) result3 = step3.run(result2) I wanted the response time for a large number of requests against its API, which I can get easily from the client ...

This week I was working with a developer to make a change to a legacy codebase in an area neither of us are very familiar with. The need is easy to describe in general and some occurrences of the behaviour we want to alter are common, straightforward to identify in use, and clear in the code. Unfortunately, the logic in the application is complex, the data used is domain-specialised and the behaviour we are interested in can occur in extremely specific combinations that are hard for a layperson to predict. I had no confidence that cases we knew were all of the cases. My colleague did a round of work and asked me to take a look. I exercised the application in a few ways while inspecting its logs in real time so that I could see the effect of the changes (or not) immediately. This gave up some rarer examples which had not been covered in the code. I added a couple of tests to characterise them and he identified another code change. Next, rather than continue exercising the product lookin...

My team has been building a new service over the last few months. Until recently all the data it needs has been ingested at startup and our focus has been on the logic that processes the data, architecture, and infrastructure. This week we introduced a couple of new endpoints that enable the creation (through an HTTP POST) and update (PUT) of the fundamental data type (we call it a definition ) that the service operates on. I picked up the task of smoke testing the first implementations. I started out by asking the system under test to show me what it can do by using Postman to submit requests and inspecting the results. It was the kinds of things you'd imagine, including: submit some definitions (of various structure, size, intent, name, identifiers, etc) resubmit the same definitions (identical, sharing keys, with variations, etc) retrieve the submitted definitions (using whatever endpoints exist to show some view of them) compare definitions I submitted fro...

  So much of what I want to do with automation is repetition of slight variants. I was reminded of it again last week when I was looking for a way to list the versions of certain dependencies across many versions of a product my team owns. These dependencies are exposed through the running product's API on an "about" endpoint. They can also be found in our Gradle configuration file, build.gradle , something like this: thingVersion = System.getenv( 'THING_VERSION' ) ?: '4.0' otherVersion = System.getenv( 'OTHER_VERSION' ) ?: '2.3.1' anotherVersion = System.getenv( 'ANOTHER_VERSION' ) ?: "7.6.0" Conceptually I need to run through released versions of the product, interrogate each for the  dependencies, and aggregate them into some easily-parsable format. If I was only after a couple of versions of the product I'd just check out the code for those versions, inspect the build.gradle files, and copy...

The product I'm working on at the moment is a Ada, a symptom checking app . The basic idea is that users enter a few details about themselves and their current symptoms and are then guided through a series of questions which leads to them being given a list of probability-ranked conditions they might have based on their answers. As you'd expect in this field, with this kind of data, a lot of care and concern is taken to understand how the app performs and what caused any changes in that performance across releases. There are many layers of testing. In one of those layers we have medical test cases. (And these are literally cases in the medical sense.) Each one represents data about an individual who might present to a doctor with a particular set of symptoms, a given medical history, possible comorbidities and so on. Each also comes with a set of acceptable condition suggestions and other expectations about how the software should behave when asking this kind of user questions...

I've been working on my talk for the Testing, Diversity, AI conference run by the Software Testing interest group of the British Computer Society.  In it, I'm thinking about the tooling I built to help me explore a chatbot API. It exploits random choice to walk through the extremely large space of possible chats in a medical symptom checking application. As I reflected on the combination of tools and testing I found it convenient to label three activities that involve both. Navigat e Navigation is about finding a path to an endpoint. While navigating I am very interested to notice assumptions I'm making, workarounds that are required, and any questions that come to mind, but my main focus is on reaching the goal. In the first instance, on this project, I needed a basic framework that would enable my code to start a chat, walk through all of the interactions with the service, and stop. As I was writing code to create my initial ...

This week I've been playing with altwalker , a model-based testing tool. To get the hang of it, I attempted to build a very simple model of a workflow that is supported by the service my team owns. Hacking away at the example code, and looking frequently at the docs, I was able to get up and running in a few hours, creating: a basic model: nodes for system states, edges for operations simple assertions: mainly consistency checks on the states client: HTTP client to implement the operations against the service's API I configured this so that altwalker will perform a random walk of the model, starting state data is randomised, and the client will choose randomly whenever offered an option. Why so much randomness? Because it means that, over successive runs, more of the infinite space of possible workflow executions will be covered. Once I had that basically working I wrote a shell script that would run this loop a number of times: call altwalker ...

Ben Dowen pinged me and others on Twitter last week , asking for "a nice concise resource to link to for a blog post - about taking good Testing notes." I didn't have one so I thought I'd write a few words on how I'm doing it at the moment for my work at Ada Health, alongside Ben. You may have read previously that I use a script to upload Markdown-based text files to Confluence . Here's the template that I start from: # Date + Title # Mission # Summary WIP! # Notes Then I fill out what I plan to do. The Mission can be as high or low level as I want it to be. Sometimes, if deeper context might be valuable I'll add a Background subsection to it. I don't fill in the Summary section until the end. It's a high-level overview of what I did, what I found, risks identified, value provided, and so on. Between the Mission and Summary I hope that a reader can see what I initially intended and what actually...

On Fridays I pair with doctors from Ada 's medical quality team. It's a fun and productive collaboration where I gain deeper insight into the way that diagnostic information is encoded in our product and they get to see a testing perspective unhindered by domain knowledge. We meet at the same time each week and decide late on our focus, choosing something that one of us is working on that's in a state where it can be shared. This week we picked up a task that I'd been hoping to get to for a while: exploring an API which takes a list of symptoms and returns a list of potential medical conditions that are consistent with those symptoms.  I was interested to know whether I could find small input differences that led to large output differences. Without domain knowledge, though, I wasn't really sure what "small" and "large" might mean. I prepared an input payload and wrote a simple shell script which did the following: make a...

Last week I started a new job. The team I've joined owns a back-end service and, along with all the usual onboarding process, inevitable IT hassles, and necessary context-gathering, one of my goals for my first week was to get a local instance of it running and explore the API. Which I did. Getting the service running was mostly about ensuring the right tools and dependencies were available on my machine. Fortunately the team has wiki checklists for that stuff, and my colleagues were extremely helpful when something was missing, out of date, or needed an extra configuration tweak. Starting to explore the service was boosted by having ReDoc for the endpoints and a Postman collection of example requests against them. I was able to send requests, inspect responses, compare both to the doc, and then make adjustments to see what effects they had. If that's testing of any kind, it's probably what I call pathetic testing : There's this mental ima...

This week I wanted to monitor several pieces of software that talk to one another via HTTP and HTTPS. All are running on the same machine, three are Linux services, and one is a standalone script. I was interested in being able to see all of the communications between them in one place, in time order. I know a couple of ways of capturing this kind of data: proxying and network sniffing .  My default approach would be to have the applications configured to proxy via Fiddler running on my laptop inside the work network. Easy, right? Err, no, because I had forgotten that the machine in question is on a network that isn't considered secure and firewalls prevent that connection. In my proof of concept experiment, the standalone script just hung failing to find the proxy I had specified. Interesting behaviour, and I later reported it, but not what I needed right then. Next! As all of the software is on the same machine, capturing network traffic int...