Skip to main content

What Did I Do?


What did I do? 

It's a natural question to ask when you get made redundant, which I just did, and so I am. 

But I'm not pondering it in a why me? kind of a way. I'm good at my job and was valuable to the company and I understand that the business will always do what it believes benefits the business.

But if there's no soul-searching right now, there is sadness, just as there was in previous rounds of downsizing at the company. Then, I was sad for those who were let go but this time I find that I'm feeling all the feelings for the people I've left behind.

What did I do?

It's still an interesting question to reflect on, but in the sense of what have I accomplished? What brought me pleasure? What did I learn? Perhaps the answers will influence where I go next, or perhaps it'll just be useful prep for future recruitment rounds.

What did I do? 

If you've been reading this blog for a while you'll suspect that there might be some data to review ... and there is. 

I made Ada's bureaucratic annual review process work for me by creating a spreadsheet formatted for easy copy-pasting into the forms and Miro board I was required to fill. I added to it incrementally across the year and I carried on even when the reviews fizzled out because I'm an instinctive collector.

I might need that data later!

Well, later has arrived and I started to use the data in my job search prep. I self-identify as a tester and my role title at Ada was Senior Expert Quality Engineer (!) but the data reinforces what I always knew: I have a lot of experience and I range widely.

I thought it might be interesting to share some highlights here.

What did I do?

At a high level, I did what I always do: try to understand the context, look for risks, seek opportunities to add value, make connections within and outside my team, share my knowledge and learn from others, support my colleagues by trying to help them make the best version of whatever they're working on given what we know and all the constraints. Also, take my share of whatever crap work just has to be done with a smile. (My Frame Your Thing) 

But at a more granular level, here's three of the categories I factored out of the spreadsheet.

Testing

Testing is my happy place. I love the mix of social, technical and intellectual challenges that testing brings, and it suits me so well. 
 
In my second week at Ada I ran a session in Community Day showing a tool I'd built for interactively exploring the search engine built into the main product. To be honest that's bread and butter for me and the results were the point, but for my colleagues a tester building a tool was something of a novelty.
  
I built many more tools, mostly ad hoc but some longer-lived, mostly personal but some that made it into general use. An example of the latter is the model-based walker that drives Ada's assessment engine using randomness to cover the state space, making assertions as it goes. I've lost count of the number of issues this tool found and the amount of reassurance the team took when it didn't.
 
Another innovation I am particularly proud of is a suite that took arbitrary combinations of product and complex configuration and ran a battery of checks to see whether core medical functionality was preserved. I had to argue strongly, over a long period, and through many technical conversations about the difficulties, to get it built. But it was built, and it worked.
 
Having a good instinct for when it might be worth following a thread comes with experience. On one occasion, after noticing an anomaly in some prod telemetry data, I dove deep into a set of issues that crossed all sorts of boundaries in the company. That included product (a performance issue), user behaviour (a potential safeguarding concern) and process (a hole in test coverage). I followed each issue to conclusion across the groups involved.

There are numerous other examples of self-initiated analysis that either had immediate effect or proved invaluable as background later. Examples include Ada's search technology, managing translations through the stack, representing age in different components, which assumptions are made where in the medical reasoning and how safe they are, gating logic in critical path pipelines, and building a map of the complex configuration possibilities of one of the products. 
 

Artificial Intelligence 

Ada introduced AI across the board reasonably early in both product and working practice. From the beginning I explored, with my medical colleagues, ways of testing our AI-enabled products including metamorphic approaches, statistics, adversarial AI, adversarial human, and large eval sets.  

Dial-a-Persona was a tool I conceived, prototyped in a hackathon, and then ran as a semi-official side-project with a cross-functional team of developers, medical staff, and data analysts. It lets domain experts configure very personas to be simulated by AI and run against the Ada product at scale, enabling bulk testing of fine-grained or helicopter-view features of the medical modelling that simply isn't possible with only human effort. 

I built skills for test planning based on my blog and five years of test notes, that's theory and practice, which I iterated aggressively to shape them into something useful and shared with colleagues. 

I used AI to help me build test tools, and as a tool itself for controlling the tools it built for me. I found interesting benefits from a deterministic tool with lots of telemetry and data creation driven by an agent under my direction. I used this approach frequently for exploration, amplifying my capacity to cover potentially interesting scenarios, but always giving me a way to directly check any claim made.

Some related posts: Is the Variability Variable?, How do I Test AI?,  Exploratory Testing Reimagined  

Knowledge and Sharing It

I've been around the block, within and outside testing, with roles in development, tech support, and technical writing as an individual contributor and a manager, plus all the stuff you just do being present at the start of a startup. Here, I've pulled out a few testing nuggets.

In my role at Ada I presented, demoed, built, documented, and paired all the time, as my default. I've learned that information tends to be absorbed better if it's solving a right-now problem so that's what I aimed for with topics including data-driven testing, property-based testing, concurrency, exploratory testing, heuristics, oracles, data, and visualisation.

Sometimes I was summoned to provide expertise. I recall an E2E Task Force that had wound itself into a knot of misconceptions about testing and service monitoring. I helped them to untangle and decide what they actually wanted before they did anything else. The conceptual blocker, which I've also seen elsewhere, was the use of a "test library" bringing a load of what-testing-is baggage with it. 

I set up pairing schemes within my team (including managers) and across teams, the goal being to share and learn from each other and build relationships. I've also had mentoring arrangements with colleagues tailored to whatever they felt I could help them with, and their preference for approaching it.

I paired at the drop of a hat on ad hoc tasks with anyone on anything, typically for testing ongoing work. I also facilitated weekly group testing sessions with the medical quality engineers where they'd bring a task or I'd suggest an exercise and we'd work out ways of approaching it given their context. The feedback on these sessions was amazing.

Talking of feedback, I give it if requested, including to the CEO and including things the requestor might consider negative. Asking is important: I would not usually offer unbidden feedback so I made a point of setting up an open invitation for it in both directions with my immediate managers.

Some related posts: Paving the Way, Giving Back, Collect, Arrange, and Slice, Means Testing 

So, what did I do?

Simply put: I did what I do. Loads of it, much more than what I've excerpted here, always identifying risks, finding information, building tools, building relationships, sharing, learning, and seeking to add value at a proportionate cost.  

I really do like this stuff. I like helping to build interesting things with interesting constraints, ideally as part of a small team that cares to do a good job. I like being trusted to seek out problems and opportunities to improve, wherever they might lie. And I like reflecting on those things. That's what I do.

If you're a tester who starts and ends at the acceptance criteria, I hope I've shown that testing can be so much more (and so much more fun) than that. You could do what I do.

And if you're a hiring manager, I'm available to talk. Perhaps I could do what I do for you? Give me a shout on LinkedIn.
Image: Canva 

Labels:

Popular posts from this blog

Meet Me Halfway?

  The Association for Software Testing is crowd-sourcing a book,  Navigating the World as a Context-Driven Tester , which aims to provide  responses to common questions and statements about testing from a  context-driven perspective . It's being edited by  Lee Hawkins  who is  posing questions on  Twitter ,   LinkedIn , Mastodon , Slack , and the AST  mailing list  and then collating the replies, focusing on practice over theory. I've decided to  contribute  by answering briefly, and without a lot of editing or crafting, by imagining that I'm speaking to someone in software development who's acting in good faith, cares about their work and mine, but doesn't have much visibility of what testing can be. Perhaps you'd like to join me?   --00-- "Stop answering my questions with questions." Sure, I can do that. In return, please stop asking me questions so open to interpretation that any answ...
Read more

How do I Test AI?

  Recently a few people have asked me how I test AI. I'm happy to share my experiences, but I frame the question more broadly, perhaps something like this: what kinds of things do I consider when testing systems with artificial intelligence components .  I freestyled liberally the first time I answered but when the question came up again I thought I'd write a few bullets to help me remember key things. This post is the latest iteration of that list. Caveats: I'm not an expert; what you see below is a reminder of things to pick up on during conversations so it's quite minimal; it's also messy; it's absolutely not a guide or a set of best practices; each point should be applied in context; the categories are very rough; it's certainly not complete.  Also note that I work with teams who really know what they're doing on the domain, tech, and medical safety fronts and some of the things listed here are things they'd typically do some or all of. Testing ...
Read more

Notes on Testing Notes

Ben Dowen pinged me and others on Twitter last week , asking for "a nice concise resource to link to for a blog post - about taking good Testing notes." I didn't have one so I thought I'd write a few words on how I'm doing it at the moment for my work at Ada Health, alongside Ben. You may have read previously that I use a script to upload Markdown-based text files to Confluence . Here's the template that I start from: # Date + Title # Mission # Summary WIP! # Notes Then I fill out what I plan to do. The Mission can be as high or low level as I want it to be. Sometimes, if deeper context might be valuable I'll add a Background subsection to it. I don't fill in the Summary section until the end. It's a high-level overview of what I did, what I found, risks identified, value provided, and so on. Between the Mission and Summary I hope that a reader can see what I initially intended and what actually...
Read more

The Best Programmer Dan Knows

  I was pairing with my friend Vernon at work last week, on a tool I've been developing. He was smiling broadly as I talked him through what I'd done because we've been here before. The tool facilitates a task that's time-consuming, inefficient, error-prone, tiresome, and important to get right. Vern knows that those kinds of factors trigger me to change or build something, and that's why he was struggling not to laugh out loud. He held himself together and asked a bunch of sensible questions about the need, the desired outcome, and the approach I'd taken. Then he mentioned a talk by Daniel Terhorst-North, called The Best Programmer I Know, and said that much of it paralleled what he sees me doing. It was my turn to laugh then, because I am not a good programmer, and I thought he knew that already. What I do accept, though, is that I am focussed on the value that programs can give, and getting some of that value as early as possible. He sent me a link to the ta...
Read more

On Herding Cats

Last night I was at the Cambridge Tester meetup for a workshop on leadership. It was a two-parter with Drew Pontikis facilitating conversation about workplace scenarios followed by an AMA with a group of experienced managers. I can't come to work this week, my cat died. Drew opened by asking us what our first thoughts would be as managers on seeing that sentence. Naturally, sadness and sympathy,  followed by a week ? for a cat ? and I only got a day for my gran! Then practicalities such as maybe there's company policy that covers that , and then the acknowledgement that it's contextual: perhaps this was a long-time emotional support animal . Having established that management decisions are a mixture of emotion, logic, and contingency Drew noted that most of us don't get training in management or leadership then split us into small groups and confronted us with three situations to talk through: Setting personal development goals for others. Dropping a clange...
Read more

Reasonable Doubt

In Your job is to deliver code you have proven to work  Simon Willison writes: As software engineers we ... need to deliver code that works — and we need to include proof that it works as well.  He is coming at this from the perspective of LLM-assisted coding, but most of what he says applies in general. I think this is a reasonable consise summary of his requirements for developers: Manual happy paths: get the system into an initial state, exercise the code, check that it has the desired effect on the state. Manual edge cases: no advice given, just a note that skill here is a sign of a senior engineer.  Automated tests: should demonstrate the change like Manual happy paths  but also fail if the change is reverted.  He notes that, even though LLM tooling can write automated tests, it's humans who are accountable for the code and it's on us to "include evidence that it works as it should." Coincidentally, just the week before I read his post I told one of my...
Read more

Why Test, Test What, Then How?

There was a time when testing was all about the mnemonics . Well, we had no AI back then so thinking about our human craft and how to share what we had learned using our human intelligence with other engaged humans for later recall in their human heads seemed important. But this post isn't about dumping on AI. It isn't about mnemonics either even if WTTWTH does look like something that'd fit well into that ancient world.  No, this post is just a snappier version of what I said to my team this week when I was walking through some testing I'd done the day before. It was concerned with a change to a particular turn in the dialog system we're building where multiple variables are in play, some populated by an external call to an LLM service.  I wanted to make the point that the bulk of the testing work was done in the research I did and the spreadsheet I made, not in the interaction with our product. That spreadsheet was the result of me looking in our service's cod...
Read more

Exploratory Testing Reimagined

Maaret Pyhäjärvi gave a talk, Exploratory Testing Reimagined , last week.  There's a  blog post  to go with it, so I won't say much on top of what's in my sketchnotes above, just pull out a few reflections. I don't have the data or wide range of experience reports that Maaret does, but my own anecdotal results from using AI tools for testing align closely with what she says in the talk. I have created skills to assist me, and my colleagues, in our testing work. For example, I used the contents of this blog and five years of test notes in my current role to distil a conversational workflow, facilitated by Claude, which runs through risk identification, test idea generation, and test mission creation. All four skills (the facilitation is its own skill) are backed by a fifth, on general testing theory and practice, and a sixth about how I like to interact with the AI: which notably includes that I want it to be concise, evidence-backed, well-structured, and critical. I st...
Read more

Vanilla Flavour Testing

I have been pairing with a new developer colleague recently. In our last session he asked me "is this normal testing?" saying that he'd never seen anything like it anywhere else that he'd worked. We finished the task we were on and then chatted about his question for a few minutes. This is a short summary of what I said. I would describe myself as context-driven . I don't take the same approach to testing every time, except in a meta way. I try to understand the important questions, who they are important to, and what the constraints on the work are. With that knowledge I look for productive, pragmatic, ways to explore whatever we're looking at to uncover valuable information or find a way to move on. I write test notes as I work in a format that I have found to be useful to me, colleagues, and stakeholders. For me, the notes should clearly state the mission and give a tl;dr summary of the findings and I like them to be public while I'm working not just w...
Read more

For Fix Sake

The latest episode of the Vernon Richard Show, New QA Job? How We'd Decide Where To Start , answers this question from a listener, Shady: When joining a new organisation, how do you decide which QA domino piece to tackle first for the rest to easily fall into place? For example, you might find flaky test automation, poor testability in user stories, no integration tests, limited test management tools, missing test data, and an ever growing regression suite. With so many improvement opportunities, how do you prioritise what to fix first? As usual the conversation ranged freely and I've picked out some of the key points with a little commentary here: It Depends : usually the right answer or, rather, a small part of the right answer. The rest should be what it depends on and how that affects your answer. Investigate the Context First : exploring to understand the context is a reasonable default in a new job. You'll want to understand the key ...
Read more