Testing is inherently about looking because, put simply:
If you don't look, you're not likely to find.
The interesting challenge is to look in the right way at the right place at the right time.
This is what motivates any kind of intentional testing: how can we put ourselves in a strong position to look for, see, and — crucially — recognise the things that matter to the people that matter, when they matter?
--00--
Tools can help with this mission: using tools we can look more deeply, more broadly, for more complex patterns, for harder-to-spot traces, faster, more often, more efficiently, and so on.
There's a trade-off, naturally. As the tool takes us further from the material being worked on we must either trust it more or check its results more thoroughly ... to the extent that we care about the results.
At a crude level, think of it as a spectrum. At one end we might have a knife. It's a tool I can use to cut things. I could eat my pizza without cutting it, but sliced segments make it easier to pick up and bite, and creates less mess than trying to tear pieces off or shovel the whole thing into my face.
There's a tool here, and it has value, but the tool use is very analogous to hand use and the tool user is not far removed from the material, the pizza.
In the middle of the spectrum somewhere are search tools. We all use them on a regular basis and they indisputably have value, but they take us further from the source material. Ctrl-F in a document you are writing maintains proximity but searching the web is a different proposition — what was there, but missed?
At the spectrum's far end we find much of today's AI tooling. We know that they are flawed: they don't answer our request but instead generate a mathematically and linguistically plausible facsimile of an answer. We also know that this can be sufficient, and indeed genuinely useful, in some cases.
There's a tipping point on the spectrum at which we need to think about how sure we can be that tools will do the things we want and intend them to do. When we ask AI to summarise the requirements for a task by looking through a Jira ticket, its parent, siblings and related Confluence pages, how far should we trust the bullet points it returns?
A summary is more than the "basic" information retrieval of a search task: it involves judgements about what to include, how to resolve conflicts, what level of granularity to use, and other factors. Can we have confidence that the editorial choices, wider context, and critical thinking align with what we want, expect, and need? And if we can today, does that carry over to tomorrow?
Well, no, we can't and it doesn't necessarily, and the less visibility of the source material we have and the less we choose to even look at it, the easier it will be for us to accept something as correct when it is not.
--00--
But clearly tools are able to amplify, replace, or extend our puny human capabilities. They can do things humans can't, find things humans miss, generate data for humans to review, very reliably take on low-risk, mundane, predictable, repetitive tasks, and operate at a scale humans will never reach alone. And, even if we agree that tools can't be relied upon to be perfect, surely we aren't going to argue that humans can?
Go tools! Ditch the meatsacks!
Well, let's not be hasty. People, or people controlling tools, or people supervising tools, can still have value.
The map is not the territory: any abstraction, such as an AI summary, is not the raw material. Any summary of raw material, however accurate on whatever metrics we care to use, only takes into account that raw material. Humans bring context, and make connections outside of the material. In the Jira example, perhaps the connection is to a project they worked on last year, or a conversation about priorities that they overheard last week, or the current state of the market, and perhaps it invalidates some assumptions that motivate the work.
Let me give you an example: sometimes for that kind of task, in important, complex cases, I retype the requirements. Not an AI summary, not even copy-paste, but literally retyping them, gathering them from multiple places, giving them identifiers, sorting them into groups, standardising terminology, breaking complex phrases down into testable atomic chunks.
Too slow? Too boring? Someone else's job? Perhaps, but also forcing me to take my time to understand, analyse, correlate, cross-reference, contextualise. To learn. Looking is learning, and with that depth of learning I feel better equipped to direct my testing to the important areas now, and I've got context for later.
It's the same when I come to testing the application itself. Sure, I could instruct an agent to see whether it can run an end-to-end user journey including the new endpoint (and I have done) but when I do that I am, at best, squinting at it in low lighting through a pinhole without my specs on.
If that's all I do, I am giving myself no chance of spotting inconsistencies between this and another service in our product range, noticing poor naming conventions, feeling the pain of a developer trying to use this API, seeing opportunities for seams to exploit in further testing, and so on.
There is friction in engaging with a product directly and incrementally building a mental model of its behaviour, but there is learning in that friction: in the active looking.
--00--
The value proposition for automation is this: automate when the results matter but the details don't, and the cost is reasonable.
I could buy an automatic pizza slicing machine, but it's expensive, overkill for portioning the occasional pizza and, while the 10-year-old that still lives in my head loves the idea, it wouldn't even fit in my house. Nope.
On the other hand, searches cost me nothing and I trust that the algorithms are largely a solved problem so I'm happy to delegate search to automation and allow myself to focus on the queries that will give me the data I need.
But what about those AI summaries of the work requested, pulling in multiple perspectives and making judgements on my behalf? Erm, no. In general, if I'm going to work from those requirements, I care about the details and I want to look explicitly at what we know and give my peripheral vision the opportunity to make sideways connections.
But the details are just one of the considerations I mentioned. Cost is the other. It costs me nothing to use a company tool to create the summary or, at least, I'm not personally out of pocket. But there are other costs of sub-contracting work: over time our skills rust, our memory atrophies, and the data that feeds our intuition gets dated.
I've heard many people describe using AI agents for work as being like management and I do see parallels: one person orchestrating action by others, distance from the actual work, the potential for the same kinds of costs that I've just mentioned. But do we all want to be managers? Do we all want to be the kind of micromanagers that today's AI tooling requires?
As with all tooling, intentionality is key. What do you need? What cost is reasonable for you, right now, to get it? What about in the medium and long term?
--00--
There's a further consideration when considering tool use: what brings you joy?
I enjoy testing. I regard it as a craft and I am unapologetically precious about it. As I have said many times on this blog, I use tools, including AI tools, to help me in my craft. I am not blind to the fact that historically crafts dwindle in the face of technology, but I do have sympathy for the position of Jeremy Atkinson, the last traditional clog maker in England:
"I often get asked whether I’m worried about competing with machinery. Traditional clog making started dying out as early as the 1950s, and while I will never keep up with machines when it comes to quantity, my bespoke shoes will fit better."
He stays close to his materials. He looks. He really sees the wood and the leather, and he sees his customers, and he sees his customers' context, and that produces better quality, if lower throughput.
--00--
My point here isn't that tools in general or AI specifically should not be used for testing. My point is that AI is at the far end of the spectrum representing a tool user's distance from source material. AI tools bring great potential but they are the darkest of black boxes with a paradoxically apparently open user experience and, importantly, low friction at the point of use.
All tools sit on that same spectrum, but AI's particular feature combination requires users to exercise particular caution. Not doing so would increase the likelihood that significant results are missed, system knowledge is eroded, and we let our testing skills wither.
This risk can be mitigated to some extent by intentional use and by taking care to stay close enough to the data, to the application, and to the system under test for the context. That is, by looking.
Because testing is inherently about looking and the best testers are the ones who look good. (Even while eating pizza.)
Image:Andreea Pop on Unsplash