In it, I'm thinking about the tooling I built to help me explore a chatbot
API. It exploits random choice to walk through the extremely large space of
possible chats in a medical symptom checking application.
As I reflected on the combination of tools and testing I found it convenient
to label three activities that involve both.
Navigate
Navigation is about finding a path to an endpoint. While navigating I am very
interested to notice assumptions I'm making, workarounds that are required,
and any questions that come to mind, but my main focus is on reaching the
goal.
In the first instance, on this project, I needed a basic framework that would enable my code to start a chat, walk through all of the interactions with the service, and stop. As I was writing code to create my initial implementation I sacrificed generality in favour of the end-to-end path. For example, the first version presented as the same user every time.
I found myself asking questions like What would happen if I ... ? How can I
... ? Why is this ...? For example: why are these two schemas
similar but different (and so I need two routines to parse them)? Unless I
needed to resolve them to proceed, I just parked those questions for later.
Survey
Surveying looks at the known paths and their environments with lenses that increase knowledge about them. While surveying I'm trying to find information that supports and extends what I've learned so far or points to somewhere that might merit exploration.
On this project, I added assertions on API payloads matching schemas, fields that should be in some relationship to one another, and bad requests getting error responses. I also asserted things that I thought would hold generally but wasn't sure in the hope that, across enough runs I would find places that the assertion was violated.
When an assertion fails I get to wonder whether my expectation is wrong,
whether my coding is wrong, or whether the service is wrong.
Because I was using randomness to make choices, I sometimes couldn't assert very specifically. In this case, the assertions might be that values should be within a specific range or set, as in property-based testing.
Explore
Exploration seeks unknowns in the areas off any paths. While exploring,
I will typically start with open questions and move in directions that look
promising in the moment. I need to be comfortable backtracking, finding
alternative routes, and dealing with ambiguity. My main focus is on answering
the question, and I'll use tools to do that where I think it makes sense.
When you explore you don’t know if you’ll find anything, and if you find
something you won’t know whether it’s relevant, and if it’s relevant you won’t
know whether it’s important.
On this project I'd ask a question like are there dialog states where aborting
the chat will be problematic? I'd then look for a way to hack or configure the
code to abort regularly during chats, and log what happened when it tried. I
could then run thousands of chats, generate logs, and look for patterns of
behaviour, gaps in the places that were aborted from, and so on using a
spreadsheet or tools I created specifically for that.
Thoughts
I think that in navigation I'm biasing towards tooling and finding ideas that
I can take into testing. When exploring I'm starting with a test idea and
looking for tools that can help me to investigate it. Often the tools are code
(Python and Bash on this project).
Surveying falls somewhere between the other two, but the boundaries between
all three are pretty fuzzy. Where the distinction helps is to remind me of my
intent and keep me on mission. When I want to build a thing — to navigate —
then I probably won't follow up every interesting-looking observation. I'll
save them until I explore, unless the potential value looks high in which case
I can intentionally suspend one mission and start another.
Also, although I've listed them in a particular order here, I'm not saying that they need to occur in that order. You might explore to find places to navigate between, surveying can be the first steps in exploring, attempts to navigate can help you to understand what makes sense to survey.
I've seen cartographical analogies for exploration before but I don't
recall seeing this kind of breakdown for the relationship between tooling and
testing. I'd be interested to know of precedents or similar ideas.
Comments
Post a Comment