I love that.
The most recent instance for me was a couple of weeks ago when asked what problems I have with test data at work. That's when it became obvious that I have three broad categories of test data, and want something different from each of them.
Of course I do! Duh.
And why didn't I realise this before? Not sure. Perhaps I've never framed that kind of high-level abstract question, Maybe more often I'm interested in test data for the actual problem in front of me? Whatever, here's my initial draft of what I think I think...
Data for specific logic. I use this in traditional regression tests that need a static expected result: unit tests, for instance. I see a lot of examples that have clearly grabbed runtime application data that's the right shape and bunged it into the test. This is convenient for the writer but can obscure the test intent and make later understanding and failure investigation more costly.
As I usually want to focus on particular aspects relevant to a specific check, I'll more often apply my Real vs Clear heuristics and curate the data carefully.
Data for performance evaluation. Here I want production(-like) data in whatever dimensions are important for the kind of evaluation I'm after. Say it's a latency test, then I'll probably need to think about data that has a realistic distribution for the scenario I'm testing in terms of variability, size, shape, arrival rate, peak load, and so on.
If I'm not able to see prod data myself I try to work with more privileged people because then at least metrics about the data are obtainable and can be used to create synthetic data. Whatever the provenance, validation and maintenance of performance test data can be tiresome unless both the data format and the usage patterns are relatively static.
One other way in which this type of data can be challenging is when it's divided into training and test sets. I understand why but the temptation to peek at the test data can be powerful.
Data for exploration. "What data will drive the application to the state I need for this experiment?" That's a question I ask myself a lot and I have a bunch of ways to approach it, including using data from the two categories above, either directly or as a starting point to iterate from.
A good knowledge of the application or product, its dependents and dependencies, and the ecosystem it sits in can be helpful because this type of information reveals levers that I can pull to engineer the scenarios I'm interested in.
I look for access to logging, monitoring, and observability tooling because I can search there to find a pattern indicative of the state I want to get to and then try to work backwards to the data that provoked it.
The code itself is, of course, a potential source of data but not always an easy one for me to decipher. LLMs are a recent addition to my toolbox and can help enormously, for example to get an idea of the codebase geography, jump to specific logic, and even extract helpers that generate the data I want using the application code as a model. Keep an eye on what they're doing though, eh?
Another code-based strategy I use is to alter the application to make the state I want easier to reach. This can take many different forms, but includes side-stepping data altogether to throw a specific exception deliberately to force the application into a state, hard-coding or side-loading data from a file I supply so that I can control things very precisely, or forcing use of a mock for dependencies, with the same aim.
It's also possible to bypass up-front data creation without changing the software by generating it at runtime. When I do this, I am typically going to run many experiments and see if any of them end up in interesting places and be careful to record a lot of detail while my experiment is running so that I can explore it and get back to specific states later.
--00--
Reading this post back, I find it a useful if not very well-defined division. As I already said, I might use unit test data to explore, but I will also explore to make data for static or performance tests and something like property testing straddles the lines. In spite of those fuzzy edges, though, I feel happy I've captured a reasonable chunk of what I didn't know I knew that I can reflect on and refine later.
So I'll stop there for now because, while I'm happy to have got that out of my head, I've already started chewing this over:
What data will drive the application to the state I need? - if you're not asking yourself this on a regular basis, you're not exploring.
It just slipped out when I started writing about data for exploration and now I'm wondering if that's really what I believe, and whether it might actually be true.
Did I mention that I love this stuff?
Image: Greg Jewett on Unsplash