Transforming Theory and Practice

When Sneha Bhat asked if I'd present with her at CEWT #5 the talk we produced was Theoreticus Prime vs Praktikertron. In this essay we've tidied up the notes we wrote in preparation and included a few of the sketches we made when we were developing our model. The title comes from the Transformers we gave the participants at CEWT to explore in an attempt to illustrate different kinds of theory being discovered and shared.

CEWT #5 asked this question: theory over practice or practice over theory? It's an age-old conundrum, represented in popular culture by memes like these that you would have seen as you avoid both theory and practice by grazing on social media when you should be working:

In theory, there is no difference between theory and practice. But, in practice, there is. (Wiki)

Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. In our lab, theory and practice are combined: nothing works and no one knows why. (Crazy Proverbs)

In the way that all communities look for another to kick against — an outgroup to their ingroup — it’s not hard to find instances of theorists saying that practitioners don’t know why they’re doing what they do, and practitioners saying theorists couldn’t do it if they had to. But let’s not be drawn into this kind of bickering, Instead, let’s step back and try to draw out some distinction between these terms, between theory and practice.

A theory seeks to explain observations. In science it tends to be interpreted as being backed up by a weight of evidence, but in casual conversation this isn’t the case.

Practice also has a couple of primary senses, but both of them are strongly about doing, about activity: either repeatedly exercising a skill, or applying some idea.

So theory is some kind of thing, a thing that is produced, whereas practice is an activity, perhaps an engine for production. That already suggests an intriguing possible relationship, one where practice might drive the generation of theory. But how else might practice relate to theory? You don’t have to look hard to find suggestions and we’ve picked out just three that piqued our interest when we were researching our CEWT talk.

Pete Walen, a tester, writing in Considering Conferences: A Reflection, argues for a causal relationship, that theory only has value if it changes practice.

W. Edwards Deming, in The New Economics, wants theory to invoke practices that will confirm it or contradict it, and either way, increase the sum total of theory.

Steve Klabnik, a well-known thinker and contributor to the Ruby and Rust communities, thinks that the key is finding people who can explain the value of theory to practitioners and get practitioners to point out its flaws to the theorists.

Our intuition is that there’s truth in all of these perspectives, and we want to capture it in a model.  Assuming that theorists and practitioners can be convinced of the value of each other’s contributions, the primary aspects are:

• Theory should guide practice.
• Practice should help to refine theory.

And if that looks like a loop to you, it does to us too. But where should we enter it? Jack Cohen and Graham Medley provide a suggestion in their book, Stop Working and Start Thinking:

This cannot be said too often or emphasised too much. Ignorance, recognised, is the most valuable starting place

So we imagine some kind of process for accomplishing a goal which goes like this:

• Do I have theory that gives me what I need?
• If yes, then stop.
• If no, then practice to generate theory.
• Repeat.

And that seems interesting, maybe even plausible, but perhaps lacking in some important respects. For example, we’ve talked about both theorists and practitioners but while the latter are represented in this pseudocode the former are not. Where might they fit?

Our proposal is that they are present. Our proposal is that what you might call theorists, in fact, are practitioners. We’ve said that theory is a thing and practice is an activity. Theorists think about things. Thinking about things is an activity. Ergo, theorists are practitioners! They might not be working with tangible objects like a plumber would, but then perhaps neither are software testers.

To us, the model looks less naive now. But there’s another wrinkle: practice can generate all manner of data, but we guess that only some of it will become theory. To accommodate that, we think of theory as something that doesn’t necessarily have explanatory power but is simply the data that we care to keep track of.

The loop now makes more intuitive sense. Here is it again, with a theory/data distinction:

• Do I have theory that gives me what I need?
• If yes, then stop.
• If no, then practice to generate data.
• Repeat.

When starting a project we wonder whether we have data that answers whatever question the project is set up to answer. If not, then we set out to generate that data. This generation may be by manipulating the data we have (traditionally the realm of theorists) or manipulating some external system (traditionally practice).

It may be that both types of practice are required. Whichever it is, the theory that we have to start with, and ideas about how to get it, drive whatever practices are carried out. The data ultimately underpins everything. As Rikard Edgren warns in the Little Book of Testing Wisdom:

Software is messy. That's why empirical evidence is so valuable.

Let’s explore how we think this model suggests explanations for some phenomena that we see around testing practice and theory. Take a very simple case where a tester is exploring an application which takes letters as input. Perhaps she finds that:

• entering data item A results in output X, B results in output Y, and C is disallowed,
• the text box in the UI restricts entry to one letter at a time but there is an API through which bulk entry is possible.

The tester has asked a question — “what can this application do?” — and found answers, or data. The data that she tracks is her theory. The theory influences further interactions. In this case, we perceive two broad types of theory: behavioural (what the system does) and practical (how to exercise the system).  In a typical project the first of these is likely to get reported back to the team in general, but the latter is likely to remain local with either this tester or her peers when they share information that helps them to do their job.

We might consider these behavioural and practical flavours of theory to represent expertise and experience respectively. For us, both can usefully guide practice, even in the world where the theorist/practitioner distinction that we reject holds some sway:

It's possible to have degrees of [experience or expertise], or both, or to have neither in some area. Apart from its utility as a heuristic for remembering to look to test at different levels, it serves to remind me that the people on any project have different combinations of each and their mental models reflect that.

The manager knows there's a hammer that can be applied to the current problem: just how hard can it be? The implementer knows what to hit, and where, and just how hard it can be.

As testers we’ll naturally be alert to edge cases, the less usual scenarios. A typical edge would be the initial starting condition for a system. Let’s look at that now, and see how the model we’re building might handle it.

Imagine a tester joining a project that’s working on a product that she has never heard of in a market she didn’t know existed for users that she has no knowledge of nor immediately empathy for. She has no expertise in this particular system but she has experience of systems in general. Perhaps she decides that her mission is to make a mental model of the product and begins to explore it, using cues from other software systems she has known as an initial comparison point.

Her theory in this situation is biased to the practical. Or, perhaps more accurately in our conception of it, the subset of her theory that she chooses to use to guide her initial practice is biased towards experience. As she works she generates more data, and the pieces she cares to keep track of become new theory for her.

Another tester dropped into the same project at the same point might have chosen a different route, even with the same mission. He might have begun by reading all of the available project documentation, stories, bug reports, and support tickets, by talking to all of the team members, by reviewing the code and so on. This is also practice, to us, and again it is based on experience rather than product or project expertise (because this tester has none), and it also generates data of which some portion becomes theory for this tester.

In this example, two testers faced with the same situation chose two different approaches. We made it sound like conscious choices, but that needn’t be the case. We’re all human, so factors such as our biases, available time or other resources, pressure from others, incorrect analyses, inattention, preferences, and familiarity with tools can all impact on the choice too.

Again, in this example, although we made the scenario the same we didn’t restrict the personal theory that the testers carried with them into the project. This can, and will, influence actions alongside any shared theory in the project. Ease of sharing is an extremely positive aspect of theory. Often, in fact, theory can be shared in much less than the time taken to generate the data which lead to the theory. In a software project, we might imagine a team of testers reporting their findings to a product owner who takes from it and works on it to generate her own data and hence theory.

This PO is a practitioner. She might not practice on a physical system but her interactions with data have the same potential for experience and expertise as any interaction with a physical system:

experience: she finds Excel a convenient way to combine the reports on this project given that they are generated as csv files from session-based test management notes by the team members.

expertise: she recognises different terminology in several reports as referring to the same kind of phenomenon and combines them, and rates that issue more important because it’s clearly affecting a significant portion of the product.

Her report is theory for her, and potentially for whoever she chooses to share it with. It might go back to the team, to other teams, to a project manager, senior management and so on. Within the team, her theory will guide action. For example, the testers might be asked to run new experiments or change the way they work, or the PO might decide that it's time to ship. Outside of the team, it may be added to the set of theory for someone else, or it might be ignored or lost, or read carefully without the importance being recognised.

Having access to data is not the same as understanding or exploiting data. A practitioner will not report all of the data they collect nor all of the theory they generate to the PO in this scenario. Even if trying to be exhaustive, they will naturally provide only an abstraction of their complete knowledge, and (if our experience is anything to go by) it will tend to be biased more towards the expertise, to how the system works. In another scenario, such as a community of practice or a peer conference, they might provide a different abstraction, perhaps biased more towards experience, how to work the system. We find David Kolb’s Learning Cycles interesting here.

Also interesting to us, in Why is data hard? Helen L Kupp identifies four levels of abstraction around data: infrastructure, metrics and dimensions, exploration and tools, and insights:

Having lots of data doesn’t make it immediately valuable. And ... not only is leveraging data and metrics well critical to effective scaling, but it is also that much harder because you are “building the plane as it is flying”.

At each level she describes how practitioners operate on the data to generate value, the theory. Interestingly, she also talks about how these layers interact with and feed back to each other:

This is an appealing potential elaboration on the distinction between theorists (nearer the top) and practitioners (towards the bottom) which also emphasises data as the key thing that binds the various parties, and the different ways in which those parties act on the data or in pursuit of theory.

A distinction that we haven’t yet covered is that between tacit and explicit theory. All of the examples we’ve given involve theory that’s out in the open, known about and deliberately shared or stored, explicit. But theory can also be tacit, that is, internalised and difficult to describe. Think how easy it is to ride a bike once you’ve got it, but how hard it would be to explain all of the things you do when riding, and why.

Tacit theory can be both advantageous and problematic. An expert practitioner may add enormous value to a project by virtue of being able to divine the causes of problems, find workarounds for them, and gauge the potential impacts on important scenarios. But they may find it hard to explain how they arrived at their conclusions and asking them to try might impact on the extent to which the theory remains tacit and hence directly and intuitively accessible to them.

This kind of tacit knowledge is learned over time, and with experience, and likely also with mistakes. There is some mythology around the status of mistakes with respect to learning, Scott Berkun says “you can only learn from a mistake after you admit you’ve made it” and it’s not uncommon to hear people claim “don’t worry, you learn more from your mistakes” when you’ve messed up.

We’re not sure we agree with this. Mistakes are actions, which may generate data, some of which may become theory. At the time a practitioner takes an action and has the chance to see that data they may not realise there was a mistake, but there is still learning to be had. Perhaps the insight here is that mistakes tend to generate how-to data (experience) because the results of a mistake are less likely to be shareable (so of less expertise benefit).

In the kinds of situations we find ourselves in day-to-day there are many actors operating in a given system, for instance testers, developers, technical authors, application specialists, Scrum masters, project managers. Each of them is using their practical skills to take some input, make some observations, create some data, and generate some theory.

There are no clear levels between which data and theory are passed. There are multiple types of theory. There is no place in which all data and theory are, or could be, shared (because a substantial portion of it is tacit). Beware, though, because as theory is generated further and further away from the point at which its underlying data was gathered, the risk of successive abstractions being too narrow increases. This can be seen in the classic reporting funnel where information gets less and less tied to reality the further up a reporting chain it goes.

Back to the original question, then. It theory primary? Is practice primary? We think that data is primary and both theory and practice serve to guide the generation and exploitation of data:

• Practice generates data.
• Data makes theory.
• Theory guides practice.
• Practice generates data
• ...

The required data, rather than theory or practice, is the starting point, and also the ending point. Once the required data is in hand, the loop can stop. Which, we hope you’ll agree, is a neat, ahem, theory. But what practical value does it have? For us, there are a few useful pointers:

The theorist/practitioner distinction has no real validity. We’re all on Team Practice, although we may have different levels of interest, ability, and inclination in practising in particular ways.

You can consciously make yourself more widely useful by developing skills in practices that apply across contexts and having theory around those practices which is transferable. The flip side is that depth of theory in a particular area will likely be reduced.

Don’t just pick up your favourite tool and begin hacking away. Whatever your biases, you can start any task by asking what data you need, then making a conscious choice about how to get it.
Image: Helen Kupp