Testing vs Chicken

At CEWT #6 we were asking what constitutes good testing. Here's prettied-up versions of the notes I used for my talk.

One line of argument that I've heard in this context and others is that even though it's hard to say what good testing is, we know it when we see it. But there's an obvious potential chicken and egg problem with that.

I'm going to simplify things by assuming that the chicken came first. In my scenario, we'll assert that we can know what good testing would look like for a project which is very similar to projects we've done before, where the results of testing were praised by stakeholders. The problem I'll address is that we don't have anyone who can do that project, so we have to hire. The question is:

What can we do as recruiters, during recruitment, to understand the potential for a candidate to deliver that good testing?

I've been recruiting technical staff for around 15 years, predominantly testers in the last ten, and my approach has changed enormously in that time. Back in the early days, I would largely base my phone screen interviews around a chronological traversal of the candidate's CV asking confirmatory questions. Checking the candidate, if you like.

These days, a CV to me is more of a filter, and a source of potential topics for exploration. I have also spent a lot of time thinking about how I want to advertise positions, and about the kinds of information I want to give and get from each stage of the process, and how I'll attempt to do that.

I have a simple model to help me. I call it the Egg of Testing Recruitment.

The yolk is the core testing stuff; crucial to our identified needs. The white is the other stuff; important to functioning in our context. It supports the yolk.

Some people will tell you that eggs are easy to cook. Some people also think that recruitment is straightforward: identify a need, describe it, find the person who best matches it, hire them, relax. But eggs don't always come out the way the chef intended.

And recruitment likewise. Here'a few reasons why:

• multiple important factors
• limited time and engagements
• a space of reasonable outcomes
• a dynamic feedback system

That last one is particularly interesting: as a recruiter, be aware candidates will be looking to align themselves with your needs. If, for example, you do and say things that suggest you favour exploratory testing, then don't be surprised when answers which support their exploratory testing skills start to come.

But  recruitment is starting to sound a lot like testing: the extraction of relevant information from an arbitrarily complex system under various constraints. And, if it's testing, I'll want a mission. And if I had a mission I might phrase it something like this:

The kinds of materials you can usually expect in standard hiring rounds are:

• a cover letter
• a CV
• a phone screen
• a technical exercise
• a face-to-face interview

And then there's others that are reasonably common these days, including:

• social media
• blog
• personal website
• open source projects

All of these hold data about the system under test ... erm, about the candidate. I know that some recruiters disregard the cover letter. I love a cover letter. First, it's more data. Second, it is an opportunity for the candidate to speak direct to me, in their own time, in their own words, about the fit of this role to them and them to this role.

When it comes to conversation and exercises, I use the Egg of Testing Recruitment to remind me of what I'm after.

The yolk: when I can interact with the candidate I tend to want to explore core skills that can only really be demonstrated interactively. I'll want to put the candidate in a position where they can't know everything, where there's ambiguity, and see how they deal with it.

Do they ask for clarification, do they tell me what their assumptions are, do they offer caveated answers, do they say "in this context ..", do they use safety language? In this respect I regard interviews as more like an audition -- asking the candidate to perform something like a testing task, and being able to explain their thought processes around it.

The white: I'll be looking for reporting, presentation, consistency and the like in the written material. I'll also be noting stuff that could be ways in to understanding other aspects, particular technical expertise that I can ask about, for example. I can't ask for demonstration of all skills, but I can ask behavioural questions such as "can you tell me about a time when someone doubted the value of testing or when someone asked you to justify your testing?"

In the real world, of course, the egg model looks very different.

The yolk and egg cannot be separated so cleanly. But that's OK. In the interview, I can be testing both at once. For example, on any answer the candidate gives I can be looking for consistency. I can gauge the correctness or reasonableness or depth of a series of answers and use them as checks on the candidate's reliability of answering.

Having explored the candidate using conversation and exercises I need to evaluate them. A job advert that reflects what you actually want helps here. (It's worth remembering that when you're writing it.)

This evaluation is again like testing; you've stopped because you've spent the time that is available. Of course you could have done more. Of course you could have taken alternative routes. But you didn't and now you have to report: what you did, what you found, and the values and risks associated with that.

In your day job this probably goes to a stakeholder who ultimately makes a decision. In recruitment scenarios, you may well also be the stakeholder. But that shouldn't alter the way you go about your business, unless it makes you care even more than you would normally to do a good job.

I think there's three major points here. To put yourself in a position to recruit testers who can do the kind of good testing you're after:

• understand your mission
• treat interviews as auditions
• explore the candidate

Here's my slides: