Bog-Standard Testing

Bear with me, there is testing here but first we need to talk about my toilet. 

Yeah, my toilet. Its flush was getting weak and though I was pleased to have a preliminary diagnosis (siphon membrane) I wasn’t happy about the likely treatment. I’ve changed siphons before and it’s always taken me ages, with a lot of pieces to disassemble and reassemble. 

Worse, I have never managed to get all the bits refitted on my first attempt without a drip appearing somewhere. That’s usually how the second attempt goes too. Worse still, on this toilet all of the moving parts are hidden behind panels. How was I supposed to do anything with that?

But I know from past procrastination that I get nothing accomplished without starting, so I began by looking and feeling, wondering whether I’d need to apply some force to flex a sprung catch, undo a screw thread that hasn’t moved since Noah was a lad, or pull a stiff doodah out of a tight watchamacallit.  

I’ve learned over the years that it’s worth compromising between the desire for ingress and the risk of breaking something with misapplied effort. When I find myself reaching for a screwdriver to muscle open a joint that no screwdriver fits into, I try to remember to take a step back and decide consciously to go ahead, or not.

The vertical front panels on my toilet seemed likely to give better access than the horizontal button panel so I inspected them for signs of previous movement or fixings, but found nothing. Pushing and pulling didn’t yield anything either, and the screwdriver stayed firmly in the toolbox.

I moved on to the button panel. With a little light pressure I could tell the plate would open up somehow and eventually determined that it popped off if I slid it right and lifted, freeing four plastic catches holding it down and releasing a spring keeping it in place.

Now, with my big talk about siphons and leaks and all that, I may have sounded knowledgeable, but let me clear here: my mental model of this toilet was very flawed. I was expecting to see something mechanical under the buttons, like this:

 

Instead it looked pneumatic, with buttons pushing air down the tubes to what I still assumed was the siphon. (Spoiler: I was wrong about that too.)

I had a look inside the hole to see what access would be like (it was tight, duh!) and noted that the front panels were screwed on and I could remove them later if I wanted or needed to (good). The back of the plate had a maker’s name and some other details:

Decision point. Tinker a little, call a plumber, or see what I could find online for this kind of cistern setup? With no urgent requirement for a solution I decided to spend a few minutes gathering information. 

There was an email address in all the guff at the bottom right of the panel, at the domain asseur.com. Chuckling gently to myself about the rear-endedness of the URL for a toilet company, I ran into this:

Perhaps they had changed to a domain with less unintentional comedy potential? I backed off to a general web search and got a couple of hits on cylex-uk.co.uk, a site I’d never heard of but which turns out to be some kind of company directory. 

The first hit gives up a postal address and a link apparently to Ideal Standard, a company that I know manufactures bathroom fittings.

Click! Bingo! 

I wondered whether Ideal Standard might have taken over American Standard but if they did, they’re not advertising it very prominently:

No matter, there’s a link to a spares site at the bottom of the page but, booooo!, a broken redirect:

The second Cylex hit was a dead end except for a phone number. I saved that for later.

My next line of attack was searching YouTube for American Standard toilets. At that point, I was trying to find something that would help me decide how to proceed. Maybe someone had the same model with the same problem and demonstrated their solution in a video?

No such luck, but I did discover that American Standard is still a going concern and can be found at americanstandard-us.com. That “-us” suffix is interesting; perhaps UK business was dropped at some point which would be consistent with an Ideal Standard takeover.

There was a spare parts link here too. No error, but it was sooooo sloooooow that I abandoned that investigation. Again, I could always try again if I needed to.

I continued with YouTube, using search terms related to the parts of the toilet and the problem I had such as cistern, siphon (and syphon), flush weak, not flushing, replace, fix, and so on. 

I wasn’t finding much that was directly relevant but I did skip through some of the videos to see whether they featured any parts that looked similar to what was in front of me.

Next I tried another couple of sources I’ve found useful in the past: Amazon and eBay. The beauty of these sites is that sellers often list parts that fit multiple makes and models of an appliance and list all of the makes and models.

This gave something interesting, some more references to Ideal Standard and a part that looks close to the one I could see through the hole under the button panel, second right in the image below.

Time to pause. I wasn’t getting anywhere fast searching for American Standard and it looked like there might be a link between them and Ideal Standard. If I could understand the link, perhaps I could refine my search. So to Wikipedia!

Yes, here’s the result I was after: 

The kitchen and bathroom division was sold off ... the global business [also] became Ideal Standard.

That event is dated 2007. I’ve only been living in this house for 12 months so I don’t know the age or provenance of the toilet but the amount of dust and grime around the cistern suggests it could’ve been in place for years. 

At that point I decided to work on the assumption that Ideal Standard information and maybe also parts were what I needed. I recalled that the reverse of the button plate had some part numbers on it and hit paydirt with the next search!

In fact, I also found that the same results come up with just the part numbers which means I could have got to this point immediately had I started with them. 

Boooooo? Well, no. I didn’t consider the work I’d done up to then wasted. I didn’t spend many minutes on it and I came away with reasonable confidence that the Ideal Standard parts are relevant. 

Hindsight is a tool for learning, not for beating yourself up with.

In any case, I didn’t yet know what I was looking for. I was still trying to understand the problem, what scope I’d got for fixing it right then, what parts I needed and might be able to replace myself, and whether I'd need to call someone in.

I decided to search for all three of the part numbers on the back of the button plate and then skimmed the descriptions and photos across multiple sites. I’ve found over the years that the product listings, and especially the purchaser reviews, at different vendor sites are often complementary and you can build up a bigger picture by combining them. 

The results below are for SVO4567 and I scanned them for photos from different angles, how-to videos, dimensions, or maybe even manufacturer instruction sheets.

I struck gold in an Amazon product description where, unexpectedly, someone had dropped diagnostic clues:

If your Ideal Standard Pneumatic flush valve isn't flushing properly ... it can be this bellows washer ... You can test [this]. Simply disconnect the Pneumatic tube that it attached to your push button and blow down the tube. If the valve doesn't lift properly as you're blowing down it, it will more than likely be this bellows washer. If the valve lifts with ease, it could very well be an issue with the button instead. (sic)

Visual inspection and comparison of the photos and description to the pieces of toilet in front of me suggested that I was in the right area. Bonus, I learned some terminology; the thing I had been calling a siphon is actually a valve:

Second bonus: the discovery that the term of art for hidden cisterns is concealed. Using that word I could get straight to a video on plate removal that would have been handy when I started. Again, them’s the breaks. 

Having the right lexicon also helps with credibility when talking to others. I’ve found that I get more information and time from an expert when I can show I’ve looked into their area rather than just called them up to say “boo hoo thingy broken costy fixy?”

Time to take stock again. At this point I wasn’t afraid to tinker because I knew that even if I broke the pieces in front of me, parts were available and the toilet could still be flushed with a bucket of water if needed. 

So I dismantled all the accessible pieces, pulling the tubes off the top of the valves, pulling the valve caps off, removing the rubber bellows (the black corrugated discs you can see in the SVO4567 photos above) beneath them, and so on. I found that there’s a cylinder inside each valve and pushing them down provokes a powerful flush so I was confident that the problem was between the button and the valve top somewhere. Better yet, I wouldn’t need a bucket to flush the loo if I did break something.

The two buttons, and correspondingly the valve cylinders, are for full flush and half flush. I couldn’t see a significant difference between the flushes but as I’d read at least one product reviewer saying that they had the same experience I didn’t investigate further.

In passing, I wondered how to compare flushes and found that test poo is a thing (!)

There’s a diagram on the back of the button plate that shows which valve is for full flushes. When I took all the pieces off, I’d noticed that the big (full flush) button was connected to the half-flush cylinder. Did I mention that I took photos before removing any pieces so I know how to put them back?

My next move was to try connecting the buttons up in the recommended configuration on the hypothesis that someone had previously noticed a weak flush and simply swapped the tubes over. Nice idea, I had been wondering if I might be able to do that myself as a quick and dirty fix.

It did appear that the power of the flushes was worse that way, so it’s possible that’s what had happened. There were few enough combinations of button, bellows, and cylinder that I could try them all, so I did, but none showed a significant improvement. 

I also tried blowing down the pneumatic tubes as the diagnostic text at Amazon had suggested. I learned something there too: it takes a lot of air pressure to force the valve cylinder down and the only thing getting flushed was me.

Where had I got to at this point? Well, I was reasonably confident that I knew a set of components that contained the problem and I knew that I could order a service pack (SVO4567) for that set. The existence of a specific pack made me think that it was worth replacing all of them together, and that’s what I decided to do.

I wasn’t quite finished with the research though. I wanted to make sure I bought genuine parts from a vendor with a good reputation and paid the going rate with delivery in a reasonable time. I had a list of sites (yeah, I bookmarked them as I went) so that just took a few minutes and around 30 quid. 

Within a couple of days the parts were at my house and I was confident that I could fit them and that I’d know quickly whether they’d improved the situation. While I was fiddling I again tried fitting the big button to the full-flush cylinder but my impression was that it gave a marginally weaker flush so perhaps the valve is on the way out too. I’m not bothered though, the flush is strong enough that it’s not a problem so I left that alone for now.

--00-- 

Great, you’re probably thinking, nice story and I’m really pleased your toilet is working again, but how is this relevant to me? Where’s the testing here?

Good questions and I’d answer that it’s about the approach, the skills, the tool selection, the framing, the intent, the engagement, the decision-making, the documentation. What I did in the bathroom overlaps considerably with how I approach testing.

Imagine I’d described this scenario:

I was presented with a system I hadn’t seen before and told that it had a problem. I have tested similar products so the observed symptoms gave me a working hypothesis but, without knowledge of the specific system, I had difficulty even getting into a position to frame an experiment.

I moved into discovery mode and explored the external surfaces of the system under test, eventually discovering a way to expose information about its back end. It turned out to be an old technology and I had to do research, which included wandering down some dead ends, before finding that it had an API in common with a modern library.

Unfortunately, the API wasn’t well documented but by carefully stitching together information from a bunch of blogs, open source projects, and YouTube videos while exercising the API using a client I know well, I was able to gather enough information to understand some of what I was looking at.

I could now tell that my original hypothesis about the cause of the issue was incorrect but I also had the tools and knowledge to run some experiments which resulted in the discovery that the system under test’s front-end HTTP module was outdated and so partially incompatible with the back-end. 

Upgrading it was easy enough and now the SUT’s client and server are talking happily again. I also have a reasonable mental model of part of the system and I’ve learned about some technologies I was unfamiliar with.

That sounds more like testing, right? But it’s exactly the structure of the investigation I did for my toilet. It also explains why you hear people make analogies between testers and journalists, detectives, and scientists — those roles, done well, also require the ability to explore a space of possibilities, seek the unknown unknowns, and make connections between disparate data.

Jon Bach drew a wonderful representation of his exploratory testing that speaks to me about this:

The Process of Design Squiggle by Damien Newman has similar characteristics. Designers explore too.

I’ve tried to represent the non-linearity and iteration and reiteration in data gathering and hypothesis generation and experimentation that I see and feel in testing multiple times. Here’s one attempt:

It’s exploration, and that’s how I like to test: by exploring. 

--00-- 

The thing about exploration is that it’s not easily predictable, not deterministic, and heavily dependent on who’s exploring, what they’re trying to do, the order they do it, what they encounter along the way, and how they react to all of that.

In any non-trivial exploration there will be choices to make. With each choice there is an opportunity cost: by choosing to do some particular thing (right now) you choose not to do many possible something elses. (“Decision point. Tinker a little, call a plumber, or see what I could find online for this kind of cistern setup?”)

I try to be alert to choice points because intent is a crucial component of testing for me. At each stage I’ll understand what my current mission is, and then use my choices to deliberately guide my exploration towards the completion of that mission. (“At that point, I was trying to find something that would help me decide how to proceed.”)

This kind of guided or managed exploration is a wonderful way to bootstrap knowledge, trading breadth and depth of investigation, avoiding rabbit holes and shallow time-wasting to navigate towards value in the time available. 

One way to create explicit choice points is time-boxing. I think of this as a way of managing my investment. How much time am I prepared to invest in this next step? How important is it to me to find out the thing I am looking for here? If that’s difficult, can I find a smaller step that will help me to answer it and invest a small amount of time in that? (“I decided to spend a few minutes gathering information.”)

When I explore, even with intent, I often find that in retrospect I could have taken a more direct route to some result. That’s normal. I treat it as learning and try to use it to inform my testing choices in the future. (“I could have got to this point immediately.”)

Learning won’t just be for the future, though. Each step, each iteration, each connection made can incrementally add or adjust knowledge (“my mental model of this toilet was very flawed”). Again, this is normal and in fact desirable. There is no shame in finding that a hypothesis was incorrect and updating perceptions as a result. 

Keeping track of what I’ve learned, the routes I’ve tried, and potential new starting points helps to guide the exploration and enhances my ability to cross-reference. (“I took photos before removing any pieces”, “I bookmarked them as I went.”) Different degrees of record-keeping will be appropriate depending on the mission and the explorer.

Using the system under test and other sources as mutual oracles can be immensely valuable. Cross-referencing observations of the SUT with information found elsewhere gives three options: a problem in the system, some duff information, or both. 

Exploration in and of itself says nothing about the tools used to explore with. Knowing tools that I use frequently well, and knowing about the existence of tools for tasks I’ve never performed, helps me to have options. I was telling my friend Neil about the work I’d done and he asked whether I’d used a reverse image search on the button panel. I hadn’t and it hadn’t even occurred to me to try. The tool is now in my kit for next time. 

Analogies are never perfect, though. What might make my toilet story and my testing mission different? Well, there’s no developer character in my story. If I’d called a plumber in, someone with deep hands-on experience, that would have been similar to asking the developer of the system I’m working on. In this case, an experienced plumber would likely have diagnosed and fixed my toilet in about, umm, 30 seconds instead of a couple of hours.

If the toilet had been leaking gallons of water into the bathroom I would definitely have called a plumber. But the stakes weren’t high, and I explored in such a way that the stakes remained low (“the toilet could still be flushed with a bucket of water if needed”).

When I have had tradespeople in, I’ve sometimes asked if it would be OK to watch them work, or asked them to explain how they diagnosed the problem once they’ve finished. I’ve seen developers be amazing at exploration too  — I’m not saying it’s a tester superpower — and I’ve learnt much sitting at the shoulder of a colleague while we talk about what the problem is, where the faults might lie, how we might experiment to discover more, how we might work around what we find, and so on. 

I’m interested in, and get great pleasure from, the investigation, and in broadening my skills and knowledge. As in life, in testing; my default is generally to look into a product issue I’ve observed before reporting it, for an appropriate amount of time given the context. 

If your testing extends from picking up a ticket through to exercising precisely the acceptance criteria as described, and closing the ticket then I’m afraid you’re not seeing the intellectual, social, and emotional pleasures of exploration, you’re probably treading and retreading the same paths, and, at best, you’ll be doing bog-standard testing. 

--00-- 

Selected further reading:

• I spoke about How I Test Anything at OnlineTestConf 2020. 
• Maaret Pyhäjärvi promotes exploratory testing and maintains an Exploratory Testing Index.
• Session-Based Test Management is one formal way to manage exploration.   

Thank you to Conor Fitzgerald and Neil Younger for the reviews and suggestions.