On a Hiding to Nothing

Is it a bug when you're offered the chance to see something similar to nothing and, when you take it, nothing is displayed?

Or is it a bug to be offering the chance in the first place?

Image: http://flic.kr/p/5WvWMF

Sheet Happens

You don't have to look very hard to find definitions of quality. Some of them aren't very good (boom! boom!) and one with a lot of currency runs along these lines: 

Quality is value to some person (who matters)

in this case quoted from James Bach's Rapid Software Testing slides. 

Here's the price ticket for toilet roll I saw in the supermarket this morning. It adds a new dimension to the phrase spending a penny: 

Is there a quality issue here? I'm not sure whether I matter much to Tesco and it's arguable whether any value was lost by the use of this unfortunate abbreviation - I still bought some toilet paper - but if your customers are laughing at you rather than with you, there's probably a quality issue somewhere.

Image: winnond / FreeDigitalPhotos.net

Defect Three

Purely by chance I encountered three takes on perfection in the space of a couple of days.

"Perfection", Run DMC said, "is quite essential". A laudable aspiration but, as their later career showed handsomely, hard to achieve. Compromise has a lot of very persuasive arguments.

"This software is bug-free. It is perfect", Fast Company said of the Space Shuttle's control system, "as perfect as human beings have achieved ... the last three versions of the program - each 420,000 lines long - had just one error each." Skipping over how they knew that there was just one error, and why they didn't fix it, NASA had obvious reasons to strive for perfection, including the need to certify "that the software will not endanger the shuttle" before launch. Their approach was to be meticulous in the meta-work that supports software production as well as the production itself. But they still had errors.

"We test", Weinberg said, "because people are not perfect".  Weinberg's book, Perfect Software And Other Illusions About Testing, is a classification of the ways in which the human factor - in the coding, the development process, the testing, the management or anywhere else in the creation of a piece of software - rules out the possibility of perfection.

So what lessons are there for us?

• Software should generally be tested (the extent and rigour determined by the context) because the people who made it are unlikely to have made it defect free.
• Testing is not just an engagement with the software but an implicit interaction with the people involved in the software. 
• We often put ourselves in the mind of the user but we should also be putting ourselves in the mind of the software creators, the business development team, the people who designed the process, the GUI designer, the QA manager and the rest. They aren't black boxes, so what do we know about them, their strengths, frailties, foibles, blindspots, that will give us an insight to exploit in tests?
• Stopping at Raising Hell would have been perfect.

Image: http://flic.kr/p/9Ca3Z4

Bogof Testing

We all like a bargain and, if there was such a thing, we'd tuck into the free lunches with gusto too. But bargains needs to be viewed with caution and free lunch is usually only about half right - it probably is lunch.

Automated testing is often thought to be a bargain or a free lunch but even limited experience will have shown you that it's not something for nothing and not even always cheap, especially in terms of maintenance. You should think carefully about whether regression tests are appropriate, what you hope to achieve, how, using what resources and attempt some level of cost/benefit analysis (using whatever data, estimates, gut feeling you have available) before you start on an automation project to try to maximise its value.

But existing infrastructure can have unexploited value. For example, it is sometimes possible to double up regression test suites and test something other than what was originally intended.

In my company we have both a traditional software development team and a resource development team that produces plug-ins, such as configuration files for processing different source data formats. A software regression test will usually use a static set of data inputs with a variable executable, like so:

The harness will essentially run the executable on the inputs to generate results, then diff these against the  golds (the expected results) to generate the report. A change needs investigation to see whether it's intended or accidental.

However, we can use exactly the same machinery to provide a regression test for the resource development team to use by making the configuration files variable too - perhaps we'll check the latest revision out of version control before running:

Now we have two variables in the suite but as we're already running the suite with a single variable (the executable) we can easily tell whether changes are due to the executable. Any changes from the second suite are due to the configuration files (or in rare cases a bad interaction of the new files and the new build). The cost of this? The expense of checking out the latest configs and one more run of the suite in your overnight regression set.

The same harness can be used with a static executable. When the resource development team are creating materials for a previous release, being able to run against an unchanging branch build is valuable. But remember that you need to be sure your golds are relevant to that build. (You do branch your test code base in sync with the product, don't you?)

Another way to exploit existing infrastructure is when you have two product components in a producer-consumer relationship. The first creates a resource and the second uses it. If you have a regression suite for validation of the consumer, you can reuse it in the test suite for the producer.

By configuring the suites to consume or produce the "same" resource (i.e. one that if nothing has changed in the software will be equivalent) we can re-run the consumption test suite on a dynamically-produced resource to give feedback on whether the resource creation has been successful - although note that this only tests the dimensions that the consumer cares about.

If this shows changes we may have accidentally broken the producer. This has more value still because the resource, once changes are validated, become the next set of static resource for the consumption suite, which saves us having to regenerate it manually:

Note that this is different from test suites that rely on the output from another suite for their primary input. While that might be a valid approach (e.g. because it is expensive to generate the data), it creates a dependency that you might prefer not to have and timings may become an issue. For instance, if you're running all your suites nightly and the precursor suite takes twice as long as usual, any downstream suite that depends on its output being in place by a certain time may fail for lack of data.

You might never get your lunch paid for, but you could cover the cost of the starters if you always look for  value: plan your test suites, cost them, build them efficiently and for extensibility when you can, run them as often as makes sense and look out for special offers like buy one, get one free.
Image: Salvatore Vuono / FreeDigitalPhotos.net

Wrong is not not Right

I saw a blue being from another dimension clawing its way through the ceiling the other day.

If only.

The building I work in is being redecorated and on the way to the canteen  I noticed that the painters had covered a smoke detector with a rubber glove. You don't need to be a health and safety officer to realise that this is not standard policy and potentially risky.

But, paint fumes can set off smoke detectors and there were plenty of people around and there are many other smoke detectors in the building and the lack of other detectors covered with gloves suggested that decorators would be likely to remove it when they'd moved further down the building so I decided to do nothing about it. A clear spec violation, but mitigated by other factors. A reasonable short-term solution to an acute issue, but on the way home I still walked that way and checked that it had gone.

You'll sometimes hear testers described as pedantic and it's true that the detail and process-orientated aspects of our work are apt to be misinterpreted as (or can actually slide into) the rigid adherence to requirements.

It's not worth getting worked up about the term itself because, for example, to people who operate at helicopter altitudes on projects, the details are often just that, the details, the potential stains on the big picture and not their main focus or responsibility. And let's not kid ourselves. It works both ways and I'm sure you've got a few favoured adjectives for the teams you work with on a regular basis.

It is worth thinking about whether the description has some validity, and trying to demonstrate that it's not a fair blanket description of testers all the time. Look at yourself and wonder whether (a) now is the right forum and time for raising questions of detail; (b) whether any particular wrong you're concerned about really is not right given all the context or (c) whether despite being wrong it's also perhaps not not right for now and (d) whether you should monitor it and/or record the need to change it later, when it's more likely to be invalid and risky to have a spot fix buried in the codebase.