A Gradual Decline into Disorder

I like to listen to podcasts on my walk to work and I try to interleave testing stuff with general science and technology. The other day a chap from Cambridge University was talking about entropy and, more particularly, the idea that the natural state of things is to drift towards disorder.

Entropy: "Historically, the concept of entropy evolved in order to explain why some processes (permitted by conservation laws) occur spontaneously while their time reversals (also permitted by conservation laws) do not; systems tend to progress in the direction of increasing entropy." 

In an ironic reversal of its naturally confused state, my brain spontaneously organised a couple of previously distinct notions (entropy in the natural world and the state of a code base) and started churning out ideas:

• Is the development of a piece of code over time an analogue of entropy in the universe? Could we say that as more commits are made, the codebase becomes more fragmented and any original order becomes slowly obscured?
• Is it possible that a metric based on entropy could be derived to measure the "organisedness" of a codebase, something like a cyclomatic complexity analysis? And, if so, could this metric predict software quality?
• Could there be strategies, perhaps again analogues of the natural world, for holding back the increasing entropy in the software?
• Could testing, with an aim of creating order at the external interface to an application, actually make the internal entropy worse by forcing more edits?

I strolled into the office feeling quite chuffed, made a cup of tea, turned my computer on, noted the 500 emails, ignored them, searched for "entropy testing" and found that, as usual, the good ship Original Thought had long since sailed. My day began its gradual decline into disorder.

Credit where it's due:

• Software Entropy
• Entropy as a metric on the known disorder of the system 
• Testing to reduce the entropy in an application 
• Testing increases entropy 
• Minimise the impact of testing on entropy

Image: graur razvan ionut / FreeDigitalPhotos.net

The Ass in the Assumptions

Assume and make an ass out of u and me is how the old saw goes. But when we're testing we're always making assumptions. We can't test everything so we take a view on the least risky areas and test them least. We're assuming that our risk analysis is reasonable, based on whatever evidence is available. We're up front about it - we may even have been directed to do it - and stakeholders have visibility of it.

However, often assumptions aren't prominent. This may be because we didn't think it was worth documenting them, or that we didn't even know we were making them. The second set are the more troublesome. We need to be clear in our own mind what we think it is we are doing and what information we think we are going to get from it.

For instance, if we're testing a new feature and we think it affects components X, Y and Z, we need to be aware that what we're doing is restricting our test space and we need to state that assumption up front because otherwise no-one will have any reason to tell us that it's critical in component W too. Other people will generally assume we know what we're doing, and that we know what they know. (You'll do it yourself too - another assumption to watch out for.)

Or let's say we're running some profiling experiments that generate a lot of data so we decide to write it to the shared "big disk". Are we thinking about whether that's a network disk, whether other people are accessing it at the same time, what its RAID level is, whether it's NFS or ZFS? Are we thinking about whether any of those things make any difference to the validity of our test for the result we're trying to obtain, such as whether it's what the previous figures used?

If we don't think about and share what is implicit in our testing we'll end up the ass in our assumptions, u and me.
Image: Juan Gnecco / FreeDigitalPhotos.net

The U In User

I ask my test team to be proxy users. They can and do report issues over, above, outside, around, across and regardless of any spec, convention or any other consideration if they feel that customers will find it an issue. To help to keep the team in step with customer thinking, all QA engineers are watchers on support traffic. (This has other benefits too, but that's for another time.)

When you're doing this, especially in exploratory work, you have to be careful not to confuse your user with your self. You just won't have a single model user and, even if you did, the chances of it or any particular user having your own set of behaviours and prejudices is small (although obviously your behaviours and prejudices are the optimal set of behaviours and prejudices to have).

So, if you're navigating the product and you like to use short cuts, don't just use short cuts. Take the slow route with mouse clicks too because some of your users will. Hover over all the components you usually ignore. Perhaps the infotips or tooltips are wrong. When was the last time you looked, I mean actually read?  How about all those features that have been in the product so long you barely remember what they do? Your new users don't know they're deprecated so dust them off. Delete all your product preferences. Yes, especially the ones that make it pleasant to use.  Take the time and effort to put yourself in a different frame of mind.

When you're testing you're there for the user, but the user isn't you.
Image licensed by GFDL

Every Second Counts

You're busy and you'd like to squeeze some extra time out of the day without spending more time in the office? You should look for ways to make small repetitive tasks more efficient. I'm not talking about test automation (although that's certainly something you should be open to and looking for) but the kinds of things you do all day every day, probably without even thinking about them.

You almost certainly swap between applications under test, test harnesses, email clients, editors, command lines, word processors, spreadsheets, bug trackers, browsers and so on tens or hundreds of times a day. If you're a mouse jockey you probably spend a few seconds mousing to the task bar, clicking on the next application, mousing back up and clicking in the application to get focus. Did you know that Alt-Tab throws up a quick task switcher?

You probably have to edit and run scripts at the command line. Do you find yourself repeatedly opening an editor, editing a script, closing the editor, typing the command to run the test, opening the editor on a results file,  closing the editor, opening the editor on the script, editing the script ...? Then you should consider (a) having a couple of shells open at once, one to run in and one to edit in, (b) becoming more familiar with your editor so that you can have multiple buffers open at once and (c) learning enough about shells to know that tab-completion, short cuts and cursor keys can optimise command line interactions.

Any time you find yourself doing the same thing again and again, ask whether you can cut it out: use the keyboard for copying and pasting, install helper applications for missing features (e.g. wikEd for wiki editing) use utilities like grep rather than eyeballing documents, use bookmarks on your web browser for all the sites you usually browse to from your intranet home page ...

Time not spent on the process of  getting to the next task is time not wasted and, more important, time where you're not losing context. In another installment of an occasional series on the software development philosophies of celebrities, I'm with Paul Daniels when he says Every Second Counts.

Sympathy for the Dev Team

We had to update a mature regression test suite recently. The task was effectively moving a set of tests from a source data set that was no longer supported to its replacement. For product reasons this wasn't a trivial undertaking, but we could anticipate most of the problems and decide in advance on a strategy that would make the migration as straightforward as possible.

Unfortunately, as the source had been in the test system for several years dependencies had built up around it. Dependencies that were reasonable pragmatic choices at the time, or forced short-cuts to meet a deadline or just plain wrong but dependencies nonetheless. Of course, as soon as we retired the old source data, a selection of apparently unrelated tests started falling over.

Because this is the real world, our remedial action has only removed the dependencies in some cases and just made the suites run again in others. We're filing bug tickets for ourselves (you do maintain a bug database for your tests don't you?) against the latter cases and we'll hopefully have opportunity to really fix them before we go through this again.

This is a pain for us, but think about your colleagues in Development for a minute. Their codebase is probably older and more mature than yours, much larger and more diverse than yours, gets hacked about by more hands than yours and changes much more frequently than yours ever will. Even with careful handling and refactoring, unwanted dependencies are introduced. Every little bug you report has the potential to bring something seemingly unrelated crashing down around their ears, and they know it.

In this area, as in so many others in testing, me and Mick Jagger see things the same way: we've got sympathy for the Dev team.