Thursday, August 3, 2017

On Mapping Non-testable Papers

The Test team book club at Linguamatics read On Testing Non-testable Programs by Elaine Weyuker this week. As usual, the discussion was interesting and, as usual, the reading material was only the starting point and, as usual, we found ourselves exploring our own context and sharing our own experiences and ideas.

I find this kind of interaction invigorating and energising. It remains fascinating to me that we each bring common and unique perspectives to these meetings and I thrive on hearing others on my team talk about how they see the topic, I covet the time I spend thinking about how I do, and then I enjoy immensely contrasting the two.

I had wondered, while reading the paper, whether I could extract some kind of ontology of oracles from it. Informally, it seemed that Weyuker had structured her analysis in this way: programs are testable or not; these are characteristics of non-testable programs; non-testable programs are of three types; these approaches to oracles can be used with such and such a type, ...

So, to explore that notion, I went back to the paper and gave myself an hour to re-read it, watch a short video by Cem Kaner which references it, and make a mind map. (I'm also interested in experimenting with getting value from mind maps.)

Unsurprisingly, you might say, when studying the paper more closely, with a particular purpose in mind, I found that my informal analysis was a simplistic analysis. In order to map the details of the structure I was interested in, I had to think harder than when I was merely absorbing the higher-level points.

To give one example, Weyuker breaks non-testable programs into three types early in the paper:
It is interesting to attempt to identify classes of programs which are non-testable. These include: (1) programs which were written to determine the answer. If the correct answer were known, there would have been no need to write the program; (2) programs which produce so much output that it is impractical to verify all of it; (3) programs for which the tester has a misconception.
Then, later, in a section about approaches for dealing with two of those types, she says
For those programs deemed non-testable due to a lack of knowledge of the correct answer ... In the case of programs which produce excessive amount of output ... One other class of non-testable programs deserves mention. These are programs for which not only an oracle is lacking, but it is not even possible to determine the plausibility of the output.
Is this third class the same as the original third class? Or different? (And, as a writer, what can I learn from that?)

Here's the map I produced in my timebox:


Edit: Doug Hoffman published an article, A Taxonomy for Test Oracles, back in 1998. 

No comments:

Post a Comment