Red Testing Hood

Angie Jones, The Build That Cried Broken

Like the boy who cried wolf, the build that’s repeatedly red for no good reason will not be trusted. Worse, it will likely end up in a persistent red state as people stop looking at the regular failures. Eventually something important will also fail ... and no-one will notice.

At CAST 2021, Angie told us this and other stories about a team she was on, one who found themselves in that bad place but used their wits to escape and lived happily ever after. 

Well, perhaps that’s an exaggeration: they tolerated living in a short-term bearable state where a reliable kernel of tests supported development and deployment and a set of flaky tests were worked on to the side.

Separating off the flakes was a good move but it was supported by others, including assigning a team member to investigate all failures and setting limits on the percentage of tests that could be in the flaky set before all other activity stopped. Focus was firmly on fixing today’s problems rather than grand plans for tomorrow.

The team also made Jira tickets for each failure investigation and prioritised each one explicitly. As a rule of thumb, Angie would delete and rewrite from scratch any case that had been edited significantly more than three times. 

If there's one principle over all of these, it's to prefer fewer relevant, reliable, and important tests over simple large numbers. Very much not "Oh Grandma, what a large test suite you've got!"

AI Needs Testers

Tariq King, Towards Better Software: How Testers Can Revolutionize AI and Machine Learning

Tariq issued a powerful call to arms in his talk at CAST 2021. Yes, AI/ML is new technology and, yes, it is solving new problems, and so, yes, testers may find themselves out of their comfort zones in some respect.

But there are parallels between machine learning and testing that should give testers confidence that they have something valuable to contribute right now: mapping inputs to outcomes for black boxes, partitioning a possibility space, and data wrangling. 

Learning is part and parcel of testing, so why think we can’t learn what’s needed for working with AI and ML systems? And don’t forget that testers come preloaded with universally-valuable skills such as exploring, questioning, and assessing.

Without some kind of challenge, Tariq says, AI is going to continue powering the runaway train of bad software. Testers can and should be standing squarely in front of it.

Laugh Don't Cry

Laurie Sirois, Quality Isn’t Funny

Why didn’t we find this bug in testing? 

Without a sense of humour, hearing that kind of question repeatedly could bring people to tears. At CAST 2021 Laurie Sirois encouraged us to deploy laughter to defuse those tense situations, improve our relationships, and grow other people’s perceptions of us. As a manager in particular, lightening the mood can do wonders for a team’s morale, creativity, and sense of safety.

Care needs to be taken over the style and timing of the humour used. Sarcasm and inside jokes might work well with trusted peers but may not be appropriate when delivering feedback. Even self-deprecating  humour can make others uncomfortable in the wrong context.

Sounds challenging? Don't worry. If you’re not a natural stand-up simply smiling more frequently is a good start and it turns out that sharing surprising insights (the aha!) can be as effective as a joke (the haha!)

I have a bit of history in this area myself. At EuroSTAR 2015 I talked about how testing and joking share some significant features. For example, the punchline for a joke could be a violation of some expectation, the exposure of some ambiguity, an observation that no one else has made, or just making a surprising connection. 

Put this together with Laurie's presentation and we can wonder whether testers are especially well-suited to using humour at work.

Scale Model

Greg Sypolt, Building a Better Tomorrow with Model-Based Testing

As he was telling us at CAST 2021, Greg’s team have built a model-based testing system and integrated it into continuous integration infrastructure which has scaled to be capable of exhaustively exercising the 30 or 40 deployment scenarios that each of their products supports.

The models bring advantages such as describing system behaviour in reusable chunks that are independent of implementation details, making maintenance straightforward, and broadening coverage.

Sounds great, and is, but it comes at a price. Getting buy-in for this kind of approach — from both management and the team — can be tricky and  there’s a lot of up-front effort, unfamiliar concepts and technology, and steep learning curves.

The models Greg needs can be quite simple because each product is basically a linear navigation through a sequence of pages. When the system runs, test cases are generated from the models, permutations of cases are created for each environment, and then Cypress and Applitools exercise the systems under test.

Interestingly, the team makes “models” for steps they want to perform, as well as for the system behaviour. This is theoretically questionable but a pragmatic and practical approach to, for example, invoking Applitools or waiting for page load completion. 

Perhaps more interesting, there is sufficient confidence in this approach in this context that exploratory testing is crowd-sourced, with the team relying heavily on the framework to perform a (simulated) mouse, (simulated) keyboard, and visual check on each component via the model.

Testing Hats Can Be White

Rajni Hatti, Ethical Hacking for Testers

Testers should not feel excluded from exploring security concerns just because specialists are available or tooling (such as the ZAP scanner) is running in continuous integration. 

Why? Rajni gave three reasons at CAST 2021:

• Testers tend to have a big-picture perspective and so perhaps ideas about where there might be vulnerabilities outside of standard attack vectors. 
• Testers are more likely to be involved in the design of features and so able to ask security questions or influence the priority of security in development backlogs.
• Security is a process not a product, and so regular review throughout the cycle is desirable versus throwing a build over the wall to some other team.

Naturally, there is opportunity cost associated with additional security testing, so the choice about what to do and how much time to spend on it should be determined by context. Similarly, any risk assessment should be a living document and oracles should be appropriate for the application under test at the time of testing for the purposes of the testing.

Rajni talked us through a project where she based a risk assessment on the OWASP Top Ten, looked at static code analysis tools for vulnerability scanning, at dynamic tools for fuzzing, and at design review to identify sensitive data obscured by naming convention.

Her experience was that even with a team of white hats at the company, the testing hat can still add value. And when security professionals are available, testers should be looking to collaborate with and learn from them to improve their own work.

Don't Just Check, Please

Ben Simo, Computer-Assisted Testing

Ben kicked off CAST 2021 with a brief history lesson, tracing the use of the term checking as a tactic in software testing back to at least Daniel McCracken’s Digital Computer Programming from 1957 and through into his own recent model. Checking for him is a confirmatory activity, focusing on knowns and attempting to demonstrate that what was known yesterday has not changed today. 

Checking need not be performed by machine but it’s a common target for automation because it comes with a straightforward decision function: the previously-known state.  In fact, this is for many all of what “test automation” is or can be and numerous regression test frameworks exist to facilitate that kind of work. 

Ben would, I think, reject the both the term and the limited thinking about where computer tooling can be valuable for testers. In his model, testing is an exploratory, investigative activity, focused on unknowns, providing challenges to the system under test and processing its responses in ways that make human assessment of them tractable.

I find myself less bothered about the term, enjoying the model a great deal, and very interested in breaking through the idea that automation, and tooling more generally, has nothing to contribute to exploration.

The remainder of Ben’s talk was a tour-de-force of ways in which he has built tools that assisted testers in their testing. These were not one-size-fits all utilities, but chosen to suit the context in which their need arose. 

In one example, the tool asked the tester for the constraints they were interested in imposing on a system under test, generated test data that fitted, ran with that data, looked for anomalies, and collected trace data to help diagnosis. In a second, a similar starting point resulted in the creation of a test environment.

For another project, the tool ran the same sets of keystrokes on multiple deployments of a product in different tech stacks. There were no asserts, simply comparison of the responses. If they differed then there might be a problem and a human was asked to investigate. As patterns emerged, and permissible differences were identified, the system was enhanced with heuristics that would suppress requests for human inspection.

Other approaches built and explored state models; implemented sophisticated logic into a log aggregator by creating derived columns and querying over them; and harvested production data to drive load tests.

Even if this kind of material were not right in my wheelhouse (see e.g. Exploratory Tooling or Exploring to the Choir) I'd still see Ben's examples as powerful illustrations of the value of context-driven testing by a skilled practitioner.  

The Eye of the Beholder

With the wrong lens, or the wrong position relative to the lens, your vision will be compromised.

And this is true no matter how hard or how long you look.

You will not see all that you could see and what you do see may not be what you think it is.

But you don't wear glasses?

We all wear glasses; literally, metaphorically, or both.

Every time you engage with a system under test you are viewing it through a lens.

So what are some things you can do? 

Accept that your view is always mediated by the way you are viewing.

Do your best to understand the lenses available to you.

Try to choose a lens that is compatible with your intent. 

Take opportunities to to test an observation through multiple lenses.

Use those tests to gather data on the lenses as well as the system

Be open to trying a range of lenses.

Attempt to use any given lens in a range of ways.

Remember always that observation, like beauty, is in the eye of the beholder.

Image: https://flic.kr/p/8s82M2