Thursday, October 11, 2012
We've all done it. We've found an issue in passing and reported it quickly and it got triaged and it was P3 and no-one died and that was that. Until the underlying problem, of which we'd only observed a relatively benign symptom, burst out somewhere else in the product and turned out to be P1 critical. Late in the project, natch.
We all know it. We can't test everything, even if we wanted to. But we're naturally cautious and we'd always prefer to, when we see the P3 in passing, wonder whether there's a cheap check to give some confirmation of our assessment. And if there is, we can use our testing instinct to decide whether to run it.
We all know it doesn't work like this every time: on a system I was looking at once, I noticed that a particular account was being rejected on login. The cause of the rejection was understood and it was P3 all the way. But account issues can be sensitive and there was a parallel operation I could look at, for low cost. So I logged in with a different account and then switched to the original one. Rather than being rejected, the operation was permitted and presented an obviously inconsistent state to the user. Ding! P1.