Saturday, August 1, 2020

The Other Sides

A developer I don't work with often made a change, tested it against the problem case that motivated it, and then asked me to have a look.

I started by reading his description of the testing he'd done, reviewed his commits, and then ran the application with and without the change. After testing for a short while I went back to him having found a major problem.

This isn't a blog post slagging off how developers (don't) test, it's also not about how testers should strive to create a quality culture and coach the team, and it's not about how clever I was at finding the problem either.

This blog post is about the response the developer gave when he heard my report: "what made you think to test it that way?"

Which is a great question from someone who wants to improve themselves and their work. I didn't want to put him off ever asking me anything ever again by taking the opportunity of an apparently receptive audience to lard on the testing know-how. So I said something along these lines:

There are basically two sides to testing a fix like this, and you need to look at both:
    • checking for the intended effect
    • looking for unintended consequences 
When I'm testing I will often try to find ways of cheaply giving myself a chance to see unintended side-effects. In this case, it was to force the fix to be exercised lots of times on very little data. This makes the outcome easy to scan and problems easier to see, for example wrong results or missing application of the fix. 

It is easy to be indignant or critical when finding a problem quickly, but I do know that when developing it's easy to skip or lightly touch the looking for unintended consequences. Hell, just last week I committed changes I didn't mean to because I didn't look first, unjustifiably confident in my shallow checks, keen to capture the good work I'd done.

So while this is a blog post about the developer's excellent attitude (and I told him I thought it was excellent) I'll slip in a note-to-self too: remember to have empathy for the person whose work I'm looking at. Yes, I found an issue here but there's other sides to every situation.

Sunday, July 26, 2020

Are Your Latch On?

The other week I found myself locked out of our shed and subsequently learned more than I ever expected to about Yale locks, or night latches as I now know they're called.

The image at the top is a pretty standard night latch. It is opened from the outside with a key and from the inside with the handle. The latch (the gold tongue on the top left image) is sprung, which means that simply closing the door will push the latch onto the striker plate and into the box (both top right), locking it.  A deadlock which stops the latch from moving can be applied from the inside using the button (or, more correctly, the snib).

Night latches are an old technology, insecure, and make it easy to accidentally lock yourself out. The snib helps with the last of these by being able to hold the latch back inside the body of the lock. This means that even if the door closes, the latch can't engage and the door remains unlocked.

Which is nice to know, but my problem was that I couldn't get in. I tried a bunch of things, searched the web and tried some more things, thought about trying to drill the lock ... and then called a locksmith.

When he turned up he unlocked the door in 10 minutes with some airbags that forced the door slowly open while he aggressively turned the key back and forth. You gotta know where to tap.

Once in, we could tell that the door had been closed with the snib in the deadlock position. The manufacturer's intention is that the deadlock is used when the door is shut but there is nothing to stop it being used with the door open. Even so, that usually won't be a problem because the deadlocked latch will bump against the strike plate instead of sliding in, and the door will stay open.

However, if the door doesn't fit particularly well, and if it's slammed shut, the latch could be forced home and then the lock won't open even with the key. There's a reverse scenario too, where the latch is snibbed inside the body of the lock and the door is pulled to without locking, leaving a gaping security hole.

We usually use the snib to keep the door on the latch when we're going in and out of the shed but I have no way of knowing how many times we might have been at risk of either of those unwanted outcomes.

As testers, this feels like familiar territory doesn't it? We have some functionality which is ...
  • ... intended to be used a particular way
  • ... and effective when used in the intended way
  • ... but possible to be used in an unintended way
  • ... and generally harmless when used in the unintended way
  • ... until sometimes when it turns out to be harmful.

More familiar territory, thinking of options and their trade-offs. Here's some:
  • Change the lock for something more modern. OK, but costs money and short-term hassle.
  • Stop using the snib at all. OK, cheap but means we need to carry a key all the time.
  • Don't make a mistake with the snib. OK, cheap and easy, but we are human.

Right now, I've gone with the last one by (familiar territory again) taking inspiration from Jerry Weinberg:

Image: Yale

Saturday, July 18, 2020

Community Building

David Högberg tagged me on a thread about teams in the Rapid Software Testing Slack the other day. I've paraphrased the conversation here:
Our team has expanded to include groups working on many different products. I'm thinking of starting a Testing Community of Practice with a Slack channel and perhaps a monthly meeting. I'd like to share things like articles, how we test our applications, interesting bugs, what we've learned, the business problems our products solve, who our users are, and so on. Looking for thoughts, ideas, advice, articles, etc.
I've spoken to David about this stuff in the past, and I'm an agreeable kind of chap, so I started listing some of the things my team at Linguamatics has done over the years until it became apparent that I had quite a lot to say and I'd be better off typing it up in a proper editor and posting it somewhere other than Slack. So here it is.

-- 00 --

Our setup is a little different to the scenario above in that we are a standalone Test team whose members support multiple development teams by working in them or very closely with them on a day-to-day basis. However, despite being nominally part of the same functional team, it can be a challenge for testers to keep abreast of what's going on elsewhere in the company (in terms of product or testing), to that get broader context for their work, or contextualise the work of others.

Because we recognise this challenge, we've experimented with a bunch of different devices for collaboration, information sharing, and empathy building over the years. Here's some of them.

We have a weekly team meeting. It's a brief sharing of status and then a presentation on something by a member of the team or, occasionally, a guest from another part of the company. There's no formal rota on presenting but it's part of our culture that we'll each regularly present, and that includes me.

Topics are not heavily policed but are expected to be relevant to our work. For example, the tester on each core product feature will give an overview of it, the motivation for it, the implementation, how it was tested, and so on. Team members who've been to conferences will summarise something back to the team about what they learned, or found interesting. If someone's been experimenting with a new tool or approach, they might do a demo of it.

The information exchange is useful in these meetings but there are valuable side benefits too, including a safe space to practice reporting on testing, to get experience presenting, and to build empathy with team mates.

On the other four days of the week we have a stand up meeting. This is intended to be short and is a place to give relevant updates, flag issues, or ask for help. We also make sure that any test suite failure investigations that haven't been picked up are allocated in this meeting. If conversation here starts to go deep, we ask for it to be taken to some other forum. When someone has something they'd like to share quickly, they'll often offer to do it for 5-10 minutes after stand up, as an optional thing for anyone who wants to stick around.

In stand up we have to regularly remind ourselves to keep focused on outcomes over outputs. It's so easy to slip into a laundry list of "I did this, I did that, I did the other" which consumes time and achieves little of benefit. As a manager, I also resist turning them into status reports from others to me by scheduling those conversations separately.

Around Christmas I run a session that I (jokingly) call Testing Can Be Fun. This takes the form of some kind of puzzle or game which requires skills that we might also use in testing. I've written about some of these over the years, for example in Merry Cryptmas, Going On Ahead, and One Way To Test. It's good to see each other in non-standard situations, be a bit more relaxed with each other, and watch how we all approach the same problems in an area where we're equally lacking in context and expertise.

One of my team started an occasional session called What I Don't Know About ... in which someone would present what they knew about an area of one of our products and then invite the rest of the team to fill in the holes. This benefitted both the presenter and the participants equally in terms of learning, but additionally showed us that it's OK to expose our lack of knowledge and to ask for help.

We have a reading group, again facilitated by someone on the team, which meets every three weeks or so and is currently working through categories of test ideas from The Little Black Book on Test Design. We're discussing a category and then trying to apply it as a kind of mob to one of the applications we work on. Again, I've written about some of the things we've looked at in the past, e.g. Observability, Safety, and Checklists.

Some of the team are running a Python coding club that meets a couple of times a month. The current project is a tool to extract information from our continuous integration system and chart some build statistics.

We have a quarterly retrospective for our team in which we try to talk about issues (positive and negative, but naturally more of the latter) that are on us rather than part of a specific project or product. This aims to be a place where we can reflect on and constructively criticise what we do. It's also a place where people get a chance to practice facilitation in a friendly environment. In a similar vein, one of the team has recently begun a series of events to build a Team SWOT analysis.

On a regular basis, over lunch, another team member sets up TestSphere Card of the Week. The format changes on a regular basis but always involves the drawing of TestSphere cards in some combination to provoke conversation about testing.

A couple of years ago we ran an internal test conference where everyone from the team had the opportunity to present on some aspect of testing. We budgeted it as a training day, but one in which we didn't have to pay for a trainer, and set aside some work time for preparation. When we retrospected on it, more prep time would have been appreciated, but spending a day together talking about our craft was appreciated. I blogged about my own presentation in Exploring It!

Katrina Clokie inspired a team member to start a pairing experiment. Anyone who signed up would get another tester to work with once a week for a few months. It ran for a couple of years and, while pairing isn't happening all day every day on our team, it's certainly been normalised. I carried on when the experiment ended and have been pairing with a different member of my team each quarter for a long time now.

In common with many places these days, we've got chat software (currently Microsoft Teams) in which we've made a few channels for standard work topics but also an informal channel (currently called Tea, Coffee, Beer, and Wine) in which we can chew the fat about very non-work stuff. To give just one example, I've been posting pictures of the unusual crisps I've bought during lockdown.

When we grew big enough to have line managers in the team, we started Community of Practice for line management. It ran its course for us but has evolved to become a company-wide thing. We also introduced a brown bag lunch meeting that we call Team Eating and which has been running for five years now, at around one session a month. It brings together colleagues from across the company to build inter-team relationships.

We've recently been trying to set up a Community of Practice with other test groups in our parent company. This is proving more challenging for us as we're geographically distant with little shared context. Three of us from Cambridge visited a group in Brighton last year and we were intending to host a return visit before circumstances got in the way. We've managed to have several virtual meetings but nothing regular has stuck yet. I feel that there's still enthusiasm for this kind of thing, but we need to find the right format and frequency.

If that feels like a looooong list don't be fooled into thinking that we're all doing all of those things all of the time. Everything except for the daily and weekly meetings and Testing Can Be Fun is optional, and there's no negative impact or offense taken by non-attendance.

Even the meetings that are non-optional are missed by all of us from time to time, when something else needs our attention, and we trust each other to gauge when that's appropriate.

I've found that it's natural for enthusiasm to wax and wane, and that it's OK to let the initiatives that have run out of energy stop or change direction. I was delighted when Drew Pontikis said that his experience was the same in his talk, Leading Communities of Practice, recently.

Also, some of these initiatives just won't work, and all of them will likely receive criticism and scepticism from some members of the team. I wouldn't say don't listen, because you might learn something useful by doing that, but I would definitely take the perspective that the people who turn up and participate are the right people to be there.

That doesn't mean I'd judge anyone who doesn't attend negatively: it's their choice and they'll have their reasons. I once gave a presentation to one person at a local meetup (even the organiser didn't turn up that night!) and, yes, of course it was disappointing in some respects but the person who attended was very engaged and we had a great conversation.

As the team manager I try to attend everything of this kind that anyone in my team organises. I want to show support for them as an individual, and to give them confidence that they can attempt something and it will be taken seriously. If they have an idea that they'd like to try something I'll encourage them to set it up and see what happens, and how they feel about it.

Finally, while I'm very interested in sharing ideas and experience, retrospecting, and learning together, there's a secondary aim here that I don't discount at all: the chance to see each other as rounded personalities rather than as one-dimensional words in a bug report, the opportunities to build trust and personal relationships with colleagues, and places to see that our team has shared purpose and values and to discover what they are.

Friday, July 10, 2020

Stuck in the Middle

This week I wanted to monitor several pieces of software that talk to one another via HTTP and HTTPS. All are running on the same machine, three are Linux services, and one is a standalone script. I was interested in being able to see all of the communications between them in one place, in time order.

I know a couple of ways of capturing this kind of data: proxying and network sniffing

My default approach would be to have the applications configured to proxy via Fiddler running on my laptop inside the work network. Easy, right? Err, no, because I had forgotten that the machine in question is on a network that isn't considered secure and firewalls prevent that connection. In my proof of concept experiment, the standalone script just hung failing to find the proxy I had specified. Interesting behaviour, and I later reported it, but not what I needed right then. Next!

As all of the software is on the same machine, capturing network traffic into a pcap file using tcpdump should have been relatively straightforward and I could import it into Fiddler for viewing. Result! Err, no, because HTTPS traffic is not decrypted with this approach so I only got some of the comms. Next!

What if there was something like tcpdump for HTTPS? A bit of googling and I found ssldump. Result! Err, no, because although it was simple to install, I couldn't make it work quickly and the data I was trying to gather was not important enough to invest enormous amounts of time in learning a new tool. (Although knowing that this tool exists might be very useful to me in future.) Next!

Back to proxying. What about if I run a proxy on the machine itself? I remembered playing with mitmproxy a long time ago and its web page says it deals with HTTPS so I installed it. Result! Err, no, because the latest version won't run due to a C library incompatibility on this machine. A quick search on the developer forums suggests that this is a known and accepted issue:
We build our releases using PyInstaller on GitHub CI runners, and that combo doesn't allow us to support earlier glibc versions. Please go bother RedHat.
I have been burned before by trying to upgrade Linux C libraries and, again, today is not the day for deep diving into infrastructure that only facilitates a potentially interesting but not crucial experiment. Next!

Hold on, I'm not ready to give up on mitmproxy yet. Might there be an older version that depends on an earlier C library? Is there a page I can download historical versions from? There is. Result! And this time, after stepping back a major version at a time, I got 2.0 running on the box. Double result! Next!

Next is to repeat the proof of concept test with the standalone script. The script has no proxy configuration options but I know it's running Python's requests library and an earlier search told me that it should respect the Linux HTTP proxy environment variables

So, in one shell I started mitmdump, a flavour of mitmproxy:
$ ./mitmdump
Proxy server listening at
In another shell, I set the environment variables to point at the proxy's URL and ran the script:
$ export http_proxy=
$ export https_proxy=
$ ./myscript
At this stage, I don't care to know which of the proxy variables requests will respect, so I simply set them both. 

Result! HTTPS traffic appears in mitmdump's console and it's the kind of traffic I expect. POST https://myserver:8000/endpoint
              << 201 Created 91b GET https://myserver:8000/endpoint/result_data
              << 200 OK 20b

Next was to get the various services configured to proxy through the same mitm instance too. Unfortunately I found that they do not have proxy configuration options. I wondered whether they would respect the Linux environment variables but didn't know how to set them in the environments that the services ran in. I pinged the testers for those services in chat and  in parallel did some more googling. 

It seems that it's possible to set environment variables in an override file per service. Result! So I invoked the service editor and entered the runes required to set the same variables for one of the services:  
$ systemctl edit myservice


$ systemctl restart myservice

I ran the script again and this time saw traffic from both it and outbound from the service it was speaking to. Result! I quickly configured the other services in the same way and had the monitoring that I needed: all the traffic from all pieces of the infrastructure I cared about, aggregated in one time-ordered location.

In total, this took about an hour, and I spent another few minutes writing the steps up on our wiki for future reference. (Years ago I set up a wiki page called Log More From ... where we've documented the various tricks we've used over the years to get access to data around our products and tooling.)

A moment of reflection, then: I had a mission here. I didn't state it explicitly, but it was something like this: explore setting up HTTP/HTTPS monitoring using whatever tools work to get inter-application monitoring data for a specific analysis. The experiment I was engaged in was a nice-to-have. I was already reasonably confident that the right functional things were happening, and I had access to HTTP logs for some of the pieces of the infrastructure, so I didn't want this to be a time sink.

This framed the way I approached the problem. I have some background here, so I tried approaches that I was familiar with first. I used something like the plunge-in-and-quit heuristic, which I first read about in Lessons Learned in Software Testing, and which James Bach describes succinctly as: 
...pick something that looks hard to do and just attempt to do it right away. It doesn’t matter what you pick. The point is to try to do something now. You can always quit if it’s not getting done.
This mindset helps to stop me from disappearing down intellectually interesting or technically challenging rabbit holes: if it's not working, step back and try another way. 

Another thing that helps is having been in these kinds of situations before. My experience helps me to judge when to quit and when to continue plunging. Unfortunately, there's no substitute for experience. And the truth is that the judgements made even with experience can still be wrong: if I'd spent another couple of minutes working out what I was doing wrong with ssldump, perhaps I'd have satisfied my need ten minutes in? Yes, perhaps, or perhaps I'd have burned the whole hour fiddling.

On the whole I'm happy with how this went. I got what I needed in a proportionate time, I learned a little more about mitmproxy, I learned a new trick for configuring the environment of Linux services, and I learned of the existence of ssldump which could be just the tool I need in some future situation. Result! 

Sunday, June 28, 2020

Une Oeuf Blogging

When I started Hiccupps I challenged myself to do 50 posts in the first year. If I could manage one piece a week, I reckoned, I'd have a decent idea whether I really wanted to write and whether the effort I was putting in was worth it. 

At the end of that initial 12 months I found that I really did want to write and, that as far as I was concerned, the effort was worth it. In the almost nine years since then I've maintained an annual average of around 50 posts and as I've reached the half-century or October I've taken a moment to reflect on how things are going. This post will be the 456th — with something like the 400th awful pun of a title — and so now seems like a good time as any to look back down the track again.

Blog is a contraction of weblog and it occurs to me now, in a way that I'm not sure it has before, that Hiccupps genuinely is a log of stuff I've been pondering or doing or reading or viewing. The posts feel like a wander through my learning in the order I attempted it. Here's three highlights:

I continue to practice sketchnoting just enough to stop getting too rusty. Recently, I've also asked myself to write up concise notes of talks I've attended soon afterwards. Being able to get the essence of a thing down quickly is a skill I find extremely handy at work and this is a good opportunity to practice. The Cambridge Agile Exchange has been a great source of material for sketchnoting, and my favourite of the year was their Practice Makes Perfect because I got my sketchnotes down cleanly and I wrote my textual notes up within minutes of the talk finishing.

Finally, I got around to reading Billy Vaughn Koen's Definition of the Engineering Method and found that it set off fireworks in my head. No less than five posts were spun out of what is really only a pamphlet but which is packed to the gunnels with thoughtful review of how practitioners practice heuristically and the creation of a model of the shared context of the players in any implementation effort.

While I'm very pleased with my sketchnotes and reading Koen, the post I'm most proud of this year is Unsung Love Song, my review of a book written by my friend Laurence Dillon, about the rent that being castrated tore in his life. I have read my words back a couple of times since I wrote it, and it makes me cry. The book is brilliant, you should buy it.

Nine years in, then, have I had enough? No.

Sunday, June 21, 2020

For Against Method

So I've just given up on Paul Feyerabend's Against Method. It's by no means the first book I've stopped reading; only recently I skipped big chunks of Accelerate by Nicole Forsgren, Jez Humble, and Gene Kim, shortly before that I dumped Douglas Rushkoff's Team Human, and further back Women, Fire, and Dangerous Things by George Lakoff moved off my bedside table and back onto the book shelves. 

Why? Well sometimes I feel like I know the content already, sometimes I feel like I don't or won't have a context in which the information is useful, sometimes I'm just not feeling it for whatever reason. So I stop, or skip, or cherry-pick, or all three. 

What was the problem with Feyerabend's book? It was too dense, referring to too much that I don't know without looking it up, and I have too much on my mind to be able to concentrate at the depth needed to consume it. Also, Feyerabend helpfully provided a summary of his argument, broken down by chapters, right at the start. It's essentially a self-authored crib sheet that gives me the high-level insight that the friend who recommended the book thought would interest me. Maybe its time will come again later. For now, it's sitting next to Plato and a Platypus Walk Into a Bar (which I did finish) waiting.

Boiled down, the insight that I took from the book is that:
  • scientific theories always have holes
  • ... and we generally understand this
  • ... and historically use theories pragmatically (e.g. classical and quantum mechanics)
  • but we do risk becoming wedded to a theory
  • ... and accept its weaknesses as just the way things are
  • ... to the extent that we stop questioning and work only within its constraints
  • we can break out of this potential stranglehold
  • ... by actively looking for alternatives
  • ... and not just scientific alternatives based on observation, but metaphysical, hypothetical, and even self-contradictory ones
Counterinduction  is what Feyerabend calls this process of deliberately challenging by comparison. Crucially, counterinduction does not require the challenger to have the same explanatory power as the theory being challenged: if it can fill some gap in some way, it's worthy of consideration. 

If the challenger is radically different from the challenged, such as when religious doctrine is set against scientific method, so much the better for jolting those people running on theoretical rails off them. For Feyerabend, significant scientific advances tend to come through these left-field challenges rather than incremental refinement of a dominant ideology.

I was unfamiliar with the term counterinduction, and am still unfamiliar with the bulk of the philosophy and the details of the scientists and theory that Feyerabend calls on to back up his claims, but I find the idea appealing. My intuition has long been that I want to pick and choose methods that I think could be helpful in any given situation. 

I am suspicious of people who want to dichotomise the world into the right answer and everything else. For me, a pluralistic perspective will provide more potential avenues to explore and doesn't rule out the "right answer" either; it's always an available option. 

This is one of the reasons that I was so enamoured with Koen's Definition of the Engineering Method recently: the heuristic mindset is an open mindset, implicitly challenging available techniques against potential alternatives and being alert to the prospect of failure in any given application and context.

So, yes, I gave up on Against Method yet still got something new from it to slot into my model of how I work and how I want to work. I count that as a win rather than a loss for both me and Paul Feyerabend.
Image: Amazon

Thursday, June 18, 2020

Practice Makes Perfect

Drew Pontikis spoke about Leading Communities of Practice at Cambridge Agile Exchange tonight. Some might think it controversial that a CoP should have a leader, but it's one of the four things that Drew thinks are required, along with regularity, a purpose, and the investment of (at least) time.

Crucially, though, note that it's leadership rather than management, that Drew wants for a CoP. His own experience is that trying too hard to force things, direct behaviour, or control the structure or content of the group will lead to failure. He told the story of a team where management made CoP attendance mandatory and how awful it was to be interacting with a group who largely would rather be somewhere else.

Leadership in a CoP should keep the group grounded during the early stages when motivation is high and motivated during the inevitable downturn in enthusiasm that will come. The leader is a peer with knowledge, respect, and integrity, and also with contacts and the ability to act as a catalyst to enable others.

A leader should also be ready to step back from heavy involvement in a CoP when it's clear that they are no longer needed, and shouldn't be discouraged when the group inevitably winds down or changes direction. Ideally that will be because whatever need caused it to start has been met.