Hiccupps

In What We Know We Don't Know , Hillel Wayne crisply summarises a handful of research findings about software development, describes how the research is carried out and reviewed and how he explores it, and contrasts those evidence-based results with the pronouncements of charismatic thought leaders. He also notes how and why this kind of research is hard in the software world. I won't pull much from the talk because I want to encourage you to watch it. Go on, it's reasonably short, it's comprehensible for me at 1.25x, and you can skip the section on Domain-Driven Design (the talk was at DDD Europe) if that's not your bag. Let me just give the same example that he opens with: research shows that most code reviews focus more on the first file presented to reviewers rather than the most important file in the eye of the developer. What we should learn: flag the starting and other critical files to receive more productive reviews. You never even thought about that possi...

  One of my chores takes me outside early in the morning and, if I time it right, I get to hear a charming chorus of birdsong from the trees in the gardens down our road, a relaxing layered soundscape of tuneful calls, chatter, and chirrupping. Interestingly, although I can tell from the number and variety of trills that there must be a large number of birds around, they are tricky to spot. I have found that by staring loosely at something, such as the silhouette of a tree's crown against the slowly brightening sky, I see more birds out of the corner of my eye than if I scan to look for them. The reason seems to be that my peripheral vision picks up movement against the wider background that direct inspection can miss. An optometrist I am not, but I do find myself staring at data a great deal, seeking relationships, patterns, or gaps. I idly wondered whether, if I filled my visual field with data, I might be able to exploit my peripheral vision in that quest. I have a wide monito...

  Last month I started thinking about slicing , my instinctive approach to looking for perspectives on a problem, an opinion, an observation, or anything else. This time around I've got an example to talk about. On Fridays I ensemble with a group of medical quality engineers and medical knowledge engineers. We learn from each other, about testing and about the domain. On and off recently we've looked at a project of theirs which aims to understand better what work they do, how they do it, and why it's that way, and then write it up for internal and external consumption. In one early session, with a wider group in the company, there was an extremely open and exciting conversation about what should be covered in this effort.  It was the kind of discussion that greenfield projects often have, before scope is nailed down, where the world seems ripe with possibility, no difficulties have been identified, and there is no talk of who will taking respons...

  I demoed some of my testing to Maaret Pyhäjärvi the other day. She asked a bunch of incisive questions, offered some helpful ideas on framing my results, and then suggested that I should consider speaking and writing about how I partition the testing space when I work.  She followed up on Mastodon :  ... I learn so much hearing how [James] dissects the exact necessary information and combines it with an approach that provides better results than what is usual for testing. Which, naturally, made me blush, but also reminded me that others have observed similar things. Back in 2016, during my presentation at MEWT #5 , Iain McCowatt tweeted this : [James] wields distinctions like a surgeon wields a scalpel. So now I am wondering about how I can begin to externalise more of this aspect of my approach to testing. I don't have the answer yet. Today's task is to collect some data on what I've said in the past. I'm lis...

  I was in the shed earlier on, spraying the footrest I've made for my daughter a metallic purple.  As I ghosted the can back and forth, a waft of paint drifted onto the workbench adding another colourful contribution to the happenstance Pollock that's been building up since I fished the boards out of a skip in the centre of town *cough* years ago. That's not to mention the various scratches, cuts, dents, dinks, and drill holes that pepper the surface. Yes, I thought to myself, this bench has a real patina . On seeing my bench, a fellow maker and fixer would recognise it. The layers and shapes are archeological evidence of the variety of activities at different times, with different materials, operated on by different tools.  Which got me thinking: how and where am I building up patina in my work in testing? And how would anyone ever see it? And would they be able to appreciate it if they did? Edit: I followed up with a few ideas in A Testing Patina

In recent times I've spoken and written about how much fun and how productive it's been to build random walkers to help me to test services I've been working on: Walking the Talk webinar , A Model Student , and Navigate, Survey, and Explore . The walkers are clients which use dice rolls to make decisions as they navigate paths through a service, asserting things about the state as they go. Traditional unit tests tend to be extremely specific. They imagine the system in a particular state with a hard-coded input and an expected output.  In my walkers, the assertions are relatively generic, for example: a service response payload conforms to a schema the value of a field representing progress will not decrease across a series of interactions some values in service responses must be in a particular relationship to others, or to the input. I doubt this is a technically-correct use of the term but I've been thinking of th...

I am totally loving Oddly Influenced, Brian Marick's new podcast. The latest episoide covers ways in which schools of thought and practice can inhibit the cross-fertilisation of ideas.  It includes a case study in experimental physics from Peter Galison's book, Image and Logic , where two different approaches to the same particle analysis problem seem to run on separate, parallel tracks: In the 'head to world' tradition, you use your head to carefully construct situations that allow the world to express its subtle truths ... In the 'world to head' tradition, you make yourself ever more sensitive to the world’s self-expressed truths ... The first of these wants to theorise and then craft an experiment using statistics while the latter wants to gather data and try to understand it visually. Marick is pessimistic about the scope for crossover in this kind of situation: How do you bridge traditions that differ on aesthetics, on different standards of what counts as...

  In episode three of Oddly Influenced, jUnit and What Makes a Successful Tool , Brian Marick is speculating about the relative lack of adoption of test-driven development compared to the tooling that supported it.  After putting forward a particular theory he wonders whether it's "true" and says: That ... gives me an opportunity to state a theme that’s been in my work for decades. I read books ... about how people do their work. I’m not so concerned if the theories are true as if they are suggestive – that is, do they give me ideas about how software people should do our work. [The] next task is trying out whether those ideas have good results, in our work. Because everyone’s theory about people is somewhere between fully wrong and fully right, and is always incomplete. I like this perspective a lot. It puts me in mind of Paul Feyerabend's Against Method, a book I failed to finish because it was so dense and widely-read, bu...

A month or two ago, after seeing how I was taking notes and sharing information, a colleague pointed me at Tiego Forte's blog on Building a Second Brain : [BASB is] a methodology for saving and systematically reminding us of the ideas, inspirations, insights, and connections we’ve gained through our experience. It expands our memory and our intellect... That definitely sounded like my kind of thing so I ordered the upcoming book, waited for it to arrive, and then read it in a couple of sittings. Very crudely, I'd summarise it something like this: notes are atomic items, each one a single idea, and are not just textual notes should capture what your gut tells you could be valuable notes should capture what you think you need right now notes should preserve important context for restarting work notes on a topic are bundled in a folder for a Project, Area, or Resource and moved into Archive when they're done. ( PARA ) ...

Last week I hosted a webinar, Steel Yourselves , at the Association for Software Testing .  I'd been thinking about the format for a while and waiting for an opportunity to try it out with speakers that I knew would be able to do it justice. There's an interesting debating tactic in which the idea is ...  ... to help one's opponent to construct the strongest form of their argument. This may involve removing flawed assumptions that could be easily refuted, for example, so that one produces the best argument for the 'core' of one's opponent's position. I thought it would be interesting to have speakers take a testing concept they are opposed to and construct a case for it, present it, defend it during a Q&A, then talk about what they learned and how they felt. I thought it would be challenging for the speaker — a chance to find something out about their views — and inspiring for the audience — a chance to see someone p...

  If only we knew ... When our testing is essentially the same eyes looking through the same lens at the same thing in the same context in same way over and again then we are limiting the extent to which we can learn about our product. If we aspire to know more, then we should use different eyes and different lenses and look in different ways at different aspects of the system under test in different scenarios. Easy to say, of course. But how to do ? Engage with what's happening elsewhere in our teams, in other teams, in our companies, in our domains, in the tech our companies use, in the testing community and industry, and in other areas relevant to our contexts. If only we new ...  We will find people, tooling, approaches, or knowledge that could help us do a better job. We will make opportunities to try those new things and invest in the ones that look promising.  We will find that what we knew wasn't as much as we thought. Image: https://flic.kr/p/2mz3krH

And this is how it goes. One thing to another ... At the weekend I  was listening to Gene Kim's Idealcast interviews with Dr. Nicole Forsgren and Jez Humble . Jez Humble was talking about the importance of rapid feedback and referenced a  Brett Victor  talk that had stuck with him over the years: ... he built this little JavaScript game and he was changing parameters and the game was changing in real time. And his whole comment is like, if you can change something and see the change as you are changing it, it's incredibly powerful.   And so I looked it up in the show notes and watched it. Wow ... Inventing in Principle shows examples of experimental tooling for creative activities, particularly those that include a temporal dimension. The essential idea is to reduce the friction of having to maintain a model outside of the tool. In the image at the top, the left side is a traditional IDE and the right side is a dynamic visualisation of the function being develo...

Much of  Definition of The Engineering Method  by Billy Vaughn Koen chimes with what I've come to believe about testing.  In part, I think, this is because thinkers who have influenced my thinking were themselves influenced by Koen's thoughts. In part, also, it's because some of my self-learned experience of testing is prefigured in the article. In part, finally, it's because I'm reading it through biased lenses, wanting to find positive analogy to something I care about.  I recognise that this last one is dangerous. As Richard Feynman said in Surely You're Joking, Mr. Feynman!: "I could find a way of making up an analog with any subject ... I don’t consider such analogs meaningful.”  This is a short series of posts which will take an aspect of Definition of The Engineering Method that I found interesting and explore why, taking care not to over-analogise. Getting Myself Koen Sota so Good Meta is Better The Tester as Engineer? --00-- So...

In knowledge work it's usual for ideas to need to be shared. In software development a common direction of travel for concepts is from one set of people with a vision for a feature to another set of people who will bring it to life. Sounds simple, right? But it isn't simple and many ways exist to try to address that lack of simplicity, including thick wodges of functional specification and layers of sign-offs through to terse sticky notes, a chat, and iteration. Whatever the approach there's one constant: a commitment of time from the idea holders. An anti-pattern common in our industry is that these individuals often also have emerging and urgent tasks which trump the merely important ones ... such as the sharing of ideas. Despite our best intentions we've seen it at our place too, and it impacts the quality of the thing that we can build in the time available to us. Naturally, this kind of problem can be tackled at all sorts of levels. At one level, I was in...

Behind the hyperbole of repeatedly saying that we don't know what we're doing in my talk at CEWT #7 a couple of weeks ago, my core message was that, for me: In order to be a great tester you have to embrace that not knowing. You have to be able to work within uncertainty, without being confident that anything will stand still, taking into account that your lack of knowledge of something might be the key issue. The project without uncertainty never existed and will never exist. You won't be sure you ran the experiment you thought you did, or generated the data you were hoping to, or analysed the results without imposing your own biases. Stakeholders often won't know what they really want, developers often won't be sure they understood what the stakeholders said they wanted, you often won't be sure that the developer implemented the thing they intended to. The company might appear confident that the bets it is laying this quarter will pay off, b...

When I asked Twitter this: Anyone know of a course on risk analysis that could work as in-house training for software testers?   It's a topic we're interested in but the stuff I'm turning up is more to do with corporate analysis, register building , mitigation at the business level #testing #risk Paul Hankin  suggested a book, Superforecasting, The Art of Science and Prediction , by Philip Tetlock and Dan Gardner. As the title suggests the book is about prognosis rather than peril, but the needs of prediction and risk analysis overlap in interesting ways: understanding possible outcomes, identifying factors that contribute to those outcomes, and weighting the factors and their interactions. Tetlock and Gardner study forecasting and forecasters. They use a metric called the Brier score to assess the accuracy of forecasts, and, over time, and with repeated forecasts, it becomes clear that some people tend to make better forecasts than others. The Brier score...

We've all been there: frustrated by a request from a stakeholder for what we take to be significant new work without regard for the scale of it, the time it would take, or the current backlog. Recently, a colleague in that situation and ready to scream "NOOOOO!!!!" asked for my advice. What I said boiled down to this: Step back and think of at least three ways that the request could be interpreted. Sketch rough ideas for how you could do each of them, at what cost, with what compromises.  Share them with your stakeholder to clarify their desires and help them to guide the next steps. This is essentially Jerry Weinberg's  rule of three  and orange juice test  so I claim no great novelty here. What I do claim is that I feel a lot better when I follow these steps than when I instinctively reject some request based on poor assumptions and no conversation, landing myself in a needlessly defensive position. P.S. just to make this harder, don't forg...

How about if it wasn't Venn diagram, but instead it was a  When diagram? Would we know when things were going to get done if we had one of them? I'm sure I'm not the first person to make that phonetic connection, but perhaps I'm the first who has  sufficiently little shame to mention it publicly on Twitter : If I asked you to produce a "When diagram" for the project you're running, what would you think I was after? What kind of picture/chart/diagram would you draw for me? A handful of kind folk responded, each with something different: @joelmonte : When, in the project perspective, sounds like a milestones diagram.  It can be interconnected, and showing the dependencies of events...  But this is only my sunday-morning-wild-guess.  Do we also have "Why", "How" and most importantly "Who is to blame" diagrams? @always_fearful : Gant chart @hairyhatfield : A venn diagram with sets 'sooner'  'l...