I Can Manage

For work reasons, I've recently become interested in resources for those new to line management. I put out an appeal for suggestions on Twitter and Managing The Unmanageable was recommended by Thomas Ponnet, with a little cautious reservation:

@qahiccupps Hope you enjoy it. I don't agree with everything but that comes with the job description. Not all translates for my context.

— Thomas Ponnet (@ThomasPonnet) August 15, 2016

This quote from the book's preface sets up the authors' intent nicely:

There is no methodology for the newly anointed development manager charged with managing, leading, guiding, and reviewing the performance of a team of programmers — often, the team he was on just days before. There are no off-the-shelf approaches. Unlike project managers, who devote hours and hours of study toward certifcation in their chosen career path, development managers often win their management roles primarily from having been stellar coders while displaying a modicum of people skills.

The book is long - over-long for my taste - and, rather than try to rehash the whole thing, I'll take the liberty of making an exceedingly crude precis:

• people are all different
• ... but there are broad classes of characteristics that it can useful to acknowledge and look for
• people are motivated by a relatively small set of important things
• .. and, after a certain level is reached, salary is not usually the most important thing
• hiring well is crucial, and can be extremely difficult
• ... and a manager should be thinking about it even when they are not actively hiring
• to do well, a manager  needs to be organised
• ... even more organised than you probably think
• to command respect from a team, a manager should be able to demonstrate relevant skills
• ... and need to know when is a good time to do that and when to step back
• to enjoy the support of a team, a manager needs to show empathy and give protection
• ... and that sometimes means letting them fail; but shouldn't mean setting them up to fail
• to function well within a company a manager needs to establish relationships and communicate well
• ... in all directions: down, up, and across, and in different media
• a good manager will reflect on their own actions
• ... and look to improve themselves
• the source of a team culture is the manager
• ... and, once established, it requires nurturing

Perhaps these things seem self-evident. Perhaps some of them are self-evident. Broadly speaking I think I'd agree with them, based on my own experience. And, in my own experience I find that I learned many of them only incrementally and some of them the hard way.

Which is where a book like this can help - it's a brain dump of wisdom from the two authors mostly, but also from a bunch of others who offer nugget-sized bites of experience such as

Managers must manage - Andy Grove

with associated commentary:

I’ve used Andy Grove’s phrase innumerable times to coach my managers and directors of programming teams. When confronted with a problem, they can’t just "raise a red flag." I’m always available when needed, but good software managers find ways to solve problems without my involvement or executive management direction.

And here's handful of others that chimed with me:

Don’t let the day-to-day eat you up - David Dibble 

David made this statement to make the point to his management team that managers have "real" work to do; that the seemingly urgent—e-mail, meetings, the routine—could easily fill a day. Only by being intentional about how we use our days can managers overcome letting that happen 

If you’re a people manager, your people are far more important than anything else you're working on - Tim Swihart 

Tim notes, "If a team member drops by at an awkward time and wants to chat, set aside what you’re doing and pay attention. They may be building up the courage to tell you something big. I’ve noticed this to be especially true when the sudden chatter isn’t somebody who normally drops by for idle conversation." 

Managers who use one-on-one meetings consistently fnd them one of the most effective and productive uses of their management time - Johanna Rothman and Esther Derby 

The statement is a match for our own experience.

We have two ears and one mouth. Use them in this ratio - Kimberly Wiefling 

While I love theory and can happily spend time in talking shops, dissecting semantics and splitting hairs, as my recent MEWT experience showed ...

@qahiccupps wields distinctions like a surgeon wields a scalpel #mewt

— Iain McCowatt (@imccowatt) April 9, 2016

... I also recognise the value of activity to explore, inform, test, and back up the theory. I like to think of myself, still, as a practitioner, and Managing the Unmanageable is a book written by practitioners and grounded in their practice, with examples drawn liberally from it.

It's unlikely, as Thomas Ponnet suggested, and I'd agree, to fit exactly with everything that you're doing right now with the team you have in the place you're working - especially as some of it is very specific to managing software developers. Parts of it will probably jar too. For instance, I found the suggested  approach to levels of seniority too simplistic.

But what it can do is give you another perspective, or inspiration, or perhaps fire warning shots across your bow from some position not too dissimilar to yours, and rooted in the real world of managing people in technical disciplines.
Image: http://www.managingtheunmanageable.net/

It's Complicated

In a recent episode of Rationally Speaking, Samuel Arbesman, a complexity scientist, talks about complexity in technology. Here's a few quotes I particularly enjoyed.

On levels of understanding of systems:

Technology very broadly is becoming more and more complicated ... actually so complex that no one, whether you're an expert or otherwise, fully understands these things ... They have enormous number of parts that are all interacting in highly nonlinear ways that are subject to emerging phenomena.

We're going to have bugs and glitches and failures. And if we think we understand these things well and we don’t, there's going to be tons of gap between how we think we understand the system and how it actually does behave.

On modelling reality with a system and then creating a model of that system:

... the world is messy and complex. Therefore, often, in order to capture all that messiness and complexity, you need a system that effectively is often of equal level of messiness and complexity ... whether or not it's explicitly including all the rules and exceptions and kind of the edge cases, or a system that learns these kinds of things in some sort of probabilistic, counterintuitive manner.

It might be hard to understand all the logic in [a] machine learning system, but it still captures a lot of that messiness. I think you can see the situation where in machine learning, the learning algorithm might be fairly understandable. But then the end result ... You might be able to say, theoretically, I can step through the mathematical logic in each individual piece of the resulting system, but effectively there's no way to really understand what's going on.

On "physics" and "biological" thinking:

[Physics:] A simple set of equations explains a whole host of phenomena. So you write some equations to explain gravity, and it can explain everything from the orbits, the planets, the nature of the tides ...  It has this incredibly explanatory power. It might not explain every detail, but it maybe it could explain the vast majority of what's going on within a system. That's the physics. The physics thinking approach, abstracting away details, deals with some very powerful insights.

[Biology:] Which is the recognition that oftentimes ... the details not only are fun and enjoyable to focus on, but they're also extremely important. They might even actually make up the majority of the kinds of behavior that the system can exhibit. Therefore, if you sweep away the details and you try to create this abstracted notion of the system, you're actually missing the majority of what is going on.

Oftentimes I think people in their haste to understand technology ... because technologies are engineered things ... think of them as perhaps being more the physics thinking side of the spectrum.

On robustness:

There's this idea within complexity science ... "robust yet fragile," and the idea behind this is that a lot of these very complex systems are highly robust. They've been tested thoroughly. They had a lot of edge cases and exceptions built in and baked into the system. They're robust to an enormously large set of things but oftentimes, they're only the set of things that have been anticipated by the engineers. However, they're actually quite fragile to the unanticipated situations.

Side note: I don't think there's an attempt in this discussion to draw a distinction between complex and complicated, which some do.
Image: https://flic.kr/p/Q2tMz

Giving 'Back

The Test team book club at Linguamatics is currently reading What Did You Say? The Art of Giving and Receiving Feedback. Here's a couple of quotes that I picked out for our last session:

• If you’re really interested in helping people, you’ll do well to start your feedback by opening your own motives to inspection.
• Even when it’s given at the receiver’s request, feedback describes the giver more than the receiver.
• When the data and their model don’t match, most people discard the data.

I recall an instance when, engaged in discussion with a colleague I'll call Russell, about the data analysis he was presenting, I spotted an opportunity to offer feedback. It was about something that I knew Russell wanted to change. It was about something that I knew was open to me to give feedback on, because we had talked about it. It was about something that I thought would be beneficial for Russell in multiple ways and, I hoped, would provide some insight into a particular behaviour pattern that he had.

However, it was also the first time that I had seen this particular thing. A data set of size one. I had no evidence, yet, that it would lead to the end point that Russell desired to alter. A data set of size zero.

Against this: my instinct, my gut, and my experience. And a sense of goodwill, built up over time, over repeated interactions, over sometimes difficult sessions where I had tried to demonstrate that I do care to assist and support and advise because I want to help Russell to be the best he can be, in the respects that matter to him and for his work.

But I was still cautious. I have unwittingly burned and been burned enough times over the years to know that each of these conversations carries with it risks. Risks of misreading the context, risks of misreading the agreements, risks of misreading the mood, risks, risks, risks, ...

But I went ahead anyway. The potential benefit and the goodwill in the bank outweighed the risks, I calculated, on this occasion. And I gave my feedback. And Russell agreed with me. And I breathed a deep internal sigh of relief.

Comparing this anecdote to the quotes I pulled from the book:

• My motives, I think, were good: I wanted to help Russell achieve a personal goal.
• But the feedback does reflect something about me: an interest in reducing unnecessary complexity, an interest in making presentation clear, the ego that is required to believe that my colleagues will want to listen to any advice from me, ...
• In this case, it turned out my suggestion didn't contradict Russell's model but exposed it, and in any case I had little concrete data to present.

I use this episode as an example not because it ended well, particularly, but because it's an illustration for me of how much I have been influenced by What Did You Say? in the couple of years since I first read it. I consciously I go about my day-to-day business, doing my best to be careful about when I choose to offer feedback, about when I deliberately choose not to, and about picking up and picking up on any feedback that's coming my way in return.

I try to treat this as a testing task where I can, in the sense that I try hard to observe my own actions and the responses they generate, and I think about ways in which they might be related and how I might approach things differently in the next exchange, or at another time, with this person, or someone else.

Easier said than done, of course, so I'll finish with another quote from the book, another quote that I've taken to heart and act on, that regularly helps guide me with pretty much everything that I've said above:

Don’t concentrate on giving feedback; concentrate on being congruent–responding to the other person, to yourself, and to the here-and-now situation. Don’t go around hunting for opportunities to give feedback, because feedback is effective only when the need arises naturally out of congruent interactions.

Some details have been changed.
Image: Leanpub

Reporting, a Novel Approach

 

There's a girl in the park playing with an enormous bunch of balloons. She's running around, clearly very happy to have such a pretty and fun toy. She seems entranced by the way the balloons have life of their own: they hold themselves up, needing no support from her, and animatedly jostle one another as she moves. Her grip on the strings, twined together in her fist, is quite loose, and she's in danger of losing them if she's not careful. And, of course, she isn't and she does. 

The balloons float up and up and up from her released grasp, past a tall tree in which two nude men are arm wrestling. On their wrists each sports a watch showing ten minutes past ten, despite the time being 12:57. With their free arms they reach out and catch the balloons as they bobble by, bursting every last one of them, and smiling.

In the second exercise of Mira Nair's Storytelling workshop, which ran at last night's Cambridge Tester Meetup, we were asked to write a story that included three items chosen at random from a selection. I had balloons, entwined arms with hands clasped, and a clock showing 10:10. Prior to that we'd been provided with a story structure and asked to fill it with content. Later, to take existing stories and compress them to tweet length.

The workshop's practical aspects focused mostly on structure, including on the STAR mnemonic which is intended to help interviewees give good answers to behaviour questions such as "Give me an example of when you ...". The letters stand for Situation, Task, Action, and Result and a story according to them should run like this:

• Situation: define the background
• Task: explain the mission
• Action: describe the work done
• Result: enumerate the outcomes 

The first exercise in the workshop gave us that skeleton and asked us to fit a recent episode to it. Some found it liberating ("The structure draws the story out of whatever I put in") while others struggled ("I couldn't find something that I thought would make a good story"). At work, Mira suggests, we'll more often have the problem of something to present and needing a structure to bring the best out of it than the reverse.

This was reinforced by the second exercise which provided content but, interestingly, didn't require any structure of us. More people found this more straightforward, the content anchoring everything else.

In the story I gave at the beginning I experimented with another narrative device Mira talked about, the False Start, in which an apparently predictable beginning leads to an unexpected ending and can result (with judicious use) in increased audience engagement.

The final exercise cut the content problem a different way: take an existing description (a couple of software bugs were provided) and summarise it in 140 characters or less. As testers, we author reports of potential issues regularly, and part of the skill of transmitting those reports is finding a way to quickly engage our audience, which will often mean extracting the essentials and conveying them efficiently.

Different approaches were taken here: I boiled the descriptions down as I might at work; others took the Twitter aspects and used hashtags as shorthand, effectively importing context cheaply, and others used humour to convey a sense of violated expectation.

One of the things I look for in these kinds of events is the questions they generate, the trains of thought I can follow at my leisure, the connections I can make, ... Here's a few:

• what really distinguishes a story from any other prose, if that's possible? 
• is the "storyness" in the eye of the author or the audience?
• what techniques are there for picking out the relevant content for a story?
• what aspects of storytelling are important beyond structure?
• even with strong structure, stories can be poor, boring, uninformative.
• readers assume much when reading a story, filling in missing details, assuming causation, intent and so on.
• what about the inadvertent stories we tell all the time; that bored expression, that casual gesture, that throwaway remark?
• stories don't need to be true, but my reports as a tester generally need to be true (to what I understand the situation I'm describing to be).
• stories can be input to and output from testing.
• don't forget the relative rule: the audience and the time are important to the effect a story will have.
• there was discussion about tailoring stories for an audience ("stories should not be the same each time you tell them") but once written down, a story is static. 
• I find that writing helps me to generate and understand the content. I'll often start writing before I know the story myself.
• finding a perspective can help to make the story compelling, and that perspective can be the author's, the readers', or that of a third party.
• I like the testing story heuristic I took from RST: status, how you tested, value/risks. But this is a content heuristic more than a delivery heuristic, although I find that order to generally be useful.

I have written before about the C's I look for in communication: conciseness, completeness, correctness, clarity, context. I realised in this workshop that I can add another: compelling.
Image: https://flic.kr/p/97ba7K

Tools: Take Your Pick Part 4

Back in Part 1 I started this series of posts one Sunday morning with a very small thought on tooling. (Thinking is a tool.) I let my mind wander over the topic and found that I had opinions, knowledge, ideas, and connections that I hadn't made explicit before, or perhaps only in part. (Mind-wandering is a tool.)

I wrote the stuff that I was thinking down. (Writing is a tool.) Actually, I typed the stuff that I was thinking up.¹ I have recently been teaching myself to touch type in a more traditional style to (a) stop the twinges I was feeling in my right hand from over-extension for some key combinations and (b) become a faster, more consistent, typist so that my thoughts are less mediated in their transmission to a file. (Typing is a tool.)

I reviewed and organised my thoughts and notes. With each review, each categorisation, each classification, each arrangement, each period of reflection away from the piece of writing, I found more thoughts. (Reviewing and rationalisation and reflection are tools.) I challenged myself to explore ideas further, to tease out my intuitions, to try to understand my motivations, to dig deeper into whatever point it was I thought I was making. (Exploration is a tool.)

The boundaries between some of these tools are not clear some of the time. And that doesn't matter, some of the time. For me, in this work, it doesn't matter at all. My goal is to get my thoughts in order and hopefully learn something about myself, for me, and perhaps also something more general that I can share with my team, my colleagues, the wider readership of this blog.

That the boundaries are not clear is an interesting observation, I find, and it came about only because I was trying to list a set of tools used somewhat implicitly here. (Lists are tools.) Not knowing which tool we are using suggests that we can use tools without needing to know that we are using them at all. Part 2 started off with this:

   When all you have is a hammer, everything looks like a nail.

And I discussed how this does not necessarily mean that every use of the hammer is mindless. But I now also wonder whether it's possible to proceed without even realising that you have a hammer. With non-physical tools - such as those I've listed above - this seems to me a distinct risk. Side-effects of it might include that you don't know how you arrived at solutions and so can't easily generalise them, you don't realise that you are missing out large parts of the search space and so can't consider it; you don't provide yourself with the opportunity to look for improvement, ...

I think that reflection and introspection help me to mitigate those risks, to some extent. Although, of course, some of the risks themselves will be unknown unknowns and so less amenable to mitigation without additional effort being made to make them known. (Another problem to be solved. But which tool to use?)

The more I practise that kind of reflection the more practised I become at recognising simultaneously the problem and meta problems around it, or the problem space and the context of which is it a part, or the specific problem instance and the general problem class. I have had my fingers burned trying to verbalise these things to others, and I probably now over-compensate through clunky conversational tactics to try to make clear that I'm shifting modes of thinking.

Another thought on sub-conscious problem-solving approaches: perhaps the recognition (correct or not) of an instance of a nail-like problem triggers a particular hammer (tacit or explicit):

When it looks enough like a nail, I hit it.

But every decision to use a tool is also an opportunity to make a different decision. Deciding to think about how and why a decision was made gives insight that can be useful next time a decision is made, including the realisation that a decision is being made.

Part 1 of this essay was in the domain of cleaning,  Part 2 more general and theoretical, and Part 3 focused back on work. They are grouped that way and written in that order because I found it helpful and convenient, but the way the thoughts came to me was messier, non-linear, fragmentary. I used tools to note down the thoughts: my phone, scraps of paper, my notebook, text files on the computer, ... Tools to preserve the output of tools to provide input to tools that will themselves generate output on the topic of tools.

I find myself chuckling to and at myself as I write this last part of the summary. While attempting to pull together the threads (attempting to pull together threads is a tool) I realise that in this piece which is ostensibly about hammers and nails - and having observed that perhaps we sometimes don't recognise the hammer - there may actually be no nail.

When I started out I had no particular problem to solve - beyond my own interest in exploring a thought - and so no particular need for a tool, although I have deployed several, deliberately and not. But. ironies aside, that's fine with me: quite apart from any benefits that may accrue (and there may be none, to me or you, y'know) the process itself is enjoyable, intellectually challenging, and satisfying for me. And exercising with my tools keeps me and them in some kind of working order too.
Image: https://flic.kr/p/qjYUm

Footnote

1.  Writing down but typing up? There's a thought for another day.

Tools: Take Your Pick Part 3

In Part 1 of this series I observed my behaviour in identifying problems, choosing tools, and finding profitable ways to use them when cleaning my bathroom at home. The introspection broadened out in Part 2 to consider tool selection more generally. I speculated that, although we may see someone apparently thoughtlessly viewing every problem as a nail and hence treatable with the same hammer, that simple action can hide deeper conscious and unconscious thought processes. In Part 3 I find myself with these things in mind, reflecting on the tools I use in my day-to-day work.

One class of problems that I apply tools to involves a route to the solution being understood and a desire to get there quickly. I think of these as essentially productivity or efficiency problems and one of the tools I deploy to resolve them is a programming or scripting language.

Programming languages are are tools, for sure, but they are also tool factories. When I have some kind of task which is repetitive or tiresome, or which is substantially the same in a bunch of different cases, I'll look for an opportunity to write a script - or fabricate a tool - which does those things for me. For instance, I frequently clone repositories from different branches of our source code using Mercurial. I could type this every time:

$ hg clone -r branch_that_I_want https://our.localrepo.com/repo_that_I_want

... and swear a lot when I forget that this is secure HTTP or mistype localrepo again. Or I could write a simple bash script, like this one, and call it hgclone:

#!/bin/bash

hg clone -r $1 https://our.localrepo.com/$2

and then call it like this whenever I need to clone:

$ hgclone branch_that_I_want repo_that_I_want

Now I'm left dealing with the logic of my need but not the implementation details. This keeps me in flow (if you're a believer in that kind of thing) or just makes me less likely to make a mistake (you're certainly a believer in mistakes, right?) and, in the aggregrate, saves me significant time, effort and pain.

Your infrastructure will often provide hooks for what I sometimes think of as micro tools too. An example of this might be aliases and environment variables. In Linux, because that's what I use most often, I have set things up so that:

• commands I like to run a particular way are aliased to always run that way.
• some commands I run a lot are aliased to single characters.
• some directory paths that I need to use frequently are stored as environment variables.
• I can search forwards and backwards in my bash history to reuse commands easily.

One of the reasons that I find writing (and blogging, although I don't blog anything like as much as I write) such a productive activity is that the act of doing it - for me - provokes further thoughts and connections and questions. In this case, writing about micro tools I realise that I have another kind of helper, one that I could call a skeleton tool.

Those scripts that you return to again and again as starting points for some other piece of work, they're probably useful because of some specific piece of functionality within them. You hack out the rest and replace them in each usage, but keep that generally useful bit. That bit is the skeleton. I have one in particular that is so useful I've made a copy of it with only the bits that I was reusing to make it easier to hack.

Another class of problem I bump into is more open-ended. Often I'll have some idea of the kind of thing I'd like to be able to do because I'm chasing an issue. I may already have a tool but its shortcomings, or my shortcomings as a user, are getting in the way. I proceed here in a variety of ways, including:

• analogy: sometimes I can think of a domain where I know of an answer, as I did with folders in Thunderbird.
• background knowledge: I keep myself open for tool ideas even when I don't need tools for a task. 
• asking colleagues: because often someone has been there before me.
• research: that frustrated lament "if only I could ..." is a great starting point for a search. Choosing sufficient context to make the search useful is a skill. 
• reading the manual: I know, old-fashioned, but still sometimes pays off.

On one project, getting the data I needed was possible but frustratingly tiresome. I  had tried to research solutions myself, had failed to get anything I was happy with, and so asked for help:

#Testers: what tools for monitoring raw HTTP? I'm using tcpdump/Wireshark and Fiddler. I got networks of servers, including proxies #testing

— James Thomas (@qahiccupps) March 26, 2016

This lead to a couple of useful, practical findings: that Fiddler will read pcap files, and that chaosreader can provide raw HTTP in a form that can be grepped. I logged these findings in another tool - our company wiki - categorised so that others stand a chance of finding them later.

Re-reading this now, I notice that in that Twitter thread I am casting the problem in terms of the solution that I am pursuing:

I would like a way to dump all HTTP out of .pcap. Wireshark cuts it up into TCP streams. 

Later, I recast the problem (for myself) in a different way:

I would like something like tcpdump for HTTP.

The former presupposes that I have used tcpdump to capture raw comms and now want to inspect the HTTP contained within it, because that was the kind of solution I was already using. The latter is agnostic about the method, but uses analogy to describe the shape of the solution I'm looking for. More recently still, I have refined this further:

I would like to be able to inspect raw HTTP in real time, and simultaneously dump it to a file, and possibly modify it on the fly, and not have to configure my application to use an external proxy (because that can change its behaviour).

Having this need in mind means that when I happen across a tool like mitmproxy (as I did recently) I can associate it with the background problem I have. Looking into mitmproxy, I bumped into HTTPolice, which can be deployed alongside it and used to lint my product's HTTP.  Without the background thinking I might not have picked up on mitmproxy when it floated past me; without picking up on mitmproxy I would not have found HTTPolice or, at least, not found it so interesting at that time.

Changing to a new tool can give you possibilities that you didn't know were there before. Or expose a part of the space of possible solutions that you hadn't considered, or change your perspective so that you see the problem differently and a different class of solutions becomes available.

Sometimes the problem is that you know of multiple tools that you could start a task in, but you're unsure of the extent of the task, or the time that you'll need to spend on it, whether you'll need to work and rework or this is a one-shot effort and other meta problems of the problem itself. I wondered about this a while ago on Twitter:

With experience I become more interested in - where other constraints permit - setting up tooling to facilitate work before starting work.

— James Thomas (@qahiccupps) December 5, 2015

And where that's not possible (e.g. JFDI) doing in a way that I hope will be conducive to later retrospective tooling.

— James Thomas (@qahiccupps) December 5, 2015

And I mean "tooling" in a very generic sense. Not just programming.

— James Thomas (@qahiccupps) December 5, 2015

And when I say "where other constraints permit" I include contextual factors, project expectations, mission, length etc not just budget

— James Thomas (@qahiccupps) December 5, 2015

Gah. I should've started this at https://t.co/DWcsnKiSfS. Perhaps tomorrow.

— James Thomas (@qahiccupps) December 5, 2015

I wonder if this is irony.

— James Thomas (@qahiccupps) December 5, 2015

A common scenario for me at a small scale is, when gathering data, whether I should start in text file, or Excel, or an Excel table. Within Excel, these days, I usually expect to switch to tables as soon as it becomes apparent I'm doing something more than inspecting data.

Most of my writing starts as plain text. Blog posts usually start in Notepad++ because I like the ease of editing in a real editor, because I save drafts to disk, because I work offline. (I'm writing this in Notepad++ now, offline because the internet connection where I am is flaky.) Evil Tester wrote about his workflow for blogging and his reasons for using offline editors too.

When writing in text files I also have heuristics about switching to a richer format. For instance, if I find that I'm using a set of multiply-indented bullets that are essentially representing two-dimensional data it's a sign that the data I am describing is richer than the format I'm using. I might switch to tabulated formatting in the document (if the data is small and likely to remain that way), I might switch to wiki table markup (if the document is destined for the wiki), or I might switch to a different tool altogether (either just for the data or for everything.)

At the command line I'll often start in shell, then move to bash script, then move to a more sophisticated scripting language.  If I think I might later add what I'm writing to a test suite I might make a different set of decisions to writing a one-off script. If I know I'm searching for repro steps I'll generally work in a shell script, recording various attempts as I go and commenting them out each time so that I can easily see what I did that lead to what. But if I think I'm going to be doing a lot of exploration in an area I have little idea about I might be more interactive but use script to log my attempts.

At a larger scale, I will try to think through workflows for data in the project: what will we collect, how will we want to analyse it, who will want to receive it, how will they want to use it? Data includes reports: who are we reporting to, how would they like to receive reports, who else might be interested? I have a set of defaults here: use existing tooling, use existing conventions, be open about everything.

Migration between tools is also interesting to me, not least because it's not always a conscious decision. I find I've begun to use Notepad++ more on Windows whereas for years I was an Emacs user on that platform. In part this is because my colleagues began to move that way and I wanted to be conversant in the same kinds of tools as them in order to share knowledge and experience. On the Linux command line I'll still use Emacs as my starting point, although I've begun to teach myself vi over the last two or three years. I don't want to become dependent on a tool to the point where I can't work in common, if spartan, environments. Using different tools for the same task has the added benefit of opening my mind to different possibilities and seeing how different concepts repeat across tools, and what doesn't, or what differs.

But some migrations take much longer, or never complete at all: I used to use find and grep together to identify files with certain characteristics and search them. Now I often use ack. But I'll continue to use find when I want to run a command on the results of the search, because I find its -exec option a more convenient tool than the standalone xargs.

Similarly I used to use grep and sed to search and filter JSON files. Now I often use jq when I need to filter cleanly, but I'll continue with grep as a kind of gross "landscaping" tool, because I find that the syntax is easier to remember even if the output is frequently dirtier.

On the other hand, there are sometimes tools that change the game instantly, In the past I used Emacs as a way to provide multiple command lines inside a single connection to a Linux server. (Aside: putty is the tool I use to connect to Linux servers from Windows.) When I discovered screen I immediately ditched the Emacs approach. Screen gives me something that Emacs could not: persistence across sessions. That single attribute is enough for me to swap tools. I didn't even know that kind of persistence was possible until I happened to be moaning about it to one of our Ops team. Why didn't I look for a solution to a problem that was causing me pain?

I don't know the answer to that.

I do know about Remote Desktop so I could have made an analogy and begun to look for the possibility of command line session persistence. I suspect that I just never considered it to be a possibility. I should know better. I am not omniscient. (No, really.) I don't have to imagine a solution in order to find one. I just have to know that I perceive a problem.

That's a lesson that, even now, I learn over and over. And here's another: even if there's not a full solution to my problem there may be partial solutions that are improvements on the situation I have.

In Part 4 I'll try to tie together the themes from this and the preceding two posts.
Image: https://flic.kr/p/5mPY4G
Syntax highlighting: http://markup.su/highlighter

Tools: Take Your Pick Part 2

In Part 1, I described my Sunday morning Cleaning the Bathroom problem and how I think about the tools I'm using, the way I use them, and why.  In particular I talked about using a credit card as a scraper for the grotty build up around the sides of the bath and sink. On the particular Sunday that kicked this chain of thoughts off, I noticed myself picking the card up and using a corner of it vertically, rather than its edge horizontally, to remove some guff that was collecting around the base of a tap.

This is something I've been doing regularly for a while now but, before I got the scraper, it was a job I used an old toothbrush for. In Part 1 I recounted a number of conscious decisions around the way I keep the littlest room spic and span, but switching to use the card on the tap wasn't one I could recall making.

Observing myself taking a tool I'd specifically obtained for one purpose and using it for another put me in mind of this old saw:

When all you have is a hammer, everything looks like a nail.

Until I looked it up¹ just now I hadn't heard this saying called The Law of the Instrument nor come across the slightly subtler formulation that Wikipedia attributes to Maslow:

I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.

Given the first of those two variants, it's easy to imagine that the universal application of the hammer is a mindless option - and we've all probably seen instances of that - but I think that, generally, tools are used in places where they are inappropriate or sub-optimal for a variety of reasons, and temptations, as Maslow would have it.

There are three explicit variables in play in this space: the problem, the tool, and the person using the tool. Here's one way I explored it, by considering the possible scenarios involving the tool and choice of tool, and trying to understand how the person might have got there:

I recognise the shape of the problem, and I have a tool that I know fits it

•  ... but I use my favourite tool instead because I'm more familiar with it.
•  ... but I use something else because of politics in the office, my boss, my colleagues, ...
•  ... but I use something novel because I want to own this problem and be the expert in it.
•  ... but I use something else to prevent an increase in our already large tool set.
•  ... but I won't use it because of ethical or moral issues I have with the tool vendor.
•  ... but I won't use it because of previous bad experiences with the tool, or others similar to it in some way.
•  ... but the context changed since I last looked, and I didn't notice, so I'll continue to use the existing tool.
•  ...

I recognise the shape of the problem, but I don't have a tool that I know fits it

• ... so I'll use the tool that I have invested heavily in anyway because sunk cost fallacy. 
• ... so I'll use the tool I do have that looks closest.
• ... so I'll use the tool that I have in my hand right now because there's no context-switching cost.
• ... so I'll continue to use the tool I am using now, despite its flaws, because I believe there is some benefit.
• ... so I'll use a tool I do have because there's no time/budget/desire to look for others.
• ... so I'll use something I do have because I'm uncertain of my ability to choose a new tool well.
• ... so I'll continue to use a tool I have because I'm worried about the cost of getting a new tool wrong.
• ... so I'll use whatever is to hand because I don't really care about doing a good job.
• ...

I don't recognise the shape of the problem

• ... so I'll try the tools I've got and see where they get me,
• ... or make a tool,
• ... or use no tool,
• ... or try break the problem down into pieces that can be attacked with tools I do know.
• ...

The latter class can be particularly galling because it contains two sub-cases:

•  I don't recognise the shape of the problem, and - even if I did - I don't have a tool that fits it.
•  I don't recognise the shape of the problem, but - if I did - I would find that I have a tool that fits it.

And much wailing and gnashing of teeth have been caused by the hindsight searchlight focusing on the second of those. Your wailing and gnashing of teeth, right? And the same is true of these scenarios:

I don't, or am not prepared to,  recognise the existence of a problem

• ... so I make no decisions about tool use at all
• ... which means that I might stay as I am or unconsciously drift into some other behaviour.
• ...

I recognise that there is no problem

• ... but I have an agenda that I am pushing and so force tool use anyway.
• ... but I just love to try new things so I'll go ahead and use a tool for its own sake.
• ... but I'm putting off some other work so I'll do needless work here.
• ... but I haven't got enough to do so I'll try this tool out to look busy.
• ...

As I enumerate these cases, I begin to think that they apply not just to the person with just the hammer, but to all of us, every time we do or not use any tool for any task.

In using any tool at all you are making a decision - implicitly or explicitly. When you enter three commands into the shell to get something to run you are either accepting that you will not use a script to do it for you, and avoid the typos, being in the wrong directory and so on, or you are missing out on the knowledge that a script could help you, perhaps because you don't care to avoid that time being spent on typing and typos.

In choosing to use the same tool that you always use for editing a file you are missing out on the chance to learn that there is something better out there. But also avoiding paying the costs of learning that new thing. Do you do that consciously?

I started trying to map these kinds of observations back onto my own exploration of ways in which I could satisfy my bathroom mission. As I did this, I came to realise that I have mostly cast the problem recognition and tool choice as something that is done from a position of knowledge of the problem. But my own experience shows me that it's less clear-cut than that.

In this area, I love Weinberg's definition of a problem:

A problem is a difference between things as desired and things as perceived.

I love it not least because it reminds me that there are multiple levers that can be pulled when confronted with a problem. If I am struggling with the shape of the problem I can change it, change my view of it, change my desires about what it should be. Opening out this way in turn reminds me that exploration of the space is an incredibly useful way to begin to understand which of those levers I can and/or should be pulling: perhaps if I can remove the things that look like nails, I can even put down my hammer.
 
Sometimes I find that I can learn more about the shape of the problem by using the tools I have and discovering their weaknesses. Sometimes I can only imagine a possible solution by attempting to resolve the problem the wrong way. If I do that tacitly, deliberately, then I'm here:

I recognise the shape of the problem, but I don't have a tool that I know fits it ... so I will explore the problem space with the tools I have, deliberately experimenting and hoping to learn more about the tools, the space, the problem, myself.

And I might find that I'm then saying "aha, if only I had something which could ..." or "oh, so perhaps I don't really need ..."

But this means deliberately deciding to spend some of whatever budget is available for problem solving on the meta task of understanding the problem. Stated as baldly as this it seems obvious that someone with a problem might consider that, doesn't it? But how many times have you seen something else happen?

How many times have you seen only a tiny fraction of the capacity of some tool being exploited? For anything more complicated than a hammer, it's easy not to know that there are capabilities not being used. The Law of the Instrument can be applied within tools too. If I don't know that Word can do mail merge, I might find myself buying a new tool to do it, for example.

On the other hand, creative reuse can be a good way to get additional value from an existing tool. A hammer can be used for things other than hitting a nail - as a door stop, as a lever, to encourage some seized machinery to become separated, as a counterbalance, as a pendulum weight, as a goal post, and might be a sufficiently good substitute for the "proper" tool in any given situation, at any given time. And, in fact, imagining ways to reuse a tool can itself be a useful way to get creative juices flowing.

But contexts change - the problem might alter, views of it might alter, the available tools might alter. Being open to reconsidering decisions is important in getting a good outcome. Doing nothing, reconsidering nothing - that pretty much guarantees at best standing still or perhaps applying a solution to a problem that no longer exists or applying the wrong solution to the problem that does.

Tool use is inherent in software development and the kinds of choices I've described above are being made all the time for all sorts of reasons, including those that I've given. It was interesting to me, as I enumerated these thoughts, that although in my bathroom cleaning example I have no reason to be anything other than on the level - there are no bathroom politics in our house and the stakes are not high in any dimension - and despite doing my best to be as clear to myself about what I'm thinking and trying at any given time as I can, I still proceeded to make choices unconsciously, to not take account of useful evidence, and to continue with one line of exploration past the point at which its utility was exhausted.

In Part 3 I'll try and recast these thoughts in terms of some practical examples from the world of work rather than bathroom cleaning.
Image: https://flic.kr/p/eFAoHQ

Footnote

1. Given where I come from, and its traditional rivalry with Birmingham, I'm amused that the hammer that's applied to every problem is sometimes called a Birmingham Screwdriver.