The Test team book club is reading Guide: Achieving Observability from Honeycomb, a high-level white paper outlining what observability of a system means, why you might want it, and factors relevant to achieving and getting value from it.
It's not a particularly technical piece but it's sketched out to sufficient depth that our conversations have compared the content of the guide to the approaches taken in some of our internal projects, the problems they present, and our current solutions to them.
While I enjoy that practical stuff a great deal, I also enjoy chewing over the semantics of the terminology and making connections between domains. Here's a couple of first thoughts in that area.
The guide distinguishes between monitoring and observability.
- monitoring: "Monitoring .. will tell you when something you know about but haven't fixed yet happens again" and "... you are unable to answer any questions you didn’t predict in advance. Monitoring ... discard[s] all the context of your events".
- observability: "Observability is all about answering questions about your system using data", "rapidly iterate through hypothesis after hypothesis" and "strengthen the human element, the curiosity element, the ability to make connections."
Increasing observability will generally result in more data being collected. A couple of dimensions are important here:
- high cardinality: refers to keys which can take many values. User, request, or session identifiers in a busy web service might be a good example. Being able to slice out data according to these kinds of variables allows analysis to take place at appropriate resolutions for given questions.
- richness: refers to the variety of data collected or, to put it another way, the amount of context that's associated with any "event" that the system records.
When assessing evidence, it is helpful to see a full data matrix, all observations for all variables, those private numbers from which the public displays are constructed. No telling what will turn up. (p. 45)This is one of the reasons that, at a small scale, I think Excel's pivot tables are such a valuable tool for quick, convenient, exploratory data analysis, and why one day I'll write a post about that.