Hiccupps

I'm definitely no expert in Large Language Models but expertise is not required, given all the publicity, to know that variation in the responses from the technology is expected.  To illustrate that, I asked ChatGPT  "In one sentence, why do LLM responses vary on the same prompt?" and got three different, although clearly related and overlapping, answers: LLM responses vary on the same prompt due to the model's stochastic nature, diverse training data, and sensitivity to slight changes in input phrasing.  LLM responses vary on the same prompt due to the model's probabilistic nature, employing diverse sampling techniques and leveraging its extensive training data to generate a range of contextually plausible and grammatically coherent outputs. LLM responses vary on the same prompt due to the model's inherent stochasticity and diverse training data, leading to dif...

  I was listening to the Radiolab podcast Stochasticity yesterday as I walked to the shop. The presenters were talking to two women, both named Laura Buxton, who became friends years ago because one of them released a balloon with their name and address attached from their back garden and the other found it 150 miles away in their back garden.  After Laura Two got in touch with Laura One they discovered other incredible similarities: they were both tall for their age, had brown hair and blue eyes, both had a labrador, black, a rabbit, grey, and a guinea pig with orange markings. They ended up in the local newspaper, and on national and even international television talking about how glad they were that fate had brought them together. If that sounds astonishing, how about this? As I strolled along the river and the podcast turned to the question of the probability of Laura-like events, David Spiegelhalter , a renowned statistician, jogged past me. What are the chances? The po...

When I asked Twitter this: Anyone know of a course on risk analysis that could work as in-house training for software testers?   It's a topic we're interested in but the stuff I'm turning up is more to do with corporate analysis, register building , mitigation at the business level #testing #risk Paul Hankin  suggested a book, Superforecasting, The Art of Science and Prediction , by Philip Tetlock and Dan Gardner. As the title suggests the book is about prognosis rather than peril, but the needs of prediction and risk analysis overlap in interesting ways: understanding possible outcomes, identifying factors that contribute to those outcomes, and weighting the factors and their interactions. Tetlock and Gardner study forecasting and forecasters. They use a metric called the Brier score to assess the accuracy of forecasts, and, over time, and with repeated forecasts, it becomes clear that some people tend to make better forecasts than others. The Brier score...

Fiona Charles ran a workshop on business risk analysis for my team at Linguamatics last week. Across the day we covered risk-based testing , how it can help with prioritisation, and how it is often overlooked as a factor in test design. We also looked at how the presentation of risks and their potential impact to someone who matters can be a way to engage stakeholders in the testing effort. Hopefully, this would in turn encourage contribution to activities such as test idea generation, triage, and attempts to mitigate risk elsewhere during design and development. Stakeholders often expect a level of testing we can't deliver. (Fiona Charles) The approach to risk assessment that Fiona outlined has some similarity to a pre-mortem . Essentially: assume the system has been implemented then look for ways in which it could go wrong. It's important to understand who the relevant stakeholders are — they are more than just your users — and to solicit diverse perspectives in you...

This week Sime suggested  Tidy Data by Hadley Wickham  for the Test team book club: A huge amount of effort is spent cleaning data to get it ready for analysis, but there has been little research on how to make data cleaning as easy and effective as possible. This paper tackles a small, but important, component of data cleaning: data tidying. Tidy datasets are easy to manipulate, model and visualize, and have a specific structure: each variable is a column, each observation is a row, and each type of observational unit is a table.  Messy data needn't be bad data, but it might not be in a format that makes it easy to process. Many tables used for data presentation will contain implicit variables, such as person or result in the table here: If you've ever generated, aggregated, or inherited data of any scale for analysis you're almost certainly already familiar with the basic ideas. You've probably also done informally, with much cursing, copy-pasting, and...

Skimming a recent issue of Significance , the journal of the Royal Statistical Society, I saw a reference to a paper called We Need Both Exploratory and Confirmatory by John Tukey, published in the 1980 in The American Statistician. I hunted it down and found much that felt familiar. Here's a few quotes: Exploratory data analysis is an attitude, a flexibility, and a reliance on display, NOT a bundle of techniques, and should be so taught. Confirmatory data analysis, by contrast, is easier to teach and easier to computerize. ... to implement the confirmatory paradigm properly we need to do a lot of exploratory work. Neither exploratory nor confirmatory is sufficient alone. To try to replace either by the other is madness. We need them both Science DOES NOT BEGIN WITH A TIDY QUESTION. Nor does it end with a tidy answer. No catalogue of techniques can convey a willingness to look for what can be seen, whether or not anticipated. Yet this is at the heart of exploratory d...

I2E, the flagship product at Linguamatics , is a text mining engine and so sits in the broad space of search tools such as grep , Ctrl-F , and even Google. In that world, evaluating "how good" or "how relevant", or the "correctness" of a set of search results is interesting for a number of reasons, including: it may be hard to define what those terms mean, in general cases. it may be possible to calculate some kind of metric on well-understood, small, data sets but less so at scale.  it may be possible to calculate some kind of metric for simple searches, but less so for complex ones. on different occasions the person searching may have different intent and needs from the same search. But today we'll concentrate on two standard metrics that can be easily defined and which have agreed definitions: precision (roughly "how useful the search results are")  and recall (roughly "how complete the results are"). Imagine we wan...

Spoiler alert: there is no happy ending here. Often a blog post or article will talk about some inspiration gained from outside of testing which was applied successfully to the author's testing and yielded useful results. Those are great posts to read  ... but this isn't one of them. This one starts with inspiration from outside of testing followed by research, discussion, criticism and thought experiment but ultimately no application. The yield here is the documentation of all of that. Hopefully it's useful and still a good read. What was the inspiration? I've long had the notion - sadly dormant and largely sidelined - that I'd like to look into the use of statistics in testing. (One of my earliest posts here was on entropy .) As testers we talk a lot about such things as the scientific method and about data, about running experiments, about crafting experiments such that they generate usable and valuable data, about making decisions on where to test next ...

I was watching Stuart Hunter's  What is Design of Experiments?  ( 1 ,  2 ) lectures this morning. They were made in the mid-60s for Westinghouse Electric and, after a demonstration of the exploration of a space to find the interesting location guided by a cycle of experiment, data collection and analysis, the final sentences run like this: The purpose of statistics is to analyse the data in such a way as to lead to new conjecture ... the data are expensive and the data are variable and we don't know the true response function ... All of this is part and parcel of the arts of statistics.  But they're really not very valuable unless we can call upon the experience that people such as [you, the audience] have which comes only from working in an industrial environment. And our real objective is to wed together your experience with the practical uses of engineering and production and development and research with the arts of statistics so that you will be more ...

I took the National Institutes of Health 's course Protecting Human Research Participants  this week. It's aimed at people setting up experiments involving human subjects and covers areas such as risk (including identification, minimisation, compared to benefits - personal and communal), recruitment (including coercion, balance), rights of the participants (including consent, welfare, vulnerable groups) and statutes (including international research, differences between definitions in different US bodies). In a section on the design of a clinical trial (where interventions - such as drugs - are being compared for efficacy in treating some illness in humans), the term equipoise was introduced. The definition given is: Substantial scientific uncertainty about which treatments will benefit subjects most, or a lack of consensus in the field that one intervention is superior to another. and the course notes say: A state of equipoise is required for conducting research that...

Happenstance. Ideally, I wouldn't have been reading two books at once. But I'd already started the one when the Dev manager chucked the other onto my desk as he stalked past, wild-eyed from some meeting, en route to the kitchen for a caffeine salve to the throbbing vein in his forehead. Intrigued (with the book not the vein, I've seen that plenty of times now), I started flicking through it, got hooked and then alternated between it and my own over the next few days, seeing the connections that both made to the, ah, idea that ideas (or lack of them) can be a problem. The Dev manager's book , Fooled by Randomness by Nassim Nicholas Taleb, talks about consequences of an insufficient grasp of potential outcomes. Taleb's domain is financial markets and the instruments that populate them and he attributes the false confidence of many traders - and the population at large, by generalisation - to a lack of understanding of first possible scenarios and second the abil...