Wednesday, August 23, 2017

Quite the Reverse

Cohen and Medley, in Stop Working & Start Thinking, say:
Simple tests are not experiments ... A chef will bake a cake at different temperatures and find the one that gives the best results ... [but] we should only include [this test] in classical "science" if the "normal" situation is included as a control ... Every careful observation of a puzzling or new phenomenon should be matched to similar observations of well-understood or classical material.
They go on to introduce some useful terminology: variables are the things that you will aim to alter in the experiment; all other factors that could vary, but which you will aim not to vary, are parameters.

And they then describe three types of experiment concerned with investigating the possibility of a causal relationship between a variable, A, and an outcome, X.
Deficit: run one experiment with A and one without A. Monitor the presence of X in both cases. If X is seen with A but not without A then perhaps we have some evidence that lack of A inhibits X.
Result reversal: run a single experiment during which A is first removed and then reintroduced, holding all parameters constant, If X disappears when A is removed and reappears when A is brought back, then there is stronger evidence of a causal relationship between A and X.
Demi-reversal: when it's not possible to control the parameters in the experimental context  — for example in real world ecological investigations — then introducing A into an environment which is already understood and looking for changes in X can be indicative of a causal relationship too.
Result reversal is convincing, they say, because "there are so many more ways to lose function than there are to regain it. You can only fix your car by repairing the fault, but your car can fail for a billion and one reasons."

Once you have a hypothesis, its value will be dependent upon its power of prediction of the outcomes in new experiments. When you can't control the parameters enough to run new experiments, you may be able to use retrodiction, where historical data is analysed to see whether the hypothesis can be seen to have held in relevant situations in the past.

It's sometimes dense and dry, and it gives deeper-than-is-helpful-to-a-non-scientist detail about some of the experiments described. Despite this, I really liked Stop Working & Start Thinking as a basic guide to experimental practice, and a reminder to step back from the coal face, written by seasoned practitioners. I pulled out a couple of other aspects that I found particularly enjoyable in A Different Class and Ignorance, Recognised.

No comments:

Post a Comment