The function of repeatabilty in scientific experiments Many thanks for your excellent answer. It's taken me a while to get back to it.
If you repeat a measurement under the same conditions in an experiment, the goal of that is usually to take an average; establishing concordance and forming a variance reduced estimate of the true value you're measuring. — fdrake
:nod:
If you repeat a measurement under different conditions in an experiment, in part that's trying to find out how the measured response varies with the stimulus/treatment, in part that's trying to find out how that response varies with contextual factors, in part (nowadays) that's trying to assess whether and how the stimulus/treatment's response itself varies with contextual factors. On this level, "repeating a measurement" is pretty much the core of a controlled experiment. — fdrake
Indeed.
If you're repeating an entire experiment, there's some wiggle room in practice regarding what counts as a repeat. There's the hypothetical "exact replication", which is where you do literally everything the same, the "conceptual replication", which is where you try to ape the experimental conditions to be the same but can't do it exactly. I doubt those are an exhaustive typology of replication results, but the purpose of both isn't easily reducible to confirming or testing a previously held hypothesis in most cases, and that follows just because the overall set up in the initial experiment isn't identical, or necessarily even equivalent in all relevant respects, to the replication attempt. — fdrake
I hadn't thought of that specifically, thanks.
That "lack of identity" (arguably) shows up in the difference in replication rates between papers where the initial researcher group is represented in the reproduction team and where they are not. — fdrake
That's interesting.
I would make the claim that the function of reproduction attempts/replication attempts in science isn't to check the reliability of any individual result; most results are false and over-simplifications and everyone knows this; the overall function is to make the process of scientific discovery in the aggregate not spend too long on "clear" falsehoods and inaccuracies, it's a quality control thing. What counts as a "clear falsehood" only makes sense in light of reproducibility. — fdrake
Sure. I think by 'results' you mean conclusions/interpretations rather than data?
Another angle on repeatability is that if you're repeating the experiment, manage it exactly, and the effect doesn't show up the same as before, that doesn't necessarily mean the conclusions of the initial experiment were false - it might be that the response is contextually variable, it might be a contextual interaction - both experiments could be samples of a distribution associated with the "true effect" indexed by contexts and their variables. The latter approach, to my understanding, is the one favoured by Gelman and his group. — fdrake
Another very good point I hadn't thought of.
I think that depends too, the role of a non-repeat, if you see it in the context of a contextually variable interaction, it's not a refutation but evidence that the effect is contextual if it exists (and that starts a process of compensation of making it smaller compared to context induced imprecision, "exaggeration factors" "the garden of forking paths", and analysing true power of the study/broader scientific endeavour), if you see it in the context of everything's really set up exactly the same, the effect's probably not there as it was theorised - but if the "exact replication" must reproduce the contextual ambiguities of the initial one? It still doesn't mean the effect's not there/is 0* if the second one comes out, it could be that the ambiguities realised differently in both experiments.
In that kind of case, if the ambiguities are enough to swamp the signal, it's reasonable to say the treatment as intended or the effect as theorised has little to no evidence that it exists... Probably. — fdrake
Yes, I completely glossed over all that nuance. My initial post popped into my head by considering vague memories of my philosophy of science on falsificationism and confirmationism. I found all that really interesting at the time but forgot most of it.
Thank you, that was really interesting and helpful.