One conclusion seems to be that seeing successive replications of an experiment that you believe to be flawed does not mean that one should eventually lose one's doubt. Unfortunately, it also does not mean that one's doubt is justified, and that the results can safely be ignored. Rather, it means that at some point (if you want to improve your knowledge) you have to figure out some way to analyze the experiment from another angle and remove (or confirm) the base cause of the doubt.
"Again, they’re placing the original study in a privileged position. There’s nothing special about the original study, relative to the replication. The original study came first, that’s all. What we should really care about is what is happening in the general population."
There are two very different questions about replication.
One is whether the study got its results by chance, including forced-chance techniques like forking paths and salami slicing. This can be handled with either preregistration or exact replication. (And at p < .05, replication is a must because 5% of un-forced results will still be off!)
But the other is whether the study got its non-chance results by methodological flaws or an actual insight about the world. Exact replications are no good for this - doing the wrong thing twice is no better than doing it once. The power poses study, for instance, used testosterone sampling procedures that introduced known confounders. What would help is a study equivalent of N-version programming: settle, preferably preregistered, on multiple tests for the same effect. If they all work, you win. If some work (repeatably) and others don't, you've either made a design error or found a different effect than the one you were looking for.
This also explains how to work your confidence levels (a topic discussed in that SSC thread). You can't replicate a study endlessly and gain confidence every time. Given a prior for P(effect), exact replications boost P(effect ∪ bad study), and your P(effect) belief is bounded by the odds of a methodology error. It's a point I'd never considered until that SSC post, and one a lot of actual researchers still seem to miss.
That's a great article. Scott Alexander is always worth a read.
I especially love this bit:
> Two experimenters in the same laboratory, using the same apparatus, having no contact with the subjects except to introduce themselves and flip a few switches – and whether one or the other was there that day completely altered the result. For a good time, watch the gymnastics they have to do to in the paper to make this sound sufficiently sensical to even get published. This is the only journal article I’ve ever read where, in the part of the Discussion section where you’re supposed to propose possible reasons for your findings, both authors suggest maybe their co-author hacked into the computer and altered the results.
"For example, Savva et al was listed as an “exact replication” of Bem, but it was performed in 2004 – seven years before Bem’s original study took place. I know Bem believes in precognition, but that’s going too far."
Oh man, I've been looking for this article for ages. When the new Bem article came out yesterday I hunted for it again and couldn't find it. Thank you so much!
