Peterson, Priestcraft, Profits and the Paranormal

[Physics Guy]

Even without any hacking, the 95% confidence threshold just isn’t very high. This poses a basic problem.

If there are twenty studies, on average one will show something that would only occur by chance one time in twenty. The other 19 are likely not published, because they find nothing significant, and so the only published research on the topic has shown a significant effect, even though there isn’t really anything there at all.

This may sound like such a basic and trivial problem that it couldn’t possibly really be that simple, because that would make much of medical and sociological research completely unreliable. Alas, though, this simple problem is indeed all too real, and it really does make a lot of empirical science unreliable.

People try to deal with the problem. In the so-called hard sciences, the standard is much higher than 95%. The only case I know in physics where any specific confidence level is even mentioned is for the detection of a new kind of particle, where the threshold is about 99.99994% (5 standard deviations). Otherwise the standard is basically, “If you need to ask about this, you need more data.” The hard sciences should really be called the easy sciences, though. You can’t do medical trials with a billion subjects, but a billion atoms are almost nothing. So in practice we’re stuck with much lower standards, in order at least to make educated guesses about important questions.

One important practice is to publish every attempt to examine a question, even if its results are negative. This means publishing a lot of boring papers that don’t surprise anyone or seem to deserve much attention, but efforts are made to have lower-tier repositories for registering negative results, just to give a more realistic picture. It’s still kind of tough, because often the negative studies really are boring, and their results seem obvious in advance; that lucky twentieth study only seemed noteworthy because of its surprisingly unusual finding. Publishing negative results is still coming to be seen as best practice, though.

The main solution is just to try to replicate anything that seems surprising. If it was actually just a fluke of chance, then the next dozen attempts to reproduce it won’t show anything, and the result will get dismissed, as it should be, as a fluke.

The fact that exactly that happens a lot is the “replication crisis”. It isn’t really a crisis. It’s how things should work. Everyone just has to realize that one 95%-confident finding does not equal a solid discovery, and chill out about individual papers, until enough replications have followed to confirm the result.

Needless to say, parapsychology is full of legitimately published papers that might not have been cooked or hacked to reach their 95% confidence threshold, but that have never been replicated.