Might have found the source for the Dales' skeptical prior.
https://blogs.scientificamerican.com/cr ... redirect=1Horgan: What’s so great about Bayes’ Theorem?
Yudkowsky: For one thing, Bayes's Theorem is incredibly deep. So it's not easy to give a brief answer to that.
I might answer that Bayes's Theorem is a kind of Second Law of Thermodynamics for cognition. If you obtain a well-calibrated posterior belief that some proposition is 99% probable, whether that proposition is milk being available at the supermarket or global warming being anthropogenic, then you must have processed some combination of sufficiently good priors and sufficiently strong evidence. That's not a normative demand, it's a law. In the same way that a car can't run without dissipating entropy, you simply don't get an accurate map of the world without a process that has Bayesian structure buried somewhere inside it, even if the process doesn't explicitly represent probabilities or likelihood ratios. You had strong-enough evidence and a good-enough prior or you wouldn't have gotten there.
On a personal level, I think the main inspiration Bayes has to offer us is just the fact that there are rules, that there are iron laws that govern whether a mode of thinking works to map reality. Mormons are told that they'll know the truth of the Book of Mormon through feeling a burning sensation in their hearts.
Let's conservatively set the prior probability of the Book of Mormon at one to a billion (against). We then ask about the likelihood that, assuming the Book of Mormon is false, someone would feel a burning sensation in their heart after being told to expect one. If you understand Bayes's Rule you can see at once that the improbability of the evidence is not commensurate with the improbability of the hypothesis it's trying to lift. You don't even have to make up numbers to see that the numbers don't add up - as Philip Tetlock found in his study of superforecasters, superforecasters often know Bayes's Rule but they rarely make up specific probabilities. On some level, it's harder to be fooled if you just realize on a gut level that there is math, that there is some math you'd do to arrive at the exact strength of the evidence and whether it sufficed to lift the prior improbability of the hypothesis. That you can't just make stuff up and believe what you want to believe because that doesn't work.