Interpretering Bayesian Analysis

[Billy Shears]

Background
On the Interpreter blog, Dr. Kyler Rasmussen Ph.D. has begun a series of posts “summarizing and evaluating Book of Mormon-related evidence from a Bayesian statistical perspective.” While the moderators of the comment section of that website have been very generous allowing me to post my criticisms there, the formatting is subpar. I thought I’d post some comments regarding his methodology here.

To explain my point, I’m going to present a simple example of how Bayesian analysis is supposed to work that should be easy to understand. I’ll then use that as an analogy for what Dr. Rasmussen is doing wrong.

Simple Example
Say I have a coin. I have the hypothesis that the coin is fair, i.e. that the probability of flipping heads is 50%. In contrast, Kyler thinks the coin is weighted, and thinks the probability of flipping heads is 55%. How could we settle this?

We necessarily need a prior assumption, and I set the assumption to me being 99% sure I am right—coins are generally fair.

We then flip the coin 30 times, and end up with the following basket of evidence:

TTHTHHTTHTHTTHTHTTHHTTTTHTHHTT

(That is 18 heads and 12 tails, in that order)

We then need to ask ourselves two questions:

1- What are the chances we’d get that basket of evidence if I’m right?
2- What are the chances we’d get that basket of evidence if Kyler is right?

It turns out that if the probability of getting that string of results with a fair coin is 0.000000000931 (i.e. 1 in 1.07 billion). In contrast, the probability of getting that string of results with a coin weighted 55:45 towards heads is 0.000000001462 (i.e. 1 in 680 million). Knowing that, we can do some algebra and see that the results moved in Kyler’s direction. It isn’t overwhelming, but we can update the probability of the coin being weighted from 1% to 1.56%. At that point we could conduct some more experiments, gather more evidence, and update the probability further. Eventually we would converge on the correct answer.

[continued]

[Billy Shears]

Commentary
I want to point out here is that regardless of what the probability is of flipping a head, getting that particular basket of evidence is very, very, unlikely. That’s the nature of reality. Richard Feynman sarcastically made the point when he said “You know, the most amazing thing happened to me tonight. I saw a car with the license plate ARW 357. Can you imagine? Of all the millions of license plates in the state, what was the chance that I would see that particular one tonight? Amazing!”

[Billy Shears]

What Kyler Should Do
In his analysis, Kyler ought to collect 23 baskets of evidence. Each basket of evidence should be completely independent of the others. And they ought to collectively encompass all of the evidence that exists regarding the Book of Mormon, both favorable and unfavorable.

He then ought to establish two clear hypotheses that broadly encompass all possibilities:

H0: The hypothesis that the Book of Mormon is 19th century fiction
H1: The hypothesis that the Book of Mormon is an accurate translation of an authentic ancient manuscript

He then ought to go through the evidence and ask two questions:

1- What is the probability I’d see this basket of evidence if H0 is true?
2- What is the probability I’d see this basket of evidence if H1 is true?

He can then do the algebra.

[Dr Moore]

Welcome, Billy Shears!

Your comments above are nice illustration of controlled experimental process and why I continue to rib Kyler over his sloppy wielding of powerful statistical tools.

[Billy Shears]

What Kyler Is Doing: Problem 1
The first fundamental problem is that on every episode, he changes his hypothesis to suite the evidence he is looking for. On the first one, the hypotheses were:

H0: Joseph Smith was author
H1: Ancient collection of authors

On the second episode:

H0: The first vision story was fraudulent
H1: The first vision story is based on an actual event

On the third:

H0: The Book of Mormon was either automatic writing or fraudulent dictation
H1: The Book of Mormon dictation was created by “authentic dictation”

He needs to choose what, specifically, he is testing. He then needs to rigorously stick to that. He can’t test these various things and update a “posterior probability” of something that is meaningful.

[Analytics]

Welcome Billy!

[Gadianton]

Welcome back Billy.

ETA:
letting you finish your analysis before I respond.

[Billy Shears]

What Kyler is Doing: Problem 2
As stated above, Kyler needs to ask two questions in each episode for the basket of evidence featured in that episode:

1- What is the probability I’d see this basket of evidence if H0 is true?
2- What is the probability I’d see this basket of evidence if H1 is true?

Kyler isn’t consistently using the same basket of evidence. In episode 1, he used page counts of 19th Century literature as the basket of evidence when evaluating H0, and he used the page counts of the book of the Bible for evaluating H1. Those are two different things.

Likewise in episode 3, for H0 he looked at how quickly people dictate automatic writing and some numbers he made. For H1, he didn’t look at any evidence at all.

The next problem is the most fundamental one.

[Billy Shears]

What Kyler Is Doing: Problem 3
The biggest problem is that he isn’t comparing the hypothesis that the Book of Mormon is 19th Century fiction to another testable hypothesis.

Referring back to my analogy, rather than hypothesizing the probability of flipping a heads is 55%, Dr. Rasmussen is saying that by divine intervention, God chose the flips of the coin to be precisely:

TTHTHHTTHTHTTHTHTTHHTTTTHTHHTT

But he isn't making that hypothesis until after the coin was tossed 30 times.

That is specifically why in Episode 3 he came to the conclusion that the Book of Mormon being dictated in 65 days is means it is 100,000 times more likely to be authentic. He says the probability of fiction being dictated in 65 days is 0.000000154. Okay. Whatever. As I emphasized earlier, reality is unlikely. But he then compares that to the probability of authentic scripture being dictated in 65 days. What is the probability authentic scripture is dictated in 65 days? According to Kyler, it is 100%.

[Doctor CamNC4Me]

crap, man. Billy Shears is in the house!

Dr. Rasmussen,

I’ll lay $1,000 on the line, right now, if you come to this thread and defend your analysis just successful enough so that Lemmie has to begrudgingly admit your process and math, as it were, is sound.

That’s all. That’s all you have to do, and since you’ve already done the hard work, I’m sure it won’t be too onerous to do it. $1,000 to you or your charity of choice.

- Doc