The Interpreter; Bayes Theorem; Nephites and Mayans

[_Philo Sofee]

Very good summary Physics Guy. Thak you for this. In all this, I note that it is not Bayes Theorem that is at fault, it is the crackpot usage and assumptions about it that are wrong. Richard Carrier has discussed this aspect of Bayes as well in his book "Proving History," (still the best, most comprehensive discussion of HOW to use Bayes properly in print) of which the Dales should acquaint themselves with before endeavoring on any future usages of a signal science using silly means. It is not Bayes Theorem that gives the ridiculous subjective answers, it is the Dale's lack of honest peer review and desire to get the real answer, istead of the church answer perhaps realizing deep down what the real answer will be. This is yet another piece of evidence that Mormon scholarship is not interested in truth, it is interested in "disciple scholarship", that is, in testimony confirmation regardless of what means used, or what truths ignored contradictory to their desires results. It is, to be pointed about it, phony.

Another lesson in why apologetics simply cannot ever gain the truck to be a valid enterprise of scholarship.

I wonder if the powers at Interpreter even have the mental capacity to grasp that this thread, this one right here critiquing the Interpreter's articles, IS PEER REVIEW? Would the Interpreter wish to have actual peer review it would present the idea and paper it wishes to publish right here and have it looked over and commented on, and then could give valid corrections and ideas. Wer all know that would be more miraculous than Elohim actually coming down and dwelling among us normal folk in his levis and t shirts, sipping lemonade on hot days with us chatting about the meaning of life.

[_Analytics]

Very good summary Physics Guy. Thak you for this. In all this, I note that it is not Bayes Theorem that is at fault, it is the crackpot usage and assumptions about it that are wrong. Richard Carrier has discussed this aspect of Bayes as well in his book "Proving History," (still the best, most comprehensive discussion of HOW to use Bayes properly in print) of which the Dales should acquaint themselves with before endeavoring on any future usages of a signal science using silly means. It is not Bayes Theorem that gives the ridiculous subjective answers, it is the Dale's lack of honest peer review and desire to get the real answer...

Well said. For those who have used the Dales as an segue to mock Carrier, the Dales came up with the probability of the Book of Mormon being a fake as 1.27x10^-137 (or whatever), while Carrier said the probability of Jesus being historical is less than 33%. And while the Dales say that they have conclusively proven that the Book of Mormon his historical, case closed, Carrier says that Bayesian framework gives people who disagree with his conclusions a proper framework to make their case. You can drink Carrier's Jägermeister without drinking the Dales' Kool Aid.

[_Analytics]

Here is my summary of the critique so far. I'm going to have to give up on this issue now and get some actual work done ...

1) The Dales' likelihood ratios aren't calculated at all, but just chosen, by subjective assessment, from a small set of allowed values. As Shumway pointed out, the Dales often seem to assign likelihood ratios as low as 0.1 when a value as high as 0.9 might be at least as plausible. If we raise 0.1 to the 132nd power, we get the insanely low odds of 1 in 10^132, but if instead we raise 0.9 to the 132nd power, we get odds of only just under 1 in a million. That's a factor of 10^126 which is subjective because of subjectivity raised to a high power. The subjectivity does not tend to average out over the long haul of 132 subjective assessments because the subjective assessments by the Dales are not random. Their subjective assessments are consistently biased in favor of the Book of Mormon being true.

2) The Dales' likelihood ratios are all assigned from among a small set of allowed values, and this set of ratios does not go higher than 50 or lower than 1/50. So the logical impact of strong evidence is arbitrarily capped by the Dales at a very modest level. This arbitrary capping artificially prevents discrepancies like horses or steel from having the decisive effects that they should have. Since the features that the Dales consider consist of many debatable "hits" plus a few glaring discrepancies, capping the impact of the discrepancies grossly inflates the final likelihood of the Book of Mormon being authentic. The artificial inflation factor in the final odds due to arbitrary capping might plausibly be as large as 10^60 or so.

3) The Dales multiply a large number of small likelihood ratios together, but this would only be valid if the likelihood ratios were independent and they obviously aren't. A fraudulent Book of Mormon would never be composed by making independent random guesses for every feature of the fictional Nephite world. Instead that fictional world would be invented by consistently copying from a model society (such as Biblical Israelite culture) and model geography (such as New England), with only a few independent choices to be made according to the author's fancy. So even if the Dales' subjectively assessed likelihood ratios for fraud were all fair for each individual feature, it would be extremely wrong to multiply them all together as if they were independent. Ignoring their correlation is such a severe error that it can easily inflate the odds against fraudulence by a factor of 10^100, perhaps even by much more.

Conclusion
The Dales' final estimate of the odds against the Book of Mormon being fraudulent is very high, but this is only because the severe flaws in their methodology have produced very large false inflation factors. The Dales' very high odds against the Book of Mormon being fraudulent are thus entirely an artifact of bad methodology. If their calculation is redone with the subjectively assessed likelihoods being chosen more skeptically, with adverse evidence allowed its full weight instead of being artificially capped, and with many highly correlated choices not being treated as independent guesses, then the Dales' conclusion can easily be completely reversed.

The Dales have not only produced an extremely optimistic Mormon odds estimate that is very different from the estimate that skeptics could make. They have also made major objective errors in capping likelihoods and ignoring many strong correlations. Beyond even that, they have confidently presented their analysis as rigorous; their disastrously invalid assumptions have been made without any admission that the assumptions are even debatable. Without that overconfidence this paper might have been a scientific work that was just badly mistaken, but the overconfidence marks it as a crackpot production of which Interpreter should be ashamed.

Great summary, but let me add one thing to it that hasn't received enough attention here.

The Dales selected their evidence by looking at things that were mentioned in both the Book of Mormon and in The Maya. If something wasn't mentioned in both books it was excluded from consideration. They claim they did this because it was the only way to objectively select what was going to be considered.

However, only selecting information that is in both places biases the results towards hits. This is actually proven in their own "control." According to the evidence they selected and the weights they gave, the View of the Hebrews has 15 positive hits and only 9 negative hits. In aggregate, according to how the Dales weight this, this increases the odds of View of the Hebrews being an accurate, ancient reflection of the Mayan world by 65:1. Those are the results from their control.

They covered up how this exposes a flaw in their methodology by giving the "hoax" hypothesis a priori odds of 1-billion to 1, which thus overwhelms the actual evidence which is, again, that the View of the Hebrews is in fact a Mayan document. That is according to their biased analysis and calculations. In their control!

[_DrW]

Here is my summary of the critique so far. I'm going to have to give up on this issue now and get some actual work done ...

1) The Dales' likelihood ratios aren't calculated at all, but just chosen, by subjective assessment, from a small set of allowed values. As Shumway pointed out, the Dales often seem to assign likelihood ratios as low as 0.1 when a value as high as 0.9 might be at least as plausible. If we raise 0.1 to the 132nd power, we get the insanely low odds of 1 in 10^132, but if instead we raise 0.9 to the 132nd power, we get odds of only just under 1 in a million. That's a factor of 10^126 which is subjective because of subjectivity raised to a high power. The subjectivity does not tend to average out over the long haul of 132 subjective assessments because the subjective assessments by the Dales are not random. Their subjective assessments are consistently biased in favor of the Book of Mormon being true.

2) The Dales' likelihood ratios are all assigned from among a small set of allowed values, and this set of ratios does not go higher than 50 or lower than 1/50. So the logical impact of strong evidence is arbitrarily capped by the Dales at a very modest level. This arbitrary capping artificially prevents discrepancies like horses or steel from having the decisive effects that they should have. Since the features that the Dales consider consist of many debatable "hits" plus a few glaring discrepancies, capping the impact of the discrepancies grossly inflates the final likelihood of the Book of Mormon being authentic. The artificial inflation factor in the final odds due to arbitrary capping might plausibly be as large as 10^60 or so.

3) The Dales multiply a large number of small likelihood ratios together, but this would only be valid if the likelihood ratios were independent and they obviously aren't. A fraudulent Book of Mormon would never be composed by making independent random guesses for every feature of the fictional Nephite world. Instead that fictional world would be invented by consistently copying from a model society (such as Biblical Israelite culture) and model geography (such as New England), with only a few independent choices to be made according to the author's fancy. So even if the Dales' subjectively assessed likelihood ratios for fraud were all fair for each individual feature, it would be extremely wrong to multiply them all together as if they were independent. Ignoring their correlation is such a severe error that it can easily inflate the odds against fraudulence by a factor of 10^100, perhaps even by much more.

Conclusion
The Dales' final estimate of the odds against the Book of Mormon being fraudulent is very high, but this is only because the severe flaws in their methodology have produced very large false inflation factors. The Dales' very high odds against the Book of Mormon being fraudulent are thus entirely an artifact of bad methodology. If their calculation is redone with the subjectively assessed likelihoods being chosen more skeptically, with adverse evidence allowed its full weight instead of being artificially capped, and with many highly correlated choices not being treated as independent guesses, then the Dales' conclusion can easily be completely reversed.

The Dales have not only produced an extremely optimistic Mormon odds estimate that is very different from the estimate that skeptics could make. They have also made major objective errors in capping likelihoods and ignoring many strong correlations. Beyond even that, they have confidently presented their analysis as rigorous; their disastrously invalid assumptions have been made without any admission that the assumptions are even debatable. Without that overconfidence this paper might have been a scientific work that was just badly mistaken, but the overconfidence marks it as a crackpot production of which Interpreter should be ashamed.

Great summary, but let me add one thing to it that hasn't received enough attention here.

The Dales selected their evidence by looking at things that were mentioned in both the Book of Mormon and in The Maya. If something wasn't mentioned in both books it was excluded from consideration. They claim they did this because it was the only way to objectively select what was going to be considered.

However, only selecting information that is in both places biases the results towards hits. This is actually proven in their own "control." According to the evidence they selected and the weights they gave, the View of the Hebrews has 15 positive hits and only 9 negative hits. In aggregate, according to how the Dales weight this, this increases the odds of View of the Hebrews being an accurate, ancient reflection of the Mayan world by 65:1. Those are the results from their control.

They covered up how this exposes a flaw in their methodology by giving the "hoax" hypothesis a conjugate prior of 1-billion to 1, which thus overwhelms the actual evidence which is, again, that the View of the Hebrews is in fact a Mayan document. That is according to their biased analysis and calculations. In their control!

It just keeps getting better, does it not?

[_Water Dog]

A good summary, PG. I would add one thing though. I think some of y'all are still confused about where these LR numbers are coming from. Perhaps I'm wrong, but here's what I'm seeing. They are indeed arbitrary, but I believe they are relying on Harold Jeffreys to derive their arbitrary numbers by working his K table in reverse.

From the blog this stuck out to me,

Now, in the real world, we usually don’t experience the kind of mathematically well-defined probabilities that rolling dice offers to us. Instead, we usually deal with “odds” or “probabilities”, many of which are somewhat subjective. By “subjective”, I mean that the individual must decide for himself/herself what constitutes strong evidence, what evidence is positive but not especially strong, and what evidence is just barely worth mentioning.

Bayesian statistics provides one approach to this kind of problem and is well-explained in this introductory article.

....

In the Bayesian approach, we write the likelihood ratio...

....

We can assign a likelihood ratio or “Bayes factor” to each statement of fact associated with the revelations given to Joseph. This likelihood ratio is the probability that the statement is true given the assumption that he was just guessing divided by the probability that the statement is true given the assumption that he was a prophet. This likelihood ratio therefore represents the strength of the evidence, in this case the evidence against Joseph being a prophet.

The terminology they use, "strong," "positive," "barely worth a mention" is well known. Y'all keep debating this though so I dug a little further. From the paper,

In the real world, we usually don’t experience the mathematically well-defined probabilities that rolling dice offers. Instead, we usually deal with “odds” or “likelihoods,” many of which are somewhat subjective. By subjective, we mean the person performing the test must decide for him or herself what constitutes strong evidence, what evidence is positive, and what evidence is supportive but not particularly strong. These are the three relative strengths of evidence summarized above: (1) specific (Bayesian “supportive”), (2) specific and detailed, (Bayesian “positive”) and (3) specific, detailed, and unusual (Bayesian “strong”).
...
We have assigned one of three different likelihood ratios to each correspondence. The specific Bayes factor or likelihood assigned to each correspondence is based on our assessment as to whether the correspondence is (1) specific or “supportive” according to Bayesian nomenclature (0.5); (2) specific and detailed, or Bayesian “positive” (0.10); or (3) specific, detailed, and unusual, or Bayesian “strong” (0.02), as described above and given in the literature.23

There's that terminology again, but they replace "barely worth a mention" with "supportive" which is synonymous in the literature. Following 23 leads to this paper.

https://www.stat.washington.edu/raftery ... ss1995.pdf

Where, low and behold, we find Harrold Jeffreys' K table, but modified slightly.



How did they come up with Likelihood = 50? They arbitrarily decided one hypothesis was "strong" and assigned it a value of 150. They then arbitrarily decided the competition was "barely worth a mention" or "supportive" and assigned it a value of 3. 150 / 3 = 50. Done.

[Guest]

Dog,

I had time last night to sit back and read all the comments I hadn't seen over the last few days as well as look in on the comments at Interpreter. I found the material explaining the K table. It's good you pointed that out because that does need to be considered. I would stick with the point that if they were to sit down, and even working backwards, begin to plug data in as if they were deriving their data, and think about what the numbers could mean (especially if a pattern forms) it might give them some pause, when comparing to real medical examples. It's actually very telling that they found a great justification to circumvent a process that could provide a self-check on their thinking.



Of course, if they were going to compare what they were doing to a real medical example, most telling would be they'd unlikely find a real medical or crime example where 156 data pieces of data were multiplied together. If they were to go into a doctor to test for a disease, and the doctor were to say he wants to run 156 tests, but it's a shame because more data points would be better, they'd turn and run.

Imagine if in the discussion on The Late War we had, we assigned likelihood values to every n-gram hit and then jumped on the multiplication fright train to get a stupendous Tree[3] number. Funny enough, an Interpreter stats guy jumped in on this board back then to tell us that Bayes theorem put constraints on how significant the find was. I thought about hunting that guy down as he seemed sincere, but decided against it as not fair to put his professional credibility at stake on something he knows is wrong but feels obliged to support.

[_Lemmie]

The Dales made a comment about their choices:

To clarify a bit more about the Kaas and Raftery paper, here is a bit more explanation.

Recall that the hypothesis we are testing is that the Book of Mormon is false. We consider three different strengths of evidence in support of that hypothesis.

As given in the paper by Kaas and Raftery, the Bayes factor range (B10) for evidence supporting the hypothesis that is judged “not worth more than a bare mention” is 1 to 3, the Bayes factor range for “positive” evidence in support of the hypothesis is 3 to 20 and the Bayes factor range for “strong” evidence is 20-150.

In the paper, we call these three different strengths of evidence “supportive”, “positive” and “strong” and we use the numerical values 2, 10 and 50 in our calculations to represent an approximate midpoint value of the range for each of the three different strengths of evidence in support of the hypothesis that the Book of Mormon is false.

some key issues:

For the range 1 to 3, which the paper they quote defines as "hardly worth mentioning," they now disingenuously define the midpoint 2 (or 1/2) as "supportive," as Water Dog noted.

Given that 1/5 of the 131 positive correspondences score 1/2, this means not only did they arbitrarily assign at least 26 as "positive," the actual scale from the paper actually defines those as "hardly worth mentioning."

Also, the Dales claim they are using the Kass paper's results. No, they are not:

...and so these categories are not a calibration of the Bayes factor, but rather a rough descriptive statement about standards of evidence in scientific investigation.

So the categories describe the data; this does NOT justify the Dales' paper arbitrarily assigning their particular data a value equal to their approximate midpoints of the categories. The dales did no calculations, but rather simply assigned values to questions to manipulate the result.

And, as always, their gross mis-use of independence to multiply likelihood ratios further invalidates their results.

[_Gadianton]

a thought for the road:

Because of the difficulty in determining whether or not two events are statistically independent, statisticians often take this as the *definition* of statistical independence: two events are statistically independent if the probability of their simultaneous occurrence is the product of the probabilities of the two events taken in isolation.

I think you won’t find very many correspondences that are not independent...But there are scores of other correspondences that definitely are independent

placed sequentially in his "definitely independent list":

1.24 Political power exercised by family dynasties
1.12 Royal or elite marriages for political purposes

I'm really curious what his idea of independence is. Both are common in societies and shudder the thought that family dynasty would arrange a marriage. But okay,

2.2 Active interchange of ideas and things among the elite

So a small chance that a society would have a dynasty that arranges marriage and exchange ideas? okay. switch up the intuition behind independence a little and throw in bees.

6.4 Domesticated bees

so a very small chance that a dynastic society etc. would also keep bees?

3.7 Belief in a resurrection

odds are getting really low for a dynasty to exchange ideas, keep bees, and believe in resurrection.

If there are 7 things 50% likely for any civilization to have, if they are truly independent, to have all of them gets below 1%. You could have a really short paper if you could find really good hits.

is the idea of multiplying independent events the same as the idea of multiplying LRs, because the LR tries to control for dependency?

[_Lemmie]

This goes back to the response the Dales' gave to my post that Chino Blanco posted on the Interpreter site. It surprised me, because the article seemed include this as facts "B":

>>131 true statements from Coe's book, plus

>>18 false statements Coe also commented on.

However, their comment was this:

Hi Mr. Blanco,

Your core criticism here seems to be “For example, elements B could be every statement in the Book of Mormon, both those known as factual, those known as nonfactual, those not known if they are either, etc. Every element needs to be part of the experiment.”


The elements B are every statement of fact in the Book of Mormon text itself about the physical, political, geographical, religious, military, technological, and cultural environment. Every element B is part of the study, we did not exclude any portion of the Book of Mormon text from scrutiny. We did exclude statements made in the chapter headings, the introduction, and in any subsequent commentary as all of those are outside the text itself.

We did not limit ourselves to 131 pieces of evidence, nor did we define them as true. After searching all B we found 149 (not 131 as you repeatedly and mistakenly claim) statements of fact, B, about which there were corresponding statements of fact in The Maya. Of those 149 statements of fact in B for which corresponding statements of fact in The Maya existed we found that 131 of them agreed and 18 did not.

From the article however, it is clear that these 131 +18 statements of fact do NOT constitute every eelement B from the Book of Mormon:

Note: only statements of fact which are dealt with by both books can be rationally admitted to the analysis; on statements of fact where one or the other book is silent, we cannot factually assume either agreement or disagreement. There is no rational scientific basis for doing so.

And also:

.... We then compare 131 separate positive correspondences or points of evidence between the Book of Mormon and Dr. Coe’s book. We also analyze negative points of evidence between the Book of Mormon and The Maya, between the Book of Mormon and a 1973 Dialogue article written by Dr. Coe, and between the Book of Mormon and a series of Mormon Stories podcast interviews given by Dr. Coe to Dr. John Dehlin.

And finally, regarding assuming the 131 statements are true:

Thus we take the statements of fact in The Maya as essentially true, and we compare the “guesses” in the Book of Mormon with these statements of fact. To repeat, for purposes of our Bayesian statistical analysis, we accept the universe of facts summarized by Dr. Coe in The Maya as essentially true.

Then, by definition, the denominator

P(B true, given Book of Mormon non-fiction)

MUST be 1.

There is no consistency in their comments at all.

[_Physics Guy]

The apparent source for the Dales' particular set of allowed likelihood ratios is an interesting detail that might be worth adding. They might defend their use of such a limited range of likelihood ratios as being some kind of standard practice. It could only be a valid standard practice in some limited domain, however, and whatever domain that is, the present topic isn't part of it. That's probably worth saying.

The bias involved in considering only points that are explicitly mentioned in both texts is a subtle point but it may well be big enough that it should get included with the other three. What it means is that errors of omission are not penalized at all, but of course they really should be. A three-hundred-page story purportedly set in the contemporary world, but in which no-one ever used a cell phone or saw an airplane, could only be fiction. So it should be counted against the Book of Mormon—probably quite heavily—that it fails to mention some important aspects of Mayan society. The Book doesn't have to explicitly contradict those aspects of Mayan society to deserve this hit to its odds of being authentic. Simply failing to mention something that it really ought to have mentioned, if it were authentic, should significantly lower its odds of being authentic.

The sin of omission can be committed on both sides, too. A story purportedly set in the 1950's, but in which the whole economy is portrayed as being based on herding longhorn cattle and posting YouTube videos, would also have to be fiction, even though no real recent history will bother to mention the absence of longhorns and YouTube in the 1950's. So features of Nephite society that are prominent in the Book of Mormon but never even mentioned one way or the other in Coe's book—like Protestantism!—should also count heavily against the Book of Mormon being authentic.

Can we come up with a ballpark estimate of how big a factor might be at stake in the final odds, due to errors of omission?