The Interpreter; Bayes Theorem; Nephites and Mayans

[_Lemmie]

From the authors yesterday:

...We have answered the objections ad nauseam. Our statistical methodology is neither flawed nor arbitrary, as some have claimed without evidence. Our Bayesian methodology is based largely on the 1995 Kass and Raftery paper. We use the likelihood ratios given for the midpoint of the ranges of the three principal strengths of evidence that Kass and Raftery cite.

If you have issues with their paper, or the various Bayesian strengths of Bayesian likelihoods that it lays out, take it up with Kass and Raftery or with the many thousands of scientists and statisticians who have cited their paper as authoritative.

If you don’t want to take the Bayesian methodology seriously (or you want to avoid the necessary work—my null hypothesis regarding the motivations of some of the commentators)...

[_honorentheos]

Yeah, the "everyone who disagrees with us must be too lazy to do the simple math" statement was excellent. The reply to Billy whooshed by as such a complete misunderstanding of his point it's difficult to believe it is legitimate rather than they see it and are trying to shift away from the purpose of the paper to just squabble over whether or not wheat existed in Central America but squash wasn't worth mentioning because it was unrelated to the core message of the Book of Mormon. That being Proto Christian Hebrews settled in the Americas among the Maya and not only absorbed but possibly are responsible for much of Mayan culture but didn't intermarry enough to leave DNA evidence, who didn't speak Hebrew or adhere to the law of Moses but did something else no matter what we think the book says, while including hebraism and chiasms written in Egyptian logoglyphic characters but not in Hebrew...wait, what was the claim again?

[_Physics Guy]

For the Dales to defend their work by citing that paper by Kass and Raftery is like defending a plan to go to the moon in a plywood rocket by citing a standard textbook on aircraft design. The textbook is sound and the rocketeers may have copied it closely but it is about vehicles that will stay in the atmosphere and it presumes materials stronger than wood.

My plywood rocket analogy was inspired by a concept of Richard Feynman's but the original might well be better.

Feynman coined the term "Cargo Cult science" by analogy with the South Pacific "Cargo Cults" that flourished for a while after WW2. Some islanders tried to bring back the wartime air traffic by reproducing the wartime conditions as well as they could. They built large bamboo structures in the shapes of radio antennas and radar dishes. Feynman noted that pseudo-scientists often imitate superficial features of science, but fail to appreciate how much more science requires.

If the Dales think that following Kass and Raftery will make their results valid then they are doing Cargo Cult statistics.


EDIT: Googling "Kass Rafferty 1995" (even with Raftery's name wrong like that) turns up a review article in the Journal of the American Statistical Association which I presume is the paper Dale and Dale cite. It has over 12,000 citations so it would indeed seem to be an authoritative reference—for what it actually says. Its abstract seems to indicate a relatively narrow focus, however. In particular the abstract says that the paper will consider contexts of "genetics, sports, ecology, sociology, and psychology". That seems unlikely to include assessments of the historical accuracy of the Book of Mormon.

I got a free copy of the article from JSTOR, so I'll skim through it to see if I can spot any clear statements of basic requirements that the Dales violate. If Kass and Raftery 1995 is these guys' Bayesian Bible then we may be able to quote it clearly against them. On the other hand descending to authority quotes in a case of gaping logical fallacies is really going all-in to mudwrestle the pig. People who are capable of recognizing basic errors like the ones on which this paper is based don't write papers like this one in the first place.

[_Physics Guy]

Skimming through Kass and Raftery 1995 turned out to be easier but less helpful than I thought. I find two points.

1)
The association of qualitative terms like "not worth more than a bare mention", "substantial", "strong", etc. for specific ranges of likelihood ratio is indeed discussed in section 3.2 of KR95. This association is NOT used, however, as a rule for assigning quantitative likelihood ratios to subjective assessments of evidence. It's exactly the reverse: KR95 offers the qualitative terms as interpretations that can fairly be made of numerical values in various ranges. There is no suggestion whatever that Bayesian likelihood ratios should ever be restricted to a small set of specific values representing different subjective assessments of evidence strength. That is absolutely turning KR95 on its head. Furthermore KR95 includes the qualitative categories "decisive" or "very strong" for likelihood ratios greater than 100 or 150—with no upper limit. This clearly implies that Bayesian likelihood ratios CANNOT be capped in advance at 50.

2)
If the Dales are trying to deflect criticism by citing KR95 then they are simply trying to snow their critics, because KR95 is twenty pages long and only a single column of it deals with anything remotely relevant to the Dales' paper. That single column is the section 3.2 that I mentioned in 1), which invalidates the Dales' basic method of assigning likelihood ratios based on qualitative assessments of evidence. All the rest of KR95 is far more technical. As in, about half of it concerns techniques for numerical integration in higher dimensions. The Dales saying their method is based on KR95 is like first graders claiming that they did their subtraction homework according to Gödel's paper on undecidability. Yeah, nice try, guys.

[_Lemmie]

The authors have mentioned the Kass and Raftery paper a couple of times. Earlier i quoted one of their comments, here is the relevant part:

To clarify a bit more about the Kaas and Raftery paper, here is a bit more explanation...

Recall that the hypothesis we are testing is that the Book of Mormon is false. We consider three different strengths of evidence in support of that hypothesis...

In the paper, we call these three different strengths of evidence “supportive”, “positive” and “strong” and we use the numerical values 2, 10 and 50 in our calculations...

Note that they describe their elements as "THESE three different strengths," based on their claim they are using the Kass paper's results. But again from the Kass paper:

...and so these categories are not a calibration of the Bayes factor, but rather a rough descriptive statement about standards of evidence in scientific investigation.

So the categories describe the standards of evidence; this does NOT justify the Dales' paper arbitrarily assigning their particular data to a category, nor does it justify giving cardinal values to their self-defined ordinal list.

[_Physics Guy]

Exactly. So I guess we can condemn them out of their own scripture on at least this one topic.

And they can be nailed for their response of trying to stand on Kass and Raftery. That was a blatant attempt to snow their critics by citing a long, technical paper that is 97.5% irrelevant to their work, only touches their work at one place—and at that one place contradicts them.

[_Gadianton]

Exactly. So I guess we can condemn them out of their own scripture on at least this one topic.

No because you clearly didn't understand the scripture. Go back and read it again without anti-Mormon bias. I'm sure Kass and Raftery would be thrilled to learn what their work has led to.

[_Physics Guy]

In this case there really is only half a page from Kass and Raftery that these guys could conceivably cite, and it's non-technical enough that there's not much room for blustering about how to interpret it. For the rest of KR95, well, I'd like to see either Dale try to explain how appreciating their work depends on understanding the merits of adaptive Gaussian quadrature.

But okay, they can always retreat to debating the meaning of "is". I don't hold real hope that either Dale will ever see the light here. It might be possible to make clear to some of their readers how totally wrong they are, though. The fields are white for harvest, as it were. Something like that anyway.

I'm sure Kass and Raftery would be thrilled to learn what their work has led to.

That's a thought. There's an outside chance that one of them might be willing to comment. Since the chance can't be less than 2% no matter what, it could be worth a try.

They are both still alive and apparently active, at CMU and UW, respectively. Now that I have the idea I'm faltering, however. What else could they do, on the remote chance that they wanted to do anything, besides reach down to crush some poor little incompetent buggers with a comment? If I were in their shoes I'd just feel bad about that. This topic is not important enough to shatter anyone's ego over it.

[_Dr Exiled]

What a crap show this turned out to be. Clearly, this wasn't the revolutionary paper DCP claimed. It's just empty testimony dressed up in a bayesian dark suit with a white shirt and tie.

[_Lemmie]

In this case there really is only half a page from Kass and Raftery that these guys could conceivably cite, and it's non-technical enough that there's not much room for blustering about how to interpret it....

And it's the same Kass and Raftery citation, page and section that every article I've seen that discusses the use of bayes factors scaling in diagnostic testing uses, so I really doubt they are doing anything but copying the source from one of those.

Those articles all discuss n sample sizes, where the results are used to determine the probabilities in the likelihood ratios, which are THEN compared to the scale.

The Dales, on thre other hand, just take a sample of ONE, pick a part of the scale they want the LR to match based on their subjective feelings about how meaningful it is, and then assign it that number. Not even the underlying probabilities, just the ratio. Unreal.