The Interpreter; Bayes Theorem; Nephites and Mayans

[_honorentheos]

Brant is apparently the moderator for The Interpreter. He emailed me to clarify a request made regarding a comment. I believe what Arc described is that a message addressed to Dr. Bruce Dale was forwarded through Brant Gardner on to Bruce. That's how I read it, anyway.

That said, I don't get the impression Brant is thrilled with the paper or it's methodology. He's made it clear he thinks the use of statistics didn't add to the argument for a Mesoamerican setting, preferring his books as the more appropriate approach. He may have said something behind closed doors not too different from what Arc suggests was the message passed along.

[_Arc]

As suggested to Dr. Bruce Dale in a PM via Brant Gardner, the paper should be formally retracted, in writing, and taken down from the Interpreter site. The Interpreter could take the extraordinary step of also removing the comments. The URL should be left with the retraction alone.

The paper cannot be salvaged as a scholarly work any more than a paper describing a flat Earth or a perpetual motion machine could be salvaged.

WOW!

Brant Gardner sent a PM to Dr. Bruce Dale requesting the paper should be formally retracted in writing and taken down from the Interpreter site?!

My respect for Brant Gardner was already high, but now it is through the roof! It's reassuring to know there are still apologists out there like Brant who place some value on academic honesty and integrity.

This will not sit well with Priestcraft Peter$on.

Again, the PM was sent via Brant Gardner. Brant was clearly very professional in handling the request, especially in seeking clarification/ confirmation from a third party (Honorentheos). However, Brant was not the author of the PM. The PM did not mince words. Honorentheos accurately described the process.

[_Gadianton]

They would never pull this paper. Their best bet is to let it die, silently lock the comments, and go on as if it didn't happen, as they do with most of their other papers. Pretty much everything published there is a one-off topic that falls down the memory hole anyway.

Because there are so many fully objective problems with the paper and because so much has gone into defending the paper by the Interpreter staff, insisting anyone critical of it doesn't understand it, insisting the sky is purple when everyone knows it's blue, and not giving an inch on any one of the dozens of factual and methodological problems, this paper may very well be to date the most glaring single piece of evidence of the extreme confirmation bias and institutional group-think in the history of Mopologetics. When Philip Jenkins asked, "what is the piece of evidence for Bigfoot so significant, that it would urge me to take the Bigfoot hypothesis seriously?" -- had he asked instead about the piece of evidence for the group-think of Mopologetics, this paper would serve as that representative initial piece of evidence. I'm thinking of showing up to the next FAIR conference wearing a t-shirt with a 2.89 x 10^-152 printed on it.

Dr. Peterson's "Dr. Jekyll / Mr. Hyde" hypothesis, that scholars who do respectable work in their profession surely do not just turn off their brains when they take up Mopologetics has been utterly destroyed by this incident and showed it's not just the authors, but the whirlwind-like self-reinforcement at an institutional level. The only way it could be worse, is if they pulled the paper. While certainly, on the one hand, it would show that Interpreter is working harder to be responsible, it would also be tantamount to admitting the entire foundation of Mopologetics up to this point in time has been a sham.

[Guest]

Something about Lemmie's find has been gnawing at me, and I wanted to raise it as a question.

In the quote Lemmie found, the commentor named Bruce said,

If a true positive is much more likely than a false positive, the posterior odds increases relative to the prior odds. If a true positive is about as likely as a false positive, the posterior odds stays about the same as the prior odds.

...In my opinion, sensitivity is decreased and specificity is increased by at least two features of the Johnsons’ study:

1. the massive search model tends to produce false positives.

2. the dependence of weights on a randomly selected corpus (from books of many genres between 1500 and 1830) tends to affect sensitivity and specificity in unpredictable ways; I can conceive of ways in which sensitivity is decreased and specificity is increased.

I've been puzzling over his comment since reading it and am more confused than when I started in trying to guess at why he came to this conclusion? My understanding is when he says, "sensitivity is decreased and specificity is increased" he is saying that the two items he notes lead to fewer true hits and more false hits. And he said this in reference to 1) the Johnsons having used an algorithm to search over 100,000 books for unusual 4 word phrases (4-grams) used in the Book of Mormon and determined to be rare based on a frequency analysis of a 5,000 book sample. And 2), the weights that Bruce the Commentor takes issue with seem to be in reference to their having "Tak(en) the inverse baseline frequency (e.g. if the phrase "having been born of" shows up 6 times in all of the books sample for the baseline, then the inverse baseline frequency would be 1/6=0.167.)" and summing the results divided by the total work count of a book to reduce the influence of a longer book outscoring as shorter book. The result being a score that reflected both the number and rarity of shared 4-grams between a book and the Book of Mormon.

It seems to me Bruce the Commentor believes using as large a sample of books as possible to identify unusual phrases, and then to weight those that have the least frequent use over those found used more frequently should result in many more false hits and fewer positive hits. Instinctively, if this were true it would imply that the study should have identified numerous books that were candidates for influencing Smith's writing of the Book of Mormon.

But that isn't what the Johnsons experienced as I recall. The Late War was a surprise "hit" that they had never heard of, but the statistical model they used helped it be identified as a close match to the language of the Book of Mormon among tens of thousands of other books written prior to the publication of the Book of Mormon because the number of unusual phrases that matched was such an anomaly it couldn't be missed among the tens of thousands of books included in the analysis.

If my understanding is correct, what is he even trying to say?

[_honorentheos]

I'm thinking of showing up to the next FAIR conference wearing a t-shirt with a 2.89 x 10^-152 printed on it.

Damn.

I genuinely laughed at that. Brilliant.

[_Gadianton]

the Johnsons having used an algorithm to search over 100,000 books for unusual 4 word phrases

is that what they did? I thought this study was as barebones as it gets. Start with first nephi, and take the first four words, then shift up 1 word and take the next four words. I don't think they did any statistical analysis whatsoever, something taking the find to the next level would require, but we mere mortals don't dare do it ourselves because we know more than one bad trade has been made on a killer candlestick chart.

I don't know what the best test is, but certainly something like the likelihood test would seem to make sense -- there were 3 books if I recall that all had pretty impressive hits. The runner up -- something about napoleon -- was the book they were thinking would win. So yes, that the Late War crushed it was an unexpected result from doing an actual, real study with actual, real data; something the authors of the Interpreter paper didn't even come close to doing.

What would be interesting, for starters, is when controlling for rarity -- however it is you do that -- would the Late War really win over the Napoleon book? But from there, we still have an impressive correlation (or rather we've found the single most correlated book, if the most correlated book is really most correlated with statistical significance), but what does it actually mean?

I have a hard time believing it's meaningless; I would put my money it does tell us something about the "Bible writing" culture the Book of Mormon comes from. And more than just the n-grams, the facile one-dimensional characters and plots where the good people can do no wrong and the bad people can do no right is just as entertaining to think about. But did the author go so far as to literally plagiarize the Late War? and if so, why not also the Napoleon book because those hits were very impressive also, and it was much shorter (If I recall correctly)?

[_Lemmie]

Something about Lemmie's find has been gnawing at me, and I wanted to raise it as a question.

In the quote Lemmie found, the commentor named Bruce said,

If a true positive is much more likely than a false positive, the posterior odds increases relative to the prior odds. If a true positive is about as likely as a false positive, the posterior odds stays about the same as the prior odds.

...In my opinion, sensitivity is decreased and specificity is increased by at least two features of the Johnsons’ study:

1. the massive search model tends to produce false positives.

2. the dependence of weights on a randomly selected corpus (from books of many genres between 1500 and 1830) tends to affect sensitivity and specificity in unpredictable ways; I can conceive of ways in which sensitivity is decreased and specificity is increased.

I've been puzzling over his comment since reading it and am more confused than when I started in trying to guess at why he came to this conclusion? My understanding is when he says, "sensitivity is decreased and specificity is increased" he is saying that the two items he notes lead to fewer true hits and more false hits. And he said this in reference to 1) the Johnsons having used an algorithm to search over 100,000 books for unusual 4 word phrases (4-grams) used in the Book of Mormon and determined to be rare based on a frequency analysis of a 5,000 book sample. And 2), the weights that Bruce the Commentor takes issue with seem to be in reference to their having "Tak(en) the inverse baseline frequency (e.g. if the phrase "having been born of" shows up 6 times in all of the books sample for the baseline, then the inverse baseline frequency would be 1/6=0.167.)" and summing the results divided by the total work count of a book to reduce the influence of a longer book outscoring as shorter book. The result being a score that reflected both the number and rarity of shared 4-grams between a book and the Book of Mormon.

It seems to me Bruce the Commentor believes using as large a sample of books as possible to identify unusual phrases, and then to weight those that have the least frequent use over those found used more frequently should result in many more false hits and fewer positive hits. Instinctively, if this were true it would imply that the study should have identified numerous books that were candidates for influencing Smith's writing of the Book of Mormon.

But that isn't what the Johnsons experienced as I recall. The Late War was a surprise "hit" that they had never heard of, but the statistical model they used helped it be identified as a close match to the language of the Book of Mormon among tens of thousands of other books written prior to the publication of the Book of Mormon because the number of unusual phrases that matched was such an anomaly it couldn't be missed among the tens of thousands of books included in the analysis.

If my understanding is correct, what is he even trying to say?

No time so I'll expand on this later, but I read that as Bruce the Commenter focusing more on his second point, that there is a way to get around issues:

2. the dependence of weights on a randomly selected corpus (from books of many genres between 1500 and 1830) tends to affect sensitivity and specificity in unpredictable ways; I can conceive of ways in which sensitivity is decreased and specificity is increased.

[Guest]

the Johnsons having used an algorithm to search over 100,000 books for unusual 4 word phrases

is that what they did? I thought this study was as barebones as it gets. Start with first nephi, and take the first four words, then shift up 1 word and take the next four words. I don't think they did any statistical analysis whatsoever, something taking the find to the next level would require, but we mere mortals don't dare do it ourselves because we know more than one bad trade has been made on a killer candlestick chart.

It's been a long while since I read their website, but what I recall was they used 5,000 books randomly selected from the larger sample to first identify rare 4-grams. What you describe may be how they created the initial list of 4-grams but I don't recall reading that so I can say. But what I described above was taken from the summary Duane Johnson shared of their approach. There was a threshold that divided rare vs. common 4-grams, they eliminated 4-grams common to the KJV of the Bible for what seem like obvious reasons, and assigned weights that were inverse to the frequency of the 4-gram in their 5,000 book sample.

I have a hard time believing it's meaningless; I would put my money it does tell us something about the "Bible writing" culture the Book of Mormon comes from. And more than just the n-grams, the facile one-dimensional characters and plots where the good people can do no wrong and the bad people can do no right is just as entertaining to think about. But did the author go so far as to literally plagiarize the Late War? and if so, why not also the Napoleon book because those hits were very impressive also, and it was much shorter (If I recall correctly)?

This graphic was shared by Duane Johnson in the thread I linked to earlier, around page 29 or so, and shows the relationship between The Late War, other LDS scripture, and other high scoring potential candidates. It helps illustrate just how much of an anomaly it was:



My personal favorite post in the entire original thread about the Late War, and most memorable to me, was made by EA regarding the fairy photo hoaxes from the early 1900s. I went back and tracked it down to be quoted below. Of course it was towards the end of a 76 page thread:

Is everyone here familiar with the Cottingley Fairies photos? It was a series of 5 photos taken in 1917 in England.

This is the most famous one:



It has influenced depictions of fairies even to this day.

Quaint as it might seem now, there was a great deal of controversy over them with many people, including Sir Arthur Conan Doyle, taking them as evidence of spiritual beings. There were experts in photography who declared them as legitimate.

Interestingly, I once saw a show on the Cottingley Fairies filmed in the 1970's that covered this as a controversy over the existence of spirit beings. Skeptics were brought on to offer a variety of theories for how the photographs were produced. All of them involved some relatively sophisticated photographic techniques for girls in 1917 to be using or remarkable coincidences. The show, being fundamentally interested in portraying this as a supernatural controversy, rightly cast doubt on accepting any of those theories as correct. No one explanation seemed in particular likely.

Then something marvelous happened. In the 1980's, the girls who produced the photos admitted it was a hoax. They also described how they did it. It turns out they cut out pictures of fairies from a book, stuck them to hatpins, and took pictures of them. That's it. That's what they did. So much for acid etched engravings and complicated exposures.

This story has long stuck with me for two reasons. First, whenever I see complicated and remote explanations for unusual phenomena and potential hoaxes, I'm always reminded that the reality can be devastatingly more simple. Second, while everyone was right to reject those complicated theories for how the photographs were produced, it's always fascinated me that people lost sight of the fact that even though those theories were unlikely, the explanation that entailed the photos were of actual fairies was vastly, vastly more unlikely than that. You can't prove extraordinary supernatural claims simply by attacking somewhat unlikely natural explanations.

This story does inform what I see in Book of Mormon debates. I personally am skeptical of theories of authorship that do not involve Smith. Elaborate plagiarism hypotheses have always struck me as strained. And while I find myself on the same side as believers when seeing this, I also see them as having a huge blind-spot for not appreciating just how much more implausible the supernatural tall-tale version of events is than the authorship theories they are finding without sufficient basis.

What I've highlighted in blue is one of those fundamental points that transcends the topic of any thread.

I was never a proponent of the claim The Late War showed plagiarism. Fortunately, I'm on record in that thread with my view being it was more useful as showing the claims for unique ancient, Semitic linguistic features were absolutely available to Smith in the 19thC and as such, put to rest the whole, "How could Joseph have known?" debate. I'm minimally concerned with the details of how Smith composed the Book of Mormon or if there were others involved, though I feel the fact the pace and content seemed to be affected by the arrival of Oliver Cowdery suggests he had a role to play if the details may be lost to time. Whatever the source, the facts are such I'm more than comfortable believing the Book of Mormon is a product of the 19thC US frontier, period. That seems to be able to resist my internal urge towards skeptism's prodding and poking.

[_Gadianton]

There was a threshold that divided rare vs. common 4-grams, they eliminated 4-grams common to the KJV of the Bible for what seem like obvious reasons, and assigned weights that were inverse to the frequency of the 4-gram in their 5,000 book sample.

got it, thanks h.

[_honorentheos]

No time so I'll expand on this later, but I read that as Bruce the Commenter focusing more on his second point, that there is a way to get around issues:

2. the dependence of weights on a randomly selected corpus (from books of many genres between 1500 and 1830) tends to affect sensitivity and specificity in unpredictable ways; I can conceive of ways in which sensitivity is decreased and specificity is increased.

Thanks, Lemmie. I look forward to your thoughts, and hope you're enjoying the extended weekend rather than spending time on it.

I try to apply the principle of greatest charity to an argument with which I may disagree, and that requires taking the time to see the argument from the inside as the person making it is doing so. This one alludes me, though, as I can't do so in good faith. My closest attempt requires assuming Bruce the Commenter is looking at the results of the Johnson's analysis of The Late War, assuming there is no valid source outside of the Bible and biblically influenced books, and thinking that it's inevitable that the Johnsons would have found hits if they looked at 100,000 books so it's meaningless. The Late War may be the strongest hit, but that's only because one of the inevitable numerous false positives would have the highest score. It is otherwise meaningless to the Book of Mormon or environment in which is came out of in 1829/30.

In other words, the most charitable explanation for the comment I can come up with is there had to be a hit that was closer to the Book of Mormon than any other because that's how numbers work, and by looking at a lot of books that increased the cluster of "hits" among which the closest one is situated so the possibility one of them would score fairly high was plausible but more an artifact of the methodology than that the methodology resulted in sifting out something unusual from the otherwise ocean of books being analyzed.

But it just doesn't make sense given that TLW isn't floating slightly above a bunch of other "false positives". It was a largely unknown book that the analysis uncovered. There aren't a number of other "false positives" clustered just below it, and as long as it was in the sample of books selected it wouldn't have affected it's relative position to the Book of Mormon if the sample had been 5, 5000, or 5 million books.

But perhaps that's what I'm not really taking into account - he didn't look at the results in detail, and made assumptions based on an arm chair thought experiment on how the results could arise given the Book of Mormon is in fact ancient and historical according to his prior beliefs. Perhaps the flaw is entirely with my attempt to be charitable in this case.