Response to Jockers, Criddle, et al., Now Available

[_MCB]

There is remarkably good agreement for attribution
of Book of Mormon chapters between open-set
NSC and Burrows’ delta method with its threshold
for eliminating false positives. However, open-set
NSC has advantages in that it produces posterior
authorship probabilities rather than probability
rankings, it can be modified to deal with heterogeneous
sizes of test texts, and its open-set capabilities
are based on theory rather than simply empirical
evidence. When using open-set NSC, or any authorship
attribution procedure, results need to be interpreted
in light of the stylistic features used, the
genres of the training texts, and other subtle properties
of the problem. Even so, the method has great
potential for future careful use in authorship
attribution.

[_harmony]

After reading and the article, are we allowed to quote any portion of it in a discussion?

Glenn

Just use appropriate cite.

[_Nomad]

I had no problem downloading the paper and saving a copy of it to my local hard drive.

I also read the whole thing. The authors left no stone unturned in their impressive rebuttal of the grasping-at-straws Jockers/Criddle study. Imagine then my surprise to read MCB’s comment on this thread:

… the jury is still out.

No, MCB, the jury has long since left the building. Oxford’s Journal of Literary and Linguistic Computing has, with the publication of this article, essentially admitted that their previous publication of the Jockers/Criddle study was an error in judgment and an example of the failure of adequate peer review. The methodology of the Jockers/Criddle study has been shown to be fatally flawed. The conclusions of the Jockers/Criddle study have been shown to be fatally flawed. To borrow a phrase from Greg Smith, the brief resurrection of the Spalding/Rigdon theory of Book of Mormon production has been re-entombed beneath Mt. Doom, hopefully never to rise again.

[_MCB]

Scholarly debate does not work like that. There will be a response, addressing those issues.

[_mikwut]

This is certainly a devastating critique. Critics coming from a hopeful Spalding construction should reassess that hope it doesn't do the believer (who could be wrong and shown the error of his way) any good, and it doesn't do the critic any good.

my regards, mikwut

[_Ray A]

As test data, we
calculated the same features for the 51 Federalist
papers authored by Alexander Hamilton. We then
naïvely used the closed-set NSC procedure to calculate
posterior probabilities and classifications for the
Hamilton texts (as if they were anonymous). The
130 literary features included relative frequencies
of 93 non-contextual words, 35 word-pattern
ratios (Hilton 1990), and 2 vocabulary richness
measures (Holmes 1992).
Early or late Rigdon was falsely chosen as the
author of 28 of the 51 Hamilton texts with inflated
posterior probabilities ranging as high as 0.9999
(Fig. 2). Pratt was falsely chosen as the author of
12 of the papers, and Cowdery was falsely chosen as
the author of the remaining 11 papers. These results
dramatically demonstrate the danger of misapplying
closed-set NSC.

I think the supporters of Spalding/Rigdon will have to address this one.

[_Benjamin McGuire]

MCB writes:

No--- the jury is still out. Based on the data that your team analyzed, the initial Jockers et al study is not conclusive.

But, we might also quote this MCB:

However, there actually is strong disagreement between the closed-set NSC results and the delta results. This is because delta-z scores should not be taken seriously unless they are very small (i.e. very negative). Burrows (2003) found that a threshold of -1.9 separated most false positives from true attributions for a set of 17th-century poets. Jockers et al. (2008) failed to do this. In the Jockers et al. (2008) study, only 16 of the 239 chapters had delta-z values as small as -1.9 (Fig. 9). Ten of these 16 chapters were essentially verbatim quotations of Isaiah/Malachi, and all 10 were correctly attributed to Isaiah/Malachi. Four additional chapters were attributed to Isaiah/Malachi and the others to Rigdon and Spalding. The remaining 223 chapters had large delta-z values and were thus apparently false positive. Hence, the delta results of Jockers et al. (2008) actually say little more than what is already uncontroversial about Book of Mormon authorship: that some of the chapters are quotations of Isaiah and Malachi. The delta-z results do not, in fact, attribute sizeable precentages of the chapters to Rigdon, Spalding, or Cowdery.

And this -

We then carried out the extended open-set NSC procedure. Of the 239 Book of Mormon chapters, 175 (73.2%) were classified to one or more unobserved authors, 35 (14.6%) were classified to Isaiah, 17 (7.1%) were classified to early or late Rigdon, 8 (3.4%) were classified to Smith, 2 (0.8%) were classified to Cowdery.

Now let me explain this in a way that might be easier for you to understand. The process involved adding in a factor that would determine the likelihood that an author not in the list of test cases is the real author. As the first quote points out, of the chapters analyzed in the original study, nearly all the chapters were presented as having been given a relatively confident attribution. In fact, this studies shows that only 16 of the chapters could arguably be shown to provide a reliable attribution - and 14 of those were attributed to Isaiah/Malachi. This does not produce confidence.

Further, a reasonable "distance" was used to distinguish between an unknown author and real authors. When tested in this way, NO chapters of the Book of Mormon were attributed to Spalding. Not a single one. If I believed as you seem to that Spalding was involved in some way in the authorship of the Book of Mormon, then I would think I would find some trace of Spalding on a significant statistical level - but here we don't. And in fact, if we go back to the Jockers study, the implication is that Spalding was the best match for those chapters associated with him. That means that all of the other potential authors were even farther away from the Book of Mormon text for those chapters than Spalding was - and Spalding wasn't even close.

The only way to plausibly suggest that there are chapters written by Spalding is to suggest that Spalding wrote in another style so disparate from the one in which he wrote the extent manuscript, that the two texts could not reasonably be analyzed stylometrically with a result that they were by the same author. And if we were to suggest something like this, then we might as well throw all of the stylometric analysis out the window (since such analysis is predicated on the idea that authors do not generally produce works with such differences). A rather self-defeating premise for those holding to the Jockers study as evidence.

And this study made it through the same peer review process as the earlier one ...

And of course, one of the more interesting features is that they applied the closed and open methods to the Hamilton papers, using the proposed authors - Joseph Smith, early Sidney Rigdon, late Sidney Rigdon, Solomon Spalding, Oliver Cowdery, and Parley Pratt. The results? of the 51 Hamilton papers, the closed NSC method (Jockers method) assigned Rigdon as the author of 28 of them (with probabilities ranging as high as 0.9999), Pratt became the author of 12, and Cowdery got the other 11. When the open NSC method was used, 2 of them were assigned to Rigdon, and the other 49 were assigned to an unknown author.

No matter how you slice the bread, the Jockers method is fatally flawed. And no amount of adding other possibilities or better controls will somehow make the Book of Mormon text come closer to Spalding or Rigdon. Adding more authors won't somehow make the language more similar.

Ben McGuire

[_Benjamin McGuire]

I want to add a couple of comments.

The approach that Jockers and company took was, I think, quite valuable. It was something new - used some good tools, and so on. It had some flaws. This new paper does not destroy the Jockers methodology. It corrects it. It still uses most of the framework that the Jockers study erected. What it does is show that the flaws in the methodology resulted in bad conclusions in this particular test case. When the Jockers study has the actual author in the mix, it does very, very well (just as the open NSC method proposed in the paper under the same circumstances). When the author is not in the mix, it does very poorly - and there was no mechanism to determine whether or not the author was in the mix. What this paper does is introduce a mechanism to do just that - and in doing so allows the process to spit back a result that says that all of the potential authors are dissimilar enough to the test material to suggest that none of them are good candidates for authorship.

[_bschaalje]

Scholarly debate does not work like that. There will be a response, addressing those issues.

I hope there will be a response. As Ben mentioned, the Jockers/Criddle paper did a lot of good. For me, itreawakened my dormant interest in statistical authorship attribution and generated some really interesting and important questions. I think the scholarly debate has been healthy, leading to a lot of new insights. And I hope that there will be a response.
On the other board, Dale Broadhurst said

I think that it would be fair to say that the three Stanford researchers DO NOT have an argument with Dr. Schaalje, on the issue of a "closed author set" in relation to an "open author set." The latter is clearly that a limitation of the 2008 Stanford study (and of Dr. Jockers' more recent study, which included the word-print of Joseph Smith, Jr.).

If that is truly the case I am elated. In my first on-line discussions with Craig Criddle, he said that he had not addressed the idea that his method ‘forced’ a choice among his potentially poor candidates because he didn’t think there was any merit in the idea. In my last discussion with him, he simply stated that the ‘Jockers et al. study is what it is,’ and said he would have nothing more to say to me on the subject. I would love to see a paper from the Stanford group that uses open-set classification, whatever the conclusions of the study are.
I do agree that the current paper is not the end of work in this area. Your comments in this board are helpful in charting new directions.

[_CaliforniaKid]

When the Jockers study has the actual author in the mix, it does very, very well (just as the open NSC method proposed in the paper under the same circumstances).

I'm not yet persuaded of that. Cross-genre tests still seem to be an intractable problem for the method.