I just came across this post from the aptly named Recovery from Mormonism board reference FARMS and its "peer review" process:
Added by elee Aug 2004:
http://en.wikipedia.org/wiki/Peer_review
And some tasty excerpts:
A rationale for peer review is that it is rare for an individual author or research team to spot every mistake or flaw in a complicated piece of work. This is not because deficiencies represent needles in a haystack, but because in a new and perhaps eclectic intellectual product, an opportunity for improvement may stand out only to someone with special expertise or experience. Therefore showing work to others increases the probability that weaknesses will be identified, and with advice and encouragement, fixed. The anonymity and independence of reviewers is intended to foster unvarnished criticism and discourage cronyism in funding and publication decisions.
...
Typically referees are not selected from among the authors' close colleagues, relatives, or friends. Referees are supposed to inform the editor of any conflict of interests that might arise. Journals or individual editors often invite a manuscript's authors to name people whom they consider qualified to referee their work. Authors are sometimes also invited to name natural candidates who should be disqualified, in which case they may be asked to provide justification (typically expressed in terms of conflict of interest).
...
I wanted to give my input on the Daniel Peterson’s claim that the FARMS review process is as rigorous as that of mainline academic journals.
In his very sarcastic and condescending response to the original RfM thread on this topic, Peterson attempts to buttress his arguments by stating his bonifides. So in the same spirit, please allow me to state my bonifides, so that you might make appropriate comparisons between his observations and mine. I worked for several years in academics. I have published dozens of articles in peer-reviewed scientific journals, I have reviewed dozens of articles on behalf of several peer-reviewed journals, and I was editor of a peer-reviewed journal for several years. (One might not ever guess I was an editor judging by the sloppy spelling and grammar in my posts; however, when I write posts, I write very fast, and I never proof read—I don’t have the time. As an academic, I was an obsessive proofreader, typically taking my manuscripts through well over 15 drafts before submitting them for publication.)
So, with this in mind, here’s my response to Peterson.
What Peterson describes is not “peer” review, it is “editor” review. The peer review process is anonymous. An editor sends out a manuscript to, typically, 1-3 “blind” reviewers. The reviewers do not know whose manuscript they are reviewing, and the author does not know the identity of the reviewers. This is done expressly for the purpose of ensuring objectivity in the review, reflecting the very reasonable concern that knowledge of the identity of the author might compromise the objectivity of the review, plus it protects the reviewer from retaliation by the author, again helping to ensure greater objectivity. The system is not perfect. Frequently reviewers can guess who the author is (particularly if it is a narrow field or subfield) and the author can guess the reviewers. There is also a good ol’ boy system that ensures that established scholars get easier treatment than Assistant Professors who lack reputations. (This is similar to the NBA, in which, say, Greg Ostertag gets called for traveling while Shaquille O’Neal almost never does, regardless of how blatantly he actually travels.) But, all in all, the system works reasonably well.
The above is distinguished from “editor” review, in which the manuscript is reviewed by the editorial board. In an editor review, there is no pretence of anonymity, and the standards in editor reviewed journals tend to be significantly lower than peer reviewed journals. In a top tier research university, editor reviewed publications count almost zip, and in some cases less than zip, towards tenure and promotion, precisely because they are known to have lower standards, generally speaking, than peer reviewed publications. In my case, I might have had 20 editor reviewed publications when I came up for tenure, and I still would have been denied tenure. (As it was, I had several publications, many in top rated journals, so I earned tenure.) So, as rigorous as Peterson claims his review process is, if the same rigorous process were used by other editor reviewed journals, it still wouldn’t matter worth crap to a top tier research university. Why? Because what matters is that manuscripts be OBJECTIVELY reviewed according to rigorous standards, but also rigorous standards applied by PEERS, who are presumed to be the foremost experts on the “state of the art” in the discipline.
Peterson also proudly points to the rigorous proofing of texts and checking of citations. What Peterson describes is “copy editing” and “source editing.” These are editorial functions, not review functions. Few reviewers take the time to nitpick over spelling and grammar (unless really poor) but focus more on issues such as the soundness of theoretical constructs, methodology, interpretation, and conclusions. It is the editor’s job to do the copy and source editing. Yet in my opinion these functions, while important, are subsidiary to the peer review, which focuses on substantive issues. Peterson can rightly be proud about the rigor of his copy and source editing, but this is a Red Herring, it has little to do with whether the conclusions, methodology, or theoretical framework, of the manuscript is any good.
In a post on the topic, Brian B. quoted something from an online source of the peer review process. If I remember correctly, the gist of the quote was the peer review is inherently conservative and stifles innovative thinking or challenges to orthodoxy. In my experience, this is a gross overstatement. True, there is at times a tendency for reviewers to be resistant to new arguments and evidence that challenge received wisdom, but this fails to explain the often radical evolution in theory that one finds over time in virtually every academic field. Take economics for example. Decades ago, Keynesian economics dominated academics; today Keynesianism is an anachronism having been succeeded by monetarism and several other “isms” in their time. There has been significant change in organization and behavioral theory over time. In the social sciences and humanities, Post Modernism, Feminist Critique, and several other challenges to the orthodoxy have arisen, gained substantial credibility and followings, and are now being challenged by other theories. In international development it seems there is a new theory of underdeveloped that gains precedence every few years only to fade out after a while to be replaced by another theory. In my case, I wrote an article that challenged a predominant theoretical framework in my own field—the framework made famous at the school where I earned my Ph.D.—and my article was published by the #1 journal in the field. In sum, I see little evidence that the peer review process has stymied innovation and new ideas in academics. The competitive marketplace of ideas is alive and well in academics.
What Peterson avoids mentioning, and this is in my opinion the central point, is how FARMS publications would be evaluated in a true peer review, that is anonymous, objective reviewers who are experts in their fields, and who do not have a vested interest in proving Mormonism to be true. He fails to answer the most fundamental criticism of FARMS research—that it is not truly peer reviewed. I think we all know the answer why it is not. Any article submitted to a peer-reviewed scientific journal that posited a civilization numbered in the millions that live in MesoAmerica, worshiped Jesus Christ, wrote in Reformed Egyptian, drove in Chariots, wielding steel swords, rode horses, domesticated oxen, etc. would be summarily rejected by any competent, knowledgeable peer. Any article arguing that an ordinary funerary text contains writings by an ancient prophet of God (whom scholars doubt existed anyway) would be summarily rejected by any competent, knowledgeable peer. Few FARMS publications would survive a true peer review process, regardless of how carefully and well argued, because they reside within a totally invalid theoretical or empirical framework. One can craft the most tightly reasoned defense of the Book of Mormon, with every single conclusion following logically from the underlying assumptions, impeccably copy and source edited, and it would still be rejected summarily by a true peer, because the foundational assumptions have no basis in known reality. FARMS, and those who write in its employ, would quickly become a laughing stock in the field. No wonder they do not risk the rigors of true peer review.
One final comment. The nature of the FARMS review process guarantees no real innovations in learning, because it holds as inviolable the foundational assumptions underlying the research—that the Mormon church is, ex ante and prima facie, and therefore so are the Book of Mormon, Book of Abraham, etc., etc. There can be innovations within this framework, (e.g., limited geography theory), but the framework itself cannot successfully be challenged, as happens all the time in academic research submitted through a true peer review process. FARMS engages in counterfeit scholarship; counterfeit in that the conclusions are predetermined. It is one massive exercise in circular reasoning, where every argument, every bit of evidence circles back around to support the foundational assumptions. If there were such thing as a truth in labeling law for research, FARMS, and by extension Peterson, would be guilty of breaking the law. They label their work as scholarly, and claim to use a peer review process, but their work is neither scholarly nor is it subject to true peer review in any legitimately understood sense of the word.
I suppose the difference beteen the peer review process that takes place as described above, and the one that takes place at FARMS is simply a procedural one. At FARMS, they do not feel the need to conduct their "peer review" in the same manner as described above.
Very Respectfully,
Doctor CamNC4Me