If peer review is as sacred as it is made out to be -- and believe me, it is invoked with more regularity and sanctity than Allah for the Muslim, Jehovah for the Jew, or Jesus for the Christian within the orthodoxy of academia -- then surely, SURELY, it should hold up to its own process, yes?
So let's review peer review--but not blindly. Let us see what its premises are, its pretensions, and its practices. We will find at the core of peer reviewed publishing the pride of academia--both its security and also its vanity. We may also find that it has worked pretty well for a long time and is fairly reliable. But let us not mistake a practice that's been adequate for one context as an absolute to be lorded over another. That would be monstrous.
I am not taking pot shots. I assume a cavalier tone to provoke attention, but this should not be mistaken as ignorance of peer review or any lack of seriousness in appreciating it. I will be claiming that peer review is dangerously broken, working contrary to its original goals, and overdue for a radical rebirth--but this is coming not from an uninformed pundit throwing rocks at the ivory tower. I'm an academic insider, and I'm blowing the whistle.
I first began editing an academic journal in 1990, and most recently finished five years as the Liberal Arts & Sciences editor for BYU Studies, a multi-disciplinary academic journal with 50 years of issues on the shelf. Peer review was the axis upon which our operation turned. We editors screened mountains of manuscripts, rejecting and explaining our rejections, then networking and researching to find the right peer reviewers for the articles that passed our initial review. I saw the system working at its best--when the feedback from reviewers contributed to substantial revisions that in turn yielded a higher quality manuscript that all could be proud of--much as Gregory Clark described the process in the previous post.
And over my academic career I have routinely peer reviewed manuscripts for editors who have sought me out for my specialty knowledge (ironically, I've been deemed an authority within traditional publishing due largely to my nontraditional digital publications...). I have tried to be a fair and thorough reviewer, if not always a prompt one. At the other end of that, as an editor, I have dogged and nagged reviewers to return manuscripts, and I have spent countless hours with individual authors explaining why their work did not meet standards and suggesting ways that it could. I have been an agent of the system on every side--working with editors who showed interest in my own work, and dealing with the routine disappointment of my work being rejected.
I've edited books, too, gathering in manuscripts and then re-dispersing them to reviewers, working under the restraints of press runs, budgets, publishing schedules, and editorial policies. It's a lot of hard work, and as I look at the publications that I've authored or helped to produce, there is satisfaction in knowing something went from being disorganized and of middling quality to something one could be proud of. Others have had much more experience, but I think mine sufficiently representative of academic publishing to speak with some authority. No, there are no pot shots here. I've been in the belly of the beast.
So when you look at that cartoonish image that heads this post -- a ghoulish beast ripping apart the words "PEER REVIEW"-- this should not be mistaken for me, some reckless iconoclast, comically trying to tear down the very foundations of academic knowledge. No, the fiend is peer review itself, first laying waste shoddy arguments and half-baked scholarship, but not stopping there; it has grown up into something beastly and dangerous.
How could something so necessary as expert evaluation possibly turn into a monster? Listen to me betray the brotherhood and I will tell you: Peer review has grown dangerous through the unchecked power of anonymity and its use more as an instrument of academic power than as a filter of scholarly quality. That is how peer review has become both an idol and a bogeyman.
Peer review is like a lab lizard from a B-movie -- originally meant to be used as a control for objectivity, but after getting sprayed with the gamma rays of institutional authority and grown to adulthood on the atomic fuel of profit-driven publishing, it has ginormousized itself into something so revered, so feared, that it rampages like Kong in Manhattan. Movie poster slogan: Peer Review--Though it may check error, its terror is unchecked. That's right, it destroys with ravenous abandon, stamping to jelly the very beings who give it life, indiscriminate in its faux discriminatory acumen. We feed the beast; we feed others to the beast; and then, the beast eats us.
Comical? A caricature? Over the top? Ask enough of its victims (not hard to find!) and they will tell you I have understated its cruelty and downplayed its abuses and inefficiencies. Well, I intend to expose the monster, to rip off the mask and reveal it for the fraud that it was never intended to be but which it has unfortunately become. I challenge my readers to revisit their devotion to this false god. I dare you to question if conventional peer review truly serves its original purposes and to be honest about whether better alternatives don't exist. The unexamined peer review is not worth conducting.
There will be many, like the Pharisees at the time of Christ, who are so invested in the priestly offices of the establishment that they will see any challenge to the status quo as an attack, as a threat. Part of the beastliness of the beast is the protective pact made by among its adherents. Measure this by how readily establishment scholars are dismissive of alternatives, even when there is a good case for new modes of publishing and new kinds of review to meet the ends for which conventional peer review and academic publishing were invented. That sort of defensive reaction only proves my point: those within academia who are unwilling to have their principles and processes re-examined are not serious stewards of knowledge. Those who believe everything except peer review is subject to rigorous examination are hypocrites whose loyalty is not to advancing knowledge or extending the reach and impact of learning; no, their primary loyalty is to institutions and to the power structures fueled by peer review. Any knowledge system without humility is untrustworthy--especially any system whose priority is perpetuating itself rather than the loftier aims it has purported to serve.
Let's talk about vanity publishing.
A vanity press -- let's go back before the Internet -- was a publishing house which relied not on book sales but on the authors paying to have their work published. With no gatekeeping check on quality, vanity presses have truly stained the world with atrocious misuses of ink and paper. These are the intellectual white elephants of the book world that sit cheek by jowl with personal poetry chapbooks and memoirs of pets.
Then came the Internet. Starting in the early 1990s one could use a website service like Geocities and fling one's underdeveloped or overstated ideas across the ether. The Internet set loose vanity publishing on a global scale. The vanity comes through the self-flattering of "being published" by oneself, and also by such stuff being vain or fruitless in making real contributions to anything. With the indisputable bounteousness of poor-to-bad online knowledge, a flight-to-quality rationale has made peer-reviewed publishing seem all the more august and earnest. Only peer review is the guarantor of rigor, of meeting the standards of scholarship within a given discipline. Anything that bypasses that system has no warrant, no authoritative standing. Thus, even the most thoughtful academic experiment with new media online is no more worthy of serious attention than Ryan and Jennifer's 7th-grade "Harry Potter Roolz!" fansite.
From the lofty view of peer reviewed publishing, the Internet is one galactic vanity press, a maelstrom of shallow propositions, half-baked ideas, untested theories, biased opinion, and crap. The only islands of light in this intellectual wasteland--the only discourse worthy of the name of knowledge-- are those peer-reviewed journals whose information has passed muster with The Brotherhood.
So here's their math: Internet or electronic publication = self-publishing = unvetted, unreliable, inferior knowledge. It is no wonder websites and blogs are considered a profound waste of time or even a threat to the careers of academics. In fact, electrons are so tainted that even journals following conventional peer review processes have been deemed suspect if distributed online instead of in print.
From the traditional point of view, electronic publishing is vanity publishing. The nature of digital publishing deserves very careful examination (see my series on the subject), but let's see the various ways traditional, peer-reviewed publishing rates quite high on the vanity scale:
- Elitism is vanity
Academic journals are highly restricted to those who are highly educated (usually in specialized fields with their own esoteric vocabulary), and to those who are rich. One need not go to the extreme examples of those $20,000/year neurosurgery journals, either, to see what a select audience is privileged to see any academic knowledge. A journal that costs $50-100 annually puts it out of reach of the better part of the world.
Moreover, because academic publishing is a commercial endeavor, the editorial policies for these publications are slanted toward the concerns of the rich that are the commercial targets for the work. Malaria is more abundant today than it ever was, yet medical journals are more likely to publish works about Cialis or whatever other big-money drug funds the ads that keep that journal afloat.
Even ad-free academic publishing is hobbled by its elitest hyper-specialization. The oligarchic system of specialists vetting specialists ends up cutting off knowledge not only from the general public, but even from academics with offices in the same hallway. Alas, that guarantor of objectivity and quality, peer review, does not prevent the vanity of elitist academic publishing; it sustains and enforces it, believing this to be a mark of its very quality.
- Blind review is vanity
We flatter ourselves that authors and reviewers being ignorant of one another removes potential prejudice and gives knowledge a chance to be objectively evaluated. In reality, anonymity removes responsibility, providing the appearance of objectivity without any checks upon the system. Editors can claim something has been reviewed externally when it has only undergone internal review. A "peer" can be mismatched with the subject being reviewed. This is commonplace. In my own field, rhetoric, there are overlapping disciplines. I've submitted something for publication and have had blind comments returned to me from reviewers that indicate the editors didn't know the field of rhetoric well enough to send the manuscript to someone in the humanities branch of this field rather than the speech communications side. But the blind review disallows any contact between author and reviewers, no chance to clarify expectations or even to orient to the proper sub-discipline. I know this as an editor; sometimes we have had to go on little information about a reviewer and have not always properly aligned subjects and specialists. But we couldn't very well ask the authors to help us find the best way to assess their work, could we? Blind peer review can also be a way to abuse privilege. Someone with a score to settle can do so by using the blind review process punitively. Any system that places the power to make or break careers in the hands of those who never have to answer for their evaluations invites large-scale corruption. Pretending otherwise is a vanity of its own.
- Delay is vanity
Knowledge thrives through interchange. Peer review is shamefully sluggish. A peer-reviewed publication which I submitted in 1995 waited 12 years before it finally got into print. I have long since moved on from that corner of my specialty and see the publication as just something for the record, evidence that I had succeeded in certain research, but my article and my attention are now so far apart that my scholarship there is a fossil, an historical record, not a living thing. That publication resulted in a recent invitation to speak at a conference--an invitation that would have been meaningful in 1996 but was comical in 2007.
There are so many examples of harmful, shameful delays within the traditional peer review and publishing process that it constitutes a dreadful epic of wasted opportunity and languishing knowledge. One book manuscript which I sent in for review to Oxford University Press received a positive response from the editor, went out for review, then came back for revisions. I was asked to recast my approach, which I painstakingly did, returning the manuscript so it could go out to external reviewers again. Finally, mixed reviews caused the publisher to decide against publication. That process took 18 months, effectively killing the project. Am I alone in such tales? When the means are at hand for rapidly getting ideas out to where they can be used and reviewed, but academics insist on keeping knowledge out of sight until it is perfected and vetted through suspect control mechanisms, this is vanity.
- Single instance review is vanity.
Those who tout intellectual rigor and responsibility as merits of peer review (as my colleague articulated in the last post) need to take their own standard more seriously. Traditional peer review is single instance. There may be multiple reviewers, but the knowledge is vetted only once, prior to publication. The web has made possible ongoing, open review by people across disciplines. Digital metrics make possible usage studies and feedback mechanisms that can create an open-ended and collaborative feedback loop. And open access publication can expose published knowledge to orders of magnitude more people from more specialties. I know this because my own academic websites have received an ongoing stream of feedback which has fed productively into my revision plans. Each revision to Silva Rhetoricae or the Mormon Literature & Creative Arts database is markedly superior to the previous one due to monitoring their use (a kind of review), and to responding to feedback from both specialists and the general public. I call this a more responsible kind of review, far more thorough than the handful of specialists that might evaluate a project once and from a very limited point of view.
When the means of opening knowledge to such a profoundly more vast and varied group of people is at hand, but one clings to the notion that a single evaluation by only two or three specialists is adequate, this is an embarrassing vanity--sustained not by logic or a true desire to improve the quality of the publication, but by a desire to uphold a traditional mode of evaluation. That is intellectually shallow and irresponsible. Why do academics continue to buy into an inferior mode of peer review? Again, their interest is in sustaining the institutions and power structures already set up. They are not interested in curating knowledge or serving any public good. Traditional peer review is vanity publishing.
By attacking traditional peer review, I am not attacking the need for review itself. I only assert that the conventional print-paradigm mode of blind peer review is inadequate--to the point that it has become counter-productive. It is an instrument unfit for the digitally mediated and interactive world we now live in. We don't need less peer review; we actually need more. But conventional peer review gives us so much less than emerging alternatives.
Why will scholars and institutions cling to traditional peer review well beyond its utility? Because peer review is not simply a mechanism for publishing good scholarship; it is a mechanism for establishing and sustaining the authority of those restricting access and building careers based on that game. This is the real reason why Open Access is a threat. Academic publishing is not about the knowledge; it's about the power. If it were about knowledge being a public good, the choke hold on learned publishing would have given way long ago. But the beast will continue throttling good ideas and their makers, leaving a trail of dead bodies behind it as the academics applaud themselves for having safeguarded knowledge.
Vanity, vanity. All is vanity.
This post is a first run at Chapter 4 of Academic Evolution: The Book--see the Table of Contents. I'd love your vigorous feedback.