The time has come to call academia's darling child out onto the carpet. That's right. It's time to review peer review.
If peer review is as sacred as it is made out to be -- and believe me, it is invoked with more regularity and sanctity than Allah for the Muslim, Jehovah for the Jew, or Jesus for the Christian within the orthodoxy of academia -- then surely, SURELY, it should hold up to its own process, yes?
So let's review peer review--but not blindly. Let us see what its premises are, its pretensions, and its practices. We will find at the core of peer reviewed publishing the pride of academia--both its security and also its vanity. We may also find that it has worked pretty well for a long time and is fairly reliable. But let us not mistake a practice that's been adequate for one context as an absolute to be lorded over another. That would be monstrous.
I am not taking pot shots. I assume a cavalier tone to provoke attention, but this should not be mistaken as ignorance of peer review or any lack of seriousness in appreciating it. I will be claiming that peer review is dangerously broken, working contrary to its original goals, and overdue for a radical rebirth--but this is coming not from an uninformed pundit throwing rocks at the ivory tower. I'm an academic insider, and I'm blowing the whistle.
I first began editing an academic journal in 1990, and most recently finished five years as the Liberal Arts & Sciences editor for BYU Studies, a multi-disciplinary academic journal with 50 years of issues on the shelf. Peer review was the axis upon which our operation turned. We editors screened mountains of manuscripts, rejecting and explaining our rejections, then networking and researching to find the right peer reviewers for the articles that passed our initial review. I saw the system working at its best--when the feedback from reviewers contributed to substantial revisions that in turn yielded a higher quality manuscript that all could be proud of--much as Gregory Clark described the process in the previous post.
And over my academic career I have routinely peer reviewed manuscripts for editors who have sought me out for my specialty knowledge (ironically, I've been deemed an authority within traditional publishing due largely to my nontraditional digital publications...). I have tried to be a fair and thorough reviewer, if not always a prompt one. At the other end of that, as an editor, I have dogged and nagged reviewers to return manuscripts, and I have spent countless hours with individual authors explaining why their work did not meet standards and suggesting ways that it could. I have been an agent of the system on every side--working with editors who showed interest in my own work, and dealing with the routine disappointment of my work being rejected.
I've edited books, too, gathering in manuscripts and then re-dispersing them to reviewers, working under the restraints of press runs, budgets, publishing schedules, and editorial policies. It's a lot of hard work, and as I look at the publications that I've authored or helped to produce, there is satisfaction in knowing something went from being disorganized and of middling quality to something one could be proud of. Others have had much more experience, but I think mine sufficiently representative of academic publishing to speak with some authority. No, there are no pot shots here. I've been in the belly of the beast.
So when you look at that cartoonish image that heads this post -- a ghoulish beast ripping apart the words "PEER REVIEW"-- this should not be mistaken for me, some reckless iconoclast, comically trying to tear down the very foundations of academic knowledge. No, the fiend is peer review itself, first laying waste shoddy arguments and half-baked scholarship, but not stopping there; it has grown up into something beastly and dangerous.
How could something so necessary as expert evaluation possibly turn into a monster? Listen to me betray the brotherhood and I will tell you: Peer review has grown dangerous through the unchecked power of anonymity and its use more as an instrument of academic power than as a filter of scholarly quality. That is how peer review has become both an idol and a bogeyman.
Peer review is like a lab lizard from a B-movie -- originally meant to be used as a control for objectivity, but after getting sprayed with the gamma rays of institutional authority and grown to adulthood on the atomic fuel of profit-driven publishing, it has ginormousized itself into something so revered, so feared, that it rampages like Kong in Manhattan. Movie poster slogan: Peer Review--Though it may check error, its terror is unchecked. That's right, it destroys with ravenous abandon, stamping to jelly the very beings who give it life, indiscriminate in its faux discriminatory acumen. We feed the beast; we feed others to the beast; and then, the beast eats us.
Comical? A caricature? Over the top? Ask enough of its victims (not hard to find!) and they will tell you I have understated its cruelty and downplayed its abuses and inefficiencies. Well, I intend to expose the monster, to rip off the mask and reveal it for the fraud that it was never intended to be but which it has unfortunately become. I challenge my readers to revisit their devotion to this false god. I dare you to question if conventional peer review truly serves its original purposes and to be honest about whether better alternatives don't exist. The unexamined peer review is not worth conducting.
There will be many, like the Pharisees at the time of Christ, who are so invested in the priestly offices of the establishment that they will see any challenge to the status quo as an attack, as a threat. Part of the beastliness of the beast is the protective pact made by among its adherents. Measure this by how readily establishment scholars are dismissive of alternatives, even when there is a good case for new modes of publishing and new kinds of review to meet the ends for which conventional peer review and academic publishing were invented. That sort of defensive reaction only proves my point: those within academia who are unwilling to have their principles and processes re-examined are not serious stewards of knowledge. Those who believe everything except peer review is subject to rigorous examination are hypocrites whose loyalty is not to advancing knowledge or extending the reach and impact of learning; no, their primary loyalty is to institutions and to the power structures fueled by peer review. Any knowledge system without humility is untrustworthy--especially any system whose priority is perpetuating itself rather than the loftier aims it has purported to serve.
Let's talk about vanity publishing.
A vanity press -- let's go back before the Internet -- was a publishing house which relied not on book sales but on the authors paying to have their work published. With no gatekeeping check on quality, vanity presses have truly stained the world with atrocious misuses of ink and paper. These are the intellectual white elephants of the book world that sit cheek by jowl with personal poetry chapbooks and memoirs of pets.
Then came the Internet. Starting in the early 1990s one could use a website service like Geocities and fling one's underdeveloped or overstated ideas across the ether. The Internet set loose vanity publishing on a global scale. The vanity comes through the self-flattering of "being published" by oneself, and also by such stuff being vain or fruitless in making real contributions to anything. With the indisputable bounteousness of poor-to-bad online knowledge, a flight-to-quality rationale has made peer-reviewed publishing seem all the more august and earnest. Only peer review is the guarantor of rigor, of meeting the standards of scholarship within a given discipline. Anything that bypasses that system has no warrant, no authoritative standing. Thus, even the most thoughtful academic experiment with new media online is no more worthy of serious attention than Ryan and Jennifer's 7th-grade "Harry Potter Roolz!" fansite.
From the lofty view of peer reviewed publishing, the Internet is one galactic vanity press, a maelstrom of shallow propositions, half-baked ideas, untested theories, biased opinion, and crap. The only islands of light in this intellectual wasteland--the only discourse worthy of the name of knowledge-- are those peer-reviewed journals whose information has passed muster with The Brotherhood.
So here's their math: Internet or electronic publication = self-publishing = unvetted, unreliable, inferior knowledge. It is no wonder websites and blogs are considered a profound waste of time or even a threat to the careers of academics. In fact, electrons are so tainted that even journals following conventional peer review processes have been deemed suspect if distributed online instead of in print.
From the traditional point of view, electronic publishing is vanity publishing. The nature of digital publishing deserves very careful examination (see my series on the subject), but let's see the various ways traditional, peer-reviewed publishing rates quite high on the vanity scale:
- Elitism is vanity
Academic journals are highly restricted to those who are highly educated (usually in specialized fields with their own esoteric vocabulary), and to those who are rich. One need not go to the extreme examples of those $20,000/year neurosurgery journals, either, to see what a select audience is privileged to see any academic knowledge. A journal that costs $50-100 annually puts it out of reach of the better part of the world.
Moreover, because academic publishing is a commercial endeavor, the editorial policies for these publications are slanted toward the concerns of the rich that are the commercial targets for the work. Malaria is more abundant today than it ever was, yet medical journals are more likely to publish works about Cialis or whatever other big-money drug funds the ads that keep that journal afloat.
Even ad-free academic publishing is hobbled by its elitest hyper-specialization. The oligarchic system of specialists vetting specialists ends up cutting off knowledge not only from the general public, but even from academics with offices in the same hallway. Alas, that guarantor of objectivity and quality, peer review, does not prevent the vanity of elitist academic publishing; it sustains and enforces it, believing this to be a mark of its very quality. - Blind review is vanity
We flatter ourselves that authors and reviewers being ignorant of one another removes potential prejudice and gives knowledge a chance to be objectively evaluated. In reality, anonymity removes responsibility, providing the appearance of objectivity without any checks upon the system. Editors can claim something has been reviewed externally when it has only undergone internal review. A "peer" can be mismatched with the subject being reviewed. This is commonplace. In my own field, rhetoric, there are overlapping disciplines. I've submitted something for publication and have had blind comments returned to me from reviewers that indicate the editors didn't know the field of rhetoric well enough to send the manuscript to someone in the humanities branch of this field rather than the speech communications side. But the blind review disallows any contact between author and reviewers, no chance to clarify expectations or even to orient to the proper sub-discipline. I know this as an editor; sometimes we have had to go on little information about a reviewer and have not always properly aligned subjects and specialists. But we couldn't very well ask the authors to help us find the best way to assess their work, could we? Blind peer review can also be a way to abuse privilege. Someone with a score to settle can do so by using the blind review process punitively. Any system that places the power to make or break careers in the hands of those who never have to answer for their evaluations invites large-scale corruption. Pretending otherwise is a vanity of its own.
- Delay is vanity
Knowledge thrives through interchange. Peer review is shamefully sluggish. A peer-reviewed publication which I submitted in 1995 waited 12 years before it finally got into print. I have long since moved on from that corner of my specialty and see the publication as just something for the record, evidence that I had succeeded in certain research, but my article and my attention are now so far apart that my scholarship there is a fossil, an historical record, not a living thing. That publication resulted in a recent invitation to speak at a conference--an invitation that would have been meaningful in 1996 but was comical in 2007.
There are so many examples of harmful, shameful delays within the traditional peer review and publishing process that it constitutes a dreadful epic of wasted opportunity and languishing knowledge. One book manuscript which I sent in for review to Oxford University Press received a positive response from the editor, went out for review, then came back for revisions. I was asked to recast my approach, which I painstakingly did, returning the manuscript so it could go out to external reviewers again. Finally, mixed reviews caused the publisher to decide against publication. That process took 18 months, effectively killing the project. Am I alone in such tales? When the means are at hand for rapidly getting ideas out to where they can be used and reviewed, but academics insist on keeping knowledge out of sight until it is perfected and vetted through suspect control mechanisms, this is vanity. - Single instance review is vanity.
Those who tout intellectual rigor and responsibility as merits of peer review (as my colleague articulated in the last post) need to take their own standard more seriously. Traditional peer review is single instance. There may be multiple reviewers, but the knowledge is vetted only once, prior to publication. The web has made possible ongoing, open review by people across disciplines. Digital metrics make possible usage studies and feedback mechanisms that can create an open-ended and collaborative feedback loop. And open access publication can expose published knowledge to orders of magnitude more people from more specialties. I know this because my own academic websites have received an ongoing stream of feedback which has fed productively into my revision plans. Each revision to Silva Rhetoricae or the Mormon Literature & Creative Arts database is markedly superior to the previous one due to monitoring their use (a kind of review), and to responding to feedback from both specialists and the general public. I call this a more responsible kind of review, far more thorough than the handful of specialists that might evaluate a project once and from a very limited point of view.
When the means of opening knowledge to such a profoundly more vast and varied group of people is at hand, but one clings to the notion that a single evaluation by only two or three specialists is adequate, this is an embarrassing vanity--sustained not by logic or a true desire to improve the quality of the publication, but by a desire to uphold a traditional mode of evaluation. That is intellectually shallow and irresponsible. Why do academics continue to buy into an inferior mode of peer review? Again, their interest is in sustaining the institutions and power structures already set up. They are not interested in curating knowledge or serving any public good. Traditional peer review is vanity publishing.
By attacking traditional peer review, I am not attacking the need for review itself. I only assert that the conventional print-paradigm mode of blind peer review is inadequate--to the point that it has become counter-productive. It is an instrument unfit for the digitally mediated and interactive world we now live in. We don't need less peer review; we actually need more. But conventional peer review gives us so much less than emerging alternatives.
Why will scholars and institutions cling to traditional peer review well beyond its utility? Because peer review is not simply a mechanism for publishing good scholarship; it is a mechanism for establishing and sustaining the authority of those restricting access and building careers based on that game. This is the real reason why Open Access is a threat. Academic publishing is not about the knowledge; it's about the power. If it were about knowledge being a public good, the choke hold on learned publishing would have given way long ago. But the beast will continue throttling good ideas and their makers, leaving a trail of dead bodies behind it as the academics applaud themselves for having safeguarded knowledge.
Vanity, vanity. All is vanity.
This post is a first run at Chapter 4 of Academic Evolution: The Book--see the Table of Contents. I'd love your vigorous feedback.
I've been invited to lead a workshop on academic publishing in a Ghanaian university - I'm Ghanaian. In preparing for this task I've found this paper very helpful in the sessions in which I'll be dealing with peer review, and I definitely will be recommending it to my audience. It is also entertaining. Thank you
Posted by: Lawrence Owusu-Ansah | March 18, 2009 at 10:05 AM
Thanks for commenting, but let me paraphrase myself so you can see that I did in fact intend those words. Essentially, peer review has been set up as a way of testing and verifying knowledge. This is rightly seen as necessary to advancing knowledge. To exempt peer review itself from the same scrutiny demonstrates that those holding it up do not actually believe in peer review's ostensible purpose, and I am claiming that the peer review process is, for them, actually more about sustaining the existing knowledge exchange system than the actual advancement of learning. A threat to conventional peer review is a threat to those institutions built upon it, not to learning or to sound ideas. If one believes that only institutionalized knowledge counts as knowledge or advances the cause of learning, then my argument will not make sense. But I am trying to appeal to those whose loyalty is more to intellectual progress than to institutional preservation.
Posted by: Gideon Burton | March 13, 2009 at 12:46 PM
I haven't finished reading this yet, but you seem to have some editing errors that are interfering with your meaning:
"Those who believe everything except peer review is subject to rigorous examination are hypocrites whose loyalty is not to advancing knowledge or extending the reach and impact of learning; no, their primary loyalty is to institutions and to the power structures fueled by peer review."
I think you mean "nothing except peer review".
Posted by: M | March 13, 2009 at 10:00 AM
In part this blog post is a 21st century approach to Wayne Booth's 1982 MLA address "The Credo of an English Teacher." One of the more memorable lines from the address comes when Booth rebukes scholars who forget the public: "When we fail to test our scholarship by making its most important results accessible to non-specialists," he says, "we also lose our capacity to address, and thus recreate in each generation, the literate public who can understand its stake in what we do." To me, this is the center of Gideon's argument. While I'm not a huge proponent of the cavalier tone he employs (though I can see his motives for using it), I think that Gideon's fight to make scholarship more accessible is very needed. I should say, I need it.
As I've tried to enter the academic discourse I've felt continually excluded. As an undergraduate and even as a graduate student I find that most scholars tend to be talking to their peers, not to me. (Booth's work is a rare exception.) My perception of scholars so far has not been good. I like them in person more than in writing. I feel that I've honestly tried, am trying, to enter the discourse, but to me most scholarship seems very aware of being intentionally difficult, hard to access, exclusive. Gideon's efforts work in opposition to this trend. As a student I appreciate this very much. To me, any work that is influencing me to be a better student is real and is needed. Any work that obfuscates meaning or is obscure and abstract in its rhetorical approach is phony and unneeded.
I hope, above all, that the changes in the digital age will lead to the emergence of more accessible and more useful scholarship. The quest for a more literate public is, in my mind, a most important goal for any scholar.
Posted by: Jon Ogden | February 07, 2009 at 02:41 PM
I'm beginning to wonder if part of the problem might be the great divide between the sciences and the liberal arts. (My thinking on this subject is informed in part by an interesting essay by web guru Paul Graham: http://www.paulgraham.com/essay.html .)
While I think the blind peer-review methodology works fairly well in the sciences, I can see how it is somewhat more problematic in the liberal arts. The sciences of course benefit greatly from the objective testing of evidence. However, the liberal arts as benefit much more from the free and open exchange of ideas. I like Natalie Goldberg's metaphor of the compost pile from which good writing/ideas bloom. There's a lot of value in half-baked ideas that can one day grow up to become serious scholarship. And I can see how the new digital age can facilitate that by providing quick and timely feedback.
Incidentally, I enjoy a blog post defending the peer-review process by Steve Peck, a member of the biology department at BYU: http://sciencebysteve.net/?p=45 It strikes me that he would be a good guest on your podcast if you wanted someone with whom you could debate this issue and who has a background in science. In any case, his blog is pretty entertaining in its own right.
Posted by: Alyssa | February 06, 2009 at 01:59 PM
Silvia:
I think you really get where I'm coming from. Thanks for a thoughtful reply. Why didn't I know about that Zuniga book? I'll have to get it right away. That point about bypassing the gatekeepers was something I was getting at in my "Dear Students" post earlier. I'm searching for others, like you or Zuniga, who have thought these things through.
In a future post I do plan to give attention to various alternative models to conventional peer review. That British journal's efforts sound interesting--do you recall the name of that periodical? I had a very positive experience with open peer review from JIME that I plan to discuss.
Thanks for the ongoing thought on this. I appreciate it.
Posted by: Gideon Burton | February 05, 2009 at 07:51 PM
First of all, Waking Tiger, I love your lurid pulp fiction style of prose! It actually makes your points stronger and it's a very creative example of an academically unacceptable writing style used to support an academically rigorous argument.
I think what you are mostly railing at is the fact that academia isn't even willing to open the conversation about alternatives to traditional peer review. But as Markos Zuniga says in his brilliant book "Taking on the System: Rules for Radical Change in a Digital Era", the new realities allow us to sometimes simply bypass the gatekeepers. What is going to happen when the best and brightest academics of the new generation are putting more and more of their work into new media?
I, like all academics, have greatly benefited but also unjustly suffered at the hands of reviewers. It can just be the luck of the draw. Sometimes they have no clue about your methodology and critique it based on the rules of another paradigm. Or American reviewers ask for corrections that are ethnocentric (e.g., asking a Canadian author to change "blacks" to "African-Americans").
I have seen some glimpses of different peer review models under the traditional system. One colleague submitted to a British journal on group practice that really coaches and mentors authors to bring their articles up to publication standards. They want to encourage practitioners to submit, not only academics.
But like you, I am incredibly excited about the possibilities for new forms of peer review that may bring a broader and more immediate kind of review to our work.
Bravo for taking on this beast!
Posted by: Silvia | February 05, 2009 at 04:19 PM