Transitioning Peer Review from the Print Paradigm

Lest my readers suppose that the testy tone of my prior post critical of peer review suggests I do not take it seriously enough or that I'm simply out of touch with academic realities, I offer below the abstract which I just sent in as a proposed paper for the International Symposium on Peer Reviewing coming up in July. I highly recommend glancing over their program, as its organizers have thought very thoroughly about the various issues involved as they attempt to apply peer review to peer review itself. Their own process for vetting the proposed papers is a unique combination of blind and open review. I was invited to submit the names of three potential reviewers as part of this. Here's my abstract, to which I invite open review:

Methods for authenticating knowledge are not absolute epistemologies; they are dependent upon the technologies that make possible certain kinds of verification. Printing made it possible to arrange, distribute, and verify knowledge in novel ways. These novelties (such as scholarly editions, indexes, footnotes, or tabular data) made possible scholarly conventions (such as documenting sources), scholarly genres (such as lab reports), and intellectual practices (such as source checking) that gradually became required for scholarly knowledge to be recognized as such. Modern scholarship and publishing, as well as the academic superstructure built upon it, exist within a knowledge economy so dependent upon such historical conventions and practices that these now seem natural to knowledge itself, not as byproducts of a technology that evolved certain modes of authenticating knowledge while supplanting others.

The problems with peer review today are rooted in trying to maintain print paradigm concepts within a knowledge economy that operates upon different principles and that is quickly evolving its own conventions for ascertaining validity and quality. The existence of “peers” presupposes modes of isolated expertise at odds with social intelligence evident in recommendation systems, web metrics, and the simple exposure of open content to multiple disciplines and points of view. “Publishing” presupposes the perfecting of knowledge before it is shared, while the nature of the digital medium pushes toward knowledge being “perpetually in beta.” The delays of traditional peer review are unacceptable in the post-and-comment world of easy blogging and pre-print servers. Single-instance review becomes intellectually indefensible when knowledge shared broadly invites continuous open review. Conventional peer review is tied to the control mechanisms of editorial boards and toll-access academic journals, while Open Access publishing invites transparent, interactive, and rapid editorial and vetting practices. Traditional peer review treats knowledge in terms of intellectual products vying for the privilege to be marketed by publishers who assume copyright of those products; but Creative Commons licensing defies the restrictive and proprietary commodification of knowledge. The re-use of knowledge (post-publication) may be as much a sign of its quality as any pre-publication assessment. Traditional peer review favors discursive genres of scholarship, while the Internet focuses value on reusable digital objects such as data sets, software programs, and media that can serve as the tools building the cyberinfrastructure of a discipline. Peer review is traditionally divorced from layout and design, while the value of digital scholarship necessarily correlates with how well it is digitally optimized for search engines and the semantic web through XML-structured content, appropriate metadata, and hyperlinks.

We must begin with acknowledging that the new medium calls forth its own modes of authenticating knowledge and attempt more to see where the digital realm leads than to insist upon propagating practices no longer consistent with the primary intellectual medium.

Comments

[Glowing Willow]

Thanks for sharing this. I didn't understand at first, that the tone in some of your earlier posts were to initiate a conversation and, at times, that you were being an antagonist to your own protagonist views. This is a good conference and I needed to read your point of view. I enjoy being made to think outside the box!

[Stephen Humphrey]

Outstanding. Wish I could be there.

So an interesting extension to these new debates on NewMediaPeerReview is going to be validation of the reviewers; what are we going to do to prevent "gaming the system" (namely, parallels to astroturfing and similar underworld phenomena we've already seen in political new media)? If the journal is today acting as the validator of the anonymous reviewers, who serves that function in new media?

The answer clearly will involve some form of reputation economies, but I'm not sure what forms that might take (although they will surely be something more rigorous than eBay ratings or Amazon starred reviews).

I guess a first-level approximation of the problem is that today the journal (and its editorial staff) acts almost as a form of group pseudonym to protect the article from biasing influence between the author and the reviewers. Can we build similar anonymity mediators in an online world when we are simultaneously promoting more open reputation systems?