Scholarly Communications will Transform via Cybermetrics

Continuing my series on how scholarly communications must transform, I will argue here that scholarship is about to see "webometrics" or "cybermetrics" supplant traditional bibliometrics for gauging the impact of scholarship. But this is just the beginning. Cybermetrics applied to scholarship will revitalize traditional academic publishing and pave the way for new uses and genres of intellectual work. As scholars and their institutions begin to use cybermetrics they can enrich scholarly productivity and maximize the influence of their intellectual output.

The Impact Factor Factor
Impact is a big deal to scholars and their sponsors. Really big. The Impact Factor of the journal in which one publishes adds or subtracts value from one's publications. This algorithm is derived from a calculation based chiefly on the number of citations a publication generates. It has become a prominent determining factor in securing grants, academic posts, tenure, and advancement. And why not? Don't we want scholars to be making an impact?  With today's info glut, isn't it even more important to preserve and promote systems that help us to know what information should be given more authority?

I will first look into the history of Impact Factor and will claim that this early effort to grapple with information overload has improperly become institutionalized and is neither trustworthy nor adequate for today's information culture. Then I will open the discussion of what can or should be measured through cybermetrics with online scholarly communication. Academia has some very good places to go with its treasure trove of existing and ongoing scholarship; it can't get there by clinging to authority systems based on pre-Internet bibliometrics.

Origin of Impact Factor

Where does Impact Factor comes from?  The photo here links to the 1955 article in Science in which Eugene Garfield first proposed the creation of a citation index for science. He was trying to solve a problem we face on a bigger scale today. With so much information circulating and the great need to sift the wheat from the chaff (even 40 years before the Internet), Garfield proposed that citations be used as a quality filter. The reasoning here is sound. After all, when people reference others' research, it shows they consider that research important. So, if lots of people cite the same things, the idea goes, then those things obviously have more influence. Quantifying citations has become the post-publication counterpart to pre-publication peer review. Peer review evaluates work before it is in print; cumulative citations show how researchers are filtering for quality by way of what they read and cite.

Could it be that today's web page hits or inbound links are the analogs to traditional scholarly citation indexes? Yes and no. But before getting into the details of webometrics or cybermetrics, it's important to appreciate the impact of the impact factor upon academia and the problems this has created. What started as a good faith effort to systematize scholarly selectivity has turned into a system that abuses that good faith. It has led some scientists to claim that "Garfield's impact factor is now being used by others in ways that threaten to destroy scientific inquiry as we know it" (Roger A. Brumback). Less apocalyptic but equally earnest protests are widespread. The British Medical Journal has even commissioned a series of articles analyzing problems with Impact Factor, and in a stinging editorial from the Journal of Cell Biology, the inadequacies of Impact Factor are laid bare.

What could go wrong? Well, first of all, Impact Factor simply is not the objective instrument academic evaluators treat it as. Eugene Garfield commercialized his Science Citation Index, creating the Institute for Scientific Information -- thus, the "ISI Impact Factor" for which it is so famous.  ISI carefully guards the secret formula(s) that yield their Impact Factor. Unlike the scholarship rated by Impact Factor, this proprietary instrument itself is not subject to review or analysis. That discrepancy did not go unnoticed by editors of the Journal of Cell Biology. They rightly claimed that "opaque data" is contrary to editorial policies and scientific transparency. "Just as scientists would not accept the findings in a scientific paper without seeing the primary data," they concluded, "so should they not rely on Thomson Scientific's impact factor, which is based on hidden data." Consider the intellectually shaky ground of basing a whole system of authenticating knowledge upon a reputation algorithm that is not open to review. Science doesn't work this way, but somehow the system that warrants scientific authority gets to do so.

This secrecy is a recipe for abuse, and bias becomes even more likely when one takes into account that ISI and its valuable Impact Factor were purchased by Thomson Corporation, a publisher of scientific journals (which are rated by ISI). Thomson has aggressively marketed the ISI metric along with its scientific publications that garner $600 million annually. This glaring conflict of interest is ignored by academic institutions because the value of using the ISI Impact Factor in the academic evaluation system is greater than the potential threat of internal bias within Thomson.

And there lies the true problem. The abuse of Impact Factor has come through institutional over-reliance upon it. From the beginning Impact Factor was never meant to be a primary or sole criterion for judging academic quality. But as many others have pointed out, Impact Factor has gone beyond its original purpose and has become an all-too convenient proxy for quality within the academic evaluation system. No one needs to read a scholar's work to gauge its importance: the fact that his or her work has appeared in a journal with a certain Impact Factor warrants its quality. As one college dean complains:

The problem is that Impact Factors were never designed to be used for evaluating the quality of scientists or their research projects. So, why are they being used for this? It is a cop-out by reviewers, referees and, perhaps more importantly, administrators who are distributing funds — it is a brutish, simplistic quantitative tool that allows them to avoid hard decisions in the judgement of quality. (Graeme Martin, "Impact Factor=Rip-Off Factor")

Sadly, there are very strong forces at play to keep the Impact Factor as the prime mover in the evaluation of scholarly publications. For Thomson, obviously, the sale of high impact journals is extremely lucrative, so the reputation of the index they publish through ISI directly affects their bottom line. For academics, Impact Factor provides an attractive simplicity that is hard to let go of. Impact Factor works for promotion and tenure committees or for granting bodies the same way SAT and GPA scores work for college admissions: a number gives evaluators a rapid and efficient way to make hard decisions.

The critics have not had trouble punching holes in ISI's bibliometrics. Out of tens of thousands of peer-reviewed journals, why are only a few thousand indexed? How can a new journal ever compete with an older one for citations? Why are only very recent publications factored in? What scholarly items make the count in deciding impact -- proceedings? notes? or just complete articles? What if in one field proceedings have more weight and articles have more weight in another? How does one compensate for self-citation? Doesn't this system encourage rhetorical footnoting? Isn't it true that faulty scholarship is often highly discussed, giving it a faux authority because it gets cited so much?

Impact Factor's authoritative and proprietary nature is not consistent with the ascendant open paradigm of the online knowledge commons. It is unsurprising to find an Open Access outlet like Public Library of Science ONE rejecting ISI's Impact Factor altogether. Earlier this year, this journal announced it would be exposing all sorts of data about the articles it publishes in an effort to encourage innovative and article-level metrics. We are entering a new era of data availability, and it would be as foolish not to look to new metrics based on that data as it would be to try to keep the leaks patched in ISI's Impact Factor. It is time for cybermetrics.

Cybermetrics

While cybermetrics is in its infancy, the principles informing it are readily understood, and we can look at current efforts to apply cybermetrics in order to get our bearings on the future. 

Early webometrics paralleled bibliometrics. The various links back to a given page can be quantified into a measure of impact or authority. But such a "Web Impact Factor" (WIF) has really just been a Web1.0 effort. Today, cybermetrics is growing up as people are understanding and evolving uses of networked information and developing general and commercial Internet metrics for online information. This is the first thing to understand about cybermetrics: it promises not simply an electronic updating of Impact Factor, but a suite of analytical tools that can be variously applied and customized. And whereas Impact Factor has been completely journal-centered, the focus of Cybermetrics is both broader and more granaular, as I will explain.

In what follows, I'm drawing heavily from work being done by the Consejo Superior de Investigaciones Científicas (CSIC), the largest public research body in Spain. This group also publishes the journal Cybermetrics, "devoted to the study of the quantitative analysis of scholarly and scientific communications in the Internet." I see many problems of scope and methodology in what they are attempting, but the difference from ISI is they have made their methodology explicit and have created their journal as an organ for publicly discussing and improving those methods. In other words, CSIC is in keeping with principles of transparency and collaboration that we are recognizing as superior values in the information economy of the 21st century.

Beyond Journals: Measuring Institutional Impact

Since 2004, the Ranking Web of World Universities has been published semi-annually in an effort to showcase the overall intellectual output of universities and to encourage best practices for electronic publishing and for exposing archives intelligently online. While the salient outcome of their project is a numerical ranking that may seem as simplistic for universities as ISI is for journals, the premises upon which CSIC has developed its alogrithm are an intriguing challenge to traditional measurements of scholarship.

The novel assumption here is that universities can and should be measured in terms of their intellectual output and overall presence online. Universities have long been ranked (as with US News & World Report's annual college rankings), but the Ranking Web ignores cultural reputation or alumni surveys in favor of a data-driven index that reveals not campus life or teacher-student ratios, but an institution's intellectual web presence. 

Why shouldn't universities have a measurable online presence -- one that is not simply the product of company PR or cultural reputation, something independently measured and based upon overall intellectual output? Consider how this might change the way universities are viewed generally (or how they might view their contributors internally) once it becomes evident that certain kinds of activities can boost one's intellectual imprint on the world. Think of how funding agencies, state legislatures, and philanthropic organizations might be looking for more bang for their research buck, calibrating grants and support to an institution's net contribution to the information commons of the world. Such measurements could materially affect institutional practices and increase productive competition. It could also throw traditional academic measurement systems into a tailspin.

The criteria and weighting of factors within CSIC's university rankings bear careful discussion. Are the many informal web pages and all that non-scholarly content universities put on the web to be thrown in with peer-reviewed publications in evaluating a school's intellectual contributions? Rather than opening that can of worms right now, I'd like to look just at the most traditional measurement of intellectual output, peer-reviewed publications. Let's consider what it might mean to create some sort of aggregate measure for institutions of its overall output of conventional scholarship.

Tracking an Institution's Overall Scholarly Output

It will be surprising to non-scholars to learn that universities do not track the scholarly output of their faculty members. There may be some internal fact finding and summary reports prepared for accrediting agencies, but until recently, there has been little effort to systematically gather and centralize the scholarship produced by an entire institution. This is due to the fact that faculty are evaluated individually for tenure and promotion within their specialties. And while a university may feature selected scholarship within its public relations, there has been no motive to try to gather faculty work en masse. It has been the function of journals and libraries to gather and preserve scholarship.

But the Internet is changing that. Now, universities and research centers are creating institutional or disciplinary repositories to warehouse scholarship for permanent access. This is driven by the Open Access movement. Now, leading schools like Harvard and Stanford have put in place policies requiring faculty to retain copyright of their publications and to grant the school a non-exclusive license to permanently host copies of all their scholarship in a local archive. 

Here is where cybermetrics is going to prove very interesting. Right now such archiving is being led by librarians who see the virtues of organizing knowledge with metadata and preserving permanent open access to it. Faculty (and most administrators) have not yet recognized that this is far more consequential than simply preserving a record. Archiving means a kind of permanent publishing, and increasingy sophisticated data harvesting tools and data mining activities are already creating a longer and richer life for publications that in the past have only been valued at the point of publication. The after market for academic content is going to be huge, and the true demand of that content is just beginning to make itself felt. Metrics will signal the demand and change these repositories into active assets of the university. Record labels and non-academic publishers have found that Long Tail economics, recommendation systems, and social media have combined to give renewed attention to products once lingering in obscurity. How long will academia resist the cultural and technological forces pushing for the robust discovery and use of its legacy content? About as long as it takes for administrators to read usage metrics and see the value of awakening academia's dormant intellectual assets.

While composing this post I received an email alerting me that a scholarly presentation I once gave on Beowulf was downloaded from where it is hosted online. Suddenly, I am wondering about giving that presentation a revision. After all, now that I can get reports on the use of my work, it changes how I see it and configures my future projects. What happened to me incidentally today is what is happening on a more organized basis at the University of Nebraska - Lincoln through their institutional repository, Digital Commons. There, metrics on the use of archived scholarship have begun to change how faculty view their work. Rather than simply seeing to the publication of their work in a respectable journal, faculty are now seeing the value of their publications being housed where they will be continuously used and where they will get reports each month about the frequency with which their various publications have been downloaded (or commented upon). It is the pattern of things to come.

I foresee the day that both institutions and individual scholars will take as keen an interest in the ongoing life of publications as they now do in placing those publications in respectable journals, precisely because metrics will drive interest that way. If an assistant professor can show that an article published in a relatively minor venue is generating lots of traffic on the university's institutional archive or has created productive attention to the university, will this not be relevant to how that professor is evaluated? Or, if publications in highly respected journals prove to attract nary a hit or generate hosts of negative comments, will this not change the way reputation and rewards work in academia? We have barely scratched the surface of the coming impact of institutionally based scholarship metrics.

These metrics will change the role of academic journals, too, taking from them much of the authority they have traditionally held as arbiters of quality. Peer-review that happens pre-publication will be in competition with the emerging values that attend the use of scholarship once it is published and exposed to the web. Cybermetrics will be driving that change.

We will see the day when universities will try to maximize the value of their intellectual endowment by actively monitoring and marketing their scholarship (old and new). I broached this topic earlier by introducing the concept of Scholarly Inquiry Optimization to parallal Search Engine Optimization (SEO). SEO and web analytics are now firmly entrenched in enterprise, and the value this monitoring adds will at some point become centrally relevant to scholarship. The sort of data now available to those monitoring web assets for business is astounding -- from granular reports of the location of web visitors to the type of platform used to access the data, to a whole host of attention data as clickstreams are tracked and crunched. This makes for a robust way to fine tune marketing and to better match the needs of customers with services and products. Such data begs to be used; the utility is self-evident. And why wouldn't universities begin to capitalize (perhaps even literally) on being able to monitor and direct the attention of its web visitors? Won't this also affect how resource decisions are made on campuses? Once demand can be measured and tailored to customers, we'll have new criteria for organizing and prioritizing scholarly activity.

Measuring Scholarly Media, Metering Scholarly Data

The Internet is spawning the use of multimedia in every aspect of culture, including scholarly inquiry, and we will soon see media-rich born-digital scholarship complementing and competing with traditional articles and books. And as Open Access begins to predominate within scholarly publishing, the effect will do far more than increase the exposure and use of traditional scholarly publications; we will see value accruing through opening scholarly media -- old and new -- to remixing and reuse.

There will be new things to measure when a social scientific study includes the publication of protocols and data that can be separated from the original scholarly study for separate significant use. CSIC measures "rich content" web pages, which for now might simply be limited to the presence of images, audio, or video. But it is not hard to imagine putting in place ways to measure the creative reuse of open scholarly media. A service such as TubeMogul already offers the ability for mainstream media to detect and measure pirated video (by tracking a video's "DNA") in order to provide content producers a sense of the overall impact of their content (which sometimes goes much further through informal and illegitimate channels than through the channels chosen by the content producer). When academic institutions abandon the foolishness of propertization of intellectual output and realize the enormous value of creative commons licensing, cybermetrics will be there to track how broadly and variously researchers' work is used. And as scholars turn to audio and video formats for disseminating their ideas, services like TubeMogul can provide granular analysis of user attention. It will be possible to pinpoint the exact second within a lecture when user interest wanes. This sort of data will provide feedback loops to improve academic media and keep existing content in play longer. Cybermetrics will reveal better ways to use, re-use, and improve intellectual content -- including the application of scholarly content to pedagogical uses (and teaching media to scholarly purposes).

We will see open data becoming increasingly important as an asset in the knowledge commons. As researchers optimize their results by drawing upon publicly available data sets, cybermetrics will be there to meter usage of that data. Suddenly, that study that is a decade old will have its data refreshed as another researcher employs it in a different study. This will reflect positively on the original researchers and funders. Such extended life for data hinges upon metrics that document and encourage the extended life of scholarly products.

Cybermetrics and Structured Knowledge

Cybermetrics will encourage scholars to observe the protocols of the semantic web. The more machine-readable information is, the more readily it can be shared, re-used, and measured. XML and standarized schema will enrich the life and usage of intellectual content. Scholars can therefore be measured not just on their actual research or their arguments, but on how efficiently they have structured their work to be further used and built upon. We will be measuring how well knowledge is structured to articulate with the knowledge commons, how open it will be to data harvesters and data mining. We will measure the degree to which scholars are building the cyberinfrastructure by making their work not just an end in itself, but a building block for future work.

Cybermetrics and Informal Knowledge

Scholarship today is understood largely in terms of the formal genres of publication (the scholarly monograph, the academic article), and certainly traditional bibliometrics are conducted only upon these sorts of finished products. But within the new paradigm there will increasingly be more value placed upon informal and in-process knowledge: research logs, field notes, lab notebooks, and a whole range of teaching resources and media. Part of this new value will come precisely because the usage of such content can and will be measured.

What is going to happen when a teacher's online syllabus or that professor's blog or that open science notebook can be as readily monitored as the usage of a peer-reviewed article? We will see things that have normally been undervalued or seen as ephemeral having increased permanence, publicity, and use. And as faculty contribute Open Educational Resources as readily as they do Open Access publications to their institutional repositories, there wil be creative overlap (as well as vexing competition) between those intellectual assets aimed at teaching and those aimed at scholarship. Again, the measurement of usage will be central to the reshaping and evolution of various kinds of intellectual output.

Cybermetrics and Real-Time Scholarship

Data drives knowledge, and the measurement of data now possible on the web will continue to reshape the forms of knowledge and the formats of scholarship -- especially because of the ease and efficiency of measurement tools. Those tools are driving faster feedback and more granular analysis of user behavior, leading to cycles of analysis and genesis that approach real-time. A tool like Reinvigorate extends Google Analytics to make real-time feedback loops viable. Much more remains to be thought through on the topic of real-time scholarship, but I raise it here as an emerging phenomenon that, like so many other things online today, is driven not just by emerging communication tools like microblogging, but by web metrics. The networked environment is continuously evolving improved ways of tracking behavior, providing feedback, and creating value. Even though these phenomena are mostly in the enterprise space, they will certainly not remain there.

Scholarly communication -- as academia in general -- will be changing because of how knowledge and data can be measured. Cybermetrics is not some curiosity, nor is it anything close to being a modest update of bibliometrics or the ISI Impact Factor. Print-based metrics will be rapidly transcended by a richer, deeper set of measurements that will help broaden the concepts, genres, and uses of all our intellectual work.