This post is only tangentially about open science. It is more directly about the process of peer review and how it might be improved. I am working on a follow-up post about how these points can be addressed in an open publishing environment.

A recent paper on the arXiv got me thinking about the sticking points in the publishing pipeline. As it stands, most scientists have a pretty good understanding of how peer reviewed publishing is supposed to work. Once an author—or more likely, a group of authors—decides that a manuscript is ready for action, the following series of events will occur:

  1. the authors submit the manuscript to the journal of choice;
  2. the journal’s editor makes a general decision about whether the article is appropriate for the journal;
  3. in the affirmative case, the editor selects referees for the manuscript and sends them the text for review;
  4. the referees return reviews of the manuscript (the referees are not typically identified to the authors);
  5. the editor makes the decision to reject the manuscript, accept it with minor revisions, or accept it with major revisions. Rejected manuscripts usually start over the process in another journal. Minor revisions to accepted manuscripts are usually made quickly and publication proceeds. In the case of major revisions, the suggested changes are made, if possible, and the manuscript is returned to the editor. At this point, the referees may get a second crack at the material (but not necessarily), before the editor makes a final accept/reject decision based on the feedback from the referees.

Peer review of manuscripts exists for several reasons. For one, self-regulation determines the suitability of the material for publication if it was not already obvious to the editor of the journal. Having peer reviewers also improves the material and its presentation. Furthermore, having expert reviewers lends credibility to the work and insures that misleading, wrong, or crackpot material does not receive the stamp of credibility. Finally, finding appropriately skilled referees spreads the workload beyond the editors, who may not have the resources to evaluate every paper arriving at their desk.

Though peer review has a storied history, it also has its drawbacks. First, and perhaps foremost, the process is often a slow one, with many months elapsing during even one round of communications between the authors, the editor, and the referees. Peer review is not always an objective process either: referees have the power to delay, or outright reject, work that their competitors have completed, and thus they may lose their impartiality in the process. Additionally, the publishing process does not reveal the feedback process that occurs between authors and referees, which can be a scientifically and pedagogically valuable exchange.

One proposal to address the shortcomings of the peer review process (alluded to in the first paragraph) was posted by Sergey Bozhevolnyi on the arXiv, a pre-publication website for many physics-related manuscripts. Bozhevolnyi calls his model of publishing Rapid, Impartial, and Comprehensive (RIC) publishing. To him, “rapid” means that editors should approve or reject manuscripts before the manuscripts are sent to the referees for review. Then, “impartial” means that referees, who might otherwise have an interest in rejecting a perfectly fine paper, lose the power to dictate whether or not a manuscript is published. Instead, the referees critique the paper without assessing whether it is publication-worthy. Lastly, “comprehensive” involves publishing everything having to do with the manuscript. That is, all positive and negative reviews are published in conjunction with the all versions of a manuscript.

The primary benefit of RIC, according to Bozhevolnyi, is that it saves the energies of authors, editors, and referees, thus allowing them all to do more research and less wrangling. Since most papers are ultimately accepted somewhere, then we should not cause additional delays in publishing by first rejecting them in multiple places. Instead, collate the manuscript and the reviews and publish them all together, along with any revisions to the manuscript. Having the reviews be publicly viewable will encourage referees to be more careful about writing their critiques and supporting their assertions, and the process as a whole will be more transparent than it currently is.

Before I critique the RIC publishing proposal, I should point out that some aspects of the proposal are very appealing. I particularly like the idea of publishing all reviews in addition to the manuscript. That said, I find it difficult to believe that the incentives for authors and referees change for the better under this proposal. For example, what happens if authors receive feedback, do not wish to invest the time to address the critique, and subsequently allow the original manuscript and the reviews to stand as they are? This situation seems like a moral hazard for authors that does a disservice to the quality of scientific literature. On the part of the referees, does removing decision-making authority make reviewing less appealing? Disempowering the referees by potentially ignoring their critique and only counting it as a minor part of the publishing process will not motivate them to write better reviews. In the case of editors, what makes us believe that an editor, or an editorial board, has the background to properly evaluate the suitability of work for acceptance into a journal? The reason we have referees in the current peer review system is because they have the very expertise and familiarity needed for this task.

Does the fact that Bozhevolnyi’s RIC proposal does not make sense mean that peer review is fine as it is? I do not think so. Instead, it is worth asking what parts of peer review we like and what parts we would like to improve. I posit that rejection, or the threat of rejection, is the greatest motivator for authors to make necessary changes to their manuscript. As such, rejection by peers is still the best way to require and receive revisions. Though I think that referees should retain their rejecting power (and their anonymity!), I feel strongly that the entire peer review process would benefit from the increased transparency and accountability that publishing unsigned reviews would add. As far as editors, they play a role in shaping the kind of journal they run by selecting appropriate material on a general level, but they should not play too large a role in determining the “important” research in any field. The model used by the journal [Public Library of Science One][] is a promising one in this regard, with the only acceptance criterion being whether the science is sound.

The amount of time that it takes to publish is one of the most frustrating aspects of peer review, however. Journals could voluntarily publish time-to-publication figures, a number which could then be used by authors—along with impact factors and acceptance rates—to decide which journals to submit to. For instance, an editor of the Journal of Orthodontics writes about just this fact in an editorial. A Google search for “journal time to publication” reveals that people have been thinking about this problem for a while (e.g. computer science comparisons), but no general standard exists across journals. In fact, I suspect these are numbers most journals are afraid will hurt them more than help them. Nevertheless, journals acknowledge the demand for rapid publication when they offer services like [Springer’s Fast Track publishingfasttrackpub or Physical Review’s Rapid Communications.

Ultimately, it may not matter what journals do because authors are routing around this problem via pre-publication archives such as the arXiv for physics-related subject matter. Though not without complications, especially in the health sciences (see, for example, “The Promise and Perils of Pre-Publication Review”), pre-publication allows authors to communicate results and establish priority without stressing about getting through the peer review process as fast as possible. Instead, the process takes its normal, slower course while authors move along their on-going research.

I will conclude by leaving an open question that I may address in a future post: how do you encourage peer reviewers to do the best possible job, in a timely manner, without only relying on their altruism to doing good science and being good members of the community? It is this question about peer review, I feel, that is the most fraught with complication and subject to the law of unintended consequences if the incentives are changed.