Q&A: Re-review opt-out and painless publishing

Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication

The debate on whether the peer-review system is in crisis has been heated recently. A variety of alternative systems have been proposed to improve the system and make it sustainable. However, we lack sufficient evidence and data related to these issues. Here we used a previously developed agent-based model of the scientific publication and peer-review system calibrated with empirical data to compare the efficiency of five alternative peer-review systems with the conventional system. We modelled two systems of immediate publication, with and without online reviews (crowdsourcing), a system with only one round of reviews and revisions allowed (re-review opt-out) and two review-sharing systems in which rejected manuscripts are resubmitted along with their past reviews to any other journal (portable) or to only those of the same publisher but of lower impact factor (cascade). The review-sharing systems outperformed or matched the performance of the conventional one in all peer-review efficiency, reviewer effort and scientific dissemination metrics we used. The systems especially showed a large decrease in total time of the peer-review process and total time devoted by reviewers to complete all reports in a year. The two systems with immediate publication released more scientific information than the conventional one but provided almost no other benefit. Re-review opt-out decreased the time reviewers devoted to peer review but had lower performance on screening papers that should not be published and relative increase in intrinsic quality of papers due to peer review than the conventional system. Sensitivity analyses showed consistent findings to those from our main simulations. We recommend prioritizing a system of review-sharing to create a sustainable scientific publication and peer-review system.

The Global Burden of Journal Peer Review in the Biomedical Literature: Strong Imbalance in the Collective Enterprise

The growth in scientific production may threaten the capacity for the scientific community to handle the ever-increasing demand for peer review of scientific publications. There is little evidence regarding the sustainability of the peer-review system and how the scientific community copes with the burden it poses. We used mathematical modeling to estimate the overall quantitative annual demand for peer review and the supply in biomedical research. The modeling was informed by empirical data from various sources in the biomedical domain, including all articles indexed at MEDLINE. We found that for 2015, across a range of scenarios, the supply exceeded by 15% to 249% the demand for reviewers and reviews. However, 20% of the researchers performed 69% to 94% of the reviews. Among researchers actually contributing to peer review, 70% dedicated 1% or less of their research work-time to peer review while 5% dedicated 13% or more of it. An estimated 63.4 million hours were devoted to peer review in 2015, among which 18.9 million hours were provided by the top 5% contributing reviewers. Our results support that the system is sustainable in terms of volume but emphasizes a considerable imbalance in the distribution of the peer-review effort across the scientific community. Finally, various individual interactions between authors, editors and reviewers may reduce to some extent the number of reviewers who are available to editors at any point.

Why training and specialization is needed for peer review: a case study of peer review for randomized controlled trials

BACKGROUND

The purpose and effectiveness of peer review is currently a subject of hot debate, as is the need for greater openness and transparency in the conduct of clinical trials. Innovations in peer review have focused on the process of peer review rather than its quality.

DISCUSSION

The aims of peer review are poorly defined, with no evidence that it works and no established way to provide training. However, despite the lack of evidence for its effectiveness, evidence-based medicine, which directly informs patient care, depends on the system of peer review. The current system applies the same process to all fields of research and all study designs. While the volume of available health related information is vast, there is no consistent means for the lay person to judge its quality or trustworthiness. Some types of research, such as randomized controlled trials, may lend themselves to a more specialized form of peer review where training and ongoing appraisal and revalidation is provided to individuals who peer review randomized controlled trials. Any randomized controlled trial peer reviewed by such a trained peer reviewer could then have a searchable 'quality assurance' symbol attached to the published articles and any published peer reviewer reports, thereby providing some guidance to the lay person seeking to inform themselves about their own health or medical treatment.

SUMMARY

Specialization, training and ongoing appraisal and revalidation in peer review, coupled with a quality assurance symbol for the lay person, could address some of the current limitations of peer review for randomized controlled trials.