A core-item reviewer evaluation (CoRE) system for manuscript peer review

The cross-subsidy and buy-one-give-one models of compensated peer review: A comparative study for mission-driven journals

A financial compensation model could incentivise peer reviewers to provide the optimal amount of effort by rewarding them for their quality reports. Therefore, in this article, we consider peer-reviewed journals with the mission to financially compensate reviewers for delivering quality reports and to ensure affordable access to peer review for every author. In a cross-subsidy scenario of compensated peer review, the mission-driven journals expect all authors to pay for either the standard or a premium peer review. However, using this strategy, the journals offer standard peer review to low-income authors at lower prices, compared with the premium price high-income authors pay for higher-quality peer review. In addition, we also present a buy-one-give-one model of compensated peer review for the mission-driven journals. In this alternative setting, when a high-income author pays for a premium peer review, the journals would donate a certain number of free peer reviews to the low-income authors who are unable to afford them. In this two-tiered system, scholars with access to more funding receive premium treatment, while low-income authors can still access standard peer review similar to what authors currently receive for free. In this article, we show a comparative study between the cross-subsidy and buy-one-give-one models of compensated peer review. We find that a buy-one-give-one scenario provides a higher total sum of financial and social gain than the cross-subsidy scenario either when the mission-driven journals are highly socially responsible or when the social gap between the high-income and the low-income authors is large enough.

A formal study of co-opetition in scholarly publishing

In this article, we model and study a cooperative peer-review scenario in scholarly publishing. In this scenario, the peer-reviewed journals cooperate in the necessary investment for the peer-review system. However, the final decision on what to publish in each journal would rest with the journal’s editor, and the journals still compete in their quality standards for accepting papers. This simultaneously cooperative and competitive relationship between peer-reviewed journals is co-opetition in scholarly publishing. From the comparison between a benchmark scenario of competition between journals and a cooperative peer-review setting, we find that by sharing the cost of providing a common peer-review system, the peer-reviewed journals could offer a higher review quality in the manuscript evaluation process than they would otherwise be able to achieve individually. Furthermore, we find the conditions under which the competing academic journals using cooperative peer review could increase their expected quality levels, their standards for accepting articles, and their peer-review quality, which establishes the benefit from co-opetition between peer-reviewed journals. Nevertheless, a threshold cost-sharing factor exists above which the benefit from cooperative peer review disappears.

Reimagining peer review as an expert elicitation process

Journal peer review regulates the flow of ideas through an academic discipline and thus has the power to shape what a research community knows, actively investigates, and recommends to policymakers and the wider public. We might assume that editors can identify the 'best' experts and rely on them for peer review. But decades of research on both expert decision-making and peer review suggests they cannot. In the absence of a clear criterion for demarcating reliable, insightful, and accurate expert assessors of research quality, the best safeguard against unwanted biases and uneven power distributions is to introduce greater transparency and structure into the process. This paper argues that peer review would therefore benefit from applying a series of evidence-based recommendations from the empirical literature on structured expert elicitation. We highlight individual and group characteristics that contribute to higher quality judgements, and elements of elicitation protocols that reduce bias, promote constructive discussion, and enable opinions to be objectively and transparently aggregated.

Can a paid model for peer review be sustainable when the author can decide whether to pay or not?

Given how hard it is to recruit good reviewers who are aligned with authors in their functions, journal editors could consider the use of better incentives, such as paying reviewers for their time. In order to facilitate a speedy turn-around when a rapid decision is required, the peer-reviewed journal can also offer a review model in which selected peer reviewers are compensated to deliver high-quality and timely peer-review reports. In this paper, we consider a peer-reviewed journal in which the manuscript’s evaluation consists of a necessary peer review component and an optional speedy peer review component. We model and study that journal under two different scenarios to be compared: a paid peer-reviewing scenario that is considered as the benchmark; and a hybrid peer-review scenario where the manuscript’s author can decide whether to pay or not. In the benchmark scenario of paid peer-reviewing, the scholarly journal expects all authors to pay for the peer review and charges separately for the necessary and the optional speedy peer-review components. Alternatively, in a hybrid peer-review scenario, the peer-reviewed journal gives the option to the authors to not pay for the necessary peer review if they are not able to pay. This will determine an altruistic amplification of pay utility. However, the no-pay authors cannot avail of the optional speedy peer review, which determines a restriction-induced no-pay utility reduction. In this paper, we find that under the hybrid setting of compensated peer review where the author can decide whether to pay or not, the optimal price and review quality of the optional speedy peer review are always higher than under the benchmark scenario of paid peer-reviewing, due to the altruistic amplification of pay utility. Our results show that when the advantage of adopting the hybrid mode of compensated peer review is higher due to the higher difference between the altruistic author utility amplification and the restriction-induced no-pay utility reduction, the journal can increase its profitability by increasing the price for the necessary peer review above that in the benchmark scenario of paid peer review. A key insight from our results is the journal’s capability to increase the number of paying authors by giving the option to the authors to not pay for the necessary peer review if they are not able to pay.

Duration and quality of the peer review process: the author's perspective

To gain insight into the duration and quality of the scientific peer review process, we analyzed data from 3500 review experiences submitted by authors to the SciRev.sc website. Aspects studied are duration of the first review round, total review duration, immediate rejection time, the number, quality, and difficulty of referee reports, the time it takes authors to revise and resubmit their manuscript, and overall quality of the experience. We find clear differences in these aspects between scientific fields, with Medicine, Public health, and Natural sciences showing the shortest durations and Mathematics and Computer sciences, Social sciences, Economics and Business, and Humanities the longest. One-third of journals take more than 2 weeks for an immediate (desk) rejection and one sixth even more than 4 weeks. This suggests that besides the time reviewers take, inefficient editorial processes also play an important role. As might be expected, shorter peer review processes and those of accepted papers are rated more positively by authors. More surprising is that peer review processes in the fields linked to long processes are rated highest and those in the fields linked to short processes lowest. Hence authors' satisfaction is apparently influenced by their expectations regarding what is common in their field. Qualitative information provided by the authors indicates that editors can enhance author satisfaction by taking an independent position vis-à-vis reviewers and by communicating well with authors.

On the Nature and Role of Peer Review in Mathematics

For the past three decades, peer review practices have received much attention in the literature. But although this literature covers many research fields, only one previous systematic study has been devoted to the practice of peer review in mathematics, namely a study by Geist, Löwe, and Van Kerkhove from 2010. This lack of attention may be due to a view that peer review in mathematics is more reliable, and therefore less interesting as an object of study, than peer review in other fields. In fact, Geist, Löwe, and Van Kerkhove argue that peer review in mathematics is relatively reliable. At the same time, peer review in mathematics differs from peer review in most, if not all, other fields in that papers submitted to mathematical journals are usually only reviewed by a single referee. Furthermore, recent empirical studies indicate that the referees do not check the papers line by line. I argue that, in spite of this, mathematical practice in general and refereeing practices in particular are such that the common practice of mathematical journals of using just one referee is justified from the point of view of proof validity assessment. The argument is based on interviews I conducted with seven mathematicians.

Validation of the CoRE Questionnaire for a Medical Journal Peer Review

If a peer review instrument asks concrete questions (defined as items that can only generate disagreement if reviewers have different degrees of expertise), then questionnaires could become more meaningful in terms of resolving subjectivity thus leading to more reviewer agreement. A concrete item questionnaire with well-chosen questions can also help resolve disagreement when reviewers have the same level of expertise. We have recently created the core-item reviewer evaluation (CoRE) questionnaire for which decision-threshold score levels have been created, but which have not been validated. This prospective validation of these thresholds for the CoRE questionnaire demonstrated strong agreement between reviewer recommendations and their reported score levels when tested prospectively at Clinical Medicine and Research. We conclude that using the CoRE questionnaire will help reduce peer reviewer disagreement. More importantly, when reviewer expertise varies, editors can more easily detect this and decide which opinion reflects the greater expertise.

How to review a scientific paper

Scientific observations must survive the scrutiny of experts before they are disseminated to the broader community because their publication in a scientific journal provides a stamp of validity. Although critical review of a manuscript by peers prior to publication in a scientific journal is a central element in this process, virtually no formal guidance is provided to reviewers about the nature of the task. In this article, the essence of peer review is described and critical steps in the process are summarized. The role of the peer reviewer as an intermediary and arbiter in the process of scientific communication between the authors and the readers via the vehicle of the particular journal is discussed and the responsibilities of the reviewer to each of the three parties (the author/s, readers, and the Journal editor) are defined. The two formal products of this activity are separate sets of reviewer comments to the editor and the authors and these are described. Ethical aspects of the process are considered and rewards accruing to the reviewer summarized.