A multi-disciplinary perspective on emergent and future innovations in peer review

Ponderings on peer review. Part 2. Manuscript critiques

In part 1 of this Perspective, I discussed general principles of scientific peer review in the biomedical sciences aimed at early-stage investigators (i.e., graduate students, postdoctoral fellows, and junior faculty). Here in part 2, I share my thoughts specifically on the topic of peer review of manuscripts. I begin by defining manuscript peer review and discussing the goals and importance of the concept. I then describe the organizational structure of the process, including the two distinct stages involved. Next, I emphasize several important considerations for manuscript reviewers, both general points and key considerations when evaluating specific types of papers, including original research manuscripts, reviews, methods articles, and opinion pieces. I then advance some practical suggestions for developing the written critique document, offer advice for making an overall recommendation to the editor (i.e., accept, revise, reject), and describe the unique issues involved when assessing a revised manuscript. Finally, I comment on how best to gain experience in the essential academic research skill of manuscript peer review. In part 3 of the series, I will discuss the topic of reviewing grant applications submitted to research funding agencies.

Unblocking recognition: A token system for acknowledging academic contribution

Here we present a blockchain-backed token recognition system to reward the contributions that academics make to the scientific ecosystem. Recognition is important in science but current methods, systems and incentives are limited. Specifically, the traditional focus on narrow publication metrics means diverse contributions are not captured, while bias toward senior, established scientists is common. To tackle this challenge, we explore the potential of harnessing blockchain’s collaborative, decentralised and trust-brokering properties to develop a token reward system for use by research funders. Academics would be awarded tokens for undertaking common but vital tasks such as peer review, sitting on funding committees and submitting reports. These tokens would not be tradable or specifically monetisable but would serve as a validated record of scientific contribution. They would have value in professional recruitment and job placement, support grant and award applications, and inform performance appraisals and file reviews. Coordination and cooperation across multiple funding agencies in developing the platform would provide an opportunity to aggregate and standardise recognition, given academics often work with several funders. This system’s goals are to expand recognition metrics, promote efficiencies, improve the robustness of professional assessments and enable cross-funder collaboration, thereby optimising research processes and practices in a decentralised and democratised manner.

Which peer reviewers voluntarily reveal their identity to authors? Insights into the consequences of open-identities peer review

Identifying reviewers is argued to improve the quality and fairness of peer review, but is generally disfavoured by reviewers. To gain some insight into the factors that influence when reviewers are willing to have their identity revealed, I examined which reviewers voluntarily reveal their identities to authors at the journal Functional Ecology, at which reviewer identities are confidential unless reviewers sign their comments to authors. I found that 5.6% of reviewers signed their comments to authors. This proportion increased slightly over time, from 4.4% in 2003-2005 to 6.7% in 2013-2015. Male reviewers were 1.8 times more likely to sign their comments to authors than were female reviewers, and this difference persisted over time. Few reviewers signed all of their reviews; reviewers were more likely to sign their reviews when their rating of the manuscript was more positive, and papers that had at least one signed review were more likely to be invited for revision. Signed reviews were, on average, longer and recommended more references to authors. My analyses cannot distinguish cause and effect for the patterns observed, but my results suggest that 'open-identities' review, in which reviewers are not permitted to be anonymous, will probably reduce the degree to which reviewers are critical in their assessment of manuscripts and will differentially affect recruitment of male and female reviewers, negatively affecting the diversity of reviewers recruited by journals.

Open Accessibility in Education Research: Enhancing the Credibility, Equity, Impact, and Efficiency of Research

Openness is a foundational principle in science. Making the tools and products of scientific research openly accessible advances core aims and values of education researchers, such as the credibility, equity, impact, and efficiency of research. The digital revolution has expanded opportunities for providing greater access to research. In this article, we examine three open-science practices-open data and code, open materials, and open access-that education researchers can use to increase accessibility to the tools and products of research in the field. For each open-science practice, we discuss what the practice is and how it works, its primary benefits, some important limitations and challenges, and two thorny issues.

Peer review analysis in the field of radiation oncology: results from a web-based survey of the Young DEGRO working group

Purpose

To evaluate the reviewing behaviour in the German-speaking countries in order to provide recommendations to increase the attractiveness of reviewing activity in the field of radiation oncology.

Methods

In November 2019, a survey was conducted by the Young DEGRO working group (jDEGRO) using the online platform “eSurveyCreator”. The questionnaire consisted of 29 items examining a broad range of factors that influence reviewing motivation and performance.

Results

A total of 281 responses were received. Of these, 154 (55%) were completed and included in the evaluation. The most important factors for journal selection criteria and peer review performance in the field of radiation oncology are the scientific background of the manuscript (85%), reputation of the journal (59%) and a high impact factor (IF; 40%). Reasons for declining an invitation to review include the scientific background of the article (60%), assumed effort (55%) and a low IF (27%). A double-blind review process is preferred by 70% of respondents to a single-blind (16%) or an open review process (14%). If compensation was offered, 59% of participants would review articles more often. Only 12% of the participants have received compensation for their reviewing activities so far. As compensation for the effort of reviewing, 55% of the respondents would prefer free access to the journal’s articles, 45% a discount for their own manuscripts, 40% reduced congress fees and 39% compensation for expenses.

Conclusion

The scientific content of the manuscript, reputation of the journal and a high IF determine the attractiveness for peer reviewing in the field of radiation oncology. The majority of participants prefer a double-blind peer review process and would conduct more reviews if compensation was available. Free access to journal articles, discounts for publication costs or congress fees, or an expense allowance were identified to increase attractiveness of the review process.

Supplementary Information

The online version of this article (10.1007/s00066-020-01729-2) contains supplementary material, which is available to authorized users.

The state of the art in peer review

Scholarly communication is in a perpetual state of disruption. Within this, peer review of research articles remains an essential part of the formal publication process, distinguishing it from virtually all other modes of communication. In the last several years, there has been an explosive wave of innovation in peer review research, platforms, discussions, tools and services. This is largely coupled with the ongoing and parallel evolution of scholarly communication as it adapts to rapidly changing environments, within what is widely considered as the ‘open research’ or ‘open science’ movement. Here, we summarise the current ebb and flow around changes to peer review and consider its role in a modern digital research and communications infrastructure and suggest why uptake of new models of peer review appears to have been so low compared to what is often viewed as the ‘traditional’ method of peer review. Finally, we offer some insight into the potential futures of scholarly peer review and consider what impacts this might have on the broader scholarly research ecosystem. In particular, we focus on the key traits of certification and reputation, moderation and quality control and engagement incentives, and discuss how these interact with socio-technical aspects of peer review and academic culture.

Collaboration, competition and publication in toxicology: views of British Toxicology Society members

BTS members offer their opinions and interesting suggestions for improvement around resourcing, collaboration, competition, infrastructure and peer review in toxicology.

To ascertain attitudes to resourcing, collaboration and publication in toxicology, a survey was developed and distributed to British Toxicology Society (BTS) members. The survey comprised 14 questions with 5 response options (strongly agree; agree; conflicted; disagree; strongly disagree) and a free text box. One hundred completed surveys were received by the cut-off date for data analysis. Unsurprisingly, 60% of participants disagreed or strongly disagreed that toxicology research is adequately funded in the UK; only 12% agreed with this statement. A similar proportion of participants (53%) disagreed with the statement that funding councils give equal opportunity to toxicology whereas 31% were conflicted on this point. An overwhelming 97% of respondents agreed that collaboration is important in driving toxicology research whereas only 38% agreed that competition is important. When this question was broadened out beyond the discipline of toxicology, a similar profile was seen suggesting that participants held similar views on toxicology versus other types of research. Many respondents were conflicted regarding the role of competition both in toxicology and in other research disciplines. Free text comments suggested that some competition is good to drive quality but can be counterproductive when competing for limited resources. Most participants were in favour of making toxicology research data openly available (86%) and in favour of open access publication (89%) although there were reservations about the cost of open access. Many (60%) thought the current system of peer review is fair but 65% also supported the idea of double-blind peer review (where both reviewer and author are anonymized). Others suggested a step in the opposite direction towards increased transparency (revealing and holding reviewers to account) would be preferable. Overall, there was a broad theme in free text responses that the need for experienced toxicologists has increased at a time when training and investment in the discipline has declined. However, not all respondents held that view with some noting that toxicology both as a research and as an applied discipline is strong within the UK scientific community.

Ten Hot Topics around Scholarly Publishing

The changing world of scholarly communication and the emerging new wave of ‘Open Science’ or ‘Open Research’ has brought to light a number of controversial and hotly debated topics. Evidence-based rational debate is regularly drowned out by misinformed or exaggerated rhetoric, which does not benefit the evolving system of scholarly communication. This article aims to provide a baseline evidence framework for ten of the most contested topics, in order to help frame and move forward discussions, practices, and policies. We address issues around preprints and scooping, the practice of copyright transfer, the function of peer review, predatory publishers, and the legitimacy of ‘global’ databases. These arguments and data will be a powerful tool against misinformation across wider academic research, policy and practice, and will inform changes within the rapidly evolving scholarly publishing system.

Is Biomedical Research Protected from Predatory Reviewers?

Authors endure considerable hardship carrying out biomedical research, from generating ideas to completing their manuscripts and submitting their findings and data (as is increasingly required) to a journal. When researchers submit to journals, they entrust their findings and ideas to editors and peer reviewers who are expected to respect the confidentiality of peer review. Inherent trust in peer review is built on the ethical conduct of authors, editors and reviewers, and on the respect of this confidentiality. If such confidentiality is breached by unethical reviewers who might steal or plagiarize the authors' ideas, researchers will lose trust in peer review and may resist submitting their findings to that journal. Science loses as a result, scientific and medical advances slow down, knowledge may become scarce, and it is unlikely that increasing bias in the literature will be detected or eliminated. In such a climate, society will ultimately be deprived from scientific and medical advances. Despite a rise in documented cases of abused peer review, there is still a relative lack of qualitative and quantitative studies on reviewer-related misconduct, most likely because evidence is difficult to come by. Our paper presents an assessment of editors' and reviewers' responsibilities in preserving the confidentiality of manuscripts during the peer review process, in response to a 2016 case of intellectual property theft by a reviewer. Our main objectives are to propose additional measures that would offer protection of authors' intellectual ideas from predatory reviewers, and increase researchers' awareness of the responsible reviewing of journal articles and reporting of biomedical research.

Improving transparency and scientific rigor in academic publishing

Progress in basic and clinical research is slowed when researchers fail to provide a complete and accurate report of how a study was designed, executed, and the results analyzed. Publishing rigorous scientific research involves a full description of the methods, materials, procedures, and outcomes. Investigators may fail to provide a complete description of how their study was designed and executed because they may not know how to accurately report the information or the mechanisms are not in place to facilitate transparent reporting. Here, we provide an overview of how authors can write manuscripts in a transparent and thorough manner. We introduce a set of reporting criteria that can be used for publishing, including recommendations on reporting the experimental design and statistical approaches. We also discuss how to accurately visualize the results and provide recommendations for peer reviewers to enhance rigor and transparency. Incorporating transparency practices into research manuscripts will significantly improve the reproducibility of the results by independent laboratories. SIGNIFICANCE: Failure to replicate research findings often arises from errors in the experimental design and statistical approaches. By providing a full account of the experimental design, procedures, and statistical approaches, researchers can address the reproducibility crisis and improve the sustainability of research outcomes. In this piece, we discuss the key issues leading to irreproducibility and provide general approaches to improving transparency and rigor in reporting, which could assist in making research more reproducible.

The effect of publishing peer review reports on referee behavior in five scholarly journals

To increase transparency in science, some scholarly journals are publishing peer review reports. But it is unclear how this practice affects the peer review process. Here, we examine the effect of publishing peer review reports on referee behavior in five scholarly journals involved in a pilot study at Elsevier. By considering 9,220 submissions and 18,525 reviews from 2010 to 2017, we measured changes both before and during the pilot and found that publishing reports did not significantly compromise referees' willingness to review, recommendations, or turn-around times. Younger and non-academic scholars were more willing to accept to review and provided more positive and objective recommendations. Male referees tended to write more constructive reports during the pilot. Only 8.1% of referees agreed to reveal their identity in the published report. These findings suggest that open peer review does not compromise the process, at least when referees are able to protect their anonymity.

Preprints in Scholarly Communication: Re-Imagining Metrics and Infrastructures

Digital scholarship and electronic publishing within scholarly communities change when metrics and open infrastructures take center stage for measuring research impact. In scholarly communication, the growth of preprint repositories as a new model of scholarly publishing over the last three decades has been one of the major developments. As it unfolds, the landscape of scholarly communication is transitioning—with much being privatized as it is made open—and turning towards alternative metrics, such as social media attention, author-level, and article-level metrics. Moreover, the granularity of evaluating research impact through new metrics and social media changes the objective standards of evaluating research performance. Using preprint repositories as a case study, this article situates them in a scholarly web, examining their salient features, benefits, and futures. Moves towards scholarly web development and publishing on the semantic and social web with open infrastructures, citations, and alternative metrics—how preprints advance building the web as data—is discussed. We determine that this will viably demonstrate new metrics and, by enhancing research publishing tools in the scholarly commons, facilitate various communities of practice. However, for preprint repositories to be sustainable, scholarly communities and funding agencies should support continued investment in open knowledge, alternative metrics development, and open infrastructures in scholarly publishing.

Improving transparency and scientific rigor in academic publishing

Progress in basic and clinical research is slowed when researchers fail to provide a complete and accurate report of how a study was designed, executed, and the results analyzed. Publishing rigorous scientific research involves a full description of the methods, materials, procedures, and outcomes. Investigators may fail to provide a complete description of how their study was designed and executed because they may not know how to accurately report the information or the mechanisms are not in place to facilitate transparent reporting. Here, we provide an overview of how authors can write manuscripts in a transparent and thorough manner. We introduce a set of reporting criteria that can be used for publishing, including recommendations on reporting the experimental design and statistical approaches. We also discuss how to accurately visualize the results and provide recommendations for peer reviewers to enhance rigor and transparency. Incorporating transparency practices into research manuscripts will significantly improve the reproducibility of the results by independent laboratories.

The changing forms and expectations of peer review

The quality and integrity of the scientific literature have recently become the subject of heated debate. Due to an apparent increase in cases of scientific fraud and irreproducible research, some have claimed science to be in a state of crisis. A key concern in this debate has been the extent to which science is capable of self-regulation. Among various mechanisms, the peer review system in particular is considered an essential gatekeeper of both quality and sometimes even integrity in science. However, the allocation of responsibility for integrity to the peer review system is fairly recent and remains controversial. In addition, peer review currently comes in a wide variety of forms, developed in the expectation they can address specific problems and concerns in science publishing. At present, there is a clear need for a systematic analysis of peer review forms and the concerns underpinning them, especially considering a wave of experimentation fuelled by internet technologies and their promise to improve research integrity and reporting. We describe the emergence of current peer review forms by reviewing the scientific literature on peer review and by adding recent developments based on information from editors and publishers. We analyse the rationale for developing new review forms and discuss how they have been implemented in the current system. Finally, we give a systematisation of the range of discussed peer review forms. We pay detailed attention to the emergence of the expectation that peer review can maintain 'the integrity of science's published record', demonstrating that this leads to tensions in the academic debate about the responsibilities and abilities of the peer review system.

The peer review game: an agent-based model of scientists facing resource constraints and institutional pressures

This paper looks at peer review as a cooperation dilemma through a game-theory framework. We built an agent-based model to estimate how much the quality of peer review is influenced by different resource allocation strategies followed by scientists dealing with multiple tasks, i.e., publishing and reviewing. We assumed that scientists were sensitive to acceptance or rejection of their manuscripts and the fairness of peer review to which they were exposed before reviewing. We also assumed that they could be realistic or excessively over-confident about the quality of their manuscripts when reviewing. Furthermore, we assumed they could be sensitive to competitive pressures provided by the institutional context in which they were embedded. Results showed that the bias and quality of publications greatly depend on reviewer motivations but also that context pressures can have a negative effect. However, while excessive competition can be detrimental to minimising publication bias, a certain level of competition is instrumental to ensure the high quality of publication especially when scientists accept reviewing for reciprocity motives.

Reputation or peer review? The role of outliers

We present an agent-based model of paper publication and consumption that allows to study the effect of two different evaluation mechanisms, peer review and reputation, on the quality of the manuscripts accessed by a scientific community. The model was empirically calibrated on two data sets, mono- and multi-disciplinary. Our results point out that disciplinary settings differ in the rapidity with which they deal with extreme events-papers that have an extremely high quality, that we call outliers. In the mono-disciplinary case, reputation is better than traditional peer review to optimize the quality of papers read by researchers. In the multi-disciplinary case, if the quality landscape is relatively flat, a reputation system also performs better. In the presence of outliers, peer review is more effective. Our simulation suggests that a reputation system could perform better than peer review as a scientific information filter for quality except when research is multi-disciplinary and in a field where outliers exist.

Journal peer review: a bar or bridge? An analysis of a paper's revision history and turnaround time, and the effect on citation

Journal peer review lies at the heart of academic quality control. This article explores the journal peer review process and seeks to examine how the reviewing process might itself contribute to papers, leading them to be more highly cited and to achieve greater recognition. Our work builds on previous observations and views expressed in the literature about (a) the role of actors involved in the research and publication process that suggest that peer review is inherent in the research process and (b) on the contribution reviewers themselves might make to the content and increased citation of papers. Using data from the journal peer review process of a single journal in the Social Sciences field (Business, Management and Accounting), we examine the effects of peer review on papers submitted to that journal including the effect upon citation, a novel step in the study of the outcome of peer review. Our detailed analysis suggests, contrary to initial assumptions, that it is not the time taken to revise papers but the actual number of revisions that leads to greater recognition for papers in terms of citation impact. Our study provides evidence, albeit limited to the case of a single journal, that the peer review process may constitute a form of knowledge production and is not the simple correction of errors contained in submitted papers.

Decisions, decisions

Journals are exploring new approaches to peer review in order to reduce bias, increase transparency and respond to author preferences. Funders are also getting involved.

What is open peer review? A systematic review

Background: “Open peer review” (OPR), despite being a major pillar of Open Science, has neither a standardized definition nor an agreed schema of its features and implementations. The literature reflects this, with numerous overlapping and contradictory definitions. While for some the term refers to peer review where the identities of both author and reviewer are disclosed to each other, for others it signifies systems where reviewer reports are published alongside articles. For others it signifies both of these conditions, and for yet others it describes systems where not only “invited experts” are able to comment. For still others, it includes a variety of combinations of these and other novel methods.

Methods: Recognising the absence of a consensus view on what open peer review is, this article undertakes a systematic review of definitions of “open peer review” or “open review”, to create a corpus of 122 definitions. These definitions are systematically analysed to build a coherent typology of the various innovations in peer review signified by the term, and hence provide the precise technical definition currently lacking.

Results: This quantifiable data yields rich information on the range and extent of differing definitions over time and by broad subject area. Quantifying definitions in this way allows us to accurately portray exactly how ambiguously the phrase “open peer review” has been used thus far, for the literature offers 22 distinct configurations of seven traits, effectively meaning that there are 22 different definitions of OPR in the literature reviewed.

Conclusions: I propose a pragmatic definition of open peer review as an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the aims of Open Science, including making reviewer and author identities open, publishing review reports and enabling greater participation in the peer review process.