What is open peer review? A systematic review

Reviewing manuscripts for scientific journals: recommendations for early career scientists

Publication of articles in international scientific journals has been one of the main strategies for the communication of scientific findings and ideas. Prepublication peer review is a fundamental aspect of the publishing process in indexed scientific journals and, associated with the large growth in journals and articles, there has been a recent challenge in having adequate peer reviewers for international journals. In this article, we provide a short overview of the publishing process, give recommendations to early career researchers about writing peer reviews of adequate quality, and discuss some possibilities for the future.

Nine quick tips for open meta-analyses

Open science principles are revolutionizing the transparency, reproducibility, and accessibility of research. Meta-analysis has become a key technique for synthesizing data across studies in a principled way; however, its impact is contingent on adherence to open science practices. Here, we outline 9 quick tips for open meta-analyses, aimed at guiding researchers to maximize the reach and utility of their findings. We advocate for outlining preregistering clear protocols, opting for open tools and software, and the use of version control systems to ensure transparency and facilitate collaboration. We further emphasize the importance of reproducibility, for example, by sharing search syntax and analysis scripts, and discuss the benefits of planning for dynamic updating to enable living meta-analyses. We also recommend publication in open-access formats, as well as open data, open code, and open access publication. We close by encouraging active promotion of research findings to bridge the gap between complex syntheses and public discourse, and provide a detailed submission checklist to equip researchers, reviewers and journal editors with a structured approach to conducting and reporting open meta-analyses.

Messing with Merton: The intersection between open science practices and Mertonian values

Although adherence to Mertonian values of science (i.e., communism, universalism, organized skepticism, disinterestedness) is desired and promoted in academia, such adherence can cause friction with the normative structures and practices of Open Science. Mertonian values and Open Science practices aim to improve the conduct and communication of research and are promoted by institutional actors. However, Mertonian values remain mostly idealistic and contextualized in local and disciplinary cultures and Open Science practices rely heavily on third-party resources and technology that are not equally accessible to all parties. Furthermore, although still popular, Mertonian values were developed in a different institutional and political context. In this article, we argue that new normative structures for science need to look beyond nostalgia and consider aspirations and outcomes of Open Science practices. To contribute to such a vision, we explore the intersection of several Open Science practices with Mertonian values to flesh out challenges involved in upholding these values. We demonstrate that this intersection becomes complicated when the interests of numerous groups collide and contrast. Acknowledging and exploring such tensions informs our understanding of researchers' behavior and supports efforts that seek to improve researchers' interactions with other normative structures such as research ethics and integrity frameworks.

The Peer Review Process: Past, Present, and Future

The peer review process is a fundamental aspect of modern scientific paper publishing, underpinning essential quality control. First conceptualised in the 1700s, it is an iterative process that aims to elevate scientific literature to the highest standards whilst preventing publication of scientifically unsound, potentially misleading, and even plagiarised information. It is widely accepted that the peer review of scientific papers is an irreplaceable and fundamental aspect of the research process. However, the rapid growth of research and technology has led to a huge increase in the number of publications. This has led to increased pressure on the peer review system. There are several established peer review methodologies, ranging from single and double blind to open and transparent review, but their implementation across journals and research fields varies greatly. Some journals are testing entirely novel approaches (such as collaborative reviews), whilst others are piloting changes to established methods. Given the unprecedented growth in publication numbers, and the ensuing burden on journals, editors, and reviewers, it is imperative to improve the quality and efficiency of the peer review process. Herein we evaluate the peer review process, from its historical origins to current practice and future directions.

Cross-Sectional Evaluation of Open Science Practices at Imaging Journals: A Meta-Research Study

Objective: To evaluate open science policies of imaging journals, and compliance to these policies in published articles. Methods: From imaging journals listed we extracted open science policy details: protocol registration, reporting guidelines, funding, ethics and conflicts of interest (COI), data sharing, and open access publishing. The 10 most recently published studies from each journal were assessed to determine adherence to these policies. We calculated the proportion of open science policies into an Open Science Score (OSS) for all journals and articles. We evaluated relationships between OSS and journal/article level variables. Results: 82 journals/820 articles were included. The OSS of journals and articles was 58.3% and 31.8%, respectively. Of the journals, 65.9% had registration and 78.1% had reporting guideline policies. 79.3% of journals were members of COPE, 81.7% had plagiarism policies, 100% required disclosure of funding, and 97.6% required disclosure of COI and ethics approval. 81.7% had data sharing policies and 15.9% were fully open access. 7.8% of articles had a registered protocol, 8.4% followed a reporting guideline, 77.4% disclosed funding, 88.7% disclosed COI, and 85.6% reported ethics approval. 12.3% of articles shared their data. 51% of articles were available through open access or as a preprint. OSS was higher for journal with DOAJ membership (80% vs 54.2%; P < .0001). Impact factor was not correlated with journal OSS. Knowledge synthesis articles has a higher OSS scores (44.5%) than prospective/retrospective studies (32.6%, 30.0%, P < .0001). Conclusion: Imaging journals endorsed just over half of open science practices considered; however, the application of these practices at the article level was lower.

Multifaceted Scoping Review of Black/African American Transportation and Land Use Expert Recommendations on Activity-Friendly Routes to Everyday Destinations

Promoting physical activity (PA) at the community level is a complex, multisector approach requiring researchers and practitioners to impact the individual, interpersonal, environment, and policy levels. One such strategy aiming to impact systems, policies, and environments is the Centers for Disease Control and Prevention's Activity Friendly Routes to Everyday Destinations (Routes to Destinations). This strategy specifically aims to connect pedestrian, bicycle, and public transportation systems with built environment and land use destinations. This article examines Black/African American transportation and land use experts' perspectives and concerns-across multiple mediums-around inequities that have discouraged PA among Black/African American persons specifically Routes to Destinations strategies. In March 2021, a multifaceted scoping review was conducted of peer-reviewed literature, gray literature, and social media authored by Black/African American transportation and land use experts focusing on policy, system, and environmental changes which promote or discourage equitable and inclusive access to physical activity. Themes from peer-reviewed and gray literature resources included: (1) Assessing Racism, Discrimination, and Segregation; (2) Addressing Equity and Inclusion Through Policy; (3) Community Engagement and Place-Based Interventions; (4) Infrastructure Changes; (5) Safety; and (6) Reporting Health Disparities. Twitter topic models suggested the main topics included elements of race/racism, equity, safety, infrastructure, and advancing social justice. Experts called for systemic and systematic change through new policies and implementation of existing policies as well as enhanced community inclusion in decision-making through ownership of policy and built environment change. Safety was discussed differently between peer-reviewed and gray literature and Twitter discussions indicating a publication bias.

Research misconduct as a challenge for academic institutions and scientific journals

Research misconduct refers to a set of unethical practices in research and publication and is the main reason for retraction of articles published in the academic literature. Research misconduct has negative consequences and has generated public scepticism about research, which has led to increasing distrust in science. In this context, a joint effort by the scientific community, academic institutions, scientific journals and research funders is needed to identify and prevent research misconduct. In this paper, we will evaluate what has already been done and what is needed to do to better address research misconduct. The focus of this paper will be on the actions taken by academic institutions, as the first line of defence, and scientific journals, as the gatekeepers of science. However, scientific journals and academic institutions are only a part of a much larger and multistakeholder effort needed to address the challenges scientific research is facing.

The consistency of peer-reviewers: Assessment of separate parts of the manuscripts vs final recommendations

Delving into the review reports, this paper is aimed at analyzing reviewers` attitudes toward different sections of the manuscripts they review. The research focuses on the consistency of reviewers` evaluation through analysis of their assessment of separate parts of a paper, if it corresponds with the recommendations they made to the editors and whether a paper needs revision or should be accepted/rejected. It is assumed that the assessment of separate parts of a paper should be consistent with the final decision regarding the acceptance or rejection of a manuscript. Based on the analysis presented in this paper it can be concluded that the assessments of separate parts of articles in the evaluation sheets do not fully reflect the final recommendations of the reviewers. The results showed that the most correlated and therefore the most significant sections for the reviewers are the main text and the conclusions. The conditional probability analysis showed that the decision of reviewers, when number of points in the evaluation sheet is taken into consideration, is slightly unpredictable. No significant differences in the reviewers` recommendations based on gender or country of origin of the reviewers were found.

Open peer review, pros and cons from the perspective of an early career researcher

Peer review is considered by many to be a fundamental component of scientific publishing. In this context, open peer review (OPR) has gained popularity in recent years as a tool to increase transparency, rigor, and inclusivity in research. But how does OPR really affect the review process? How does OPR impact specific groups, such as early career researchers? This editorial explores and discusses these aspects as well as some suggested actions for journals.

A health sciences researcher's experience of manuscript review comments, 2020-2022

BACKGROUND

 Peer review frequently improves a manuscript, but authors may consider some reviewer feedback negative, inappropriate or unclear. This study aims to summarise and analyse review comments received by authors.

METHODS

 This longitudinal study included all submissions of which the researcher was an author, reviewed by any journal during 2020-2022. First-round reviews were retrieved from emails and documents received by the authors or the faculty's medical editors or the journal platforms. A confidential datasheet with review items compiled from literature and the researcher's experience as author and reviewer was completed for each submission. Review comments were noted verbatim for subjective items such as rude or vague statements.

RESULTS

 The 65 submissions received 118 reviews from 36 journals, mainly in the form of unstructured narrative reports (59%). The majority of first-round reviews (58%), including those for rejected submissions, contained some positive comments. Reviewers frequently (75% of reviews, 88% of submissions) required some expansion of information. Vague and incorrect statements occurred in 15% and 18% of reviews, respectively. Only two reviews contained statements that could be considered rude. The types of comments made were associated with the review format.

CONCLUSION

 The majority of reviews contained some positive comments and rude comments were extremely rare. Reviewers frequently requested the expansion of information provided.Contribution: This study gives insight to authors, reviewers and editors regarding the type and tone of review comments. This could guide authors during manuscript preparation and authors, reviewers and editors during the review process.

Preprint Peer Review Enhances Undergraduate Biology Students' Disciplinary Literacy and Sense of Belonging in STEM

Education about scientific publishing and manuscript peer review is not universally provided in undergraduate science courses. Since peer review is integral to the scientific process and central to the identity of a scientist, we envision a paradigm shift where teaching peer review becomes integral to undergraduate science education. We hypothesize that teaching undergraduates how to peer review scientific manuscripts may facilitate their development of scientific literacy and identity formation. To this end, we developed a constructivist, service-learning curriculum for biology undergraduates to learn about the mechanisms of peer review using preprints and then to write and publish their own peer reviews of preprints as a way to authentically join the scientific community of practice. The curriculum was implemented as a semester-long intervention in one class and, in another class, as an embedded module intervention. Students' scientific literacy and peer review ability were assessed using quantitative methods. Student's perceptions of their scientific literacy and identity were assessed using thematic analysis of students' reflective writing. Here, we present data on the improvement in the peer review ability of undergraduates in both classes and data on the curriculum's interrelated impact on students' development of scientific literacy, identity, and belonging in peer and professional discourse spaces. These data suggest that undergraduates can and should be trained in peer review to foster the interrelated development of their scientific literacy, scientific identity, and sense of belonging in science.

How to improve scientific peer review: Four schools of thought

Peer review plays an essential role as one of the cornerstones of the scholarly publishing system. There are many initiatives that aim to improve the way in which peer review is organized, resulting in a highly complex landscape of innovation in peer review. Different initiatives are based on different views on the most urgent challenges faced by the peer review system, leading to a diversity of perspectives on how the system can be improved. To provide a more systematic understanding of the landscape of innovation in peer review, we suggest that the landscape is shaped by four schools of thought: The Quality & Reproducibility school, the Democracy & Transparency school, the Equity & Inclusion school, and the Efficiency & Incentives school. Each school has a different view on the key problems of the peer review system and the innovations necessary to address these problems. The schools partly complement each other, but we argue that there are also important tensions between them. We hope that the four schools of thought offer a useful framework to facilitate conversations about the future development of the peer review system.

Recent Issues in Medical Journal Publishing and Editing Policies: Adoption of Artificial Intelligence, Preprints, Open Peer Review, Model Text Recycling Policies, Best Practice in Scholarly Publishing 4th Version, and Country Names in Titles

In Korea, many editors of medical journal are also publishers; therefore, they need to not only manage peer review, but also understand current trends and policies in journal publishing and editing. This article aims to highlight some of these policies with examples. First, the use of artificial intelligence tools in journal publishing has increased, including for manuscript editing and plagiarism detection. Second, preprint publications, which have not been peer-reviewed, are becoming more common. During the COVID-19 pandemic, medical journals have been more willing to accept preprints to adjust rapidly changing pandemic health issues, leading to a significant increase in their use. Third, open peer review with reviewer comments is becoming more widespread, including the mandatory publication of peer-reviewed manuscripts with comments. Fourth, model text recycling policies provide guidelines for researchers and editors on how to appropriately recycle text, for example, in the background section of the Introduction or the Methods section. Fifth, journals should take into account the recently updated 4th version of the Principles of Transparency and Best Practice in Scholarly Publishing, released in 2022. This version includes more detailed guidelines on journal websites, peer review processes, advisory boards, and author fees. Finally, it recommends that titles of human studies include country names to clarify the cultural context of the research. Each editor must decide whether to adopt these six policies for their journals. Editor-publishers of society journals are encouraged to familiarize themselves with these policies so that they can implement them in their journals as appropriate.

Community consensus on core open science practices to monitor in biomedicine

The state of open science needs to be monitored to track changes over time and identify areas to create interventions to drive improvements. In order to monitor open science practices, they first need to be well defined and operationalized. To reach consensus on what open science practices to monitor at biomedical research institutions, we conducted a modified 3-round Delphi study. Participants were research administrators, researchers, specialists in dedicated open science roles, and librarians. In rounds 1 and 2, participants completed an online survey evaluating a set of potential open science practices, and for round 3, we hosted two half-day virtual meetings to discuss and vote on items that had not reached consensus. Ultimately, participants reached consensus on 19 open science practices. This core set of open science practices will form the foundation for institutional dashboards and may also be of value for the development of policy, education, and interventions.

role of peer review in the evaluation of research in Italy. Some remarks on the evaluation of PRINs

This contribution proposes some remarks on the evaluation and financing mechanisms of PRINs – Progetti di Rilevante Interesse Nazionale, promoted in Italy by the MUR - Ministry of University and Research, in the context of the critical issues and evolution prospects of peer review, of which a summary state of the art is presented. Starting from the partial and incomplete data made available on the MUR website dedicated to PRINs, are listed and examined the projects financed for the current disciplinary sector M-STO/08 (Archival Science, Bibliography and Librarianship), in the years between 1996 and 2020, and those included in other disciplinary areas that have as their subject matters related to the contents of the academic field M-STO/08.

Open peer review: some considerations on the selection and management of reviewers

Open peer review (OPR) is a type of review that has long since made space alongside the more well-known single-blind and double-blind peer reviews. Despite this, we still do not have a shared definition by the scientific community and publishers. The purpose of this paper is to offer some reflections on the selection of reviewers in OPR, a process that in its current configuration is traced back to the 19th century. After a brief overview of the best-known definitions of open peer review, the paper continues with an analysis of some aspects of reviewer selection carried out with the help of data from a recent survey.

Open peer review: the point of view of scientific journal editors

Academic journals have been incorporating several elements of open science: open access (since 2000), later, the deposit of research data of the articles published, the dissemination of preprints before the publication of the paper and, finally, the open peer review (OPR). While open access is well-established and the inclusion of research data is increasingly widespread, the OPR is just at the beginning of its incorporation as a real alternative to the double-blind model, which is the most widespread and consolidated. The objective of our article is to analyse the opinion of the editors of Spanish scientific journals about the advantages and disadvantages or barriers for the implementation of the OPR. This is a qualitative study that has been carried out from the open answers of a questionnaire sent to the 1875 editors of the Spanish academic journals that appear in the database Dulcinea and that obtained a response of 22.4%. Regarding the limitations, the study is based on the opinions and experience of the editors of Spanish scientific journals, which are mostly published by academic institutions and are in the field of social sciences and humanities. The results focus on delving into the advantages and disadvantages. Among the encouraging factors, the editors point out that to have open reports is very useful for the scientific community, that it recognizes the role of the reviewer, makes it possible to control the arbitrariness of some reviewers, and that it promotes the reviewer-author dialogue. The main barriers discussed are the following: a possible lack of objectivity and rigor, resistance to change a consolidated system (“double-blind”), knowing the author benefits established authors and harms novices, more difficulties for finding reviewers, increases costs and can lengthen the review process.

Peer review ethics in Iranian LIS scholarly journals: a comparison between views of reviewers and authors

Peer review is one of the most efficient ways to ensure the quality of papers for possible publication in scholarly journals. However, the process of peer review is not free of bias and disorders. Many reviewers are unaware of how their attitudes towards the evaluation of scholarly papers may violate Peer Review Ethics (PRE). This paper attempts to analyze the different ethical issues influencing the job of reviewing. The research sample for this study included 7 Iranian library and information journals, 124 Iranian peer reviewers, and 34 authors. Peer reviewers and authors were asked to evaluate the most important ethical elements of peer review in Iranian LIS journals through two different questionnaires based on Rajabali Beglou et al. (2019) research. Findings showed that there was no difference among authors and reviewers in terms of gender in most PRE elements. Also, the level of experience of the authors was not significant in terms of understanding and acceptance of the PRE among reviewers and authors. However, review experiences regarding some PRE elements were significant in respondents’ viewpoints. The experiences reviewers had already gained were influential on their views about PRE. In addition, results showed that there were significant differences among reviewers and authors about the PRE elements in LIS journals. Authorship experiences had not effect on the PRE elements and the dual role of peer reviewing and authorship had no impact on their views.

[Open Science: Emergency Response or the New Normal?]

N/a.

Fostering openness in open science: An ethical discussion of risks and benefits

Transformation of science by embracing the concepts of open science presents a very attractive strategy to enhance the reliability of science. Open science policies embody the concepts of open data and open access that encompass sharing of resources, dissemination of ideas, and synergizing the collaborative forums of research. Despite the opportunities in openness, however, there are grave ethical concerns too, and they present a dual-use dilemma. Access to sensitive information is seen as a security risk, and it also possesses other concerns such as confidentiality, privacy, and affordability. There are arguments that open science can be harmful to marginalized groups. Through this study, we aim to discuss the opportunities of open science, as well as the ethical and security aspects, which require further deliberation before full-fledged acceptance in the science community.

A validation study on the factors affecting the practice modes of open peer review

In this paper we conduct a validation study on the factors affecting the practice modes of open peer review. Taking the Open Access Journals (OAJ) in Directory of Open Access Journals (DOAJ) as the research objects, we crawled the internet to gather their relevant data. Based on the method of categorical variable assignment, a quantitative analysis was performed on the qualitative factors that affect the practice modes of open peer review. A multi-dimensional analysis chart is used to illustrate the relationships between the factors. Optimal scale regression modeling and discriminant analysis were also employed to reveal the degrees of influences by the factors. The public categories of “type of open peer review” and “reviewer identity” are closely related to each other. “Reviewer identity” has evident positive influence on “type of open peer review”, and the degree of influence is the highest. Therefore, “reviewer identity” is the primary and most crucial factor affecting open peer review practice modes. “Review report” and “order of review report and publication” are the secondary ones. Whether or not the identities of review experts are open has become the most important factor affecting the practice modes of open peer review. Transparent peer review is currently the most effective practice mode of open peer review. Technologies like block chain can be used to address the psychological uneasiness for the peer review experts who are concerned with privacy issues. The fact that most OAJs use “pre-publication review” shows that open peer review still plays the traditional role of “academic goalkeeper”. Publication of peer review reports actually helps peer review experts augment their reputation, which in turn practically promotes the development of open peer review.

Investing in the Academic Writing: Training Future Reviewers and Sustaining Efficient and Quality Peer Review

Peer reviewers are considered gatekeepers in academic writing who play a pivotal and essential role during the publication process. Excellent manuscripts invariably need excellent reviewers. Producing peer reviewers with such caliber is time-consuming albeit necessary for the progress and continuity of academia. Despite the popular belief that an experienced author invariably makes a good reviewer, the reality is far-fetched. This suggests the need for peer reviewer training, which should be effective, logistically affordable, and demonstrate long-lasting positive impacts. Open review, co-review, and several reviewer training programs are already in place for this purpose with varying efficiencies. This narrative review discusses the current modalities available to a junior reviewer to improve his/her review skills and proposes a reviewer residency concept that could be adopted as a part of peer reviewer training.

The Role of Publons in the Context of Open Peer Review

Publons was a peer reviewer rewards platform that aimed to recognize the contribution that academics made during peer review to a journal. For about 10 years of its existence, Publons became the most popular service among peer reviewers. Having gained traction and popularity, Publons was purchased in 2017 by Clarivate Analytics (now Clarivate), and many academics, journals and publishers invested time and effort to participate in Publons. Using Publons, various peer review-related experiments or pilot programs were initiated by some academic publishers regarding the introduction of open peer review into their journals’ editorial processes. In this paper, we examine pertinent literature related to Publons, and reflect on its benefits and flaws during its short-lived history. In mid-August 2022, Clarivate fused Publons into the Web of Science platform. Publons, as a brand peer review service, has now ceased to exist but some of the functionality remains in Web of Science while other aspects that used to be open and free at Publons are now paid-for services. We reflect on the effect of such experiments, which initially had bold and ambitious academic objectives to fortify peer review, on academics’ trust, especially when such projects become commercialized.

Innovating peer review, reconfiguring scholarly communication: an analytical overview of ongoing peer review innovation activities

Purpose

The study aims to provide an analytical overview of current innovations in peer review and their potential impacts on scholarly communication.

Design/methodology/approach

The authors created a survey that was disseminated among publishers, academic journal editors and other organizations in the scholarly communication ecosystem, resulting in a data set of 95 self-defined innovations. The authors ordered the material using a taxonomy that compares innovation projects according to five dimensions. For example, what is the object of review? How are reviewers recruited, and does the innovation entail specific review foci?

Findings

Peer review innovations partly pull in mutually opposed directions. Several initiatives aim to make peer review more efficient and less costly, while other initiatives aim to promote its rigor, which is likely to increase costs; innovations based on a singular notion of “good scientific practice” are at odds with more pluralistic understandings of scientific quality; and the idea of transparency in peer review is the antithesis to the notion that objectivity requires anonymization. These fault lines suggest a need for better coordination.

Originality/value

This paper presents original data that were analyzed using a novel, inductively developed, taxonomy. Contrary to earlier research, the authors do not attempt to gauge the extent to which peer review innovations increase the “reliability” or “quality” of reviews (as defined according to often implicit normative criteria), nor are they trying to measure the uptake of innovations in the routines of academic journals. Instead, they focus on peer review innovation activities as a distinct object of analysis.

Open science at the science-policy interface: bringing in the evidence?

Part of the current enthusiasm about open science stems from its promises to reform scientific practice in service of the common good, to ensure that scientific outputs will be found and reused more easily, and to enhance scientific impact on policy and society. With this article, we question this optimism by analysing the potential for open science practices to enhance research uptake at the science–policy interface. Science advice is critical to help policy-makers make informed decisions. Likewise, some interpretations of open science hold that making research processes and outputs more transparent and accessible will also enhance the uptake of results by policy and society at large. However, we argue that this hope is based on an unjustifiably simplistic understanding of the science–policy interface that leaves key terms (“impact”, “uptake”) undefined. We show that this understanding—based upon linear models of research uptake—likewise grounds the influential “evidence–policy gap” diagnosis which holds that to improve research uptake, communication and interaction between researchers and policy-makers need to be improved. The overall normative stance of both discussions has sidelined empirical description of the science–policy interface, ignoring questions about the underlying differences between the policy domain and academia. Importantly, both open science and literature on closing the evidence–policy gap recommend improving communication (in terms of either the content or the means) as a viable strategy. To correct some of these views, we combine insights from policy theory with a narrative review of the literature on the evidence–policy gap in the health domain and find that removing barriers to access by itself will not be enough to foster research uptake.

Keep calm and carry on: moral panic, predatory publishers, peer review, and the emperor's new clothes

The moral panic over the impact of so-called predatory publishers continues unabated. It is important, however, to resist the urge to simply join in this crusade without pausing to examine the assumptions upon which such concerns are based. It is often assumed that established journals are almost sacrosanct, and that their quality, secured by peer review, is established. It is also routinely presumed that such journals are immune to the lure of easy money in return for publication. Rather than looking at the deficits that may be apparent in the practices and products of predatory publishers, this commentary invites you to explore the weaknesses that have been exposed in traditional academic journals but are seldom discussed in the context of predatory publishing. The inherent message for health and medical services staff, researchers, academics, and students is, as always, to critically evaluate all sources of information, whatever their provenance.

Types, limitations, and possible alternatives of peer review based on the literature and surgeons’ opinions via Twitter: a narrative review

This review aimed to illustrate the types, limitations, and possible alternatives of peer review (PR) based on a literature review together with the opinions of a social media audience via Twitter. This study was conducted via the #OpenSourceResearch collaborative platform and combined a comprehensive literature search on the current PR system with the opinions of a social media audience of surgeons who are actively engaged in the current PR system. Six independent researchers conducted a literature search of electronic databases in addition to Google Scholar. Electronic polls were organized via Twitter to assess surgeons’ opinions on the current PR system and potential alternative approaches. PR can be classified into single-blind, double-blind, triple-blind, and open PR. Newer PR systems include interactive platforms, prepublication and postpublication commenting or review, transparent review, and collaborative review. The main limitations of the current PR system are its allegedly time-consuming nature and inconsistent, biased, and non-transparent results. Suggestions to improve the PR process include employing an interactive, double-blind PR system, using artificial intelligence to recruit reviewers, providing incentives for reviewers, and using PR templates. The above results offer several concepts for possible alternative approaches and modifications to this critically important process.

Transparent, Open, and Reproducible Prevention Science

The field of prevention science aims to understand societal problems, identify effective interventions, and translate scientific evidence into policy and practice. There is growing interest among prevention scientists in the potential for transparency, openness, and reproducibility to facilitate this mission by providing opportunities to align scientific practice with scientific ideals, accelerate scientific discovery, and broaden access to scientific knowledge. The overarching goal of this manuscript is to serve as a primer introducing and providing an overview of open science for prevention researchers. In this paper, we discuss factors motivating interest in transparency and reproducibility, research practices associated with open science, and stakeholders engaged in and impacted by open science reform efforts. In addition, we discuss how and why different types of prevention research could incorporate open science practices, as well as ways that prevention science tools and methods could be leveraged to advance the wider open science movement. To promote further discussion, we conclude with potential reservations and challenges for the field of prevention science to address as it transitions to greater transparency, openness, and reproducibility. Throughout, we identify activities that aim to strengthen the reliability and efficiency of prevention science, facilitate access to its products and outputs, and promote collaborative and inclusive participation in research activities. By embracing principles of transparency, openness, and reproducibility, prevention science can better achieve its mission to advance evidence-based solutions to promote individual and collective well-being.

An overview of innovations in the external peer review of journal manuscripts

Background: There are currently numerous innovations in peer review and quality assurance in scholarly publishing. The Research on Research Institute conducted a programme of co-produced projects investigating these innovations. This literature review was part of one such project 'Experiments in peer review' which created an inventory and framework of peer review innovations. The aim of this literature review was to aid the development of the inventory by identifying innovations in the external peer review of journal manuscripts reported in the scholarly literature and by providing a summary of the different approaches. This did not include interventions in editorial processes. Methods: This review of reviews is based on data identified from Web of Science and Scopus limited from 2010 to 2021. A total of 291 records were screened, with six review articles chosen for the focus of the literature review. Items were selected that described approaches to innovating peer review or illustrated examples.   Results: The overview of innovations are drawn from six review articles. The innovations are divided into three high-level categories: approaches to peer review, reviewer focussed initiatives and technology to support peer review with sub-categories of results presented in tabular form and summarised. A summary of all innovations found is also presented. Conclusions: From a simple synthesis of the review authors' conclusions, three key messages are presented: observations on current practice; authors' views on the implications of innovations in peer review; and calls for action in peer review research and practice.

Innovations in peer review in scholarly publishing: a meta-summary

Background: There are currently numerous innovations in peer review and quality assurance in scholarly publishing. The Research on Research Institute conducted a programme of co-produced projects investigating these innovations. This literature review was part of one such project 'Experiments in peer review' which created an inventory and framework of peer review innovations. The aim of this literature review was to aid the development of the inventory by identifying innovations in peer review reported in the scholarly literature and by providing a summary of the different approaches. Methods: This meta-summary is based on data identified from Web of Science and Scopus limited from 2010 to 2021. A total of 247 papers were screened, with 6 review articles chosen for the focus of the literature review. Items were selected that described approaches to innovating peer review or illustrated examples.   Results: The summary of innovations are drawn from 6 review articles. The innovations are divided into three high-level categories: approaches to peer review, reviewer focussed initiatives and technology to support peer review with sub-categories of results presented in tabular form and summarised. A summary of all innovations found is also presented. Conclusions: From a simple synthesis of the review authors' conclusions, three key messages are presented: observations on current practice; authors' views on the implications of innovations in peer review; and calls for action in peer review research and practice.

Transparency in peer review: Exploring the content and tone of reviewers' confidential comments to editors

Purpose

Recent calls to improve transparency in peer review have prompted examination of many aspects of the peer-review process. Peer-review systems often allow confidential comments to editors that could reduce transparency to authors, yet this option has escaped scrutiny. Our study explores 1) how reviewers use the confidential comments section and 2) alignment between comments to the editor and comments to authors with respect to content and tone.

Methods

Our dataset included 358 reviews of 168 manuscripts submitted between January 1, 2019 and August 24, 2020 to a health professions education journal with a single blind review process. We first identified reviews containing comments to the editor. Then, for the reviews with comments, we used procedures consistent with conventional and directed qualitative content analysis to develop a coding scheme and code comments for content, tone, and section of the manuscript. For reviews in which the reviewer recommended “reject,” we coded for alignment between reviewers’ comments to the editor and to authors. We report descriptive statistics.

Results

49% of reviews contained comments to the editor (n = 176). Most of these comments summarized the reviewers’ impression of the article (85%), which included explicit reference to their recommended decision (44%) and suitability for the journal (10%). The majority of comments addressed argument quality (56%) or research design/methods/data (51%). The tone of comments tended to be critical (40%) or constructive (34%). For the 86 reviews recommending “reject,” the majority of comments to the editor contained content that also appeared in comments to the authors (80%); additional content tended to be irrelevant to the manuscript. Tone frequently aligned (91%).

Conclusion

Findings indicate variability in how reviewers use the confidential comments to editor section in online peer-review systems, though generally the way they use them suggests integrity and transparency to authors.

Open Science in regulatory environmental risk assessment

A possible way to alleviate the public skepticism toward regulatory science is to increase transparency by making all data and value judgments used in regulatory decision making accessible for public interpretation, ideally early on in the process, and following the concepts of Open Science. This paper discusses the opportunities and challenges in strengthening Open Science initiatives in regulatory environmental risk assessment (ERA). In this discussion paper, we argue that the benefits associated with Open Science in regulatory ERA far outweigh its perceived risks. All stakeholders involved in regulatory ERA (e.g., governmental regulatory authorities, private sector, academia, and nongovernmental organizations), as well as professional organizations like the Society of Environmental Toxicology and Chemistry, can play a key role in supporting the Open Science initiative, by promoting the use of recommended reporting criteria for reliability and relevance of data and tools used in ERA, and by developing a communication strategy for both professionals and nonprofessionals to transparently explain the socioeconomic value judgments and scientific principles underlying regulatory ERA. Integr Environ Assess Manag 2021;17:1229-1242. © 2021 The Authors. Integrated Environmental Assessment and Management published by Wiley Periodicals LLC on behalf of Society of Environmental Toxicology & Chemistry (SETAC).

Which peer reviewers voluntarily reveal their identity to authors? Insights into the consequences of open-identities peer review

Identifying reviewers is argued to improve the quality and fairness of peer review, but is generally disfavoured by reviewers. To gain some insight into the factors that influence when reviewers are willing to have their identity revealed, I examined which reviewers voluntarily reveal their identities to authors at the journal Functional Ecology, at which reviewer identities are confidential unless reviewers sign their comments to authors. I found that 5.6% of reviewers signed their comments to authors. This proportion increased slightly over time, from 4.4% in 2003-2005 to 6.7% in 2013-2015. Male reviewers were 1.8 times more likely to sign their comments to authors than were female reviewers, and this difference persisted over time. Few reviewers signed all of their reviews; reviewers were more likely to sign their reviews when their rating of the manuscript was more positive, and papers that had at least one signed review were more likely to be invited for revision. Signed reviews were, on average, longer and recommended more references to authors. My analyses cannot distinguish cause and effect for the patterns observed, but my results suggest that 'open-identities' review, in which reviewers are not permitted to be anonymous, will probably reduce the degree to which reviewers are critical in their assessment of manuscripts and will differentially affect recruitment of male and female reviewers, negatively affecting the diversity of reviewers recruited by journals.

Motivations for performing scholarly prepublication peer review: A scoping review

Prepublication peer review is a cornerstone of science. Overburdened reviewers invest millions of hours in this voluntary activity. In this scoping review, we aimed at identifying motivations for performing prepublication peer review of scholarly manuscripts. Original research studies investigating actual peer reviewers' motivations were included. We excluded modeling studies, studies related to other types of peer review, guidelines, peer review processes in particular journals. Medline, WoS, and Scopus were searched in February 2016, with no language or time limitations, and the search was updated in July 2019. The search yielded 5,250 records, and 382 were chosen for full text analysis, out of which 10 were appropriate for synthesis. Reference snowballing identified one eligible study. Eleven studies were appropriate for synthesis: four qualitative, four mixed qualitative/quantitative, and three qualitative studies, published from 1998 to 2018, involving 6,667 respondents. Major internal incentive was "communal obligations and reciprocity." Major external incentives were "career advancement," "being recognized as an expert," and "building relationships with journals and editors." Major disincentive was the "lack of time." Editors could incentivize peer review process by choosing highest quality articles, improving communication with peer reviewers, in order to make the process of peer review as short and efficient as possible. The gaps in research concern disincentives to review.

Attitudes and practices of open data, preprinting, and peer-review-A cross sectional study on Croatian scientists

Attitudes towards open peer review, open data and use of preprints influence scientists’ engagement with those practices. Yet there is a lack of validated questionnaires that measure these attitudes. The goal of our study was to construct and validate such a questionnaire and use it to assess attitudes of Croatian scientists. We first developed a 21-item questionnaire called Attitudes towards Open data sharing, preprinting, and peer-review (ATOPP), which had a reliable four-factor structure, and measured attitudes towards open data, preprint servers, open peer-review and open peer-review in small scientific communities. We then used the ATOPP to explore attitudes of Croatian scientists (n = 541) towards these topics, and to assess the association of their attitudes with their open science practices and demographic information. Overall, Croatian scientists’ attitudes towards these topics were generally neutral, with a median (Md) score of 3.3 out of max 5 on the scale score. We also found no gender (P = 0.995) or field differences (P = 0.523) in their attitudes. However, attitudes of scientist who previously engaged in open peer-review or preprinting were higher than of scientists that did not (Md 3.5 vs. 3.3, P<0.001, and Md 3.6 vs 3.3, P<0.001, respectively). Further research is needed to determine optimal ways of increasing scientists’ attitudes and their open science practices.

Open science saves lives: lessons from the COVID-19 pandemic

In the last decade Open Science principles have been successfully advocated for and are being slowly adopted in different research communities. In response to the COVID-19 pandemic many publishers and researchers have sped up their adoption of Open Science practices, sometimes embracing them fully and sometimes partially or in a sub-optimal manner. In this article, we express concerns about the violation of some of the Open Science principles and its potential impact on the quality of research output. We provide evidence of the misuses of these principles at different stages of the scientific process. We call for a wider adoption of Open Science practices in the hope that this work will encourage a broader endorsement of Open Science principles and serve as a reminder that science should always be a rigorous process, reliable and transparent, especially in the context of a pandemic where research findings are being translated into practice even more rapidly. We provide all data and scripts at https://osf.io/renxy/ .

Honest signaling in academic publishing

Academic journals provide a key quality-control mechanism in science. Yet, information asymmetries and conflicts of interests incentivize scientists to deceive journals about the quality of their research. How can honesty be ensured, despite incentives for deception? Here, we address this question by applying the theory of honest signaling to the publication process. Our models demonstrate that several mechanisms can ensure honest journal submission, including differential benefits, differential costs, and costs to resubmitting rejected papers. Without submission costs, scientists benefit from submitting all papers to high-ranking journals, unless papers can only be submitted a limited number of times. Counterintuitively, our analysis implies that inefficiencies in academic publishing (e.g., arbitrary formatting requirements, long review times) can serve a function by disincentivizing scientists from submitting low-quality work to high-ranking journals. Our models provide simple, powerful tools for understanding how to promote honest paper submission in academic publishing.

Peer review analysis in the field of radiation oncology: results from a web-based survey of the Young DEGRO working group

Purpose

To evaluate the reviewing behaviour in the German-speaking countries in order to provide recommendations to increase the attractiveness of reviewing activity in the field of radiation oncology.

Methods

In November 2019, a survey was conducted by the Young DEGRO working group (jDEGRO) using the online platform “eSurveyCreator”. The questionnaire consisted of 29 items examining a broad range of factors that influence reviewing motivation and performance.

Results

A total of 281 responses were received. Of these, 154 (55%) were completed and included in the evaluation. The most important factors for journal selection criteria and peer review performance in the field of radiation oncology are the scientific background of the manuscript (85%), reputation of the journal (59%) and a high impact factor (IF; 40%). Reasons for declining an invitation to review include the scientific background of the article (60%), assumed effort (55%) and a low IF (27%). A double-blind review process is preferred by 70% of respondents to a single-blind (16%) or an open review process (14%). If compensation was offered, 59% of participants would review articles more often. Only 12% of the participants have received compensation for their reviewing activities so far. As compensation for the effort of reviewing, 55% of the respondents would prefer free access to the journal’s articles, 45% a discount for their own manuscripts, 40% reduced congress fees and 39% compensation for expenses.

Conclusion

The scientific content of the manuscript, reputation of the journal and a high IF determine the attractiveness for peer reviewing in the field of radiation oncology. The majority of participants prefer a double-blind peer review process and would conduct more reviews if compensation was available. Free access to journal articles, discounts for publication costs or congress fees, or an expense allowance were identified to increase attractiveness of the review process.

Supplementary Information

The online version of this article (10.1007/s00066-020-01729-2) contains supplementary material, which is available to authorized users.

Journal policies and editors' opinions on peer review

Peer review practices differ substantially between journals and disciplines. This study presents the results of a survey of 322 editors of journals in ecology, economics, medicine, physics and psychology. We found that 49% of the journals surveyed checked all manuscripts for plagiarism, that 61% allowed authors to recommend both for and against specific reviewers, and that less than 6% used a form of open peer review. Most journals did not have an official policy on altering reports from reviewers, but 91% of editors identified at least one situation in which it was appropriate for an editor to alter a report. Editors were also asked for their views on five issues related to publication ethics. A majority expressed support for co-reviewing, reviewers requesting access to data, reviewers recommending citations to their work, editors publishing in their own journals, and replication studies. Our results provide a window into what is largely an opaque aspect of the scientific process. We hope the findings will inform the debate about the role and transparency of peer review in scholarly publishing.

Peer review and preprint policies are unclear at most major journals

Clear and findable publishing policies are important for authors to choose appropriate journals for publication. We investigated the clarity of policies of 171 major academic journals across disciplines regarding peer review and preprinting. 31.6% of journals surveyed do not provide information on the type of peer review they use. Information on whether preprints can be posted or not is unclear in 39.2% of journals. 58.5% of journals offer no clear information on whether reviewer identities are revealed to authors. Around 75% of journals have no clear policy on co-reviewing, citation of preprints, and publication of reviewer identities. Information regarding practices of open peer review is even more scarce, with <20% of journals providing clear information. Having found a lack of clear information, we conclude by examining the implications this has for researchers (especially early career) and the spread of open research practices.

Open science practices in higher education: Discussion of survey results from research and teaching staff in Germany

Aspects of open science and scholarly practices are often discussed with a focus on research and research dissemination processes. There is currently less discussion on open science and its influence on learning and teaching in higher education, and reversely. This paper discusses open science in relation to educational practices and resources and reports on a study to investigate current educational practices from the perspective of open science. We argue that offering students opportunities via open educational practices raises their awareness of future open science goals and teaches them the skills needed to reach those goals. We present online survey results from 210 participants with teaching responsibility at higher education institutions in Germany. While some of them try to establish more open learning and teaching settings, most respondents apply rather traditional ways of learning and teaching. 60% do not use open educational resources – many have not even heard of them – nor do they make their courses open for an online audience. Participants’ priority lies in resource accuracy and quality and we still see a gap between the benefit of open practices and their practicability and applicability. The paper contributes to the general discussion of open practices in higher education by looking at open science practices and their adaptation to the learning and teaching environment. It formulates recommendations for improvements of open practice support and infrastructure.

Open up: a survey on open and non-anonymized peer reviewing

BACKGROUND

Our aim is to highlight the benefits and limitations of open and non-anonymized peer review. Our argument is based on the literature and on responses to a survey on the reviewing process of alt.chi, a more or less open review track within the so-called Computer Human Interaction (CHI) conference, the predominant conference in the field of human-computer interaction. This track currently is the only implementation of an open peer review process in the field of human-computer interaction while, with the recent increase in interest in open scientific practices, open review is now being considered and used in other fields.

METHODS

We ran an online survey with 30 responses from alt.chi authors and reviewers, collecting quantitative data using multiple-choice questions and Likert scales. Qualitative data were collected using open questions.

RESULTS

Our main quantitative result is that respondents are more positive to open and non-anonymous reviewing for alt.chi than for other parts of the CHI conference. The qualitative data specifically highlight the benefits of open and transparent academic discussions. The data and scripts are available on https://osf.io/vuw7h/, and the figures and follow-up work on http://tiny.cc/OpenReviews.

CONCLUSION

While the benefits are quite clear and the system is generally well-liked by alt.chi participants, they remain reluctant to see it used in other venues. This concurs with a number of recent studies that suggest a divergence between support for a more open review process and its practical implementation.

Automatically detecting open academic review praise and criticism

Purpose

Peer reviewer evaluations of academic papers are known to be variable in content and overall judgements but are important academic publishing safeguards. This article introduces a sentiment analysis program, PeerJudge, to detect praise and criticism in peer evaluations. It is designed to support editorial management decisions and reviewers in the scholarly publishing process and for grant funding decision workflows. The initial version of PeerJudge is tailored for reviews from F1000Research's open peer review publishing platform.

Design/methodology/approach

PeerJudge uses a lexical sentiment analysis approach with a human-coded initial sentiment lexicon and machine learning adjustments and additions. It was built with an F1000Research development corpus and evaluated on a different F1000Research test corpus using reviewer ratings.

Findings

PeerJudge can predict F1000Research judgements from negative evaluations in reviewers' comments more accurately than baseline approaches, although not from positive reviewer comments, which seem to be largely unrelated to reviewer decisions. Within the F1000Research mode of post-publication peer review, the absence of any detected negative comments is a reliable indicator that an article will be ‘approved’, but the presence of moderately negative comments could lead to either an approved or approved with reservations decision.

Originality/value

PeerJudge is the first transparent AI approach to peer review sentiment detection. It may be used to identify anomalous reviews with text potentially not matching judgements for individual checks or systematic bias assessments.