Open peer review: a randomised controlled trial

Twenty-Five Years of Progress-Lessons Learned From JMIR Publications to Address Gender Parity in Digital Health Authorships: Bibliometric Analysis

BACKGROUND

Digital health research plays a vital role in advancing equitable health care. The diversity of research teams is thereby instrumental in capturing societal challenges, increasing productivity, and reducing bias in algorithms. Despite its importance, the gender distribution within digital health authorship remains largely unexplored.

OBJECTIVE

This study aimed to investigate the gender distribution among first and last authors in digital health research, thereby identifying predicting factors of female authorship.

METHODS

This bibliometric analysis examined the gender distribution across 59,980 publications from 1999 to 2023, spanning 42 digital health journals indexed in the Web of Science. To identify strategies ensuring equality in research, a detailed comparison of gender representation in JMIR journals was conducted within the field, as well as against a matched sample. Two-tailed Welch 2-sample t tests, Wilcoxon rank sum tests, and chi-square tests were used to assess differences. In addition, odds ratios were calculated to identify predictors of female authorship.

RESULTS

The analysis revealed that 37% of first authors and 30% of last authors in digital health were female. JMIR journals demonstrated a higher representation, with 49% of first authors and 38% of last authors being female, yielding odds ratios of 1.96 (95% CI 1.90-2.03; P<.001) and 1.78 (95% CI 1.71-1.84; P<.001), respectively. Since 2008, JMIR journals have consistently featured a greater proportion of female first authors than male counterparts. Other factors that predicted female authorship included having female authors in other relevant positions and gender discordance, given the higher rate of male last authors in the field.

CONCLUSIONS

There was an evident shift toward gender parity across publications in digital health, particularly from the publisher JMIR Publications. The specialized focus of its sister journals, equitable editorial policies, and transparency in the review process might contribute to these achievements. Further research is imperative to establish causality, enabling the replication of these successful strategies across other scientific fields to bridge the gender gap in digital health effectively.

An overview of the peer review process in biomedical sciences

OBJECTIVE

This paper aims to provide an introductory resource for beginner peer reviewers in psychiatry and the broader biomedical science field. It will provide a concise overview of the peer review process, alongside some reviewing tips and tricks.

CONCLUSION

The peer review process is a fundamental aspect of biomedical science publishing. The model of peer review offered varies between journals and usually relies on a pool of volunteers with differing levels of expertise and scope. The aim of peer review is to collaboratively leverage reviewers' collective knowledge with the objective of increasing the quality and merit of published works. The limitations, methodology and need for transparency in the peer review process are often poorly understood. Although imperfect, the peer review process provides some degree of scientific rigour by emphasising the need for an ethical, comprehensive and systematic approach to reviewing articles. Contributions from junior reviewers can add significant value to manuscripts.

The trustworthiness of the cumulative knowledge in industrial/organizational psychology: The current state of affairs and a path forward

The goal of industrial/organizational (IO) psychology, is to build and organize trustworthy knowledge about people-related phenomena in the workplace. Unfortunately, as with other scientific disciplines, our discipline may be experiencing a "crisis of confidence" stemming from the lack of reproducibility and replicability of many of our field's research findings, which would suggest that much of our research may be untrustworthy. If a scientific discipline's research is deemed untrustworthy, it can have dire consequences, including the withdraw of funding for future research. In this focal article, we review the current state of reproducibility and replicability in IO psychology and related fields. As part of this review, we discuss factors that make it less likely that research findings will be trustworthy, including the prevalence of scientific misconduct, questionable research practices (QRPs), and errors. We then identify some root causes of these issues and provide several potential remedies. In particular, we highlight the need for improved research methods and statistics training as well as a re-alignment of the incentive structure in academia. To accomplish this, we advocate for changes in the reward structure, improvements to the peer review process, and the implementation of open science practices. Overall, addressing the current "crisis of confidence" in IO psychology requires individual researchers, academic institutions, and publishers to embrace system-wide change.

Overcoming the gender bias in ecology and evolution: is the double-anonymized peer review an effective pathway over time?

Male researchers dominate scientific production in science, technology, engineering, and mathematics (STEM). However, potential mechanisms to avoid this gender imbalance remain poorly explored in STEM, including ecology and evolution areas. In the last decades, changes in the peer-review process towards double-anonymized (DA) have increased among ecology and evolution (EcoEvo) journals. Using comprehensive data on articles from 18 selected EcoEvo journals with an impact factor >1, we tested the effect of the DA peer-review process in female-leading (i.e., first and senior authors) articles. We tested whether the representation of female-leading authors differs between double and single-anonymized (SA) peer-reviewed journals. Also, we tested if the adoption of the DA by previous SA journals has increased the representativeness of female-leading authors over time. We found that publications led by female authors did not differ between DA and SA journals. Moreover, female-leading articles did not increase after changes from SA to DA peer-review. Tackling female underrepresentation in science is a complex task requiring many interventions. Still, our results highlight that adopting the DA peer-review system alone could be insufficient in fostering gender equality in EcoEvo scientific publications. Ecologists and evolutionists understand how diversity is important to ecosystems' resilience in facing environmental changes. The question remaining is: why is it so difficult to promote and keep this "diversity" in addition to equity and inclusion in the academic environment? We thus argue that all scientists, mentors, and research centers must be engaged in promoting solutions to gender bias by fostering diversity, inclusion, and affirmative measures.

A validation study on the factors affecting the practice modes of open peer review

In this paper we conduct a validation study on the factors affecting the practice modes of open peer review. Taking the Open Access Journals (OAJ) in Directory of Open Access Journals (DOAJ) as the research objects, we crawled the internet to gather their relevant data. Based on the method of categorical variable assignment, a quantitative analysis was performed on the qualitative factors that affect the practice modes of open peer review. A multi-dimensional analysis chart is used to illustrate the relationships between the factors. Optimal scale regression modeling and discriminant analysis were also employed to reveal the degrees of influences by the factors. The public categories of “type of open peer review” and “reviewer identity” are closely related to each other. “Reviewer identity” has evident positive influence on “type of open peer review”, and the degree of influence is the highest. Therefore, “reviewer identity” is the primary and most crucial factor affecting open peer review practice modes. “Review report” and “order of review report and publication” are the secondary ones. Whether or not the identities of review experts are open has become the most important factor affecting the practice modes of open peer review. Transparent peer review is currently the most effective practice mode of open peer review. Technologies like block chain can be used to address the psychological uneasiness for the peer review experts who are concerned with privacy issues. The fact that most OAJs use “pre-publication review” shows that open peer review still plays the traditional role of “academic goalkeeper”. Publication of peer review reports actually helps peer review experts augment their reputation, which in turn practically promotes the development of open peer review.

Removing anonymity protection and utilization review decisions: a real-world case under a single-payer health system

The effects of anonymity on utilization review has never been examined in the real world. This study aimed to evaluate the impact of removing anonymity protection for claims reviewers on their review decisions. Using a single-blinded repeated measures design, we randomly selected 1457 claims cases (with 12,237 orders) that had been anonymously reviewed and reimbursed in 2016 and had them re-reviewed in a signed review program in 2017 under the Taiwanese National Health Insurance scheme. The signed review policy significantly decreased the likelihood of a deduction decision at the case and the order level (P < 0.001). Furthermore, signed reviewers tended to make more "too lenient" decisions, and were less likely to make "too harsh" decisions. Removing anonymity protection dramatically reduced the deduction rate and overturned the tendency of decisions from "too harsh" to "too lenient". However, whether to maintain the anonymity of utilization reviews is a challenge for health authorities around the globe.

Transparency in peer review: Exploring the content and tone of reviewers' confidential comments to editors

Purpose

Recent calls to improve transparency in peer review have prompted examination of many aspects of the peer-review process. Peer-review systems often allow confidential comments to editors that could reduce transparency to authors, yet this option has escaped scrutiny. Our study explores 1) how reviewers use the confidential comments section and 2) alignment between comments to the editor and comments to authors with respect to content and tone.

Methods

Our dataset included 358 reviews of 168 manuscripts submitted between January 1, 2019 and August 24, 2020 to a health professions education journal with a single blind review process. We first identified reviews containing comments to the editor. Then, for the reviews with comments, we used procedures consistent with conventional and directed qualitative content analysis to develop a coding scheme and code comments for content, tone, and section of the manuscript. For reviews in which the reviewer recommended “reject,” we coded for alignment between reviewers’ comments to the editor and to authors. We report descriptive statistics.

Results

49% of reviews contained comments to the editor (n = 176). Most of these comments summarized the reviewers’ impression of the article (85%), which included explicit reference to their recommended decision (44%) and suitability for the journal (10%). The majority of comments addressed argument quality (56%) or research design/methods/data (51%). The tone of comments tended to be critical (40%) or constructive (34%). For the 86 reviews recommending “reject,” the majority of comments to the editor contained content that also appeared in comments to the authors (80%); additional content tended to be irrelevant to the manuscript. Tone frequently aligned (91%).

Conclusion

Findings indicate variability in how reviewers use the confidential comments to editor section in online peer-review systems, though generally the way they use them suggests integrity and transparency to authors.

Which peer reviewers voluntarily reveal their identity to authors? Insights into the consequences of open-identities peer review

Identifying reviewers is argued to improve the quality and fairness of peer review, but is generally disfavoured by reviewers. To gain some insight into the factors that influence when reviewers are willing to have their identity revealed, I examined which reviewers voluntarily reveal their identities to authors at the journal Functional Ecology, at which reviewer identities are confidential unless reviewers sign their comments to authors. I found that 5.6% of reviewers signed their comments to authors. This proportion increased slightly over time, from 4.4% in 2003-2005 to 6.7% in 2013-2015. Male reviewers were 1.8 times more likely to sign their comments to authors than were female reviewers, and this difference persisted over time. Few reviewers signed all of their reviews; reviewers were more likely to sign their reviews when their rating of the manuscript was more positive, and papers that had at least one signed review were more likely to be invited for revision. Signed reviews were, on average, longer and recommended more references to authors. My analyses cannot distinguish cause and effect for the patterns observed, but my results suggest that 'open-identities' review, in which reviewers are not permitted to be anonymous, will probably reduce the degree to which reviewers are critical in their assessment of manuscripts and will differentially affect recruitment of male and female reviewers, negatively affecting the diversity of reviewers recruited by journals.

Effects of Experimental Interventions to Improve the Biomedical Peer-Review Process: A Systematic Review and Meta-Analysis

Background

Quality of the peer‐review process has been tested only in small studies. We describe and summarize the randomized trials that investigated interventions aimed at improving peer‐review process of biomedical manuscripts.

Methods and Results

All randomized trials comparing different peer‐review interventions at author‐, reviewer‐, and/or editor‐level were included. Differences between traditional and intervention‐modified peer‐review processes were pooled as standardized mean difference (SMD) in quality based on the definitions used in the individual studies. Main outcomes assessed were quality and duration of the peer‐review process. Five‐hundred and seventy‐five studies were retrieved, eventually yielding 24 randomized trials. Eight studies evaluated the effect of interventions at author‐level, 16 at reviewer‐level, and 3 at editor‐level. Three studies investigated interventions at multiple levels. The effects of the interventions were reported as mean change in review quality, duration of the peer‐review process, acceptance/rejection rate, manuscript quality, and number of errors detected in 13, 11, 5, 4, and 3 studies, respectively. At network meta‐analysis, reviewer‐level interventions were associated with a significant improvement in review quality (SMD, 0.20 [0.06 to 0.33]), at the cost of increased duration of the review process (SMD, 0.15 [0.01 to 0.29]), except for reviewer blinding. Author‐ and editor‐level interventions did not significantly impact peer‐review quality and duration (respectively, SMD, 0.17 [−0.16 to 0.51] and SMD, 0.19 [−0.40 to 0.79] for quality, and SMD, 0.17 [−0.16 to 0.51] and SMD, 0.19 [−0.40 to 0.79] for duration).

Conclusions

Modifications of the traditional peer‐review process at reviewer‐level are associated with improved quality, at the price of longer duration. Further studies are needed.

Registration

URL: https://www.crd.york.ac.uk/prospero; Unique identifier: CRD42020187910.

Peer reviewing made easier: your questions answered

Peer reviewing is a hugely important part of the scientific process that ensures published articles are of sufficient quality to deserve dissemination to the wider scientific community. Building on a previous article published in this journal, this article addresses topics that potential or practising peer reviewers may find useful. These include what peer reviewing is, why do peer reviews, how to become a reviewer, what to write in a review and where to find more information. It includes a template for writing a review, and lists various websites and guidelines that can help ease the entire process depending on what type of article is being reviewed. Peer reviewing can be enormously rewarding and help clinicians diversify their scope of work while also benefiting the scientific community by contributing to the quality control of published work.

Open science saves lives: lessons from the COVID-19 pandemic

In the last decade Open Science principles have been successfully advocated for and are being slowly adopted in different research communities. In response to the COVID-19 pandemic many publishers and researchers have sped up their adoption of Open Science practices, sometimes embracing them fully and sometimes partially or in a sub-optimal manner. In this article, we express concerns about the violation of some of the Open Science principles and its potential impact on the quality of research output. We provide evidence of the misuses of these principles at different stages of the scientific process. We call for a wider adoption of Open Science practices in the hope that this work will encourage a broader endorsement of Open Science principles and serve as a reminder that science should always be a rigorous process, reliable and transparent, especially in the context of a pandemic where research findings are being translated into practice even more rapidly. We provide all data and scripts at https://osf.io/renxy/ .

Quality of peer review reports submitted to a specialty psychiatry journal

BACKGROUND

Though peer review is at the heart of scholarly publishing, peer review reports are not commonly investigated. We aimed to analyse the quality and structure of review reports submitted to the Indian Journal of Psychological Medicine (IJPM).

METHODS

We systematically analysed the structure, tone, and quality of peer review reports of all original articles submitted to the journal between January 1, 2018 to May 15, 2020. Quality assessment was done using the 8-item Review Quality Instrument (RQI).

RESULTS

A total of 527 review reports from 291 original articles were analysed. More than two-thirds of review reports were provided as inline comments (n = 368, 69.8 %). Most of the review reports were not well-structured; only a few provided a summary (n = 64, 13.2 %) or divided the comments into major and minor ones (n = 12, 2.5 %). Nearly a quarter had negative wordings (n = 117, 24.1 %) and a minority had a frankly unprofessional tone (n = 43, 8.8 %). The global rating was "poor" (n = 266, 50.5 %) or "below average" (n = 203, 38.5 %) for most reports.

CONCLUSION

Most of the peer reviews submitted to the IJPM were not structured and obtained low scores on the RQI domains. Concerted efforts are needed to improve the quality of peer reviews and to provide training for reviewers.

Why Participate in Peer Review?

The scholarly publishing enterprise is currently undergoing a "crisis," likely exacerbated by the global pandemic, in which peer reviewers are increasingly less available to perform reviews at the same time the flow of submitted manuscripts has not subsided. This editorial considers possible reasons why scholars might decline to participate in the peer review process, including the lack of compensation for this time-consuming and effort-laden service activity; questions about the fairness, validity, and efficacy of peer review; a commonly experienced dearth of training in peer review skills; and the fact that a lack of diversity in the sciences, academia, and the professions is reflected in the makeup of scholarly publishing leadership such that peer review is not necessarily conducted by one's "peers." Potential considerations are also offered on the other side of the ledger. These include the benefits that accrue to our own scholarship and publishing acumen when we review the work of others; the value of peer review to the quality of our journals and the excellence of our field; the positive contributions that thoughtful and educative reviews can make to the work of our colleagues; recent initiatives designed to increase representativeness, reduce bias, and guard against conflicts of interest in the peer reviewing process; the availability of guides and tutorials to assist emerging scholars to develop the relevant skills and acumen; and the ways in which peer reviewing can set the stage for professional growth and entry into leadership positions in the field of scholarly publishing.

Journal policies and editors' opinions on peer review

Peer review practices differ substantially between journals and disciplines. This study presents the results of a survey of 322 editors of journals in ecology, economics, medicine, physics and psychology. We found that 49% of the journals surveyed checked all manuscripts for plagiarism, that 61% allowed authors to recommend both for and against specific reviewers, and that less than 6% used a form of open peer review. Most journals did not have an official policy on altering reports from reviewers, but 91% of editors identified at least one situation in which it was appropriate for an editor to alter a report. Editors were also asked for their views on five issues related to publication ethics. A majority expressed support for co-reviewing, reviewers requesting access to data, reviewers recommending citations to their work, editors publishing in their own journals, and replication studies. Our results provide a window into what is largely an opaque aspect of the scientific process. We hope the findings will inform the debate about the role and transparency of peer review in scholarly publishing.

Open versus blind peer review: is anonymity better than transparency?

Peer review is widely accepted as essential to ensuring scientific quality in academic journals, yet little training is provided in the specifics of how to conduct peer review. In this article we describe the different forms of peer review, with a particular focus on the differences between single-blind, double-blind and open peer review, and the advantages and disadvantages of each. These illustrate some of the challenges facing the community of authors, editors, reviewers and readers in relation to the process of peer review. We also describe other forms of peer review, such as post-publication review, transferable review and collaborative review, and encourage clinicians and academics at all training stages to engage in the practice of peer review as part of continuing professional development.

Quantifying professionalism in peer review

Background

The process of peer-review in academia has attracted criticism surrounding issues of bias, fairness, and professionalism; however, frequency of occurrence of such comments is unknown.

Methods

We evaluated 1491 sets of reviewer comments from the fields of “Ecology and Evolution” and “Behavioural Medicine,” of which 920 were retrieved from the online review repository Publons and 571 were obtained from six early career investigators. Comment sets were coded for the occurrence of “unprofessional comments” and “incomplete, inaccurate or unsubstantiated critiques” using an a-prior rubric based on our published research. Results are presented as absolute numbers and percentages.

Results

Overall, 12% (179) of comment sets included at least one unprofessional comment towards the author or their work, and 41% (611) contained incomplete, inaccurate of unsubstantiated critiques (IIUC).

Conclusions

The large number of unprofessional comments, and IIUCs observed could heighten psychological distress among investigators, particularly those at an early stage in their career. We suggest that development and adherence to a universally agreed upon reviewer code of conduct is necessary to improve the quality and professional experience of peer review.

Does the use of open, non-anonymous peer review in scholarly publishing introduce bias? Evidence from the F1000Research post-publication open peer review publishing model

As part of moves towards open knowledge practices, making peer review open is cited as a way to enable fuller scrutiny and transparency of assessments around research. There are now many flavours of open peer review in use across scholarly publishing, including where reviews are fully attributable and the reviewer is named. This study examines whether there is any evidence of bias in two areas of common critique of open, non-anonymous (named) peer review – and used in the post-publication, peer review system operated by the open-access scholarly publishing platform F1000Research. First, is there evidence of potential bias where a reviewer based in a specific country assesses the work of an author also based in the same country? Second, are reviewers influenced by being able to see the comments and know the origins of a previous reviewer? Based on over 4 years of open peer review data, we found some weak evidence that being based in the same country as an author may influence a reviewer’s decision, while there was insufficient evidence to conclude that being able to read an existing published review prior to submitting a review encourages conformity. Thus, while immediate publishing of peer review reports appears to be unproblematic, caution may be needed when selecting same-country reviewers in open systems if other studies confirm these results.

Open up: a survey on open and non-anonymized peer reviewing

BACKGROUND

Our aim is to highlight the benefits and limitations of open and non-anonymized peer review. Our argument is based on the literature and on responses to a survey on the reviewing process of alt.chi, a more or less open review track within the so-called Computer Human Interaction (CHI) conference, the predominant conference in the field of human-computer interaction. This track currently is the only implementation of an open peer review process in the field of human-computer interaction while, with the recent increase in interest in open scientific practices, open review is now being considered and used in other fields.

METHODS

We ran an online survey with 30 responses from alt.chi authors and reviewers, collecting quantitative data using multiple-choice questions and Likert scales. Qualitative data were collected using open questions.

RESULTS

Our main quantitative result is that respondents are more positive to open and non-anonymous reviewing for alt.chi than for other parts of the CHI conference. The qualitative data specifically highlight the benefits of open and transparent academic discussions. The data and scripts are available on https://osf.io/vuw7h/, and the figures and follow-up work on http://tiny.cc/OpenReviews.

CONCLUSION

While the benefits are quite clear and the system is generally well-liked by alt.chi participants, they remain reluctant to see it used in other venues. This concurs with a number of recent studies that suggest a divergence between support for a more open review process and its practical implementation.

Automatically detecting open academic review praise and criticism

Purpose

Peer reviewer evaluations of academic papers are known to be variable in content and overall judgements but are important academic publishing safeguards. This article introduces a sentiment analysis program, PeerJudge, to detect praise and criticism in peer evaluations. It is designed to support editorial management decisions and reviewers in the scholarly publishing process and for grant funding decision workflows. The initial version of PeerJudge is tailored for reviews from F1000Research's open peer review publishing platform.

Design/methodology/approach

PeerJudge uses a lexical sentiment analysis approach with a human-coded initial sentiment lexicon and machine learning adjustments and additions. It was built with an F1000Research development corpus and evaluated on a different F1000Research test corpus using reviewer ratings.

Findings

PeerJudge can predict F1000Research judgements from negative evaluations in reviewers' comments more accurately than baseline approaches, although not from positive reviewer comments, which seem to be largely unrelated to reviewer decisions. Within the F1000Research mode of post-publication peer review, the absence of any detected negative comments is a reliable indicator that an article will be ‘approved’, but the presence of moderately negative comments could lead to either an approved or approved with reservations decision.

Originality/value

PeerJudge is the first transparent AI approach to peer review sentiment detection. It may be used to identify anomalous reviews with text potentially not matching judgements for individual checks or systematic bias assessments.

Open peer review: promoting transparency in open science

Open peer review (OPR), where review reports and reviewers’ identities are published alongside the articles, represents one of the last aspects of the open science movement to be widely embraced, although its adoption has been growing since the turn of the century. This study provides the first comprehensive investigation of OPR adoption, its early adopters and the implementation approaches used. Current bibliographic databases do not systematically index OPR journals, nor do the OPR journals clearly state their policies on open identities and open reports. Using various methods, we identified 617 OPR journals that published at least one article with open identities or open reports as of 2019 and analyzed their wide-ranging implementations to derive emerging OPR practices. The findings suggest that: (1) there has been a steady growth in OPR adoption since 2001, when 38 journals initially adopted OPR, with more rapid growth since 2017; (2) OPR adoption is most prevalent in medical and scientific disciplines (79.9%); (3) five publishers are responsible for 81% of the identified OPR journals; (4) early adopter publishers have implemented OPR in different ways, resulting in different levels of transparency. Across the variations in OPR implementations, two important factors define the degree of transparency: open identities and open reports. Open identities may include reviewer names and affiliation as well as credentials; open reports may include timestamped review histories consisting of referee reports and author rebuttals or a letter from the editor integrating reviewers’ comments. When and where open reports can be accessed are also important factors indicating the OPR transparency level. Publishers of optional OPR journals should add metric data in their annual status reports.

Open Science Practices at the Journal of Traumatic Stress

This editorial describes new initiatives designed to promote and maintain open science practices (OSP) at the Journal of Traumatic Stress, to be enacted beginning January 2020. Following a brief description of the rationale underlying the argument for conducting and reporting research in ways that maximize transparency and replicability, this article summarizes changes in Journal submission and publication procedures that are designed to foster and highlight such practices. These include requesting an Open Science Practices Statement from authors of all accepted manuscripts, which will be published as supplementary material for each article, and providing authors with the opportunity to earn OSP badges for preregistering studies, making data available to other researchers by posting on a third party archive, and making available research materials and codes used in the study.

Towards the enhancement of quality publication practices in clinical psychological science

Addressing the 'replication crisis' and questionable research practices are at the forefront of international research agendas in clinical psychological science. The aim of this paper is to consider how the quality of research practices can be improved by a specific focus on publication practices. Currently, the responsibility for documenting quality research practices is primarily placed on authors. However, barriers to improved quality publication practices cut across all levels of the research community and require a broader approach that shares the burden for ensuring the production of high quality publications. We describe a framework that is intended to be ambitious and aspirational and encourage discussion and adoption of strategies to improve quality publication practices (QPPs). The framework cuts across multiple stakeholders and is designed to enhance (a) the quality of reporting; (b) adherence to protocols and guidelines; (c) timely accessibility of study materials and data. We discuss how QPPs might be improved by (a) funding bodies considering formally supporting QPPs; (b) research institutions encouraging a research culture that espouses quality research practices, and internally supporting QPP review processes and professional development in QPPs; (c) journals expanding editorial teams to include reviewers with design and statistical expertise, considering strategies to enhance QPP adherence during the peer review process, and committing to ongoing assessment and development of QPP training for peer reviewers; and (d) authors and peer reviewers integrating QPPs during the manuscript preparation/peer review process, engaging in ongoing QPP training, and committing to openness and transparency initiatives. We discuss the current state and potential next steps within each stage of the framework and provide information and resources to enhance QPPs. We hope that the suggestions offered here inspire research institutions, leaders and faculty to discuss, reflect on, and take action towards, integrating these, or other, QPPs into their research practice and workplace.

Predatory publishing and the Ghana experience: A call to action for information professionals

Researchers in developing countries are more likely to publish in predatory journals (Xia et al., 2015). This study investigates the understanding that research scientists in Ghana, a developing country, have about predatory journals and their publishing practices. Using a mixed methods approach, research scientists within one cluster of research organizations in Ghana were asked about their awareness of the characteristics of predatory journals, based on their own experience as a researcher. Their publications were also examined. The results indicate that most of the research scientists in this study are aware of predatory journals and are often solicited by them, but are less aware of tools they can use to determine the quality of a particular publication. In addition, 12% of the articles published that make up 24% of the unique journals in which these researchers published could be considered “predatory”. The findings of this research are significant because they indicate that research scientists may have more awareness of predatory journals than is expected, but that they may lack the training or tools necessary for deciding whether or not a journal is legitimate.

Opening and Reusing Transparent Peer Reviews with Automatic Article Annotation

An increasing number of scientific publications are created in open and transparent peer review models: a submission is published first, and then reviewers are invited, or a submission is reviewed in a closed environment but then these reviews are published with the final article, or combinations of these. Reasons for open peer review include giving better credit to reviewers, and enabling readers to better appraise the quality of a publication. In most cases, the full, unstructured text of an open review is published next to the full, unstructured text of the article reviewed. This approach prevents human readers from getting a quick impression of the quality of parts of an article, and it does not easily support secondary exploitation, e.g., for scientometrics on reviews. While document formats have been proposed for publishing structured articles including reviews, integrated tool support for entire open peer review workflows resulting in such documents is still scarce. We present AR-Annotator, the Automatic Article and Review Annotator which employs a semantic information model of an article and its reviews, using semantic markup and unique identifiers for all entities of interest. The fine-grained article structure is not only exposed to authors and reviewers but also preserved in the published version. We publish articles and their reviews in a Linked Data representation and thus maximise their reusability by third party applications. We demonstrate this reusability by running quality-related queries against the structured representation of articles and their reviews.

The effect of publishing peer review reports on referee behavior in five scholarly journals

To increase transparency in science, some scholarly journals are publishing peer review reports. But it is unclear how this practice affects the peer review process. Here, we examine the effect of publishing peer review reports on referee behavior in five scholarly journals involved in a pilot study at Elsevier. By considering 9,220 submissions and 18,525 reviews from 2010 to 2017, we measured changes both before and during the pilot and found that publishing reports did not significantly compromise referees' willingness to review, recommendations, or turn-around times. Younger and non-academic scholars were more willing to accept to review and provided more positive and objective recommendations. Male referees tended to write more constructive reports during the pilot. Only 8.1% of referees agreed to reveal their identity in the published report. These findings suggest that open peer review does not compromise the process, at least when referees are able to protect their anonymity.

Quality of the Reviews Submitted by Attendees of a Workshop on Peer Review

OBJECTIVE

The objective of the study was to study the methodological quality and error detection of the review by the participants of a peer review workshop.

METHODS

All participants of the workshop were invited to peer review a randomized controlled trial. The manuscript was E-mailed to them after introducing eight deliberate errors to it. Specific instructions and a deadline were provided. All the reviews were analyzed using review quality instrument (RQI). Furthermore, the rate and the type of errors identified were recorded.

RESULTS

Of 25 participants, 16 (64%) returned the reviews. The mean total score on RQI was 4.12 (standard deviation 0.70, 95% confidence interval 3.74-4.50); the items which most reviewers did not discuss where the importance of research question and originality of the paper. The number of errors correctly identified varied from 0 to 6 (median 3), the most common being a wrong conclusion (87.5%), randomization procedure (50%), written informed consent (50%), ethics committee approval (42.8%), and masking (31.2%). Only 5 (31.2%) gave an overall recommendation on whether the manuscript should be accepted or not.

CONCLUSIONS

Major errors were readily identified by the reviewers; however, the need for training was felt in some areas in which the review quality was modest.

The perceived feasibility of methods to reduce publication bias

Publication bias is prevalent within the scientific literature. Whilst there are multiple ideas on how to reduce publication bias, only a minority of journals have made substantive changes to address the problem. We aimed to explore the perceived feasibility of strategies to reduce publication bias by gauging opinions of journal editors (n = 73) and other academics/researchers (n = 160) regarding nine methods of publishing and peer-reviewing research: mandatory publication, negative results journals/articles, open reviewing, peer-review training and accreditation, post-publication review, pre-study publication of methodology, published rejection lists, research registration, and two-stage review. Participants completed a questionnaire asking both quantitative (multiple choice or Likert scales) and qualitative (open-ended) questions regarding the barriers to implementing each suggestion, and their strengths and limitations. Participants were asked to rate the nine suggestions, then choose the method they felt was most effective. Mandatory publication was most popularly selected as the 'most effective' method of reducing publication bias for editors (25%), and was the third most popular choice for academics/researchers (14%). The most common selection for academics/researchers was two-stage review (26%), but fewer editors prioritised this (11%). Negative results journals/articles were the second and third most common choices for academics/researchers (21%) and editors (16%), respectively. Editors more commonly chose research registration as 'most effective' (21%), which was favoured by only 6% of academics/researchers. Whilst mandatory publication was generally favoured by respondents, it is infeasible to trial at a journal level. Where suggestions have already been implemented (e.g. negative results journals/articles, trial registration), efforts should be made to objectively assess their efficacy. Two-stage review should be further trialled as its popularity amongst academics/researchers suggests it may be well received, though editors may be less receptive. Several underlying barriers to change also emerged, including scientific culture, impact factors, and researcher training; these should be further explored to reduce publication bias.

Open Peer Review in Scientific Publishing: A Web Mining Study of PeerJ Authors and Reviewers

Purpose

To understand how authors and reviewers are accepting and embracing Open Peer Review (OPR), one of the newest innovations in the Open Science movement.

Design/methodology/approach

This research collected and analyzed data from the Open Access journal PeerJ over its first three years (2013–2016). Web data were scraped, cleaned, and structured using several Web tools and programs. The structured data were imported into a relational database. Data analyses were conducted using analytical tools as well as programs developed by the researchers.

Findings

PeerJ, which supports optional OPR, has a broad international representation of authors and referees. Approximately 73.89% of articles provide full review histories. Of the articles with published review histories, 17.61% had identities of all reviewers and 52.57% had at least one signed reviewer. In total, 43.23% of all reviews were signed. The observed proportions of signed reviews have been relatively stable over the period since the Journal’s inception.

Research limitations

This research is constrained by the availability of the peer review history data. Some peer reviews were not available when the authors opted out of publishing their review histories. The anonymity of reviewers made it impossible to give an accurate count of reviewers who contributed to the review process.

Practical implications

These findings shed light on the current characteristics of OPR. Given the policy that authors are encouraged to make their articles’ review history public and referees are encouraged to sign their review reports, the three years of PeerJ review data demonstrate that there is still some reluctance by authors to make their reviews public and by reviewers to identify themselves.

Originality/value

This is the first study to closely examine PeerJ as an example of an OPR model journal. As Open Science moves further towards open research, OPR is a final and critical component. Research in this area must identify the best policies and paths towards a transparent and open peer review process for scientific communication.

Collister L, Pikas CK, Crick T, Masuzzo P, Caravaggi A, Berg DR, Niemeyer KE, Ross-Hellauer T, Mannheimer S, Rigling L, Katz DS, Greshake Tzovaras B, Pacheco-Mendoza J, Fatima N, Poblet M, Isaakidis M, Irawan DE, Renaut S, Madan CR, Matthias L, Nørgaard Kjær J, O'Donnell DP, Neylon C, Kearns S, Selvaraju M, Colomb J. A multi-disciplinary perspective on emergent and future innovations in peer review

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

Collister L, Pikas CK, Crick T, Masuzzo P, Caravaggi A, Berg DR, Niemeyer KE, Ross-Hellauer T, Mannheimer S, Rigling L, Katz DS, Greshake Tzovaras B, Pacheco-Mendoza J, Fatima N, Poblet M, Isaakidis M, Irawan DE, Renaut S, Madan CR, Matthias L, Nørgaard Kjær J, O'Donnell DP, Neylon C, Kearns S, Selvaraju M, Colomb J. A multi-disciplinary perspective on emergent and future innovations in peer review

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of Web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform current models while avoiding as many of the biases of existing systems as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that, at least partially, resolves many of the technical and social issues associated with peer review, and can potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

Collister L, Pikas CK, Crick T, Masuzzo P, Caravaggi A, Berg DR, Niemeyer KE, Ross-Hellauer T, Mannheimer S, Rigling L, Katz DS, Greshake Tzovaras B, Pacheco-Mendoza J, Fatima N, Poblet M, Isaakidis M, Irawan DE, Renaut S, Madan CR, Matthias L, Nørgaard Kjær J, O'Donnell DP, Neylon C, Kearns S, Selvaraju M, Colomb J. A multi-disciplinary perspective on emergent and future innovations in peer review

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

The credibility crisis in research: Can economics tools help?

The issue of nonreplicable evidence has attracted considerable attention across biomedical and other sciences. This concern is accompanied by an increasing interest in reforming research incentives and practices. How to optimally perform these reforms is a scientific problem in itself, and economics has several scientific methods that can help evaluate research reforms. Here, we review these methods and show their potential. Prominent among them are mathematical modeling and laboratory experiments that constitute affordable ways to approximate the effects of policies with wide-ranging implications.

Efficacy of Double-Blind Peer Review in an Imaging Subspecialty Journal

BACKGROUND AND PURPOSE

Many scientific journals use double-blind peer review to minimize potential reviewer bias concerning publication recommendations. However, because neuroradiology is a relatively small subspecialty, this process may be limited by prior knowledge of the authors' work or associated institutions. We sought to investigate the efficacy of reviewer blinding and determine the impact that unblinding may have on manuscript acceptance.

MATERIALS AND METHODS

For manuscripts submitted to the American Journal of Neuroradiology (AJNR) from January through June 2015, reviewers completed a brief anonymous questionnaire after submitting their evaluations, assessing whether they were familiar with the research or had knowledge of the authors or institutions from which the work originated.

RESULTS

The response rate for 1079 questionnaires was 98.8%; 12.9% of reviewers knew or suspected that they knew authors, and 15.3% knew or suspected that they knew the associated institutions. Reviewers correctly identified the authors in 90.3% of cases and correctly stated the institutions in 86.8% of cases. Unblinding resulted from self-citation in 34.1% for both authorship and institutions. The acceptance rate when reviewers knew or suspected that they knew the authors was 57/137 (41.6%) and 262/929 (28.2%) when reviewers did not. The acceptance rate when reviewers knew or suspected that they knew the institutions was 60/163 (36.8%) and 259/903 (28.7%) when they did not. The Fisher exact test showed that author (P < .038) and institution (P < .039) familiarity was associated with greater manuscript acceptance.

CONCLUSIONS

While the AJNR process of double-blind peer review minimizes reviewer bias, perceived knowledge of the author and institution is associated with a higher rate of manuscript acceptance.

A manifesto for reproducible science

Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.

Referees’ Attitudes toward Open Peer Review and Electronic Transmission of Papers

A survey was mailed to 293 referees from the review board of Food Science and Technology International with the following personal characteristics: ages: 35–45 (35%), 45–55 (37%), and 55–65 (27%); 93% PhD graduates; 69% male, 98% researchers, 82% teachers too, 85% review for other journals as well to assess reviewers’ attitudes or preferences in favor of or against masking their identity, and toward the electronic transmission of papers for review. The reviewers were mainly from Europe, North America, and South America. The questionnaire was anonymous and asked if respondents were in favor of an open review or masking of the reviewers, and if they agreed with the electronic transmission of the papers for their review (both from the point of view of author and reviewer). The response rate was 35% (103 respondents). The consistency between the answers as being authors or reviewers when asked by the peer review process was significant ( P<0.001) without significant differences in terms of gender or age. Seventy-five percent were in favor of masking reviewers, and 17% completely favored unblinded review. The consistency between the answers for paper transmission was significant ( P<0.001) without significant differences in terms of gender or age. Seventy-five percent were in favor of electronic transmission, 25% were against it. There was a significant association between the answers in favor of or against e-transmission and the age either as reviewers ( P=0.009) or as authors ( P= 0.031). The other associations between the system of review and gender or age were not significant. There was a preference among the participants for masking the reviewers, and a tendency to use the Web as the transmission medium because it is considered faster, easier, simpler, and more economic.

How open science helps researchers succeed

Open access, open data, open source and other open scholarship practices are growing in popularity and necessity. However, widespread adoption of these practices has not yet been achieved. One reason is that researchers are uncertain about how sharing their work will affect their careers. We review literature demonstrating that open research is associated with increases in citations, media attention, potential collaborators, job opportunities and funding opportunities. These findings are evidence that open research practices bring significant benefits to researchers relative to more traditional closed practices.

Sensitivity analysis after multiple imputation under missing at random: a weighting approach

Multiple imputation (MI) is now well established as a flexible, general, method for the analysis of data sets with missing values. Most implementations assume the missing data are `missing at random' (MAR), that is, given the observed data, the reason for the missing data does not depend on the unseen data. However, although this is a helpful and simplifying working assumption, it is unlikely to be true in practice. Assessing the sensitivity of the analysis to the MAR assumption is therefore important. However, there is very limited MI software for this. Further, analysis of a data set with missing values that are not missing at random (NMAR) is complicated by the need to extend the MAR imputation model to include a model for the reason for dropout. Here, we propose a simple alternative. We first impute under MAR and obtain parameter estimates for each imputed data set. The overall NMAR parameter estimate is a weighted average of these parameter estimates, where the weights depend on the assumed degree of departure from MAR. In some settings, this approach gives results that closely agree with joint modelling as the number of imputations increases. In others, it provides ball-park estimates of the results of full NMAR modelling, indicating the extent to which it is necessary and providing a check on its results. We illustrate our approach with a small simulation study, and the analysis of data from a trial of interventions to improve the quality of peer review.

The Influence of Peer Review on the Research Assessment Exercise

The use of peer review within both the scholarly communication system and the UK’s Research Assessment Exercise is reviewed. The common denominator is that of peer-reviewed academic journals, since peer review is used by referees to aid publication decisions and by RAE panel members to evaluate a department’s research performance. We propose that since academic research is now subject to peer review at all stages of evaluation, it is becoming an accepted method of rewarding (by funding) research. The growth of electronic publications (both toll-access and open access) provides possibilities for changes to some of the process of peer review and RAE, but the fundamental model of peer review to reduce the number of poor quality publications will remain. The paper concludes that because of the many criticisms of peer review, it is unwise to base funding decisions on second level peer review of articles that have already undergone peer review.