Who is afraid of reviewers' comments? Or, why anything can be published and anything can be cited

Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature

BACKGROUND

Preprint usage is growing rapidly in the life sciences; however, questions remain on the relative quality of preprints when compared to published articles. An objective dimension of quality that is readily measurable is completeness of reporting, as transparency can improve the reader's ability to independently interpret data and reproduce findings.

METHODS

In this observational study, we initially compared independent samples of articles published in bioRxiv and in PubMed-indexed journals in 2016 using a quality of reporting questionnaire. After that, we performed paired comparisons between preprints from bioRxiv to their own peer-reviewed versions in journals.

RESULTS

Peer-reviewed articles had, on average, higher quality of reporting than preprints, although the difference was small, with absolute differences of 5.0% [95% CI 1.4, 8.6] and 4.7% [95% CI 2.4, 7.0] of reported items in the independent samples and paired sample comparison, respectively. There were larger differences favoring peer-reviewed articles in subjective ratings of how clearly titles and abstracts presented the main findings and how easy it was to locate relevant reporting information. Changes in reporting from preprints to peer-reviewed versions did not correlate with the impact factor of the publication venue or with the time lag from bioRxiv to journal publication.

CONCLUSIONS

Our results suggest that, on average, publication in a peer-reviewed journal is associated with improvement in quality of reporting. They also show that quality of reporting in preprints in the life sciences is within a similar range as that of peer-reviewed articles, albeit slightly lower on average, supporting the idea that preprints should be considered valid scientific contributions.

Peer Review and Transparency in Evidence-Source Selection in Value and Health Technology Assessment

OBJECTIVES

Value and health technology assessment (V/HTA) is often used in clinical, access, and reimbursement decisions. V/HTA data-source selection may not be transparent, which is a necessary element for stakeholder understanding and trust and for fostering accountability among decision makers. Peer review is considered one mechanism for judging data trustworthiness. Our objective was (1) to use publicly available documentation of V/HTA methods to identify requirements for inclusion of peer-reviewed evidence sources, (2) to compare and contrast US and non-US approaches, and (3) to assess evidence sources used in published V/HTA reports.

METHODS

Publicly available methods documentation from 11 V/HTA organizations in North America and Europe were manually searched and abstracted for descriptions of requirements and recommendations regarding search strategy and evidence-source selection. The bibliographies of a subset of V/HTA reports published in 2018 were manually abstracted for evidence-source types used in each.

RESULTS

Heterogeneity in evidence-source retrieval and selection was observed across all V/HTA organizations, with more pronounced differences between US and non-US organizations. Not all documentation of organizations' methods address the evidence-source selection processes (7 of 11), and few explicitly reference peer-reviewed sources (3 of 11). Documentation of the evidence-source selection strategy was inconsistent across reports (6 of 13), and the level of detail provided varied across organizations. Some information on evidence-source selection was often included in confidential documentation and was not publicly available.

CONCLUSIONS

Disparities exist among V/HTA organizations in requirements and guidance regarding evidence-source selection. Standardization of evidence-source selection strategies and documentation could help improve V/HTA transparency and has implications for decision making based on report findings.

A generalized view of self-citation: direct, co-author, collaborative, and coercive induced self-citation

The phenomenon of self-citation can present in many different forms, including direct, co-author, collaborative, and coercive induced self-citation. It can also pertain to the citation of single scientists, groups of scientists, journals, and institutions. This article presents some case studies of extreme self-citation practices. It also discusses the implications of different types of self-citation. Self-citation is not necessarily inappropriate by default. In fact, usually it is fully appropriate but often it is even necessary. Conversely, inappropriate self-citation practices may be highly misleading and may distort the scientific literature. Coercive induced self-citation is the most difficult to discover. Coercive Induced self-citation may happen directly from reviewers of articles, but also indirectly from reviewers of grants, scientific advisors who steer a research agenda, and leaders of funding agencies who may espouse spending disproportionately large funds in research domains that perpetuate their own self-legacy. Inappropriate self-citation can be only a surrogate marker of what might be much greater distortions of the scientific corpus towards conformity to specific opinions and biases. Inappropriate self-citations eventually affect also impact metrics. Different impact metrics vary in the extent to which they can be gamed through self-citation practices. Citation indices that are more gaming-proof are available and should be more widely used. We need more empirical studies to dissect the impact of different types of inappropriate self-citation and to examine the effectiveness of interventions to limit them.

Potentially coercive self-citation by peer reviewers: a cross-sectional study

OBJECTIVE

Peer reviewers sometimes request that authors cite their work, either appropriately or via coercive self-citation to highlight the reviewers' work. The objective of this study was to determine in peer reviews submitted to one biomedical journal (1) the extent of peer reviewer self-citation; (2) the proportion of reviews recommending revision or acceptance versus rejection that included reviewer self-citations; and (3) the proportion of reviewer self-citations versus citations to others that included a rationale.

METHODS

Peer reviews for manuscripts submitted in 2012 to the Journal of Psychosomatic Research were evaluated. Data extraction was performed independently by two investigators.

RESULTS

There were 616 peer reviews (526 reviewers; 276 manuscripts), of which 444 recommended revision or acceptance and 172 rejection. Of 428 total citations, there were 122 peer reviewer self-citations (29%) and 306 citations to others' work (71%). Self-citations were more common in reviews recommending revision or acceptance (105 of 316 citations; 33%) versus rejection (17/112; 15%; p<0.001). The percentage of self-citations with no rationale (26 of 122; 21%) was higher than for citations to others' work (15 of 306; 5%; p<0.001).

CONCLUSIONS

Self-citation in peer reviews is common and may reflect a combination of appropriate citation to research that should be cited in published articles and inappropriate citation intended to highlight the work of the peer reviewer. Providing instructions to peer reviewers about self-citation and asking them to indicate when and why they have self-cited may help to limit self-citation to appropriate, constructive recommendations.

How to review a scientific paper

Scientific observations must survive the scrutiny of experts before they are disseminated to the broader community because their publication in a scientific journal provides a stamp of validity. Although critical review of a manuscript by peers prior to publication in a scientific journal is a central element in this process, virtually no formal guidance is provided to reviewers about the nature of the task. In this article, the essence of peer review is described and critical steps in the process are summarized. The role of the peer reviewer as an intermediary and arbiter in the process of scientific communication between the authors and the readers via the vehicle of the particular journal is discussed and the responsibilities of the reviewer to each of the three parties (the author/s, readers, and the Journal editor) are defined. The two formal products of this activity are separate sets of reviewer comments to the editor and the authors and these are described. Ethical aspects of the process are considered and rewards accruing to the reviewer summarized.

Biomedical journals lack a consistent method to detect outcome reporting bias: a cross-sectional analysis

WHAT IS KNOWN AND OBJECTIVE

An increasing amount of recently published literature has implicated outcome reporting bias (ORB) as a major contributor to skewing data in both randomized controlled trials and systematic reviews; however, little is known about the current methods in place to detect ORB. This study aims to gain insight into the detection and management of ORB by biomedical journals.

METHODS

This was a cross-sectional analysis involving standardized questions via email or telephone with the top 30 biomedical journals (2012) ranked by impact factor. The Cochrane Database of Systematic Reviews was excluded leaving 29 journals in the sample.

RESULTS

Of 29 journals, 24 (83%) responded to our initial inquiry of which 14 (58%) answered our questions and 10 (42%) declined participation. Five (36%) of the responding journals indicated they had a specific method to detect ORB, whereas 9 (64%) did not have a specific method in place. The prevalence of ORB in the review process seemed to differ with 4 (29%) journals indicating ORB was found commonly, whereas 7 (50%) indicated ORB was uncommon or never detected by their journal previously. The majority (n = 10/14, 72%) of journals were unwilling to report or make discrepancies found in manuscripts available to the public. Although the minority, there were some journals (n = 4/14, 29%) which described thorough methods to detect ORB.

WHAT IS NEW AND CONCLUSION

Many journals seemed to lack a method with which to detect ORB and its estimated prevalence was much lower than that reported in literature suggesting inadequate detection. There exists a potential for overestimation of treatment effects of interventions and unclear risks. Fortunately, there are journals within this sample which appear to utilize comprehensive methods for detection of ORB, but overall, the data suggest improvements at the biomedical journal level for detecting and minimizing the effect of this bias are needed.

Reliability of reviewer ratings in the manuscript peer review process: an opportunity for improvement

Accountability to authors and readers cannot exist without proper peer review practices. Thus, the information a journal seeks from its peer reviewers and how it makes use of this information is paramount. Disagreement amongst peer reviewers can be considerable, resulting in very diverse comments to authors. Incorporating a clear scoring system for key concrete items and requiring referees to provide justification for scores may ensure that reviewers contribute in a consistently fair and effective manner. This article evaluates information collected from reviewers and proposes an example of a system that aims to improve accountability, while having the potential to make it easier for reviewers to perform a more objective review.

Best peer reviewers and the quality of peer review in biomedical journals

Current scholarly publications heavily rely on high quality peer review. Peer review, albeit imperfect, is aimed at improving science writing and editing. Evidence supporting peer review as a guarantor of the quality of biomedical publications is currently lacking. Its outcomes are largely dependent on the credentials of the reviewers. Several lines of evidence suggest that predictors of the best contributors to the process include affiliation to a good University and proper research training. Though the options to further improve peer review are currently limited, experts are in favor of formal education and courses on peer review for all contributors to this process. Long-term studies are warranted to assess the strengths and weaknesses of this approach.

Designing next-generation platforms for evaluating scientific output: what scientists can learn from the social web

Traditional pre-publication peer review of scientific output is a slow, inefficient, and unreliable process. Efforts to replace or supplement traditional evaluation models with open evaluation platforms that leverage advances in information technology are slowly gaining traction, but remain in the early stages of design and implementation. Here I discuss a number of considerations relevant to the development of such platforms. I focus particular attention on three core elements that next-generation evaluation platforms should strive to emphasize, including (1) open and transparent access to accumulated evaluation data, (2) personalized and highly customizable performance metrics, and (3) appropriate short-term incentivization of the userbase. Because all of these elements have already been successfully implemented on a large scale in hundreds of existing social web applications, I argue that development of new scientific evaluation platforms should proceed largely by adapting existing techniques rather than engineering entirely new evaluation mechanisms. Successful implementation of open evaluation platforms has the potential to substantially advance both the pace and the quality of scientific publication and evaluation, and the scientific community has a vested interest in shifting toward such models as soon as possible.

Establishing a new journal for systematic review products

Welcome to a new age in publishing systematic reviews. We hope the launch of Systematic Reviews will resonate with a broad spectrum of readers interested in using them in a variety of ways, such as providing comprehensive and up to date evidence for patient management, informing health policy, and developing rigorous practice guidelines. Systematic reviews are increasingly popular. Our journal is committed to publishing a wide variety of well conducted and transparently reported systematic reviews and associated research. We are open access and electronic and not confined by space and so offer scope for publishing reviews in detail and providing a modern and innovative approach to publishing. We look forward to participating in the voyage with all of our readers.

Editorial: Methodology in judgment and decision making research

In this introduction to the special issue on methodology, we provide background on its original motivation and a systematic overview of the contributions. The latter are discussed with correspondence to the phase of the scientific process they (most strongly) refer to: Theory construction, design, data analysis, and cumulative development of scientific knowledge. Several contributions propose novel measurement techniques and paradigms that will allow for new insights and can thus avail researchers in JDM and beyond. Another set of contributions centers around how models can best be tested and/or compared. Especially when viewed in combination, the papers on this topic spell out vital necessities for model comparisons and provide approaches that solve noteworthy problems prior work has been faced with.

Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network

Although current electronic methods of scientific publishing offer increased opportunities for publishing all research studies and describing them in sufficient detail, health research literature still suffers from many shortcomings. These shortcomings seriously undermine the value and utility of the literature and waste scarce resources invested in the research. In recent years there have been several positive steps aimed at improving this situation, such as a strengthening of journals' policies on research publication and the wide requirement to register clinical trials.The EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network is an international initiative set up to advance high quality reporting of health research studies; it promotes good reporting practices including the wider implementation of reporting guidelines. EQUATOR provides free online resources http://www.equator-network.org supported by education and training activities and assists in the development of robust reporting guidelines. This paper outlines EQUATOR's goals and activities and offers suggestions for organizations and individuals involved in health research on how to strengthen research reporting.