Credibility of preprints: an interdisciplinary survey of researchers

Reimagining bioRxiv and preprint servers as platforms for academic learning

A popular preprint server, bioRxiv, is important as a tool for increased visibility for life science research. If used properly, however, bioRxiv can also be an important tool for training, as it may expose trainees (degree-seeking students undertaking research or internships directly related to their field of study) to the peer review process. Here, we offer a comprehensive guide to using bioRxiv as a training tool, as well as offer suggestions for improvements in bioRxiv, including confusion that may be caused by bioRxiv articles appearing on PubMed.

The use and acceptability of preprints in health and social care settings: A scoping review

BACKGROUND

Preprints are open and accessible scientific manuscript or report that is shared publicly, through a preprint server, before being submitted to a journal. The value and importance of preprints has grown since its contribution during the public health emergency of the COVID-19 pandemic. Funders and publishers are establishing their position on the use of preprints, in grant applications and publishing models. However, the evidence supporting the use and acceptability of preprints varies across funders, publishers, and researchers. The scoping review explored the current evidence on the use and acceptability of preprints in health and social care settings by publishers, funders, and the research community throughout the research lifecycle.

METHODS

A scoping review was undertaken with no study or language limits. The search strategy was limited to the last five years (2017-2022) to capture changes influenced by COVID-19 (e.g., accelerated use and role of preprints in research). The review included international literature, including grey literature, and two databases were searched: Scopus and Web of Science (24 August 2022).

RESULTS

379 titles and abstracts and 193 full text articles were assessed for eligibility. Ninety-eight articles met eligibility criteria and were included for full extraction. For barriers and challenges, 26 statements were grouped under four main themes (e.g., volume/growth of publications, quality assurance/trustworthiness, risks associated to credibility, and validation). For benefits and value, 34 statements were grouped under six themes (e.g., openness/transparency, increased visibility/credibility, open review process, open research, democratic process/systems, increased productivity/opportunities).

CONCLUSIONS

Preprints provide opportunities for rapid dissemination but there is a need for clear policies and guidance from journals, publishers, and funders. Cautionary measures are needed to maintain the quality and value of preprints, paying particular attention to how findings are translated to the public. More research is needed to address some of the uncertainties addressed in this review.

The experiences of COVID-19 preprint authors: a survey of researchers about publishing and receiving feedback on their work during the pandemic

The COVID-19 pandemic caused a rise in preprinting, triggered by the need for open and rapid dissemination of research outputs. We surveyed authors of COVID-19 preprints to learn about their experiences with preprinting their work and also with publishing their work in a peer-reviewed journal. Our research had the following objectives: 1. to learn about authors' experiences with preprinting, their motivations, and future intentions; 2. to consider preprints in terms of their effectiveness in enabling authors to receive feedback on their work; 3. to compare the impact of feedback on preprints with the impact of comments of editors and reviewers on papers submitted to journals. In our survey, 78% of the new adopters of preprinting reported the intention to also preprint their future work. The boost in preprinting may therefore have a structural effect that will last after the pandemic, although future developments will also depend on other factors, including the broader growth in the adoption of open science practices. A total of 53% of the respondents reported that they had received feedback on their preprints. However, more than half of the feedback was received through "closed" channels-privately to the authors. This means that preprinting was a useful way to receive feedback on research, but the value of feedback could be increased further by facilitating and promoting "open" channels for preprint feedback. Almost a quarter of the feedback received by respondents consisted of detailed comments, showing the potential of preprint feedback to provide valuable comments on research. Respondents also reported that, compared to preprint feedback, journal peer review was more likely to lead to major changes to their work, suggesting that journal peer review provides significant added value compared to feedback received on preprints.

Science in motion: A qualitative analysis of journalists' use and perception of preprints

This qualitative study explores how and why journalists use preprints-unreviewed research papers-in their reporting. Through thematic analysis of interviews conducted with 19 health and science journalists in the second year of the COVID-19 pandemic, it applies a theoretical framework that conceptualizes COVID-19 preprint research as a form of post-normal science, characterized by high scientific uncertainty and societal relevance, urgent need for political decision-making, and value-related policy considerations. Findings suggest that journalists approach the decision to cover preprints as a careful calculation, in which the potential public benefits and the ease of access preprints provided were weighed against risks of spreading misinformation. Journalists described viewing unreviewed studies with extra skepticism and relied on diverse strategies to find, vet, and report on them. Some of these strategies represent standard science journalism, while others, such as labeling unreviewed studies as preprints, mark a departure from the norm. However, journalists also reported barriers to covering preprints, as many felt they lacked the expertise or the time required to fully understand or vet the research. The findings suggest that coverage of preprints is likely to continue post-pandemic, with important implications for scientists, journalists, and the publics who read their work.

Ensuring an Accurate Scientific Record in an Era of Pre-print Servers

BACKGROUND

Pre-prints have become an increasing part of the biomedical landscape. For example, during the first month of operation, July 2019, medRxiv received 176 submissions, one year later, in June 2020, including the first few months of COVID-19, it received 1866 submissions. The current relevant question is how to ensure an accurate scientific record given that there may be important differences between a pre-print and the peer-reviewed publication.

METHODS

Based upon the experience of the authors, conversations with editors, and a focused selective review of the literature, including the recommendations of some professional groups, a limited number of practical recommendations were formulated.

RESULTS

Peer-reviewed journals should request that authors indicate if the submitted manuscript has been posted on a pre-print server; ensure this is noted in the article if it is published by including the digital object identifier (DOI); and detail any major differences in the conclusions between the pre-print and the article. Pre-print servers should ensure that all content is marked as not peer-reviewed and be prepared to retract any pre-print that is fundamentally flawed within days that could influence clinical or public health recommendations that have therapeutic implications.

CONCLUSION

Authors, those responsible for pre-print servers, and editors of peer- reviewed journals are responsible for ensuring an accurate scientific record.

Forecasting the publication and citation outcomes of COVID-19 preprints

Many publications on COVID-19 were released on preprint servers such as medRxiv and bioRxiv. It is unknown how reliable these preprints are, and which ones will eventually be published in scientific journals. In this study, we use crowdsourced human forecasts to predict publication outcomes and future citation counts for a sample of 400 preprints with high Altmetric score. Most of these preprints were published within 1 year of upload on a preprint server (70%), with a considerable fraction (45%) appearing in a high-impact journal with a journal impact factor of at least 10. On average, the preprints received 162 citations within the first year. We found that forecasters can predict if preprints will be published after 1 year and if the publishing journal has high impact. Forecasts are also informative with respect to Google Scholar citations within 1 year of upload on a preprint server. For both types of assessment, we found statistically significant positive correlations between forecasts and observed outcomes. While the forecasts can help to provide a preliminary assessment of preprints at a faster pace than traditional peer-review, it remains to be investigated if such an assessment is suited to identify methodological problems in preprints.

Caution, Preprint! Brief Explanations Allow Nonscientists to Differentiate Between Preprints and Peer-Reviewed Journal Articles

A growing number of psychological research findings are initially published as preprints. Preprints are not peer reviewed and thus did not undergo the established scientific quality-control process. Many researchers hence worry that these preprints reach nonscientists, such as practitioners, journalists, and policymakers, who might be unable to differentiate them from the peer-reviewed literature. Across five studies in Germany and the United States, we investigated whether this concern is warranted and whether this problem can be solved by providing nonscientists with a brief explanation of preprints and the peer-review process. Studies 1 and 2 showed that without an explanation, nonscientists perceive research findings published as preprints as equally credible as findings published as peer-reviewed articles. However, an explanation of the peer-review process reduces the credibility of preprints (Studies 3 and 4). In Study 5, we developed and tested a shortened version of this explanation, which we recommend adding to preprints. This explanation again allowed nonscientists to differentiate between preprints and the peer-reviewed literature. In sum, our research demonstrates that even a short explanation of the concept of preprints and their lack of peer review allows nonscientists who evaluate scientific findings to adjust their credibility perception accordingly. This would allow harvesting the benefits of preprints, such as faster and more accessible science communication, while reducing concerns about public overconfidence in the presented findings.

Impact of computational approaches in the fight against COVID-19: an AI guided review of 17 000 studies

SARS-CoV-2 caused the first severe pandemic of the digital era. Computational approaches have been ubiquitously used in an attempt to timely and effectively cope with the resulting global health crisis. In order to extensively assess such contribution, we collected, categorized and prioritized over 17 000 COVID-19-related research articles including both peer-reviewed and preprint publications that make a relevant use of computational approaches. Using machine learning methods, we identified six broad application areas i.e. Molecular Pharmacology and Biomarkers, Molecular Virology, Epidemiology, Healthcare, Clinical Medicine and Clinical Imaging. We then used our prioritization model as a guidance through an extensive, systematic review of the most relevant studies. We believe that the remarkable contribution provided by computational applications during the ongoing pandemic motivates additional efforts toward their further development and adoption, with the aim of enhancing preparedness and critical response for current and future emergencies.

Reporting all results efficiently: A RARE proposal to open up the file drawer

While the social sciences have made impressive progress in adopting transparent research practices that facilitate verification, replication, and reuse of materials, the problem of publication bias persists. Bias on the part of peer reviewers and journal editors, as well as the use of outdated research practices by authors, continues to skew literature toward statistically significant effects, many of which may be false positives. To mitigate this bias, we propose a framework to enable authors to report all results efficiently (RARE), with an initial focus on experimental and other prospective empirical social science research that utilizes public study registries. This framework depicts an integrated system that leverages the capacities of existing infrastructure in the form of public registries, institutional review boards, journals, and granting agencies, as well as investigators themselves, to efficiently incentivize full reporting and thereby, improve confidence in social science findings. In addition to increasing access to the results of scientific endeavors, a well-coordinated research ecosystem can prevent scholars from wasting time investigating the same questions in ways that have not worked in the past and reduce wasted funds on the part of granting agencies.

Reaping the benefits of open science in scholarly communication

Regardless of multiple efforts carried out across many countries to disseminate the ideas and the practice of open science, most scholars in the early 2020s do not self-archive their research articles and do not publish research papers in preprint form. Having received no education and training on open science, researchers are often puzzled on what to do, in practice, to start reaping the benefits of open science. This study offers a succinct vademecum on how to benefit from the open science approach to scholarly communication, no matter whether in natural or in humanistic and social sciences.

Adjusting the use of preprints to accommodate the 'quality' factor in response to COVID-19

Preprints are typically crude precursors of peer-reviewed papers that are placed almost immediately, save for some superficial screening, on an open-access repository to allow the information to reach readers quickly, circumventing the long-drawn process typically associated with processing in peer-reviewed journals. For early-career researchers who might be enthusiastic about obtaining some recognition for their efforts, or wanting open and public input about their work, preprints are certainly a useful publication choice. However, if health-related data and information have not been carefully scrutinised, they may pose a risk and may even serve as a source of public health misinformation. Surging growth and competition among preprint servers, coupled with a massive volume of COVID-19-related preprints, mainly on bioRxiv and medRxiv, as well as select indexing now being tested on PubMed, suggests that preprints are being increasingly used in the biomedical sciences. Stronger and more robust ethical policies are needed to screen preprints before they are released to the public, and even if this implies a slight delay in publication, it may increase academics' trust in this form of scientific information and communication. Clear and stringent ethical policies need to be urgently introduced by ethics groups such as COPE and the ICMJE, whose many member journals allow preprints to be posted before traditional peer review. Stringent ethical guidelines that treat misconduct equally in preprints and peer-reviewed papers will boost the integrity of academic publishing.

Engaging the next generation of editorial talent through a hands-on fellowship model

Peer-review and subject-matter editing is the backbone of scientific publishing. However, early-career researchers (ECRs) are given few opportunities to participate in the editorial process beyond reviewing articles. Thus, a disconnect exists: science needs high-quality editorial talent to conduct, oversee and improve the publishing process, yet we dedicate few resources to building editorial talent nor giving ECRs formal opportunities to influence publishing from within. ECRs can contribute to the publishing landscape in unique ways given their insight into new and rapidly developing publishing trends (e.g. open science). Here, we describe a two-way fellowship model that gives ECRs a "seat" at the editorial table of a field-leading journal. We describe both the necessary framework and benefits that can stem from editorial fellowships for ECRs, editors, journals, societies, and the ​broader scientific community.