Robustness of evidence reported in preprints during peer review

Framing COVID-19 Preprint Research as Uncertain: A Mixed-Method Study of Public Reactions

During the COVID-19 pandemic, journalists were encouraged to convey uncertainty surrounding preliminary scientific evidence, including mentioning when research is unpublished or unverified by peer review. To understand how public audiences interpret this information, we conducted a mixed method study with U.S. adults. Participants read a news article about preprint COVID-19 vaccine research in early April 2021, just as the vaccine was becoming widely available to the U.S. public. We modified the article to test two ways of conveying uncertainty (hedging of scientific claims and mention of preprint status) in a 2 × 2 between-participants factorial design. To complement this, we collected open-ended data to assess participants' understanding of the concept of a scientific preprint. In all, participants who read hedged (vs. unhedged) versions of the article reported less favorable vaccine attitudes and intentions and found the scientists and news reporting less trustworthy. These effects were moderated by participants' epistemic beliefs and their preference for information about scientific uncertainty. However, there was no impact of describing the study as a preprint, and participants' qualitative responses indicated a limited understanding of the concept. We discuss implications of these findings for communicating initial scientific evidence to the public and we outline important next steps for research and theory-building.

Making science public: a review of journalists' use of Open Access research

Science journalists are uniquely positioned to increase the societal impact of open research outputs by contextualizing and communicating findings in ways that highlight their relevance and implications for non-specialist audiences. Yet, it is unclear to what degree journalists use open research outputs, such as open access publications or preprints, in their reporting; what factors motivate or constrain this use; and how the recent surge in openly available research seen during the COVID-19 pandemic has affected this. This article examines these questions through a review of relevant literature published from 2018 onwards-particularly literature relating to the COVID-19 pandemic-as well as seminal articles outside the search dates. We find that research that explicitly examines journalists' engagement with open access publications or preprints is scarce, with existing literature mostly addressing the topic tangentially or as a secondary concern, rather than a primary focus. Still, the limited body of evidence points to several factors that may hamper journalists' use of these outputs and thus warrant further exploration. These include an overreliance on traditional criteria for evaluating scientific quality; concerns about the trustworthiness of open research outputs; and challenges using and verifying the findings. We also find that, while the COVID-19 pandemic encouraged journalists to explore open research outputs such as preprints, the extent to which these explorations will become established journalistic practices remains unclear. Furthermore, we note that current research is overwhelmingly authored and focused on the Global North, and the United States specifically. We conclude with recommendations for future research that attend to issues of equity and diversity, and more explicitly examine the intersections of open access and science journalism.

A systematic review and meta-analysis of the global prevalence and determinants of COVID-19 vaccine acceptance and uptake in people living with HIV

People living with HIV (PLHIV) are at higher risk of poor outcomes of SARS-CoV-2 infection. Here we report the pooled prevalence of COVID-19 vaccine acceptance/uptake and determinants among this vulnerable population of PLHIV based on a systematic review and meta-analysis of studies published by 25 August 2023. Among the 54 included studies (N = 167,485 participants), 53 (N = 166,455) provided data on vaccine acceptance rate, while 27 (N = 150,926) provided uptake data. The global prevalences of COVID-19 vaccine acceptance and uptake were 67.0% and 56.6%, respectively. Acceptance and uptake rates were 86.6% and 90.1% for the European Region, 74.9% and 71.6% for the Region of the Americas, 62.3% and 78.9% for the South-East Asian Region, 64.6% and 19.3% for the Eastern Mediterranean Region, 58.0% and 35.5% for the African Region, and 57.4% and 44.0% for the Western Pacific Region. The acceptance rate increased from 65.9% in 2020 to 71.0% in 2022, and the uptake rate increased from 55.9% in 2021 to 58.1% in 2022. Men, PLHIV aged ≥40 years and those who had recently received the influenza vaccine were more likely to accept and receive the COVID-19 vaccine. Factors associated with lower uptake included Black race, other races (Latinx/Hispanic/mixed race), low education level and being unemployed. Vaccine-related factors associated with higher acceptance included belief in vaccine effectiveness, vaccine trust, perceived high susceptibility to SARS-CoV-2 infection and fear of potential COVID-19 effect in PLHIV. Sustained efforts and targeted interventions are needed to reduce regional disparities in COVID-19 vaccine uptake among PLHIV.

A Systematic Review Of COVID-19 Misinformation Interventions: Lessons Learned

Governments, public health authorities, and social media platforms have employed various measures to counter misinformation that emerged during the COVID-19 pandemic. The effectiveness of those misinformation interventions is poorly understood. We analyzed fifty papers published between January 1, 2020, and February 24, 2023, to understand which interventions, if any, were helpful in mitigating COVID-19 misinformation. We found evidence supporting accuracy prompts, debunks, media literacy tips, warning labels, and overlays in mitigating either the spread of or belief in COVID-19 misinformation. However, by mapping the different characteristics of each study, we found levels of variation that weaken the current evidence base. For example, only 18 percent of studies included public health-related measures, such as intent to vaccinate, and the misinformation that interventions were tested against ranged considerably from conspiracy theories (vaccines include microchips) to unproven claims (gargling with saltwater prevents COVID-19). To more clearly discern the impact of various interventions and make evidence actionable for public health, the field urgently needs to include more public health experts in intervention design and to develop a health misinformation typology; agreed-upon outcome measures; and more global, more longitudinal, more video-based, and more platform-diverse studies.

Characterization and Reach of Orthopaedic Research Posted to Preprint Servers: Are We "Undercooking" Our Science?

BACKGROUND

Although biomedical preprint servers have grown rapidly over the past several years, the harm to patient health and safety remains a major concern among several scientific communities. Despite previous studies examining the role of preprints during the Coronavirus-19 pandemic, there is limited information characterizing their impact on scientific communication in orthopaedic surgery.

QUESTIONS/PURPOSES

(1) What are the characteristics (subspecialty, study design, geographic origin, and proportion of publications) of orthopaedic articles on three preprint servers? (2) What are the citation counts, abstract views, tweets, and Altmetric score per preprinted article and per corresponding publication?

METHODS

Three of the largest preprint servers (medRxiv, bioRxiv, and Research Square) with a focus on biomedical topics were queried for all preprinted articles published between July 26, 2014, and September 1, 2021, using the following search terms: "orthopaedic," "orthopedic," "bone," "cartilage," "ligament," "tendon," "fracture," "dislocation," "hand," "wrist," "elbow," "shoulder," "spine," "spinal," "hip," "knee," "ankle," and "foot." Full-text articles in English related to orthopaedic surgery were included, while nonclinical studies, animal studies, duplicate studies, editorials, abstracts from conferences, and commentaries were excluded. A total of 1471 unique preprints were included and further characterized in terms of the orthopaedic subspecialty, study design, date posted, and geographic factors. Citation counts, abstract views, tweets, and Altmetric scores were collected for each preprinted article and the corresponding publication of that preprint in an accepting journal. We ascertained whether a preprinted article was published by searching title keywords and the corresponding author in three peer-reviewed article databases (PubMed, Google Scholar, and Dimensions) and confirming that the study design and research question matched.

RESULTS

The number of orthopaedic preprints increased from four in 2017 to 838 in 2020. The most common orthopaedic subspecialties represented were spine, knee, and hip. From 2017 to 2020, the cumulative counts of preprinted article citations, abstract views, and Altmetric scores increased. A corresponding publication was identified in 52% (762 of 1471) of preprints. As would be expected, because preprinting is a form of redundant publication, published articles that are also preprinted saw greater abstract views, citations, and Altmetric scores on a per-article basis.

CONCLUSION

Although preprints remain an extremely small proportion of all orthopaedic research, our findings suggest that nonpeer-reviewed, preprinted orthopaedic articles are being increasingly disseminated. These preprinted articles have a smaller academic and public footprint than their published counterparts, but they still reach a substantial audience through infrequent and superficial online interactions, which are far from equivalent to the engagement facilitated by peer review. Furthermore, the sequence of preprint posting and journal submission, acceptance, and publication is unclear based on the information available on these preprint servers. Thus, it is difficult to determine whether the metrics of preprinted articles are attributable to preprinting, and studies such as the present analysis will tend to overestimate the apparent impact of preprinting. Despite the potential for preprint servers to function as a venue for thoughtful feedback on research ideas, the available metrics data for these preprinted articles do not demonstrate the meaningful engagement that is achieved by peer review in terms of the frequency or depth of audience feedback.

CLINICAL RELEVANCE

Our findings highlight the need for safeguards to regulate research dissemination through preprint media, which has never been shown to benefit patients and should not be considered as evidence by clinicians. Clinician-scientists and researchers have the most important responsibility of protecting patients from the harm of potentially inaccurate biomedical science and therefore must prioritize patient needs first by uncovering scientific truths through the evidence-based processes of peer review, not preprinting. We recommend all journals publishing clinical research adopt the same policy as Clinical Orthopaedics and Related Research® , The Bone & Joint Journal, The Journal of Bone and Joint Surgery, and the Journal of Orthopaedic Research , removing any papers posted to preprint servers from consideration.

Current concerns on journal article with preprint: Anesthesia and Pain Medicine perspectives

Preprints are preliminary research reports that have not yet been peer-reviewed. They have been widely adopted to promote the timely dissemination of research across many scientific fields. In August 1991, Paul Ginsparg launched an electronic bulletin board intended to serve a few hundred colleagues working in a subfield of theoretical high-energy physics, thus launching arXiv, the first and largest preprint platform. Additional preprint servers have since been implemented in different academic fields, such as BioRxiv (2013, Biology; www.biorxiv.org) and medRxiv (2019, Health Science; www.medrxiv.org). While preprint availability has made valuable research resources accessible to the general public, thus bridging the gap between academic and non-academic audiences, it has also facilitated the spread of unsupported conclusions through various media channels. Issues surrounding the preprint policies of a journal must be addressed, ultimately, by editors and include the acceptance of preprint manuscripts, allowing the citation of preprints, maintaining a double-blind peer review process, changes to the preprint's content and authors' list, scoop priorities, commenting on preprints, and preventing the influence of social media. Editors must be able to deal with these issues adequately, to maintain the scientific integrity of their journal. In this review, the history, current status, and strengths and weaknesses of preprints as well as ongoing concerns regarding journal articles with preprints are discussed. An optimal approach to preprints is suggested for editorial board members, authors, and researchers.

Robustness of evidence reported in preprints during peer review

Scientists have expressed concern that the risk of flawed decision making is increased through the use of preprint data that might change after undergoing peer review. This Health Policy paper assesses how COVID-19 evidence presented in preprints changes after review. We quantified attrition dynamics of more than 1000 epidemiological estimates first reported in 100 preprints matched to their subsequent peer-reviewed journal publication. Point estimate values changed an average of 6% during review; the correlation between estimate values before and after review was high (0·99) and there was no systematic trend. Expert peer-review scores of preprint quality were not related to eventual publication in a peer-reviewed journal. Uncertainty was reduced during peer review, with CIs reducing by 7% on average. These results support the use of preprints, a component of biomedical research literature, in decision making. These results can also help inform the use of preprints during the ongoing COVID-19 pandemic and future disease outbreaks.