Open synthesis and the coronavirus pandemic in 2020

The transparency and reproducibility of systematic reviews in forensic science

Systematic reviews are indispensable tools for both reliably informing decision-makers about the state of the field and for identifying areas that need further study. Their value, however, depends on their transparency and reproducibility. Readers should be able to determine what was searched for and when, where the authors searched, and whether that search was predetermined or evolved based on what was found. In this article, we measured the transparency and reproducibility of systematic reviews in forensic science, a field where courts, policymakers, and legislators count on systematic reviews to make informed decisions. In a sample of 100 systematic reviews published between 2018 and 2021, we found that completeness of reporting varied markedly. For instance, 50 % of reviews claimed to follow a reporting guideline and such statements were only modestly related to compliance with that reporting guideline. As to specific reporting items, 82 % reported all of the databases searched, 22 % reported the review's full Boolean search logic, and just 7 % reported the review was registered. Among meta-analyses (n = 23), only one stated data was available and none stated the analytic code was available. After considering the results, we end with recommendations for improved regulation of reporting practices, especially among journals. Our results may serve as a useful benchmark as the field evolves.

COVID-19 scenarios for a long-term strategy under fundamental uncertainty

Early on, scientists have pointed out that coronavirus disease 2019 is most likely here to stay, although its course and development are uncertain. This requires a long-term strategy of living with the virus. However, the urgency of new waves of infection and the emergence of new variants have invoked an approach of acute crisis management over and over, hindering the design of a structural approach for the long term. Exploratory scenarios can provide scientific strategic guidance to policy processes to be better prepared in this situation of fundamental uncertainty. We have therefore developed five scenarios, which describe the possible long-term development of the pandemic from an epidemiological, virological, and broader societal perspective. These scenarios are based on four driving forces that are both important and uncertain: immunity, vaccination, mutations, and human behavior. The scenarios are (1) return to normal, (2) flu+, (3) external threat, (4) continuous struggle, and (5) worst case. Working with scenarios is crucial for appropriate public communication and provides guidance for anticipating the various conceivable possibilities for the further course of the pandemic.

Feasibility study for interactive reporting of network meta-analysis: experiences from the development of the MetaInsight COVID-19 app for stakeholder exploration, re-analysis and sensitivity analysis from living systematic reviews

BACKGROUND

Network meta-analysis (NMA) has been increasingly adopted worldwide by Cochrane reviews, guideline developers and decision-making bodies to identify optimal treatment choices. However, NMA results are often produced statically, not allowing stakeholders to 'dig deeper' and interrogate with their own judgement. Additionally, amid the COVID-19 pandemic, unnecessary or duplicated reviews have been proposed which analyse from the same pool of evidence. We developed the 'MetaInsight COVID-19' app as a prototype for an interactive platform to eliminate such duplicated efforts, by empowering users to freely analyse the data and improve scientific transparency.

METHODS

MetaInsight COVID-19 ( https://crsu.shinyapps.io/metainsightcovid/ ) was developed to conduct NMA with the evolving evidence on treatments for COVID-19. It was updated weekly between 19th May - 19th Oct 2020, incorporating new evidence identified from a living systematic review.

RESULTS

The app includes embedded functions to facilitate study selection based on study characteristics, and displays the synthesised results in real time. It allows both frequentist and Bayesian NMA to be conducted as well as consistency and heterogeneity assessments. A demonstration of the app is provided and experiences of building such a platform are discussed.

CONCLUSIONS

MetaInsight COVID-19 allows users to take control of the evidence synthesis using the analytic approach they deem appropriate to ascertain how robust findings are to alternative analysis strategies and study inclusion criteria. It is hoped that this app will help avoid many of the duplicated efforts when reviewing and synthesising the COVID-19 evidence, and, in addition, establish the desirability of an open platform format such as this for interactive data interrogation, visualisation, and reporting for any traditional or 'living' NMA.

Challenges of evidence synthesis during the 2020 COVID pandemic: a scoping review

AIM

The objectives of this scoping review are to identify the challenges to conducting evidence synthesis during the COVID-19 pandemic and to propose some recommendations addressing the identified gaps.

METHODS

A scoping review methodology was followed to map the literature published on the challenges and solutions of conducting evidence synthesis using the Joanna Briggs Methodology of performing scoping review. We searched several databases from the start of the Pandemic in December 2019 until 10th June 2021.

RESULTS

A total of 28 publications was included in the review. The challenges cited in the included studies have been categorised into four distinct but interconnected themes including: upstream, Evidence synthesis, downstream and contextual challenges. These challenges have been further refined into issues with primary studies, databases, team capacity, process, resources, and context. Several proposals to improve the above challenges included: transparency in primary studies registration and reporting, establishment of online platforms that enables collaboration, data sharing and searching, the use of computable evidence and coordination of efforts at an international level.

CONCLUSION

This review has highlighted the importance of including artificial intelligence, a framework for international collaboration and a sustained funding model to address many of the shortcomings and ensure we are ready for similar challenges in the future.

Methodological assessment of systematic reviews and meta-analyses on COVID-19: A meta-epidemiological study

RATIONALE, AIMS, AND OBJECTIVES

COVID-19 has caused an ongoing public health crisis. Many systematic reviews and meta-analyses have been performed to synthesize evidence for better understanding this new disease. However, some concerns have been raised about rapid COVID-19 research. This meta-epidemiological study aims to methodologically assess the current systematic reviews and meta-analyses on COVID-19.

METHODS

We searched in various databases for systematic reviews with meta-analyses published between 1 January 2020 and 31 October 2020. We extracted their basic characteristics, data analyses, evidence appraisal, and assessment of publication bias and heterogeneity.

RESULTS

We identified 295 systematic reviews on COVID-19. The median time from submission to acceptance was 33 days. Among these systematic reviews, 73.9% evaluated clinical manifestations or comorbidities of COVID-19. Stata was the most used software programme (43.39%). The odds ratio was the most used effect measure (34.24%). Moreover, 28.14% of the systematic reviews did not present evidence appraisal. Among those reporting the risk of bias results, 14.64% of studies had a high risk of bias. Egger's test was the most used method for assessing publication bias (38.31%), while 38.66% of the systematic reviews did not assess publication bias. The I2 statistic was widely used for assessing heterogeneity (92.20%); many meta-analyses had high values of I2 . Among the meta-analyses using the random-effects model, 75.82% did not report the methods for model implementation; among those meta-analyses reporting implementation methods, the DerSimonian-Laird method was the most used one.

CONCLUSIONS

The current systematic reviews and meta-analyses on COVID-19 might suffer from low transparency, high heterogeneity, and suboptimal statistical methods. It is recommended that future systematic reviews on COVID-19 strictly follow well-developed guidelines. Sensitivity analyses may be performed to examine how the synthesized evidence might depend on different methods for appraising evidence, assessing publication bias, and implementing meta-analysis models.