The quality of systematic reviews and other synthesis in the time of COVID-19

The quality of COVID-19 systematic reviews during the coronavirus 2019 pandemic: an exploratory comparison

BACKGROUND

The unprecedented volume and speed at which COVID-19-related systematic reviews (SRs) may have been produced has raised questions regarding the quality of this evidence. It is feasible that pandemic-related factors may have led to an impairment in quality (reduced internal validity, increased risk of bias [RoB]). This may have serious implications for decision-making related to public health and individual healthcare.

OBJECTIVE

The primary objective was to compare the quality of SRs published during the pandemic that were related to COVID-19 with SRs published during the pandemic that were unrelated to COVID-19 (all of which were fully appraised in the KSR Evidence database of SRs in healthcare). Our secondary objective was to compare the quality of SRs published during the pandemic (regardless of research topic), with SRs published pre-pandemic.

METHODS

We compared all SRs related to COVID-19 to all SRs unrelated to COVID-19 that (i) were published during the pandemic (between 1st March 2020 and September 14, 2022), (ii) were included in KSR Evidence, and (iii) had been appraised using the ROBIS tool. We then compared all SRs published during the pandemic (regardless of research topic) with a pre-pandemic sample of SRs.

RESULTS

For SRs published during the pandemic, we found there was no statistically significant difference in quality between those SRs tagged as being related to COVID-19 and those that were not [relative risk (RR) of low RoB for COVID-19 versus COVID-19-unrelated reviews: 0.94; 95% confidence interval (CI): 0.66 to 1.34]. Generally, COVID-19 SRs and COVID-19-unrelated SRs were both of low quality with only 10% of COVID-19 reviews and 11% of COVID-19-unrelated reviews rated as low RoB. However, SRs (regardless of topic) published during the pandemic were of lower quality than those published pre-pandemic (RR for low RoB for 'during pandemic' versus 'pre-pandemic': 0.30; 95% CI: 0.26 to 0.34) with 11% of pandemic and 36% of pre-pandemic SRs rated as low RoB.

CONCLUSION

These results suggest COVID-19 and COVID-19-unrelated SRs published during the pandemic are equally of low quality. SRs published during the pandemic were generally lower quality compared with SRs published pre-pandemic irrespective of COVID-19 focus. Moreover, SR quality in general is seriously lacking, and considerable efforts need to be made to substantially improve the quality and rigour of the SR process.

Are COVID-19 systematic reviews up to date and can we tell? A cross-sectional study

BACKGROUND

COVID-19 led to a rapid acceleration in the number of systematic reviews. Readers need to know how up to date evidence is when selecting reviews to inform decisions. This cross-sectional study aimed to evaluate how easily the currency of COVID-19 systematic reviews published early in the pandemic could be determined and how up to date these reviews were at the time of publication.

METHODS

We searched for systematic reviews and meta-analyses relevant to COVID-19 added to PubMed in July 2020 and January 2021, including any that were first published as preprints. We extracted data on the date of search, number of included studies, and date first published online. For the search date, we noted the format of the date and where in the review this was reported. A sample of non-COVID-19 systematic reviews from November 2020 served as a comparator.

RESULTS

We identified 246 systematic reviews on COVID-19. In the abstract of these reviews, just over half (57%) reported the search date (day/month/year or month/year) while 43% failed to report any date. When the full text was considered, the search date was missing from 6% of reviews. The median time from last search to publication online was 91 days (IQR 63-130). Time from search to publication was similar for the subset of 15 rapid or living reviews (92 days) but shorter for the 29 reviews published as preprints (37 days). The median number of studies or publications included per review was 23 (IQR 12-40). In the sample of 290 non-COVID SRs, around two-thirds (65%) reported the search date while a third (34%) did not include any date in the abstract. The median time from search to publication online was 253 days (IQR 153-381) and each review included a median of 12 studies (IQR 8-21).

CONCLUSIONS

Despite the context of the pandemic and the need to easily ascertain the currency of systematic reviews, reporting of the search date information for COVID-19 reviews was inadequate. Adherence to reporting guidelines would improve the transparency and usefulness of systematic reviews to users.

Characteristics of registered and published systematic reviews focusing on the prevention of COVID-19: a meta-research study

OBJECTIVE

We investigated characteristics of systematic reviews (SRs) assessing measures to prevent COVID-19 by (1) identifying SR registrations in Prospective Register of Systematic Reviews (PROSPERO), (2) identifying published SRs in COVID-19 Living Overview of the Evidence (L-OVE) and (3) estimating the proportion of PROSPERO registrations published as full SR between 8 and 16 months after registration.

STUDY DESIGN

This meta-research study is part of the German CEOsys project, www.covid-evidenz.de. We searched PROSPERO entries registered between 1 January 2020 and 31 August 2020, and we searched COVID-19 L-OVE for published SRs (search date: 5 May 2021) focusing on measures to prevent COVID-19 and SARS-CoV-2 transmission. The two samples were screened for eligibility and key characteristics were extracted and summarised.

RESULTS

Of 612 PROSPERO registrations, 47 focused on prevention and were included. The preventive measures included public health interventions (20), followed by personal protective equipment (10), vaccinations (9) and others (8). In total, 13 of 47 (28%) PROSPERO registrations had been published as full SR (as preprint only (6), as peer-reviewed article only (6), as preprint and peer-reviewed article (1)). Median time between PROSPERO registration and publication was 5 months for peer-reviewed SRs and 2 months for preprints.Of the 2182 entries identified in COVID-19 L-OVE, 51 published SRs focused on prevention and were included. Similar to the PROSPERO sample, most published SRs focused on public health interventions (21). The number of included primary studies ranged between 0 and 64 (median: 7). Nine published SRs did not include any studies because of a lack of primary studies.

CONCLUSION

Considering the urgent information needs of policymakers and the public, our findings reveal the high-speed publication of preprints and lack of primary studies in the beginning of the COVID-19 crisis. Further meta-research on COVID-19 SRs is important to improve research efficiency among researchers across the world.

PROSPERO REGISTRATION NUMBER

CRD42021240423.

The currency and completeness of specialized databases of COVID-19 publications

Objective

Several specialized collections of COVID-19 literature have been developed during the global health emergency. These include the WHO COVID-19 Global Literature Database, Cochrane COVID-19 Study Register, CAMARADES COVID-19 SOLES, Epistemonikos’ COVID-19 L-OVE, and LitCovid. Our objective was to evaluate the completeness of these collections and to measure the time from when COVID-19 articles are posted to when they appear in the collections.

Study Design and Setting

We tested each selected collection for the presence of 440 included studies from 25 COVID-19 systematic reviews. We sampled 112 journals and prospectively monitored their websites until a new COVID-19 article appeared. We then monitored for 2 weeks to see when the new articles appeared in each collection. PubMed served as a comparator.

Results

Every collection provided at least one record not found in PubMed. Four records (1%) were not in any of the sources studied. Collections contained between 83% and 93% of the primary studies with the WHO database being the most complete. By 2 weeks, between 60% and 78% of tracked articles had appeared.

Conclusion

Our findings support the use of the best performing COVID-19 collections by systematic reviews to replace paywalled databases.

Most published systematic reviews of remdesivir for COVID-19 were redundant and lacked currency

Objective

To investigate the completeness and currency of published systematic reviews of remdesivir for COVID-19 and to compare this with a living guidelines approach.

Study Design and Setting

In this cross-sectional study, we searched Europe PMC on May 20, 2021 for systematic reviews of remdesivir (including preprints, living review updates). Completeness and currency were based on the inclusion of four major randomized trials of remdesivir available at the time of publication of the review (including as preliminary results and preprints).

Results

We included 38 reviews (45 reports), equivalent to a new publication every 9 days. 23 (51%) reports were out of date at the time of publication. Eleven reviews that were current on publication had a median survival time of 10 days (range 4–57). A third of reviews cited other systematic reviews, but only four provided justifications for why another review was necessary. Eight (21%) of the reviews were registered in PROSPERO. The Australian COVID-19 Clinical Evidence Taskforce living guidelines were updated within 14 days for three of the remdesivir trials, and within 28 days for the fourth.

Conclusion

There was considerable duplication of systematic reviews of remdesivir, and half were already out of date at the time of publication.

A Comprehensive Meta-Analysis of COVID-19 in Latin America

BACKGROUND

Latin America has become the epicenter of the coronavirus disease 2019 (COVID-19) pandemic. We aim to perform a systematic comparative review of the clinical characteristics that are associated with this disease in Latin American countries.

METHODS

We conducted a systematic review of published articles, journal and/or epidemiological reports of confirmed COVID-19 cases in Latin America. Data were obtained either through publicly available information from Ministries of Health, published journal reports and/or unpublished datasets. We analyzed data from SARS-CoV-2 positive patients evaluated at healthcare centers and hospitals of 8 countries including Brazil, Peru, Mexico, Argentina, Colombia, Venezuela, Ecuador, and Bolivia, between March 1^st and July 30^th, 2020. These countries consist of a total population that exceeds 519 million. Demographics, comorbidities, and clinical symptoms were collected. Statistical descriptive analysis and correlation analyses of symptoms, comorbidities and mortality were performed.

RESULTS

A total of 728,282 COVID-19 patients were included in this study. Of these, 52.6% were female. The average age was 48.4 years. Peru had the oldest cohort with 56.8 years and highest rate of females (56.8%) while Chile had the youngest cohort (39 years old). Venezuela had the highest male prevalence (56.7%). Most common symptoms were cough with 60.1% (Bolivia had the highest rate 78%), fatigue/tiredness with 52.0%, sore throat with 50.3%, and fever with 44.2%. Bolivian patients had fever as the top symptom (83.3%). GI symptoms included diarrhea which was highest in Mexico with 22.9%. Hypertension was among the top (12.1%) comorbidities, followed by diabetes with 8.3% and obesity at 4.5%. In multivariate analyses, the leading and significant comorbidities were hypertension (r = 0.83, p = 0.02), diabetes (r = 0.91, p = 0.01), and obesity (r = 0.86, p = 0.03). Mortality was highest in Mexico (16.6%) and lowest in Venezuela (0.9%) among the analyzed cohorts.

CONCLUSION

Overall, COVID-19 patients in Latin America display cough, fatigue, and fever as main symptoms. Up to 53% of patients with COVID-19 have GI manifestations. Different clinical symptoms were associated with COVID-19 in Latin American countries. Metabolic syndrome components were the main comorbidities associated with poor outcome. Country-specific management and prevention plans are needed and can be established from this meta-analysis.