Outpatient Treatment of Confirmed COVID-19: A Living, Rapid Review for the American College of Physicians

Sensitivity and Specificity of Using GPT-3.5 Turbo Models for Title and Abstract Screening in Systematic Reviews and Meta-analyses

BACKGROUND

Systematic reviews are performed manually despite the exponential growth of scientific literature.

OBJECTIVE

To investigate the sensitivity and specificity of GPT-3.5 Turbo, from OpenAI, as a single reviewer, for title and abstract screening in systematic reviews.

DESIGN

Diagnostic test accuracy study.

SETTING

Unannotated bibliographic databases from 5 systematic reviews representing 22 665 citations.

PARTICIPANTS

None.

MEASUREMENTS

A generic prompt framework to instruct GPT to perform title and abstract screening was designed. The output of the model was compared with decisions from authors under 2 rules. The first rule balanced sensitivity and specificity, for example, to act as a second reviewer. The second rule optimized sensitivity, for example, to reduce the number of citations to be manually screened.

RESULTS

Under the balanced rule, sensitivities ranged from 81.1% to 96.5% and specificities ranged from 25.8% to 80.4%. Across all reviews, GPT identified 7 of 708 citations (1%) missed by humans that should have been included after full-text screening at the cost of 10 279 of 22 665 false-positive recommendations (45.3%) that would require reconciliation during the screening process. Under the sensitive rule, sensitivities ranged from 94.6% to 99.8% and specificities ranged from 2.2% to 46.6%. Limiting manual screening to citations not ruled out by GPT could reduce the number of citations to screen from 127 of 6334 (2%) to 1851 of 4077 (45.4%), at the cost of missing from 0 to 1 of 26 citations (3.8%) at the full-text level.

LIMITATIONS

Time needed to fine-tune prompt. Retrospective nature of the study, convenient sample of 5 systematic reviews, and GPT performance sensitive to prompt development and time.

CONCLUSION

The GPT-3.5 Turbo model may be used as a second reviewer for title and abstract screening, at the cost of additional work to reconcile added false positives. It also showed potential to reduce the number of citations before screening by humans, at the cost of missing some citations at the full-text level.

PRIMARY FUNDING SOURCE

None.

Nirmatrelvir/ritonavir for COVID-19: an overview of systematic reviews

OBJECTIVES

A reappraisal of the validity of the conclusions of systematic reviews (SRs) related to nirmatrelvir/ritonavir for the treatment of COVID-19.

METHODS

An overview of SRs (umbrella review). The methodological quality of the SRs was assessed using the AMSTAR 2 checklist; quality of the evidence from the trials included in each SR was appraised following the GRADE approach.

RESULTS

Sixteen SRs with meta-analysis published between 2020 and 2023 were included in this overview. The SRs reported data from 108 overlapping reports, based on 43 individual primary studies [3 randomized clinical trials (RCTs), 40 non-RCTs]. In outpatient settings the use of nirmatrelvir/ritonavir reduced overall mortality, hospital admission and progression of disease compared with controls (from moderate to low certainty of evidence); nirmatrelvir/ritonavir reduced mortality, hospital admission and progression of disease in both immunized and non-immunized patients. No differences in the occurrence of any adverse events between groups were observed in the large majority of SRs; serious adverse events, including adverse events requiring discontinuation of treatment, were reported with lower prevalence in nirmatrelvir recipients compared with controls (from low to moderate certainty of evidence).

CONCLUSIONS

There is low to moderate certainty of evidence from SRs that nirmatrelvir/ritonavir reduces mortality, clinical progression and hospitalization rate in COVID-19 patients compared with controls, without increasing the occurrence of overall and serious adverse events. Based on the overall methodological assessment, on average we can have high confidence in the quality of results generated by the SRs.

Outpatient Treatment of Confirmed COVID-19: Living, Rapid Practice Points From the American College of Physicians (Version 2)

DESCRIPTION

Evidence for the use of outpatient treatments in adults with confirmed COVID-19 continues to evolve with new data. This is version 2 of the American College of Physicians (ACP) living, rapid practice points focusing on 22 outpatient treatments for COVID-19, specifically addressing the dominant SARS-CoV-2 Omicron variant.

METHODS

The Population Health and Medical Science Committee (formerly the Scientific Medical Policy Committee) developed this version of the living, rapid practice points on the basis of a living, rapid review done by the ACP Center for Evidence Reviews at Cochrane Austria at the University for Continuing Education Krems (Danube University Krems). This topic will be maintained as living and rapid by continually monitoring and assessing the impact of new evidence.

PRACTICE POINT 1

Consider molnupiravir to treat symptomatic patients with confirmed mild to moderate COVID-19 in the outpatient setting who are within 5 days of the onset of symptoms and at a high risk for progressing to severe disease.

PRACTICE POINT 2

Consider nirmatrelvir-ritonavir combination therapy to treat symptomatic patients with confirmed mild to moderate COVID-19 in the outpatient setting who are within 5 days of the onset of symptoms and at a high risk for progressing to severe disease.

PRACTICE POINT 3

Do not use ivermectin to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting.

PRACTICE POINT 4

Do not use sotrovimab to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting.

Outpatient Treatment of Confirmed COVID-19: A Living, Rapid Evidence Review for the American College of Physicians (Version 2)

BACKGROUND

Clinicians and patients want to know the benefits and harms of outpatient treatment options for the Omicron variant of SARS-CoV-2.

PURPOSE

To assess the benefits and harms of 22 different COVID-19 treatments.

DATA SOURCES

The Epistemonikos COVID-19 L·OVE platform, the iSearch COVID-19 portfolio, and the World Health Organization (WHO) COVID-19 Research Database from 26 November 2021 to 2 March 2023.

STUDY SELECTION

Two reviewers independently screened abstracts and full texts against a priori-defined criteria.

DATA EXTRACTION

One reviewer extracted the data and assessed the risk of bias and certainty of evidence (COE). A second reviewer verified the data abstraction and assessments.

DATA SYNTHESIS

Two randomized controlled trials and 6 retrospective cohort studies were included. Nirmatrelvir-ritonavir was associated with a reduction in hospitalization due to COVID-19 (for example, 0.7% vs. 1.2%; moderate COE) and all-cause mortality (for example, <0.1% vs. 0.2%; moderate COE). Molnupiravir led to a higher recovery rate (31.8% vs. 22.6%; moderate COE) and reduced time to recovery (9 vs. 15 median days; moderate COE) but had no effect on all-cause mortality (0.02% vs. 0.04%; moderate COE) and the incidence of serious adverse events (0.4% vs. 0.3%; moderate COE). Ivermectin had no effect on time to recovery (moderate COE) and resulted in no difference in adverse events compared with placebo (low COE). Sotrovimab resulted in no difference in all-cause mortality compared with no treatment (low COE). No eligible studies for all other treatments of interest were identified.

LIMITATION

Evidence for nirmatrelvir-ritonavir and sotrovimab is based on nonrandomized studies only.

CONCLUSION

Nirmatrelvir-ritonavir and molnupiravir probably improve outcomes for outpatients with mild to moderate COVID-19.

PRIMARY FUNDING SOURCE

American College of Physicians. (PROSPERO: CRD42023406456).

Molnupiravir for the treatment of non-severe COVID-19: a systematic review and meta-analysis of 14 randomized trials with 34 570 patients

BACKGROUND

Molnupiravir has been considered a promising candidate for COVID-19. Its efficacy and safety in non-severe COVID-19 patients and the differences between patients with different risk factors need further evaluation.

METHODS

We conducted a systematic review and meta-analysis of randomized controlled trials that allocated adult patients with non-severe COVID-19 to molnupiravir or a control. We used random-effects models, and conducted subgroup analyses and meta-regression for COVID-19 patients with high-risk factors. The GRADE approach was used to rate the certainty of evidence.

RESULTS

Fourteen trials with 34 570 patients were included. Moderate- to low-certainty evidence showed that molnupiravir was associated with a reduction in the risk of hospitalization (relative risk [RR] = 0.63, 95% CI: 0.47-0.85), risk of mechanical ventilation (RR = 0.37, 95% CI: 0.19-0.72) and time to symptom resolution (mean differences [MD] = -2.91 days, 95% CI: -3.66 to -2.16). However, no significant differences were found in adverse events, all-cause mortality, rate of and time to viral clearance, or duration of hospitalization. For the rate of viral clearance, subgroup effects were found between trials with low and high risk of bias (P = 0.001) and between trials with male or female majority (P < 0.001). For admission to hospital, subgroup effects were also found between trials with ≥50% and <50% of the participants being female (P = 0.04). Meta-regression showed a significant association between higher trial mean age and elevated risk of hospitalization (P = 0.011), and female majority and elevated risk of hospitalization (P = 0.011).

CONCLUSIONS

Molnupiravir was found to be effective in non-severe COVID-19, but the efficacy varied with age and sex.

Ongoing Need for Clinical Trials and Contemporary End Points for Outpatient COVID-19

The American College of Physicians presents recommendations for the outpatient treatment of COVID-19 based on Sommer and colleagues' systematic review. The editorialists commend the authors of the recommendations and review for trying to summarize the rapidly evolving literature into clear practice points and discuss the challenges of continually updating reviews and associated recommendations as new evidence emerges and relevant outcomes evolve.

Outpatient Treatment of Confirmed COVID-19: Living, Rapid Practice Points From the American College of Physicians (Version 1)

DESCRIPTION

Strategies to manage COVID-19 in the outpatient setting continue to evolve as new data emerge on SARS-CoV-2 variants and the availability of newer treatments. The Scientific Medical Policy Committee (SMPC) of the American College of Physicians (ACP) developed these living, rapid practice points to summarize the best available evidence on the treatment of adults with confirmed COVID-19 in an outpatient setting. These practice points do not evaluate COVID-19 treatments in the inpatient setting or adjunctive COVID-19 treatments in the outpatient setting.

METHODS

The SMPC developed these living, rapid practice points on the basis of a living, rapid review done by the ACP Center for Evidence Reviews at Cochrane Austria at the University for Continuing Education Krems (Danube University Krems). The SMPC will maintain these practice points as living by monitoring and assessing the impact of new evidence.

PRACTICE POINT 1

Consider molnupiravir to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting who are within 5 to 7 days of the onset of symptoms and at high risk for progressing to severe disease.

PRACTICE POINT 2

Consider nirmatrelvir-ritonavir combination therapy to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting who are within 5 days of the onset of symptoms and at high risk for progressing to severe disease.

PRACTICE POINT 3

Consider remdesivir to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting who are within 7 days of the onset of symptoms and at high risk for progressing to severe disease.

PRACTICE POINT 4

Do not use azithromycin to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting.

PRACTICE POINT 5

Do not use chloroquine or hydroxychloroquine to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting.

PRACTICE POINT 6

Do not use ivermectin to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting.

PRACTICE POINT 7

Do not use nitazoxanide to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting.

PRACTICE POINT 8

Do not use lopinavir-ritonavir combination therapy to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting.

PRACTICE POINT 9

Do not use casirivimab-imdevimab combination therapy to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting unless it is considered effective against a SARS-CoV-2 variant or subvariant locally in circulation.

PRACTICE POINT 10

Do not use regdanvimab to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting unless it is considered effective against a SARS-CoV-2 variant or subvariant locally in circulation.

PRACTICE POINT 11

Do not use sotrovimab to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting unless it is considered effective against a SARS-CoV-2 variant or subvariant locally in circulation.

PRACTICE POINT 12

Do not use convalescent plasma to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting.

PRACTICE POINT 13

Do not use ciclesonide to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting.

PRACTICE POINT 14

Do not use fluvoxamine to treat patients with confirmed mild to moderate COVID-19 in the outpatient setting.