External validity of a randomised controlled trial on the treatment of severe infections caused by MRSA

Are we enrolling the right patients? A scoping review of external validity and generalizability of clinical trials in bloodstream infections

BACKGROUND

Having a representative population in randomized clinical trials (RCTs) improves external validity and generalizability of trial results. There are limited data examining differences between RCT-enrolled and real-world populations in bloodstream infections (BSI).

OBJECTIVES

We conducted a scoping review aiming to review studies assessing generalizability of BSI RCT populations, to identify sub-groups that have been systematically under-represented and to explore approaches to improve external validity of future RCTs.

SOURCES

MEDLINE, Embase, and Cochrane Library databases were searched for terms related to external validity or generalizability, BSI, and clinical trials in papers published up to 1 August 2023. Studies comparing enrolled versus nonenrolled patients, or papers discussing external validity or generalizability in the context of BSI RCTs were included.

CONTENT

Sixteen papers were included in the final review. Five compared RCT-enrolled and nonenrolled participants from the same source population. There were significant differences between the two groups in all studies, with nonenrolled patients having a greater comorbidity burden and consistently worse outcomes including mortality. We identified several barriers to improving generalizability of RCT populations and outlined potential approaches to reduce these barriers, such as alternative/simplified consent processes, streamlining eligibility criteria and follow-up procedures, quota-based sampling techniques, and ensuring diversity in site and study team selection.

IMPLICATIONS

Study cohorts in BSI RCTs are not representative of the general BSI patient population. As we increasingly adopt large pragmatic trials in infectious diseases, it is important to recognize the importance of maximizing generalizability to ensure that our research findings are of direct relevance to our patients.

A Pragmatic Randomized Feasibility Trial of Influenza Vaccines

BACKGROUND: The relative vaccine effectiveness (rVE) of high-dose quadrivalent influenza vaccines (QIV-HD) versus standard-dose quadrivalent influenza vaccines (QIV-SD) against hospitalizations and mortality in the general older population has not been evaluated in an individually randomized trial. Because of the large sample size required, such a trial will need to incorporate innovative, pragmatic elements. METHODS: We conducted a pragmatic, open-label, active-controlled, randomized feasibility trial in Danish citizens aged 65 to 79 years during the 2021–2022 influenza season. Participants were randomly assigned 1:1 to receive QIV-HD or QIV-SD. Randomization was integrated into routine vaccination practice, and the trial relied solely on nationwide administrative health registries for data collection. Outcomes consisted of a feasibility assessment and descriptive rVE estimates. RESULTS: We invited 34,000 persons to participate. A total of 12,477 randomly assigned participants were included in the final analyses. Mean (±SD) age was 71.7±3.9 years, and 5877 (47.1%) were women. Registry-based data collection was feasible, with complete follow-up data for 99.9% of participants. Baseline characteristics were comparable to those of the overall Danish population aged 65 to 79 years. The incidence of hospitalization for influenza or pneumonia was 10 (0.2%) of 6245 in the QIV-HD group and 28 (0.4%) of 6232 in the QIV-SD group (rVE, 64.4%; 95% confidence interval, 24.4 to 84.6). All-cause death occurred in 21 (0.3%) and 41 (0.7%) participants in the QIV-HD and QIV-SD groups, respectively (rVE, 48.9%; 95% confidence interval, 11.5 to 71.3). CONCLUSIONS: Conducting a pragmatic randomized trial of QIV-HD versus QIV-SD using existing infrastructure and registry-based data collection was feasible. The findings of lower incidence of hospitalization for influenza or pneumonia and all-cause mortality in the QIV-HD group compared with the QIV-SD group require replication in a future, fully powered trial. (Funded by Sanofi; ClinicalTrials.gov number, NCT05048589.)

How generalizable are randomized controlled trials (RCTs) in Staphylococcus aureus bacteremia? A description of the mortality gap between RCTs and observational studies

In Staphylococcus aureus bacteremia, mortality rates in randomized controlled trials (RCTs) are consistently lower than observational studies. Stringent eligibility criteria and omission of early deaths in RCTs contribute to this mortality gap. Clinicians should acknowledge the possibility of a lower treatment effect when applying RCT results to bedside care.

Systematic Review: A Comparison between Vancomycin and Daptomycin for Sepsis Infection Antibiotic Therapy

BACKGROUND: Sepsis is a dangerous condition that threatens life because of immune system dysregulation caused by an infection resulting in organ failure. One of the most common resistant strain bacteria that can cause sepsis is Methicillin-resistant Staphylococcus aureus (MRSA). Vancomycin is the first-line therapy for treating sepsis infection caused by MRSA, but recently there have been some MRSA strains that are resistant to vancomycin therapy. AIM: This study aimed to review comparison between vancomycin and daptomycin for sepsis infection antibiotics therapy. MATERIALS AND METHODS: This research was a systematic review using three databases such as PubMed, ProQuest, and ScienceDirect. The journal articles included in this study were about randomized controlled trial (RCT) studies published from 2011 to 2020. RESULTS: This research included seven RCT studies, but none of them discuss the usage of daptomycin for sepsis treatment caused by MRSA. They discuss more the effect of dose, method of administration, and side effects of vancomycin therapy in relation to the outcome of the patient. CONCLUSIONS: Because of the lack of RCT articles that conducted experiments of daptomycin usage for sepsis treatment caused by MRSA infection, this research could not compare the effectiveness between vancomycin and daptomycin. However, from some case reports included in this research, there was evidence that the usage of daptomycin base after vancomycin treatment failure will cause another treatment failure.

Inadequate reporting of participants eligible for randomized controlled trials - A systematic review and meta-analysis

OBJECTIVE

to characterize randomized controlled trials (RCTs) that did not report the overall number of participants assessed for eligibility and to identify factors associated with higher enrollment rates.

STUDY DESIGN AND SETTING

Systematic review and meta-analysis of RCTs in several pre-defined fields in internal medicine. We randomly extracted 360 articles that were published in 2017. Trials that reported numbers of assessed for eligibility patients were compared with those who did not. Recruitment rates were calculated in order to investigate whether they were associated with trial characteristics.

RESULTS

A total of 360 RCTs were included. Only 2-thirds of the trials (242/360) reported the number of patients assessed for eligibility. Trials reporting eligibility data had better methodology, reported on the tested hypothesis, included a placebo arm, evaluated soft outcomes, published their findings in higher impact journals and recruited a higher number of randomized patients than those who did not. Recruitment rates in 225 (62.5%) trials enabling their calculation, were significantly higher in trials sponsored by industry, conducted in multiple centers and countries, including inpatients, tested non-inferiority hypothesis, included a placebo arm, and evaluated surrogate outcomes.

CONCLUSIONS

Reporting of participant eligibility continues to be scarce. Inadequate reporting was associated with poor methodological characteristics in trials.

How representative is a point-of-care randomized trial? Clinical outcomes of patients excluded from a point-of-care randomized controlled trial evaluating antibiotic duration for Gram-negative bacteraemia: a multicentre prospective observational cohort study

OBJECTIVES

Patients included in randomized controlled trials (RCT) are poorly representative of the general population. We compared outcomes of patients excluded from the PIRATE trial, a point-of-care RCT evaluating antibiotic durations for Gram-negative bacteraemia, with those of enrolled patients.

METHODS

A prospective observational cohort study, 'EPCO' (Excluded Patients' Clinical Outcomes) included patients excluded from the PIRATE trial. As in PIRATE, whose patients were randomized to 7-day, 14-day, or C-reactive-protein (CRP)-guided antibiotic durations, EPCO's primary outcome was occurrence of clinical success at 30 days. We also compared baseline characteristics, outcome rates and treatment-effect estimates.

RESULTS

In all, 405 patients were included in EPCO and compared with the 503 PIRATE patients. Reasons for exclusion were mainly medical (317/405; 78%), the most frequent being complicated infection. Excluded patients had more co-morbidities (Charlson median 3 versus 1, p < 0.001). Bacteraemia was more often health-care-associated (26% versus 9%, p < 0.001). The 30-day success rate was significantly lower among EPCO patients (299/396; 76% versus 469/493; 95%, p < 0.001), but the success rate was not significantly different for those excluded for non-medical reasons (68/75; 91%, p 0.09). There was no significant difference in failure rates of EPCO patients according to their treatment duration (difference 7 days versus 14 days: p 0.75; 7 days versus CRP-correspondent: p 1.00; 14 days versus CRP-correspondent: p 1.00).

CONCLUSION

Although point-of-care-randomized trials are more inclusive and representative than traditional RCTs, they are still likely to select patients with lower failure risk. Shorter antibiotic durations were not associated with failure in either included or excluded patients, supporting the generalizability of the PIRATE trial's findings.

Assessing the external validity of the VALIDATE-SWEDEHEART trial

Aims:

The VALIDATE-SWEDEHEART trial was a registry-based randomized trial comparing bivalirudin and heparin in patients with acute myocardial infarction undergoing percutaneous coronary intervention. It showed no differences in mortality at 30 or 180 days. This study examines how well the trial population results may generalize to the population of all screened patients with fulfilled inclusion criteria in regard to mortality at 30 and 180 days.

Methods:

The standardized difference in the mean propensity score for trial inclusion between trial population and the screened not-enrolled with fulfilled inclusion criteria was calculated as a metric of similarity. Propensity scores were then used in an inverse-probability weighted Cox regression analysis using the trial population only to estimate the difference in mortality as it would have been had the trial included all screened patients with fulfilled inclusion criteria. Patients who were very likely to be included were weighted down and those who had a very low probability of being in the trial were weighted up.

Results:

The propensity score difference was 0.61. There were no significant differences in mortality between bivalirudin and heparin in the inverse-probability weighted analysis (hazard ratio 1.11, 95% confidence interval (0.73, 1.68)) at 30 days or 180 days (hazard ratio 0.98, 95% confidence interval (0.70, 1.36)).

Conclusion:

The propensity score difference demonstrated that the screened not-enrolled with fulfilled inclusion criteria and trial population were not similar. The inverse-probability weighted analysis showed no significant differences in mortality. From this, we conclude that the VALIDATE results may be generalized to the screened not-enrolled with fulfilled inclusion criteria.

Investigator-initiated Randomized Controlled Trials in Infectious Diseases: Better Value for Money for Registration Trials of New Antimicrobials

Randomized controlled trials (RCTs) conducted by the industry are expensive, especially trials conducted for registration of new drugs for multidrug-resistant (MDR) bacteria. Lower-cost investigator-initiated trials have recently been successful in recruiting patients with severe infections caused by MDR bacteria. In this viewpoint, we contrast the aims, methods, and resulting costs of industry-led and investigator-initiated trials and ask whether contemporary registration trial costs are justified. Contract research organizations, delivering and monitoring industry-sponsored trials at a significant cost, have little incentive to make trials more efficient or less expensive. The value of universal monitoring of all trial data is questionable. We propose that clinical trial networks play a more influential role in RCT design and planning, lead adaptive risk-based trial monitoring, and work with the industry to maximize efficient recruitment and lower costs in registration trials for the approval of new antimicrobials.

Excluded versus included patients in a randomized controlled trial of infections caused by carbapenem-resistant Gram-negative bacteria: relevance to external validity

Background

Population external validity is the extent to which an experimental study results can be generalized from a specific sample to a defined population. In order to apply the results of a study, we should be able to assess its population external validity. We performed an investigator-initiated randomized controlled trial (RCT) (AIDA study), which compared colistin-meropenem combination therapy to colistin monotherapy in the treatment of patients infected with carbapenem-resistant Gram-negative bacteria. In order to examine the study’s population external validity and to substantiate the use of AIDA study results in clinical practice, we performed a concomitant observational trial.

Methods

The study was conducted between October 1st, 2013 and January 31st, 2017 (during the RCTs recruitment period) in Greece, Israel and Italy. Patients included in the observational arm of the study have fulfilled clinical and microbiological inclusion criteria but were excluded from the RCT due to receipt of colistin for > 96 h, refusal to participate, or prior inclusion in the RCT. Non-randomized cases were compared to randomized patients. The primary outcome was clinical failure at 14 days of infection onset.

Results

Analysis included 701 patients. Patients were infected mainly with Acinetobacter baumannii [78.2% (548/701)]. The most common reason for exclusion was refusal to participate [62% (183/295)]. Non-randomized and randomized patients were similar in most of the demographic and background parameters, though randomized patients showed minor differences towards a more severe infection. Combination therapy was less common in non-randomized patients [31.9% (53/166) vs. 51.2% (208/406), p = 0.000]. Randomized patients received longer treatment of colistin [13 days (IQR 10–16) vs. 8.5 days (IQR 0–15), p = 0.000]. Univariate analysis showed that non-randomized patients were more inclined to clinical failure on day 14 from infection onset [82% (242/295) vs. 75.5% (307/406), p = 0.042]. After adjusting for other variables, non-inclusion was not an independent risk factor for clinical failure at day 14.

Conclusion

The similarity between the observational arm and RCT patients has strengthened our confidence in the population external validity of the AIDA trial. Adding an observational arm to intervention studies can help increase the population external validity and improve implementation of study results in clinical practice.

Trial registration

The trial was registered with ClinicalTrials.gov, number NCT01732250 on November 22, 2012.

Using external data to assess the external validity of a randomised controlled trial

BACKGROUND

Few studies have addressed external validity of randomized controlled trials in infectious diseases. We aimed to assess the external validity of an investigator-initiated trial on treatment for uncomplicated urinary tract infection.

METHODS

In the original study, women (n = 513) with urinary tract infection were randomized to nitrofurantoin or fosfomycin treatment in three countries between 2013 and 2017. In the present study we compared women who were screened for enrolment but excluded to women who participated in the trial, both groups in Israel. The primary outcome was the rate of emergency department index visits resulting in hospitalization within 28 days.

RESULTS

We compared 127 included to 110 excluded patients. The most common reasons for exclusion were logistic difficulties in recruitment and antibiotic use in the preceding month. Included patients tended to be older [39 (IQR 29-59) vs. 35.5 (IQR 24-56.25 years)], more likely to have history of recurrent infection and had more urinary symptoms. Among excluded patients, 13.6% (15/110) had initial visits resulting in hospitalization compared to 3.1% (4/127) of included participants (p = .003). The rate of emergency department visits within 28 days was similar in both groups. Clinical and microbiological failures were significantly more common in included patients [26% (33/127) vs. 1.8% (2/110), p < .001; 7.9% (10/127) vs. 0% (0/110), p = .003; respectively].

CONCLUSIONS

While differences were observed between included and excluded patients, the excluded group did not represent a more 'complicated' population. The present study shows the importance of collecting data on patients excluded from randomized controlled trials.

Efficacy and safety of ceftazidime/avibactam: a systematic review and meta-analysis

Background

Ceftazidime/avibactam is approved for complicated intra-abdominal and urinary tract infections (UTIs) based on results from randomized controlled trials (RCTs). Data regarding its effectiveness in treating hospital-acquired infections or resistant pathogens have not been systematically compiled.

Methods

A systematic review and meta-analysis including RCTs evaluating ceftazidime/avibactam versus comparator for the treatment of any infection. Primary outcome was 30 day all-cause mortality. Subgroups of hospital-acquired infections and specific resistance phenotypes were planned.

Results

Seven publications (eight trials, 4093 patients) were included, reporting a baseline ∼25% of ESBL-carrying Enterobacteriaceae. No significant difference between ceftazidime/avibactam and comparator (mostly carbapenem) was demonstrated for 30 day all-cause mortality, late follow-up mortality and clinical response [relative risk (RR) 1.10, 95% CI 0.70-1.72, P = 0.69; RR 1.23, 95% CI 0.87-1.76, P = 0.25; RR 0.98, 95% CI 0.96-1.01, P = 0.21, respectively, without significant heterogeneity]. Higher microbiological response rate was demonstrated with ceftazidime/avibactam in patients with UTI (RR 1.14, 1.0-1.29, P = 0.05, I2 = 51%). No significant difference in clinical response was demonstrated for patients with ceftazidime-resistant pathogens (RR 1.02, 95% CI 0.94-1.10, P = 0.66, I2 = 0%). Results for other subgroups of resistant pathogens or hospital-acquired infection were not available. Serious adverse events (SAEs) were significantly more common with ceftazidime/avibactam (RR 1.24, 95% CI 1.00-1.54, P = 0.05, I2 = 0%).

Conclusions

Ceftazidime/avibactam is clinically and microbiologically as effective as carbapenems for treatment of infections in a setting of ∼25% ESBL-carrying Enterobacteriaceae. Safety of the drug should be further evaluated owing to a higher rate of SAEs compared with carbapenems. Further studies should assess the drug's effectiveness in the treatment of carbapenemase-producing Enterobacteriaceae.

Comparable outcomes among trial and nontrial participants in a clinical trial of antibiotics for childhood pneumonia: a retrospective cohort study

OBJECTIVES

We compared characteristics and outcomes of children enrolled in a randomized controlled trial (RCT) comparing oral amoxicillin and benzyl penicillin for the treatment of chest indrawing pneumonia vs. children who received routine care to determine the external validity of the trial results.

STUDY DESIGN AND SETTING

A retrospective cohort study was conducted among children aged 2-59 months admitted in six Kenyan hospitals. Data for nontrial participants were extracted from inpatient records upon conclusion of the RCT. Mortality among trial vs. nontrial participants was compared in multivariate models.

RESULTS

A total of 1,709 children were included, of whom 527 were enrolled in the RCT and 1,182 received routine care. History of a wheeze was more common among trial participants (35.4% vs. 11.2%; P < 0.01), while dehydration was more common among nontrial participants (8.6% vs. 5.9%; P = 0.05). Other patient characteristics were balanced between the two groups. Among those with available outcome data, 14/1,140 (1.2%) nontrial participants died compared to 4/527 (0.8%) enrolled in the trial (adjusted odds ratio, 0.7; 95% confidence interval: 0.2-2.1).

CONCLUSION

Patient characteristics were similar, and mortality was low among trial and nontrial participants. These findings support the revised World Health Organization treatment recommendations for chest indrawing pneumonia.

PIRATE project: point-of-care, informatics-based randomised controlled trial for decreasing overuse of antibiotic therapy in Gram-negative bacteraemia

INTRODUCTION

Antibiotic overuse drives antibiotic resistance. The optimal duration of antibiotic therapy for Gram-negative bacteraemia (GNB), a common community and hospital-associated infection, remains unknown and unstudied via randomised controlled trials (RCTs).

METHODS AND ANALYSIS

This investigator-initiated, multicentre, non-inferiority, informatics-based point-of-care RCT will randomly assign adult hospitalised patients receiving microbiologically efficacious antibiotic(s) for GNB to (1) 14 days of antibiotic therapy, (2) 7 days of therapy or (3) an individualised duration determined by clinical response and 75% reduction in peak C reactive protein (CRP) values. The randomisation will occur in equal proportions (1:1:1) on day 5 (±1) of efficacious antibiotic therapy as determined by antibiogram; patients, their physicians and study investigators will be blind to treatment duration allocation until the day of antibiotic discontinuation. Immunosuppressed patients and those with GNB due to complicated infections (endocarditis, osteomyelitis, etc) and/or non-fermenting bacilli (Acinetobacter spp, Burkholderia spp, Pseudomonas spp) Brucella spp, Fusobacterium spp or polymicrobial growth with Gram-positive organisms will be ineligible. The primary outcome is incidence of clinical failure at day 30; secondary outcomes include clinical failure, all-cause mortality and incidence of Clostridiumdifficile infection in the 90-day study period. An interim safety analysis will be performed after the first 150 patients have been followed for ≤30 days. Given a chosen margin of 10%, the required sample size to determine non-inferiority is roughly 500 patients. Analyses will be performed on both intention-to-treat and per-protocol populations.

ETHICS AND DISSEMINATION

Ethics approval was obtained from the cantonal ethics committees of all three participating sites. Results of the main trial and each of the secondary endpoints will be submitted for publication in a peer-reviewed journal.

TRIAL REGISTRATION NUMBER

This trial is registered at www.clinicaltrials.gov (NCT03101072; pre-results).