Consequences of Inaccurate Assumptions in Coronary Stent Noninferiority Trials: A Systematic Review and Meta-analysis

Assessment of Noninferiority Margins in Cardiovascular Medicine Trials

Background

Noninferiority trials are increasingly common in cardiovascular medicine, but their reporting and interpretation are challenging, particularly when an absolute risk difference is used as noninferiority margin.

Objectives

This study aimed to investigate the effect of using absolute rather than relative noninferiority margins in cardiovascular trials.

Methods

We reviewed noninferiority trials presented at major cardiovascular conferences from 2015 to 2022 and published within the same period. Based on the actual versus anticipated event rates in the control group, we recalculated the absolute noninferiority margin and re-assessed the trial results. The primary outcome of interest was the proportion of trials with a different interpretation after recalculation. Additionally, we analyzed the conclusion statements of these trials to determine if cautionary notes for the interpretation of study results were included.

Results

We analyzed a total of 768 trials, of which 88 had a noninferiority design and 66 used an absolute noninferiority margin. Of 48 comparisons from 45 trials qualifying for the analysis, 11 (22.9%) had divergent results after recalculation of the absolute noninferiority margin based on the observed rather than anticipated event rate. Ten trials originally claiming noninferiority, did not meet it after the margin recalculation. All of them did not include statements suggesting cautionary interpretation of the study results in the conclusion section. Compared with the other trials, these displayed a larger median difference between anticipated and recalculated noninferiority margins (44.7% [IQR: 38.6%-56.7%] vs 15.3% [IQR: -1.5% to 28.9%]; P < 0.001).

Conclusions

Recalculating noninferiority margins based on actual event rates, rather than anticipated ones, led to different outcomes in approximately 1 out of 4 cardiovascular trials, with most divergent trials lacking cautionary interpretation. These findings emphasize the importance of using or supplementing the relative noninferiority margin, particularly in studies with significant deviations between observed and expected event rates. This underscores the critical need for enhanced methodological and reporting standards in noninferiority trials, especially those employing absolute margins.

Accuracy of Event Rate and Effect Size Estimation in Major Cardiovascular Trials: A Systematic Review

Importance

For the design of a randomized clinical trial (RCT), estimation of the expected event rate and effect size of an intervention is needed to calculate the sample size. Overestimation may lead to an underpowered trial.

Objective

To evaluate the accuracy of published estimates of event rate and effect size in contemporary cardiovascular RCTs.

Evidence Review

A systematic search was conducted in MEDLINE for multicenter cardiovascular RCTs associated with MeSH (Medical Subject Headings) terms for cardiovascular diseases published in the New England Journal of Medicine, JAMA, or the Lancet between January 1, 2010, and December 31, 2019. Identified trials underwent abstract review; eligible trials then underwent full review, and those with insufficiently reported data were excluded. Data were extracted from the original publication or the study protocol, and a random-effects model was used for data pooling. This review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting guideline. The primary outcome was the accuracy of event rate and effect size estimation. Accuracy was determined by comparing the observed event rate in the control group and the effect size with their hypothesized values. Linear regression was used to determine the association between estimation accuracy and trial characteristics.

Findings

Of the 873 RCTs identified, 374 underwent full review and 30 were subsequently excluded, resulting in 344 trials for analysis. The median observed event rate was 9.0% (IQR, 4.3% to 21.4%), which was significantly lower than the estimated event rate of 11.0% (IQR, 6.0% to 25.0%) with a median deviation of -12.3% (95% CI, -16.4% to -5.6%; P < .001). More than half of the trials (196 [61.1%]) overestimated the expected event rate. Accuracy of event rate estimation was associated with a higher likelihood of refuting the null hypothesis (0.13 [95% CI, 0.01 to 0.25]; P = .03). The median observed effect size in superiority trials was 0.91 (IQR, 0.74 to 0.99), which was significantly lower than the estimated effect size of 0.72 (IQR, 0.60 to 0.80), indicating a median overestimation of 23.1% (95% CI, 17.9% to 28.3%). A total of 216 trials (82.1%) overestimated the effect size.

Conclusions and Relevance

In this systematic review of contemporary cardiovascular RCTs, event rates of the primary end point and effect sizes of an intervention were frequently overestimated. This overestimation may have contributed to the inability to adequately test the trial hypothesis.

Ensuring Superior Reporting of Radiation Therapy Noninferiority Trials: A Systematic Review

Purpose

Although the frequency of noninferiority trials is increasing, the consistency of the reporting of these trials can vary. The aim of this systematic review was to assess the reporting quality of radiation therapy noninferiority trials.

Methods and Materials

The PubMed, Embase, and Cochrane databases were queried for randomized controlled radiation therapy trials with noninferiority hypotheses published in English between January 2000 and July 2022, and this was performed by an information scientist. Descriptive statistics were used to summarize data.

Results

Of 423 records screened, 59 (14%) were included after full-text review. All were published after 2003 and open label. The most common primary cancer type was breast (n = 15, 25%). Altered radiation fractionation (n = 26, 45%) and radiation de-escalation (n = 11, 19%) were the most common types of interventions. The most common primary endpoints were locoregional control (n = 17, 29%) and progression-free survival (n = 14, 24%). Fifty-three (90%) reported the noninferiority margin, and only 9 (17%) provided statistical justification for the margin. The median absolute noninferiority margin was 9% (interquartile range, 5%-10%), and the median relative margin was 1.51 (interquartile range, 1.33-2.04). Sample size calculations and confidence intervals were reported in 54 studies (92%). Both intention-to-treat and per-protocol analyses were reported in 27 studies (46%). In 31 trials (53%), noninferiority of the primary endpoint was reached.

Conclusions

There was variability in the reporting of key components of noninferiority trials. We encourage consideration of additional statistical reasoning such as guidelines or previous trials in the selection of the noninferiority margin, reporting both absolute and relative margins, and the avoidance of statistically vague or misleading language in the reporting of future noninferiority trials.