Publication of noninferiority clinical trials: changes over a 20-year interval

Assessing efficacy in non-inferiority trials with non-adherence to interventions: Are intention-to-treat and per-protocol analyses fit for purpose?

BACKGROUND

Non-inferiority trials comparing different active drugs are often subject to treatment non-adherence. Intention-to-treat (ITT) and per-protocol (PP) analyses have been advocated in such studies but are not guaranteed to be unbiased in the presence of differential non-adherence.

METHODS

The REMoxTB trial evaluated two 4-month experimental regimens compared with a 6-month control regimen for newly diagnosed drug-susceptible TB. The primary endpoint was a composite unfavorable outcome of treatment failure or recurrence within 18 months post-randomization. We conducted a simulation study based on REMoxTB to assess the performance of statistical methods for handling non-adherence in non-inferiority trials, including: ITT and PP analyses, adjustment for observed adherence, multiple imputation (MI) of outcomes, inverse-probability-of-treatment weighting (IPTW), and a doubly-robust (DR) estimator.

RESULTS

When non-adherence differed between trial arms, ITT, and PP analyses often resulted in non-trivial bias in the estimated treatment effect, which consequently under- or over-inflated the type I error rate. Adjustment for observed adherence led to similar issues, whereas the MI, IPTW and DR approaches were able to correct bias under most non-adherence scenarios; they could not always eliminate bias entirely in the presence of unobserved confounding. The IPTW and DR methods were generally unbiased and maintained desired type I error rates and statistical power.

CONCLUSIONS

When non-adherence differs between trial arms, ITT and PP analyses can produce biased estimates of efficacy, potentially leading to the acceptance of inferior treatments or efficacious regimens being missed. IPTW and the DR estimator are relatively straightforward methods to supplement ITT and PP approaches.

The INCH-trial: a multicenter randomized controlled trial comparing short- and long-term outcomes of open and laparoscopic surgery for incisional hernia repair

BACKGROUND

Laparoscopic incisional hernia repair is increasingly performed worldwide and expected to be superior to conventional open repair regarding hospital stay and quality of life (QoL). The INCisional Hernia-Trial was designed to test this hypothesis.

METHODS

A multicenter parallel randomized controlled open-label trial with a superiority design was conducted in six hospitals in the Netherlands. Patients with primary or recurrent incisional hernias were randomized by computer-guided block-randomization to undergo either conventional open or laparoscopic repair. Primary endpoint was postoperative length of hospital stay in days. Secondary endpoints included QoL, complications, and recurrences. Patients were followed up for at least 5 years.

RESULTS

Hundred-and-two patients were recruited and randomized. In total, 88 patients underwent surgery and were included in the intention-to-treat analysis (44 in the open group, 44 in the laparoscopic group). Mean age was 59.5 years, gender division was equal, and BMI was 28.8 kg/m. The trial was concluded early for futility after an unplanned interim analysis, which showed that the hypothesis needed to be rejected. There was no difference in primary outcome: length of hospital stay was 3 (range 1-36) days in the open group and 3 (range 1-12) days in the laparoscopic group (p = 0.481). There were no significant between-group differences in QoL questionnaires on the short and long term. Satisfaction was impaired in the open group. Overall recurrence rate was 19%, of which 16% in the open and 23% in the laparoscopic group (p = 0.25) at a mean follow-up of 6.6 years.

CONCLUSIONS

In a randomized controlled trial, short- and long-term outcomes after laparoscopic incisional hernia repair were not superior to open surgery. The persisting high recurrence rates, reduced QoL, and suboptimal satisfaction warrant the need for patient's expectation management in the preoperative process and individualized surgical management.

TRIAL REGISTRATION

Netherlands Trial Register NTR2808.

Prospective randomized multicenter noninferiority clinical trial evaluating the use of TFN-advancedTM proximal femoral nailing system (TFNA) for the treatment of proximal femur fracture in a Chinese population

PURPOSE

To evaluate whether the 24-weeks postoperative fracture union rate for the investigational TFNA intramedullary nail was non-inferior compared to the control product PFNA-II.

METHODS

The study was a prospective, randomized, single-blind, noninferiority dual-arm study drawing from 9 trauma centers across China, between November 2018 and September 2020, with follow-up measurements at 24 weeks after internal fixation. The full analysis data set (FAS [Intent-to-Treat]) was analyzed and is summarized here. The primary outcome was fracture union rate, a composite score combining clinical and radiographic assessment. Secondary endpoints comprised (a) clinical outcomes including (1) SF-12, (2) Harris Hip, and (3) EQ-5D Scores, (b) radiographic incidence of complications such as loosening or cut-out requiring revision, (c) revision rates, (d) reoperation rates, and (e) adverse events, including 24-weeks revision and reoperation rates.

RESULTS

Both TFNA and PFNA-II group fracture healing rates were 100% at 24 weeks; TFNA was therefore shown to be non-inferior to PFNA-II. With baseline data matched in all parameters except age in both the TFNA and PFNA-II groups, comparisons of union rates, SF-12, Harris Hip, and EQ-5D Scores yielded p values > 0.05 indicating no significant difference between the two groups, further supporting the noninferiority of TFNA. In both groups, revision and re-operation rates were 0, and the incidences of serious adverse events were 19.4% and 17.4%, respectively.

CONCLUSION

In terms of fracture union rate at 24 weeks, the DePuy Synthes Trochanteric Fixation Nail Advanced (TFNA) was not inferior to the marketed Proximal Femoral Nail Antirotation (PFNA-II) device produced by the same manufacturer. Secondary and safety outcomes showed no significant differences between the two groups.

REGISTRATION

Registration was completed at ClinicalTrials.gov NCT03635320.

New statistical approach shows that hydroxy-methionine is noninferior to DL-Methionine in 35-day-old broiler chickens

The efficacy of a new molecule is assessed in the pharmaceutical industry through noninferiority tests to establish that it is not unacceptably less efficient than the reference. This method was proposed to compare DL-Methionine (DL-Met) as reference and DL-Hydroxy-Methionine (OH-Met) as alternative, in broiler chickens. The research hypothesized that OH-Met is inferior to DL-Met. Noninferiority margins were determined using 7 datasets comparing broiler growth response between a sulfur amino acid deficient and adequate diet from 0 to 35 d. The datasets were selected from the literature and internal records of the company. The noninferiority margins were then fixed as the largest loss of effect (inferiority) acceptable when OH-Met is compared to DL-Met. Three corn/soybean meal-based experimental treatments were offered to 4,200 chicks (35 replicates of 40 birds). Birds received from 0 to 35 d 1) a negative control diet deficient in Met and Cys; the negative control treatment supplemented on equimolar basis with 2) DL-Met or 3) OH-Met in amounts allowing to reach Aviagen Met+Cys recommendations. The three treatments were adequate in all other nutrients. Growth performance, which was analysed with one-way ANOVA, showed no significant difference between DL-Met and OH-Met. The supplemented treatments improved (P < 0.0001) the performance parameters compared to the negative control. The lower limits of the confidence intervals of the difference between means for the feed intake [-1.34; 1.41], body weight [-57.3; 9.8] and daily growth [-1.64; 0.28], did not exceed the noninferiority margins. This demonstrates that OH-Met was non-inferior to DL-Met.

Shoulder stretching versus shoulder muscle strength training for the prevention of baseball-related arm injuries: a randomized, active-controlled, open-label, non-inferiority study

Glenohumeral internal rotation deficit (GIRD) and weakness in prone external rotation are risk factors for shoulder and elbow injuries in high school baseball pitchers. While a shoulder-stretching prevention program to improve GIRD decreases the injury rate, the effects of external rotation strength remain unclear. This non-inferiority (NI) study investigates the hypothesis that external rotation strength training is not inferior to sleeper stretching for shoulder and elbow injury prevention in high school baseball pitchers. Participants were randomly allocated to the stretching (n = 62; active control group) and muscle-training (n = 51) groups. Specific exercises were performed each night. Elbow and shoulder injuries were monitored for 150 days. Kaplan-Meier survival curves were generated, and the hazard ratios (HRs) for injury occurrence were calculated using multivariate Cox regression. The log-rank test was used to compare the injury-free time. A one-sided NI test using a fixed NI margin was performed (significance level, P = 0.025). The injury rates were 22.6% (n = 14) in the stretching group and 9.8% (n = 5) in the muscle-training group. The muscle-training group had a lower injury rate (P < 0.001) and a lower risk of injury than the stretching group (HR = 0.489). Therefore, external rotation muscle strength training is not inferior to stretching for preventing baseball-related arm injuries.

Consequences of Inaccurate Assumptions in Coronary Stent Noninferiority Trials: A Systematic Review and Meta-analysis

IMPORTANCE

The outcome and interpretation of noninferiority trials depend on the magnitude of the noninferiority margin and whether a relative or absolute noninferiority margin is used and may be affected by imprecision in event rate estimation.

OBJECTIVE

To assess the consequence of imprecise event rate estimations on interpretation of peer-reviewed randomized clinical trials.

DATA SOURCES

PubMed/MEDLINE was searched for articles published between January 1, 2015, and April 30, 2021.

STUDY SELECTION

Noninferiority randomized clinical trials of coronary stents published in selected journals with clinical events as the primary end point.

DATA EXTRACTION AND SYNTHESIS

Two reviewers (M.S. and F.V.) independently extracted data on trial characteristics, noninferiority assumptions, primary end point clinical outcomes, and study conclusions. Overestimation or underestimation of the control event rate was evaluated by dividing the assumed control event rate by the observed control event rate. For noninferiority end points with absolute margins, the assumed corresponding relative margin was defined as the ratio of the absolute margin and the assumed event rate, and the observed corresponding relative margin as the ratio between the absolute margin and the observed event rate in the control arm. Noninferiority comparisons with absolute margins were reanalyzed using the assumed corresponding relative margin and the Farrington-Manning score test for relative risk.

MAIN OUTCOMES AND MEASURES

Overestimation or underestimation, assumed and observed corresponding relative margins, and relative reanalysis of the primary end points of trials with absolute margins.

RESULTS

A total of 106 989 patients from 58 trials were included. The event rate in the control arms was overestimated by a median (IQR) of 28% (2%-74%). Most noninferiority trials used absolute rather than relative margins (55 of 58 trials [94.8%]). Owing to overestimation, absolute noninferiority margins became more permissive than originally assumed (median [IQR] of observed relative noninferiority margin, 1.62 [1.50-1.80] vs assumed relative noninferiority margin, 1.47 [1.39-1.55]; P < .001). Among trial comparisons that met noninferiority with an absolute noninferiority margin, 17 of 50 trials (34.0%) would not have met noninferiority with a corresponding assumed relative noninferiority margin.

CONCLUSIONS AND RELEVANCE

In this systematic review and meta-analysis, assumed event rates were often overestimated in noninferiority coronary stent trials. Because most of these trials use absolute margins to define noninferiority, such overestimation results in excessively permissive relative noninferiority margins.

Statistical methods for non-adherence in non-inferiority trials: useful and used? A systematic review

BACKGROUND

In non-inferiority trials with non-adherence to interventions (or non-compliance), intention-to-treat and per-protocol analyses are often performed; however, non-random non-adherence generally biases these estimates of efficacy.

OBJECTIVE

To identify statistical methods that adjust for the impact of non-adherence and thus estimate the causal effects of experimental interventions in non-inferiority trials.

DESIGN

A systematic review was conducted by searching the Ovid MEDLINE database (31 December 2020) to identify (1) randomised trials with a primary analysis for non-inferiority that applied (or planned to apply) statistical methods to account for the impact of non-adherence to interventions, and (2) methodology papers that described such statistical methods and included a non-inferiority trial application.

OUTCOMES

The statistical methods identified, their impacts on non-inferiority conclusions, and their advantages/disadvantages.

RESULTS

A total of 24 papers were included (4 protocols, 13 results papers and 7 methodology papers) reporting relevant methods on 26 occasions. The most common were instrumental variable approaches (n=9), including observed adherence as a covariate within a regression model (n=3), and modelling adherence as a time-varying covariate in a time-to-event analysis (n=3). Other methods included rank preserving structural failure time models and inverse-probability-of-treatment weighting. The methods identified in protocols and results papers were more commonly specified as sensitivity analyses (n=13) than primary analyses (n=3). Twelve results papers included an alternative analysis of the same outcome; conclusions regarding non-inferiority were in agreement on six occasions and could not be compared on six occasions (different measures of effect or results not provided in full).

CONCLUSIONS

Available statistical methods which attempt to account for the impact of non-adherence to interventions were used infrequently. Therefore, firm inferences about their influence on non-inferiority conclusions could not be drawn. Since intention-to-treat and per-protocol analyses do not guarantee unbiased conclusions regarding non-inferiority, the methods identified should be considered for use in sensitivity analyses.

PROSPERO REGISTRATION NUMBER

CRD42020177458.

Characteristics of Randomized Clinical Trials in Surgery From 2008 to 2020: A Systematic Review

IMPORTANCE

Randomized clinical trials (RCTs) provide the highest level of evidence to evaluate 2 or more surgical interventions. Surgical RCTs, however, face unique challenges in design and implementation.

OBJECTIVE

To evaluate the design, conduct, and reporting of contemporary surgical RCTs.

EVIDENCE REVIEW

A literature search performed in the 2 journals with the highest impact factor in general medicine as well as 6 key surgical specialties was conducted to identify RCTs published between 2008 and 2020. All RCTs describing a surgical intervention in both experimental and control arms were included. The quality of included data was assessed by establishing an a priori protocol containing all the details to extract. Trial characteristics, fragility index, risk of bias (Cochrane Risk of Bias 2 Tool), pragmatism (Pragmatic Explanatory Continuum Indicator Summary 2 [PRECIS-2]), and reporting bias were assessed.

FINDINGS

A total of 388 trials were identified. Of them, 242 (62.4%) were registered; discrepancies with the published protocol were identified in 81 (33.5%). Most trials used superiority design (329 [84.8%]), and intention-to-treat as primary analysis (221 [56.9%]) and were designed to detect a large treatment effect (50.0%; interquartile range [IQR], 24.7%-63.3%). Only 123 trials (31.7%) used major clinical events as the primary outcome. Most trials (303 [78.1%]) did not control for surgeon experience; only 17 trials (4.4%) assessed the quality of the intervention. The median sample size was 122 patients (IQR, 70-245 patients). The median follow-up was 24 months (IQR, 12.0-32.0 months). Most trials (211 [54.4%]) had some concern of bias and 91 (23.5%) had high risk of bias. The mean (SD) PRECIS-2 score was 3.52 (0.65) and increased significantly over the study period. Most trials (212 [54.6%]) reported a neutral result; reporting bias was identified in 109 of 211 (51.7%). The median fragility index was 3.0 (IQR, 1.0-6.0). Multiplicity was detected in 175 trials (45.1%), and only 35 (20.0%) adjusted for multiple comparisons.

CONCLUSIONS AND RELEVANCE

In this systematic review, the size of contemporary surgical trials was small and the focus was on minor clinical events. Trial registration remained suboptimal and discrepancies with the published protocol and reporting bias were frequent. Few trials controlled for surgeon experience or assessed the quality of the intervention.

Testing for non-inferior mortality: a systematic review of non-inferiority margin sizes and trial characteristics

OBJECTIVE

To describe the size and variability of non-inferiority margins used in non-inferiority trials of medications with primary outcomes involving mortality, and to examine the association between trial characteristics and non-inferiority margin size.

DESIGN

Systematic review.

DATA SOURCES

Medline, Medline In Process, Medline Epub Ahead of Print and Embase Classic+Embase databases from January 1989 to December 2019.

ELIGIBILITY CRITERIA

Prospective non-inferiority randomised controlled trials comparing pharmacological therapies, with primary analyses for non-inferiority and primary outcomes involving mortality alone or as part of a composite outcome. Trials had to prespecify non-inferiority margins as absolute risk differences or relative to risks of outcome and provide a baseline risk of primary outcome in the control intervention.

RESULTS

3992 records were screened, 195 articles were selected for full text review and 111 articles were included for analyses. 82% of trials were conducted in thrombosis, infectious diseases or oncology. Mortality was the sole primary outcome in 23 (21%) trials, and part of a composite primary outcome in 88 (79%) trials. The overall median non-inferiority margin was an absolute risk difference of 9% (IQR 4.2%-10%). When non-inferiority margins were expressed relative to the baseline risk of primary outcome in control groups, the median relative non-inferiority margin was 1.5 (IQR 1.3-1.7). In multivariable regression analyses examining the association between trial characteristics (medical specialty, inclusion of paediatric patients, mortality as a sole or part of a composite primary outcome, presence of industry funding) and non-inferiority margin size, only medical specialty was significantly associated with non-inferiority margin size.

CONCLUSION

Absolute and relative non-inferiority margins used in published trials comparing medications are large, allowing conclusions of non-inferiority in the context of large differences in mortality. Accepting the potential for large increases in outcomes involving mortality while declaring non-inferiority is a challenging methodological issue in the conduct of non-inferiority trials.

Methodological and reporting quality of non-inferiority randomized controlled trials comparing antibiotic therapies: a systematic review

BACKGROUND

Antibiotic non-inferiority randomized controlled trials (RCTs) are used for approval of new antibiotics and making changes to antibiotic prescribing in clinical practice. We conducted a systematic review to assess the methodological and reporting quality of antibiotic non-inferiority RCTs.

METHODS

We searched MEDLINE, Embase, the Cochrane Database of Systematic Reviews and the FDA drug database from inception until Nov 22, 2019 for non-inferiority RCTs comparing different systemic antibiotic therapies. Comparisons between antibiotic types, doses, administration routes or durations were included. Methodological and reporting quality indicators were based on the CONSORT reporting guidelines. Two independent reviewers extracted the data.

RESULTS

The systematic review included 227 studies. Of these, 135 (59.5%) studies were supported by pharmaceutical industry. Only 83 (36.6%) studies provided a justification for the non-inferiority margin. Reporting of both intention-to-treat (ITT) and per-protocol (PP) analyses were done in 165 (72.7%) studies. The conclusion was misleading in 34 (15.0%) studies. The studies funded by pharmaceutical industry were less likely to be stopped early due to logistical reasons (3.0% vs. 19.1%, OR=0.13 95% CI 0.04-0.37) and to show inconclusive results (11.1% vs. 42.9%, OR=0.17 95% CI 0.08-0.33). The quality of studies decreased over time with respect to blinding, early stopping, reporting of ITT with PP analysis and having misleading conclusions.

CONCLUSIONS

There is room for improvement in the methodology and reporting of antibiotic non-inferiority trials. Quality can be improved across the entire spectrum from investigators, funding agencies, as well as during the peer-review process.

Protocol adherence rates in superiority and noninferiority randomized clinical trials published in high impact medical journals

BACKGROUND/AIMS

Noninferiority clinical trials are susceptible to false confirmation of noninferiority when the intention-to-treat principle is applied in the setting of incomplete trial protocol adherence. The risk increases as protocol adherence rates decrease. The objective of this study was to compare protocol adherence and hypothesis confirmation between superiority and noninferiority randomized clinical trials published in three high impact medical journals. We hypothesized that noninferiority trials have lower protocol adherence and greater hypothesis confirmation.

METHODS

We conducted an observational study using published clinical trial data. We searched PubMed for active control, two-arm parallel group randomized clinical trials published in JAMA: The Journal of the American Medical Association, The New England Journal of Medicine, and The Lancet between 2007 and 2017. The primary exposure was trial type, superiority versus noninferiority, as determined by the hypothesis testing framework of the primary trial outcome. The primary outcome was trial protocol adherence rate, defined as the number of randomized subjects receiving the allocated intervention as described by the trial protocol and followed to primary outcome ascertainment (numerator), over the total number of subjects randomized (denominator). Hypothesis confirmation was defined as affirmation of noninferiority or the alternative hypothesis for noninferiority and superiority trials, respectively.

RESULTS

Among 120 superiority and 120 noninferiority trials, median and interquartile protocol adherence rates were 91.5 [81.4-96.7] and 89.8 [83.6-95.2], respectively; P = 0.47. Hypothesis confirmation was observed in 107/120 (89.2%) of noninferiority and 64/120 (53.3%) of superiority trials, risk difference (95% confidence interval): 35.8 (25.3-46.3), P < 0.001.

CONCLUSION

Protocol adherence rates are similar between superiority and noninferiority trials published in three high impact medical journals. Despite this, we observed greater hypothesis confirmation among noninferiority trials. We speculate that publication bias, lenient noninferiority margins and other sources of bias may contribute to this finding. Further study is needed to identify the reasons for this observed difference.

Meta-analysis of noninferiority and equivalence trials: ignoring trial design leads to differing and possibly misleading conclusions

OBJECTIVE

The objective of the study is to examine the analytic approach of meta-analyses that include noninferiority or equivalence (NI/EQ) trials.

STUDY DESIGN AND SETTING

We used Scopus to identify meta-analyses including NI/EQ trials. We extracted data from the meta-analyses and their included randomized clinical trials (RCTs). We used the RCT's NI/EQ margins to reinterpret the results of the meta-analyses, assessed for risk of biases unique to NI/EQ trials, and evaluated the consistency of the meta-analysis interpretation when using NI/EQ margins.

RESULTS

We identified 38 unique meta-analyses including 515 RCTs, of which 125 (24.3%) were NI/EQ trials. Fourteen meta-analyses (36.8%) reported the study design of their included trials, but only one (2.6%) interpreted their pooled estimates using NI/EQ margins and none assessed for risks of bias unique to NI/EQ trials. Nearly all NI/EQ trials (n = 116, 92.8%) included in the meta-analyses reported NI/EQ margins. The meta-analyses of 30 outcomes were reinterpreted using the NI/EQ margins; reinterpretations conflicted with the conclusion of the meta-analyses in most cases (n = 20, 66.7%).

CONCLUSION

Most meta-analyses including NI/EQ trials ignore trial design and do not assess risks of bias unique to NI/EQ studies. Meta-analyses addressing questions previously explored as NI/EQ should conduct an NI/EQ meta-analysis or use clear language when performing standard (i.e., superiority) meta-analyses.

Current Issues in Conduct and Reporting of Noninferiority Randomized Controlled Trials in Surgical Management of Cancer Patients

BACKGROUND

Serious concerns regarding quality of conduct and reporting of noninferiority trials (NITs) have been raised. Systematic analysis of the quality of the surgical NITs is lacking. Assessing the quality of conduct, reporting, and interpretation of surgical NITs in cancer patients is critical given their potential clinical impact. We aim to assess the quality of conduct, reporting, and interpretation of NITs that investigate the effects of surgical management in cancer patients.

METHODS

A cross-sectional analysis of papers identified through a comprehensive literature database search was performed. Forty papers employing a phase III noninferiority (NI) randomized trial design to study effects of surgical methodology or sequencing of surgery in patients with solid cancers were included. Papers were assessed for type of analysis, justification of the noninferiority margin (NIM), consistency of type I error with confidence intervals (CIs), ability to achieve the predefined sample size, and interpretations regarding NI.

RESULTS

Only half of the papers used both intention-to-treat and per protocol analyses; 62.5% provided no or poor justification for the NIM; 42.5% showed inconsistency of the type I error rate with CIs; 52.5% were deemed poor or fair quality, and 60.0% did not achieve the predefined sample size. One-fifth of the papers provided interpretation of the NI hypothesis that was not in concordance with the CONSORT guidelines.

CONCLUSIONS

The quality of conduct, reporting, and interpretation of surgical NITs is suboptimal, requiring further improvements through adherence to guidelines and rigorous assessment at the stages of the study approval, funding, and the peer-review process.

Noninferiority Designed Cardiovascular Trials in Highest-Impact Journals

BACKGROUND

Noninferiority trials are increasingly being performed. However, little is known about their methodological quality. We sought to characterize noninferiority cardiovascular trials published in the highest-impact journals, features that may bias results toward noninferiority, features related to reporting of noninferiority trials, and the time trends.

METHODS

We identified cardiovascular noninferiority trials published in JAMA, Lancet, or New England Journal of Medicine from 1990 to 2016. Two independent reviewers extracted the data. Data elements included the noninferiority margin and the success of studies in achieving noninferiority. The proportion of trials showing major or minor features that may have affected the noninferiority inference was determined. Major factors included the lack of presenting the results in both intention-to-treat and per-protocol/as-treated cohorts, α>0.05, the new intervention not being compared with the best alternative, not justifying the noninferiority margin, and exclusion or loss of ≥10% of the cohort. Minor factors included suboptimal blinding, allocation concealment, and others.

RESULTS

From 2544 screened studies, we identified 111 noninferiority cardiovascular trials. Noninferiority margins varied widely: risk differences of 0.4% to 25%, hazard ratios of 1.05 to 2.85, odds ratios of 1.1 to 2.0, and relative risks of 1.1 to 1.8. Eighty-six trials claimed noninferiority, of which 20 showed superiority, whereas 23 (21.1%) did not show noninferiority, of which 8 also demonstrated inferiority. Only 7 (6.3%) trials were considered low risk for all the major and minor biasing factors. Among common major factors for bias, 41 (37%) did not confirm the findings in both intention-to-treat and per-protocol/as-treated cohorts and 4 (3.6%) reported discrepant results between intention-to-treat and per-protocol analyses. Forty-three (38.7%) did not justify the noninferiority margin. Overall, 27 (24.3%) underenrolled or had >10% exclusions. Sixty trials (54.0%) were open label. Allocation concealment was not maintained or unclear in 11 (9.9%). Publication of noninferiority trials increased over time (P<0.001). Fifty-two (46.8%) were published after 2010 and had a lower risk of methodological or reporting limitations for major (P=0.03) and minor factors (P=0.002).

CONCLUSIONS

Noninferiority trials in highest-impact journals commonly conclude noninferiority of the tested intervention, but vary markedly in the selected noninferiority margin, and frequently have limitations that may impact the inference related to noninferiority.

Exploring how non-inferiority and equivalence are assessed in paediatrics: a systematic review

OBJECTIVE

To review characteristics, methodology and reporting of non-inferiority and equivalence trials in the specific context of paediatrics.

DESIGN

PubMed and Cochrane databases were searched (up to September 2016) for non-inferiority/equivalence randomised controlled trials conducted in children published in high-impact-factor journals (>5.0 for general/specialist medical journals; >2.2 for paediatric journals).

RESULTS

We found that the statistical hypothesis was inconsistent with the objective in 12 (10%) of the 125 reports included. Non-inferiority (n=98) and equivalence trials (n=27) were mostly used to evaluate interventions with easier administration (45%, n=54/120) and/or better safety profile (34%, n=41/120). All the data needed for targeted sample size recalculation were available for 39 reports (31%). The margin-representing the largest difference between arms that would be clinically acceptable-was reported in 119 (95%), and 44/119 (37%) reported the method used for margin determination. The median sample size was 268 (IQR 125-531). Margins were wider in smaller trials (<125 randomised patients) than in larger trials (p=0.04/p<0.01 for binary/continuous outcomes, respectively). We did not agree with the authors' conclusions in 11% (11/103) of the reports that provided sufficient information.

CONCLUSIONS

There is still a need to improve the quality of methodology, reporting and interpretation of non-inferiority/equivalence trials in paediatrics. In particular, the margins were often not justified and the conclusion was often not supported by the design and/or the results. As researchers have to cope with small sample size and with lack of evidence, methods for non-inferiority/equivalence trials need to be used and/or developed in this vulnerable population.

Do non-inferiority trials of reduced intensity therapies show reduced effects? A descriptive analysis

OBJECTIVES

To identify non-inferiority trials within a cohort where the experimental therapy is the same as the active control comparator but at a reduced intensity and determine if these non-inferiority trials of reduced intensity therapies have less favourable results than other non-inferiority trials in the cohort. Such a finding would provide suggestive evidence of biocreep in these trials.

DESIGN

This metaresearch study used a cohort of non-inferiority trials published in the five highest impact general medical journals during a 5-year period. Data relating to the characteristics and results of the trials were abstracted.

PRIMARY OUTCOME MEASURES

Proportions of trials with a declaration of superiority, non-inferiority and point estimates favouring the experimental therapy and mean absolute risk differences for trials with outcomes expressed as a proportion.

RESULTS

Our search yielded 163 trials reporting 182 non-inferiority comparisons; 36 comparisons from 31 trials were between the same therapy at reduced and full intensity. Compared with trials not evaluating reduced intensity therapies, fewer comparisons of reduced intensity therapies demonstrated a favourable result (non-inferiority or superiority) (58.3%vs82.2%; P=0.002) and fewer demonstrated superiority (2.8%vs18.5%; P=0.019). Likewise, point estimates for reduced intensity therapies more often favoured active control than those for other trials (77.8%vs39.7%; P<0.001) as did mean absolute risk differences (+2.5% vs -0.7%; P=0.018).

CONCLUSIONS

Non-inferiority trials comparing a therapy at reduced intensity to the same therapy at full intensity showed reduced effects compared with other non-inferiority trials. This suggests these trials may have a high rate of type 1 errors and biocreep, with significant implications for the design and interpretation of future non-inferiority trials.

Empirical Consequences of Current Recommendations for the Design and Interpretation of Noninferiority Trials

BACKGROUND

Noninferiority trials are increasingly common, though they have less standardized designs and their interpretation is less familiar to clinicians than superiority trials.

OBJECTIVE

To empirically evaluate a cohort of noninferiority trials to determine 1) their interpretation as recommended by CONSORT, 2) choice of alpha threshold and its sidedness, and 3) differences between methods of analysis such as intention-to-treat and per-protocol.

DESIGN

We searched MEDLINE for parallel-group randomized controlled noninferiority trials published in the five highest-impact general medical journals between 2011 and 2016.

MAIN MEASURES

Data abstracted included trial design parameters, results, and interpretation of results based on CONSORT recommendations.

KEY RESULTS

One hundred sixty-three trials and 182 noninferiority comparisons were included in our analysis. Based on CONSORT-recommended interpretation, 79% of experimental therapies met criteria for noninferiority, 13% met criteria for superiority, 20% were declared inconclusive, and 2% met criteria for inferiority. However, for 12% of trials, the experimental therapy was statistically significantly worse than the active control, but CONSORT recommended an interpretation of inconclusive or noninferior. A two-sided alpha equivalent of greater than 0.05 was used in 34% of the trials, and in five of these trials, the use of a standard two-sided alpha of 0.05 led to changes in the interpretation of results that disfavored the experimental therapy. In four of the five comparisons where different methods of analysis (e.g., intention-to-treat and per-protocol) yielded different results, the intention-to-treat analysis was the more conservative. In 11% of trials, a secondary advantage of the new therapy was neither reported nor could it be inferred by reviewers.

CONCLUSIONS

In this cohort, the design and interpretation of noninferiority trials led to significant and systematic bias in favor of the experimental therapy. Clinicians should exercise caution when interpreting these trials. Future trials may be more reliable if design parameters are standardized.

Methods of defining the non-inferiority margin in randomized, double-blind controlled trials: a systematic review

BACKGROUND

There is no consensus on the preferred method for defining the non-inferiority margin in non-inferiority trials, and previous studies showed that the rationale for its choice is often not reported. This study investigated how the non-inferiority margin is defined in the published literature, and whether its reporting has changed over time.

METHODS

A systematic PubMed search was conducted for all published randomized, double-blind, non-inferiority trials from January 1, 1966, to February 6, 2015. The primary outcome was the number of margins that were defined by methods other than the historical evidence of the active comparator. This was evaluated for a time trend. We also assessed the under-reporting of the methods of defining the margin as a secondary outcome, and whether this changed over time. Both outcomes were analyzed using a Poisson log-linear model. Predictors for better reporting of the methods, and the use of the fixed-margin method (one of the historical evidence methods) were also analyzed using logistic regression.

RESULTS

Two hundred seventy-three articles were included, which account for 273 non-inferiority margins. There was no statistically significant difference in the number of margins that were defined by other methods compared to those defined based on the historical evidence (ratio 2.17, 95% CI 0.86 to 5.82, p = 0.11), and this did not change over time. The number of margins for which methods were unreported was similar to those with reported methods (ratio 1.35, 95% CI 0.76 to 2.43, p = 0.31), with no change over time. The method of defining the margin was less often reported in journals with low-impact factors compared to journals with high-impact factors (OR 0.20; 95% CI 0.10 to 0.37, p < 0.0001). The publication of the FDA draft guidance in 2010 was associated with increased reporting of the fixed-margin method (after versus before 2010) (OR 3.54; 95% CI 1.12 to 13.35, p = 0.04).

CONCLUSIONS

Non-inferiority margins are not commonly defined based on the historical evidence of the active comparator, and they are poorly reported. Authors, reviewers, and editors need to take notice of reporting this critical information to allow for better judgment of non-inferiority trials.

Non-inferiority trials: are they inferior? A systematic review of reporting in major medical journals

OBJECTIVE

To assess the adequacy of reporting of non-inferiority trials alongside the consistency and utility of current recommended analyses and guidelines.

DESIGN

Review of randomised clinical trials that used a non-inferiority design published between January 2010 and May 2015 in medical journals that had an impact factor >10 (JAMA Internal Medicine, Archives Internal Medicine, PLOS Medicine, Annals of Internal Medicine, BMJ, JAMA, Lancet and New England Journal of Medicine).

DATA SOURCES

Ovid (MEDLINE).

METHODS

We searched for non-inferiority trials and assessed the following: choice of non-inferiority margin and justification of margin; power and significance level for sample size; patient population used and how this was defined; any missing data methods used and assumptions declared and any sensitivity analyses used.

RESULTS

A total of 168 trial publications were included. Most trials concluded non-inferiority (132; 79%). The non-inferiority margin was reported for 98% (164), but less than half reported any justification for the margin (77; 46%). While most chose two different analyses (91; 54%) the most common being intention-to-treat (ITT) or modified ITT and per-protocol, a large number of articles only chose to conduct and report one analysis (65; 39%), most commonly the ITT analysis. There was lack of clarity or inconsistency between the type I error rate and corresponding CIs for 73 (43%) articles. Missing data were rarely considered with (99; 59%) not declaring whether imputation techniques were used.

CONCLUSIONS

Reporting and conduct of non-inferiority trials is inconsistent and does not follow the recommendations in available statistical guidelines, which are not wholly consistent themselves. Authors should clearly describe the methods used and provide clear descriptions of and justifications for their design and primary analysis. Failure to do this risks misleading conclusions being drawn, with consequent effects on clinical practice.

The choice of the noninferiority margin in clinical trials was driven by baseline risk, type of primary outcome, and benefits of new treatment

OBJECTIVES

To explore characteristics of clinical trials that influence the choice of the noninferiority margin (NIM) when planning the trial.

STUDY DESIGN AND SETTING

We conducted an experimental survey among corresponding authors of randomized controlled trials indexed in MEDLINE. We described two hypothetical studies and asked the respondents' opinion on the largest loss of effectiveness that is clinically negligible (or the smallest lost of effectiveness that is clinically important in the superiority scenario). We randomly manipulated four study attributes in each vignette, using a factorial design.

RESULTS

A total of 364 researchers participated. The values for NIMs were significantly lower than the differences to be detected in a superiority trial. The NIM was smaller when the primary outcome was mortality compared with treatment failure, when baseline risk in the control arm was lower, and when the advantage of the new treatment was a lower cost compared with having fewer side effects. In contrast, the population age group under study and the difficulty to recruit patients showed no effect on the choice of the NIM.

CONCLUSION

In our experimental study, the factors associated with lower NIMs were mortality as a primary outcome, low baseline risk, and a less costly new treatment.

Comparison of noninferiority margins reported in protocols and publications showed incomplete and inconsistent reporting

OBJECTIVES

To compare noninferiority margins defined in study protocols and trial registry records with margins reported in subsequent publications.

STUDY DESIGN AND SETTING

Comparison of protocols of noninferiority trials submitted 2001 to 2005 to ethics committees in Switzerland and The Netherlands with corresponding publications and registry records. We searched MEDLINE via PubMed, the Cochrane Controlled Trials Register (Cochrane Library issue 01/2012), and Google Scholar in September 2013 to identify published reports, and the International Clinical Trials Registry Platform of the World Health Organization in March 2013 to identify registry records. Two readers recorded the noninferiority margin and other data using a standardized data-abstraction form.

RESULTS

The margin was identical in study protocol and publication in 43 (80%) of 54 pairs of study protocols and articles. In the remaining pairs, reporting was inconsistent (five pairs, 9%), or the noninferiority margin was either not reported in the publication (five pairs, 9%) or not defined in the study protocol (one pair). The confidence interval or the exact P-value required to judge whether the result was compatible with noninferior, inferior, or superior efficacy was reported in 43 (80%) publications. Complete and consistent reporting of both noninferiority margin and confidence interval (or exact P-value) was present in 39 (72%) protocol-publication pairs. Twenty-nine trials (54%) were registered in trial registries, but only one registry record included the noninferiority margin.

CONCLUSION

The reporting of noninferiority margins was incomplete and inconsistent with study protocols in a substantial proportion of published trials, and margins were rarely reported in trial registries.

Choice of non-inferiority (NI) margins does not protect against degradation of treatment effects on an average--an observational study of registered and published NI trials

OBJECTIVE

NI margins have to be chosen appropriately to control the risk of degradation of treatment effects in non-inferiority (NI) trials. We aimed to study whether the current choice of NI margins protects sufficiently against a degradation of treatment effect on an average.

STUDY DESIGN AND SETTING

NI trials reflecting current practice were assembled and for each trial, the NI margin was translated into a likelihood of degradation. The likelihood of degradation was calculated as the conditional probability of a treatment being harmful given that it is declared non-inferior in the trial, using simulation. Its distribution among the NI trials was then studied to assess the potential risk of degradation.

RESULTS

The median (lower/upper quartile) NI margin among 112 binary outcome NI trials corresponded to an odds ratio of 0.57(0.45, 0.66), while among 38 NI trials with continuous outcome, to a Cohen's d of -0.42(-0.54, -0.31) and a hazard ratio of 0.82(0.73, 0.86) among 24 survival outcome NI trials. Overall, the median likelihood of degradation was 56% (45%, 62%).

CONCLUSION

Only two fifths of the current NI trials had a likelihood of degradation lower than 50%, suggesting that, in majority of the NI trials, there is no sufficient protection against degradation on an average. We suggest a third hurdle for the choice of NI margins, thus contributing a sufficient degree of protection.

Systematic review on the primary and secondary reporting of the prevalence of ghostwriting in the medical literature

BACKGROUND

Ghostwriting of industry-sponsored articles is unethical and is perceived to be common practice.

OBJECTIVE

To systematically review how evidence for the prevalence of ghostwriting is reported in the medical literature.

DATA SOURCES

MEDLINE via PubMed 1966+, EMBASE 1966+, The Cochrane Library 1988+, Medical Writing 1998+, The American Medical Writers Association (AMWA) Journal 1986+, Council of Science Editors Annual Meetings 2007+, and the Peer Review Congress 1994+ were searched electronically (23 May 2013) using the search terms ghostwrit*, ghostauthor*, ghost AND writ*, ghost AND author*.

ELIGIBILITY CRITERIA

All publication types were considered; only publications reporting a numerical estimate of possible ghostwriting prevalence were included.

DATA EXTRACTION

Two independent reviewers screened the publications; discrepancies were resolved by consensus. Data to be collected included a numerical estimate of the prevalence of possible ghostwriting (primary outcome measure), definitions of ghostwriting reported, source of the reported prevalence, publication type and year, study design and sample population.

RESULTS

Of the 848 publications retrieved and screened for eligibility, 48 reported numerical estimates for the prevalence of possible ghostwriting. Sixteen primary publications reported findings from cross-sectional surveys or descriptive analyses of published articles; 32 secondary publications cited published or unpublished evidence. Estimates on the prevalence of possible ghostwriting in primary and secondary publications varied markedly. Primary estimates were not suitable for meta-analysis because of the various definitions of ghostwriting used, study designs and types of populations or samples. Secondary estimates were not always reported or cited correctly or appropriately.

CONCLUSIONS

Evidence for the prevalence of ghostwriting in the medical literature is limited and can be outdated, misleading or mistaken. Researchers should not inflate estimates using non-standard definitions of ghostwriting nor conflate ghostwriting with other unethical authorship practices. Editors and peer reviewers should not accept articles that incorrectly cite or interpret primary publications that report the prevalence of ghostwriting.

About half of the noninferiority trials tested superior treatments: a trial-register based study

OBJECTIVES

A concern that noninferiority (NI) trials pose a risk of degradation of the treatment effects is prevalent. Thus, we aimed to determine the fraction of positive true effects (superiority rate) and the average true effect of current NI trials based on data from registered NI trials.

STUDY DESIGN AND SETTING

All NI trials carried out between 2000 and 2007 analyzing the NI of efficacy as the primary objective and registered in one of the two major clinical trials registers were studied. Having retrieved results from these trials, random effects modeling of the effect estimates was performed to determine the distribution of true effects.

RESULTS

Effect estimates were available for 79 of 99 eligible trials identified. For trials with binary outcome, we estimated a superiority rate of 49% (95% confidence interval = 27-70%) and a mean true log odds ratio of -0.005 (-0.112, 0.102). For trials with continuous outcome, the superiority rate was 58% (41-74%) and the mean true effect as Cohen's d of 0.06 (-0.064, 0.192).

CONCLUSIONS

The unanticipated finding of a positive average true effect and superiority of the new treatment in most NI trials suggest that the current practice of choosing NI designs in clinical trials makes degradation on average unlikely. However, the distribution of true treatment effects demonstrates that, in some NI trials, the new treatment is distinctly inferior.