Test for qualitative interaction of clinical significance

Statistical inference on qualitative differences in the magnitude of an effect

Qualitative interactions occur when a treatment effect or measure of association varies in sign by sub-population. Of particular interest in many biomedical settings are absence/presence qualitative interactions, which occur when an effect is present in one sub-population but absent in another. Absence/presence interactions arise in emerging applications in precision medicine, where the objective is to identify a set of predictive biomarkers that have prognostic value for clinical outcomes in some sub-population but not others. They also arise naturally in gene regulatory network inference, where the goal is to identify differences in networks corresponding to diseased and healthy individuals, or to different subtypes of disease; such differences lead to identification of network-based biomarkers for diseases. In this paper, we argue that while the absence/presence hypothesis is important, developing a statistical test for this hypothesis is an intractable problem. To overcome this challenge, we approximate the problem in a novel inference framework. In particular, we propose to make inferences about absence/presence interactions by quantifying the relative difference in effect size, reasoning that when the relative difference is large, an absence/presence interaction occurs. The proposed methodology is illustrated through a simulation study as well as an analysis of breast cancer data from the Cancer Genome Atlas.

A tutorial on Bayesian model-averaged meta-analysis in JASP

Researchers conduct meta-analyses in order to synthesize information across different studies. Compared to standard meta-analytic methods, Bayesian model-averaged meta-analysis offers several practical advantages including the ability to quantify evidence in favor of the absence of an effect, the ability to monitor evidence as individual studies accumulate indefinitely, and the ability to draw inferences based on multiple models simultaneously. This tutorial introduces the concepts and logic underlying Bayesian model-averaged meta-analysis and illustrates its application using the open-source software JASP. As a running example, we perform a Bayesian meta-analysis on language development in children. We show how to conduct a Bayesian model-averaged meta-analysis and how to interpret the results.

Association between blood pressure and endovascular treatment outcomes differs by baseline perfusion and reperfusion status

We hypothesized that the association between BP and endovascular treatment (EVT) outcomes would differ by baseline perfusion and recanalization status. We identified 388 ICA or M1 occlusion patients who underwent EVT ≤ 24 h from onset with successful recanalization (TICI ≥ 2b). BP was measured at 5-min intervals from arrival and during the procedure. Systolic BPs (SBP) were summarized as dropmax (the maximal decrease over two consecutive measurements), incmax (the maximal increase), mean, coefficient of variation (cv), and standard deviation. Adequate baseline perfusion was defined as hypoperfusion intensity ratio (HIR) ≤ 0.5; infarct proportion as the volume ratio of final infarcts within the Tmax > 6 s region. In the adequate perfusion group, infarct proportion was closely associated with SBPdropmax (β ± SE (P-value); 1.22 ± 0.48, (< 0.01)), SBPincmax (1.12 ± 0.33, (< 0.01)), SBPcv (0.61 ± 0.15 (< 0.01)), SBPsd (0.66 ± 0.08 (< 0.01)), and SBPmean (0.71 ± 0.37 (0.053) before recanalization. The associations remained significant only in SBPdropmax, SBPincmax, and SBPmean after recanalization. SBPincmax, SBPcv and SBPsd showed significant associations with modified Rankin Scale score at 3 months in the pre-recanalization period. In the poor perfusion group, none of the SBP indices was associated with any stroke outcomes regardless of recanalization status. BP may show differential associations with stroke outcomes by the recanalization and baseline perfusion status.

Effectiveness and safety of EVT in patients with acute LVO and low NIHSS

Background and purpose

There is much uncertainty in endovascular treatment (EVT) decisions in patients with acute large vessel occlusion (LVO) and mild neurological deficits.

Methods

From a prospective, nationwide stroke registry, all patients with LVO and baseline NIHSS &lt;6 presenting within 24 h from the time last known well (LKW) were included. Early neurological deterioration (END) developed before EVT was prospectively collected as an increasing total NIHSS score ≥2 or any worsening of the NIHSS consciousness or motor subscores during hospitalization not related to EVT. Significant hemorrhage was defined as PH2 hemorrhagic transformation or hemorrhage at a remote site. The modified Rankin Scale (mRS) was prospectively collected at 3 months.

Results

Among 1,083 patients, 149 (14%) patients received EVT after a median of 5.9 [3.6–12.3] h after LKW. In propensity score-matched analyses, EVT was not associated with mRS 0-1 (matched OR 0.99 [0.63–1.54]) but increased the risk of a significant hemorrhage (matched OR, 4.51 [1.59–12.80]). Extraneous END occurred in 207 (19%) patients after a median of 24.5 h [IQR, 13.5–41.9 h] after LKW (incidence rate, 1.41 [95% CI, 1.23–1.62] per 100 person-hours). END unrelated to EVT showed a tendency to modify the effectiveness of EVT (P-for-interaction, 0.08), which decreased the odds of having mRS 0–1 in mild LVO patients without END (adjusted OR, 0.63 [0.40–0.99]).

Conclusions

The use of EVT in patients with acute LVO and low NIHSS scores may require the assessment of individual risks of early deterioration, hemorrhagic complications and expected benefit.

Differential Impact of Coronary Revascularization on Long-Term Clinical Outcome According to Coronary Flow Characteristics: Analysis of the International ILIAS Registry

BACKGROUND

Coronary pressure indices such as fractional flow reserve are the standard for guiding elective revascularization. However, considering additional coronary flow parameters could further individualize and optimize the decision on revascularization. We aimed to investigate the potentially differential prognostic associations of elective percutaneous coronary intervention (PCI) according to coronary flow properties represented by coronary flow reserve (CFR), coronary flow capacity (CFC), and baseline CFC (bCFC).

METHODS

From the ILIAS Registry (Inclusive Invasive Physiological Assessment in Angina Syndromes) composed of 16 hospitals globally from 7 countries, patients with obstructive coronary artery disease who underwent invasive coronary physiological assessment were included (N=2370 vessels). We assessed effect measure modifications of the association of PCI and 5-year target vessel failure according to CFR, CFC, and bCFC either assessed by Doppler-technique or thermodilution-method.

RESULTS

The mean age of the population was 63.3 years, and there were 1322 (73.6%) males. Median fractional flow reserve was 0.85, and PCI was performed in 600 (25.3%) vessels. Reduced CFR, CFC, and abnormal bCFC were defined in 988 (41.7%), 542 (22.9%), and 600 (25.3%) vessels, respectively. Significant effect measure modifications were observed by CFC either in odds ratio (P=0.0018), additive (P=0.029), and hazard ratio scale (P=0.0002). The absolute risk of 5-year target-vessel failure was higher if treated by PCI in vessels with normal CFC by 1.8 (-1.7 to 5.3) percent, while that was lower by -5.9 (-12 to -0.1) percent in those with reduced CFC. CFR and bCFC were not significant effect modifiers in any scales. Similar associations were observed in per-patient analyses, whereas the findings were less robust.

CONCLUSIONS

We observed qualitative effect measure modification of PCI and 5-year clinical outcomes according to CFC status in additive scale. CFR and bCFC were not robust effect modifiers. Therefore, CFC could be potentially used to optimize the patient selection for elective PCI treatment combined with fractional flow reserve.

Clinical outcomes of Fractional Flow Reserve-Guided Percutaneous Coronary Intervention By Coronary Flow Capacity Status in Stable Lesions

BACKGROUND

Coronary flow capacity (CFC) provides integrated information about coronary flow reserve (CFR) and hyperaemic coronary flow and is useful for identifying coronary flow limitation.

AIMS

The aim of this study was to investigate the effect of percutaneous coronary intervention (PCI) on vessel-related major adverse cardiovascular events (MACE) according to CFC status in stable coronary lesions.

METHODS

From a global, multicentre registry of comprehensive physiological assessment, a total of 1,397 patients (1,694 vessels) were analysed. Low CFC was defined for lesions with reduced CFR and inverse of hyperaemic mean transit time (1/hTmn). A predefined definition of CFC (CFR <2.0 and 1/hTmn less than the corresponding percentile) was assessed first in a multivariable marginal Cox proportional model with the interaction term between CFC status and PCI (performed or not), and then the optimal definition of CFC was explored.

RESULTS

We observed a significant interaction between predefined low CFC and PCI (p=0.067). With the optimal definition of CFC (CFR ≤3.2 and 1/hTmn ≤2.8), the HR (95% CI) of PCI was 0.278 (0.103-0.751) and 1.393 (0.783-2.478) in lesions with low and normal CFC, respectively. If lesions with fractional flow reserve (FFR) ≤0.8 and normal CFC had been deferred, the number of PCI would have decreased by 64%.

CONCLUSIONS

FFR-guided PCI for low CFC lesions was associated with reduced incidence of MACE in low CFC but not in normal CFC lesions. Our results suggest the potential use of CFC in combination with FFR for optimising the indication for PCI by reducing potentially unbeneficial PCI.

CLINICAL TRIALS REGISTRATION

https://clinicaltrials.gov/ct2/show/NCT03690713.

Evaluating the Effectiveness of Personalized Medicine With Software

We present methodological advances in understanding the effectiveness of personalized medicine models and supply easy-to-use open-source software. Personalized medicine involves the systematic use of individual patient characteristics to determine which treatment option is most likely to result in a better average outcome for the patient. Why is personalized medicine not done more in practice? One of many reasons is because practitioners do not have any easy way to holistically evaluate whether their personalization procedure does better than the standard of care, termed improvement. Our software, "Personalized Treatment Evaluator" (the R package PTE), provides inference for improvement out-of-sample in many clinical scenarios. We also extend current methodology by allowing evaluation of improvement in the case where the endpoint is binary or survival. In the software, the practitioner inputs 1) data from a single-stage randomized trial with one continuous, incidence or survival endpoint and 2) an educated guess of a functional form of a model for the endpoint constructed from domain knowledge. The bootstrap is then employed on data unseen during model fitting to provide confidence intervals for the improvement for the average future patient (assuming future patients are similar to the patients in the trial). One may also test against a null scenario where the hypothesized personalization are not more useful than a standard of care. We demonstrate our method's promise on simulated data as well as on data from a randomized comparative trial investigating two treatments for depression.

The Interaction Continuum

A common reason given for assessing interaction is to evaluate “whether the effect is larger in one group versus another”. It has long been known that the answer to this question is scale dependent: the “effect” may be larger for one subgroup on the difference scale, but smaller on the ratio scale. In this article, we show that if the relative magnitude of effects across subgroups is of interest then there exists an “interaction continuum” that characterizes the nature of these relations. When both main effects are positive then the placement on the continuum depends on the relative magnitude of the probability of the outcome in the doubly exposed group. For high probabilities of the outcome in the doubly exposed group, the interaction may be positive-multiplicative positive-additive, the strongest form of positive interaction on the “interaction continuum”. As the probability of the outcome in the doubly exposed group goes down, the form of interaction descends through ranks, of what we will refer to as the following: positive-multiplicative positive-additive, no-multiplicative positive-additive, negative-multiplicative positive-additive, negative-multiplicative zero-additive, negative-multiplicative negative-additive, single pure interaction, single qualitative interaction, single-qualitative single-pure interaction, double qualitative interaction, perfect antagonism, inverted interaction. One can thus place a particular set of outcome probabilities into one of these eleven states on the interaction continuum. Analogous results are also given when both exposures are protective, or when one is protective and one causative. The “interaction continuum” can allow for inquiries as to relative effects sizes, while also acknowledging the scale dependence of the notion of interaction itself.

Prognostic Value of Prerevascularization Fractional Flow Reserve Mediated by the Postrevascularization Level

IMPORTANCE

The prognostic value of pre-percutaneous coronary intervention (PCI) fractional flow reserve (FFR) may be associated with the post-PCI FFR and their interaction. To correctly interpret the prognostic value of pre-PCI FFR, it is essential to understand to what extent the association of pre-PCI FFR with clinical outcomes is explained by post-PCI FFR.

OBJECTIVE

To investigate the extent to which post-PCI FFR mediates the association of pre-PCI FFR with vessel-related outcomes using an international, multicenter collaboration registry.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study used pooled patient data from 4 international FFR registries. A total of 1488 patients with pre-PCI FFR of 0.80 or less who underwent elective PCI were included. Data collection was conducted from November 2011 to August 2019, and analysis was conducted from September 2019 to July 2020.

MAIN OUTCOMES AND MEASURES

The primary outcome was target vessel failure (TVF) during 2 years of follow-up. The extent to which post-PCI FFR of less than 0.90 mediated the association of pre-PCI FFR less than 0.75 (vs pre-PCI FFR of 0.75 or greater) with TVF was evaluated using a mediation analysis in a counterfactual framework.

RESULTS

Among 1488 patients, the mean (SD) age was 63.5 (9.9) years and 1161 patients (78.0%) were men. The median (interquartile range) pre-PCI and post-PCI FFR were 0.71 (0.62-0.76) and 0.88 (0.83-0.92), respectively. The direct association of low pre-PCI FFR (ie, <0.75) with TVF was significant (odds ratio, 1.81; 95% CI, 1.03-3.17; P = .04), while the mediation by post-PCI FFR level of less than 0.90 was not (indirect association: odds ratio, 1.03; 95% CI, 0.98-1.09; P = .24). In sensitivity analyses using several pre-PCI cutoffs, the mediations by post-PCI FFR were consistently weak.

CONCLUSIONS AND RELEVANCE

In this study, the association of pre-PCI FFR with TVF was not significantly mediated by post-PCI FFR. Poor prognosis due to progressed atherosclerosis, represented as low FFR, may not be reversed by successful PCI that increases FFR. Therefore, the prognostic value of pre-PCI FFR may mainly reflect the global atherosclerotic burden, not the extent of the modifiable epicardial stenosis.

Defining heterogeneity of epicardial functional stenosis with low coronary flow reserve by unsupervised machine learning

Low CFR is associated with poor prognosis, whereas it is a heterogeneous condition according to the actual coronary flow, such as high resting or low hyperemic coronary flow, which should have different physiological traits and clinical implications. This study aimed to detect and define the sub-phenotypes of vessels with low coronary flow reserve (CFR) epicardial disease by unsupervised machine-learning methods. Hierarchical clustering was applied to 376 vessels from 364 patients with CFR less than the median and fractional flow reserve ≤ 0.8 from a global, multicenter registry. Detailed features of coronary flow physiology and survivals from vessel-oriented composite outcomes (VOCO) were assessed according to the clusters. Clustering defined three distinct physiological subgroups (PS). PS1 (n = 151) were characterized by high resting coronary flow, dominantly left anterior descending artery (LAD) lesions. PS2 (n = 131) were, in contrast, low hyperemic coronary flow, mainly LAD lesions. PS3 (n = 82) mostly consisted of non-LAD lesions with similar flow status to PS1 except for the low hyperemic Pd. Survivals from VOCO were significantly different according to the clusters (p = 0.005) and PS3 had the highest rate of VOCO. In a COX proportional model predicting VOCO, there was a significant interaction between PCI and PSs, suggesting potentially different effects of PCI on outcome between PS1 and PS2. The unsupervised machine-learning approaches provided unique insights into low CFR condition. Among low CFR epicardial lesions, high resting flow with low hyperemic Pd might be related to poor prognosis, and low hyperemic flow in LAD could benefit from elective PCI. CLINICAL TRIAL REGISTRATION INFORMATION: https://clinicaltrials.gov/ct2/show/NCT03690713 , NCT03690713.

Selecting Optimal Subgroups for Treatment Using Many Covariates

We consider the problem of selecting the optimal subgroup to treat when data on covariates are available from a randomized trial or observational study. We distinguish between four different settings including: (1) treatment selection when resources are constrained; (2) treatment selection when resources are not constrained; (3) treatment selection in the presence of side effects and costs; and (4) treatment selection to maximize effect heterogeneity. We show that, in each of these cases, the optimal treatment selection rule involves treating those for whom the predicted mean difference in outcomes comparing those with versus without treatment, conditional on covariates, exceeds a certain threshold. The threshold varies across these four scenarios, but the form of the optimal treatment selection rule does not. The results suggest a move away from the traditional subgroup analysis for personalized medicine. New randomized trial designs are proposed so as to implement and make use of optimal treatment selection rules in healthcare practice.

A framework for estimating and testing qualitative interactions with applications to predictive biomarkers

An effective treatment may only benefit a subset of patients enrolled in a clinical trial. We translate the search for patient characteristics that predict treatment benefit to a search for qualitative interactions, which occur when the estimated response-curve under treatment crosses the estimated response-curve under control. We propose a regression-based framework that tests for qualitative interactions without assuming linearity or requiring pre-specified risk strata; this flexibility is useful in settings where there is limited a priori scientific knowledge about the relationship between features and the response. Simulations suggest that our method controls Type I error while offering an improvement in power over a procedure based on linear regression or a procedure that pre-specifies evenly spaced risk strata. We apply our method to a publicly available dataset to search for a subset of HER2+ breast cancer patients who benefit from adjuvant chemotherapy. We implement our method in Python and share the code/data used to produce our results on GitHub (https://github.com/jhroth/data-example).

Considerations when assessing heterogeneity of treatment effect in patient-centered outcomes research

When baseline risk of an outcome varies within a population, the effect of a treatment on that outcome will vary on at least one scale (e.g., additive, multiplicative). This treatment effect heterogeneity is of interest in patient-centered outcomes research. Based on a literature review and solicited expert opinion, we assert the following: (1) Treatment effect heterogeneity on the additive scale is most interpretable to health-care providers and patients using effect estimates to guide treatment decision-making; heterogeneity reported on the multiplicative scale may be misleading as to the magnitude or direction of a substantively important interaction. (2) The additive scale may give clues about sufficient-cause interaction, although such interaction is typically not relevant to patients' treatment choices. (3) Statistical modeling need not be conducted on the same scale as results are communicated. (4) Statistical testing is one tool for investigations, provided important subgroups are identified a priori, but test results should be interpreted cautiously given nonequivalence of statistical and clinical significance. (5) Qualitative interactions should be evaluated in a prespecified manner for important subgroups. Principled analytic plans that take into account the purpose of investigation of treatment effect heterogeneity are likely to yield more useful results for guiding treatment decisions.

Testing for qualitative heterogeneity: An application to composite endpoints in survival analysis

Composite endpoints are frequently used in clinical outcome trials to provide more endpoints, thereby increasing statistical power. A key requirement for a composite endpoint to be meaningful is the absence of the so-called qualitative heterogeneity to ensure a valid overall interpretation of any treatment effect identified. Qualitative heterogeneity occurs when individual components of a composite endpoint exhibit differences in the direction of a treatment effect. In this paper, we develop a general statistical method to test for qualitative heterogeneity, that is to test whether a given set of parameters share the same sign. This method is based on the intersection-union principle and, provided that the sample size is large, is valid whatever the model used for parameters estimation. We propose two versions of our testing procedure, one based on a random sampling from a Gaussian distribution and another version based on bootstrapping. Our work covers both the case of completely observed data and the case where some observations are censored which is an important issue in many clinical trials. We evaluated the size and power of our proposed tests by carrying out some extensive Monte Carlo simulations in the case of multivariate time to event data. The simulations were designed under a variety of conditions on dimensionality, censoring rate, sample size and correlation structure. Our testing procedure showed very good performances in terms of statistical power and type I error. The proposed test was applied to a data set from a single-center, randomized, double-blind controlled trial in the area of Alzheimer's disease.

Omega-3 Fatty Acid Status Enhances the Prevention of Cognitive Decline by B Vitamins in Mild Cognitive Impairment

A randomized trial (VITACOG) in people with mild cognitive impairment (MCI) found that B vitamin treatment to lower homocysteine slowed the rate of cognitive and clinical decline. We have used data from this trial to see whether baseline omega-3 fatty acid status interacts with the effects of B vitamin treatment. 266 participants with MCI aged ≥70 years were randomized to B vitamins (folic acid, vitamins B6 and B12) or placebo for 2 years. Baseline cognitive test performance, clinical dementia rating (CDR) scale, and plasma concentrations of total homocysteine, total docosahexaenoic and eicosapentaenoic acids (omega-3 fatty acids) were measured. Final scores for verbal delayed recall, global cognition, and CDR sum-of-boxes were better in the B vitamin-treated group according to increasing baseline concentrations of omega-3 fatty acids, whereas scores in the placebo group were similar across these concentrations. Among those with good omega-3 status, 33% of those on B vitamin treatment had global CDR scores >0 compared with 59% among those on placebo. For all three outcome measures, higher concentrations of docosahexaenoic acid alone significantly enhanced the cognitive effects of B vitamins, while eicosapentaenoic acid appeared less effective. When omega-3 fatty acid concentrations are low, B vitamin treatment has no effect on cognitive decline in MCI, but when omega-3 levels are in the upper normal range, B vitamins interact to slow cognitive decline. A clinical trial of B vitamins combined with omega-3 fatty acids is needed to see whether it is possible to slow the conversion from MCI to AD.

A proposed statistical framework for the management of subgroup analyses for large clinical trials

Planned and unplanned subgroup analyses of large clinical trials are frequently performed and the results are sometimes difficult to interpret. The source of a nominal significant finding may come from a true signal, variation of the clinical trial outcome or the observed data structure. Quantitative assessment is critical to the interpretation of the totality of the clinical data. In this article we provide a general framework to manage subgroup analyses and to interpret the findings through a set of supplement analyses to planned main (primary and secondary) analyses, as an alternative to the commonly used multiple comparison framework. The proposed approach collectively and coherently utilizes several quantitative methods and enhances the credibility and interpretability of subgroup analyses. A case study is used to illustrate the application of the proposed method.

Subgroups: time to go back to basic statistical principles?

Exploratory subgroup analyses are an increasing source of controversy as part of the interpretation of the results of clinical trials. In this article, we review the major challenges of multiplicity, statistical methods available to assess consistency of effect, and the part appropriate design plays in mitigating the risk of false conclusions from subgroup analyses. We discuss the problems associated with using definitions of consistency based on effect sizes in specific subgroups. We argue that what is required is a return to basic statistical principles, including more use of modeling techniques.

Detecting qualitative interactions in clinical trials with binary responses

This study considers the detection of treatment-by-subset interactions in a stratified, randomised clinical trial with a binary-response variable. The focus lies on the detection of qualitative interactions. In addition, the presented method is useful more generally, as it can assess the inconsistency of the treatment effects among strata by using an a priori-defined inconsistency margin. The methodology presented is based on the construction of ratios of treatment effects. In addition to multiplicity-adjusted p-values, simultaneous confidence intervals are recommended to use in detecting the source and the amount of a potential qualitative interaction. The proposed method is demonstrated on a multi-regional trial using the open-source statistical software R.

A Tutorial on Interaction

In this tutorial, we provide a broad introduction to the topic of interaction between the effects of exposures. We discuss interaction on both additive and multiplicative scales using risks, and we discuss their relation to statistical models (e.g. linear, log-linear, and logistic models). We discuss and evaluate arguments that have been made for using additive or multiplicative scales to assess interaction. We further discuss approaches to presenting interaction analyses, different mechanistic forms of interaction, when interaction is robust to unmeasured confounding, interaction for continuous outcomes, qualitative or “crossover” interactions, methods for attributing effects to interactions, case-only estimators of interaction, and power and sample size calculations for additive and multiplicative interaction.

The road not taken: transferability issues in multinational trials

BACKGROUND

National regulatory agencies often have to use cost-effectiveness (CE) data from multinational randomized controlled trials (RCTs) for national decision making on reimbursement of new drugs. We need to make the best use of these patient-level data to obtain estimates of country-specific CE. Several methods, ranging from simple to statistically complex, have existed for years. We investigated which of these methods are used to estimate CE ratios in economic evaluations performed alongside recent, multinational RCTs that enrolled at least 500 patients.

METHODS

In this systematic literature review, studies were classified based on whether resource use, unit costs, health outcomes and utility value sets were obtained from all countries, a subset of countries or one country. We recorded if the study presented trial-wide and country-specific CE results and reported the statistical analyses that were used to estimate them.

RESULTS

We included 21 studies, of which the majority used measurements of health care utilization and health outcomes from all countries to estimate CE. Thirteen studies used a one-country valuation of health care utilization; six used a multi-country valuation. Despite the availability of country-specific utility value sets, none of the studies that presented quality-adjusted life-years (QALYs) used multi-country valuation. Valuation of health care utilization and health outcomes was not always consistent within a study: three studies combined a multi-country valuation of health care utilization, with a one-country valuation of health outcomes. Most studies calculated trial-wide CE estimates, while 11 studies calculated country- or region-specific estimates. Thirteen studies used relatively simple methods, which do not take the possible interaction between the country and treatment effect on health care utilization and health outcomes into account. Eight studies used more advanced statistical methods. Three of them used a fixed-effects modeling approach. Five studies explicitly took the hierarchical structure of the data into account, which leads to more appropriate estimates of population average results and associated standard errors. In this way, they help improve transferability of the published results.

CONCLUSION

Based on this systematic review, we concluded that the uptake of more advanced statistical methods has been relatively slow, while simpler naïve methods are still routinely employed.

Detecting moderator effects using subgroup analyses

In the analysis of prevention and intervention studies, it is often important to investigate whether treatment effects vary among subgroups of patients defined by individual characteristics. These "subgroup analyses" can provide information about how best to use a new prevention or intervention program. However, subgroup analyses can be misleading if they test data-driven hypotheses, employ inappropriate statistical methods, or fail to account for multiple testing. These problems have led to a general suspicion of findings from subgroup analyses. This article discusses sound methods for conducting subgroup analyses to detect moderators. Multiple authors have argued that, to assess whether a treatment effect varies across subgroups defined by patient characteristics, analyses should be based on tests for interaction rather than treatment comparisons within the subgroups. We discuss the concept of heterogeneity and its dependence on the metric used to describe treatment effects. We discuss issues of multiple comparisons related to subgroup analyses and the importance of considering multiplicity in the interpretation of results. We also discuss the types of questions that would lead to subgroup analyses and how different scientific goals may affect the study at the design stage. Finally, we discuss subgroup analyses based on post-baseline factors and the complexity associated with this type of subgroup analysis.

Sample size and threshold estimation for clinical trials with predictive biomarkers

With the increasing availability of newly discovered biomarkers personalized drug development is becoming more commonplace. Unless evidence of the dependence of clinical benefit on biomarker classification is a priori unequivocal, personalized drug development needs to jointly investigate treatments and biomarkers in clinical trials. Motivated by the development of contemporary cancer treatments, we propose targeting three main questions sequentially in order to determine (1) whether a drug is efficacious, (2) whether a biomarker can personalize treatment, and (3) how to define personalization. For time-to-event data satisfying the Cox proportional hazards model, we show that (1) and (2) may not directly involve the variance of an interaction term but of a contrast with smaller variance. An asymptotically exact covariance matrix for the parameter vector in the CPH model is derived to construct sample size formulae and an inference approach for thresholds of continuous biomarkers. The covariance matrix also reveals strategies for greater efficiency in trial design, for example, when the biomarker is binary or does not modulate the effect of treatment in the control arm. We motivate our approach by studying the outcome of a contemporary cancer study.

Variable selection for qualitative interactions in personalized medicine while controlling the family-wise error rate

For many years, subset analysis has been a popular topic for the biostatistics and clinical trials literature. In more recent years, the discussion has focused on finding subsets of genomes which play a role in the effect of treatment, often referred to as stratified or personalized medicine. Though highly sought after, methods for detecting subsets with altering treatment effects are limited and lacking in power. In this article we discuss variable selection for qualitative interactions with the aim to discover these critical patient subsets. We propose a new technique designed specifically to find these interaction variables among a large set of variables while still controlling for the number of false discoveries. We compare this new method against standard qualitative interaction tests using simulations and give an example of its use on data from a randomized controlled trial for the treatment of depression.

Testing for qualitative interactions between stages in an adaptive study

I consider the underlying structure for a test of qualitative interaction of a treatment when assessing heterogeneity between stages in an adaptive trial. Since decisions about the clinical utility of a drug are based on the balance of risks and benefits, a quantitative interaction in treatment efficacy across different groups could lead to qualitatively different decisions. Thus, the difference between quantitative and qualitative interactions is not a true dichotomy. I show that the standard tests for qualitative interactions (Gail and Simon,Biometrics 1985; 41:361-372; Piantadosi and Gail, Statist. Med. 1993; 12:1239-1248) are very conservative in this application. Theoretical calculations in a simpler situation confirm that the published criteria are very conservative, which may help explain why the tests are known to have very low power to detect interaction. I introduce the concept of 'minimum detectable effect', which is the smallest effect that a study could identify as statistically significant. I propose that important heterogeneity between stages in an adaptive trial be identified when two criteria are met. First, at least one individual stage must be below the overall study mean by at least the minimum detectable effect. Second, using an appropriate critical value based on simulations, there must be statistically significant heterogeneity between the stages.

Detecting qualitative interaction: a Bayesian approach

Differences in treatment effects between centers in a multi-center trial may be important. These differences represent treatment by subgroup interaction. Peto defines qualitative interaction (QI) to occur when the simple treatment effect in one subgroup has a different sign than in another subgroup: this interaction is important. Interaction where the treatment effects are of the same sign in all subgroups is called quantitative and is often not important because the treatment recommendation is identical in all cases. A hierarchical model is used here with exchangeable mean responses to each treatment between subgroups. The posterior probability of QI and the corresponding Bayes factor are proposed as a diagnostic and as a test statistic. The model is motivated by two multi-center trials with binary responses. The frequentist power and size of the test using the Bayes factor are examined and compared with two other commonly used tests. The impact of imbalance between the sample sizes in each subgroup on power is examined, and the test based on the Bayes factor typically has better power for unbalanced designs, especially for small sample sizes. An exact test based on the Bayes factor is also suggested assuming the hierarchical model. The Bayes factor provides a concise summary of the evidence for or against QI. It is shown by example that it is easily adapted to summarize the evidence for 'clinically meaningful QI,' defined as the simple effects being of opposite signs and larger in absolute value than a minimal clinically meaningful effect.

Test for the consistency of noninferiority from multiple clinical trials

A testing procedure is proposed to assess the consistency of noninferiority from a collection of trials based on simultaneous t lower confidence bounds or Scheffé's lower confidence bounds. Methods for simultaneous inferences on pairwise or many-to-one comparisons among multiple noninferiority trials are also discussed. To avoid bias due to subjective trial exclusion a tuning parameter k is embedded into the testing procedure to provide flexibility to quantify the "consistency of noninferiority" when the total number of trials is large. The size and power of the proposed test are discussed. The method is illustrated using simulations and real data analysis.

Non-inferiority of new procedure to standard procedure in stratified matched-pair design

We consider the statistical testing for non-inferiority of a new treatment compared with the standard one under matched-pair setting in a stratified study or in several trials. A non-inferiority test based on the efficient scores and a Mantel-Haenszel (M-H) like procedure with restricted maximum likelihood estimators (RMLEs) of nuisance parameters and their corresponding sample size formulae are presented. We evaluate the above tests and the M-H type Wald test in level and power. The stratified score test is conservative and provides the best power. The M-H like procedure with RMLEs gives an accurate level. However, the Wald test is anti-conservative and we suggest caution when it is used. The unstratified score test is not biased but it is less powerful than the stratified score test when base-line probabilities related to strata are not the same. This investigation shows that the stratified score test possesses optimum statistical properties in testing non-inferiority. A common difference between two proportions across strata is the basic assumption of the stratified tests, we present appropriate tests to validate the assumption and related remarks.

Detecting Qualitative Interactions in Clinical Trials: An Extension of Range Test

To help interpret a treatment effect in clinical trials, investigators usually examine whether the observed treatment effect is the same in various subsets of patients. The qualitative interaction, which means that the treatment is beneficial in some subsets and harmful in others, is of major importance. In this paper, a new statistical test is developed for detecting such interactions. The new test is an extension of the well-known range test, but utilizes all observed treatment differences rather than only the maximum and the minimum values. Extensive simulations indicate that the proposed extended range test generally outperforms the range test and is even better than the likelihood ratio test in the sense that the extended range test is much more powerful than the likelihood test when one treatment is superior to the other in most subsets and yet does not lose much power otherwise. It is also illustrated through a real clinical trial example that the extended range test detects the qualitative interaction while the range test and likelihood ratio test do not.

Statistical considerations for noninferiority/equivalence trials in vaccine development

Noninferioritylequivalence designs are often used in vaccine clinical trials. The goal of these designs is to demonstrate that a new vaccine, or new formulation or regimen of an existing vaccine, is similar in terms of effectiveness to the existing vaccine, while offering such advantages as easier manufacturing, easier administration, lower cost, or improved safety profile. These noninferioritylequivalence designs are particularly useful in four common types of immunogenicity trials: vaccine bridging trials, combination vaccine trials, vaccine concomitant use trials, and vaccine consistency lot trials. In this paper, we give an overview of the key statistical issues and recent developments for noninferioritylequivalence vaccine trials. Specifically, we cover the following topics: (i) selection of study endpoints; (ii) formulation of the null and alternative hypotheses; (iii) determination of the noninferioritylequivalence margin; (iv) selection of efficient statistical methods for the statistical analysis of noninferioritylequivalence vaccine trials, with particular emphasis on adjustment for stratification factors and missing pre-or post-vaccination data; and (v) the calculation of sample size and power.

Sources of variability on the estimate of treatment effect in the PROWESS trial: Implications for the design and conduct of future studies in severe sepsis*

OBJECTIVE

To elucidate sources of variability in the estimate of treatment effects in a successful phase 3 trial in severe sepsis and to assess their implications on the design of future clinical trials.

DESIGN

Retrospective evaluation of prospectively defined subgroups from a large phase 3, placebo-controlled clinical trial (PROWESS).

SETTING

The study involved 164 medical centers.

PATIENTS

Patients were 1,690 patients with severe sepsis.

INTERVENTIONS

Drotrecogin alfa (activated) (Xigris) 24 microg/kg/hr for 96 hrs, or placebo.

MEASUREMENTS AND MAIN RESULTS

All prospectively defined subgroups were examined to identify treatment effects that potentially differed across subgroup strata (assessed by Breslow-Day p < .10). Potential interactions were identified for subgroups defined by a) presence vs. absence of a significant protocol violation (p = .07); b) original vs. amended protocol (p = .08); and c) Acute Physiology and Chronic Health Evaluation (APACHE) II quartile at baseline (p = .09). No treatment benefit was observed in patients having a protocol violation, regardless of type. There appeared to be less treatment effect in patients enrolled under the original vs. amended protocol. The risk ratio exceeded 1.0 for patients in the lowest APACHE II score quartile. A highly significant correlation was observed between the sequence of enrollment at a site, the frequency of protocol violations, and the observed treatment effect. As enrollment increased, frequency of protocol violations decreased (p < .0001) and the treatment effect improved. The correlation between the sequence of enrollment and improvement in treatment effect remained even after removal of patients with protocol violations. Removal of the first block of patients at each site from the analysis reduced the extent of interaction by protocol version and APACHE II score.

CONCLUSIONS

A learning curve appeared to be present within the PROWESS trial such that the ability to demonstrate efficacy improved with increasing site experience. This potential learning curve may have implications for design of future trials. Investigational sites may need to require a minimum level of protocol-specific experience to appropriately implement a given trial. This experience should be an important consideration in designing trials and analysis plans. Diligence by coordinating centers, site investigators, study coordinators, and sponsors is necessary to ensure that the protocol is executed as designed such that a treatment benefit, if present, will be evident.

Test for qualitative interaction in equivalence trials when the number of centres is large

We propose a generalized testing procedure to test for qualitative interaction in equivalence trails when the number of centres is large. The proposed testing procedure allows for an adaptable definition of qualitative interaction that can take into account the total number of centres. A tuning parameter k (k > or = 0) is introduced to quantify qualitative interaction. The testing procedure is proposed for equivalence trials with symmetric or asymmetric margins. In addition to the test procedure, we also provide explicit formulae for the power calculation. The proposed test is relatively easy to implement using any statistical software. Examples for detecting qualitative interaction are given to illustrate the method.

Assessing the appropriateness of combining economic data from multinational clinical trials

Because of the potential for large variability among countries in the utilization and cost of health care resources, it is important to assess the appropriateness of combining economic data across the countries in a multinational clinical economic trial. We show how available tests for interaction can be applied to economic endpoints, including cost-effectiveness ratios and net health benefits. This analysis includes a characterization of possible interactions being quantitative or qualitative in nature. In the absence of interaction, a pooled estimate of the economic endpoint should be representative of the participating countries. We explore the analytic issues by further analysing data from the Scandinavian Simvastatin Survival Study (4S).

Tests against qualitative interaction: exact critical values and robust tests

Consider a study to evaluate treatment A with a placebo in two or more groups of patients. If treatment A is beneficial to one group of patients and harmful to another, then we say that there is qualitative interaction or crossover interaction between patient groups and the treatments. Gail and Simon (1985, Biometrics 41, 361-372) developed a large-sample procedure for this testing problem. Their test has received favorable coverage in the literature. In this article, we obtain corresponding exact finite sample results for normal error distribution and provide a table of critical values. The test statistic is similar to the familiar F-ratio, and its p-value is equal to a weighted sum of tail areas of F-distributions. The computations to implement this are simple. A simulation study shows that the exact critical values provided here for normal error distribution are preferable to the asymptotic critical values for a wide range of error distributions. We also develop tests that are power robust against long-tailed error distributions. Our robust test uses M-estimators instead of the least squares estimators. We show that the efficiency robustness of the M-estimator translates to power robustness of the corresponding test. Therefore, our robust tests are better if outliers are expected. A simulation study illustrates the substantial power advantages of our robust tests.

Controversies about sponsor initiated re-analyses of clinical trial data in the licensing process

I present my experience with sponsor initiated re-analyses as a response to the regulatory agency's 'Mängelbericht', and the difficulties which arose from the fact that the regulatory agency was not involved in the plans for correcting mistakes or irregularities in the data and for the (re-)analysis of them. I also discuss the related problem when there are major discrepancies between the planned procedures laid down in the protocol and the actually applied procedures in the study report. I give examples for poor planning of a clinical study (sample size, statistical analysis, target variable(s)) and how this reduces the strength and value of a study. I introduce the phrase 'neutral party' as a means to resolve regulatory concerns about partiality in the sponsor's decisions, for example, introduction of new statistical models, different from the one planned, change in the order of the target variables, or dropping variables from the list of target variables.