Randomization inference for treatment effect variation

Modern approaches for evaluating treatment effect heterogeneity from clinical trials and observational data

In this paper, we review recent advances in statistical methods for the evaluation of the heterogeneity of treatment effects (HTE), including subgroup identification and estimation of individualized treatment regimens, from randomized clinical trials and observational studies. We identify several types of approaches using the features introduced in Lipkovich et al (Stat Med 2017;36: 136-196) that distinguish the recommended principled methods from basic methods for HTE evaluation that typically rely on rules of thumb and general guidelines (the methods are often referred to as common practices). We discuss the advantages and disadvantages of various principled methods as well as common measures for evaluating their performance. We use simulated data and a case study based on a historical clinical trial to illustrate several new approaches to HTE evaluation.

Comparing type 1 and type 2 error rates of different tests for heterogeneous treatment effects

Psychologists are increasingly interested in whether treatment effects vary in randomized controlled trials. A number of tests have been proposed in the causal inference literature to test for such heterogeneity, which differ in the sample statistic they use (either using the variance terms of the experimental and control group, their empirical distribution functions, or specific quantiles), and in whether they make distributional assumptions or are based on a Fisher randomization procedure. In this manuscript, we present the results of a simulation study in which we examine the performance of the different tests while varying the amount of treatment effect heterogeneity, the type of underlying distribution, the sample size, and whether an additional covariate is considered. Altogether, our results suggest that researchers should use a randomization test to optimally control for type 1 errors. Furthermore, all tests studied are associated with low power in case of small and moderate samples even when the heterogeneity of the treatment effect is substantial. This suggests that current tests for treatment effect heterogeneity require much larger samples than those collected in current research.

Model-based adaptive randomization procedures for heteroscedasticity of treatment responses

In clinical trials, the responses of patients usually depend on the assigned treatment as well as some important covariates, which may cause heteroscedasticity in treatment responses. As clinical trials are generally designed to demonstrate efficacy for the overall population, they are usually not adequately powered for detecting interactions. To improve the power of interaction tests, this article develops two model-based adaptive randomization procedures for heteroscedasticity of treatment responses, and derives their limiting allocation proportions, which are generalizations of the Neyman allocation. Issues of hypothesis testing and sample size estimation are also addressed. Simulation studies show that compared with complete randomization, the two model-based randomization procedures have greater power to detect differences in systematic effects, main treatment effects and treatment-covariate interactions. In addition, the validity of limiting allocation proportion is also verified through simulations.

Probing potential priming: Defining, quantifying, and testing the causal priming effect using the potential outcomes framework

Having previously seen an item helps uncover the item another time, given a perceptual or cognitive cue. Oftentimes, however, it may be difficult to quantify or test the existence and size of a perceptual or cognitive effect, in general, and a priming effect, in particular. This is because to examine the existence of and quantify the effect, one needs to compare two outcomes: the outcome had one previously seen the item vs. the outcome had one not seen the item. But only one of the two outcomes is observable. Here, we argue that the potential outcomes framework is useful to define, quantify, and test the causal priming effect. To demonstrate its efficacy, we apply the framework to study the priming effect using data from a between-subjects study involving English word identification. In addition, we show that what has been used intuitively by experimentalists to assess the priming effect in the past has a sound mathematical foundation. Finally, we examine the links between the proposed method in studying priming and the multinomial processing tree (MPT) model, and how to extend the method to study experimental paradigms involving exclusion and inclusion instructional conditions.

Assessment of the predictive power of a causal variable: An application to the Head Start impact study

In a study attempting to estimate a causal effect of a causal variable, an assessment of the predictive power of the causal variable can shed light on the heterogeneity around its average effect. Using data from the Head Start Impact Study, a randomized controlled trial of the Head Start, a nation-wide early childhood education program in the United States, we provide a parallel comparison between measures of average effect and predictive power of the Head Start on five cognitive outcomes. We observed that one year of the Head Start increased scores for all five outcomes, with effect sizes ranging from 0.12 to 0.19 standard deviations. Percent variation explained by the Head Start ranged from 0.56 to 1.62%. For binary versions of the outcomes, the overall pattern remained; the Head Start on average improved the outcomes by meaningful magnitudes. In contrast, in a fully adjusted model, the Head Start only improved area under the curve (AUC) by less than 1% and its influence on the variance of predicted probabilities was negligible. The Head-Start-only model only achieved AUC ranging from 50.22 to 55.24%. Negligible predictive power despite the significant average effect suggests that the heterogeneity in effects may be large. The average effect estimates may not generalize well to different populations or different Head Start program settings. Assessment of the predictive power of a causal variable in randomized data should be a routine practice as it can provide helpful information on the causal effect and especially its heterogeneity.

•

The average effect and predictive power of Head Start were assessed.

•

Head Start increased scores for five cognitive outcomes with effect sizes ranging from 0.12 to 0.19 standard deviations.

•

Head Start only explained less than 2% of the child-level variance in cognitive outcomes.

•

Head Start only achieved about 50% in AUC and improved less than 1% in AUC when added to a full model.

•

The impact of Head Start is meaningfully sized on average but the effect heterogeneity is large.

Assessment of heterogeneous Head Start treatment effects on cognitive and social-emotional outcomes

Head Start is a federally funded, nation-wide program in the U.S. for enhancing school readiness of children aged 3–5 from low-income families. Understanding heterogeneity in treatment effects (HTE) is an important task when evaluating programs, but most attempts to explore HTE in Head Start have been limited to subgroup analyses that rely on average treatment effects by subgroups. This study applies an extension of multilevel modelling, complex variance modelling, to data from a randomized controlled trial of Head Start, Head Start Impact Study (HSIS). The treatment effects on the variance, in addition to the mean, of nine cognitive and social-emotional outcomes were assessed for 4,442 children aged 3–4 years who were followed until their 3rd grade year. Head Start had positive short-term effects on the means of multiple cognitive outcomes while having no effect on the means of social-emotional outcomes. Head Start reduced the variances of multiple cognitive and one social-emotional outcomes, meaning that substantial HTE exists. In particular, the increased mean and decreased variance reflect the ability of Head Start to improve the outcomes and reduce their variability. Exploratory secondary analyses suggested that larger benefits for children with Spanish as a primary language and low parental educational level partly explained the reduced variability, but the HTE remained and the variability was reduced even within these subgroups. Routinely monitoring the treatment effects on the variance, in addition to the mean, would lead to a more comprehensive program evaluation that describes how a program performs on average and on the entire distribution.

Characteristics, Prognosis, and Competing Risk Nomograms of Cutaneous Malignant Melanoma: Evidence for Pigmentary Disorders

PURPOSE

Cutaneous malignant melanoma (CMM) always presents as a complex disease process with poor prognosis. The objective of the present study was to explore the influence of solitary or multiple cancers on the prognosis of patients with CMM to better understand the landscape of CMM.

METHODS

We reviewed the records of CMM patients between 2004 and 2015 from the Surveillance, Epidemiology, and End Results Program. The cumulative incidence function was used to represent the probabilities of death. A novel causal inference method was leveraged to explore the risk difference to death between different types of CMM, and nomograms were built based on competing risk models.

RESULTS

The analysis cohort contained 165,043 patients with CMM as the first primary malignancy. Patients with recurrent CMM and multiple primary tumors had similar overall survival status (p = 0.064), while their demographics and cause-specific death demonstrated different characteristics than those of patients with solitary CMM (p < 0.001), whose mean survival times are 75.4 and 77.3 months and 66.2 months, respectively. Causal inference was further applied to unveil the risk difference of solitary and multiple tumors in subgroups, which was significantly different from the total population (p < 0.05), and vulnerable groups with high risk of death were identified. The established competing risk nomograms had a concordance index >0.6 on predicting the probabilities of death of CMM or other cancers individually across types of CMM.

CONCLUSION

Patients with different types of CMM had different prognostic characteristics and different risk of cause-specific death. The results of this study are of great significance in identifying the high risk of cause-specific death, enabling targeted intervention in the early period at both the population and individual levels.

Treatment effect heterogeneity in the head start impact study: A systematic review of study characteristics and findings

There have been consistent efforts to assess treatment effect heterogeneity (TEH) of Head Start using the data from the Head Start Impact Study (HSIS), a randomized controlled trial of a federally funded child development program for a nationally representative sample of low-income parents and their 3- and 4-year-old children in the United States. Including 28 studies on TEH of Head Start, this review found that multiple high-risk subgroups (e.g., children with lower cognitive abilities, Spanish-speaking dual language learners) experienced larger gains across a range of developmental and parental outcomes, but mixed results for several subgroups. Most studies focused on subgroup analyses, cognitive and social-emotional outcomes, and short-term effects. Further studies on distributional effects, health and parental outcomes, and long-term effects are warranted. Finally, suggestions for future research on TEH of Head Start are discussed, which are applicable to other child development programs and policy evaluations.

Causal mediation analysis with latent subgroups

In biomedical studies, the causal mediation effect might be heterogeneous across individuals in the study population due to each study subject's unique characteristics. While individuals' mediation effects may differ from each other, it is often reasonable and more interpretable to assume that individuals belong to several distinct latent subgroups with similar attributes. In this article, we first show that the subgroup-specific mediation effect can be identified under the group-specific sequential ignorability assumptions. Then, we propose a simple mixture modeling approach to account for the latent subgroup structure where each mixture component corresponds to one latent subgroup in the linear structural equation model framework. Model parameters can be estimated using the standard expectation-maximization (EM) algorithm. Each individual's subgroup membership can be inferred based on the posterior probability. We propose to use the singular Bayesian information criterion to consistently select the number of latent subgroups by recognizing that the Fisher information matrix for mixture models might be singular. We then propose to use nonparametric bootstrap method to compute standard errors and confidence intervals. We conducted simulation studies to evaluate the empirical performance of our proposed method named iMed. Finally, we reanalyzed a DNA methylation data set from the Normative Aging Study and found that the mediation effects of two well-documented DNA methylation CpG sites are heterogeneous across two latent subgroups in the causal pathway from smoking behavior to lung function. We also developed an R package iMed for public use.

Causal estimands and confidence intervals associated with Wilcoxon-Mann-Whitney tests in randomized experiments

Although the P value from a Wilcoxon-Mann-Whitney test is used often with randomized experiments, it is rarely accompanied with a causal effect estimate and its confidence interval. The natural parameter for the Wilcoxon-Mann-Whitney test is the Mann-Whitney parameter, ϕ, which measures the probability that a randomly selected individual in the treatment arm will have a larger response than a randomly selected individual in the control arm (plus an adjustment for ties). We show that the Mann-Whitney parameter may be framed as a causal parameter and show that it is not equal to a closely related and nonidentifiable causal effect, ψ, the probability that a randomly selected individual will have a larger response under treatment than under control (plus an adjustment for ties). We review the paradox, first expressed by Hand, that the ψ parameter may imply that the treatment is worse (or better) than control, while the Mann-Whitney parameter shows the opposite. Unlike the Mann-Whitney parameter, ψ is nonidentifiable from a randomized experiment. We review some nonparametric assumptions that rule out Hand's paradox through bounds on ψ and use bootstrap methods to make inferences on those bounds. We explore the relationship of the proportional odds parameter to Hand's paradox, showing that the paradox may occur for proportional odds parameters between 1/9 and 9. Thus, large effects are needed to ensure that if treatment appears better by the Mann-Whitney parameter, then treatment improves responses in most individuals. We demonstrate these issues using a vaccine trial.