Assessing the sensitivity of methods for estimating principal causal effects

Tailored guidance to apply the Estimand framework to Trials within Cohorts (TwiCs) studies

Objective: The estimand framework offers a structured approach to define the treatment effect to be estimated in a clinical study. Defining the estimand upfront helps formulating the research question and informs study design, data collection and statistical analysis methods. Since the Trials within Cohorts (TwiCs) design has unique characteristics, the objective of this study is to describe considerations and provide guidance for formulating estimands for TwiCs studies. Methods: The key attributes of an estimand are the target population, treatments that are compared, the endpoint, intercurrent events and their handling, and the population-level summary measure. The estimand framework was applied retrospectively to two TwiCs studies: the SPONGE and UMBRELLA Fit trial. The aim is to demonstrate how the estimand framework can be implemented in TwiCs studies, thereby focusing on considerations relevant for defining the estimand. Three estimands were defined for both studies. For the SPONGE trial, estimators were derived. Results: Intercurrent events considered to occur exclusively or more frequently in TwiCs studies compared to conventional randomized trials included intervention refusal after randomization, misalignment of timing of routine cohort measurements and the intervention period, and participants in the control arm initiating treatments similar to the studied intervention. Considerations for handling refusal after randomization related to decisions on whether the target population should include all eligible participants or the subpopulation that would accept (or undergo) the intervention when offered. Considerations for handling treatment initiation in the control arm and misalignments of timing related to decisions on whether such events should be considered part of treatment policy or whether interest is in a hypothetical scenario where such events do not occur. Conclusion: The TwiCs study design has unique features that pose specific considerations when formulating an estimand. The examples in this study can provide guidance in the definition of estimands in future TwiCs studies.

Causal Inference Over a Subpopulation: The Effect of Malaria Vaccine in Women During Pregnancy

Preventing malaria during pregnancy is of critical importance, yet there are no approved malaria vaccines for pregnant women due to lack of efficacy results within this population. Conducting a randomized trial in pregnant women throughout the entire duration of pregnancy is impractical. Instead, a randomized trial was conducted among women of childbearing potential (WOCBP), and some participants became pregnant during the 2-year study. We explore a statistical method for estimating vaccine effect within the target subpopulation-women who can naturally become pregnant, namely, women who can become pregnant under a placebo condition-within the causal inference framework. Two vaccine effect estimators are employed to effectively utilize baseline characteristics and account for the fact that certain baseline characteristics were only available from pregnant participants. The first estimator considers all participants but can only utilize baseline variables collected from the entire participant pool. In contrast, the second estimator, which includes only pregnant participants, utilizes all available baseline information. Both estimators are evaluated numerically through simulation studies and applied to the WOCBP trial to assess vaccine effect against pregnancy malaria.

Identification of complier and noncomplier average causal effects in the presence of latent missing-at-random (LMAR) outcomes: a unifying view and choices of assumptions

The study of treatment effects is often complicated by noncompliance and missing data. In the one-sided noncompliance setting where of interest are the complier and noncomplier average causal effects, we address outcome missingness of the latent missing at random type (LMAR, also known as latent ignorability). That is, conditional on covariates and treatment assigned, the missingness may depend on compliance type. Within the instrumental variable (IV) approach to noncompliance, methods have been proposed for handling LMAR outcome that additionally invoke an exclusion restriction-type assumption on missingness, but no solution has been proposed for when a non-IV approach is used. This article focuses on effect identification in the presence of LMAR outcomes, with a view to flexibly accommodate different principal identification approaches. We show that under treatment assignment ignorability and LMAR only, effect nonidentifiability boils down to a set of two connected mixture equations involving unidentified stratum-specific response probabilities and outcome means. This clarifies that (except for a special case) effect identification generally requires two additional assumptions: a specific missingness mechanism assumption and a principal identification assumption. This provides a template for identifying effects based on separate choices of these assumptions. We consider a range of specific missingness assumptions, including those that have appeared in the literature and some new ones. Incidentally, we find an issue in the existing assumptions, and propose a modification of the assumptions to avoid the issue. Results under different assumptions are illustrated using data from the Baltimore Experience Corps Trial.

Sensitivity analysis for principal ignorability violation in estimating complier and noncomplier average causal effects

An important strategy for identifying principal causal effects (popular estimands in settings with noncompliance) is to invoke the principal ignorability (PI) assumption. As PI is untestable, it is important to gauge how sensitive effect estimates are to its violation. We focus on this task for the common one-sided noncompliance setting where there are two principal strata, compliers and noncompliers. Under PI, compliers and noncompliers share the same outcome-mean-given-covariates function under the control condition. For sensitivity analysis, we allow this function to differ between compliers and noncompliers in several ways, indexed by an odds ratio, a generalized odds ratio, a mean ratio, or a standardized mean difference sensitivity parameter. We tailor sensitivity analysis techniques (with any sensitivity parameter choice) to several types of PI-based main analysis methods, including outcome regression, influence function (IF) based and weighting methods. We discuss range selection for the sensitivity parameter. We illustrate the sensitivity analyses with several outcome types from the JOBS II study. This application estimates nuisance functions parametrically - for simplicity and accessibility. In addition, we establish rate conditions on nonparametric nuisance estimation for IF-based estimators to be asymptotically normal - with a view to inform nonparametric inference.

An SMS chatbot digital educational program to increase healthy eating behaviors in adolescence: A multifactorial randomized controlled trial among 7,890 participants in the Danish National Birth Cohort

BACKGROUND

Few cost-effective strategies to shift dietary habits of populations in a healthier direction have been identified. We examined if participating in a chatbot health education program transmitted by Short Messages Service ("SMS-program") could improve adolescent dietary behaviors and body weight trajectories. We also explored possible added effects of maternal or peer involvement.

METHODS AND FINDINGS

We conducted a randomized controlled trial (RCT) among adolescents from the Danish National Birth Cohort (DNBC). Eligible were adolescents who during 2015 to 2016 at age 14 years had completed a questionnaire assessing height, weight, and dietary habits. Two thirds were offered participation in an SMS-program, whereas 1/3 ("non-SMS group") received no offer. The SMS program aimed to improve 3 key dietary intake behaviors: sugar-sweetened beverages (SSBs), fruit and vegetables (FV), and fish. The offered programs had 3 factorially randomized schemes; the aims of these were to test effect of asking the mother or a friend to also participate in the health promotion program, and to test the effect of a 4-week individually tailored SMS program against the full 12-week SMS program targeting all 3 dietary factors. Height and weight and intakes of SSB, FV, and fish were assessed twice by a smartphone-based abbreviated dietary questionnaire completed at 6 months (m) and 18 m follow-up. Main outcome measures were (1) body mass index (BMI) z-score; and (2) an abbreviated Healthy Eating Index (mini-HEI, 1 m window, as mean of z-scores for SSB, FV, and fish). Among the 7,890 randomized adolescents, 5,260 were assigned to any SMS program; 63% (3,338) joined the offered program. Among the 7,890 randomized, 74% (5,853) and 68% (5,370) responded to follow-ups at 6 m and 18 m, respectively. Effects were estimated by intention-to-treat (ITT) analyses and inverse probability weighted per-protocol (IPW-PP) analyses excluding adolescents who did not join the program. Mean (standard deviation (SD)) mini-HEI at baseline, 6 m and 18 m was -0.01 (0.64), 0.01 (0.59), and -0.01 (0.59), respectively. In ITT-analyses, no effects were observed, at any time point, in those who had received any SMS program compared to the non-SMS group, on BMI z-score (6 m: -0.010 [95% confidence interval (CI) -0.035, 0.015]; p = 0.442, 18 m: 0.002 [95% CI -0.029, 0.033]; p = 0.901) or mini-HEI (6 m: 0.016 [95% CI -0.011, 0.043]; p = 0.253, 18m: -0.016 [95% CI -0.045, 0.013]; p = 0.286). In IPW-PP analyses, at 6 m, a small decrease in BMI z-score (-0.030 [95% CI -0.057, -0.003]; p = 0.032) was observed, whereas no significant effect was observed in mini-HEI (0.027 [95% CI -0.002, 0.056]; p = 0.072), among those who had received any SMS program compared to the non-SMS group. At 18 m, no associations were observed (BMI z-score: -0.006 [95% CI -0.039, 0.027]; p = 0.724, and mini-HEI: -0.005 [95% CI -0.036, 0.026]; p = 0.755). The main limitations of the study were that DNBC participants, though derived from the general population, tend to have higher socioeconomic status than average, and that outcome measures were self-reported.

CONCLUSIONS

In this study, a chatbot health education program delivered through an SMS program had no effect on dietary habits or weight trajectories in ITT analyses. However, IPW-PP-analyses, based on those 63% who had joined the offered SMS program, suggested modest improvements in weight development at 6 m, which had faded at 18 m. Future research should focus on developing gender-specific messaging programs including "booster" messages to obtain sustained engagement.

CLINICAL TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT02809196 https://clinicaltrials.gov/study/NCT02809196.

Multiply robust estimation of principal causal effects with noncompliance and survival outcomes

Treatment noncompliance and censoring are two common complications in clinical trials. Motivated by the ADAPTABLE pragmatic clinical trial, we develop methods for assessing treatment effects in the presence of treatment noncompliance with a right-censored survival outcome. We classify the participants into principal strata, defined by their joint potential compliance status under treatment and control. We propose a multiply robust estimator for the causal effects on the survival probability scale within each principal stratum. This estimator is consistent even if one, sometimes two, of the four working models-on the treatment assignment, the principal strata, censoring, and the outcome-is misspecified. A sensitivity analysis strategy is developed to address violations of key identification assumptions, the principal ignorability and monotonicity. We apply the proposed approach to the ADAPTABLE trial to study the causal effect of taking low- versus high-dosage aspirin on all-cause mortality and hospitalization from cardiovascular diseases.

Effects of Compliance to a Training Intervention for High School Students with ADHD

OBJECTIVE

We evaluated the effects of treatment compliance with the Challenging Horizons Program (CHP) for high school aged adolescents with attention-deficit hyperactivity disorder (ADHD).

METHOD

Participants were 185 high school aged adolescents (65% non-Hispanic White; 79% male) with a diagnosis of ADHD who were randomly assigned to either CHP or community control. Outcomes included parent-rated academic functioning, parent- and self-rated social-emotional functioning, and GPA. The complier average causal effect (CACE) was estimated using propensity-weighted models for youth engaging in ≥ 30 CHP individual sessions (15-20 min) across the academic year.

RESULTS

Most (78%) CHP participants engaged in≥30 CHP sessions. CACE analyses using latent growth curve modeling revealed significant treatment effects among treatment compliers across ratings of academic and social outcomes relative to similar control participants. For most outcomes, CACE estimates were larger than those found in intent-to-treat analyses, especially at 6-months follow-up.

CONCLUSIONS

Compliance with 30 or more individual CHP sessions appeared to be an attainable threshold associated with incremental gains across several academic and social outcomes. Effects of compliance were amplified at 6-months follow-up, supporting the hypothesized theory of change of training interventions. Future work should focus on facilitators of treatment engagement and feasibility of the CHP as delivered by high school personnel.

The survival-incorporated median vs the median in the survivors or in the always-survivors: What are we measuring? and Why?

Many clinical studies evaluate the benefit of a treatment based on both survival and other continuous/ordinal clinical outcomes, such as quality of life scores. In these studies, when subjects die before the follow-up assessment, the clinical outcomes become undefined and are truncated by death. Treating outcomes as "missing" or "censored" due to death can be misleading for treatment effect evaluation. We show that if we use the median in the survivors or in the always-survivors as estimands to summarize clinical outcomes, we may conclude that a trade-off exists between the probability of survival and good clinical outcomes, even in settings where both the probability of survival and the probability of any good clinical outcome are better for one treatment. Therefore, we advocate not always treating death as a mechanism through which clinical outcomes are missing, but rather as part of the outcome measure. To account for the survival status, we describe the survival-incorporated median as an alternative summary measure for outcomes in the presence of death. The survival-incorporated median is the threshold such that 50% of the population is alive with an outcome above that threshold. Through conceptual examples and an application to a prostate cancer treatment study, we show that the survival-incorporated median provides a simple and useful summary measure to inform clinical practice.

A systematic review of simulation studies which compare existing statistical methods to account for non-compliance in randomised controlled trials

INTRODUCTION

Non-compliance is a common challenge for researchers and may reduce the power of an intention-to-treat analysis. Whilst a per protocol approach attempts to deal with this issue, it can result in biased estimates. Several methods to resolve this issue have been identified in previous reviews, but there is limited evidence supporting their use. This review aimed to identify simulation studies which compare such methods, assess the extent to which certain methods have been investigated and determine their performance under various scenarios.

METHODS

A systematic search of several electronic databases including MEDLINE and Scopus was carried out from conception to 30th November 2022. Included papers were published in a peer-reviewed journal, readily available in the English language and focused on comparing relevant methods in a superiority randomised controlled trial under a simulation study. Articles were screened using these criteria and a predetermined extraction form used to identify relevant information. A quality assessment appraised the risk of bias in individual studies. Extracted data was synthesised using tables, figures and a narrative summary. Both screening and data extraction were performed by two independent reviewers with disagreements resolved by consensus.

RESULTS

Of 2325 papers identified, 267 full texts were screened and 17 studies finally included. Twelve methods were identified across papers. Instrumental variable methods were commonly considered, but many authors found them to be biased in some settings. Non-compliance was generally assumed to be all-or-nothing and only occurring in the intervention group, although some methods considered it as time-varying. Simulation studies commonly varied the level and type of non-compliance and factors such as effect size and strength of confounding. The quality of papers was generally good, although some lacked detail and justification. Therefore, their conclusions were deemed to be less reliable.

CONCLUSIONS

It is common for papers to consider instrumental variable methods but more studies are needed that consider G-methods and compare a wide range of methods in realistic scenarios. It is difficult to make conclusions about the best method to deal with non-compliance due to a limited body of evidence and the difficulty in combining results from independent simulation studies.

PROSPERO REGISTRATION NUMBER

CRD42022370910.

Defining estimands in clinical trials: A unified procedure

The ICH E9 (R1) addendum proposes five strategies to define estimands by addressing intercurrent events. However, mathematical forms of these targeted quantities are lacking, which might lead to discordance between statisticians who estimate these quantities and clinicians, drug sponsors, and regulators who interpret them. To improve the concordance, we provide a unified four-step procedure for constructing the mathematical estimands. We apply the procedure for each strategy to derive the mathematical estimands and compare the five strategies in practical interpretations, data collection, and analytical methods. Finally, we show that the procedure can help ease tasks of defining estimands in settings with multiple types of intercurrent events using two real clinical trials.

Sensitivity analyses for the principal ignorability assumption using multiple imputation

In the context of clinical trials, there is interest in the treatment effect for subpopulations of patients defined by intercurrent events, namely disease-related events occurring after treatment initiation that affect either the interpretation or the existence of endpoints. With the principal stratum strategy, the ICH E9(R1) guideline introduces a formal framework in drug development for defining treatment effects in such subpopulations. Statistical estimation of the treatment effect can be performed based on the principal ignorability assumption using multiple imputation approaches. Principal ignorability is a conditional independence assumption that cannot be directly verified; therefore, it is crucial to evaluate the robustness of results to deviations from this assumption. As a sensitivity analysis, we propose a joint model that multiply imputes the principal stratum membership and the outcome variable while allowing different levels of violation of the principal ignorability assumption. We illustrate with a simulation study that the joint imputation model-based approaches are superior to naive subpopulation analyses. Motivated by an oncology clinical trial, we implement the sensitivity analysis on a time-to-event outcome to assess the treatment effect in the subpopulation of patients who discontinued due to adverse events using a synthetic dataset. Finally, we explore the potential usage and provide interpretation of such analyses in clinical settings, as well as possible extension of such models in more general cases.

Effect of a Community Health Worker-Delivered Personal Sound Amplification Device on Self-Perceived Communication Function in Older Adults With Hearing Loss: A Randomized Clinical Trial

IMPORTANCE

Age-related hearing loss that impairs daily communication is associated with adverse health outcomes, but use of hearing aids by older adults is low and disparities exist.

OBJECTIVE

To test whether an affordable, accessible hearing care intervention, delivered by community health workers using over-the-counter hearing technology, could improve self-perceived communication function among older adults with hearing loss compared with a wait-list control.

DESIGN, SETTING, AND PARTICIPANTS

Open-label randomized clinical trial conducted between April 2018 and October 2019 with 3-month data collection completed in June 2020. The trial took place at 13 community sites, including affordable independent housing complexes (n = 10), senior centers (n = 2), and an older adult social club (n = 1) in Baltimore, Maryland. A total of 151 participants aged 60 years or older with hearing loss were randomized.

INTERVENTIONS

Participants were randomized to receive a community health worker-delivered hearing care intervention (n = 78) or to a wait-list control group (n = 73). The 2-hour intervention consisted of fitting a low-cost amplification device and instruction.

MAIN OUTCOMES AND MEASURES

The primary outcome was change in self-perceived communication function (Hearing Handicap Inventory for the Elderly-Screening Version [HHIE-S]; score range, 0-40; higher scores indicate poorer function) from baseline to 3 months postrandomization. The average treatment effect was estimated using the doubly robust weighted least squares estimator, which uses an outcome regression model weighted by the inverse probability of attrition to account for baseline covariate imbalance and missing data.

RESULTS

Among 151 participants randomized (mean age, 76.7 [SD, 8.0] years; 101 [67.8%] women; 65 [43%] self-identified as African American; 96 [63.6%] with low income [<$25 000 annual household income]), 136 (90.1%) completed 3-month follow-up for the primary outcome. In the intervention group, 90.5% completed the intervention session and reported at least 1 hour of daily amplification use at 3 months postrandomization. Mean scores for the HHIE-S were 21.7 (SD, 9.4) at baseline and 7.9 (SD, 9.2) at 3 months (change of -13.2 [SD, 10.3]) in the intervention group, and 20.1 (SD, 10.1) at baseline and 21 (SD, 9.1) at 3 months (change of 0.6 [SD, 7.1]) in the control group. Self-perceived communication function significantly improved in the intervention group compared with the control group, with an estimated average treatment effect of the intervention of a -12.98-point HHIE-S change (95% CI, -15.51 to -10.42). No study-related adverse events were reported.

CONCLUSIONS AND RELEVANCE

Among older adults with hearing loss, a community health worker-delivered personal sound amplification device intervention, compared with a wait-list control, significantly improved self-perceived communication function at 3 months. Findings are limited by the absence of a sham control, and further research is needed to understand effectiveness compared with other types of care delivery models and amplification devices.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT03442296.

Handling parametric assumptions in principal causal effect estimation using Gaussian mixtures

Given the latent stratum membership, principal stratification models with continuous outcomes naturally fit in the parametric estimation framework of Gaussian mixtures. However, with models that are not nonparametrically identified, relying on parametric mixture modeling has been mostly discouraged as a way of identifying principal effects. This study revisits this rather deserted use of parametric mixture modeling, which may open up various possibilities in principal stratification modeling. The main problem with using the parametric mixture modeling approach is that it is hard to assess the quality of principal effect estimates given its reliance on parametric conditions. As a way of assessing the estimation quality in this situation, this study proposes that we use parametric mixture modeling in two different ways, with and without the assurance of nonparametric identification. The key identifying assumption employed in this study is the moving exclusion restriction, a flexible version of the standard exclusion restriction assumption. This assumption is used as a temporary vehicle to help assess the quality of principal effect estimates obtained relying on parametric mixture modeling. The study presents promising results, showing the possibility of using parametric mixture modeling as an accessible tool for causal inference.

Principal stratum strategy: Potential role in drug development

A randomized trial allows estimation of the causal effect of an intervention compared to a control in the overall population and in subpopulations defined by baseline characteristics. Often, however, clinical questions also arise regarding the treatment effect in subpopulations of patients, which would experience clinical or disease related events post-randomization. Events that occur after treatment initiation and potentially affect the interpretation or the existence of the measurements are called intercurrent events in the ICH E9(R1) guideline. If the intercurrent event is a consequence of treatment, randomization alone is no longer sufficient to meaningfully estimate the treatment effect. Analyses comparing the subgroups of patients without the intercurrent events for intervention and control will not estimate a causal effect. This is well known, but post-hoc analyses of this kind are commonly performed in drug development. An alternative approach is the principal stratum strategy, which classifies subjects according to their potential occurrence of an intercurrent event on both study arms. We illustrate with examples that questions formulated through principal strata occur naturally in drug development and argue that approaching these questions with the ICH E9(R1) estimand framework has the potential to lead to more transparent assumptions as well as more adequate analyses and conclusions. In addition, we provide an overview of assumptions required for estimation of effects in principal strata. Most of these assumptions are unverifiable and should hence be based on solid scientific understanding. Sensitivity analyses are needed to assess robustness of conclusions.

On-treatment serum albumin level can guide long-term treatment in patients with cirrhosis and uncomplicated ascites

BACKGROUND & AIMS

The ANSWER study reported that long-term albumin administration in patients with cirrhosis and uncomplicated ascites improves survival. During treatment, serum albumin increased within a month and remained stable thereafter. In this post hoc analysis, we aimed to determine whether on-treatment serum albumin levels could guide therapy.

METHODS

Logistic regression was used to assess the association between baseline serum albumin and mortality, as well as to determine on-treatment factors associated with mortality and to predict the achievement of a given on-treatment serum albumin level. Survival was assessed by Kaplan-Meier estimates and second-order polynomial regression. Patients whose on-treatment serum albumin remained below normal were compared with a subset of patients from the control arm matched by principal score.

RESULTS

Baseline serum albumin was closely associated with 18-month mortality in untreated patients; albumin treatment almost effaced this relationship. On-treatment serum albumin and MELD-Na at month 1 were the sole independent variables associated with mortality. Second-order polynomial regression revealed that survival improved in parallel with increased 1-month on-treatment serum albumin. Kaplan-Meier estimations showed that any value of 1-month on-treatment serum albumin (0.1 g/dl intervals) in the range 2.5-4.5 g/dl discriminated patient survival. In the normal range of serum albumin, the best discriminant value was 4.0 g/dl. Compared to untreated patients, survival even improved in patients whose on-treatment serum albumin remained below normal.

CONCLUSION

Baseline serum albumin per se should not guide the decision to start albumin therapy. Conversely, 1-month on-treatment serum albumin levels are strongly associated with outcomes and could guide the use of albumin - 4.0 g/dl being the target threshold. However, even patients whose serum albumin remains below normal benefit from long-term albumin administration.

LAY SUMMARY

The ANSWER study has shown that long-term albumin administration improves survival and prevents the occurrence of major complications in patients with cirrhosis and ascites. This study shows that the achievement of these beneficial effects is related to a significant increase in serum albumin concentration. Even though the best results follow the achievement of a serum albumin concentration of 4 g/dl, a survival benefit is also achieved in patients who fail to normalise serum albumin.

Estimating causal effects of internet interventions in the context of nonadherence

A substantial proportion of participants who are offered internet-based psychological treatments in randomized trials do not adhere and may therefore not receive treatment. Despite the availability of justified statistical methods for causal inference in such situations, researchers often rely on analytical strategies that either ignore adherence altogether or fail to provide causal estimands. The objective of this paper is to provide a gentle nontechnical introduction to complier average causal effect (CACE) analysis, which, under clear assumptions, can provide a causal estimate of the effect of treatment for a subsample of compliers. The article begins with a brief review of the potential outcome model for causal inference. After clarifying assumptions and model specifications for CACE in the latent variable framework, data from a previously published trial of an internet-based psychological treatment for irritable bowel syndrome are used to demonstrate CACE-analysis. Several model extensions are then briefly reviewed. The paper offers practical recommendations on how to analyze randomized trials of internet interventions in the context of nonadherence. It is argued that CACE-analysis, whenever it is considered appropriate, should be carried out as a complement to the standard intention-to-treat analysis and that the format of internet-based treatments is particularly well suited to such an analytical approach.

Post-randomization Biomarker Effect Modification Analysis in an HIV Vaccine Clinical Trial

While the HVTN 505 trial showed no overall efficacy of the tested vaccine to prevent HIV infection over placebo, markers measuring immune response to vaccination were strongly correlated with infection. This finding generated the hypothesis that some marker-defined vaccinated subgroups were partially protected whereas others had their risk increased. This hypothesis can be assessed using the principal stratification framework (Frangakis and Rubin, 2002) for studying treatment effect modification by an intermediate response variable, using methods in the sub-field of principal surrogate (PS) analysis that studies multiple principal strata. Unfortunately, available methods for PS analysis require an augmented study design not available in HVTN 505, and make untestable structural risk assumptions, motivating a need for more robust PS methods. Fortunately, another sub-field of principal stratification, survivor average causal effect (SACE) analysis (Rubin, 2006) - which studies effects in a single principal stratum - provides many methods not requiring an augmented design and making fewer assumptions. We show how, for a binary intermediate response variable, methods developed for SACE analysis can be adapted to PS analysis, providing new and more robust PS methods. Application to HVTN 505 supports that the vaccine partially protected individuals with vaccine-induced T-cells expressing certain combinations of functions.

Assessing the Impact of COVID-19 on the Clinical Trial Objective and Analysis of Oncology Clinical Trials-Application of the Estimand Framework

Abstract-Coronavirus disease 2019 (COVID-19) outbreak has rapidly evolved into a global pandemic. The impact of COVID-19 on patient journeys in oncology represents a new risk to interpretation of trial results and its broad applicability for future clinical practice. We identify key intercurrent events (ICEs) that may occur due to COVID-19 in oncology clinical trials with a focus on time-to-event endpoints and discuss considerations pertaining to the other estimand attributes introduced in the ICH E9 addendum. We propose strategies to handle COVID-19 related ICEs, depending on their relationship with malignancy and treatment and the interpretability of data after them. We argue that the clinical trial objective from a world without COVID-19 pandemic remains valid. The estimand framework provides a common language to discuss the impact of COVID-19 in a structured and transparent manner. This demonstrates that the applicability of the framework may even go beyond what it was initially intended for.

Paracetamol is ineffective for acute low back pain even for patients who comply with treatment: complier average causal effect analysis of a randomized controlled trial

In 2014, the Paracetamol for Acute Low Back Pain (PACE) trial demonstrated that paracetamol had no effect compared with placebo in acute low back pain (LBP). However, noncompliance was a potential limitation of this trial. The aim of this study was to investigate the efficacy of paracetamol in acute LBP among compliers. Using individual participant data from the PACE trial (ACTN12609000966291), complier average causal effect (CACE), intention-to-treat, and per protocol estimates were calculated for pain intensity (primary), disability, global rating of symptom change, and function (all secondary) after 2 weeks of follow-up. Compliance was defined as intake of an average of at least 4 of the prescribed 6 tablets of regular paracetamol per day (2660 mg in total) during the first 2 weeks after enrolment. Exploratory analyses using alternative time points and definitions of compliance were conducted. Mean between-group differences in pain intensity on a 0 to 10 scale using the primary time point and definition of compliance were not clinically relevant (propensity-weighted CACE 0.07 [-0.37 to 0.50] P = 0.76; joint modelling CACE 0.23 [-0.16 to 0.62] P = 0.24; intention-to-treat 0.11 [-0.20 to 0.42] P = 0.49; per protocol 0.29 [-0.07 to 0.65] P = 0.12); results for secondary outcomes and for exploratory analyses were similar. Paracetamol is ineffective for acute LBP even for patients who comply with treatment. This reinforces the notion that management of acute LBP should focus on providing patients advice and reassurance without the addition of paracetamol.

Analyzing patient-reported outcome data when completion differs between arms in open-label trials: an application of principal stratification

PURPOSE

Cancer trials are often open-label and include patient-reported outcomes (PROs). Previous work has demonstrated that patients may complete PRO assessments less frequently in the control arm compared with the experimental arm in open-label trials. Such differential completion may affect PRO results. This paper sought to explore principal stratification methodology to address potential bias caused by the posttreatment intermediate variable of questionnaire completion.

METHODS

We evaluated six randomized trials (five open-label and one double-blind) of anticancer therapies with varying levels of PRO completion submitted to the Food and Drug Administration (FDA). We applied complete case analysis (CCA), multiple imputation (MI), and principal stratification to evaluate PRO results for quality of life (QOL) and the domains of physical, role, and emotional function (PF, RF, and EF). Assignment to potential principal strata was by the expectation maximization algorithm using patient baseline characteristics.

RESULTS

Completion rates in the experimental arm ranged from 66% to 94% and 51% to 95% in the control arm. Four trials had negligible completion differences between arms (1%-2%), and two had large differences favoring the experimental arm (15%-17%). For trials with negligible completion differences, principal stratification results were similar to CCA and MI results for all domains. Notable differences in point estimates may be observed in trials with large differences in completion rates. However, in the examined trials, the confidence intervals for the principal stratification estimates overlapped with the ones obtained using CCA.

CONCLUSIONS

The principal stratification estimand may be a useful additional analysis, especially if PRO completion differs between arms.

Acute and long-term cannabis use among stimulant users: Results from CTN-0037 Stimulant Reduction Intervention using Dosed Exercise (STRIDE) Randomized Control Trial

AIMS

The aim of this study was to examine the impact of vigorous intensity, high dose exercise (DEI) on cannabis use among stimulant users compared to a health education intervention (HEI) using data from the Stimulant Reduction Intervention using Dosed Exercise, National Institute of Drug Abuse National Drug Treatment Clinical Trials Network Protocol Number 0037 (STRIDE).

METHODS

Adults (N = 302) enrolled in the STRIDE randomized clinical trial were randomized to either the DEI or the HEI. Interventions included supervised sessions three times a week during the Acute phase (12 weeks) and once a week during the Follow-up phase (6 months). Cannabis use was measured at each assessment via Timeline Follow Back and urine drug screens. Cannabis use was compared between the groups during the Acute and Follow-up phases using both the intent-to-treat sample and a complier average causal effects (CACE) analysis.

FINDINGS

Approximately 43% of the sample reported cannabis use at baseline. The difference in cannabis use between the DEI and HEI groups during the Acute phase was not significant. During the Follow-up phase, the days of cannabis use was significantly lower among those in the DEI group (1.20 days) compared to the HEI group (2.15 days; p = 0.04).

CONCLUSIONS

For those who adhered to the exercise intervention, vigorous intensity, high dose exercise resulted in less cannabis use. Results suggest that there were no significant short-term differences in cannabis use between the groups. Further study on the long-term impact of exercise as a treatment to reduce cannabis use should be considered.

Impact of Availability of Telehealth Programs on Documented HIV Viral Suppression: A Cluster-Randomized Program Evaluation in the Veterans Health Administration

Background

Telehealth may improve care for people with HIV who live far from HIV specialty clinics. We conducted a cluster-randomized evaluation to determine the impact of availability of HIV telehealth programs on documented viral suppression in Veterans Administration clinics.

Methods

In 2015-2016, people who previously traveled to HIV specialty clinics were offered telehealth visits in nearby primary care clinics. Patients were cluster-randomized to immediate telehealth availability (n = 925 patients in service areas of 13 primary care clinics offering telehealth) or availability 1 year later (n = 745 patients in 12 clinics). Measures during the evaluation year included telehealth use among patients in areas where telehealth was available and documented HIV viral suppression (viral load performed and <200 copies/mL). Impact of telehealth availability was determined using intention-to-treat (ITT) analyses that compared outcomes for patients in areas where telehealth was available with outcomes for patients where telehealth was not available, regardless of telehealth use. Complier average causal effects (CACEs) compared outcomes for telehealth users with outcomes for control patients with equal propensity to use telehealth, when available.

Results

Overall, 120 (13.0%) patients utilized telehealth when it was available. Availability of telehealth programs led to small improvements in viral suppression in ITT analyses (78.3% vs 74.1%; relative risk [RR], 1.06; 95% confidence interval [CI], 1.01 to 1.11) and large improvements among telehealth users in CACE analyses (91.5% vs 80.0%; RR, 1.14; 95% CI, 1.01 to 1.30).

Conclusions

Availability of telehealth programs improved documented viral suppression. HIV clinics should offer telehealth visits for patients facing travel burdens.

Ensuring Causal, Not Casual, Inference

With innovation in causal inference methods and a rise in non-experimental data availability, a growing number of prevention researchers and advocates are thinking about causal inference. In this commentary, we discuss the current state of science as it relates to causal inference in prevention research, and reflect on key assumptions of these methods. We review challenges associated with the use of causal inference methodology, as well as considerations for hoping to integrate causal inference methods into their research. In short, this commentary addresses the key concepts of causal inference and suggests a greater emphasis on thoughtfully designed studies (to avoid the need for strong and potentially untestable assumptions) combined with analyses of sensitivity to those assumptions.

A Complier Average Causal Effect Analysis of the Stimulant Reduction Intervention using Dosed Exercise Study

Objective

Exercise is a promising treatment for substance use disorders, yet an intention-to-treat analysis of a large, multi-site study found no reduction in stimulant use for exercise versus health education. Exercise adherence was sub-optimal; therefore, secondary post-hoc complier average causal effects (CACE) analysis was conducted to determine the potential effectiveness of adequately dosed exercise.

Method

The STimulant use Reduction Intervention using Dosed Exercise study was a randomized controlled trial comparing a 12 kcal/kg/week (KKW) exercise dose versus a health education control conducted at nine residential substance use treatment settings across the U.S. that are affiliated with the National Drug Abuse Treatment Clinical Trials Network. Participants were sedentary but medically approved for exercise, used stimulants within 30 days prior to study entry, and received a DSM-IV stimulant abuse or dependence diagnosis within the past year. A CACE analysis adjusted to include only participants with a minimum threshold of adherence (at least 8.3 KKW) and using a negative-binomial hurdle model focused on 218 participants who were 36.2% female, mean age 39.4 years (SD = 11.1), and averaged 13.0 (SD = 9.2) stimulant use days in the 30 days before residential treatment. The outcome was days of stimulant use as assessed by the self-reported TimeLine Follow Back and urine drug screen results.

Results

The CACE-adjusted analysis found a significantly lower probability of relapse to stimulant use in the exercise group versus the health education group (41.0% vs. 55.7%, p < .01) and significantly lower days of stimulant use among those who relapsed (5.0 days vs. 9.9 days, p < .01).

Conclusions

The CACE adjustment revealed significant, positive effects for exercise. Further research is warranted to develop strategies for exercise adherence that can ensure achievement of an exercise dose sufficient to produce a significant treatment effect.

Reporting non-adherence in cluster randomised trials: A systematic review

BACKGROUND

Treatment non-adherence in randomised trials refers to situations where some participants do not receive their allocated treatment as intended. For cluster randomised trials, where the unit of randomisation is a group of participants, non-adherence may occur at the cluster or individual level. When non-adherence occurs, randomisation no longer guarantees that the relationship between treatment receipt and outcome is unconfounded, and the power to detect the treatment effects in intention-to-treat analysis may be reduced. Thus, recording adherence and estimating the causal treatment effect adequately are of interest for clinical trials.

OBJECTIVES

To assess the extent of reporting of non-adherence issues in published cluster trials and to establish which methods are currently being used for addressing non-adherence, if any, and whether clustering is accounted for in these.

METHODS

We systematically reviewed 132 cluster trials published in English in 2011 previously identified through a search in PubMed.

RESULTS

One-hundred and twenty three cluster trials were included in this systematic review. Non-adherence was reported in 56 cluster trials. Among these, 19 reported a treatment efficacy estimate: per protocol in 15 and as treated in 4. No study discussed the assumptions made by these methods, their plausibility or the sensitivity of the results to deviations from these assumptions.

LIMITATIONS

The year of publication of the cluster trials included in this review (2011) could be considered a limitation of this study; however, no new guidelines regarding the reporting and the handling of non-adherence for cluster trials have been published since. In addition, a single reviewer undertook the data extraction. To mitigate this, a second reviewer conducted a validation of the extraction process on 15 randomly selected reports. Agreement was satisfactory (93%).

CONCLUSION

Despite the recommendations of the Consolidated Standards of Reporting Trials statement extension to cluster randomised trials, treatment adherence is under-reported. Among the trials providing adherence information, there was substantial variation in how adherence was defined, handled and reported. Researchers should discuss the assumptions required for the results to be interpreted causally and whether these are scientifically plausible in their studies. Sensitivity analyses to study the robustness of the results to departures from these assumptions should be performed.

Measuring Model Misspecification: Application to Propensity Score Methods with Complex Survey Data

Model misspecification is a potential problem for any parametric-model based analysis. However, the measurement and consequences of model misspecification have not been well formalized in the context of causal inference. A measure of model misspecification is proposed, and the consequences of model misspecification in non-experimental causal inference methods are investigated. The metric is then used to explore which estimators are more sensitive to misspecification of the outcome and/or treatment assignment model. Three frequently used estimators of the treatment effect are considered, all of which rely on the propensity score: (1) full matching, (2) 1:1 nearest neighbor matching, and (3) weighting. The performance of these estimators is evaluated under two different sampling designs: (1) simple random sampling (SRS) and (2) a two-stage stratified survey. As the degree of misspecification of either the propensity score or outcome model increases, so does the bias and the root mean square error, while the coverage decreases. Results are similar for the simple random sample and a complex survey design.

Causal mediation analysis with multiple mediators in the presence of treatment noncompliance

Randomized experiments are often complicated because of treatment noncompliance. This challenge prevents researchers from identifying the mediated portion of the intention-to-treated (ITT) effect, which is the effect of the assigned treatment that is attributed to a mediator. One solution suggests identifying the mediated ITT effect on the basis of the average causal mediation effect among compliers when there is a single mediator. However, considering the complex nature of the mediating mechanisms, it is natural to assume that there are multiple variables that mediate through the causal path. Motivated by an empirical analysis of a data set collected in a randomized interventional study, we develop a method to estimate the mediated portion of the ITT effect when both multiple dependent mediators and treatment noncompliance exist. This enables researchers to make an informed decision on how to strengthen the intervention effect by identifying relevant mediators despite treatment noncompliance. We propose a nonparametric estimation procedure and provide a sensitivity analysis for key assumptions. We conduct a Monte Carlo simulation study to assess the finite sample performance of the proposed approach. The proposed method is illustrated by an empirical analysis of JOBS II data, in which a job training intervention was used to prevent mental health deterioration among unemployed individuals.

Racial and ethnic differences in treatment outcomes among adults with stimulant use disorders after a dosed exercise intervention

The current study examined differences in substance abuse treatment outcomes among racial and ethnic groups enrolled in the Stimulant Reduction Intervention using Dosed Exercise (STRIDE) trial, a multisite randomized clinical trial implemented through the National Institute on Drug Abuse's (NIDA's) Clinical Trials Network (CTN). STRIDE aimed to test vigorous exercise as a novel approach to the treatment of stimulant abuse compared to a health education intervention. A hurdle model with a complier average causal effects (CACE) adjustment was used to provide an unbiased estimate of the exercise effect had all participants been adherent to exercise. Among 214 exercise-adherent participants, we found significantly lower probability of use for Blacks (z = -2.45, p = .014) and significantly lower number of days of use for Whites compared to Hispanics (z = -54.87, p = <.001) and for Whites compared to Blacks (z = -28.54, p = <.001), which suggests that vigorous, regular exercise might improve treatment outcomes given adequate levels of adherence.

Racial and Ethnic Differences in the Delivery of the Resources for Enhancing Alzheimer's Caregiver Health II Intervention

OBJECTIVES

To determine whether there are racial and ethnicity group differences in Resources for Enhancing Alzheimer's Caregiver Health (REACH II) intervention delivery.

DESIGN

Randomized controlled trial.

SETTING

Community-based intervention delivered at five sites across the United States.

PARTICIPANTS

Family caregivers of persons with dementia who were randomized to the active intervention condition (N = 323).

INTERVENTION

Nine in-home sessions (90 minutes each) and three telephone sessions (30 minutes each) were intended to be delivered and designed to reduce caregiver burden and depression, improve caregiver self-care and social support, and help caregivers manage behavior problems in persons with dementia.

MEASUREMENTS

Interventionists recorded the type of intervention (home or telephone), start and stop times, and whether specific intervention content modules (e.g., stress management, social support) were administered in each session.

RESULTS

Overall, REACH II intervention delivery was high, with more than 80% of randomized caregivers completing at least five in-home sessions and receiving eight or more hours of intervention contact, but black caregivers completed fewer in-home sessions (mean 6.98) than Hispanics (mean 7.84) or whites (mean 8.25) and received less total intervention contact time (mean 683 minutes) than Hispanics (mean 842 minutes) or whites (mean 798 minutes). No significant differences in exposure to content according to race or ethnicity were found after controlling for demographic covariates.

CONCLUSION

Blacks in REACH II received significantly less intervention contact. Similar multicomponent interventions should examine whether there are systematic differences in intervention delivery across specific demographic subgroups and explore implications for treatment outcomes.

Inference for environmental intervention studies using principal stratification

Previous research has found evidence of an association between indoor air pollution and asthma morbidity in children. Environmental intervention studies have been performed to examine the role of household environmental interventions in altering indoor air pollution concentrations and improving health. Previous environmental intervention studies have found only modest effects on health outcomes and it is unclear if the health benefits provided by environmental modification are comparable with those provided by medication. Traditionally, the statistical analysis of environmental intervention studies has involved performing two intention-to-treat analyses that separately estimate the effect of the environmental intervention on health and the effect of the environmental intervention on indoor air pollution concentrations. We propose a principal stratification approach to examine the extent to which an environmental intervention's effect on health outcomes coincides with its effect on indoor air pollution. We apply this approach to data from a randomized air cleaner intervention trial conducted in a population of asthmatic children living in Baltimore, Maryland, USA. We find that among children for whom the air cleaner reduced indoor particulate matter concentrations, the intervention resulted in a meaningful improvement of asthma symptoms with an effect generally larger than previous studies have shown. A key benefit of using principal stratification in environmental intervention studies is that it allows investigators to estimate causal effects of the intervention for sub-groups defined by changes in the indoor air pollution concentration.

The Use of Propensity Scores in Mediation Analysis

Mediation analysis uses measures of hypothesized mediating variables to test theory for how a treatment achieves effects on outcomes and to improve subsequent treatments by identifying the most efficient treatment components. Most current mediation analysis methods rely on untested distributional and functional form assumptions for valid conclusions, especially regarding the relation between the mediator and outcome variables. Propensity score methods offer an alternative whereby the propensity score is used to compare individuals in the treatment and control groups who would have had the same value of the mediator had they been assigned to the same treatment condition. This article describes the use of propensity score weighting for mediation with a focus on explicating the underlying assumptions. Propensity scores have the potential to offer an alternative estimation procedure for mediation analysis with alternative assumptions from those of standard mediation analysis. The methods are illustrated investigating the mediational effects of an intervention to improve sense of mastery to reduce depression using data from the Job Search Intervention Study (JOBS II). We find significant treatment effects for those individuals who would have improved sense of mastery when in the treatment condition but no effects for those who would not have improved sense of mastery under treatment.