Structural mean models for compliance analysis in randomized clinical trials and the impact of errors on measures of exposure

The Importance of Assessing Drug Exposure and Medication Adherence in Evaluating Investigational Medications: Ensuring Validity and Reliability of Clinical Trial Results

The objective of this current opinion paper is to draw global attention to medication adherence, emphasizing its crucial role in drug trials. Frequently, trialists lean on traditional approaches to assess medication adherence, which, while comfortable, may only reveal what trialists desire rather than offering the essential insights needed for informed decision making in drug development. Understanding drug exposure and medication adherence is paramount when evaluating the effectiveness and safety of investigational medications. Without a comprehensive understanding of how patients adhere to their prescribed treatment regimens, the integrity and dependability of clinical trial results can be compromised. This paper emphasizes the need for measures that accurately and reliably assess medication intake behaviors, enabling the differentiation between minor dosing errors and significant deviations that may impact the drug's efficacy and safety. Accurate knowledge of drug exposure empowers researchers to make informed decisions, identify potential confounding factors, and appropriately interpret study outcomes, ultimately ensuring the validity and reliability of the research findings. By prioritizing drug exposure assessment and medication adherence measurement, clinical trials can enhance their scientific rigor, contribute to more accurate evaluations of investigational medications, and ultimately speed up the development process.

A systematic review of simulation studies which compare existing statistical methods to account for non-compliance in randomised controlled trials

INTRODUCTION

Non-compliance is a common challenge for researchers and may reduce the power of an intention-to-treat analysis. Whilst a per protocol approach attempts to deal with this issue, it can result in biased estimates. Several methods to resolve this issue have been identified in previous reviews, but there is limited evidence supporting their use. This review aimed to identify simulation studies which compare such methods, assess the extent to which certain methods have been investigated and determine their performance under various scenarios.

METHODS

A systematic search of several electronic databases including MEDLINE and Scopus was carried out from conception to 30th November 2022. Included papers were published in a peer-reviewed journal, readily available in the English language and focused on comparing relevant methods in a superiority randomised controlled trial under a simulation study. Articles were screened using these criteria and a predetermined extraction form used to identify relevant information. A quality assessment appraised the risk of bias in individual studies. Extracted data was synthesised using tables, figures and a narrative summary. Both screening and data extraction were performed by two independent reviewers with disagreements resolved by consensus.

RESULTS

Of 2325 papers identified, 267 full texts were screened and 17 studies finally included. Twelve methods were identified across papers. Instrumental variable methods were commonly considered, but many authors found them to be biased in some settings. Non-compliance was generally assumed to be all-or-nothing and only occurring in the intervention group, although some methods considered it as time-varying. Simulation studies commonly varied the level and type of non-compliance and factors such as effect size and strength of confounding. The quality of papers was generally good, although some lacked detail and justification. Therefore, their conclusions were deemed to be less reliable.

CONCLUSIONS

It is common for papers to consider instrumental variable methods but more studies are needed that consider G-methods and compare a wide range of methods in realistic scenarios. It is difficult to make conclusions about the best method to deal with non-compliance due to a limited body of evidence and the difficulty in combining results from independent simulation studies.

PROSPERO REGISTRATION NUMBER

CRD42022370910.

Identification of causal effects with latent confounding and classical additive errors in treatment

In this paper, we discuss the identifiability and estimation of causal effects of a continuous treatment on a binary response when the treatment is measured with errors and there exists a latent categorical confounder associated with both treatment and response. Under some widely used parametric models, we first discuss the identifiability of the causal effects and then propose an approach for estimation and inference. Our approach can eliminate the biases induced by latent confounding and measurement errors by using only a single instrumental variable. Based on the identification results, we give guidelines for determining the existence of a latent categorical confounder and for selecting the number of levels of the latent confounder. We apply the proposed approach to a data set from the Framingham Heart Study to evaluate the effect of the systolic blood pressure on the coronary heart disease.

PROPEL: implementation of an evidence based pelvic floor muscle training intervention for women with pelvic organ prolapse: a realist evaluation and outcomes study protocol

Background

Pelvic Organ Prolapse (POP) is estimated to affect 41%–50% of women aged over 40. Findings from the multi-centre randomised controlled “Pelvic Organ Prolapse PhysiotherapY” (POPPY) trial showed that individualised pelvic floor muscle training (PFMT) was effective in reducing symptoms of prolapse, improved quality of life and showed clear potential to be cost-effective. However, provision of PFMT for prolapse continues to vary across the UK, with limited numbers of women’s health physiotherapists specialising in its delivery. Implementation of this robust evidence from the POPPY trial will require attention to different models of delivery (e.g. staff skill mix) to fit with differing care environments.

Methods

A Realist Evaluation (RE) of implementation and outcomes of PFMT delivery in contrasting NHS settings will be conducted using multiple case study sites. Involving substantial local stakeholder engagement will permit a detailed exploration of how local sites make decisions on how to deliver PFMT and how these lead to service change. The RE will track how implementation is working; identify what influences outcomes; and, guided by the RE-AIM framework, will collect robust outcomes data. This will require mixed methods data collection and analysis.

Qualitative data will be collected at four time-points across each site to understand local contexts and decisions regarding options for intervention delivery and to monitor implementation, uptake, adherence and outcomes. Patient outcome data will be collected at baseline, six months and one year follow-up for 120 women. Primary outcome will be the Pelvic Organ Prolapse Symptom Score (POP-SS). An economic evaluation will assess the costs and benefits associated with different delivery models taking account of further health care resource use by the women. Cost data will be combined with the primary outcome in a cost effectiveness analysis, and the EQ-5D-5L data in a cost utility analysis for each of the different models of delivery.

Discussion

Study of the implementation of varying models of service delivery of PFMT across contrasting sites combined with outcomes data and a cost effectiveness analysis will provide insight into the implementation and value of different models of PFMT service delivery and the cost benefits to the NHS in the longer term.

A simulation study on implementing marginal structural models in an observational study with switching medication based on a biomarker

Assessing treatment effectiveness in longitudinal data can be complex when treatments are not randomly assigned and patients are allowed to switch treatment to other or no treatment, often in a manner that is driven by changes in one or more variables associated with patient or clinical characteristics. There can be confounding of the treatment effect from a time-varying variable, i.e., one which is affected by previous exposure and can in turn also influence subsequent treatment changes. Precision medicine relies on validated biomarkers to better classify patients by their probable response to treatment. However, biomarkers may be a source of time-varying confounding, which are affected by prior treatment in the evaluation and are also subject to measurement errors. The impact of switching medications based on a biomarker has received less attention. We conducted simulation studies to explore biased estimation under various scenarios when marginal structural model estimations are employed. Holding model misspecification issues constant, bias is severe in the presence of multiple switching, along with measurement error and missing data in the covariates.

A doubly robust estimator for the average treatment effect in the context of a mean-reverting measurement error

One of the main limitations of causal inference methods is that they rely on the assumption that all variables are measured without error. A popular approach for handling measurement error is simulation-extrapolation (SIMEX). However, its use for estimating causal effects have been examined only in the context of an additive, non-differential, and homoscedastic classical measurement error structure. In this article we extend the SIMEX methodology, in the context of a mean reverting measurement error structure, to a doubly robust estimator of the average treatment effect when a single covariate is measured with error but the outcome and treatment and treatment indicator are not. Throughout this article we assume that an independent validation sample is available. Simulation studies suggest that our method performs better than a naive approach that simply uses the covariate measured with error.

Measurement Error and Environmental Epidemiology: a Policy Perspective

PURPOSE OF REVIEW

Measurement error threatens public health by producing bias in estimates of the population impact of environmental exposures. Quantitative methods to account for measurement bias can improve public health decision making.

RECENT FINDINGS

We summarize traditional and emerging methods to improve inference under a standard perspective, in which the investigator estimates an exposure-response function, and a policy perspective, in which the investigator directly estimates population impact of a proposed intervention. Under a policy perspective, the analyst must be sensitive to errors in measurement of factors that modify the effect of exposure on outcome, must consider whether policies operate on the true or measured exposures, and may increasingly need to account for potentially dependent measurement error of two or more exposures affected by the same policy or intervention. Incorporating approaches to account for measurement error into such a policy perspective will increase the impact of environmental epidemiology.

An Illustration of Inverse Probability Weighting to Estimate Policy-Relevant Causal Effects

Traditional epidemiologic approaches allow us to compare counterfactual outcomes under 2 exposure distributions, usually 100% exposed and 100% unexposed. However, to estimate the population health effect of a proposed intervention, one may wish to compare factual outcomes under the observed exposure distribution to counterfactual outcomes under the exposure distribution produced by an intervention. Here, we used inverse probability weights to compare the 5-year mortality risk under observed antiretroviral therapy treatment plans to the 5-year mortality risk that would had been observed under an intervention in which all patients initiated therapy immediately upon entry into care among patients positive for human immunodeficiency virus in the US Centers for AIDS Research Network of Integrated Clinical Systems multisite cohort study between 1998 and 2013. Therapy-naïve patients (n = 14,700) were followed from entry into care until death, loss to follow-up, or censoring at 5 years or on December 31, 2013. The 5-year cumulative incidence of mortality was 11.65% under observed treatment plans and 10.10% under the intervention, yielding a risk difference of -1.57% (95% confidence interval: -3.08, -0.06). Comparing outcomes under the intervention with outcomes under observed treatment plans provides meaningful information about the potential consequences of new US guidelines to treat all patients with human immunodeficiency virus regardless of CD4 cell count under actual clinical conditions.

Evaluation and validation of social and psychological markers in randomised trials of complex interventions in mental health: a methodological research programme

BACKGROUND

The development of the capability and capacity to evaluate the outcomes of trials of complex interventions is a key priority of the National Institute for Health Research (NIHR) and the Medical Research Council (MRC). The evaluation of complex treatment programmes for mental illness (e.g. cognitive-behavioural therapy for depression or psychosis) not only is a vital component of this research in its own right but also provides a well-established model for the evaluation of complex interventions in other clinical areas. In the context of efficacy and mechanism evaluation (EME) there is a particular need for robust methods for making valid causal inference in explanatory analyses of the mechanisms of treatment-induced change in clinical outcomes in randomised clinical trials.

OBJECTIVES

The key objective was to produce statistical methods to enable trial investigators to make valid causal inferences about the mechanisms of treatment-induced change in these clinical outcomes. The primary objective of this report is to disseminate this methodology, aiming specifically at trial practitioners.

METHODS

The three components of the research were (1) the extension of instrumental variable (IV) methods to latent growth curve models and growth mixture models for repeated-measures data; (2) the development of designs and regression methods for parallel trials; and (3) the evaluation of the sensitivity/robustness of findings to the assumptions necessary for model identifiability. We illustrate our methods with applications from psychological and psychosocial intervention trials, keeping the technical details to a minimum, leaving the reporting of the more theoretical and mathematically demanding results for publication in appropriate specialist journals.

RESULTS

We show how to estimate treatment effects and introduce methods for EME. We explain the use of IV methods and principal stratification to evaluate the role of putative treatment effect mediators and therapeutic process measures. These results are extended to the analysis of longitudinal data structures. We consider the design of EME trials. We focus on designs to create convincing IVs, bearing in mind assumptions needed to attain model identifiability. A key area of application that has become apparent during this work is the potential role of treatment moderators (predictive markers) in the evaluation of treatment effect mechanisms for personalised therapies (stratified medicine). We consider the role of targeted therapies and multiarm trials and the use of parallel trials to help elucidate the evaluation of mediators working in parallel.

CONCLUSIONS

In order to demonstrate both efficacy and mechanism, it is necessary to (1) demonstrate a treatment effect on the primary (clinical) outcome, (2) demonstrate a treatment effect on the putative mediator (mechanism) and (3) demonstrate a causal effect from the mediator to the outcome. Appropriate regression models should be applied for (3) or alternative IV procedures, which account for unmeasured confounding, provided that a valid instrument can be identified. Stratified medicine may provide a setting where such instruments can be designed into the trial. This work could be extended by considering improved trial designs, sample size considerations and measurement properties.

FUNDING

The project presents independent research funded under the MRC-NIHR Methodology Research Programme (grant reference G0900678).

Multiple Imputation to Account for Measurement Error in Marginal Structural Models

BACKGROUND

Marginal structural models are an important tool for observational studies. These models typically assume that variables are measured without error. We describe a method to account for differential and nondifferential measurement error in a marginal structural model.

METHODS

We illustrate the method estimating the joint effects of antiretroviral therapy initiation and current smoking on all-cause mortality in a United States cohort of 12,290 patients with HIV followed for up to 5 years between 1998 and 2011. Smoking status was likely measured with error, but a subset of 3,686 patients who reported smoking status on separate questionnaires composed an internal validation subgroup. We compared a standard joint marginal structural model fit using inverse probability weights to a model that also accounted for misclassification of smoking status using multiple imputation.

RESULTS

In the standard analysis, current smoking was not associated with increased risk of mortality. After accounting for misclassification, current smoking without therapy was associated with increased mortality (hazard ratio [HR]: 1.2 [95% confidence interval [CI] = 0.6, 2.3]). The HR for current smoking and therapy [0.4 (95% CI = 0.2, 0.7)] was similar to the HR for no smoking and therapy (0.4; 95% CI = 0.2, 0.6).

CONCLUSIONS

Multiple imputation can be used to account for measurement error in concert with methods for causal inference to strengthen results from observational studies.

A Bayesian approach for instrumental variable analysis with censored time-to-event outcome

Instrumental variable (IV) analysis has been widely used in economics, epidemiology, and other fields to estimate the causal effects of covariates on outcomes, in the presence of unobserved confounders and/or measurement errors in covariates. However, IV methods for time-to-event outcome with censored data remain underdeveloped. This paper proposes a Bayesian approach for IV analysis with censored time-to-event outcome by using a two-stage linear model. A Markov chain Monte Carlo sampling method is developed for parameter estimation for both normal and non-normal linear models with elliptically contoured error distributions. The performance of our method is examined by simulation studies. Our method largely reduces bias and greatly improves coverage probability of the estimated causal effect, compared with the method that ignores the unobserved confounders and measurement errors. We illustrate our method on the Women's Health Initiative Observational Study and the Atherosclerosis Risk in Communities Study.

Model selection criterion for causal parameters in structural mean models based on a quasi-likelihood

Structural mean models (SMMs) have been proposed for estimating causal parameters in the presence of non-ignorable non-compliance in clinical trials. To obtain a valid causal estimate, we must impose several assumptions. One of these is the correct specification of the structural model. Building on Pan's work (2001, Biometrics 57, 120-125) on developing a model selection criterion for generalized estimating equations, we propose a new approach for model selection of SMMs based on a quasi-likelihood. We provide a formal model selection criterion that is an extension of Akaike's information criterion. Using subset selection of baseline covariates, our method allows us to understand whether the treatment effect varies across the available baseline covariate levels, and/or to quantify the treatment effect on a specific covariates level to target specific individuals to maximize treatment benefit. We present simulation results in which our method performs reasonably well compared to other testing methods in terms of both the probability of selecting the correct model and the predictive performances of the individual treatment effects. We use a large randomized clinical trial of pravastatin as a motivation.

Identification of causal effects on binary outcomes using structural mean models

Structural mean models (SMMs) were originally formulated to estimate causal effects among those selecting treatment in randomized controlled trials affected by nonignorable noncompliance. It has already been established that SMMs can identify these causal effects in randomized placebo-controlled trials under fairly weak assumptions. SMMs are now being used to analyze other types of study where identification depends on a no effect modification assumption. We highlight how this assumption depends crucially on the unknown causal model that generated the data, and so is difficult to justify. However, we also highlight that, if treatment selection is monotonic, additive and multiplicative SMMs do identify local (or complier) causal effects, but that the double-logistic SMM estimator does not without further assumptions. We clarify the proper interpretation of inferences from SMMs by means of an application and a simulation study.

Correlation between adherence rates measured by MEMS and self-reported questionnaires: a meta-analysis

Purpose

It is vital to understand the associations between the medication event monitoring systems (MEMS) and self-reported questionnaires (SRQs) because both are often used to measure medication adherence and can produce different results. In addition, the economic implication of using alternative measures is important as the cost of electronic monitoring devices is not covered by insurance, while self-reports are the most practical and cost-effective method in the clinical settings. This meta-analysis examined the correlations of two measurements of medication adherence: MEMS and SRQs.

Methods

The literature search (1980-2009) used PubMed, OVID MEDLINE, PsycINFO (EBSCO), CINAHL (EBSCO), OVID HealthStar, EMBASE (Elsevier), and Cochrane Databases. Studies were included if the correlation coefficients [Pearson (r_p) or Spearman (r_s)] between adherences measured by both MEMS and SRQs were available or could be calculated from other statistics in the articles. Data were independently abstracted in duplicate with standardized protocol and abstraction form including 1) first author's name; 2) year of publication; 3) disease status of participants; 4) sample size; 5) mean age (year); 6) duration of trials (month); 7) SRQ names if available; 8) adherence (%) measured by MEMS; 9) adherence (%) measured by SRQ; 10) correlation coefficient and relative information, including p-value, 95% confidence interval (CI). A meta-analysis was conducted to pool the correlation coefficients using random-effect model.

Results

Eleven studies (N = 1,684 patients) met the inclusion criteria. The mean of adherence measured by MEMS was 74.9% (range 53.4%-92.9%), versus 84.0% by SRQ (range 68.35%-95%). The correlation between adherence measured by MEMS and SRQs ranged from 0.24 to 0.87. The pooled correlation coefficient for 11 studies was 0.45 (p = 0.001, 95% confidence interval [95% CI]: 0.34-0.56). The subgroup meta-analysis on the seven studies reporting r_p and four studies reporting r_s reported the pooled correlation coefficient: 0.46 (p = 0.011, 95% CI: 0.33-0.59) and 0.43 (p = 0.0038, 95% CI: 0.23-0.64), respectively. No differences were found for other subgroup analyses.

Conclusion

Medication adherence measured by MEMS and SRQs tends to be at least moderately correlated, suggesting that SRQs give a good estimate of medication adherence.

Using marginal structural measurement-error models to estimate the long-term effect of antiretroviral therapy on incident AIDS or death

To estimate the net effect of imperfectly measured highly active antiretroviral therapy on incident acquired immunodeficiency syndrome or death, the authors combined inverse probability-of-treatment-and-censoring weighted estimation of a marginal structural Cox model with regression-calibration methods. Between 1995 and 2007, 950 human immunodeficiency virus-positive men and women were followed in 2 US cohort studies. During 4,054 person-years, 374 initiated highly active antiretroviral therapy, 211 developed acquired immunodeficiency syndrome or died, and 173 dropped out. Accounting for measured confounders and determinants of dropout, the weighted hazard ratio for acquired immunodeficiency syndrome or death comparing use of highly active antiretroviral therapy in the prior 2 years with no therapy was 0.36 (95% confidence limits: 0.21, 0.61). This association was relatively constant over follow-up (P = 0.19) and stronger than crude or adjusted hazard ratios of 0.75 and 0.95, respectively. Accounting for measurement error in reported exposure using external validation data on 331 men and women provided a hazard ratio of 0.17, with bias shifted from the hazard ratio to the estimate of precision as seen by the 2.5-fold wider confidence limits (95% confidence limits: 0.06, 0.43). Marginal structural measurement-error models can simultaneously account for 3 major sources of bias in epidemiologic research: validated exposure measurement error, measured selection bias, and measured time-fixed and time-varying confounding.

Related causal frameworks for surrogate outcomes

SUMMARY

Four major frameworks have been developed for evaluating surrogate markers in randomized trials: one based on conditional independence of observable variables, another based on direct and indirect effects, a third based on a meta-analysis, and a fourth based on principal stratification. The first two of these fit into a paradigm we call the causal-effects (CE) paradigm, in which, for a good surrogate, the effect of treatment on the surrogate, combined with the effect of the surrogate on the clinical outcome, allow prediction of the effect of the treatment on the clinical outcome. The last two approaches fall into the causal-association (CA) paradigm, in which the effect of the treatment on the surrogate is associated with its effect on the clinical outcome. We consider the CE paradigm first, and consider identifying assumptions and some simple estimation procedures; we then consider the CA paradigm. We examine the relationships among these approaches and associated estimators. We perform a small simulation study to illustrate properties of the various estimators under different scenarios, and conclude with a discussion of the applicability of both paradigms.

Mediation and moderation of treatment effects in randomised controlled trials of complex interventions

Complex intervention trials should be able to answer both pragmatic and explanatory questions in order to test the theories motivating the intervention and help understand the underlying nature of the clinical problem being tested. Key to this is the estimation of direct effects of treatment and indirect effects acting through intermediate variables which are measured post-randomisation. Using psychological treatment trials as an example of complex interventions, we review statistical methods which crucially evaluate both direct and indirect effects in the presence of hidden confounding between mediator and outcome. We review the historical literature on mediation and moderation of treatment effects. We introduce two methods from within the existing causal inference literature, principal stratification and structural mean models, and demonstrate how these can be applied in a mediation context before discussing approaches and assumptions necessary for attaining identifiability of key parameters of the basic causal model. Assuming that there is modification by baseline covariates of the effect of treatment (i.e. randomisation) on the mediator (i.e. covariate by treatment interactions), but no direct effect on the outcome of these treatment by covariate interactions leads to the use of instrumental variable methods. We describe how moderation can occur through post-randomisation variables, and extend the principal stratification approach to multiple group methods with explanatory models nested within the principal strata. We illustrate the new methodology with motivating examples of randomised trials from the mental health literature.

Instrumental variable analysis for estimation of treatment effects with dichotomous outcomes

Instrumental variable analyses are increasingly used in epidemiologic studies. For dichotomous exposures and outcomes, the typical 2-stage least squares approach produces risk difference estimates rather than relative risk estimates and is criticized for assuming normally distributed errors. Using 2 example drug safety studies evaluated in 3 cohorts from Pennsylvania (1994-2003) and British Columbia, Canada (1996-2004), the authors compared instrumental variable techniques that yield relative risk and risk difference estimates and that are appropriate for dichotomous exposures and outcomes. Methods considered include probit structural equation models, 2-stage logistic models, and generalized method of moments estimators. Employing these methods, in the first study the authors observed relative risks ranging from 0.41 to 0.58 and risk differences ranging from -1.41 per 100 to -1.28 per 100; in the second, they observed relative risks of 1.38-2.07 and risk differences of 7.53-8.94; and in the third, they observed relative risks of 1.45-1.59 and risk differences of 3.88-4.84. The 2-stage logistic models showed standard errors up to 40% larger than those of the instrumental variable probit model. Generalized method of moments estimation produced substantially the same results as the 2-stage logistic method. Few substantive differences among the methods were observed, despite their reliance on distinct assumptions.

Estimating dose-response effects in psychological treatment trials: the role of instrumental variables

We present a relatively non-technical and practically orientated review of statistical methods that can be used to estimate dose-response relationships in randomised controlled psychotherapy trials in which participants fail to attend all of the planned sessions of therapy. Here we are investigating the effects on treatment outcome of the number of sessions attended when the latter is possibly subject to hidden selection effects (hidden confounding). The aim is to estimate the parameters of a structural mean model (SMM) using randomisation, and possibly randomisation by covariate interactions, as instrumental variables. We describe, compare and illustrate the equivalence of the use of a simple G-estimation algorithm and two two-stage least squares procedures that are traditionally used in economics.

Modelling treatment-effect heterogeneity in randomized controlled trials of complex interventions (psychological treatments)

We describe instrumental variable (IV) methods for the estimation of the 'dose'-response effects of psychological interventions in randomized controlled trials in which there is variability in the patients' adherence to the allocated therapy (that is, variability in the actual number of sessions of therapy attended) and also variability in the strength of the therapeutic alliance between patients and their therapists. The effect of the therapy on outcome is assumed to be a function of both the number of sessions attended and the strength of the therapeutic alliance, with no intervention effects in the absence of any sessions attended (an exclusion restriction) and the effect of the strength of the alliance being represented by a multiplicative term (interaction) in the treatment-effect model. The IV methods that are described allow for: (a) hidden confounding between sessions, alliance and outcome; (b) measurement errors in the alliance; and (c) that alliance is only measured in those receiving treatment. Three two-stage estimation procedures are illustrated, and their equivalence demonstrated, through Monte Carlo simulation and analysis of the results of an actual trial.