Should a propensity score model be super? The utility of ensemble procedures for causal adjustment

Comparing methods for estimating causal treatment effects of administrative health data: A plasmode simulation study

Estimating the causal effects of health policy interventions is crucial for policymaking but is challenging when using real-world administrative health care data due to a lack of methodological guidance. To help fill this gap, we conducted a plasmode simulation using such data from a recent policy initiative launched in a deprived urban area in Germany. Our aim was to evaluate and compare the following methods for estimating causal effects: propensity score matching, inverse probability of treatment weighting, and entropy balancing, all combined with difference-in-differences analysis, augmented inverse probability weighting, and targeted maximum likelihood estimation. Additionally, we estimated nuisance parameters using regression models and an ensemble learner called superlearner. We focused on treatment effects related to the number of physician visits, total health care cost, and hospitalization. While each approach has its strengths and weaknesses, our results demonstrate that the superlearner generally worked well for handling nuisance terms in large covariate sets when combined with doubly robust estimation methods to estimate the causal contrast of interest. In contrast, regression-based nuisance parameter estimation worked best in small covariate sets when combined with singly robust methods.

Parametric and nonparametric propensity score estimation in multilevel observational studies

There has been growing interest in using nonparametric machine learning approaches for propensity score estimation in order to foster robustness against misspecification of the propensity score model. However, the vast majority of studies focused on single-level data settings, and research on nonparametric propensity score estimation in clustered data settings is scarce. In this article, we extend existing research by describing a general algorithm for incorporating random effects into a machine learning model, which we implemented for generalized boosted modeling (GBM). In a simulation study, we investigated the performance of logistic regression, GBM, and Bayesian additive regression trees for inverse probability of treatment weighting (IPW) when the data are clustered, the treatment exposure mechanism is nonlinear, and unmeasured cluster-level confounding is present. For each approach, we compared fixed and random effects propensity score models to single-level models and evaluated their use in both marginal and clustered IPW. We additionally investigated the performance of the standard Super Learner and the balance Super Learner. The results showed that when there was no unmeasured confounding, logistic regression resulted in moderate bias in both marginal and clustered IPW, whereas the nonparametric approaches were unbiased. In presence of cluster-level confounding, fixed and random effects models greatly reduced bias compared to single-level models in marginal IPW, with fixed effects GBM and fixed effects logistic regression performing best. Finally, clustered IPW was overall preferable to marginal IPW and the balance Super Learner outperformed the standard Super Learner, though neither worked as well as their best candidate model.

Tracheostomy versus prolonged intubation in moderate to severe traumatic brain injury: a multicentre retrospective cohort study

PURPOSE

Tracheostomy is a surgical procedure that is commonly performed in patients admitted to the intensive care unit (ICU). It is frequently required in patients with moderate to severe traumatic brain injury (TBI), a subset of patients with prolonged altered state of consciousness that may require a long period of mechanical respiratory assistance. While many clinicians favour the use of early tracheostomy in TBI patients, the evidence in favour of this practice remains scarce. The aims of our study were to evaluate the potential clinical benefits of tracheostomy versus prolonged endotracheal intubation, as well as whether the timing of the procedure may influence outcome in patients with moderate to severe TBI.

METHODS

We conducted a retrospective multicentre cohort study based on data from the provincial integrated trauma system of Quebec (Québec Trauma Registry). The study population was selected from adult trauma patients hospitalized between 2013 and 2019. We included patients 16 yr and older with moderate to severe TBI (Glasgow Coma Scale score < 13) who required mechanical ventilation for 96 hr or longer. Our primary outcome was 30-day mortality. Secondary outcomes included hospital and ICU mortality, six-month mortality, duration of mechanical ventilation, ventilator-associated pneumonia, ICU and hospital length of stay as well as orientation of patients upon discharge from the hospital. We used propensity score covariate adjustment. To overcome the effect of immortal time bias, an extended Cox shared frailty model was used to compare mortality between groups.

RESULTS

From 2013 to 2019, 26,923 patients with TBI were registered in the Québec Trauma Registry. A total of 983 patients who required prolonged endotracheal intubation for 96 hr or more were included in the study, 374 of whom underwent a tracheostomy and 609 of whom remained intubated. We observed a reduction in 30-day mortality (adjusted hazard ratio, 0.33; 95% confidence interval, 0.21 to 0.53) associated with tracheostomy compared with prolonged endotracheal intubation. This effect was also seen in the ICU as well as at six months. Tracheostomy, when compared with prolonged endotracheal intubation, was associated with an increase in the duration of mechanical respiratory assistance without any increase in the length of stay. No effect on mortality was observed when comparing early vs late tracheostomy procedures. An early procedure was associated with a reduction in the duration of mechanical respiratory support as well as hospital and ICU length of stay.

CONCLUSION

In this multicentre cohort study, tracheostomy was associated with decreased mortality when compared with prolonged endotracheal intubation in patients with moderate to severe TBI. This effect does not appear to be modified by the timing of the procedure. Nevertheless, the generalization and application of these results remains limited by potential residual time-dependent indication bias.

Covariate handling approaches in combination with dynamic borrowing for hybrid control studies

Borrowing data from external control has been an appealing strategy for evidence synthesis when conducting randomized controlled trials (RCTs). Often named hybrid control trials, they leverage existing control data from clinical trials or potentially real-world data (RWD), enable trial designs to allocate more patients to the novel intervention arm, and improve the efficiency or lower the cost of the primary RCT. Several methods have been established and developed to borrow external control data, among which the propensity score methods and Bayesian dynamic borrowing framework play essential roles. Noticing the unique strengths of propensity score methods and Bayesian hierarchical models, we utilize both methods in a complementary manner to analyze hybrid control studies. In this article, we review methods including covariate adjustments, propensity score matching and weighting in combination with dynamic borrowing and compare the performance of these methods through comprehensive simulations. Different degrees of covariate imbalance and confounding are examined. Our findings suggested that the conventional covariate adjustment in combination with the Bayesian commensurate prior model provides the highest power with good type I error control under the investigated settings. It has desired performance especially under scenarios of different degrees of confounding. To estimate efficacy signals in the exploratory setting, the covariate adjustment method in combination with the Bayesian commensurate prior is recommended.

What should be done and what should be avoided when comparing two treatments?

The preferred approach to compare two treatments is a randomized controlled trial (RCT). Indeed, randomization ensures that the groups compared are similar. Well-designed and well-conducted RCTs thus allow to draw causal conclusions on the relative efficacy and safety of treatments compared. However, it is not always possible to conduct RCTs for all clinical questions of interest, and observational data may also be used to infer on the relative effectiveness of treatments. In this review, we present different approaches that allow statistically valid comparisons of the effectiveness of treatments using observational data under some assumptions. Those are based on regression modelling or the propensity score. We also present the principles of target trial emulation.

Impact of Integrating Machine Learning in Comparative Effectiveness Research of Oral Anticoagulants in Patients with Atrial Fibrillation

We aimed to compare the ability to balance baseline covariates and explore the impact of residual confounding between conventional and machine learning approaches to derive propensity scores (PS). The Health Insurance Review and Assessment Service database (January 2012-September 2019) was used. Patients with atrial fibrillation (AF) who initiated oral anticoagulants during July 2015-September 2018 were included. The outcome of interest was stroke/systemic embolism. To estimate PS, we used a logistic regression model (i.e., a conventional approach) and a generalized boosted model (GBM) which is a machine learning approach. Both PS matching and inverse probability of treatment weighting were performed. To evaluate balance achievement, standardized differences, p-values, and boxplots were used. To explore residual confounding, E-values and negative control outcomes were used. In total, 129,434 patients were identified. Although all baseline covariates were well balanced, the distribution of continuous variables seemed more similar when GBM was applied. E-values ranged between 1.75 and 2.70 and were generally higher in GBM. In the negative control outcome analysis, slightly more nonsignificant hazard ratios were observed in GBM. We showed GBM provided a better ability to balance covariates and had a lower impact of residual confounding, compared with the conventional approach in the empirical example of comparative effectiveness analysis.

Medically Attended Illness due to Respiratory Syncytial Virus Infection Among Infants Born in the United States Between 2016 and 2020

BACKGROUND

Respiratory syncytial virus (RSV) is a leading cause of infant hospitalization in the United States. Preterm infants and those with select comorbidities are at highest risk of RSV-related complications. However, morbidity due to RSV infection is not confined to high-risk infants. We estimated the burden of medically attended (MA) RSV-associated lower respiratory tract infection (LRTI) among infants in the United States.

METHODS

We analyzed commercial (MarketScan Commercial [MSC], Optum Clinformatics [OC]), and Medicaid (MarketScan Medicaid [MSM]) insurance claims data for infants born between April 2016 and February 2020. Using both specific and sensitive definitions of MA RSV LRTI, we estimated the burden of MA RSV LRTI during infants' first RSV season, stratified by gestational age, comorbidity status, and highest level of medical care associated with the MA RSV LRTI diagnosis.

RESULTS

According to the specific definition 75.0% (MSC), 78.6% (MSM), and 79.6% (OC) of MA RSV LRTI events during infants' first RSV season occurred among term infants without known comorbidities.

CONCLUSIONS

Term infants without known comorbidities account for up to 80% of the MA RSV LRTI burden in the United States during infants' first RSV season. Future prevention efforts should consider all infants.

Simulation-based design of pragmatic trials in psoriatic arthritis using propensity scores

BACKGROUND/AIMS

Design of clinical trials requires careful decision-making across several dimensions, including endpoints, eligibility criteria, and subgroup enrichment. Clinical trial simulation can be an informative tool in trial design, providing empirical evidence by which to evaluate and compare the results of hypothetical trials with varying designs. We introduce a novel simulation-based approach using observational data to inform the design of a future pragmatic trial.

METHODS

We utilize propensity score-adjusted models to simulate hypothetical trials under alternative endpoints and enrollment criteria. We apply our approach to the design of pragmatic trials in psoriatic arthritis, using observational data embedded within the Tight Control of Inflammation in Early Psoriatic Arthritis study to simulate hypothetical open-label trials comparing treatment with tumor necrosis factor-α inhibitors to methotrexate. We first validate our simulations of a trial with traditional enrollment criteria and endpoints against a recently published trial. Next, we compare simulated treatment effects in patient populations defined by traditional and broadened enrollment criteria, where the latter is consistent with a future pragmatic trial. In each trial, we also consider five candidate primary endpoints.

RESULTS

Our results highlight how changes in the enrolled population and primary endpoints may qualitatively alter study findings and the ability to detect heterogeneous treatment effects between clinical subgroups. For treatments of interest in the study of psoriatic arthritis, broadened enrollment criteria led to diluted estimated treatment effects. Endpoints with greater responsiveness to treatment compared with a traditionally used endpoint were identified. These considerations, among others, are important for designing a future pragmatic trial aimed at having high external validity with relevance for real-world clinical practice.

CONCLUSION

Observational data may be leveraged to inform design decisions in pragmatic trials. Our approach may be generalized to the study of other conditions where existing trial data are limited or do not generalize well to real-world clinical practice, but where observational data are available.

Propensity Score Analysis with Partially Observed Baseline Covariates: A Practical Comparison of Methods for Handling Missing Data

(1) Background: Propensity score methods gained popularity in non-interventional clinical studies. As it may often occur in observational datasets, some values in baseline covariates are missing for some patients. The present study aims to compare the performances of popular statistical methods to deal with missing data in propensity score analysis. (2) Methods: Methods that account for missing data during the estimation process and methods based on the imputation of missing values, such as multiple imputations, were considered. The methods were applied on the dataset of an ongoing prospective registry for the treatment of unprotected left main coronary artery disease. The performances were assessed in terms of the overall balance of baseline covariates. (3) Results: Methods that explicitly deal with missing data were superior to classical complete case analysis. The best balance was observed when propensity scores were estimated with a method that accounts for missing data using a stochastic approximation of the expectation-maximization algorithm. (4) Conclusions: If missing at random mechanism is plausible, methods that use missing data to estimate propensity score or impute them should be preferred. Sensitivity analyses are encouraged to evaluate the implications methods used to handle missing data and estimate propensity score.

Formulating causal questions and principled statistical answers

Although review papers on causal inference methods are now available, there is a lack of introductory overviews on what they can render and on the guiding criteria for choosing one particular method. This tutorial gives an overview in situations where an exposure of interest is set at a chosen baseline ("point exposure") and the target outcome arises at a later time point. We first phrase relevant causal questions and make a case for being specific about the possible exposure levels involved and the populations for which the question is relevant. Using the potential outcomes framework, we describe principled definitions of causal effects and of estimation approaches classified according to whether they invoke the no unmeasured confounding assumption (including outcome regression and propensity score-based methods) or an instrumental variable with added assumptions. We mainly focus on continuous outcomes and causal average treatment effects. We discuss interpretation, challenges, and potential pitfalls and illustrate application using a "simulation learner," that mimics the effect of various breastfeeding interventions on a child's later development. This involves a typical simulation component with generated exposure, covariate, and outcome data inspired by a randomized intervention study. The simulation learner further generates various (linked) exposure types with a set of possible values per observation unit, from which observed as well as potential outcome data are generated. It thus provides true values of several causal effects. R code for data generation and analysis is available on www.ofcaus.org, where SAS and Stata code for analysis is also provided.

Adaptive treatment strategies for chronic conditions: shared-parameter G-estimation with an application to rheumatoid arthritis

Most estimation algorithms for adaptive treatment strategies assume that treatment rules at each decision point are independent from one another in the sense that they do not possess any common parameters. This is often unrealistic, as the same decisions may be made repeatedly over time. Sharing treatment-decision parameters across decision points offers several advantages, including estimation of fewer parameters and the clinical ease of a single, time-invariant decision to implement. We propose a new computational approach to estimation of shared-parameter G-estimation, which is efficient and shares the double robustness of the "unshared" sequential G-estimation. We use this approach to analyze data from the Scottish Early Rheumatoid Arthritis (SERA) Inception Cohort.

Evaluation of propensity score used in cardiovascular research: a cross-sectional survey and guidance document

BACKGROUND

Propensity score (PS) methods are frequently used in cardiovascular clinical research. Previous evaluations revealed poor reporting of PS methods, however a comprehensive and current evaluation of PS use and reporting is lacking. The objectives of the present survey were to (1) evaluate the quality of PS methods in cardiovascular publications, (2) summarise PS methods and (3) propose key reporting elements for PS publications.

METHODS

A PubMed search for cardiovascular PS articles published between 2010 and 2017 in high-impact general medical (top five by impact factor) and cardiovascular (top three by impact factor) journals was performed. Articles were evaluated for the reporting of PS techniques and methods. Data extraction elements were identified from the PS literature and extraction forms were pilot tested.

RESULTS

Of the 306 PS articles identified, most were published in Journal of the American College of Cardiology (29%; n=88), and Circulation (27%, n=81), followed by European Heart Journal (15%; n=47). PS matching was performed most often, followed by direct adjustment, inverse probability of treatment weighting and stratification. Most studies (77%; n=193) selected variables to include in the PS model a priori. A total of 38% (n=116) of studies did not report standardised mean differences, but instead relied on hypothesis testing. For matching, 92% (n=193) of articles presented the balance of covariates. Overall, interpretations of the effect estimates corresponded to the PS method conducted or described in 49% (n=150) of the reviewed articles.

DISCUSSION

Although PS methods are frequently used in high-impact medical journals, reporting of methodological details has been inconsistent. Improved reporting of PS results is warranted and these proposals should aid both researchers and consumers in the presentation and interpretation of PS methods.