Propensity score estimation: neural networks, support vector machines, decision trees (CART), and meta-classifiers as alternatives to logistic regression

Merits and caveats of propensity scores to adjust for confounding

Proper adjustment for confounding is essential when estimating the effects of treatments or risk factors on health outcomes in observational data. To this end, various statistical methods have been developed. In the past couple of years, the use of propensity scores (PSs) to control for confounding has increased. Proper understanding of this method is necessary to critically appraise research in which it is applied. In this article, we provide an overview of PS methods, explaining their concept, advantages and possible disadvantages. Furthermore, the use of PS matching, PS adjustment and PS weighting is illustrated using data from the Netherlands Cooperative Study on the Adequacy of Dialysis (NECOSAD) cohort of dialysis patients.

An alternative robust estimator of average treatment effect in causal inference

The problem of estimating the average treatment effects is important when evaluating the effectiveness of medical treatments or social intervention policies. Most of the existing methods for estimating the average treatment effect rely on some parametric assumptions about the propensity score model or the outcome regression model one way or the other. In reality, both models are prone to misspecification, which can have undue influence on the estimated average treatment effect. We propose an alternative robust approach to estimating the average treatment effect based on observational data in the challenging situation when neither a plausible parametric outcome model nor a reliable parametric propensity score model is available. Our estimator can be considered as a robust extension of the popular class of propensity score weighted estimators. This approach has the advantage of being robust, flexible, data adaptive, and it can handle many covariates simultaneously. Adopting a dimension reduction approach, we estimate the propensity score weights semiparametrically by using a non-parametric link function to relate the treatment assignment indicator to a low-dimensional structure of the covariates which are formed typically by several linear combinations of the covariates. We develop a class of consistent estimators for the average treatment effect and study their theoretical properties. We demonstrate the robust performance of the estimators on simulated data and a real data example of investigating the effect of maternal smoking on babies' birth weight.

Automated data-adaptive analytics for electronic healthcare data to study causal treatment effects

BACKGROUND

Decision makers in health care increasingly rely on nonrandomized database analyses to assess the effectiveness, safety, and value of medical products. Health care data scientists use data-adaptive approaches that automatically optimize confounding control to study causal treatment effects. This article summarizes relevant experiences and extensions.

METHODS

The literature was reviewed on the uses of high-dimensional propensity score (HDPS) and related approaches for health care database analyses, including methodological articles on their performance and improvement. Articles were grouped into applications, comparative performance studies, and statistical simulation experiments.

RESULTS

The HDPS algorithm has been referenced frequently with a variety of clinical applications and data sources from around the world. The appeal of HDPS for database research rests in 1) its superior performance in situations of unobserved confounding through proxy adjustment, 2) its predictable efficiency in extracting confounding information from a given data source, 3) its ability to automate estimation of causal treatment effects to the extent achievable in a given data source, and 4) its independence of data source and coding system. Extensions of the HDPS approach have focused on improving variable selection when exposure is sparse, using free text information and time-varying confounding adjustment.

CONCLUSION

Semiautomated and optimized confounding adjustment in health care database analyses has proven successful across a wide range of settings. Machine-learning extensions further automate its use in estimating causal treatment effects across a range of data scenarios.

Targeted Estimation of the Relationship Between Childhood Adversity and Fluid Intelligence in a US Population Sample of Adolescents

Many studies have shown inverse associations between childhood adversity and intelligence, although most are based on small clinical samples and fail to account for the effects of multiple co-occurring adversities. Using data from the 2001-2004 National Comorbidity Survey Adolescent Supplement, a cross-sectional US population study of adolescents aged 13-18 years (n = 10,073), we examined the associations between 11 childhood adversities and intelligence, using targeted maximum likelihood estimation. Targeted maximum likelihood estimation incorporates machine learning to identify the relationships between exposures and outcomes without overfitting, including interactions and nonlinearity. The nonverbal score from the Kaufman Brief Intelligence Test was used as a standardized measure of fluid reasoning. Childhood adversities were grouped into deprivation and threat types based on recent conceptual models. Adjusted marginal mean differences compared the mean intelligence score if all adolescents experienced each adversity to the mean in the absence of the adversity. The largest associations were observed for deprivation-type experiences, including poverty and low parental education, which were related to reduced intelligence. Although lower in magnitude, threat events related to intelligence included physical abuse and witnessing domestic violence. Violence prevention and poverty-reduction measures would likely improve childhood cognitive outcomes.

The impact of a statewide trauma-informed child welfare initiative on children's permanency and maltreatment outcomes

This article presents findings of a state-wide trauma informed child-welfare initiative with the goal of improving well-being, permanency and maltreatment outcomes for traumatized children. The Massachuetts Child Trauma Project (MCTP), funded by the Administration of Children and Families, Children's Bureau was a multi-year project implementing trauma-informed care into child welfare service delivery. The project's implementation design included training and consultation for mental health providers in three evidence-based treatments and training of the child-welfare workforce in trauma-informed case work practice. The learning was integrated between child-welfare and mental health with Trauma Informed Leadership Teams which included leaders from both systems and the greater community. These teams developed incremental steps toward trauma-informed system improvement. This study evaluated whether MCTP was associated with reductions in child abuse and neglect, improvements in placement stability, and higher rates of permanency during the first year of implementation. Children in the intervention group had fewer total substantiated reports of maltreatment, including less physical abuse and neglect than the comparison group by the end of the intervention year. However, children in the intervention group had more maltreatment reports (substantiated or not) and total out-of-home placements than did their counterparts in the comparison group. Assignment to MCTP, however, was not associated with an increase in kinship care or adoption. Overall, the results are promising in reinforcing the importance of mobilizing communities toward improvements in child-welfare service delivery.

Stacked generalization: an introduction to super learning

Stacked generalization is an ensemble method that allows researchers to combine several different prediction algorithms into one. Since its introduction in the early 1990s, the method has evolved several times into a host of methods among which is the "Super Learner". Super Learner uses V-fold cross-validation to build the optimal weighted combination of predictions from a library of candidate algorithms. Optimality is defined by a user-specified objective function, such as minimizing mean squared error or maximizing the area under the receiver operating characteristic curve. Although relatively simple in nature, use of Super Learner by epidemiologists has been hampered by limitations in understanding conceptual and technical details. We work step-by-step through two examples to illustrate concepts and address common concerns.

Performing an Informatics Consult: Methods and Challenges

Our health care system is plagued by missed opportunities, waste, and harm. Data generated in the course of care are often underutilized, scientific insight goes untranslated, and evidence is overlooked. To address these problems, we envisioned a system where aggregate patient data can be used at the bedside to provide practice-based evidence. To create that system, we directly connect practicing physicians to clinical researchers and data scientists through an informatics consult. Our team processes and classifies questions posed by clinicians, identifies the appropriate patient data to use, runs the appropriate analyses, and returns an answer, ideally in a 48-hour time window. Here, we discuss the methods that are used for data extraction, processing, and analysis in our consult. We continue to refine our informatics consult service, moving closer to a learning health care system.

Choice of time period to identify confounders for propensity score matching, affected the estimate: a retrospective cohort study of drug effectiveness in asthmatic children

OBJECTIVES

To control for confounding by indication in comparative (drug) effectiveness studies, propensity score (PS) methods may be used. Since childhood diseases or outcomes often present as acute events, we compared the effect of using different look-back periods in electronic health-care data, to construct PSs. This was applied in our research on the effect of a combination of inhaled corticosteroids/long-acting beta-2 agonists (ICS + LABA), either as fixed combination or used as loose combination (2 separate inhaler devices) in the prevention of severe asthma exacerbations.

METHODS

We created a cohort of children (5-17 years) diagnosed with asthma from the Dutch Integrated Primary Care information database. Within this cohort, we identified new users of ICS + LABA, either as fixed combination or loose combination (2 separate inhaler devices). The outcome of interest was severe asthma exacerbations. PSs for type of treatment were created using comorbidity and drug use history in different time windows: 1 week, 1 month, 3 months, 1 year, and full history prior to the start of treatment. PSs were used for matching subjects in both exposure groups. Time to first asthma exacerbation was analyzed with Cox proportional hazard regression. The results were compared with published clinical trials.

RESULTS

Of 39,682 asthmatic children, 3,500 (8.8%) were new users of either ICS + LABA fixed (3,324 [95.0%]) or loose (176 [5.0%]). The crude hazard ratio (HR) for a severe asthma exacerbation, comparing ICS + LABA fixed to loose was 0.37 (95% confidence interval [CI]: 0.20-0.66). PS-matched HRs (1 week, 1 month, 3 month, 1 year, and full history) were 0.48 (95% CI: 0.22-1.04); 0.60 (95% CI: 0.26-1.38), 0.69 (95% CI: 0.31-1.57), 0.56 (CI: 0.25-1.24), and 0.58 (CI: 0.24-1.36), respectively.

CONCLUSIONS

PS matching can be used to control for confounding in pediatric comparative (drug) effectiveness studies, the impact of different look-back periods to implement the PS is important. Controlling for confounders occurring in the 3 months preceding drug exposure may yield results comparable to clinical trial results.

Targeted Maximum Likelihood Estimation for Pharmacoepidemiologic Research

Supplemental Digital Content is available in the text.

Background:

Targeted maximum likelihood estimation has been proposed for estimating marginal causal effects, and is robust to misspecification of either the treatment or outcome model. However, due perhaps to its novelty, targeted maximum likelihood estimation has not been widely used in pharmacoepidemiology. The objective of this study was to demonstrate targeted maximum likelihood estimation in a pharmacoepidemiological study with a high-dimensional covariate space, to incorporate the use of high-dimensional propensity scores into this method, and to compare the results to those of inverse probability weighting.

Methods:

We implemented the targeted maximum likelihood estimation procedure in a single-point exposure study of the use of statins and the 1-year risk of all-cause mortality postmyocardial infarction using data from the UK Clinical Practice Research Datalink. A range of known potential confounders were considered, and empirical covariates were selected using the high-dimensional propensity scores algorithm. We estimated odds ratios using targeted maximum likelihood estimation and inverse probability weighting with a variety of covariate selection strategies.

Results:

Through a real example, we demonstrated the double robustness of targeted maximum likelihood estimation. We showed that results with this method and inverse probability weighting differed when a large number of covariates were included in the treatment model.

Conclusions:

Targeted maximum likelihood can be used in high-dimensional covariate settings. In high-dimensional covariate settings, differences in results between targeted maximum likelihood and inverse probability weighted estimation are likely due to sensitivity to (near) positivity violations. Further investigations are needed to gain better understanding of the advantages and limitations of this method in pharmacoepidemiological studies.

Delirium Prediction using Machine Learning Models on Preoperative Electronic Health Records Data

Electronic Health Records (EHR) are mainly designed to record relevant patient information during their stay in the hospital for administrative purposes. They additionally provide an efficient and inexpensive source of data for medical research, such as patient outcome prediction. In this study, we used preoperative Electronic Health Records to predict postoperative delirium. We compared the performance of seven machine learning models on delirium prediction: linear models, generalized additive models, random forests, support vector machine, neural networks, and extreme gradient boosting. Among the models evaluated in this study, random forests and generalized additive model outperformed the other models in terms of the overall performance metrics for prediction of delirium, particularly with respect to sensitivity. We found that age, alcohol or drug abuse, socioeconomic status, underlying medical issue, severity of medical problem, and attending surgeon can affect the risk of delirium.

The Right Tool for the Job: Choosing Between Covariate-balancing and Generalized Boosted Model Propensity Scores

Estimating the causal effect of an exposure (vs. some control) on an outcome using observational data often requires addressing the fact that exposed and control groups differ on pre-exposure characteristics that may be related to the outcome (confounders). Propensity score methods have long been used as a tool for adjusting for observed confounders in order to produce more valid causal effect estimates under the strong ignorability assumption. In this article, we compare two promising propensity score estimation methods (for time-invariant binary exposures) when assessing the average treatment effect on the treated: the generalized boosted models and covariate-balancing propensity scores, with the main objective to provide analysts with some rules-of-thumb when choosing between these two methods. We compare the methods across different dimensions including the presence of extraneous variables, the complexity of the relationship between exposure or outcome and covariates, and the residual variance in outcome and exposure. We found that when noncomplex relationships exist between outcome or exposure and covariates, the covariate-balancing method outperformed the boosted method, but under complex relationships, the boosted method performed better. We lay out criteria for when one method should be expected to outperform the other with no blanket statement on whether one method is always better than the other.

Improving causal inference with a doubly robust estimator that combines propensity score stratification and weighting

RATIONALE, AIMS AND OBJECTIVES

When a randomized controlled trial is not feasible, health researchers typically use observational data and rely on statistical methods to adjust for confounding when estimating treatment effects. These methods generally fall into 3 categories: (1) estimators based on a model for the outcome using conventional regression adjustment; (2) weighted estimators based on the propensity score (ie, a model for the treatment assignment); and (3) "doubly robust" (DR) estimators that model both the outcome and propensity score within the same framework. In this paper, we introduce a new DR estimator that utilizes marginal mean weighting through stratification (MMWS) as the basis for weighted adjustment. This estimator may prove more accurate than treatment effect estimators because MMWS has been shown to be more accurate than other models when the propensity score is misspecified. We therefore compare the performance of this new estimator to other commonly used treatment effects estimators.

METHOD

Monte Carlo simulation is used to compare the DR-MMWS estimator to regression adjustment, 2 weighted estimators based on the propensity score and 2 other DR methods. To assess performance under varied conditions, we vary the level of misspecification of the propensity score model as well as misspecify the outcome model.

RESULTS

Overall, DR estimators generally outperform methods that model one or the other components (eg, propensity score or outcome). The DR-MMWS estimator outperforms all other estimators when both the propensity score and outcome models are misspecified and performs equally as well as other DR estimators when only the propensity score is misspecified.

CONCLUSIONS

Health researchers should consider using DR-MMWS as the principal evaluation strategy in observational studies, as this estimator appears to outperform other estimators in its class.

Using classification tree analysis to generate propensity score weights

RATIONALE, AIMS AND OBJECTIVES

In evaluating non-randomized interventions, propensity scores (PS) estimate the probability of assignment to the treatment group given observed characteristics. Machine learning algorithms have been proposed as an alternative to conventional logistic regression for modelling PS in order to avoid limitations of linear methods. We introduce classification tree analysis (CTA) to generate PS which is a "decision-tree"-like classification model that provides accurate, parsimonious decision rules that are easy to display and interpret, reports P values derived via permutation tests, and evaluates cross-generalizability.

METHOD

Using empirical data, we identify all statistically valid CTA PS models and then use them to compute strata-specific, observation-level PS weights that are subsequently applied in outcomes analyses. We compare findings obtained using this framework to logistic regression and boosted regression, by evaluating covariate balance using standardized differences, model predictive accuracy, and treatment effect estimates obtained using median regression and a weighted CTA outcomes model.

RESULTS

While all models had some imbalanced covariates, main-effects logistic regression yielded the lowest average standardized difference, whereas CTA yielded the greatest predictive accuracy. Nevertheless, treatment effect estimates were generally consistent across all models.

CONCLUSIONS

Assessing standardized differences in means as a test of covariate balance is inappropriate for machine learning algorithms that segment the sample into two or more strata. Because the CTA algorithm identifies all statistically valid PS models for a sample, it is most likely to identify a correctly specified PS model, and should be considered as an alternative approach to modeling the PS.

Propensity scores based methods for estimating average treatment effect and average treatment effect among treated: A comparative study

Propensity score based statistical methods, such as matching, regression, stratification, inverse probability weighting (IPW), and doubly robust (DR) estimating equations, have become popular in estimating average treatment effect (ATE) and average treatment effect among treated (ATT) in observational studies. Propensity score is the conditional probability receiving a treatment assignment with given covariates, and propensity score is usually estimated by logistic regression. However, a misspecification of the propensity score model may result in biased estimates for ATT and ATE. As an alternative, the generalized boosting method (GBM) has been proposed to estimate the propensity score. GBM uses regression trees as weak predictors and captures nonlinear and interactive effects of the covariate. For GBM-based propensity score, only IPW methods have been investigated in the literature. In this article, we provide a comparative study of the commonly used propensity score based methods for estimating ATT and ATE, and examine their performances when propensity score is estimated by logistic regression and GBM, respectively. Extensive simulation results indicate that the estimators for ATE and ATT may vary greatly due to different methods. We concluded that (i) regression may not be suitable for estimating ATE and ATT regardless of the estimation method of propensity score; (ii) IPW and stratification usually provide reliable estimates of ATT when propensity score model is correctly specified; (iii) the estimators of ATE based on stratification, IPW, and DR are close to the underlying true value of ATE when propensity score is correctly specified by logistic regression or estimated using GBM.

Predictive model for 5-year mortality after breast cancer surgery in Taiwan residents

BACKGROUND

Few studies of breast cancer surgery outcomes have used longitudinal data for more than 2 years. This study aimed to validate the use of the artificial neural network (ANN) model to predict the 5-year mortality of breast cancer patients after surgery and compare predictive accuracy between the ANN model, multiple logistic regression (MLR) model, and Cox regression model.

METHODS

This study compared the MLR, Cox, and ANN models based on clinical data of 3632 breast cancer patients who underwent surgery between 1996 and 2010. An estimation dataset was used to train the model, and a validation dataset was used to evaluate model performance. The sensitivity analysis was also used to assess the relative significance of input variables in the prediction model.

RESULTS

The ANN model significantly outperformed the MLR and Cox models in predicting 5-year mortality, with higher overall performance indices. The results indicated that the 5-year postoperative mortality of breast cancer patients was significantly associated with age, Charlson comorbidity index (CCI), chemotherapy, radiotherapy, hormone therapy, and breast cancer surgery volumes of hospital and surgeon (all P < 0.05). Breast cancer surgery volume of surgeon was the most influential (sensitive) variable affecting 5-year mortality, followed by breast cancer surgery volume of hospital, age, and CCI.

CONCLUSIONS

Compared with the conventional MLR and Cox models, the ANN model was more accurate in predicting 5-year mortality of breast cancer patients who underwent surgery. The mortality predictors identified in this study can also be used to educate candidates for breast cancer surgery with respect to the course of recovery and health outcomes.

A doubly robust approach for cost-effectiveness estimation from observational data

Estimation of common cost-effectiveness measures, including the incremental cost-effectiveness ratio and the net monetary benefit, is complicated by the need to account for informative censoring and inherent skewness of the data. In addition, since the two components of these measures, medical costs and survival are often collected from observational claims data, one must account for potential confounders. We propose a novel doubly robust, unbiased estimator for cost-effectiveness based on propensity scores that allow the incorporation of cost history and time-varying covariates. Further, we use an ensemble machine learning approach to obtain improved predictions from parametric and non-parametric cost and propensity score models. Our simulation studies demonstrate that the proposed doubly robust approach performs well even under mis-specification of either the propensity score model or the outcome model. We apply our approach to a cost-effectiveness analysis of two competing lung cancer surveillance procedures, CT vs. chest X-ray, using SEER-Medicare data.

Predicting the incidence of portosplenomesenteric vein thrombosis in patients with acute pancreatitis using classification and regression tree algorithm

BACKGROUND AND OBJECTIVE

The accurate prediction of portosplenomesenteric vein thrombosis (PVT) in patients with acute pancreatitis(AP) is very important but may also be difficult because of our insufficient understanding of the characteristics of AP-induced PVT. The purpose of this study is to design a decision tree model that provides critical factors associated with PVT using an approach that makes use of classification and regression tree (CART) algorithm.

METHODS

The analysis included 353 patients with AP who were admitted between January 2011 and December 2015. CART model and logistic regression model were each applied to the same 50% of the sample to develop the predictive training models, and these models were tested on the remaining 50%. Statistical indexes were used to evaluate the value of the prediction in the 2 models.

RESULTS

The predicted sensitivity, specificity, positive predictive value, negative predictive value, and accuracy by CART for PVT were 78.0%, 87.2%, 64.0%, 93.2%, and 85.2%, respectively. Significant differences could be found between the CART model and the logistic regression model in these parameters. There were significant differences between the CART and logistic regression models in these parameters (P<.05). When the CART model was used to identify PVT, the area under receiver operating characteristic curve was 0.803, which demonstrated better overall properties than the logistic regression model (area under the curve=0.696) (95% confidence interval, 0.603-0.812).

CONCLUSION

The CART model based on serum amylase, d-dimer, Acute Physiology and Chronic Health Evaluation II, and prothrombin time is more likely to predict the occurrence of PVT induced by AP.

Targeted Maximum Likelihood Estimation for Causal Inference in Observational Studies

Estimation of causal effects using observational data continues to grow in popularity in the epidemiologic literature. While many applications of causal effect estimation use propensity score methods or G-computation, targeted maximum likelihood estimation (TMLE) is a well-established alternative method with desirable statistical properties. TMLE is a doubly robust maximum-likelihood-based approach that includes a secondary "targeting" step that optimizes the bias-variance tradeoff for the target parameter. Under standard causal assumptions, estimates can be interpreted as causal effects. Because TMLE has not been as widely implemented in epidemiologic research, we aim to provide an accessible presentation of TMLE for applied researchers. We give step-by-step instructions for using TMLE to estimate the average treatment effect in the context of an observational study. We discuss conceptual similarities and differences between TMLE and 2 common estimation approaches (G-computation and inverse probability weighting) and present findings on their relative performance using simulated data. Our simulation study compares methods under parametric regression misspecification; our results highlight TMLE's property of double robustness. Additionally, we discuss best practices for TMLE implementation, particularly the use of ensembled machine learning algorithms. Our simulation study demonstrates all methods using super learning, highlighting that incorporation of machine learning may outperform parametric regression in observational data settings.

The Balance Super Learner: A robust adaptation of the Super Learner to improve estimation of the average treatment effect in the treated based on propensity score matching

Consistency of the propensity score estimators rely on correct specification of the propensity score model. The propensity score is frequently estimated using a main effect logistic regression. It has recently been shown that the use of ensemble machine learning algorithms, such as the Super Learner, could improve covariate balance and reduce bias in a meaningful manner in the case of serious model misspecification for treatment assignment. However, the loss functions normally used by the Super Learner may not be appropriate for propensity score estimation since the goal in this problem is not to optimize propensity score prediction but rather to achieve the best possible balance in the covariate distribution between treatment groups. In a simulation study, we evaluated the benefit of a modification of the Super Learner by propensity score estimation geared toward achieving covariate balance between the treated and untreated after matching on the propensity score. Our simulation study included six different scenarios characterized by various degrees of deviation from the usual main term logistic model for the true propensity score and outcome as well as the presence (or not) of instrumental variables. Our results suggest that the use of this adapted Super Learner to estimate the propensity score can further improve the robustness of propensity score matching estimators.

Estimation of conditional and marginal odds ratios using the prognostic score

Introduced by Hansen in 2008, the prognostic score (PGS) has been presented as 'the prognostic analogue of the propensity score' (PPS). PPS-based methods are intended to estimate marginal effects. Most previous studies evaluated the performance of existing PGS-based methods (adjustment, stratification and matching using the PGS) in situations in which the theoretical conditional and marginal effects are equal (i.e., collapsible situations). To support the use of PGS framework as an alternative to the PPS framework, applied researchers must have reliable information about the type of treatment effect estimated by each method. We propose four new PGS-based methods, each developed to estimate a specific type of treatment effect. We evaluated the ability of existing and new PGS-based methods to estimate the conditional treatment effect (CTE), the (marginal) average treatment effect on the whole population (ATE), and the (marginal) average treatment effect on the treated population (ATT), when the odds ratio (a non-collapsible estimator) is the measure of interest. The performance of PGS-based methods was assessed by Monte Carlo simulations and compared with PPS-based methods and multivariate regression analysis. Existing PGS-based methods did not allow for estimating the ATE and showed unacceptable performance when the proportion of exposed subjects was large. When estimating marginal effects, PPS-based methods were too conservative, whereas the new PGS-based methods performed better with low prevalence of exposure, and had coverages closer to the nominal value. When estimating CTE, the new PGS-based methods performed as well as traditional multivariate regression. Copyright © 2016 John Wiley & Sons, Ltd.

A multicenter study of diet quality on birth weight and gestational age in infants of HIV-infected women

We determined factors associated with diet quality and assessed the relationship between diet quality, birth weight, and gestational age in a prospective national multicenter cohort study. We evaluated diet quality with the Healthy Eating Index (HEI, scale 0-100) in the third trimester of pregnancy with three 24-hr multiple-pass dietary recalls in 266 HIV+ women enrolled in the Pediatric HIV/AIDS Cohort Study. Covariates included demographics, food security, pre-pregnancy body mass index, HIV disease severity, substance use, and antiretroviral exposures. A two-stage multivariate process using classification and regression trees (CART) followed by multiple regression described HEI tendencies, controlled possible confounding effects, and examined the association of HEI with birth weight and gestational age. To assess the stability of the CART solution, both the HEI 2005 and 2010 were evaluated. The mean HEI scores were 56.1 and 47.5 for the 2005 and 2010 HEI, respectively. The first-stage CART analysis examined the relationship between HEI and covariates. Non-US born versus US-born mothers had higher HEI scores (15-point difference, R²  = 0.28). There was a secondary partition due to alcohol/cigarette/illicit drug usage (3.5-point difference, R²  = 0.03) among US-born women. For the second-stage CART adjusted multiple regression, birth weight z-score was positively related to HEI 2005 and 2010 (partial r's > 0.13, P's ≤ 0.0398), but not gestational age (r = 0.00). We conclude that diet quality among HIV+ women is associated with higher birth weight. Despite the influence of a large cultural effect and poor prenatal behaviors, interventions to improve diet in HIV+ women may help to increase birth weight.

Propensity Scoring after Multiple Imputation in a Retrospective Study on Adjuvant Radiation Therapy in Lymph-Node Positive Vulvar Cancer

Propensity scoring (PS) is an established tool to account for measured confounding in non-randomized studies. These methods are sensitive to missing values, which are a common problem in observational data. The combination of multiple imputation of missing values and different propensity scoring techniques is addressed in this work. For a sample of lymph node-positive vulvar cancer patients, we re-analyze associations between the application of radiotherapy and disease-related and non-related survival. Inverse-probability-of-treatment-weighting (IPTW) and PS stratification are applied after multiple imputation by chained equation (MICE). Methodological issues are described in detail. Interpretation of the results and methodological limitations are discussed.

Evaluating treatment effectiveness under model misspecification: A comparison of targeted maximum likelihood estimation with bias-corrected matching

Statistical approaches for estimating treatment effectiveness commonly model the endpoint, or the propensity score, using parametric regressions such as generalised linear models. Misspecification of these models can lead to biased parameter estimates. We compare two approaches that combine the propensity score and the endpoint regression, and can make weaker modelling assumptions, by using machine learning approaches to estimate the regression function and the propensity score. Targeted maximum likelihood estimation is a double-robust method designed to reduce bias in the estimate of the parameter of interest. Bias-corrected matching reduces bias due to covariate imbalance between matched pairs by using regression predictions. We illustrate the methods in an evaluation of different types of hip prosthesis on the health-related quality of life of patients with osteoarthritis. We undertake a simulation study, grounded in the case study, to compare the relative bias, efficiency and confidence interval coverage of the methods. We consider data generating processes with non-linear functional form relationships, normal and non-normal endpoints. We find that across the circumstances considered, bias-corrected matching generally reported less bias, but higher variance than targeted maximum likelihood estimation. When either targeted maximum likelihood estimation or bias-corrected matching incorporated machine learning, bias was much reduced, compared to using misspecified parametric models.

Best (but oft-forgotten) practices: propensity score methods in clinical nutrition research

In observational studies, treatment assignment is a nonrandom process and treatment groups may not be comparable in their baseline characteristics, a phenomenon known as confounding. Propensity score (PS) methods can be used to achieve comparability of treated and nontreated groups in terms of their observed covariates and, as such, control for confounding in estimating treatment effects. In this article, we provide a step-by-step guidance on how to use PS methods. For illustrative purposes, we used simulated data based on an observational study of the relation between oral nutritional supplementation and hospital length of stay. We focused on the key aspects of PS analysis, including covariate selection, PS estimation, covariate balance assessment, treatment effect estimation, and reporting. PS matching, stratification, covariate adjustment, and weighting are discussed. R codes and example data are provided to show the different steps in a PS analysis.

Application of Machine Learning Techniques to High-Dimensional Clinical Data to Forecast Postoperative Complications

Objective

To compare performance of risk prediction models for forecasting postoperative sepsis and acute kidney injury.

Design

Retrospective single center cohort study of adult surgical patients admitted between 2000 and 2010.

Patients

50,318 adult patients undergoing major surgery.

Measurements

We evaluated the performance of logistic regression, generalized additive models, naïve Bayes and support vector machines for forecasting postoperative sepsis and acute kidney injury. We assessed the impact of feature reduction techniques on predictive performance. Model performance was determined using the area under the receiver operating characteristic curve, accuracy, and positive predicted value. The results were reported based on a 70/30 cross validation procedure where the data were randomly split into 70% used for training the model and the 30% for validation.

Main Results

The areas under the receiver operating characteristic curve for different models ranged between 0.797 and 0.858 for acute kidney injury and between 0.757 and 0.909 for severe sepsis. Logistic regression, generalized additive model, and support vector machines had better performance compared to Naïve Bayes model. Generalized additive models additionally accounted for non-linearity of continuous clinical variables as depicted in their risk patterns plots. Reducing the input feature space with LASSO had minimal effect on prediction performance, while feature extraction using principal component analysis improved performance of the models.

Conclusions

Generalized additive models and support vector machines had good performance as risk prediction model for postoperative sepsis and AKI. Feature extraction using principal component analysis improved the predictive performance of all models.

A Case Study of the Impact of Data-Adaptive Versus Model-Based Estimation of the Propensity Scores on Causal Inferences from Three Inverse Probability Weighting Estimators

OBJECTIVE

Consistent estimation of causal effects with inverse probability weighting estimators is known to rely on consistent estimation of propensity scores. To alleviate the bias expected from incorrect model specification for these nuisance parameters in observational studies, data-adaptive estimation and in particular an ensemble learning approach known as Super Learning has been proposed as an alternative to the common practice of estimation based on arbitrary model specification. While the theoretical arguments against the use of the latter haphazard estimation strategy are evident, the extent to which data-adaptive estimation can improve inferences in practice is not. Some practitioners may view bias concerns over arbitrary parametric assumptions as academic considerations that are inconsequential in practice. They may also be wary of data-adaptive estimation of the propensity scores for fear of greatly increasing estimation variability due to extreme weight values. With this report, we aim to contribute to the understanding of the potential practical consequences of the choice of estimation strategy for the propensity scores in real-world comparative effectiveness research.

METHOD

We implement secondary analyses of Electronic Health Record data from a large cohort of type 2 diabetes patients to evaluate the effects of four adaptive treatment intensification strategies for glucose control (dynamic treatment regimens) on subsequent development or progression of urinary albumin excretion. Three Inverse Probability Weighting estimators are implemented using both model-based and data-adaptive estimation strategies for the propensity scores. Their practical performances for proper confounding and selection bias adjustment are compared and evaluated against results from previous randomized experiments.

CONCLUSION

Results suggest both potential reduction in bias and increase in efficiency at the cost of an increase in computing time when using Super Learning to implement Inverse Probability Weighting estimators to draw causal inferences.

Variable Selection for Confounder Control, Flexible Modeling and Collaborative Targeted Minimum Loss-Based Estimation in Causal Inference

This paper investigates the appropriateness of the integration of flexible propensity score modeling (nonparametric or machine learning approaches) in semiparametric models for the estimation of a causal quantity, such as the mean outcome under treatment. We begin with an overview of some of the issues involved in knowledge-based and statistical variable selection in causal inference and the potential pitfalls of automated selection based on the fit of the propensity score. Using a simple example, we directly show the consequences of adjusting for pure causes of the exposure when using inverse probability of treatment weighting (IPTW). Such variables are likely to be selected when using a naive approach to model selection for the propensity score. We describe how the method of Collaborative Targeted minimum loss-based estimation (C-TMLE; van der Laan and Gruber, 2010 [27]) capitalizes on the collaborative double robustness property of semiparametric efficient estimators to select covariates for the propensity score based on the error in the conditional outcome model. Finally, we compare several approaches to automated variable selection in low- and high-dimensional settings through a simulation study. From this simulation study, we conclude that using IPTW with flexible prediction for the propensity score can result in inferior estimation, while Targeted minimum loss-based estimation and C-TMLE may benefit from flexible prediction and remain robust to the presence of variables that are highly correlated with treatment. However, in our study, standard influence function-based methods for the variance underestimated the standard errors, resulting in poor coverage under certain data-generating scenarios.

Time Trend of Outcomes for Severe Acute Pancreatitis After Publication of Japanese Guidelines Based on a National Administrative Database

OBJECTIVES

This study aimed to investigate the recent time trend of outcomes for severe acute pancreatitis after publication of Japanese guidelines based on a national administrative database.

METHODS

A total of 10,400 patients with severe acute pancreatitis were referred to 1021 hospitals between 2010 and 2012 in Japan. We collected patients' data from the administrative database to compare in-hospital mortality (within 28 days and overall), length of stay (LOS), and medical costs during hospitalization. The study periods were categorized into 3 groups according to fiscal year: 2010 (n = 2698), 2011 (n = 3842), and 2012 (n = 3860).

RESULTS

In-hospital mortality within 28 days and overall in-hospital mortality were significantly decreased according to fiscal year (6.3% [2010] vs 5.7% [2011] vs 4.5% [2012], P = 0.005; 7.6% vs 7.1% vs 5.6%, P = 0.002, respectively). However, mean LOS and medical costs were not different between fiscal years (27.0 vs 27.1 vs 26.9 days, P = 0.218; 13,998.0 vs 14,156.4 vs 14,319.2 USD, P = 0.232, respectively).

CONCLUSIONS

This study shows that mortality of severe acute pancreatitis was reduced according to the time course, whereas LOS or medical costs were stable after publication of the Japanese guidelines.

On the use of propensity scores in case of rare exposure

Background

Observational post-marketing assessment studies often involve evaluating the effect of a rare treatment on a time-to-event outcome, through the estimation of a marginal hazard ratio. Propensity score (PS) methods are the most used methods to estimate marginal effect of an exposure in observational studies. However there is paucity of data concerning their performance in a context of low prevalence of exposure.

Methods

We conducted an extensive series of Monte Carlo simulations to examine the performance of the two preferred PS methods, known as PS-matching and PS-weighting to estimate marginal hazard ratios, through various scenarios.

Results

We found that both PS-weighting and PS-matching could be biased when estimating the marginal effect of rare exposure. The less biased results were obtained with estimators of average treatment effect in the treated population (ATT), in comparison with estimators of average treatment effect in the overall population (ATE). Among ATT estimators, PS-weighting using ATT weights outperformed PS-matching. These results are illustrated using a real observational study.

Conclusions

When clinical objectives are focused on the treated population, applied researchers are encouraged to estimate ATT with PS-weighting for studying the relative effect of a rare treatment on time-to-event outcomes.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-016-0135-1) contains supplementary material, which is available to authorized users.

The EffectLiteR Approach for Analyzing Average and Conditional Effects

We present a framework for estimating average and conditional effects of a discrete treatment variable on a continuous outcome variable, conditioning on categorical and continuous covariates. Using the new approach, termed the EffectLiteR approach, researchers can consider conditional treatment effects given values of all covariates in the analysis and various aggregates of these conditional treatment effects such as average effects, effects on the treated, or aggregated conditional effects given values of a subset of covariates. Building on structural equation modeling, key advantages of the new approach are (1) It allows for latent covariates and outcome variables; (2) it permits (higher order) interactions between the treatment variable and categorical and (latent) continuous covariates; and (3) covariates can be treated as stochastic or fixed. The approach is illustrated by an example, and open source software EffectLiteR is provided, which makes a detailed analysis of effects conveniently accessible for applied researchers.

Head to head comparison of the propensity score and the high-dimensional propensity score matching methods

BACKGROUND

Comparative performance of the traditional propensity score (PS) and high-dimensional propensity score (hdPS) methods in the adjustment for confounding by indication remains unclear. We aimed to identify which method provided the best adjustment for confounding by indication within the context of the risk of diabetes among patients exposed to moderate versus high potency statins.

METHOD

A cohort of diabetes-free incident statins users was identified from the Quebec's publicly funded medico-administrative database (Full Cohort). We created two matched sub-cohorts by matching one patient initiated on a lower potency to one patient initiated on a high potency either on patients' PS or hdPS. Both methods' performance were compared by means of the absolute standardized differences (ASDD) regarding relevant characteristics and by means of the obtained measures of association.

RESULTS

Eight out of the 18 examined characteristics were shown to be unbalanced within the Full Cohort. Although matching on either method achieved balance within all examined characteristic, matching on patients' hdPS created the most balanced sub-cohort. Measures of associations and confidence intervals obtained within the two matched sub-cohorts overlapped.

CONCLUSION

Although ASDD suggest better matching with hdPS than with PS, measures of association were almost identical when adjusted for either method. Use of the hdPS method in adjusting for confounding by indication within future studies should be recommended due to its ability to identify confounding variables which may be unknown to the investigators.

Statistical foundations for model-based adjustments

Most epidemiology textbooks that discuss models are vague on details of model selection. This lack of detail may be understandable since selection should be strongly influenced by features of the particular study, including contextual (prior) information about covariates that may confound, modify, or mediate the effect under study. It is thus important that authors document their modeling goals and strategies and understand the contextual interpretation of model parameters and model selection criteria. To illustrate this point, we review several established strategies for selecting model covariates, describe their shortcomings, and point to refinements, assuming that the main goal is to derive the most accurate effect estimates obtainable from the data and available resources. This goal shifts the focus to prediction of exposure or potential outcomes (or both) to adjust for confounding; it thus differs from the goal of ordinary statistical modeling, which is to passively predict outcomes. Nonetheless, methods and software for passive prediction can be used for causal inference as well, provided that the target parameters are shifted appropriately.

Propensity score and proximity matching using random forest

In order to derive unbiased inference from observational data, matching methods are often applied to produce balanced treatment and control groups in terms of all background variables. Propensity score has been a key component in this research area. However, propensity score based matching methods in the literature have several limitations, such as model mis-specifications, categorical variables with more than two levels, difficulties in handling missing data, and nonlinear relationships. Random forest, averaging outcomes from many decision trees, is nonparametric in nature, straightforward to use, and capable of solving these issues. More importantly, the precision afforded by random forest (Caruana et al., 2008) may provide us with a more accurate and less model dependent estimate of the propensity score. In addition, the proximity matrix, a by-product of the random forest, may naturally serve as a distance measure between observations that can be used in matching. The proposed random forest based matching methods are applied to data from the National Health and Nutrition Examination Survey (NHANES). Our results show that the proposed methods can produce well balanced treatment and control groups. An illustration is also provided that the methods can effectively deal with missing data in covariates.

Comparing high-dimensional confounder control methods for rapid cohort studies from electronic health records

Aims:

Electronic health records (EHR), containing rich clinical histories of large patient populations, can provide evidence for clinical decisions when evidence from trials and literature is absent. To enable such observational studies from EHR in real time, particularly in emergencies, rapid confounder control methods that can handle numerous variables and adjust for biases are imperative. This study compares the performance of 18 automatic confounder control methods.

Methods:

Methods include propensity scores, direct adjustment by machine learning, similarity matching and resampling in two simulated and one real-world EHR datasets.

Results & conclusions:

Direct adjustment by lasso regression and ensemble models involving multiple resamples have performance comparable to expert-based propensity scores and thus, may help provide real-time EHR-based evidence for timely clinical decisions.

Effect of dementia on outcomes of elderly patients with hemorrhagic peptic ulcer disease based on a national administrative database

BACKGROUND

Little information is available on the effect of dementia on outcomes of elderly patients with hemorrhagic peptic ulcer disease at the population level.

AIMS

This study aimed to investigate the effect of dementia on outcomes of elderly patients with hemorrhagic peptic ulcer based on a national administrative database.

METHODS

A total of 14,569 elderly patients (≥80 years) who were treated by endoscopic hemostasis for hemorrhagic peptic ulcer were referred to 1073 hospitals between 2010 and 2012 in Japan. We collected patients' data from the administrative database to compare clinical and medical economic outcomes of elderly patients with hemorrhagic peptic ulcers. Patients were divided into two groups according to the presence of dementia: patients with dementia (n = 695) and those without dementia (n = 13,874).

RESULTS

There were no significant differences in in-hospital mortality within 30 days and overall mortality between the groups (odds ratio; OR 1.00, 95 % confidence interval; CI 0.68-1.46, p = 0.986 and OR 1.02, 95 % CI 0.74-1.41, p = 0.877). However, the length of stay (LOS) and medical costs during hospitalization were significantly higher in patients with dementia compared with those without dementia. The unstandardized coefficient for LOS was 3.12 days (95 % CI 1.58-4.67 days, p < 0.001), whereas that for medical costs was 1171.7 US dollars (95 % CI 533.8-1809.5 US dollars, p < 0.001).

CONCLUSIONS

Length of stay and medical costs during hospitalization are significantly increased in elderly patients with dementia undergoing endoscopic hemostasis for hemorrhagic peptic ulcer disease.

Estimating the safety effects of lane widths on urban streets in Nebraska using the propensity scores-potential outcomes framework

A sufficient understanding of the safety impact of lane widths in urban areas is necessary to produce geometric designs that optimize safety performance for all users. The overarching trend found in the research literature is that as lane widths narrow, crash frequency increases. However, this trend is inconsistent and is the result of multiple cross-sectional studies that have issues related to lack of control for potential confounding variables, unobserved heterogeneity or omitted variable bias, or endogeneity among independent variables, among others. Using ten years of mid-block crash data on urban arterials and collectors from four cities in Nebraska, crash modification factors (CMFs) were estimated for various lane widths and crash types. These CMFs were developed using the propensity scores-potential outcomes methodology. This method reduces many of the issues associated with cross-sectional regression models when estimating the safety effects of infrastructure-related design features. Generalized boosting, a non-parametric modeling technique, was used to estimate the propensity scores. Matching was performed using both Nearest Neighbor and Mahalanobis matching techniques. CMF estimation was done using mixed-effects negative binomial or Poisson regression with the matched data. Lane widths included in the analysis included 9ft, 10ft, 11ft, and 12ft. Some of the estimated CMFs were point estimates while others were functions of traffic volume (i.e., the CMF changed depending on the traffic volume). Roadways with 10ft travel lanes were found to experience the highest crash frequency relative to other lane widths. Meanwhile, roads with 9ft travel lanes were found to experience the lowest relative crash frequency. While this may be due to increased driver caution when traveling on narrow lanes, it is possible that unobserved factors influenced this result. CMFs for target crash types (sideswipe same-direction and sideswipe opposite-direction) were consistent with the values currently used in the Highway Safety Manual (HSM).

Effects of proton pump inhibitor on outcomes of patients with severe acute pancreatitis based on a national administrative database

OBJECTIVE

This study aimed to investigate whether proton pump inhibitors (PPIs) affect the outcomes of patients with severe acute pancreatitis based on a national administrative database.

METHODS

A total of 10,400 patients with severe acute pancreatitis were referred to 1021 hospitals between 2010 and 2012 in Japan. Patients were divided into two groups: patients who used PPIs (n = 3879) and those without PPIs (n = 6521). We collected patients' data from the administrative database to compare in-hospital mortality within 7, 14, and 28 days, and overall in-hospital mortality between groups, using propensity score analysis to adjust for treatment selection bias.

RESULTS

Multiple logistic regression showed that use of PPIs did not affect in-hospital mortality within 7 and 14 days. The odds ratio (OR) for mortality within 7 days was 1.14 (95% confidence interval [CI]: 0.91-1.42, p = 0.236) while that within 14 days was 1.10 (95% CI: 0.89-1.35, p = 0.349). No significant association was observed for in-hospital mortality within 28 days and overall in-hospital mortality (OR for within 28 days: 1.12, 95% CI: 0.92-1.37, p = 0.224; OR for overall in-hospital mortality: 1.42, 95% CI: 0.97-1.87, p = 0.065).

CONCLUSIONS

This study shows that use of PPIs does not affect clinical outcomes of patients with severe acute pancreatitis. Prospective or randomized studies are needed to confirm the efficacy of PPIs on outcomes of patients with severe acute pancreatitis in the future.

Imputation approaches for potential outcomes in causal inference

BACKGROUND

The fundamental problem of causal inference is one of missing data, and specifically of missing potential outcomes: if potential outcomes were fully observed, then causal inference could be made trivially. Though often not discussed explicitly in the epidemiological literature, the connections between causal inference and missing data can provide additional intuition.

METHODS

We demonstrate how we can approach causal inference in ways similar to how we address all problems of missing data, using multiple imputation and the parametric g-formula.

RESULTS

We explain and demonstrate the use of these methods in example data, and discuss implications for more traditional approaches to causal inference.

CONCLUSIONS

Though there are advantages and disadvantages to both multiple imputation and g-formula approaches, epidemiologists can benefit from thinking about their causal inference problems as problems of missing data, as such perspectives may lend new and clarifying insights to their analyses.