Plasmode simulation for the evaluation of pharmacoepidemiologic methods in complex healthcare databases

Evaluating Flexible Modeling of Continuous Covariates in Inverse-Weighted Estimators

Correct specification of the exposure model is essential for unbiased estimation in marginal structural models with inverse-probability-of-treatment weights. However, although flexible modeling is commonplace when estimating effects of continuous covariates in outcome models, its use is less frequent in estimation of inverse probability weights. Using simulations, we assess the accuracy of the treatment effect estimates and covariate balance obtained with different exposure model specifications when the true relationship between a continuous, possibly time-varying covariate Lt and the logit of the probability of exposure is nonlinear. Specifically, we compare 4 approaches to modeling the effect of Lt when estimating inverse probability weights: a linear function, the covariate-balancing propensity score, and 2 easy-to-implement flexible methods that relax the assumption of linearity: cubic regression splines and fractional polynomials. Using data from 2 empirical studies, we compare linear exposure models with flexible exposure models to estimate the effect of sustained virological response to hepatitis C virus treatment on the progression of liver fibrosis. Our simulation results demonstrate that ignoring important nonlinear relationships when fitting the exposure model may provide poorer covariate balance and induce substantial bias in the estimated exposure-outcome associations. Analysts should routinely consider flexible modeling of continuous covariates when estimating inverse-probability-of-treatment weights.

Combining the Power of Artificial Intelligence with the Richness of Healthcare Claims Data: Opportunities and Challenges

Combinations of healthcare claims data with additional datasets provide large and rich sources of information. The dimensionality and complexity of these combined datasets can be challenging to handle with standard statistical analyses. However, recent developments in artificial intelligence (AI) have led to algorithms and systems that are able to learn and extract complex patterns from such data. AI has already been applied successfully to such combined datasets, with applications such as improving the insurance claim processing pipeline and reducing estimation biases in retrospective studies. Nevertheless, there is still the potential to do much more. The identification of complex patterns within high dimensional datasets may find new predictors for early onset of diseases or lead to a more proactive offering of personalized preventive services. While there are potential risks and challenges associated with the use of AI, these are not insurmountable. As with the introduction of any innovation, it will be necessary to be thoughtful and responsible as we increasingly apply AI methods in healthcare.

Evaluating the use of bootstrapping in cohort studies conducted with 1:1 propensity score matching-A plasmode simulation study

PURPOSE

Bootstrapping can account for uncertainty in propensity score (PS) estimation and matching processes in 1:1 PS-matched cohort studies. While theory suggests that the classical bootstrap can fail to produce proper coverage, practical impact of this theoretical limitation in settings typical to pharmacoepidemiology is not well studied.

METHODS

In a plasmode-based simulation study, we compared performance of the standard parametric approach, which ignores uncertainty in PS estimation and matching, with two bootstrapping methods. The first method only accounted for uncertainty introduced during the matching process (the observation resampling approach). The second method accounted for uncertainty introduced during both PS estimation and matching processes (the PS reestimation approach). Variance was estimated based on percentile and empirical standard errors, and treatment effect estimation was based on median and mean of the estimated treatment effects across 1000 bootstrap resamples. Two treatment prevalence scenarios (5% and 29%) across two treatment effect scenarios (hazard ratio of 1.0 and 2.0) were evaluated in 500 simulated cohorts of 10 000 patients each.

RESULTS

We observed that 95% confidence intervals from the bootstrapping approaches but not the standard approach, resulted in inaccurate coverage rates (98%-100% for the observation resampling approach, 99%-100% for the PS reestimation approach, and 95%-96% for standard approach). Treatment effect estimation based on bootstrapping approaches resulted in lower bias than the standard approach (less than 1.4% vs 4.1%) at 5% treatment prevalence; however, the performance was equivalent at 29% treatment prevalence.

CONCLUSION

Use of bootstrapping led to variance overestimation and inconsistent coverage, while coverage remained more consistent with parametric estimation.

Collaborative-controlled LASSO for constructing propensity score-based estimators in high-dimensional data

Propensity score-based estimators are increasingly used for causal inference in observational studies. However, model selection for propensity score estimation in high-dimensional data has received little attention. In these settings, propensity score models have traditionally been selected based on the goodness-of-fit for the treatment mechanism itself, without consideration of the causal parameter of interest. Collaborative minimum loss-based estimation is a novel methodology for causal inference that takes into account information on the causal parameter of interest when selecting a propensity score model. This "collaborative learning" considers variable associations with both treatment and outcome when selecting a propensity score model in order to minimize a bias-variance tradeoff in the estimated treatment effect. In this study, we introduce a novel approach for collaborative model selection when using the LASSO estimator for propensity score estimation in high-dimensional covariate settings. To demonstrate the importance of selecting the propensity score model collaboratively, we designed quasi-experiments based on a real electronic healthcare database, where only the potential outcomes were manually generated, and the treatment and baseline covariates remained unchanged. Results showed that the collaborative minimum loss-based estimation algorithm outperformed other competing estimators for both point estimation and confidence interval coverage. In addition, the propensity score model selected by collaborative minimum loss-based estimation could be applied to other propensity score-based estimators, which also resulted in substantive improvement for both point estimation and confidence interval coverage. We illustrate the discussed concepts through an empirical example comparing the effects of non-selective nonsteroidal anti-inflammatory drugs with selective COX-2 inhibitors on gastrointestinal complications in a population of Medicare beneficiaries.

Data Mining for Adverse Drug Events With a Propensity Score-matched Tree-based Scan Statistic

The tree-based scan statistic is a statistical data mining tool that has been used for signal detection with a self-controlled design in vaccine safety studies. This disproportionality statistic adjusts for multiple testing in evaluation of thousands of potential adverse events. However, many drug safety questions are not well suited for self-controlled analysis. We propose a method that combines tree-based scan statistics with propensity score-matched analysis of new initiator cohorts, a robust design for investigations of drug safety. We conducted plasmode simulations to evaluate performance. In multiple realistic scenarios, tree-based scan statistics in cohorts that were propensity score matched to adjust for confounding outperformed tree-based scan statistics in unmatched cohorts. In scenarios where confounding moved point estimates away from the null, adjusted analyses recovered the prespecified type 1 error while unadjusted analyses inflated type 1 error. In scenarios where confounding moved point estimates toward the null, adjusted analyses preserved power, whereas unadjusted analyses greatly reduced power. Although complete adjustment of true confounders had the best performance, matching on a moderately mis-specified propensity score substantially improved type 1 error and power compared with no adjustment. When there was true elevation in risk of an adverse event, there were often co-occurring signals for clinically related concepts. TreeScan with propensity score matching shows promise as a method for screening and prioritization of potential adverse events. It should be followed by clinical review and safety studies specifically designed to quantify the magnitude of effect, with confounding control targeted to the outcome of interest.

Scalable collaborative targeted learning for high-dimensional data

Robust inference of a low-dimensional parameter in a large semi-parametric model relies on external estimators of infinite-dimensional features of the distribution of the data. Typically, only one of the latter is optimized for the sake of constructing a well-behaved estimator of the low-dimensional parameter of interest. Optimizing more than one of them for the sake of achieving a better bias-variance trade-off in the estimation of the parameter of interest is the core idea driving the general template of the collaborative targeted minimum loss-based estimation procedure. The original instantiation of the collaborative targeted minimum loss-based estimation template can be presented as a greedy forward stepwise collaborative targeted minimum loss-based estimation algorithm. It does not scale well when the number p of covariates increases drastically. This motivates the introduction of a novel instantiation of the collaborative targeted minimum loss-based estimation template where the covariates are pre-ordered. Its time complexity is O ( p ) as opposed to the original O ( p 2 ) , a remarkable gain. We propose two pre-ordering strategies and suggest a rule of thumb to develop other meaningful strategies. Because it is usually unclear a priori which pre-ordering strategy to choose, we also introduce another instantiation called SL-C-TMLE algorithm that enables the data-driven choice of the better pre-ordering strategy given the problem at hand. Its time complexity is O ( p ) as well. The computational burden and relative performance of these algorithms were compared in simulation studies involving fully synthetic data or partially synthetic data based on a real world large electronic health database; and in analyses of three real, large electronic health databases. In all analyses involving electronic health databases, the greedy collaborative targeted minimum loss-based estimation algorithm is unacceptably slow. Simulation studies seem to indicate that our scalable collaborative targeted minimum loss-based estimation and SL-C-TMLE algorithms work well. All C-TMLEs are publicly available in a Julia software package.

Can We Train Machine Learning Methods to Outperform the High-dimensional Propensity Score Algorithm?

The use of retrospective health care claims datasets is frequently criticized for the lack of complete information on potential confounders. Utilizing patient's health status-related information from claims datasets as surrogates or proxies for mismeasured and unobserved confounders, the high-dimensional propensity score algorithm enables us to reduce bias. Using a previously published cohort study of postmyocardial infarction statin use (1998-2012), we compare the performance of the algorithm with a number of popular machine learning approaches for confounder selection in high-dimensional covariate spaces: random forest, least absolute shrinkage and selection operator, and elastic net. Our results suggest that, when the data analysis is done with epidemiologic principles in mind, machine learning methods perform as well as the high-dimensional propensity score algorithm. Using a plasmode framework that mimicked the empirical data, we also showed that a hybrid of machine learning and high-dimensional propensity score algorithms generally perform slightly better than both in terms of mean squared error, when a bias-based analysis is used.

Should a propensity score model be super? The utility of ensemble procedures for causal adjustment

In investigations of the effect of treatment on outcome, the propensity score is a tool to eliminate imbalance in the distribution of confounding variables between treatment groups. Recent work has suggested that Super Learner, an ensemble method, outperforms logistic regression in nonlinear settings; however, experience with real-data analyses tends to show overfitting of the propensity score model using this approach. We investigated a wide range of simulated settings of varying complexities including simulations based on real data to compare the performances of logistic regression, generalized boosted models, and Super Learner in providing balance and for estimating the average treatment effect via propensity score regression, propensity score matching, and inverse probability of treatment weighting. We found that Super Learner and logistic regression are comparable in terms of covariate balance, bias, and mean squared error (MSE); however, Super Learner is computationally very expensive thus leaving no clear advantage to the more complex approach. Propensity scores estimated by generalized boosted models were inferior to the other two estimation approaches. We also found that propensity score regression adjustment was superior to either matching or inverse weighting when the form of the dependence on the treatment on the outcome is correctly specified.

The Number of Events per Confounder for Valid Estimation of Risk Difference Using Modified Least-Squares Regression

Risk difference is a relevant effect measure in epidemiologic research. Although it is well known that when there are few events per confounder, logistic regression is not suitable for confounding control, it is not clear how many events per confounder are required for valid estimation of risk difference using linear binomial models. Because the maximum likelihood method has a convergence problem, we investigated the number of events per confounder necessary to validly estimate risk difference using modified least-squares regression in a simulation. We simulated 864 scenarios, according to the number of confounders (2-20), the number of events per confounder (2-12), marginal risk (0.5%-40%), exposure proportion (20% and 40%), and 3 sizes of risk difference. Our simulation showed that modified least-squares regression provided unbiased risk difference-regardless of the number of events per confounder-and reliable confidence intervals when more than 5 events were expected in the exposed and in the unexposed, irrespective of the number of events per confounder. We illustrated the modified least-squares regression analysis using perinatal epidemiologic data. Modified least-squares regression is considered to be a useful analytical tool for rare binary outcomes relative to the number of confounders.

Propensity-score matching with competing risks in survival analysis

Propensity‐score matching is a popular analytic method to remove the effects of confounding due to measured baseline covariates when using observational data to estimate the effects of treatment. Time‐to‐event outcomes are common in medical research. Competing risks are outcomes whose occurrence precludes the occurrence of the primary time‐to‐event outcome of interest. All non‐fatal outcomes and all cause‐specific mortality outcomes are potentially subject to competing risks. There is a paucity of guidance on the conduct of propensity‐score matching in the presence of competing risks. We describe how both relative and absolute measures of treatment effect can be obtained when using propensity‐score matching with competing risks data. Estimates of the relative effect of treatment can be obtained by using cause‐specific hazard models in the matched sample. Estimates of absolute treatment effects can be obtained by comparing cumulative incidence functions (CIFs) between matched treated and matched control subjects. We conducted a series of Monte Carlo simulations to compare the empirical type I error rate of different statistical methods for testing the equality of CIFs estimated in the matched sample. We also examined the performance of different methods to estimate the marginal subdistribution hazard ratio. We recommend that a marginal subdistribution hazard model that accounts for the within‐pair clustering of outcomes be used to test the equality of CIFs and to estimate subdistribution hazard ratios. We illustrate the described methods by using data on patients discharged from hospital with acute myocardial infarction to estimate the effect of discharge prescribing of statins on cardiovascular death.

Closed-form variance estimator for weighted propensity score estimators with survival outcome

Propensity score (PS) methods are widely used in observational studies for evaluating marginal treatment effects. PS-weighting is a popular PS-based method that allows for estimating both the average treatment effect on the overall population (ATE) and the average treatment effect on the treated population (ATT). Previous research has shown that the variance of the treatment effect is accurately estimated only if the variance estimator takes into account the fact that the propensity score is itself estimated from the available data in a first step of the analysis. In 2016, Austin showed that the bootstrap-based variance estimator was the only existing estimator resulting in approximately correct estimates of standard errors when evaluating a survival outcome and a Cox model was used to estimate a marginal hazard ratio (HR). This author stressed the need to develop a closed-form variance estimator of the marginal HR accounting for the estimation of the PS. In the present research, we developed such variance estimators both for the ATE and ATT. We evaluated their performance with an extensive simulation study and compared them to bootstrap-based variance estimators and to naive variance estimators that do not account for the estimation step. We found that the performance of the proposed variance estimators was similar to that of the bootstrap-based estimators. The proposed variance estimators provide an alternative to the bootstrap estimator, particularly interesting in situations in which time-consumption and/or reproducibility are an important issue. An implementation has been developed for the R software and is freely available (package hrIPW).

Relative Performance of Propensity Score Matching Strategies for Subgroup Analyses

Postapproval drug safety studies often use propensity scores (PSs) to adjust for a large number of baseline confounders. These studies may involve examining whether treatment safety varies across subgroups. There are many ways a PS could be used to adjust for confounding in subgroup analyses. These methods have trade-offs that are not well understood. We conducted a plasmode simulation to compare relative performance of 5 methods involving PS matching for subgroup analysis, including methods frequently used in applied literature whose performance has not been previously directly compared. These methods varied as to whether the overall PS, subgroup-specific PS, or no rematching was used in subgroup analysis as well as whether subgroups were fully nested within the main analytical cohort. The evaluated PS subgroup matching methods performed similarly in terms of balance, bias, and precision in 12 simulated scenarios varying size of the cohort, prevalence of exposure and outcome, strength of relationships between baseline covariates and exposure, the true effect within subgroups, and the degree of confounding within subgroups. Each had strengths and limitations with respect to other performance metrics that could inform choice of method.

Using Super Learner Prediction Modeling to Improve High-dimensional Propensity Score Estimation

The high-dimensional propensity score is a semiautomated variable selection algorithm that can supplement expert knowledge to improve confounding control in nonexperimental medical studies utilizing electronic healthcare databases. Although the algorithm can be used to generate hundreds of patient-level variables and rank them by their potential confounding impact, it remains unclear how to select the optimal number of variables for adjustment. We used plasmode simulations based on empirical data to discuss and evaluate data-adaptive approaches for variable selection and prediction modeling that can be combined with the high-dimensional propensity score to improve confounding control in large healthcare databases. We considered approaches that combine the high-dimensional propensity score with Super Learner prediction modeling, a scalable version of collaborative targeted maximum-likelihood estimation, and penalized regression. We evaluated performance using bias and mean squared error (MSE) in effect estimates. Results showed that the high-dimensional propensity score can be sensitive to the number of variables included for adjustment and that severe overfitting of the propensity score model can negatively impact the properties of effect estimates. Combining the high-dimensional propensity score with Super Learner was the most consistent strategy, in terms of reducing bias and MSE in the effect estimates, and may be promising for semiautomated data-adaptive propensity score estimation in high-dimensional covariate datasets.

Automated data-adaptive analytics for electronic healthcare data to study causal treatment effects

BACKGROUND

Decision makers in health care increasingly rely on nonrandomized database analyses to assess the effectiveness, safety, and value of medical products. Health care data scientists use data-adaptive approaches that automatically optimize confounding control to study causal treatment effects. This article summarizes relevant experiences and extensions.

METHODS

The literature was reviewed on the uses of high-dimensional propensity score (HDPS) and related approaches for health care database analyses, including methodological articles on their performance and improvement. Articles were grouped into applications, comparative performance studies, and statistical simulation experiments.

RESULTS

The HDPS algorithm has been referenced frequently with a variety of clinical applications and data sources from around the world. The appeal of HDPS for database research rests in 1) its superior performance in situations of unobserved confounding through proxy adjustment, 2) its predictable efficiency in extracting confounding information from a given data source, 3) its ability to automate estimation of causal treatment effects to the extent achievable in a given data source, and 4) its independence of data source and coding system. Extensions of the HDPS approach have focused on improving variable selection when exposure is sparse, using free text information and time-varying confounding adjustment.

CONCLUSION

Semiautomated and optimized confounding adjustment in health care database analyses has proven successful across a wide range of settings. Machine-learning extensions further automate its use in estimating causal treatment effects across a range of data scenarios.

Assessing the performance of the generalized propensity score for estimating the effect of quantitative or continuous exposures on survival or time-to-event outcomes

Propensity score methods are frequently used to estimate the effects of interventions using observational data. The propensity score was originally developed for use with binary exposures. The generalized propensity score (GPS) is an extension of the propensity score for use with quantitative or continuous exposures (e.g. pack-years of cigarettes smoked, dose of medication, or years of education). We describe how the GPS can be used to estimate the effect of continuous exposures on survival or time-to-event outcomes. To do so we modified the concept of the dose–response function for use with time-to-event outcomes. We used Monte Carlo simulations to examine the performance of different methods of using the GPS to estimate the effect of quantitative exposures on survival or time-to-event outcomes. We examined covariate adjustment using the GPS and weighting using weights based on the inverse of the GPS. The use of methods based on the GPS was compared with the use of conventional G-computation and weighted G-computation. Conventional G-computation resulted in estimates of the dose–response function that displayed the lowest bias and the lowest variability. Amongst the two GPS-based methods, covariate adjustment using the GPS tended to have the better performance. We illustrate the application of these methods by estimating the effect of average neighbourhood income on the probability of survival following hospitalization for an acute myocardial infarction.

Understanding and diagnosing the potential for bias when using machine learning methods with doubly robust causal estimators

Data-adaptive methods have been proposed to estimate nuisance parameters when using doubly robust semiparametric methods for estimating marginal causal effects. However, in the presence of near practical positivity violations, these methods can produce a separation of the two exposure groups in terms of propensity score densities which can lead to biased estimates of the treatment effect. To motivate the problem, we evaluated the Targeted Minimum Loss-based Estimation procedure using a simulation scenario to estimate the average treatment effect. We highlight the divergence in estimates obtained when using parametric and data-adaptive methods to estimate the propensity score. We then adapted an existing diagnostic tool based on a bootstrap resampling of the subjects and simulation of the outcome data in order to show that the estimation using data-adaptive methods for the propensity score in this study may lead to large bias and poor coverage. The adapted bootstrap procedure is able to identify this instability and can be used as a diagnostic tool.

Variable Selection for Confounding Adjustment in High-dimensional Covariate Spaces When Analyzing Healthcare Databases

BACKGROUND

Data-adaptive approaches to confounding adjustment may improve performance beyond expert knowledge when analyzing electronic healthcare databases and have additional practical advantages for analyzing multiple databases in rapid cycles. Improvements seemed possible if outcome predictors were reliably identified empirically and adjusted.

METHODS

In five cohort studies from diverse healthcare databases, we implemented a base-case high-dimensional propensity score algorithm with propensity score decile-adjusted outcome models to estimate treatment effects among prescription drug initiators. The original variable selection procedure based on the estimated bias of each variable using unadjusted associations between confounders and exposure (RRCE) and disease outcome (RRCD) was augmented by alternative strategies. These included using increasingly adjusted RRCD estimates, including models considering >1,500 variables jointly (Lasso, Bayesian logistic regression); using prediction statistics or likelihood-ratio statistics for covariate prioritization; directly estimating the propensity score with >1,500 variables (Lasso, Bayesian regression); or directly fitting an outcome model using all covariates jointly (Lasso, Ridge).

RESULTS

In five example studies, most tested augmentations of the base-case hdPS did not meaningfully change estimates in light of wide confidence intervals except for Bayesian regression and Lasso to estimate RRCD, which moved estimates minimally closer to the expectation in three of five examples. The direct outcome estimation with Lasso performed worst.

CONCLUSION

Overall, the basic heuristic of variable reduction in high-dimensional propensity score adjustment performed, as well as alternative approaches in diverse settings. Minor improvements in variable selection may be possible using Bayesian outcome regression to prioritize variables for propensity score estimation when outcomes are rare. See video abstract at, http://links.lww.com/EDE/B162.

Extending the Case-Control Design to Longitudinal Data: Stratified Sampling Based on Repeated Binary Outcomes

We detail study design options that generalize case-control sampling when longitudinal outcome data are already collected as part of a primary cohort study, but new exposure data must be retrospectively processed for a secondary analysis. Furthermore, we assume that cost will limit the size of the subsample that can be evaluated. We describe a novel class of stratified outcome-dependent sampling designs for longitudinal binary response data where distinct strata are created for subjects who never, sometimes, and always experienced the event of interest during longitudinal follow-up. Individual designs within this class are differentiated by the stratum-specific sampling probabilities. We show for parameters associated with time-varying exposures, subjects who experience the event/outcome at some but not at all of the follow-up times (i.e., those who exhibit response variation) are highly informative. If the time-varying exposure varies exclusively within individuals (i.e., intraclass correlation coefficient is 0), then sampling all subjects with response variability can yield highly precise parameter estimates even when compared with an analysis of the original cohort. The flexibility of the designs and analysis procedures also permits estimation of parameters that correspond to time-fixed covariates, and we show that with an imputation-based estimation procedure, baseline covariate associations can be estimated with very high precision irrespective of the design. We demonstrate features of the designs and analysis procedures via a plasmode simulation using data from the Lung Health Study.

Outcome-related, Auxiliary Variable Sampling Designs for Longitudinal Binary Data

BACKGROUND

Epidemiologists have long used case-control and related study designs to enhance variability of response and information available to estimate exposure-disease associations. Less has been done for longitudinal data.

METHODS

We discuss an epidemiological study design and analysis approach for longitudinal binary response data. We seek to gain statistical efficiency by oversampling relatively informative subjects for inclusion into the sample. In this methodological demonstration, we develop this concept by sampling repeatedly from an existing cohort study to estimate the relationship of chronic obstructive pulmonary disease to past-year smoking in a panel of baseline smokers. To account for oversampling, we describe a sequential offsetted regressions approach for valid inferences in this setting.

RESULTS

Targeted sampling can lead to increased statistical efficiency when combined with sequential offsetted regressions. Efficiency gains are degraded with increased prevalence of the disease response variable, with decreased association between the sampling variable and the response, and with other design and analysis parameters, providing guidance to those wishing to use these types of designs in the future.

CONCLUSIONS

These designs hold promise for efficient use of resources in longitudinal cohort studies.

Synergies From Integrating Randomized Controlled Trials and Real-World Data Analyses

Analyses using administrative claims databases or national registries provide estimates of benefits and harms of medications in real-world settings for large and diverse patient populations. Whereas claims-based nonrandomized studies and randomized-controlled trials (RCTs) have distinct limitations, their strengths are complementary. Integrating RCT and claims data offers substantial synergies. We propose obtaining routinely collected longitudinal claims data from RCT participants and discuss the added value of the novel evidence that can be derived from this "information overlap."

Using Multiple Pharmacovigilance Models Improves the Timeliness of Signal Detection in Simulated Prospective Surveillance

INTRODUCTION

Prospective pharmacovigilance aims to rapidly detect safety concerns related to medical products. The exposure model selected for pharmacovigilance impacts the timeliness of signal detection. However, in most real-life pharmacovigilance studies, little is known about which model correctly represents the association and there is no evidence to guide the selection of an exposure model. Different exposure models reflect different aspects of exposure history, and their relevance varies across studies. Therefore, one potential solution is to apply several alternative exposure models simultaneously, with each model assuming a different exposure-risk association, and then combine the model results.

METHODS

We simulated alternative clinically plausible associations between time-varying drug exposure and the hazard of an adverse event. Prospective surveillance was conducted on the simulated data by estimating parametric and semi-parametric exposure-risk models at multiple times during follow-up. For each model separately, and using combined evidence from different subsets of models, we compared the time to signal detection.

RESULTS

Timely detection across the simulated associations was obtained by fitting a set of pharmacovigilance models. This set included alternative parametric models that assumed different exposure-risk associations and flexible models that made no assumptions regarding the form/shape of the association. Times to detection generated using a simple combination of evidence from multiple models were comparable to those observed under the ideal, but unrealistic, scenario where pharmacovigilance relied on the single 'true' model used for data generation.

CONCLUSIONS

Simulation results indicate that, if the true model is not known, an association can be detected in a more timely manner by first fitting a carefully selected set of exposure-risk models and then generating a signal as soon as any of the models considered yields a test statistic value below a predetermined testing threshold.

Conducting Privacy-Preserving Multivariable Propensity Score Analysis When Patient Covariate Information Is Stored in Separate Locations

Distributed networks of health-care data sources are increasingly being utilized to conduct pharmacoepidemiologic database studies. Such networks may contain data that are not physically pooled but instead are distributed horizontally (separate patients within each data source) or vertically (separate measures within each data source) in order to preserve patient privacy. While multivariable methods for the analysis of horizontally distributed data are frequently employed, few practical approaches have been put forth to deal with vertically distributed health-care databases. In this paper, we propose 2 propensity score-based approaches to vertically distributed data analysis and test their performance using 5 example studies. We found that these approaches produced point estimates close to what could be achieved without partitioning. We further found a performance benefit (i.e., lower mean squared error) for sequentially passing a propensity score through each data domain (called the "sequential approach") as compared with fitting separate domain-specific propensity scores (called the "parallel approach"). These results were validated in a small simulation study. This proof-of-concept study suggests a new multivariable analysis approach to vertically distributed health-care databases that is practical, preserves patient privacy, and warrants further investigation for use in clinical research applications that rely on health-care databases.

Effect Estimation in Point-Exposure Studies with Binary Outcomes and High-Dimensional Covariate Data - A Comparison of Targeted Maximum Likelihood Estimation and Inverse Probability of Treatment Weighting

Inverse probability of treatment weighting (IPW) and targeted maximum likelihood estimation (TMLE) are relatively new methods proposed for estimating marginal causal effects. TMLE is doubly robust, yielding consistent estimators even under misspecification of either the treatment or the outcome model. While IPW methods are known to be sensitive to near violations of the practical positivity assumption (e. g., in the case of data sparsity), the consequences of this violation in the TMLE framework for binary outcomes have been less widely investigated. As near practical positivity violations are particularly likely in high-dimensional covariate settings, a better understanding of the performance of TMLE is of particular interest for pharmcoepidemiological studies using large databases. Using plasmode and Monte-Carlo simulation studies, we evaluated the performance of TMLE compared to that of IPW estimators based on a point-exposure cohort study of the marginal causal effect of post-myocardial infarction statin use on the 1-year risk of all-cause mortality from the Clinical Practice Research Datalink. A variety of treatment model specifications were considered, inducing different degrees of near practical non-positivity. Our simulation study showed that the performance of the TMLE and IPW estimators were comparable when the dimension of the fitted treatment model was small to moderate; however, they differed when a large number of covariates was considered. When a rich outcome model was included in the TMLE, estimators were unbiased. In some cases, we found irregular bias and large standard errors with both methods even with a correctly specified high-dimensional treatment model. The IPW estimator showed a slightly better root MSE with high-dimensional treatment model specifications in our simulation setting. In conclusion, for estimation of the marginal expectation of the outcome under a fixed treatment, TMLE and IPW estimators employing the same treatment model specification may perform differently due to differential sensitivity to practical positivity violations; however, TMLE, being doubly robust, shows improved performance with richer specifications of the outcome model. Although TMLE is appealing for its double robustness property, such violations in a high-dimensional covariate setting are problematic for both methods.

Propensity score model overfitting led to inflated variance of estimated odds ratios

OBJECTIVE

Simulation studies suggest that the ratio of the number of events to the number of estimated parameters in a logistic regression model should be not less than 10 or 20 to 1 to achieve reliable effect estimates. Applications of propensity score approaches for confounding control in practice, however, do often not consider these recommendations.

STUDY DESIGN AND SETTING

We conducted extensive Monte Carlo and plasmode simulation studies to investigate the impact of propensity score model overfitting on the performance in estimating conditional and marginal odds ratios using different established propensity score inference approaches. We assessed estimate accuracy and precision as well as associated type I error and type II error rates in testing the null hypothesis of no exposure effect.

RESULTS

For all inference approaches considered, our simulation study revealed considerably inflated standard errors of effect estimates when using overfitted propensity score models. Overfitting did not considerably affect type I error rates for most inference approaches. However, because of residual confounding, estimation performance and type I error probabilities were unsatisfactory when using propensity score quintile adjustment.

CONCLUSION

Overfitting of propensity score models should be avoided to obtain reliable estimates of treatment or exposure effects in individual studies.

Performance of the high-dimensional propensity score in adjusting for unmeasured confounders

Purpose

High-dimensional propensity scores (hdPS) can adjust for measured confounders, but it remains unclear how well it can adjust for unmeasured confounders. Our goal was to identify if the hdPS method could adjust for confounders which were hidden to the hdPS algorithm.

Method

The hdPS algorithm was used to estimate two hdPS; the first version (hdPS-1) was estimated using data provided by 6 data dimensions and the second version (hdPS-2) was estimated using data provided from only two of the 6 data dimensions. Two matched sub-cohorts were created by matching one patient initiated on a high-dose statin to one patient initiated on a low-dose statin based on either hdPS-1 (Matched hdPS Full Info Sub-Cohort) or hdPS-2 (Matched hdPS Hidden Info Sub-Cohort). Performances of both hdPS were compared by means of the absolute standardized differences (ASDD) regarding 18 characteristics (data on seven of the 18 characteristics were hidden to the hdPS algorithm when estimating the hdPS-2).

Results

Eight out of the 18 characteristics were shown to be unbalanced within the unmatched cohort. Matching on either hdPS achieved adequate balance (i.e., ASDD <0.1) on all 18 characteristics.

Conclusion

Our results indicate that the hdPS method was able to adjust for hidden confounders supporting the claim that the hdPS method can adjust for at least some unmeasured confounders.

Variance estimation when using inverse probability of treatment weighting (IPTW) with survival analysis

Propensity score methods are used to reduce the effects of observed confounding when using observational data to estimate the effects of treatments or exposures. A popular method of using the propensity score is inverse probability of treatment weighting (IPTW). When using this method, a weight is calculated for each subject that is equal to the inverse of the probability of receiving the treatment that was actually received. These weights are then incorporated into the analyses to minimize the effects of observed confounding. Previous research has found that these methods result in unbiased estimation when estimating the effect of treatment on survival outcomes. However, conventional methods of variance estimation were shown to result in biased estimates of standard error. In this study, we conducted an extensive set of Monte Carlo simulations to examine different methods of variance estimation when using a weighted Cox proportional hazards model to estimate the effect of treatment. We considered three variance estimation methods: (i) a naïve model-based variance estimator; (ii) a robust sandwich-type variance estimator; and (iii) a bootstrap variance estimator. We considered estimation of both the average treatment effect and the average treatment effect in the treated. We found that the use of a bootstrap estimator resulted in approximately correct estimates of standard errors and confidence intervals with the correct coverage rates. The other estimators resulted in biased estimates of standard errors and confidence intervals with incorrect coverage rates. Our simulations were informed by a case study examining the effect of statin prescribing on mortality. © 2016 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.

Head to head comparison of the propensity score and the high-dimensional propensity score matching methods

BACKGROUND

Comparative performance of the traditional propensity score (PS) and high-dimensional propensity score (hdPS) methods in the adjustment for confounding by indication remains unclear. We aimed to identify which method provided the best adjustment for confounding by indication within the context of the risk of diabetes among patients exposed to moderate versus high potency statins.

METHOD

A cohort of diabetes-free incident statins users was identified from the Quebec's publicly funded medico-administrative database (Full Cohort). We created two matched sub-cohorts by matching one patient initiated on a lower potency to one patient initiated on a high potency either on patients' PS or hdPS. Both methods' performance were compared by means of the absolute standardized differences (ASDD) regarding relevant characteristics and by means of the obtained measures of association.

RESULTS

Eight out of the 18 examined characteristics were shown to be unbalanced within the Full Cohort. Although matching on either method achieved balance within all examined characteristic, matching on patients' hdPS created the most balanced sub-cohort. Measures of associations and confidence intervals obtained within the two matched sub-cohorts overlapped.

CONCLUSION

Although ASDD suggest better matching with hdPS than with PS, measures of association were almost identical when adjusted for either method. Use of the hdPS method in adjusting for confounding by indication within future studies should be recommended due to its ability to identify confounding variables which may be unknown to the investigators.

The Comparison of Matching Methods Using Different Measures of Balance: Benefits and Risks Exemplified within a Study to Evaluate the Effects of German Disease Management Programs on Long-Term Outcomes of Patients with Type 2 Diabetes

OBJECTIVE

To present a case study on how to compare various matching methods applying different measures of balance and to point out some pitfalls involved in relying on such measures.

DATA SOURCES

Administrative claims data from a German statutory health insurance fund covering the years 2004-2008.

STUDY DESIGN

We applied three different covariance balance diagnostics to a choice of 12 different matching methods used to evaluate the effectiveness of the German disease management program for type 2 diabetes (DMPDM2). We further compared the effect estimates resulting from applying these different matching techniques in the evaluation of the DMPDM2.

PRINCIPAL FINDINGS

The choice of balance measure leads to different results on the performance of the applied matching methods. Exact matching methods performed well across all measures of balance, but resulted in the exclusion of many observations, leading to a change of the baseline characteristics of the study sample and also the effect estimate of the DMPDM2. All PS-based methods showed similar effect estimates. Applying a higher matching ratio and using a larger variable set generally resulted in better balance. Using a generalized boosted instead of a logistic regression model showed slightly better performance for balance diagnostics taking into account imbalances at higher moments.

CONCLUSION

Best practice should include the application of several matching methods and thorough balance diagnostics. Applying matching techniques can provide a useful preprocessing step to reveal areas of the data that lack common support. The use of different balance diagnostics can be helpful for the interpretation of different effect estimates found with different matching methods.

Regularized Regression Versus the High-Dimensional Propensity Score for Confounding Adjustment in Secondary Database Analyses

Selection and measurement of confounders is critical for successful adjustment in nonrandomized studies. Although the principles behind confounder selection are now well established, variable selection for confounder adjustment remains a difficult problem in practice, particularly in secondary analyses of databases. We present a simulation study that compares the high-dimensional propensity score algorithm for variable selection with approaches that utilize direct adjustment for all potential confounders via regularized regression, including ridge regression and lasso regression. Simulations were based on 2 previously published pharmacoepidemiologic cohorts and used the plasmode simulation framework to create realistic simulated data sets with thousands of potential confounders. Performance of methods was evaluated with respect to bias and mean squared error of the estimated effects of a binary treatment. Simulation scenarios varied the true underlying outcome model, treatment effect, prevalence of exposure and outcome, and presence of unmeasured confounding. Across scenarios, high-dimensional propensity score approaches generally performed better than regularized regression approaches. However, including the variables selected by lasso regression in a regular propensity score model also performed well and may provide a promising alternative variable selection method.

Optimal full matching for survival outcomes: a method that merits more widespread use

Matching on the propensity score is a commonly used analytic method for estimating the effects of treatments on outcomes. Commonly used propensity score matching methods include nearest neighbor matching and nearest neighbor caliper matching. Rosenbaum (1991) proposed an optimal full matching approach, in which matched strata are formed consisting of either one treated subject and at least one control subject or one control subject and at least one treated subject. Full matching has been used rarely in the applied literature. Furthermore, its performance for use with survival outcomes has not been rigorously evaluated. We propose a method to use full matching to estimate the effect of treatment on the hazard of the occurrence of the outcome. An extensive set of Monte Carlo simulations were conducted to examine the performance of optimal full matching with survival analysis. Its performance was compared with that of nearest neighbor matching, nearest neighbor caliper matching, and inverse probability of treatment weighting using the propensity score. Full matching has superior performance compared with that of the two other matching algorithms and had comparable performance with that of inverse probability of treatment weighting using the propensity score. We illustrate the application of full matching with survival outcomes to estimate the effect of statin prescribing at hospital discharge on the hazard of post‐discharge mortality in a large cohort of patients who were discharged from hospital with a diagnosis of acute myocardial infarction. Optimal full matching merits more widespread adoption in medical and epidemiological research. © 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.