Plasmode simulation for the evaluation of pharmacoepidemiologic methods in complex healthcare databases

Comparison of two propensity score-based methods for balancing covariates: the overlap weighting and fine stratification methods in real-world claims data

BACKGROUND

Two propensity score (PS) based balancing covariate methods, the overlap weighting method (OW) and the fine stratification method (FS), produce superb covariate balance. OW has been compared with various weighting methods while FS has been compared with the traditional stratification method and various matching methods. However, no study has yet compared OW and FS. In addition, OW has not yet been evaluated in large claims data with low prevalence exposure and with low frequency outcomes, a context in which optimal use of balancing methods is critical. In the study, we aimed to compare OW and FS using real-world data and simulations with low prevalence exposure and with low frequency outcomes.

METHODS

We used the Texas State Medicaid claims data on adult beneficiaries with diabetes in 2012 as an empirical example (N = 42,628). Based on its real-world research question, we estimated an average treatment effect of health center vs. non-health center attendance in the total population. We also performed simulations to evaluate their relative performance. To preserve associations between covariates, we used the plasmode approach to simulate outcomes and/or exposures with N = 4,000. We simulated both homogeneous and heterogeneous treatment effects with various outcome risks (1-30% or observed: 27.75%) and/or exposure prevalence (2.5-30% or observed:10.55%). We used a weighted generalized linear model to estimate the exposure effect and the cluster-robust standard error (SE) method to estimate its SE.

RESULTS

In the empirical example, we found that OW had smaller standardized mean differences in all covariates (range: OW: 0.0-0.02 vs. FS: 0.22-3.26) and Mahalanobis balance distance (MB) (< 0.001 vs. > 0.049) than FS. In simulations, OW also achieved smaller MB (homogeneity: <0.04 vs. > 0.04; heterogeneity: 0.0-0.11 vs. 0.07-0.29), relative bias (homogeneity: 4.04-56.20 vs. 20-61.63; heterogeneity: 7.85-57.6 vs. 15.0-60.4), square root of mean squared error (homogeneity: 0.332-1.308 vs. 0.385-1.365; heterogeneity: 0.263-0.526 vs 0.313-0.620), and coverage probability (homogeneity: 0.0-80.4% vs. 0.0-69.8%; heterogeneity: 0.0-97.6% vs. 0.0-92.8%), than FS, in most cases.

CONCLUSIONS

These findings suggest that OW can yield nearly perfect covariate balance and therefore enhance the accuracy of average treatment effect estimation in the total population.

A Principled Approach to Characterize and Analyze Partially Observed Confounder Data from Electronic Health Records

Objective

Partially observed confounder data pose challenges to the statistical analysis of electronic health records (EHR) and systematic assessments of potentially underlying missingness mechanisms are lacking. We aimed to provide a principled approach to empirically characterize missing data processes and investigate performance of analytic methods.

Methods

Three empirical sub-cohorts of diabetic SGLT2 or DPP4-inhibitor initiators with complete information on HbA1c, BMI and smoking as confounders of interest (COI) formed the basis of data simulation under a plasmode framework. A true null treatment effect, including the COI in the outcome generation model, and four missingness mechanisms for the COI were simulated: completely at random (MCAR), at random (MAR), and two not at random (MNAR) mechanisms, where missingness was dependent on an unmeasured confounder and on the value of the COI itself. We evaluated the ability of three groups of diagnostics to differentiate between mechanisms: 1)-differences in characteristics between patients with or without the observed COI (using averaged standardized mean differences [ASMD]), 2)-predictive ability of the missingness indicator based on observed covariates, and 3)-association of the missingness indicator with the outcome. We then compared analytic methods including "complete case", inverse probability weighting, single and multiple imputation in their ability to recover true treatment effects.

Results

The diagnostics successfully identified characteristic patterns of simulated missingness mechanisms. For MAR, but not MCAR, the patient characteristics showed substantial differences (median ASMD 0.20 vs 0.05) and consequently, discrimination of the prediction models for missingness was also higher (0.59 vs 0.50). For MNAR, but not MAR or MCAR, missingness was significantly associated with the outcome even in models adjusting for other observed covariates. Comparing analytic methods, multiple imputation using a random forest algorithm resulted in the lowest root-mean-squared-error.

Conclusion

Principled diagnostics provided reliable insights into missingness mechanisms. When assumptions allow, multiple imputation with nonparametric models could help reduce bias.

A methodological review of the high-dimensional propensity score in comparative-effectiveness and safety-of-interventions research finds incomplete reporting relative to algorithm development and robustness

OBJECTIVES

The use of secondary databases has become popular for evaluating the effectiveness and safety of interventions in real-life settings. However, the absence of important confounders in these databases is challenging. To address this issue, the high-dimensional propensity score (hdPS) algorithm was developed in 2009. This algorithm uses proxy variables for mitigating confounding by combining information available across several healthcare dimensions. This study assessed the methodology and reporting of the hdPS in comparative effectiveness and safety research.

STUDY DESIGN AND SETTING

In this methodological review, we searched PubMed and Google Scholar from July 2009 to May 2022 for studies that used the hdPS for evaluating the effectiveness or safety of healthcare interventions. Two reviewers independently extracted study characteristics and assessed how the hdPS was applied and reported. Risk of bias was evaluated with the Risk Of Bias In Non-randomised Studies - of Interventions (ROBINS-I) tool.

RESULTS

In total, 136 studies met the inclusion criteria; the median publication year was 2018 (Q1-Q3 2016-2020). The studies included 192 datasets, mostly North American databases (n = 132, 69%). The hdPS was used in primary analysis in 120 studies (88%). Dimensions were defined in 101 studies (74%), with a median of 5 (Q1-Q3 4-6) dimensions included. A median of 500 (Q1-Q3 200-500) empirically identified covariates were selected. Regarding hdPS reporting, only 11 studies (8%) reported all recommended items. Most studies (n = 81, 60%) had a moderate overall risk of bias.

CONCLUSION

There is room for improvement in the reporting of hdPS studies, especially regarding the transparency of methodological choices that underpin the construction of the hdPS.

Challenges and best practices in omics benchmarking

Technological advances enabling massively parallel measurement of biological features - such as microarrays, high-throughput sequencing and mass spectrometry - have ushered in the omics era, now in its third decade. The resulting complex landscape of analytical methods has naturally fostered the growth of an omics benchmarking industry. Benchmarking refers to the process of objectively comparing and evaluating the performance of different computational or analytical techniques when processing and analysing large-scale biological data sets, such as transcriptomics, proteomics and metabolomics. With thousands of omics benchmarking studies published over the past 25 years, the field has matured to the point where the foundations of benchmarking have been established and well described. However, generating meaningful benchmarking data and properly evaluating performance in this complex domain remains challenging. In this Review, we highlight some common oversights and pitfalls in omics benchmarking. We also establish a methodology to bring the issues that can be addressed into focus and to be transparent about those that cannot: this takes the form of a spreadsheet template of guidelines for comprehensive reporting, intended to accompany publications. In addition, a survey of recent developments in benchmarking is provided as well as specific guidance for commonly encountered difficulties.

Statistical plasmode simulations-Potentials, challenges and recommendations

Statistical data simulation is essential in the development of statistical models and methods as well as in their performance evaluation. To capture complex data structures, in particular for high-dimensional data, a variety of simulation approaches have been introduced including parametric and the so-called plasmode simulations. While there are concerns about the realism of parametrically simulated data, it is widely claimed that plasmodes come very close to reality with some aspects of the "truth" known. However, there are no explicit guidelines or state-of-the-art on how to perform plasmode data simulations. In the present paper, we first review existing literature and introduce the concept of statistical plasmode simulation. We then discuss advantages and challenges of statistical plasmodes and provide a step-wise procedure for their generation, including key steps to their implementation and reporting. Finally, we illustrate the concept of statistical plasmodes as well as the proposed plasmode generation procedure by means of a public real RNA data set on breast carcinoma patients.

Characterizing Imbalance in the Tails of the Propensity Score Distribution

Understanding characteristics of patients with propensity scores in the tails of the propensity score (PS) distribution has relevance for inverse-probability-of-treatment-weighted and PS-based estimation in observational studies. Here we outline a method for identifying variables most responsible for extreme propensity scores. The approach is illustrated in 3 scenarios: 1) a plasmode simulation of adult patients in the National Ambulatory Medical Care Survey (2011-2015) and 2) timing of dexamethasone initiation and 3) timing of remdesivir initiation in patients hospitalized for coronavirus disease 2019 from February 2020 through January 2021. PS models were fitted using relevant baseline covariates, and tails of the PS distribution were defined using asymmetric first and 99th percentiles. After fitting of the PS model in each original data set, values of each key covariate were permuted and model-agnostic variable importance measures were examined. Visualization and variable importance techniques were helpful in identifying variables most responsible for extreme propensity scores and may help identify individual characteristics that might make patients inappropriate for inclusion in a study (e.g., off-label use). Subsetting or restricting the study sample based on variables identified using this approach may help investigators avoid the need for trimming or overlap weights in studies.

On the role of benchmarking data sets and simulations in method comparison studies

Method comparisons are essential to provide recommendations and guidance for applied researchers, who often have to choose from a plethora of available approaches. While many comparisons exist in the literature, these are often not neutral but favor a novel method. Apart from the choice of design and a proper reporting of the findings, there are different approaches concerning the underlying data for such method comparison studies. Most manuscripts on statistical methodology rely on simulation studies and provide a single real-world data set as an example to motivate and illustrate the methodology investigated. In the context of supervised learning, in contrast, methods are often evaluated using so-called benchmarking data sets, that is, real-world data that serve as gold standard in the community. Simulation studies, on the other hand, are much less common in this context. The aim of this paper is to investigate differences and similarities between these approaches, to discuss their advantages and disadvantages, and ultimately to develop new approaches to the evaluation of methods picking the best of both worlds. To this aim, we borrow ideas from different contexts such as mixed methods research and Clinical Scenario Evaluation.

Generating synthetic data from administrative health records for drug safety and effectiveness studies

Introduction

Administrative health records (AHRs) are used to conduct population-based post-market drug safety and comparative effectiveness studies to inform healthcare decision making. However, the cost of data extraction, and the challenges associated with privacy and securing approvals can make it challenging for researchers to conduct methodological research in a timely manner using real data. Generating synthetic AHRs that reasonably represent the real-world data are beneficial for developing analytic methods and training analysts to rapidly implement study protocols. We generated synthetic AHRs using two methods and compared these synthetic AHRs to real-world AHRs. We described the challenges associated with using synthetic AHRs for real-world study.

Methods

The real-world AHRs comprised prescription drug records for individuals with healthcare insurance coverage in the Population Research Data Repository (PRDR) from Manitoba, Canada for the 10-year period from 2008 to 2017. Synthetic data were generated using the Observational Medical Dataset Simulator II (OSIM2) and a modification (ModOSIM). Synthetic and real-world data were described using frequencies and percentages. Agreement of prescription drug use measures in PRDR, OSIM2 and ModOSIM was estimated with the concordance coefficient.

Results

The PRDR cohort included 169,586,633 drug records and 1,395 drug types for 1,604,734 individuals. Synthetic data for 1,000,000 individuals were generated using OSIM2 and ModOSIM. Sex and age group distributions were similar in the real-world and synthetic AHRs. However, there were significant differences in the number of drug records and number of unique drugs per person for OSIM2 and ModOSIM when compared with PRDR. For the average number of days of drug use, concordance with the PRDR was 16% (95% confidence interval [CI]: 12%-19%) for OSIM2 and 88% (95% CI: 87%-90%) for ModOSIM.

Conclusions

ModOSIM data were more similar to PRDR than OSIM2 data on many measures. Synthetic AHRs consistent with those found in real-world settings can be generated using ModOSIM. Synthetic data will benefit rapid implementation of methodological studies and data analyst training.

Longitudinal plasmode algorithms to evaluate statistical methods in realistic scenarios: an illustration applied to occupational epidemiology

INTRODUCTION

Plasmode simulations are a type of simulations that use real data to determine the synthetic data-generating equations. Such simulations thus allow evaluating statistical methods under realistic conditions. As far as we know, no plasmode algorithm has been proposed for simulating longitudinal data. In this paper, we propose a longitudinal plasmode framework to generate realistic data with both a time-varying exposure and time-varying covariates. This work was motivated by the objective of comparing different methods for estimating the causal effect of a cumulative exposure to psychosocial stressors at work over time.

METHODS

We developed two longitudinal plasmode algorithms: a parametric and a nonparametric algorithms. Data from the PROspective Québec (PROQ) Study on Work and Health were used as an input to generate data with the proposed plasmode algorithms. We evaluated the performance of multiple estimators of the parameters of marginal structural models (MSMs): inverse probability of treatment weighting, g-computation and targeted maximum likelihood estimation. These estimators were also compared to standard regression approaches with either adjustment for baseline covariates only or with adjustment for both baseline and time-varying covariates.

RESULTS

Standard regression methods were susceptible to yield biased estimates with confidence intervals having coverage probability lower than their nominal level. The bias was much lower and coverage of confidence intervals was much closer to the nominal level when considering MSMs. Among MSM estimators, g-computation overall produced the best results relative to bias, root mean squared error and coverage of confidence intervals. No method produced unbiased estimates with adequate coverage for all parameters in the more realistic nonparametric plasmode simulation.

CONCLUSION

The proposed longitudinal plasmode algorithms can be important methodological tools for evaluating and comparing analytical methods in realistic simulation scenarios. To facilitate the use of these algorithms, we provide R functions on GitHub. We also recommend using MSMs when estimating the effect of cumulative exposure to psychosocial stressors at work.

Validating an approach to overcome the immeasurable time bias in cohort studies: a real-world example and Monte Carlo simulation study

BACKGROUND

Immeasurable time bias arises from the lack of in-hospital medication information. It has been suggested that time-varying adjustment for hospitalization may minimize this potential bias. However, whereas we examined this issue in one case study, there remains a need to assess the validity of this approach in other settings.

METHODS

Using a Monte Carlo simulation, we generated synthetic immeasurable time-varying hospitalization-related factors of duration, frequency and timing. Nine scenarios were created by combining three frequency scenarios and three duration scenarios, where the empirical cohort distribution of hospitalization was used to simulate the 'timing'. We used Korea's healthcare database and a case example of β-blocker use and mortality among patients with heart failure. We estimated the gold-standard hazard ratio (HR) with 95% CI using inpatient and outpatient drug data, and that of the pseudo-outpatient setting using outpatient data only. We assessed the validity of adjusting for time-varying hospitalization in nine different scenarios, using relative bias, confidence limit ratio (CLR) and mean squared error (MSE) compared with the empirical gold-standard estimate across bootstrap resamples.

RESULTS

With the real-world gold standard (HR 0.73; 95% CI 0.67-0.80) as the reference estimate, adjusting for time-varying hospitalization (0.71; 0.63-0.80) effectively reduced the immeasurable time bias and had the following performance metrics across the nine scenarios: relative bias (range: -7.08% to 0.61%), CLR (1.28 to 1.36) and MSE (0.0005 to 0.0031).

CONCLUSIONS

The approach of adjusting for time-varying hospitalization consistently reduced the immeasurable time bias in Monte Carlo simulated data.

An application of the Causal Roadmap in two safety monitoring case studies: Causal inference and outcome prediction using electronic health record data

Background

Real-world data, such as administrative claims and electronic health records, are increasingly used for safety monitoring and to help guide regulatory decision-making. In these settings, it is important to document analytic decisions transparently and objectively to assess and ensure that analyses meet their intended goals.

Methods

The Causal Roadmap is an established framework that can guide and document analytic decisions through each step of the analytic pipeline, which will help investigators generate high-quality real-world evidence.

Results

In this paper, we illustrate the utility of the Causal Roadmap using two case studies previously led by workgroups sponsored by the Sentinel Initiative - a program for actively monitoring the safety of regulated medical products. Each case example focuses on different aspects of the analytic pipeline for drug safety monitoring. The first case study shows how the Causal Roadmap encourages transparency, reproducibility, and objective decision-making for causal analyses. The second case study highlights how this framework can guide analytic decisions beyond inference on causal parameters, improving outcome ascertainment in clinical phenotyping.

Conclusion

These examples provide a structured framework for implementing the Causal Roadmap in safety surveillance and guide transparent, reproducible, and objective analysis.

Inverse Probability of Treatment Weighting and Confounder Missingness in Electronic Health Record-based Analyses: A Comparison of Approaches Using Plasmode Simulation

BACKGROUND

Electronic health record (EHR) data represent a critical resource for comparative effectiveness research, allowing investigators to study intervention effects in real-world settings with large patient samples. However, high levels of missingness in confounder variables is common, challenging the perceived validity of EHR-based investigations.

METHODS

We investigated performance of multiple imputation and propensity score (PS) calibration when conducting inverse probability of treatment weights (IPTW)-based comparative effectiveness research using EHR data with missingness in confounder variables and outcome misclassification. Our motivating example compared effectiveness of immunotherapy versus chemotherapy treatment of advanced bladder cancer with missingness in a key prognostic variable. We captured complexity in EHR data structures using a plasmode simulation approach to spike investigator-defined effects into resamples of a cohort of 4361 patients from a nationwide deidentified EHR-derived database. We characterized statistical properties of IPTW hazard ratio estimates when using multiple imputation or PS calibration missingness approaches.

RESULTS

Multiple imputation and PS calibration performed similarly, maintaining ≤0.05 absolute bias in the marginal hazard ratio even when ≥50% of subjects had missing at random or missing not at random confounder data. Multiple imputation required greater computational resources, taking nearly 40 times as long as PS calibration to complete. Outcome misclassification minimally increased bias of both methods.

CONCLUSION

Our results support multiple imputation and PS calibration approaches to missingness in missing completely at random or missing at random confounder variables in EHR-based IPTW comparative effectiveness analyses, even with missingness ≥50%. PS calibration represents a computationally efficient alternative to multiple imputation.

Adaptability of High Dimensional Propensity Score Procedure in the Transition from ICD-9 to ICD-10 in the US Healthcare System

Background

High-Dimensional Propensity Score procedure (HDPS) is a data-driven approach to assist control for confounding in pharmacoepidemiologic research. The transition to the International Classification of Disease (ICD-9/10) in the US health system may pose uncertainty in applying the HDPS procedure.

Methods

We assembled a base cohort of patients in MarketScan^® Commercial Claims Database who had newly initiated celecoxib or traditional NSAIDs to compare gastrointestinal bleeding risk. We then created bootstrapped hypothetical cohorts from the base cohort with predefined patient selection patterns from the ICD eras. Three strategies for HDPS deployment were tested: 1) split the cohort by ICD era, deploy HDPS twice, and pool the relative risks (pooled RR), 2) consider codes from each ICD era as a separate data dimension and deploy HDPS in the entire cohort (data dimensions) and 3) map ICD codes from both eras to Clinical Classifications Software (CCS) concepts before deploying HDPS in the entire cohort (CCS mapping). We calculated percent bias and root-mean-squared error to compare the strategies.

Results

A similar bias reduction was observed in cohorts where patient selection pattern from each ICD era was comparable between the exposure groups. In the presence of considerable disparity in patient selection, we observed a bimodal distribution of propensity scores in the data dimensions strategy, indicating instrument-like covariates. Moreover, the CCS mapping strategy resulted in at least 30% less bias than pooled RR and data dimensions strategies (RMSE: 0.14, 0.19, 0.21, respectively) in this scenario.

Conclusion

Mapping ICD codes to a stable terminology like CCS serves as a helpful strategy to reduce residual bias when deploying HDPS in pharmacoepidemiologic studies spanning both ICD eras.

Strategies to Address Current Challenges in Real-World Evidence Generation in Japan

The generation of real-world evidence (RWE), which describes patient characteristics or treatment patterns using real-world data (RWD), is rapidly growing more popular as a tool for decision-making in Japan. The aim of this review was to summarize challenges to RWE generation in Japan related to pharmacoepidemiology, and to propose strategies to address some of these challenges. We first focused on data-related issues, including the lack of transparency of RWD sources, linkage across different care settings, definitions of clinical outcomes, and the overall assessment framework of RWD when used for research purposes. Next the study reviewed methodology-related challenges. As lack of design transparency impairs study reproducibility, transparent reporting of study design is critical for stakeholders. For this review, we considered different sources of biases and time-varying confounding, along with potential study design and methodological solutions. Additionally, the implementation of robust assessment of definition uncertainty, misclassification, and unmeasured confounders would enhance RWE credibility in light of RWD source-related limitations, and is being strongly considered by task forces in Japan. Overall, the development of guidance for best practices on data source selection, design transparency, and analytical methods to address different sources of biases and robustness in the process of RWE generation will enhance credibility for stakeholders and local decision-makers.

Performance of Multiple Imputation Using Modern Machine Learning Methods in Electronic Health Records Data

BACKGROUND

Missing data are common in studies using electronic health records (EHRs)-derived data. Missingness in EHR data is related to healthcare utilization patterns, resulting in complex and potentially missing not at random missingness mechanisms. Prior research has suggested that machine learning-based multiple imputation methods may outperform traditional methods and may perform well even in settings of missing not at random missingness.

METHODS

We used plasmode simulations based on a nationwide EHR-derived de-identified database for patients with metastatic urothelial carcinoma to compare the performance of multiple imputation using chained equations, random forests, and denoising autoencoders in terms of bias and precision of hazard ratio estimates under varying proportions of observations with missing values and missingness mechanisms (missing completely at random, missing at random, and missing not at random).

RESULTS

Multiple imputation by chained equations and random forest methods had low bias and similar standard errors for parameter estimates under missingness completely at random. Under missingness at random, denoising autoencoders had higher bias than multiple imputation by chained equations and random forests. Contrary to results of prior studies of denoising autoencoders, all methods exhibited substantial bias under missingness not at random, with bias increasing in direct proportion to the amount of missing data.

CONCLUSIONS

We found no advantage of denoising autoencoders for multiple imputation in the setting of an epidemiologic study conducted using EHR data. Results suggested that denoising autoencoders may overfit the data leading to poor confounder control. Use of more flexible imputation approaches does not mitigate bias induced by missingness not at random and can produce estimates with spurious precision.

Adapting SIMEX to correct for bias due to interval-censored outcomes in survival analysis with time-varying exposure

Many clinical and epidemiological applications of survival analysis focus on interval-censored events that can be ascertained only at discrete times of clinic visits. This implies that the values of time-varying covariates are not correctly aligned with the true, unknown event times, inducing a bias in the estimated associations. To address this issue, we adapted the simulation-extrapolation (SIMEX) methodology, based on assessing how the estimates change with the artificially increased time between clinic visits. We propose diagnostics to choose the extrapolating function. In simulations, the SIMEX-corrected estimates reduced considerably the bias to the null and generally yielded a better bias/variance trade-off than conventional estimates. In a real-life pharmacoepidemiological application, the proposed method increased by 27% the excess hazard of the estimated association between a time-varying exposure, representing the 2-year cumulative duration of past use of a hypertensive medication, and the hazard of nonmelanoma skin cancer (interval-censored events). These simulation-based and real-life results suggest that the proposed SIMEX-based correction may help improve the accuracy of estimated associations between time-varying exposures and the hazard of interval-censored events in large cohort studies where the events are recorded only at relatively sparse times of clinic visits/assessments. However, these advantages may be less certain for smaller studies and/or weak associations.

Spatially-enhanced clusterwise inference for testing and localizing intermodal correspondence

With the increasing availability of neuroimaging data from multiple modalities-each providing a different lens through which to study brain structure or function-new techniques for comparing, integrating, and interpreting information within and across modalities have emerged. Recent developments include hypothesis tests of associations between neuroimaging modalities, which can be used to determine the statistical significance of intermodal associations either throughout the entire brain or within anatomical subregions or functional networks. While these methods provide a crucial foundation for inference on intermodal relationships, they cannot be used to answer questions about where in the brain these associations are most pronounced. In this paper, we introduce a new method, called CLEAN-R, that can be used both to test intermodal correspondence throughout the brain and also to localize this correspondence. Our method involves first adjusting for the underlying spatial autocorrelation structure within each modality before aggregating information within small clusters to construct a map of enhanced test statistics. Using structural and functional magnetic resonance imaging data from a subsample of children and adolescents from the Philadelphia Neurodevelopmental Cohort, we conduct simulations and data analyses where we illustrate the high statistical power and nominal type I error levels of our method. By constructing an interpretable map of group-level correspondence using spatially-enhanced test statistics, our method offers insights beyond those provided by earlier methods.

The Timing, the Treatment, the Question: Comparison of Epidemiologic Approaches to Minimize Immortal Time Bias in Real-World Data Using a Surgical Oncology Example

BACKGROUND

Studies evaluating the effects of cancer treatments are prone to immortal time bias that, if unaddressed, can lead to treatments appearing more beneficial than they are.

METHODS

To demonstrate the impact of immortal time bias, we compared results across several analytic approaches (dichotomous exposure, dichotomous exposure excluding immortal time, time-varying exposure, landmark analysis, clone-censor-weight method), using surgical resection among women with metastatic breast cancer as an example. All adult women diagnosed with incident metastatic breast cancer from 2013-2016 in the National Cancer Database were included. To quantify immortal time bias, we also conducted a simulation study where the "true" relationship between surgical resection and mortality was known.

RESULTS

24,329 women (median age 61, IQR 51-71) were included, and 24% underwent surgical resection. The largest association between resection and mortality was observed when using a dichotomized exposure [HR, 0.54; 95% confidence interval (CI), 0.51-0.57], followed by dichotomous with exclusion of immortal time (HR, 0.62; 95% CI, 0.59-0.65). Results from the time-varying exposure, landmark, and clone-censor-weight method analyses were closer to the null (HR, 0.67-0.84). Results from the plasmode simulation found that the time-varying exposure, landmark, and clone-censor-weight method models all produced unbiased HRs (bias -0.003 to 0.016). Both standard dichotomous exposure (HR, 0.84; bias, -0.177) and dichotomous with exclusion of immortal time (HR, 0.93; bias, -0.074) produced meaningfully biased estimates.

CONCLUSIONS

Researchers should use time-varying exposures with a treatment assessment window or the clone-censor-weight method when immortal time is present.

IMPACT

Using methods that appropriately account for immortal time will improve evidence and decision-making from research using real-world data.

Generating high-fidelity privacy-conscious synthetic patient data for causal effect estimation with multiple treatments

In the past decade, there has been exponentially growing interest in the use of observational data collected as a part of routine healthcare practice to determine the effect of a treatment with causal inference models. Validation of these models, however, has been a challenge because the ground truth is unknown: only one treatment-outcome pair for each person can be observed. There have been multiple efforts to fill this void using synthetic data where the ground truth can be generated. However, to date, these datasets have been severely limited in their utility either by being modeled after small non-representative patient populations, being dissimilar to real target populations, or only providing known effects for two cohorts (treated vs. control). In this work, we produced a large-scale and realistic synthetic dataset that provides ground truth effects for over 10 hypertension treatments on blood pressure outcomes. The synthetic dataset was created by modeling a nationwide cohort of more than 580, 000 hypertension patient data including each person's multi-year history of diagnoses, medications, and laboratory values. We designed a data generation process by combining an adapted ADS-GAN model for fictitious patient information generation and a neural network for treatment outcome generation. Wasserstein distance of 0.35 demonstrates that our synthetic data follows a nearly identical joint distribution to the patient cohort used to generate the data. Patient privacy was a primary concern for this study; the ϵ-identifiability metric, which estimates the probability of actual patients being identified, is 0.008%, ensuring that our synthetic data cannot be used to identify any actual patients. To demonstrate its usage, we tested the bias in causal effect estimation of four well-established models using this dataset. The approach we used can be readily extended to other types of diseases in the clinical domain, and to datasets in other domains as well.

Synthetic Negative Controls: Using Simulation to Screen Large-scale Propensity Score Analyses

The propensity score has become a standard tool to control for large numbers of variables in healthcare database studies. However, little has been written on the challenge of comparing large-scale propensity score analyses that use different methods for confounder selection and adjustment. In these settings, balance diagnostics are useful but do not inform researchers on which variables balance should be assessed or quantify the impact of residual covariate imbalance on bias. Here, we propose a framework to supplement balance diagnostics when comparing large-scale propensity score analyses. Instead of focusing on results from any single analysis, we suggest conducting and reporting results for many analytic choices and using both balance diagnostics and synthetically generated control studies to screen analyses that show signals of bias caused by measured confounding. To generate synthetic datasets, the framework does not require simulating the outcome-generating process. In healthcare database studies, outcome events are often rare, making it difficult to identify and model all predictors of the outcome to simulate a confounding structure closely resembling the given study. Therefore, the framework uses a model for treatment assignment to divide the comparator population into pseudo-treatment groups where covariate differences resemble those in the study cohort. The partially simulated datasets have a confounding structure approximating the study population under the null (synthetic negative control studies). The framework is used to screen analyses that likely violate partial exchangeability due to lack of control for measured confounding. We illustrate the framework using simulations and an empirical example.

Using Numerical Methods to Design Simulations: Revisiting the Balancing Intercept

In this paper, we consider methods for generating draws of a binary random variable whose expectation conditional on covariates follows a logistic regression model with known covariate coefficients. We examine approximations for finding a "balancing intercept," that is, a value for the intercept of the logistic model that leads to a desired marginal expectation for the binary random variable. We show that a recently proposed analytical approximation can produce inaccurate results, especially when targeting more extreme marginal expectations or when the linear predictor of the regression model has high variance. We then formulate the balancing intercept as a solution to an integral equation, implement a numerical approximation for solving the equation based on Monte Carlo methods, and show that the approximation works well in practice. Our approach to the basic problem of the balancing intercept provides an example of a broadly applicable strategy for formulating and solving problems that arise in the design of simulation studies used to evaluate or teach epidemiologic methods.

Machine learning for improving high-dimensional proxy confounder adjustment in healthcare database studies: an overview of the current literature

Controlling for large numbers of variables that collectively serve as 'proxies' for unmeasured factors can often improve confounding control in pharmacoepidemiologic studies utilizing administrative healthcare databases. There is a growing body of evidence showing that data-driven machine learning algorithms for high-dimensional proxy confounder adjustment can supplement investigator-specified variables to improve confounding control compared to adjustment based on investigator-specified variables alone. Consequently, there has been a recent focus on the development of data-driven methods for high-dimensional proxy confounder adjustment. In this paper, we discuss the considerations underpinning three areas for data-driven high-dimensional proxy confounder adjustment: 1) feature generation-transforming raw data into covariates (or features) to be used for proxy adjustment; 2) covariate prioritization, selection and adjustment; and 3) diagnostic assessment. We survey current approaches and recent advancements within each area, including the most widely used approach to proxy confounder adjustment in healthcare database studies (the high-dimensional propensity score or hdPS). We also discuss limitations of the hdPS and outline recent advancements that incorporate the principles of proxy adjustment with machine learning extensions to improve performance. We further discuss challenges and avenues of future development within each area. This manuscript is endorsed by the International Society for Pharmacoepidemiology (ISPE). This article is protected by copyright. All rights reserved.

A Framework for Using Real-World Data and Health Outcomes Modeling to Evaluate Machine Learning-Based Risk Prediction Models

OBJECTIVES

We propose a framework of health outcomes modeling with dynamic decision making and real-world data (RWD) to evaluate the potential utility of novel risk prediction models in clinical practice. Lung transplant (LTx) referral decisions in cystic fibrosis offer a complex case study.

METHODS

We used longitudinal RWD for a cohort of adults (n = 4247) from the Cystic Fibrosis Foundation Patient Registry to compare outcomes of an LTx referral policy based on machine learning (ML) mortality risk predictions to referral based on (1) forced expiratory volume in 1 second (FEV₁) alone and (2) heterogenous usual care (UC). We then developed a patient-level simulation model to project number of patients referred for LTx and 5-year survival, accounting for transplant availability, organ allocation policy, and heterogenous treatment effects.

RESULTS

Only 12% of patients (95% confidence interval 11%-13%) were referred for LTx over 5 years under UC, compared with 19% (18%-20%) under FEV₁ and 20% (19%-22%) under ML. Of 309 patients who died before LTx referral under UC, 31% (27%-36%) would have been referred under FEV₁ and 40% (35%-45%) would have been referred under ML. Given a fixed supply of organs, differences in referral time did not lead to significant differences in transplants, pretransplant or post-transplant deaths, or overall survival in 5 years.

CONCLUSIONS

Health outcomes modeling with RWD may help to identify novel ML risk prediction models with high potential real-world clinical utility and rule out further investment in models that are unlikely to offer meaningful real-world benefits.

Extent, impact, and mitigation of batch effects in tumor biomarker studies using tissue microarrays

Tissue microarrays (TMAs) have been used in thousands of cancer biomarker studies. To what extent batch effects, measurement error in biomarker levels between slides, affects TMA-based studies has not been assessed systematically. We evaluated 20 protein biomarkers on 14 TMAs with prospectively collected tumor tissue from 1448 primary prostate cancers. In half of the biomarkers, more than 10% of biomarker variance was attributable to between-TMA differences (range, 1–48%). We implemented different methods to mitigate batch effects (R package batchtma), tested in plasmode simulation. Biomarker levels were more similar between mitigation approaches compared to uncorrected values. For some biomarkers, associations with clinical features changed substantially after addressing batch effects. Batch effects and resulting bias are not an error of an individual study but an inherent feature of TMA-based protein biomarker studies. They always need to be considered during study design and addressed analytically in studies using more than one TMA.

To understand cancer, researchers need to know which molecules tumor cells use. These so-called ‘biomarkers’ tag cancer cells as being different from healthy cells, and can be used to predict how aggressive a tumor may be, or how well it might respond to treatment.

A popular technique for assessing biomarkers across multiple tumors is to use tissue microarrays. This involves taking samples from different tumors and embedding them in a block of wax, which is then cut into micro-thin slices and stained with reagents that can detect specific biomarkers, such as proteins. Each block contains hundreds of samples, which all experience the same conditions. So, any patterns detected in the staining are likely to represent real variations in the biomarkers present.

Many cancer studies, however, often compare samples from multiple tissue microarrays, which may increase the risk of technical artifacts: for example, staining may look stronger in one batch of tissue samples than another, even though the amount of biomarker present in these different arrays is roughly the same. These ‘batch effects’ could potentially bias the results of the experiment and lead to the identification of misleading patterns.

To evaluate how batch effects impact tissue microarray studies, Stopsack et al. examined 14 wax blocks which contained tumor samples from 1,448 men with prostate cancer. This revealed that for some biomarkers, but not others, there were noticeable differences between tissue microarrays that were clearly the result of batch effects. Stopsack et al. then tested six different ways of fixing these discrepancies using statistical methods. All six approaches were successful, even if the arrays included tumors with different characteristics, such as tumors that had been diagnosed more or less recently.

This work highlights the importance of considering batch effects when using tissue microarrays to study cancer. Stopsack et al. have used their statistical approaches to develop freely available software which can reduce the biases that sometimes arise from these technical artifacts. This could help researchers avoid misleading patterns in their data and make it easier to detect real variations in the biomarkers present between tumor samples.

Combining heterogeneous subgroups with graph-structured variable selection priors for Cox regression

Background

Important objectives in cancer research are the prediction of a patient’s risk based on molecular measurements such as gene expression data and the identification of new prognostic biomarkers (e.g. genes). In clinical practice, this is often challenging because patient cohorts are typically small and can be heterogeneous. In classical subgroup analysis, a separate prediction model is fitted using only the data of one specific cohort. However, this can lead to a loss of power when the sample size is small. Simple pooling of all cohorts, on the other hand, can lead to biased results, especially when the cohorts are heterogeneous.

Results

We propose a new Bayesian approach suitable for continuous molecular measurements and survival outcome that identifies the important predictors and provides a separate risk prediction model for each cohort. It allows sharing information between cohorts to increase power by assuming a graph linking predictors within and across different cohorts. The graph helps to identify pathways of functionally related genes and genes that are simultaneously prognostic in different cohorts.

Conclusions

Results demonstrate that our proposed approach is superior to the standard approaches in terms of prediction performance and increased power in variable selection when the sample size is small.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12859-021-04483-z.

The PHARMACOM-EPI Framework for Integrating Pharmacometric Modelling Into Pharmacoepidemiological Research Using Real-World Data: Application to Assess Death Associated With Valproate

In pharmacoepidemiology, it is usually expected that the observed association should be directly or indirectly related to the pharmacological effects of the drug/s under investigation. Pharmacological effects are, in turn, strongly connected to the pharmacokinetic and pharmacodynamic properties of a drug, which can be characterized and investigated using pharmacometric models. Recently, the use of pharmacometrics has been proposed to provide pharmacological substantiation of pharmacoepidemiological findings derived from real-world data. However, validated frameworks suggesting how to combine these two disciplines for the aforementioned purpose are missing. Therefore, we propose PHARMACOM-EPI, a framework that provides a structured approach on how to identify, characterize, and apply pharmacometric models with practical details on how to choose software, format dataset, handle missing covariates/dosing data, how to perform the external evaluation of pharmacometric models in real-world data, and how to provide pharmacological substantiation of pharmacoepidemiological findings. PHARMACOM-EPI was tested in a proof-of-concept study to pharmacologically substantiate death associated with valproate use in the Danish population aged ≥ 65 years. Pharmacological substantiation of death during a follow-up period of 1 year showed that in all individuals who died (n = 169) individual predictions were within the subtherapeutic range compared with 52.8% of those who did not die (n = 1,084). Of individuals who died, 66.3% (n = 112) had a cause of death possibly related to valproate and 33.7% (n = 57) with well-defined cause of death unlikely related to valproate. This proof-of-concept study showed that PHARMACOM-EPI was able to provide pharmacological substantiation for death associated with valproate use in the study population.

Invited Commentary: The Prevalent New-User Design in Pharmacoepidemiology-Challenges and Opportunities

The prevalent new-user design includes a broader study population than the traditional new-user approach that is frequently used in pharmacoepidemiologic research. In an article appearing in this issue (Am J Epidemiol. 2021;190(7):1341-1348), Webster-Clark et al. describe the treatment initiator types included in the prevalent new-user design and contrast the causal questions assessed using a prevalent new-user design versus a new-user design. They further applied a series of simulation studies showing the importance of accounting for treatment history in addition to time since initiation of the comparator in the prevalent new-user design. In this commentary, we put their findings in the broader context with a discussion of the strengths and limitations of the prevalent new-user design and settings where it would be most useful. The prevalent new-user design and new-user design both address unique questions of clinical and public health importance. Real-world evidence generated by pharmacoepidemiologic research is increasingly being used by regulators and other knowledge users to inform their decision-making. Understanding the causal questions addressed by different designs is crucial in this process; the study by Webster-Clark et al. represents an important step in addressing this issue.

Initiator Types and the Causal Question of the Prevalent New-User Design: A Simulation Study

New-user designs restricting to treatment initiators have become the preferred design for studying drug comparative safety and effectiveness using nonexperimental data. This design reduces confounding by indication and healthy-adherer bias at the cost of smaller study sizes and reduced external validity, particularly when assessing a newly approved treatment compared with standard treatment. The prevalent new-user design includes adopters of a new treatment who switched from or previously used standard treatment (i.e., the comparator), expanding study sample size and potentially broadening the study population for inference. Previous work has suggested the use of time-conditional propensity-score matching to mitigate prevalent user bias. In this study, we describe 3 "types" of initiators of a treatment: new users, direct switchers, and delayed switchers. Using these initiator types, we articulate the causal questions answered by the prevalent new-user design and compare them with those answered by the new-user design. We then show, using simulation, how conditioning on time since initiating the comparator (rather than full treatment history) can still result in a biased estimate of the treatment effect. When implemented properly, the prevalent new-user design estimates new and important causal effects distinct from the new-user design.

Accounting for Repeat Pregnancies in Risk Prediction Models

BACKGROUND

In perinatal epidemiology, the development of risk prediction models is complicated by parity; how repeat pregnancies influence the predictive accuracy of models that include obstetrical history is unclear.

METHODS

To assess the influence of repeat pregnancies on the association between predictors and the outcomes, as well as the influence of ignoring the nonindependence between pregnancies, we created four analytical cohorts using the Clinical Practice Research Datalink. The cohorts included (1) first deliveries, (2) a random sample of one delivery per woman, (3) all eligible deliveries per woman, and (4) all eligible deliveries and censoring of follow-up at subsequent pregnancies. Using Plasmode simulations, we varied the predictor-outcome association across cohorts.

RESULTS

We found minimal differences in the relative contribution of predictors to the overall predictions and the discriminative accuracy of models in the cohort of randomly sampled deliveries versus the all deliveries cohort (C-statistic: 0.62 vs. 0.63; Nagelkerke's R2: 0.03 for both). Accounting for clustering and censoring upon subsequent pregnancies also had negligible influence on model performance. We found important differences in model performance between the models developed in the cohort of first deliveries and the random sample of deliveries.

CONCLUSIONS

In our study, a model including first deliveries had the best predictive accuracy but was not generalizable to women of varying parities. Moreover, including repeat pregnancies did not improve the predictive accuracy of the models. Multiple models may be needed to improve the transportability and accuracy of prediction models when the outcome of interest is influenced by parity.

Deep Learning-based Propensity Scores for Confounding Control in Comparative Effectiveness Research: A Large-scale, Real-world Data Study

BACKGROUND

Due to the non-randomized nature of real-world data, prognostic factors need to be balanced, which is often done by propensity scores (PSs). This study aimed to investigate whether autoencoders, which are unsupervised deep learning architectures, might be leveraged to compute PS.

METHODS

We selected patient-level data of 128,368 first-line treated cancer patients from the Flatiron Health EHR-derived de-identified database. We trained an autoencoder architecture to learn a lower-dimensional patient representation, which we used to compute PS. To compare the performance of an autoencoder-based PS with established methods, we performed a simulation study. We assessed the balancing and adjustment performance using standardized mean differences, root mean square errors (RMSE), percent bias, and confidence interval coverage. To illustrate the application of the autoencoder-based PS, we emulated the PRONOUNCE trial by applying the trial's protocol elements within an observational database setting, comparing two chemotherapy regimens.

RESULTS

All methods but the manual variable selection approach led to well-balanced cohorts with average standardized mean differences <0.1. LASSO yielded on average the lowest deviation of resulting estimates (RMSE 0.0205) followed by the autoencoder approach (RMSE 0.0248). Altering the hyperparameter setup in sensitivity analysis, the autoencoder approach led to similar results as LASSO (RMSE 0.0203 and 0.0205, respectively). In the case study, all methods provided a similar conclusion with point estimates clustered around the null (e.g., HRautoencoder 1.01 [95% confidence interval = 0.80, 1.27] vs. HRPRONOUNCE 1.07 [0.83, 1.36]).

CONCLUSIONS

Autoencoder-based PS computation was a feasible approach to control for confounding but did not perform better than some established approaches like LASSO.

Use of prescription drug samples in the US and implications for pharmacoepidemiologic research: a systematic search of the literature

INTRODUCTION

Free drug samples are not captured in the pharmacy claims databases used in many pharmacoepidemiologic studies, which could lead to misclassification of drug exposure status and thus bias study results.

AREAS COVERED

We systematically searched the literature in PubMed/MEDLINE, Embase, and Scopus from database inception to August 2020 for studies assessing the magnitude of exposure misclassification in pharmacy claims data associated with uncaptured drug sample utilization. Our review identified five US-based studies with substantially different characteristics, contexts, methods, and results. Taken together, these studies suggest that the risk of sample-related bias may be higher for (1) studies of newly approved, patented brand-only drugs in specific classes and contexts; (2) studies of populations where sample use is common and the unexposed cohort is small; and (3) studies where the outcomes of interest are expected to be early-onset or acute, with non-constant hazards.

EXPERT OPINION

In light of declining overall trends in sample use, future research on sample-related exposure misclassification should focus on delineating bias across those modern contexts where sample use remains high and optimizing bias quantification methods to create a more standardized approach. Additionally, further assessment is warranted for other sources of misclassified exposure status in claims-based pharmacoepidemiology research.

Methods to Account for Uncertainty in Latent Class Assignments When Using Latent Classes as Predictors in Regression Models, with Application to Acculturation Strategy Measures

Latent class models have become a popular means of summarizing survey questionnaires and other large sets of categorical variables. Often these classes are of primary interest to better understand complex patterns in data. Increasingly, these latent classes are reified into predictors of other outcomes of interests, treating the most likely class as the true class to which an individual belongs even though there is uncertainty in class membership. This uncertainty can be viewed as a form of measurement error in predictors, leading to bias in the estimates of the regression parameters associated with the latent classes. Despite this fact, there is very limited literature treating latent class predictors as measurement error models. Most applications ignore this issue and fit a two-stage model that treats the modal class prediction as truth. Here, we develop two approaches-one likelihood-based, the other Bayesian-to implement a joint model for latent class analysis and outcome prediction. We apply these methods to an analysis of how acculturation behaviors predict depression in South Asian immigrants to the United States. A simulation study gives guidance for when a two-stage model can be safely implemented and when the joint model may be required.

Propensity score trimming mitigates bias due to covariate measurement error in inverse probability of treatment weighted analyses: A plasmode simulation

BACKGROUND

Inverse probability of treatment weighting (IPTW) may be biased by influential observations, which can occur from misclassification of strong exposure predictors.

METHODS

We evaluated bias and precision of IPTW estimators in the presence of a misclassified confounder and assessed the effect of propensity score (PS) trimming. We generated 1000 plasmode cohorts of size N = 10 000, sampled with replacement from 6063 NHANES respondents (1999-2014) age 40 to 79 with labs and no statin use. We simulated statin exposure as a function of demographics and CVD risk factors; and outcomes as a function of 10-year CVD risk score and statin exposure (rate ratio [RR] = 0.5). For 5% of the people in selected populations (eg, all patients, exposed, those with outcomes), we randomly misclassified a confounder that strongly predicted exposure. We fit PS models and estimated RRs using IPTW and 1:1 PS matching, with and without asymmetric trimming.

RESULTS

IPTW bias was substantial when misclassification was differential by outcome (RR range: 0.38-0.63) and otherwise minimal (RR range: 0.51-0.53). However, trimming reduced bias for IPTW, nearly eliminating it at 5% trimming (RR range: 0.49-0.52). In one scenario, when the confounder was misclassified for 5% of those with outcomes (0.3% of cohort), untrimmed IPTW was more biased and less precise (RR = 0.37 [SE(logRR) = 0.21]) than matching (RR = 0.50 [SE(logRR) = 0.13]). After 1% trimming, IPTW estimates were unbiased and more precise (RR = 0.49 [SE(logRR) = 0.12]) than matching (RR = 0.51 [SE(logRR) = 0.14]).

CONCLUSIONS

Differential misclassification of a strong predictor of exposure resulted in biased and imprecise IPTW estimates. Asymmetric trimming reduced bias, with more precise estimates than matching.

Two-wave two-phase outcome-dependent sampling designs, with applications to longitudinal binary data

Two-phase outcome-dependent sampling (ODS) designs are useful when resource constraints prohibit expensive exposure ascertainment on all study subjects. One class of ODS designs for longitudinal binary data stratifies subjects into three strata according to those who experience the event at none, some, or all follow-up times. For time-varying covariate effects, exclusively selecting subjects with response variation can yield highly efficient estimates. However, if interest lies in the association of a time-invariant covariate, or the joint associations of time-varying and time-invariant covariates with the outcome, then the optimal design is unknown. Therefore, we propose a class of two-wave two-phase ODS designs for longitudinal binary data. We split the second-phase sample selection into two waves, between which an interim design evaluation analysis is conducted. The interim design evaluation analysis uses first-wave data to conduct a simulation-based search for the optimal second-wave design that will improve the likelihood of study success. Although we focus on longitudinal binary response data, the proposed design is general and can be applied to other response distributions. We believe that the proposed designs can be useful in settings where (1) the expected second-phase sample size is fixed and one must tailor stratum-specific sampling probabilities to maximize estimation efficiency, or (2) relative sampling probabilities are fixed across sampling strata and one must tailor sample size to achieve a desired precision. We describe the class of designs, examine finite sampling operating characteristics, and apply the designs to an exemplar longitudinal cohort study, the Lung Health Study.

Vector-based kernel weighting: A simple estimator for improving precision and bias of average treatment effects in multiple treatment settings

Treatment effect estimation must account for observed confounding, in which factors affect treatment assignment and outcomes simultaneously. Ignoring observed confounding risks concluding that a helpful treatment is not beneficial or that a treatment is safe when actually harmful. Propensity score matching or weighting adjusts for observed confounding, but the best way to use propensity scores for multiple treatments is unknown. It is unclear when choice of a different weighting or matching strategy leads to divergent inferences. We used Monte Carlo simulations (1000 replications) to examine sensitivity of multivalued treatment inferences to propensity score weighting or matching strategies. We consider five variants of propensity score adjustment: inverse probability of treatment weights, generalized propensity score matching, kernel weights (KW), vector matching, and a new hybrid that is easily implemented-vector-based kernel weighting (VBKW). VBKW matches observations with similar propensity score vectors, assigning greater KW to observations with similar probabilities within a given bandwidth. We varied degree of propensity score model misspecification, sample size, treatment effect heterogeneity, initial covariate imbalance, and sample distribution across treatment groups. We evaluated sensitivity of results to propensity score estimation technique (multinomial logit or multinomial probit). Across simulations, VBKW performed equally or better than the other methods in terms of bias, efficiency, and covariate balance measured via prognostic scores. Our simulations suggest that VBKW is amenable to full automation and is less sensitive to PS model misspecification than other methods used to account for observed confounding in multivalued treatment analyses.

Consequences of Depletion of Susceptibles for Hazard Ratio Estimators Based on Propensity Scores

Supplemental Digital Content is available in the text.

We use simulated data to examine the consequences of depletion of susceptibles for hazard ratio (HR) estimators based on a propensity score (PS). First, we show that the depletion of susceptibles attenuates marginal HRs toward the null by amounts that increase with the incidence of the outcome, the variance of susceptibility, and the impact of susceptibility on the outcome. If susceptibility is binary then the Bross bias multiplier, originally intended to quantify bias in a risk ratio from a binary confounder, also quantifies the ratio of the instantaneous marginal HR to the conditional HR as susceptibles are depleted differentially. Second, we show how HR estimates that are conditioned on a PS tend to be between the true conditional and marginal HRs, closer to the conditional HR if treatment status is strongly associated with susceptibility and closer to the marginal HR if treatment status is weakly associated with susceptibility. We show that associations of susceptibility with the PS matter to the marginal HR in the treated (ATT) though not to the marginal HR in the entire cohort (ATE). Third, we show how the PS can be updated periodically to reduce depletion-of-susceptibles bias in conditional estimators. Although marginal estimators can hit their ATE or ATT targets consistently without updating the PS, we show how their targets themselves can be misleading as they are attenuated toward the null. Finally, we discuss implications for the interpretation of HRs and their relevance to underlying scientific and clinical questions. See video Abstract: http://links.lww.com/EDE/B727.

Bias in case-crossover studies of medications due to persistent use: A simulation study

PURPOSE

The case-crossover design is increasingly used to evaluate the effects of chronic medications; however, as traditionally implemented in pharmacoepidemiology, with referent period preceding the outcome, it may lead to bias in the presence of persistent exposures. We aimed to evaluate the extent and magnitude of bias in case-crossover analyses of chronic and persistent exposures, using simulations.

METHODS

We simulated cohorts with either 30-day, 180-day, or 2-year exposure duration; and with varying degrees of persistence (10%, 30%, 50%, 70%, or 90% of patients not stopping exposure). We evaluated all scenarios under the null and the scenario with 30% persistence under varying exposure effects (odds ratios of 0.25 to 4.0). Cohorts were analyzed using conditional logistic regression that compared the odds of exposure on the outcome day to the odds of exposure on a referent day 30 days prior to the outcome. We further implemented the case-time-control design to evaluate its ability to adjust for bias from persistence.

RESULTS

Case-crossover analyses produced unbiased estimates across all scenarios without persistent users, regardless of exposure duration. In scenarios where some patients persisted on treatment, case-crossover analyses resulted in upward bias, which increased with increasing proportion of persistent users, but did not vary substantially in relation to the magnitude of the true effect. Case-time-control analyses removed bias in all scenarios.

CONCLUSIONS

Investigators should be aware of bias due to treatment persistence in unidirectional case-crossover analyses of chronic medications, which can be remedied with a control group of similarly persistent noncases.

Evaluating the Utility of Coarsened Exact Matching for Pharmacoepidemiology Using Real and Simulated Claims Data

Coarsened exact matching (CEM) is a matching method proposed as an alternative to other techniques commonly used to control confounding. We compared CEM with 3 techniques that have been used in pharmacoepidemiology: propensity score matching, Mahalanobis distance matching, and fine stratification by propensity score (FS). We evaluated confounding control and effect-estimate precision using insurance claims data from the Pharmaceutical Assistance Contract for the Elderly (1999-2002) and Medicaid Analytic eXtract (2000-2007) databases (United States) and from simulated claims-based cohorts. CEM generally achieved the best covariate balance. However, it often led to high bias and low precision of the risk ratio due to extreme losses in study size and numbers of outcomes (i.e., sparse data bias)-especially with larger covariate sets. FS usually was optimal with respect to bias and precision and always created good covariate balance. Propensity score matching usually performed almost as well as FS, especially with higher index exposure prevalence. The performance of Mahalanobis distance matching was relatively poor. These findings suggest that CEM, although it achieves good covariate balance, might not be optimal for large claims-database studies with rich covariate information; it might be ideal if only a few (<10) strong confounders must be controlled.

Safety surveillance and the estimation of risk in select populations: Flexible methods to control for confounding while targeting marginal comparisons via standardization

We consider the critical problem of pharmacosurveillance for adverse events once a drug or medical product is incorporated into routine clinical care. When making inference on comparative safety using large-scale electronic health records, we often encounter an extremely rare binary adverse outcome with a large number of potential confounders. In this context, it is challenging to offer flexible methods to adjust for high-dimensional confounders, whereas use of the propensity score (PS) can help address this challenge by providing both confounding control and dimension reduction. Among PS methods, regression adjustment using the PS as a covariate in an outcome model has been incompletely studied and potentially misused. Previous studies have suggested that simple linear adjustment may not provide sufficient control of confounding. Moreover, no formal representation of the statistical procedure and associated inference has been detailed. In this paper, we characterize a three-step procedure, which performs flexible regression adjustment of the estimated PS followed by standardization to estimate the causal effect in a select population. We also propose a simple variance estimation method for performing inference. Through a realistic simulation mimicking data from the Food and Drugs Administration's Sentinel Initiative comparing the effect of angiotensin-converting enzyme inhibitors and beta blockers on incidence of angioedema, we show that flexible regression on the PS resulted in less bias without loss of efficiency, and can outperform other methods when the PS model is correctly specified. In addition, the direct variance estimation method is a computationally fast and reliable approach for inference.

Comparison of Statistical Tests and Power Analysis for Phosphoproteomics Data

Advances in protein tagging and mass spectrometry have enabled generation of large quantitative proteome and phosphoproteome data sets, for identifying differentially expressed targets in case-control studies. The power study of statistical tests is critical for designing strategies for effective target identification and control of experimental cost. Here, we develop a simulation framework to generate realistic phospho-peptide data with known changes between cases and controls. Using this framework, we quantify the performance of traditional t-tests, Bayesian tests, and the ranking-by-fold-change test. Bayesian tests, which share variance information among peptides, outperform the traditional t-tests. Although ranking-by-fold-change has similar power as the Bayesian tests, its type I error rate cannot be properly controlled without proper permutation analysis; therefore, simply relying on the ranking likely brings false positives. Two-sample Bayesian tests considering dependencies between intensity and variance are superior for data sets with complex variance. While increasing the sample size enhances the statistical tests' performance, balanced controls and cases are recommended over a one-side weighted group. Further, higher peptide standard deviations require higher fold changes to achieve the same statistical power. Together, these results highlight the importance of model-informed experimental design and principled statistical analyses when working with large-scale proteomics and phosphoproteomics data.

Cost-Constrained feature selection in binary classification: adaptations for greedy forward selection and genetic algorithms

BACKGROUND

With modern methods in biotechnology, the search for biomarkers has advanced to a challenging statistical task exploring high dimensional data sets. Feature selection is a widely researched preprocessing step to handle huge numbers of biomarker candidates and has special importance for the analysis of biomedical data. Such data sets often include many input features not related to the diagnostic or therapeutic target variable. A less researched, but also relevant aspect for medical applications are costs of different biomarker candidates. These costs are often financial costs, but can also refer to other aspects, for example the decision between a painful biopsy marker and a simple urine test. In this paper, we propose extensions to two feature selection methods to control the total amount of such costs: greedy forward selection and genetic algorithms. In comprehensive simulation studies of binary classification tasks, we compare the predictive performance, the run-time and the detection rate of relevant features for the new proposed methods and five baseline alternatives to handle budget constraints.

RESULTS

In simulations with a predefined budget constraint, our proposed methods outperform the baseline alternatives, with just minor differences between them. Only in the scenario without an actual budget constraint, our adapted greedy forward selection approach showed a clear drop in performance compared to the other methods. However, introducing a hyperparameter to adapt the benefit-cost trade-off in this method could overcome this weakness.

CONCLUSIONS

In feature cost scenarios, where a total budget has to be met, common feature selection algorithms are often not suitable to identify well performing subsets for a modelling task. Adaptations of these algorithms such as the ones proposed in this paper can help to tackle this problem.

Visualization tool of variable selection in bias-variance tradeoff for inverse probability weights

PURPOSE

Inversed probability weighted (IPW) estimators are commonly used to adjust for time-fixed or time-varying confounders. However, in high-dimensional settings, including all identified confounders may result in unstable weights leading to higher variance. We aimed to develop a visualization tool demonstrating the impact of each confounder on the bias and variance of IPW estimates, as well as the propensity score overlap.

METHODS

A SAS macro was developed for this visualization tool and we demonstrate how this tool can be used to identify potentially problematic confounders of the association of statin use after myocardial infarction on one-year mortality in a plasmode simulation study using a cohort of 39,792 patients from the UK (1998-2012).

RESULTS

Through the tool's output, we can identify problematic confounders (two instrumental variables) and important confounders by comparing the estimated psuedo MSE with that from the fully adjusted model and propensity score overlap plot.

CONCLUSION

Our results suggest that the analytic impact of all confounders should be considered carefully when fitting IPW estimators.

Missing Data in Marginal Structural Models: A Plasmode Simulation Study Comparing Multiple Imputation and Inverse Probability Weighting

BACKGROUND

The use of marginal structural models (MSMs) to adjust for time-varying confounding has increased in epidemiologic studies. However, in the setting of MSMs, recommendations for how best to handle missing data are contradictory. We present a plasmode simulation study to compare the validity and precision of MSMs estimates using complete case analysis (CC), multiple imputation (MI), and inverse probability weighting (IPW) in the presence of missing data on time-independent and time-varying confounders.

MATERIALS AND METHODS

Simulations were based on a cohort substudy using data from the Osteoarthritis Initiative which estimated the marginal causal effect of intra-articular injection use on yearly changes in knee pain. We simulated 81 scenarios with parameter values varied on missing mechanisms (MCAR, MAR, and MNAR), percentages of missing (10%, 20%, and 30%), type of confounders (time-independent, time-varying, either or both), and analytical approaches (CC, IPW, and MI). The performance of CC, IPW, and MI methods was compared using relative bias, mean squared error of the estimates of interest, and empirical power.

RESULTS

Across scenarios defined by missing data mechanism, extent of missing data, and confounder type, MI generally produced less biased estimates (range: 1.2%-6.7%) with better precision (range: 0.17-0.18) compared with IPW (relative bias: -5.3% to 8.0%; precision: 0.19-0.53). Empirical power was constant across the scenarios using MI.

CONCLUSIONS

Under simple yet realistically constructed scenarios, MI seems to confer an advantage over IPW in MSMs applications.

Evaluating large-scale propensity score performance through real-world and synthetic data experiments

Background

Propensity score adjustment is a popular approach for confounding control in observational studies. Reliable frameworks are needed to determine relative propensity score performance in large-scale studies, and to establish optimal propensity score model selection methods.

Methods

We detail a propensity score evaluation framework that includes synthetic and real-world data experiments. Our synthetic experimental design extends the 'plasmode' framework and simulates survival data under known effect sizes, and our real-world experiments use a set of negative control outcomes with presumed null effect sizes. In reproductions of two published cohort studies, we compare two propensity score estimation methods that contrast in their model selection approach: L1-regularized regression that conducts a penalized likelihood regression, and the 'high-dimensional propensity score' (hdPS) that employs a univariate covariate screen. We evaluate methods on a range of outcome-dependent and outcome-independent metrics.

Results

L1-regularization propensity score methods achieve superior model fit, covariate balance and negative control bias reduction compared with the hdPS. Simulation results are mixed and fluctuate with simulation parameters, revealing a limitation of simulation under the proportional hazards framework. Including regularization with the hdPS reduces commonly reported non-convergence issues but has little effect on propensity score performance.

Conclusions

L1-regularization incorporates all covariates simultaneously into the propensity score model and offers propensity score performance superior to the hdPS marginal screen.