Deep Learning-based Propensity Scores for Confounding Control in Comparative Effectiveness Research: A Large-scale, Real-world Data Study

Unmasking bias in artificial intelligence: a systematic review of bias detection and mitigation strategies in electronic health record-based models

OBJECTIVES

Leveraging artificial intelligence (AI) in conjunction with electronic health records (EHRs) holds transformative potential to improve healthcare. However, addressing bias in AI, which risks worsening healthcare disparities, cannot be overlooked. This study reviews methods to handle various biases in AI models developed using EHR data.

MATERIALS AND METHODS

We conducted a systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-analyses guidelines, analyzing articles from PubMed, Web of Science, and IEEE published between January 01, 2010 and December 17, 2023. The review identified key biases, outlined strategies for detecting and mitigating bias throughout the AI model development, and analyzed metrics for bias assessment.

RESULTS

Of the 450 articles retrieved, 20 met our criteria, revealing 6 major bias types: algorithmic, confounding, implicit, measurement, selection, and temporal. The AI models were primarily developed for predictive tasks, yet none have been deployed in real-world healthcare settings. Five studies concentrated on the detection of implicit and algorithmic biases employing fairness metrics like statistical parity, equal opportunity, and predictive equity. Fifteen studies proposed strategies for mitigating biases, especially targeting implicit and selection biases. These strategies, evaluated through both performance and fairness metrics, predominantly involved data collection and preprocessing techniques like resampling and reweighting.

DISCUSSION

This review highlights evolving strategies to mitigate bias in EHR-based AI models, emphasizing the urgent need for both standardized and detailed reporting of the methodologies and systematic real-world testing and evaluation. Such measures are essential for gauging models' practical impact and fostering ethical AI that ensures fairness and equity in healthcare.

Matching by OS Prognostic Score to Construct External Controls in Lung Cancer Clinical Trials

External controls (eControls) leverage historical data to create non-randomized control arms. The lack of randomization can result in confounding between the experimental and eControl cohorts. To balance potentially confounding variables between the cohorts, one of the proposed methods is to match on prognostic scores. Still, the performance of prognostic scores to construct eControls in oncology has not been analyzed yet. Using an electronic health record-derived de-identified database, we constructed eControls using one of three methods: ROPRO, a state-of-the-art prognostic score, or either a propensity score composed of five (5Vars) or 27 covariates (ROPROvars). We compared the performance of these methods in estimating the overall survival (OS) hazard ratio (HR) of 11 recent advanced non-small cell lung cancer. The ROPRO eControls had a lower OS HR error (median absolute deviation (MAD), 0.072, confidence interval (CI): 0.036-0.185), than the 5Vars (MAD 0.081, CI: 0.025-0.283) and ROPROvars eControls (MAD 0.087, CI: 0.054-0.383). Notably, the OS HR errors for all methods were even lower in the phase III studies. Moreover, the ROPRO eControl cohorts included, on average, more patients than the 5Vars (6.54%) and ROPROvars cohorts (11.7%). The eControls matched with the prognostic score reproduced the controls more reliably than propensity scores composed of the underlying variables. Additionally, prognostic scores could allow eControls to be built on many prognostic variables without a significant increase in the variability of the propensity score, which would decrease the number of matched patients.

High-throughput target trial emulation for Alzheimer's disease drug repurposing with real-world data

Target trial emulation is the process of mimicking target randomized trials using real-world data, where effective confounding control for unbiased treatment effect estimation remains a main challenge. Although various approaches have been proposed for this challenge, a systematic evaluation is still lacking. Here we emulated trials for thousands of medications from two large-scale real-world data warehouses, covering over 10 years of clinical records for over 170 million patients, aiming to identify new indications of approved drugs for Alzheimer's disease. We assessed different propensity score models under the inverse probability of treatment weighting framework and suggested a model selection strategy for improved baseline covariate balancing. We also found that the deep learning-based propensity score model did not necessarily outperform logistic regression-based methods in covariate balancing. Finally, we highlighted five top-ranked drugs (pantoprazole, gabapentin, atorvastatin, fluticasone, and omeprazole) originally intended for other indications with potential benefits for Alzheimer's patients.

Genetic matching for time-dependent treatments: a longitudinal extension and simulation study

BACKGROUND

Longitudinal matching can mitigate confounding in observational, real-world studies of time-dependent treatments. To date, these methods have required iterative, manual re-specifications to achieve covariate balance. We propose a longitudinal extension of genetic matching, a machine learning approach that automates balancing of covariate histories. We examine performance by comparing the proposed extension against baseline propensity score matching and time-dependent propensity score matching.

METHODS

To evaluate comparative performance, we developed a Monte Carlo simulation framework that reflects a static treatment assigned at multiple time points. Data generation considers a treatment assignment model, a continuous outcome model, and underlying covariates. In simulation, we generated 1,000 datasets, each consisting of 1,000 subjects, and applied: (1) nearest neighbour matching on time-invariant, baseline propensity scores; (2) sequential risk set matching on time-dependent propensity scores; and (3) longitudinal genetic matching on time-dependent covariates. To measure comparative performance, we estimated covariate balance, efficiency, bias, and root mean squared error (RMSE) of treatment effect estimates. In scenario analysis, we varied underlying assumptions for assumed covariate distributions, correlations, treatment assignment models, and outcome models.

RESULTS

In all scenarios, baseline propensity score matching resulted in biased effect estimation in the presence of time-dependent confounding, with mean bias ranging from 29.7% to 37.2%. In contrast, time-dependent propensity score matching and longitudinal genetic matching achieved stronger covariate balance and yielded less biased estimation, with mean bias ranging from 0.7% to 13.7%. Across scenarios, longitudinal genetic matching achieved similar or better performance than time-dependent propensity score matching without requiring manual re-specifications or normality of covariates.

CONCLUSIONS

While the most appropriate longitudinal method will depend on research questions and underlying data patterns, our study can help guide these decisions. Simulation results demonstrate the validity of our longitudinal genetic matching approach for supporting future real-world assessments of treatments accessible at multiple time points.

Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology

Uncertainty estimation is crucial for understanding the reliability of deep learning (DL) predictions, and critical for deploying DL in the clinic. Differences between training and production datasets can lead to incorrect predictions with underestimated uncertainty. To investigate this pitfall, we benchmarked one pointwise and three approximate Bayesian DL models for predicting cancer of unknown primary, using three RNA-seq datasets with 10,968 samples across 57 cancer types. Our results highlight that simple and scalable Bayesian DL significantly improves the generalisation of uncertainty estimation. Moreover, we designed a prototypical metric-the area between development and production curve (ADP), which evaluates the accuracy loss when deploying models from development to production. Using ADP, we demonstrate that Bayesian DL improves accuracy under data distributional shifts when utilising 'uncertainty thresholding'. In summary, Bayesian DL is a promising approach for generalising uncertainty, improving performance, transparency, and safety of DL models for deployment in the real world.

Propensity score analysis with missing data using a multi-task neural network

BACKGROUND

Propensity score analysis is increasingly used to control for confounding factors in observational studies. Unfortunately, unavoidable missing values make estimating propensity scores extremely challenging. We propose a new method for estimating propensity scores in data with missing values.

MATERIALS AND METHODS

Both simulated and real-world datasets are used in our experiments. The simulated datasets were constructed under 2 scenarios, the presence (T = 1) and the absence (T = 0) of the true effect. The real-world dataset comes from LaLonde's employment training program. We construct missing data with varying degrees of missing rates under three missing mechanisms: MAR, MCAR, and MNAR. Then we compare MTNN with 2 other traditional methods in different scenarios. The experiments in each scenario were repeated 20,000 times. Our code is publicly available at https://github.com/ljwa2323/MTNN .

RESULTS

Under the three missing mechanisms of MAR, MCAR and MNAR, the RMSE between the effect and the true effect estimated by our proposed method is the smallest in simulations and in real-world data. Furthermore, the standard deviation of the effect estimated by our method is the smallest. In situations where the missing rate is low, the estimation of our method is more accurate.

CONCLUSIONS

MTNN can perform propensity score estimation and missing value filling at the same time through shared hidden layers and joint learning, which solves the dilemma of traditional methods and is very suitable for estimating true effects in samples with missing values. The method is expected to be broadly generalized and applied to real-world observational studies.

Target Trial Emulation: A Design Tool for Cancer Clinical Trials

PURPOSE

To apply target trial emulation to explore the potential impact of eligibility criteria on the primary outcome of a randomized controlled trial.

METHODS

Simulations of a real-world explanatory trial of transarterial radioembolization for advanced unresectable hepatocellular carcinoma with portal vein invasion were performed to examine the effects of cohort specification on survival outcomes and patient sample size. Simulations comprised 24 different permutations of the trial varied on three disease nonspecific eligibility parameters. Treatment and control arms for these emulated trials were drawn from the National Cancer Database and matched by treatment propensity. Target trial emulation served as the causal framework for this analysis, allowing the architecture of a true controlled experiment to address forms of bias routinely encountered in comparative effectiveness studies involving real-world observational data.

RESULTS

Twenty-four propensity score-matched cohorts comprising a wider clinical spectrum of patients than specified by the original target trial were successfully generated using the National Cancer Database. The arms for each of the emulated trials demonstrated exchangeability across all eligibility criteria and other clinical covariates. Significant treatment benefits were associated with only a narrow range of eligibility criteria, indicating that the original target trial was well specified.

CONCLUSION

The impact of patient selection on treatment outcomes can be studied using target trial emulation. This analytical framework can furthermore serve to leverage existing real-world data to inform the task of cohort specification for a randomized controlled trial, facilitating a more data-driven approach for this important step in clinical trial design.

In-hospital mortality risk stratification of Asian ACS patients with artificial intelligence algorithm

BACKGROUND

Conventional risk score for predicting in-hospital mortality following Acute Coronary Syndrome (ACS) is not catered for Asian patients and requires different types of scoring algorithms for STEMI and NSTEMI patients.

OBJECTIVE

To derive a single algorithm using deep learning and machine learning for the prediction and identification of factors associated with in-hospital mortality in Asian patients with ACS and to compare performance to a conventional risk score.

METHODS

The Malaysian National Cardiovascular Disease Database (NCVD) registry, is a multi-ethnic, heterogeneous database spanning from 2006-2017. It was used for in-hospital mortality model development with 54 variables considered for patients with STEMI and Non-STEMI (NSTEMI). Mortality prediction was analyzed using feature selection methods with machine learning algorithms. Deep learning algorithm using features selected from machine learning was compared to Thrombolysis in Myocardial Infarction (TIMI) score.

RESULTS

A total of 68528 patients were included in the analysis. Deep learning models constructed using all features and selected features from machine learning resulted in higher performance than machine learning and TIMI risk score (p < 0.0001 for all). The best model in this study is the combination of features selected from the SVM algorithm with a deep learning classifier. The DL (SVM selected var) algorithm demonstrated the highest predictive performance with the least number of predictors (14 predictors) for in-hospital prediction of STEMI patients (AUC = 0.96, 95% CI: 0.95-0.96). In NSTEMI in-hospital prediction, DL (RF selected var) (AUC = 0.96, 95% CI: 0.95-0.96, reported slightly higher AUC compared to DL (SVM selected var) (AUC = 0.95, 95% CI: 0.94-0.95). There was no significant difference between DL (SVM selected var) algorithm and DL (RF selected var) algorithm (p = 0.5). When compared to the DL (SVM selected var) model, the TIMI score underestimates patients' risk of mortality. TIMI risk score correctly identified 13.08% of the high-risk patient's non-survival vs 24.7% for the DL model and 4.65% vs 19.7% of the high-risk patient's non-survival for NSTEMI. Age, heart rate, Killip class, cardiac catheterization, oral hypoglycemia use and antiarrhythmic agent were found to be common predictors of in-hospital mortality across all ML feature selection models in this study. The final algorithm was converted into an online tool with a database for continuous data archiving for prospective validation.

CONCLUSIONS

ACS patients were better classified using a combination of machine learning and deep learning in a multi-ethnic Asian population when compared to TIMI scoring. Machine learning enables the identification of distinct factors in individual Asian populations to improve mortality prediction. Continuous testing and validation will allow for better risk stratification in the future, potentially altering management and outcomes.

Machine learning for improving high-dimensional proxy confounder adjustment in healthcare database studies: an overview of the current literature

Controlling for large numbers of variables that collectively serve as 'proxies' for unmeasured factors can often improve confounding control in pharmacoepidemiologic studies utilizing administrative healthcare databases. There is a growing body of evidence showing that data-driven machine learning algorithms for high-dimensional proxy confounder adjustment can supplement investigator-specified variables to improve confounding control compared to adjustment based on investigator-specified variables alone. Consequently, there has been a recent focus on the development of data-driven methods for high-dimensional proxy confounder adjustment. In this paper, we discuss the considerations underpinning three areas for data-driven high-dimensional proxy confounder adjustment: 1) feature generation-transforming raw data into covariates (or features) to be used for proxy adjustment; 2) covariate prioritization, selection and adjustment; and 3) diagnostic assessment. We survey current approaches and recent advancements within each area, including the most widely used approach to proxy confounder adjustment in healthcare database studies (the high-dimensional propensity score or hdPS). We also discuss limitations of the hdPS and outline recent advancements that incorporate the principles of proxy adjustment with machine learning extensions to improve performance. We further discuss challenges and avenues of future development within each area. This manuscript is endorsed by the International Society for Pharmacoepidemiology (ISPE). This article is protected by copyright. All rights reserved.