Robust Machine Learning Variable Importance Analyses of Medical Conditions for Health Care Spending

Constructing prediction models and analyzing factors in suicidal ideation using machine learning, focusing on the older population

Suicide among the older population is a significant public health concern in South Korea. As the older individuals have long considered suicide before committing suicide trials, it is important to analyze the suicidal ideation that precedes the suicide attempt for intervention. In this study, six machine learning algorithms were employed to construct a predictive model for suicidal thinking and identify key variables. A traditional logistic regression analysis was supplementarily conducted to test the robustness of the results of machine learning. All analyses were conducted using a hierarchical approach to compare the model fit of each model in both machine learning and logistic regression. Three models were established for analysis. In Model 1, socioeconomic, residential, and health behavioral factors were incorporated. Model 2 expanded upon Model 1 by integrating physical health status, and Model 3 further incorporated mental health conditions. The results indicated that the gradient boosting algorithm outperformed the other machine learning techniques. Furthermore, the household income quintile was the most important feature in Model 1, followed by subjective health status, oral health, and exercise ability in Model 2, and anxiety and depression in Model 3. These results correspond to those of the hierarchical logistic regression. Notably, economic and residential vulnerabilities are significant factors in the mental health of the older population with higher instances of suicidal thoughts. This hierarchical approach could reveal the potential target population for suicide interventions.

Classifying clinical phenotypes of functional recovery for acute traumatic spinal cord injury. An observational cohort study

PURPOSE

Identify patient subgroups with different functional outcomes after SCI and study the association between functional status and initial ISNCSCI components.

METHODS

Using CART, we performed an observational cohort study on data from 675 patients enrolled in the Rick-Hansen Registry(RHSCIR) between 2014 and 2019. The outcome was the Spinal Cord Independence Measure (SCIM) and predictors included AIS, NLI, UEMS, LEMS, pinprick(PPSS), and light touch(LTSS) scores. A temporal validation was performed on data from 62 patients treated between 2020 and 2021 in one of the RHSCIR participating centers.

RESULTS

The final CART resulted in four subgroups with increasing totSCIM according to PPSS, LEMS, and UEMS: 1)PPSS < 27(totSCIM = 28.4 ± 16.3); 2)PPSS ≥ 27, LEMS < 1.5, UEMS < 45(totSCIM = 39.5 ± 19.0); 3)PPSS ≥ 27, LEMS < 1.5, UEMS ≥ 45(totSCIM = 57.4 ± 13.8); 4)PPSS ≥ 27, LEMS ≥ 1.5(totSCIM = 66.3 ± 21.7). The validation model performed similarly to the original model. The adjusted R-squared and F-test were respectively 0.556 and 62.2(P-value <0.001) in the development cohort and, 0.520 and 31.9(P-value <0.001) in the validation cohort.

CONCLUSION

Acknowledging the presence of four characteristic subgroups of patients with distinct phenotypes of functional recovery based on PPSS, LEMS, and UEMS could be used by clinicians early after tSCI to plan rehabilitation and establish realistic goals. An improved sensory function could be key for potentiating motor gains, as a PPSS ≥ 27 was a predictor of a good function.

Evaluating the robustness of targeted maximum likelihood estimators via realistic simulations in nutrition intervention trials

Several recently developed methods have the potential to harness machine learning in the pursuit of target quantities inspired by causal inference, including inverse weighting, doubly robust estimating equations and substitution estimators like targeted maximum likelihood estimation. There are even more recent augmentations of these procedures that can increase robustness, by adding a layer of cross-validation (cross-validated targeted maximum likelihood estimation and double machine learning, as applied to substitution and estimating equation approaches, respectively). While these methods have been evaluated individually on simulated and experimental data sets, a comprehensive analysis of their performance across real data based simulations have yet to be conducted. In this work, we benchmark multiple widely used methods for estimation of the average treatment effect using ten different nutrition intervention studies data. A nonparametric regression method, undersmoothed highly adaptive lasso, is used to generate the simulated distribution which preserves important features from the observed data and reproduces a set of true target parameters. For each simulated data, we apply the methods above to estimate the average treatment effects as well as their standard errors and resulting confidence intervals. Based on the analytic results, a general recommendation is put forth for use of the cross-validated variants of both substitution and estimating equation estimators. We conclude that the additional layer of cross-validation helps in avoiding unintentional over-fitting of nuisance parameter functionals and leads to more robust inferences.

Residual Confounding in Health Plan Performance Assessments: Evidence From Randomization in Medicaid

BACKGROUND

Risk adjustment is used widely in payment systems and performance assessments, but the extent to which it distinguishes plan or provider effects from confounding due to patient differences is typically unknown.

OBJECTIVE

To assess the degree to which risk-adjusted measures of health plan performance adequately adjust for the variation across plans that arises because of differences in patient characteristics (residual confounding).

DESIGN

Comparison between plan performance estimates based on enrollees who made plan choices (observational population) and estimates based on enrollees assigned to plans (randomized population).

SETTING

Natural experiment in which more than two thirds of a state's Medicaid population in 1 region was randomly assigned to 1 of 5 plans.

PARTICIPANTS

137 933 enrollees in 2013 to 2014, of whom 31.1% selected a plan and 68.9% were randomly assigned to 1 of the same 5 plans.

MEASUREMENTS

Annual total spending (that is, payments to providers), primary care use, dental care use, and avoidable emergency department visits, all scored as plan-specific deviations from the "average" plan performance within each population.

RESULTS

Enrollee characteristics were appreciably imbalanced across plans in the observational population, as expected, but were not in the randomized population. Annual total spending varied across plans more in the observational population (SD, $147 per enrollee) than in the randomized population (SD, $70 per enrollee) after accounting for baseline differences in the observational and randomized populations and for differences across plans. On average, a plan's spending score (its deviation from the "average" performance) in the observational population differed from its score in the randomized population by $67 per enrollee in absolute value (95% CI, $38 to $123), or 4.2% of mean spending per enrollee (P = 0.009, rejecting the null hypothesis that this difference would be expected from sampling error). The difference was reduced modestly by risk adjustment to $62 per enrollee (P = 0.012). Residual confounding was similarly substantial for most other performance measures. Further adjustment for social factors did not materially change estimates.

LIMITATION

Potential heterogeneity in plan effects between the 2 populations.

CONCLUSION

Residual confounding in risk-adjusted performance assessments can be substantial and should caution policymakers against assuming that risk adjustment isolates real differences in plan performance.

PRIMARY FUNDING SOURCE

Arnold Ventures.

Identifying undercompensated groups defined by multiple attributes in risk adjustment

Objective

To identify undercompensated groups in plan payment risk adjustment that are defined by multiple attributes with a systematic new approach, improving on the arbitrary and inconsistent nature of existing evaluations.

Methods

Extending the concept of variable importance for single attributes, we construct a measure of ‘group importance’ in the random forests algorithm to identify groups with multiple attributes that are undercompensated by current risk adjustment formulas. Using 2016–2018 IBM MarketScan and 2015–2018 Medicare claims and enrolment data, we evaluate two risk adjustment scenarios: the risk adjustment formula used in the individual health insurance Marketplaces and the risk adjustment formula used in Medicare.

Results

A number of previously unidentified groups with multiple chronic conditions are undercompensated in the Marketplaces risk adjustment formula, while groups without chronic conditions tend to be overcompensated in the Marketplaces. The magnitude of undercompensation when defining groups with multiple attributes is many times larger than with single attributes. No complex groups were found to be consistently undercompensated or overcompensated in the Medicare risk adjustment formula.

Conclusions

Our method is effective at identifying complex undercompensated groups in health plan payment risk adjustment where undercompensation creates incentives for insurers to discriminate against these groups. This work provides policy-makers with new information on potential targets of discrimination in the healthcare system and a path towards more equitable health coverage.

Using Deep Learning to Identify High-Risk Patients with Heart Failure with Reduced Ejection Fraction

Background: Deep Learning (DL) has not been well-established as a method to identify high-risk patients among patients with heart failure (HF). Objectives: This study aimed to use DL models to predict hospitalizations, worsening HF events, and 30-day and 90-day readmissions in patients with heart failure with reduced ejection fraction (HFrEF). Methods: We analyzed the data of adult HFrEF patients from the IBM® MarketScan® Commercial and Medicare Supplement databases between January 1, 2015 and December 31, 2017. A sequential model architecture based on bi-directional long short-term memory (Bi-LSTM) layers was utilized. For DL models to predict HF hospitalizations and worsening HF events, we utilized two study designs: with and without a buffer window. For comparison, we also tested multiple traditional machine learning models including logistic regression, random forest, and eXtreme Gradient Boosting (XGBoost). Model performance was assessed by area under the curve (AUC) values, precision, and recall on an independent testing dataset. Results: A total of 47 498 HFrEF patients were included; 9427 with at least one HF hospitalization. The best AUCs of DL models without a buffer window in predicting HF hospitalizations and worsening HF events in the total patient cohort were 0.977 and 0.972; with a 7-day buffer window the best AUCs were 0.573 and 0.608, respectively. The best AUCs in predicting 30- and 90-day readmissions in all adult patients were 0.597 and 0.614, respectively. An AUC of 0.861 was attained for prediction of 90-day readmission in patients aged 18-64. For all outcomes assessed, the DL approach outperformed traditional machine learning models. Discussion: The DL approach can automate feature engineering during the model learning, which can increase the clinical applicability and lead to comparable or better model performance. However, the lack of granular clinical data, and sample size and imbalance issues may have limited the model's performance. Conclusions: A DL approach using Bi-LSTM was shown to be a feasible and useful tool to predict HF-related outcomes. This study can help inform the future development and deployment of predictive tools to identify high-risk HFrEF patients and ultimately facilitate targeted interventions in clinical practice.

Intersections of machine learning and epidemiological methods for health services research

The field of health services research is broad and seeks to answer questions about the health care system. It is inherently interdisciplinary, and epidemiologists have made crucial contributions. Parametric regression techniques remain standard practice in health services research with machine learning techniques currently having low penetrance in comparison. However, studies in several prominent areas, including health care spending, outcomes and quality, have begun deploying machine learning tools for these applications. Nevertheless, major advances in epidemiological methods are also as yet underleveraged in health services research. This article summarizes the current state of machine learning in key areas of health services research, and discusses important future directions at the intersection of machine learning and epidemiological methods for health services research.

Estimating treatment effects with machine learning

OBJECTIVE

To demonstrate the performance of methodologies that include machine learning (ML) algorithms to estimate average treatment effects under the assumption of exogeneity (selection on observables).

DATA SOURCES

Simulated data and observational data on hospitalized adults.

STUDY DESIGN

We assessed the performance of several ML-based estimators, including Targeted Maximum Likelihood Estimation, Bayesian Additive Regression Trees, Causal Random Forests, Double Machine Learning, and Bayesian Causal Forests, applying these methods to simulated data as well as data on the effects of right heart catheterization.

PRINCIPAL FINDINGS

In Monte Carlo studies, ML-based estimators generated estimates with smaller bias than traditional regression approaches, demonstrating substantial (69 percent-98 percent) bias reduction in some scenarios. Bayesian Causal Forests and Double Machine Learning were top performers, although all were sensitive to high dimensional (>150) sets of covariates.

CONCLUSIONS

ML-based methods are promising methods for estimating treatment effects, allowing for the inclusion of many covariates and automating the search for nonlinearities and interactions among variables. We provide guidance and sample code for researchers interested in implementing these tools in their own empirical work.

Predicting patient-reported outcomes following hip and knee replacement surgery using supervised machine learning

BACKGROUND

Machine-learning classifiers mostly offer good predictive performance and are increasingly used to support shared decision-making in clinical practice. Focusing on performance and practicability, this study evaluates prediction of patient-reported outcomes (PROs) by eight supervised classifiers including a linear model, following hip and knee replacement surgery.

METHODS

NHS PRO data (130,945 observations) from April 2015 to April 2017 were used to train and test eight classifiers to predict binary postoperative improvement based on minimal important differences. Area under the receiver operating characteristic, J-statistic and several other metrics were calculated. The dependent outcomes were generic and disease-specific improvement based on the EQ-5D-3L visual analogue scale (VAS) as well as the Oxford Hip and Knee Score (Q score).

RESULTS

The area under the receiver operating characteristic of the best training models was around 0.87 (VAS) and 0.78 (Q score) for hip replacement, while it was around 0.86 (VAS) and 0.70 (Q score) for knee replacement surgery. Extreme gradient boosting, random forests, multistep elastic net and linear model provided the highest overall J-statistics. Based on variable importance, the most important predictors for post-operative outcomes were preoperative VAS, Q score and single Q score dimensions. Sensitivity analysis for hip replacement VAS evaluated the influence of minimal important difference, patient selection criteria as well as additional data years. Together with a small benchmark of the NHS prediction model, robustness of our results was confirmed.

CONCLUSIONS

Supervised machine-learning implementations, like extreme gradient boosting, can provide better performance than linear models and should be considered, when high predictive performance is needed. Preoperative VAS, Q score and specific dimensions like limping are the most important predictors for postoperative hip and knee PROMs.