A Framework for Using Real-World Data and Health Outcomes Modeling to Evaluate Machine Learning-Based Risk Prediction Models

Predicting surgical outcomes of acute diffuse peritonitis: Updated risk models based on real-world clinical data

Aim

The existing predictive risk models for the surgical outcome of acute diffused peritonitis (ADP) need renovation by adding relevant variables such as ADP's definition or causative etiology to pursue outstanding data collection reflecting the real world. We aimed to revise the risk models predicting mortality and morbidities of ADP using the latest Japanese Nationwide Clinical Database (NCD) variable set.

Methods

Clinical dataset of ADP patients who underwent surgery, and registered in the NCD between 2016 and 2019, were used to develop a risk model for surgical outcomes. The primary outcome was perioperative mortality.

Results

After data cleanup, 45 379 surgical cases for ADP were derived for analysis. The perioperative and 30-day mortality were 10.6% and 7.2%, respectively. The prediction models have been created for the mortality and 10 morbidities associated with the mortality. The top five relevant predictors for perioperative mortality were age >80, advanced cancer with multiple metastases, platelet count of <50 000/mL, serum albumin of <2.0 g/dL, and unknown ADP site. The C-indices of perioperative and 30-day mortality were 0.859 and 0.857, respectively. The predicted value calculated with the risk models for mortality was highly fitted with the actual probability from the lower to the higher risk groups.

Conclusions

Risk models for postoperative mortality and morbidities with good predictive performance and reliability were revised and validated using the recent real-world clinical dataset. These models help to predict ADP surgical outcomes accurately and are available for clinical settings.

Statistical plasmode simulations-Potentials, challenges and recommendations

Statistical data simulation is essential in the development of statistical models and methods as well as in their performance evaluation. To capture complex data structures, in particular for high-dimensional data, a variety of simulation approaches have been introduced including parametric and the so-called plasmode simulations. While there are concerns about the realism of parametrically simulated data, it is widely claimed that plasmodes come very close to reality with some aspects of the "truth" known. However, there are no explicit guidelines or state-of-the-art on how to perform plasmode data simulations. In the present paper, we first review existing literature and introduce the concept of statistical plasmode simulation. We then discuss advantages and challenges of statistical plasmodes and provide a step-wise procedure for their generation, including key steps to their implementation and reporting. Finally, we illustrate the concept of statistical plasmodes as well as the proposed plasmode generation procedure by means of a public real RNA data set on breast carcinoma patients.

Longitudinal plasmode algorithms to evaluate statistical methods in realistic scenarios: an illustration applied to occupational epidemiology

INTRODUCTION

Plasmode simulations are a type of simulations that use real data to determine the synthetic data-generating equations. Such simulations thus allow evaluating statistical methods under realistic conditions. As far as we know, no plasmode algorithm has been proposed for simulating longitudinal data. In this paper, we propose a longitudinal plasmode framework to generate realistic data with both a time-varying exposure and time-varying covariates. This work was motivated by the objective of comparing different methods for estimating the causal effect of a cumulative exposure to psychosocial stressors at work over time.

METHODS

We developed two longitudinal plasmode algorithms: a parametric and a nonparametric algorithms. Data from the PROspective Québec (PROQ) Study on Work and Health were used as an input to generate data with the proposed plasmode algorithms. We evaluated the performance of multiple estimators of the parameters of marginal structural models (MSMs): inverse probability of treatment weighting, g-computation and targeted maximum likelihood estimation. These estimators were also compared to standard regression approaches with either adjustment for baseline covariates only or with adjustment for both baseline and time-varying covariates.

RESULTS

Standard regression methods were susceptible to yield biased estimates with confidence intervals having coverage probability lower than their nominal level. The bias was much lower and coverage of confidence intervals was much closer to the nominal level when considering MSMs. Among MSM estimators, g-computation overall produced the best results relative to bias, root mean squared error and coverage of confidence intervals. No method produced unbiased estimates with adequate coverage for all parameters in the more realistic nonparametric plasmode simulation.

CONCLUSION

The proposed longitudinal plasmode algorithms can be important methodological tools for evaluating and comparing analytical methods in realistic simulation scenarios. To facilitate the use of these algorithms, we provide R functions on GitHub. We also recommend using MSMs when estimating the effect of cumulative exposure to psychosocial stressors at work.

Recent evidence of economic evaluation of artificial intelligence in ophthalmology

PURPOSE OF REVIEW

Health economic evaluation (HEE) is essential for assessing value of health interventions, including artificial intelligence. Recent approaches, current challenges, and future directions of HEE of artificial intelligence in ophthalmology are reviewed.

RECENT FINDINGS

Majority of recent HEEs of artificial intelligence in ophthalmology were for diabetic retinopathy screening. Two models, one conducted in the rural USA (5-year period) and another in China (35-year period), found artificial intelligence to be more cost-effective than without screening for diabetic retinopathy. Two additional models, which compared artificial intelligence with human screeners in Brazil and Thailand for the lifetime of patients, found artificial intelligence to be more expensive from a healthcare system perspective. In the Thailand analysis, however, artificial intelligence was less expensive when opportunity loss from blindness was included. An artificial intelligence model for screening retinopathy of prematurity was cost-effective in the USA. A model for screening age-related macular degeneration in Japan and another for primary angle close in China did not find artificial intelligence to be cost-effective, compared with no screening. The costs of artificial intelligence varied widely in these models.

SUMMARY

Like other medical fields, there is limited evidence in assessing the value of artificial intelligence in ophthalmology and more appropriate HEE models are needed.

The Lung Allocation Score and Other Available Models Lack Predictive Accuracy for Post-Lung Transplant Survival

BACKGROUND

Improved predictive models are needed in lung transplantation in the setting of a proposed allocation system that incorporates longer-term post-transplant survival in the United States. Allocation systems require accurate mortality predictions to justly allocate organs.

METHODS

Utilizing the United Network for Organ Sharing database (2005-2017), we fit models to predict 1-year mortality based on the Lung Allocation Score (LAS), the Chan, et al, 2019 model, a novel "clinician" model (a priori clinician selection of pre-transplant covariates), and two machine learning models (Least Absolute Shrinkage and Selection Operator; LASSO and Random Forests) for predicting 1-year and 3-year post-transplant mortality. We compared predictive accuracy among models. We evaluated the calibration of models by comparing average predicted probability vs observed outcome per decile. We repeated analyses fit for 3-year mortality, disease category, including donor covariates, and LAS era.

RESULTS

The area under the cure for all models was low, ranging from 0.55 to 0.62. All exhibited reasonable negative predictive values (0.87-0.90), but the positive predictive value for was poor (all <0.25). Evaluating LAS calibration found 1-year post-transplant estimates consistently overestimated risk of mortality, with greater differences in higher deciles. LASSO, Random Forests, and clinician models showed no improvement when evaluated by disease category or with the addition of donor covariates and performed worse for 3-year outcomes.

CONCLUSIONS

The LAS overestimated patients' risk of post-transplant death, thus underestimating transplant benefit in the sickest candidates. Novel models based on pre-transplant recipient covariates failed to improve prediction. There should be wariness in post-transplant survival predictions from available models.