Applied Informatics Decision Support Tool for Mortality Predictions in Patients With Cancer

Prediction of prognosis in lung cancer using machine learning with inter-institutional generalizability: A multicenter cohort study (WJOG15121L: REAL-WIND)

OBJECTIVES

Predicting the prognosis of lung cancer is crucial for providing optimal medical care. However, a method to accurately predict the overall prognosis in patients with stage IV lung cancer, even with the use of machine learning, has not been established. Moreover, the inter-institutional generalizability of such algorithms remains unexplored. This study aimed to establish machine learning-based algorithms with inter-institutional generalizability to predict prognosis.

MATERIALS AND METHODS

This multicenter, retrospective, hospital-based cohort study included consecutive patients with stage IV lung cancer who were randomly categorized into the training and independent test cohorts with a 2:1 ratio, respectively. The primary metric to assess algorithm performance was the area under the receiver operating characteristic curve in the independent test cohort. To assess the inter-institutional generalizability of the algorithms, we investigated their ability to predict patient outcomes in the remaining facility after being trained using data from 15 other facilities.

RESULTS

Overall, 6,751 patients (median age, 70 years) were enrolled, and 1,515 (22 %) showed mutated epidermal growth factor receptor expression. The median overall survival was 16.6 (95 % confidence interval, 15.9-17.5) months. Algorithm performance metrics in the test cohort showed that the areas under the curves were 0.90 (95 % confidence interval, 0.88-0.91), 0.85 (0.84-0.87), 0.83 (0.81-0.85), and 0.85 (0.82-0.87) at 180, 360, 720, and 1,080 predicted survival days, respectively. The performance test of 16 algorithms for investigating inter-institutional generalizability showed median areas under the curves of 0.87 (range, 0.84-0.92), 0.84 (0.78-0.88), 0.84 (0.76-0.89), and 0.84 (0.75-0.90) at 180, 360, 720, and 1,080 days, respectively.

CONCLUSION

This study developed machine learning algorithms that could accurately predict the prognosis in patients with stage IV lung cancer with high inter-institutional generalizability. This can enhance the accuracy of prognosis prediction and support informed and shared decision-making in clinical settings.

Comparing Clinician Estimates versus a Statistical Tool for Predicting Risk of Death within 45 Days of Admission for Cancer Patients

OBJECTIVES

 While clinical practice guidelines recommend that oncologists discuss goals of care with patients who have advanced cancer, it is estimated that less than 20% of individuals admitted to the hospital with high-risk cancers have end-of-life discussions with their providers. While there has been interest in developing models for mortality prediction to trigger such discussions, few studies have compared how such models compare with clinical judgment to determine a patient's mortality risk.

METHODS

 This study is a prospective analysis of 1,069 solid tumor medical oncology hospital admissions (n = 911 unique patients) from February 7 to June 7, 2022, at Memorial Sloan Kettering Cancer Center. Electronic surveys were sent to hospitalists, advanced practice providers, and medical oncologists the first afternoon following a hospital admission and they were asked to estimate the probability that the patient would die within 45 days. Provider estimates of mortality were compared with those from a predictive model developed using a supervised machine learning methodology, and incorporated routine laboratory, demographic, biometric, and admission data. Area under the receiver operating characteristic curve (AUC), calibration and decision curves were compared between clinician estimates and the model predictions.

RESULTS

 Within 45 days following hospital admission, 229 (25%) of 911 patients died. The model performed better than the clinician estimates (AUC 0.834 vs. 0.753, p < 0.0001). Integrating clinician predictions with the model's estimates further increased the AUC to 0.853 (p < 0.0001). Clinicians overestimated risk whereas the model was extremely well-calibrated. The model demonstrated net benefit over a wide range of threshold probabilities.

CONCLUSION

 The inpatient prognosis at admission model is a robust tool to assist clinical providers in evaluating mortality risk, and it has recently been implemented in the electronic medical record at our institution to improve end-of-life care planning for hospitalized cancer patients.

Using Electronic Health Records to Facilitate Precision Psychiatry

The use of clinical prediction models to produce individualized risk estimates can facilitate the implementation of precision psychiatry. As a source of data from large, clinically representative patient samples, electronic health records (EHRs) provide a platform to develop and validate clinical prediction models, as well as potentially implement them in routine clinical care. The current review describes promising use cases for the application of precision psychiatry to EHR data and considers their performance in terms of discrimination (ability to separate individuals with and without the outcome) and calibration (extent to which predicted risk estimates correspond to observed outcomes), as well as their potential clinical utility (weighing benefits and costs associated with the model compared to different approaches across different assumptions of the number needed to test). We review 4 externally validated clinical prediction models designed to predict psychosis onset, psychotic relapse, cardiometabolic morbidity, and suicide risk. We then discuss the prospects for clinically implementing these models and the potential added value of integrating data from evidence syntheses, standardized psychometric assessments, and biological data into EHRs. Clinical prediction models can utilize routinely collected EHR data in an innovative way, representing a unique opportunity to inform real-world clinical decision making. Combining data from other sources (e.g., meta-analyses) or enhancing EHR data with information from research studies (clinical and biomarker data) may enhance our abilities to improve the performance of clinical prediction models.

Longitudinal clinical data improve survival prediction after hematopoietic cell transplantation using machine learning

ABSTRACT

Serial prognostic evaluation after allogeneic hematopoietic cell transplantation (allo-HCT) might help identify patients at high risk of lethal organ dysfunction. Current prediction algorithms based on models that do not incorporate changes to patients' clinical condition after allo-HCT have limited predictive ability. We developed and validated a robust risk-prediction algorithm to predict short- and long-term survival after allo-HCT in pediatric patients that includes baseline biological variables and changes in the patients' clinical status after allo-HCT. The model was developed using clinical data from children and young adults treated at a single academic quaternary-care referral center. The model was created using a randomly split training data set (70% of the cohort), internally validated (remaining 30% of the cohort) and then externally validated on patient data from another tertiary-care referral center. Repeated clinical measurements performed from 30 days before allo-HCT to 30 days afterwards were extracted from the electronic medical record and incorporated into the model to predict survival at 100 days, 1 year, and 2 years after allo-HCT. Naïve-Bayes machine learning models incorporating longitudinal data were significantly better than models constructed from baseline variables alone at predicting whether patients would be alive or deceased at the given time points. This proof-of-concept study demonstrates that unlike traditional prognostic tools that use fixed variables for risk assessment, incorporating dynamic variability using clinical and laboratory data improves the prediction of mortality in patients undergoing allo-HCT.

A review of the utility of prognostic tools in predicting 6-month mortality in cancer patients, conducted in the context of voluntary assisted dying

BACKGROUND

Eligibility to access the Victorian voluntary assisted dying (VAD) legislation requires that people have a prognosis of 6 months or less (or 12 months or less in the setting of a neurodegenerative diagnosis). Yet prognostic determination is frequently inaccurate and prompts clinician discomfort. Based on functional capacity and clinical and biochemical markers, prognostic tools have been developed to increase the accuracy of life expectancy predictions.

AIMS

This review of prognostic tools explores their accuracy to determine 6-month mortality in adults when treated under palliative care with a primary diagnosis of cancer (the diagnosis of a large proportion of people who are requesting VAD).

METHODS

A systematic search of the literature was performed on electronic databases Medline, Embase and Cinahl.

RESULTS

Limitations of prognostication identified include the following: (i) prognostic tools still provide uncertain prognoses; (ii) prognostic tools have greater accuracy predicting shorter prognoses, such as weeks to months, rather than 6 months; and (iii) functionality was often weighted significantly when calculating prognoses. Challenges of prognostication identified include the following: (i) the area under the curve (a value that represents how well a model can distinguish between two outcomes) cannot be directly interpreted clinically and (ii) difficulties exist related to determining appropriate thresholds of accuracy in this context.

CONCLUSIONS

Prognostication is a significant aspect of VAD, and the utility of the currently available prognostic tools appears limited but may prompt discussions about prognosis and alternative means (other than prognostic estimates) to identify those eligible for VAD.

Deep Learning Prediction Model for Patient Survival Outcomes in Palliative Care Using Actigraphy Data and Clinical Information

(1) Background: Predicting the survival of patients in end-of-life care is crucial, and evaluating their performance status is a key factor in determining their likelihood of survival. However, the current traditional methods for predicting survival are limited due to their subjective nature. Wearable technology that provides continuous patient monitoring is a more favorable approach for predicting survival outcomes among palliative care patients. (2) Aims and objectives: In this study, we aimed to explore the potential of using deep learning (DL) model approaches to predict the survival outcomes of end-stage cancer patients. Furthermore, we also aimed to compare the accuracy of our proposed activity monitoring and survival prediction model with traditional prognostic tools, such as the Karnofsky Performance Scale (KPS) and the Palliative Performance Index (PPI). (3) Method: This study recruited 78 patients from the Taipei Medical University Hospital's palliative care unit, with 66 (39 male and 27 female) patients eventually being included in our DL model for predicting their survival outcomes. (4) Results: The KPS and PPI demonstrated an overall accuracy of 0.833 and 0.615, respectively. In comparison, the actigraphy data exhibited a higher accuracy at 0.893, while the accuracy of the wearable data combined with clinical information was even better, at 0.924. (5) Conclusion: Our study highlights the significance of incorporating clinical data alongside wearable sensors to predict prognosis. Our findings suggest that 48 h of data is sufficient for accurate predictions. The integration of wearable technology and the prediction model in palliative care has the potential to improve decision making for healthcare providers and can provide better support for patients and their families. The outcomes of this study can possibly contribute to the development of personalized and patient-centered end-of-life care plans in clinical practice.

Machine learning models for 180-day mortality prediction of patients with advanced cancer using patient-reported symptom data

PURPOSE

The objective of the current study was to develop and test the performances of different ML algorithms which were trained using patient-reported symptom severity data to predict mortality within 180 days for patients with advanced cancer.

METHODS

We randomly selected 630 of 689 patients with advanced cancer at our institution who completed symptom PRO measures as part of routine care between 2009 and 2020. Using clinical, demographic, and PRO data, we trained and tested four ML algorithms: generalized regression with elastic net regularization (GLM), extreme gradient boosting (XGBoost) trees, support vector machines (SVM), and a single hidden layer neural network (NNET). We assessed the performance of algorithms individually as well as part of an unweighted voting ensemble on the hold-out testing sample. Performance was assessed using area under the receiver-operating characteristic curve (AUROC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV).

RESULTS

The starting cohort of 630 patients was randomly partitioned into training (n = 504) and testing (n = 126) samples. Of the four ML models, the XGBoost algorithm demonstrated the best performance for 180-day mortality prediction in testing data (AUROC = 0.69, sensitivity = 0.68, specificity = 0.62, PPV = 0.66, NPV = 0.64). Ensemble of all algorithms performed worst (AUROC = 0.65, sensitivity = 0.65, specificity = 0.62, PPV = 0.65, NPV = 0.62). Of individual PRO symptoms, shortness of breath emerged as the variable of highest impact on the XGBoost 180-mortality prediction (1-AUROC = 0.30).

CONCLUSION

Our findings support ML models driven by patient-reported symptom severity as accurate predictors of short-term mortality in patients with advanced cancer, highlighting the opportunity to integrate these models prospectively into future studies of goal-concordant care.

On the importance of interpretable machine learning predictions to inform clinical decision making in oncology

Machine learning-based tools are capable of guiding individualized clinical management and decision-making by providing predictions of a patient's future health state. Through their ability to model complex nonlinear relationships, ML algorithms can often outperform traditional statistical prediction approaches, but the use of nonlinear functions can mean that ML techniques may also be less interpretable than traditional statistical methodologies. While there are benefits of intrinsic interpretability, many model-agnostic approaches now exist and can provide insight into the way in which ML systems make decisions. In this paper, we describe how different algorithms can be interpreted and introduce some techniques for interpreting complex nonlinear algorithms.

Development of Machine Learning Algorithms Incorporating Electronic Health Record Data, Patient-Reported Outcomes, or Both to Predict Mortality for Outpatients With Cancer

PURPOSE

Machine learning (ML) algorithms that incorporate routinely collected patient-reported outcomes (PROs) alongside electronic health record (EHR) variables may improve prediction of short-term mortality and facilitate earlier supportive and palliative care for patients with cancer.

METHODS

We trained and validated two-phase ML algorithms that incorporated standard PRO assessments alongside approximately 200 routinely collected EHR variables, among patients with medical oncology encounters at a tertiary academic oncology and a community oncology practice.

RESULTS

Among 12,350 patients, 5,870 (47.5%) completed PRO assessments. Compared with EHR- and PRO-only algorithms, the EHR + PRO model improved predictive performance in both tertiary oncology (EHR + PRO v EHR v PRO: area under the curve [AUC] 0.86 [0.85-0.87] v 0.82 [0.81-0.83] v 0.74 [0.74-0.74]) and community oncology (area under the curve 0.89 [0.88-0.90] v 0.86 [0.85-0.88] v 0.77 [0.76-0.79]) practices.

CONCLUSION

Routinely collected PROs contain added prognostic information not captured by an EHR-based ML mortality risk algorithm. Augmenting an EHR-based algorithm with PROs resulted in a more accurate and clinically relevant model, which can facilitate earlier and targeted supportive care for patients with cancer.

Palliative Care Exposure Relative to Predicted Risk of Six-Month Mortality in Hospitalized Adults

CONTEXT

The optimal strategy for implementing mortality-predicting algorithms to facilitate clinical care, prognostic discussions, and palliative care interventions remains unknown.

OBJECTIVES

To develop and validate a real-time predictive model for 180 day mortality using routinely available clinical and laboratory admission data and determine if palliative care exposure varies with predicted mortality risk.

METHODS

Adult admissions between October 1, 2013 and October.1, 2017 were included for the model derivation. A separate cohort was collected between January 1, 2018 and July 31, 2020 for validation. Patients were followed for 180 days from discharge, and logistic regression with selected variables was used to estimate patients' risk for mortality.

RESULTS

In the model derivation cohort, 7963 events of 180 day mortality (4.5% event rate) were observed. Median age was 53.0 (IQR 24.0-66.0) with 92,734 females (52.5%). Variables with strongest association with 180 day mortality included: Braden Score (OR 0.83; 95% CI 0.82-0.84); admission Do Not Resuscitate orders (OR 2.61; 95% CI 2.43-2.79); admission service and admission status. The model yielded excellent discriminatory ability in both the derivation (c-statistic 0.873; 95% CI 0.870-0.877; Brier score 0.04) and validation cohorts (c-statistic 0.844; 95% CI 0.840-0.847; Brier score 0.072). Inpatient palliative care consultations increased from 3% of minimal-risk encounters to 41% of high-risk encounters (P < 0.01).

CONCLUSION

We developed and temporally validated a predictive mortality model for adults from a large retrospective cohort, which helps quantify the potential need for palliative care referrals based on risk strata. Machine learning algorithms for mortality require clinical interpretation, and additional studies are needed to design patient-centered and risk-specific interventions.

Development of a Machine Learning Model Using Limited Features to Predict 6-Month Mortality at Treatment Decision Points for Patients With Advanced Solid Tumors

Patients with advanced solid tumors may receive intensive treatments near the end of life. This study aimed to create a machine learning (ML) model using limited features to predict 6-month mortality at treatment decision points (TDPs).

Predicting 6-month mortality at treatment decisions for patients with advanced solid tumors.

Impact of Behavioral Nudges on the Quality of Serious Illness Conversations Among Patients With Cancer: Secondary Analysis of a Randomized Controlled Trial

PURPOSE

Serious Illness Conversations (SICs) are structured conversations between clinicians and patients about prognosis, treatment goals, and end-of-life preferences. Although behavioral interventions may prompt earlier or more frequent SICs, their impact on the quality of SICs is unclear.

METHODS

This was a secondary analysis of a randomized clinical trial (NCT03984773) among 78 clinicians and 14,607 patients with cancer testing the impact of an automated mortality prediction with behavioral nudges to clinicians to prompt more SICs. We analyzed 318 randomly selected SICs matched 1:1 by clinicians (159 control and 159 intervention) to compare the quality of intervention vs. control conversations using a validated codebook. Comprehensiveness of SIC documentation was used as a measure of quality, with higher integer numbers of documented conversation domains corresponding to higher quality conversations. A conversation was classified as high-quality if its score was ≥ 8 of a maximum of 10. Using a noninferiority design, mixed effects regression models with clinician-level random effects were used to assess SIC quality in intervention vs. control groups, concluding noninferiority if the adjusted odds ratio (aOR) was not significantly < 0.9.

RESULTS

Baseline characteristics of the control and intervention groups were similar. Intervention SICs were noninferior to control conversations (aOR 0.99; 95% CI, 0.91 to 1.09). The intervention increased the likelihood of addressing patient-clinician relationship (aOR = 1.99; 95% CI, 1.23 to 3.27; P < .01) and decreased the likelihood of addressing family involvement (aOR = 0.56; 95% CI, 0.34 to 0.90; P < .05).

CONCLUSION

A behavioral intervention that increased SIC frequency did not decrease their quality. Behavioral prompts may increase SIC frequency without sacrificing quality.

The Serious Illness Care Program in Oncology: Evidence, Real-World Implementation and Ongoing Barriers

The Serious Illness Care Program (SICP), designed by Ariadne Labs, is a multicomponent intervention to improve conversations about values and goals for patients with a life-limiting illness. In oncology, implementation of the SICP achieved more, earlier, and better-quality conversations and reduced anxiety and depression among patients with advanced cancer. In this commentary, we describe the SICP, including results from the cluster-randomized trial, provide examples of real-world implementation of this program, and highlight ongoing challenges and barriers that are preventing widespread adoption of this intervention into routine practice. For the SICP to be successfully embedded into routine patient care, it will require significant effort, including ongoing leadership support and training opportunities, champions from all sectors of the interdisciplinary team, and adaptation of the program to a wider range of patients. Future research should also investigate how early conversations can be translated into personalized care plans for patients.

Oncologist phenotypes and associations with response to a machine learning-based intervention to increase advance care planning: Secondary analysis of a randomized clinical trial

BACKGROUND

While health systems have implemented multifaceted interventions to improve physician and patient communication in serious illnesses such as cancer, clinicians vary in their response to these initiatives. In this secondary analysis of a randomized trial, we identified phenotypes of oncology clinicians based on practice pattern and demographic data, then evaluated associations between such phenotypes and response to a machine learning (ML)-based intervention to prompt earlier advance care planning (ACP) for patients with cancer.

METHODS AND FINDINGS

Between June and November 2019, we conducted a pragmatic randomized controlled trial testing the impact of text message prompts to 78 oncology clinicians at 9 oncology practices to perform ACP conversations among patients with cancer at high risk of 180-day mortality, identified using a ML prognostic algorithm. All practices began in the pre-intervention group, which received weekly emails about ACP performance only; practices were sequentially randomized to receive the intervention at 4-week intervals in a stepped-wedge design. We used latent profile analysis (LPA) to identify oncologist phenotypes based on 11 baseline demographic and practice pattern variables identified using EHR and internal administrative sources. Difference-in-differences analyses assessed associations between oncologist phenotype and the outcome of change in ACP conversation rate, before and during the intervention period. Primary analyses were adjusted for patients' sex, age, race, insurance status, marital status, and Charlson comorbidity index. The sample consisted of 2695 patients with a mean age of 64.9 years, of whom 72% were White, 20% were Black, and 52% were male. 78 oncology clinicians (42 oncologists, 36 advanced practice providers) were included. Three oncologist phenotypes were identified: Class 1 (n = 9) composed primarily of high-volume generalist oncologists, Class 2 (n = 5) comprised primarily of low-volume specialist oncologists; and 3) Class 3 (n = 28), composed primarily of high-volume specialist oncologists. Compared with class 1 and class 3, class 2 had lower mean clinic days per week (1.6 vs 2.5 [class 3] vs 4.4 [class 1]) a higher percentage of new patients per week (35% vs 21% vs 18%), higher baseline ACP rates (3.9% vs 1.6% vs 0.8%), and lower baseline rates of chemotherapy within 14 days of death (1.4% vs 6.5% vs 7.1%). Overall, ACP rates were 3.6% in the pre-intervention wedges and 15.2% in intervention wedges (11.6 percentage-point difference). Compared to class 3, oncologists in class 1 (adjusted percentage-point difference-in-differences 3.6, 95% CI 1.0 to 6.1, p = 0.006) and class 2 (adjusted percentage-point difference-in-differences 12.3, 95% confidence interval [CI] 4.3 to 20.3, p = 0.003) had greater response to the intervention.

CONCLUSIONS

Patient volume and time availability may be associated with oncologists' response to interventions to increase ACP. Future interventions to prompt ACP should prioritize making time available for such conversations between oncologists and their patients.

Impact of Augmented Intelligence on Utilization of Palliative Care Services in a Real-World Oncology Setting

PURPOSE

For patients with advanced cancer, timely referral to palliative care (PC) services can ensure that end-of-life care aligns with their preferences and goals. Overestimation of life expectancy may result in underutilization of PC services, counterproductive treatment measures, and reduced quality of life for patients. We assessed the impact of a commercially available augmented intelligence (AI) tool to predict 30-day mortality risk on PC service utilization in a real-world setting.

METHODS

Patients within a large hematology-oncology practice were scored weekly between June 2018 and October 2019 with an AI tool to generate insights into short-term mortality risk. Patients identified by the tool as being at high or medium risk were assessed for a supportive care visit and further referred as appropriate. Average monthly rates of PC and hospice referrals were calculated 5 months predeployment and 17 months postdeployment of the tool in the practice.

RESULTS

The mean rate of PC consults increased from 17.3 to 29.1 per 1,000 patients per month (PPM) pre- and postdeployment, whereas the mean rate of hospice referrals increased from 0.2 to 1.6 per 1,000 PPM. Eliminating the first 6 months following deployment to account for user learning curve, the mean rate of PC consults nearly doubled over baseline to 33.0 and hospice referrals increased 12-fold to 2.4 PPM.

CONCLUSION

Deployment of an AI tool at a hematology-oncology practice was found to be feasible for identifying patients at high or medium risk for short-term mortality. Insights generated by the tool drove clinical practice changes, resulting in significant increases in PC and hospice referrals.

Machine learning models for mitral valve replacement: A comparative analysis with the Society of Thoracic Surgeons risk score

BACKGROUND

Current Society of Thoracic Surgeons (STS) risk models for predicting outcomes of mitral valve surgery (MVS) assume a linear and cumulative impact of variables. We evaluated postoperative MVS outcomes and designed mortality and morbidity risk calculators to supplement the STS risk score.

METHODS

Data from the STS Adult Cardiac Surgery Database for MVS was used from 2008 to 2017. The data included 383,550 procedures and 89 variables. Machine learning (ML) algorithms were employed to train models to predict postoperative outcomes for MVS patients. Each model's discrimination and calibration performance were validated using unseen data against the STS risk score.

RESULTS

Comprehensive mortality and morbidity risk assessment scores were derived from a training set of 287,662 observations. The area under the curve (AUC) for mortality ranged from 0.77 to 0.83, leading to a 3% increase in predictive accuracy compared to the STS score. Logistic Regression and eXtreme Gradient Boosting achieved the highest AUC for prolonged ventilation (0.82) and deep sternal wound infection (0.78 and 0.77) respectively. EXtreme Gradient Boosting performed the best with an AUC of 0.815 for renal failure. For permanent stroke prediction all models performed similarly with an AUC around 0.67. The ML models led to improved calibration performance for mortality, prolonged ventilation, and renal failure, especially in cases of reconstruction/repair and replacement surgery.

CONCLUSIONS

The proposed risk models complement existing STS models in predicting mortality, prolonged ventilation, and renal failure, allowing healthcare providers to more accurately assess a patient's risk of morbidity and mortality when undergoing MVS.

Research and Application of Artificial Intelligence (AI) based on Electronic Health Records from Patients with Cancer: a Systematic Review (Preprint)

Background

With the accumulation of electronic health records and the development of artificial intelligence, patients with cancer urgently need new evidence of more personalized clinical and demographic characteristics and more sophisticated treatment and prevention strategies. However, no research has systematically analyzed the application and significance of artificial intelligence based on electronic health records in cancer care.

Objective

The aim of this study was to conduct a review to introduce the current state and limitations of artificial intelligence based on electronic health records of patients with cancer and to summarize the performance of artificial intelligence in mining electronic health records and its impact on cancer care.

Methods

Three databases were systematically searched to retrieve potentially relevant papers published from January 2009 to October 2020. Four principal reviewers assessed the quality of the papers and reviewed them for eligibility based on the inclusion criteria in the extracted data. The summary measures used in this analysis were the number and frequency of occurrence of the themes.

Results

Of the 1034 papers considered, 148 papers met the inclusion criteria. Cancer care, especially cancers of female organs and digestive organs, could benefit from artificial intelligence based on electronic health records through cancer emergencies and prognostic estimates, cancer diagnosis and prediction, tumor stage detection, cancer case detection, and treatment pattern recognition. The models can always achieve an area under the curve of 0.7. Ensemble methods and deep learning are on the rise. In addition, electronic medical records in the existing studies are mainly in English and from private institutional databases.

Conclusions

Artificial intelligence based on electronic health records performed well and could be useful for cancer care. Improving the performance of artificial intelligence can help patients receive more scientific-based and accurate treatments. There is a need for the development of new methods and electronic health record data sharing and for increased passion and support from cancer specialists.

Machine Learning for the Prediction of Amyloid Positivity in Amnestic Mild Cognitive Impairment

BACKGROUND

Amyloid-β (Aβ) evaluation in amnestic mild cognitive impairment (aMCI) patients is important for predicting conversion to Alzheimer's disease. However, Aβ evaluation through Aβ positron emission tomography (PET) is limited due to high cost and safety issues.

OBJECTIVE

We therefore aimed to develop and validate prediction models of Aβ positivity for aMCI using optimal interpretable machine learning (ML) approaches utilizing multimodal markers.

METHODS

We recruited 529 aMCI patients from multiple centers who underwent Aβ PET. We trained ML algorithms using a training cohort (324 aMCI from Samsung medical center) with two-phase modelling: model 1 included age, gender, education, diabetes, hypertension, apolipoprotein E genotype, and neuropsychological test scores; model 2 included the same variables as model 1 with additional MRI features. We used four-fold cross-validation during the modelling and evaluated the models on an external validation cohort (187 aMCI from the other centers).

RESULTS

Model 1 showed good accuracy (area under the receiver operating characteristic curve [AUROC] 0.837) in cross-validation, and fair accuracy (AUROC 0.765) in external validation. Model 2 led to improvement in the prediction performance with good accuracy (AUROC 0.892) in cross validation compared to model 1. Apolipoprotein E genotype, delayed recall task scores, and interaction between cortical thickness in the temporal region and hippocampal volume were the most important predictors of Aβ positivity.

CONCLUSION

Our results suggest that ML models are effective in predicting Aβ positivity at the individual level and could help the biomarker-guided diagnosis of prodromal AD.

Two-Stage Approaches to Accounting for Patient Heterogeneity in Machine Learning Risk Prediction Models in Oncology

PURPOSE

Machine learning models developed from electronic health records data have been increasingly used to predict risk of mortality for general oncology patients. But these models may have suboptimal performance because of patient heterogeneity. The objective of this work is to develop a new modeling approach to predicting short-term mortality that accounts for heterogeneity across multiple subgroups in the presence of a large number of electronic health record predictors.

METHODS

We proposed a two-stage approach to addressing heterogeneity among oncology patients of different cancer types for predicting their risk of mortality. Structured data were extracted from the University of Pennsylvania Health System for 20,723 patients of 11 cancer types, where 1,340 (6.5%) patients were deceased. We first modeled the overall risk for all patients without differentiating cancer types, as is done in the current practice. We then developed cancer type-specific models using the overall risk score as a predictor along with preselected type-specific predictors. The overall and type-specific models were compared with respect to discrimination using the area under the precision-recall curve (AUPRC) and calibration using the calibration slope. We also proposed metrics that characterize the degree of risk heterogeneity by comparing risk predictors in the overall and type-specific models.

RESULTS

The two-stage modeling resulted in improved calibration and discrimination across all 11 cancer types. The improvement in AUPRC was significant for hematologic malignancies including leukemia, lymphoma, and myeloma. For instance, the AUPRC increased from 0.358 to 0.519 (∆ = 0.161; 95% CI, 0.102 to 0.224) and from 0.299 to 0.354 (∆ = 0.055; 95% CI, 0.009 to 0.107) for leukemia and lymphoma, respectively. For all 11 cancer types, the two-stage approach generated well-calibrated risks. A high degree of heterogeneity between type-specific and overall risk predictors was observed for most cancer types.

CONCLUSION

Our two-stage modeling approach that accounts for cancer type-specific risk heterogeneity has improved calibration and discrimination than a model agnostic to cancer types.

Augmented intelligence to predict 30-day mortality in patients with cancer

Aim: An augmented intelligence tool to predict short-term mortality risk among patients with cancer could help identify those in need of actionable interventions or palliative care services. Patients & methods: An algorithm to predict 30-day mortality risk was developed using socioeconomic and clinical data from patients in a large community hematology/oncology practice. Patients were scored weekly; algorithm performance was assessed using dates of death in patients' electronic health records. Results: For patients scored as highest risk for 30-day mortality, the event rate was 4.9% (vs 0.7% in patients scored as low risk; a 7.4-times greater risk). Conclusion: The development and validation of a decision tool to accurately identify patients with cancer who are at risk for short-term mortality is feasible.

Electronic patient-reported outcomes and machine learning in predicting immune-related adverse events of immune checkpoint inhibitor therapies

BACKGROUND

Immune-checkpoint inhibitors (ICIs) have introduced novel immune-related adverse events (irAEs), arising from various organ systems without strong timely dependency on therapy dosing. Early detection of irAEs could result in improved toxicity profile and quality of life. Symptom data collected by electronic (e) patient-reported outcomes (PRO) could be used as an input for machine learning (ML) based prediction models for the early detection of irAEs.

METHODS

The utilized dataset consisted of two data sources. The first dataset consisted of 820 completed symptom questionnaires from 34 ICI treated advanced cancer patients, including 18 monitored symptoms collected using the Kaiku Health digital platform. The second dataset included prospectively collected irAE data, Common Terminology Criteria for Adverse Events (CTCAE) class, and the severity of 26 irAEs. The ML models were built using extreme gradient boosting algorithms. The first model was trained to detect the presence and the second the onset of irAEs.

RESULTS

The model trained to predict the presence of irAEs had an excellent performance based on four metrics: accuracy score 0.97, Area Under the Curve (AUC) value 0.99, F1-score 0.94 and Matthew's correlation coefficient (MCC) 0.92. The prediction of the irAE onset was more difficult with accuracy score 0.96, AUC value 0.93, F1-score 0.66 and MCC 0.64 but the model performance was still at a good level.

CONCLUSION

The current study suggests that ML based prediction models, using ePRO data as an input, can predict the presence and onset of irAEs with a high accuracy, indicating that ePRO follow-up with ML algorithms could facilitate the detection of irAEs in ICI-treated cancer patients. The results should be validated with a larger dataset. Trial registration Clinical Trials Register (NCT3928938), registration date the 26th of April, 2019.

Machine learning for subtype definition and risk prediction in heart failure, acute coronary syndromes and atrial fibrillation: systematic review of validity and clinical utility

BACKGROUND

Machine learning (ML) is increasingly used in research for subtype definition and risk prediction, particularly in cardiovascular diseases. No existing ML models are routinely used for cardiovascular disease management, and their phase of clinical utility is unknown, partly due to a lack of clear criteria. We evaluated ML for subtype definition and risk prediction in heart failure (HF), acute coronary syndromes (ACS) and atrial fibrillation (AF).

METHODS

For ML studies of subtype definition and risk prediction, we conducted a systematic review in HF, ACS and AF, using PubMed, MEDLINE and Web of Science from January 2000 until December 2019. By adapting published criteria for diagnostic and prognostic studies, we developed a seven-domain, ML-specific checklist.

RESULTS

Of 5918 studies identified, 97 were included. Across studies for subtype definition (n = 40) and risk prediction (n = 57), there was variation in data source, population size (median 606 and median 6769), clinical setting (outpatient, inpatient, different departments), number of covariates (median 19 and median 48) and ML methods. All studies were single disease, most were North American (n = 61/97) and only 14 studies combined definition and risk prediction. Subtype definition and risk prediction studies respectively had limitations in development (e.g. 15.0% and 78.9% of studies related to patient benefit; 15.0% and 15.8% had low patient selection bias), validation (12.5% and 5.3% externally validated) and impact (32.5% and 91.2% improved outcome prediction; no effectiveness or cost-effectiveness evaluations).

CONCLUSIONS

Studies of ML in HF, ACS and AF are limited by number and type of included covariates, ML methods, population size, country, clinical setting and focus on single diseases, not overlap or multimorbidity. Clinical utility and implementation rely on improvements in development, validation and impact, facilitated by simple checklists. We provide clear steps prior to safe implementation of machine learning in clinical practice for cardiovascular diseases and other disease areas.

Has the Flood Entered the Basement? A Systematic Literature Review about Machine Learning in Laboratory Medicine

This article presents a systematic literature review that expands and updates a previous review on the application of machine learning to laboratory medicine. We used Scopus and PubMed to collect, select and analyse the papers published from 2017 to the present in order to highlight the main studies that have applied machine learning techniques to haematochemical parameters and to review their diagnostic and prognostic performance. In doing so, we aim to address the question we asked three years ago about the potential of these techniques in laboratory medicine and the need to leverage a tool that was still under-utilised at that time.

Systematic literature review of machine learning methods used in the analysis of real-world data for patient-provider decision making

BACKGROUND

Machine learning is a broad term encompassing a number of methods that allow the investigator to learn from the data. These methods may permit large real-world databases to be more rapidly translated to applications to inform patient-provider decision making.

METHODS

This systematic literature review was conducted to identify published observational research of employed machine learning to inform decision making at the patient-provider level. The search strategy was implemented and studies meeting eligibility criteria were evaluated by two independent reviewers. Relevant data related to study design, statistical methods and strengths and limitations were identified; study quality was assessed using a modified version of the Luo checklist.

RESULTS

A total of 34 publications from January 2014 to September 2020 were identified and evaluated for this review. There were diverse methods, statistical packages and approaches used across identified studies. The most common methods included decision tree and random forest approaches. Most studies applied internal validation but only two conducted external validation. Most studies utilized one algorithm, and only eight studies applied multiple machine learning algorithms to the data. Seven items on the Luo checklist failed to be met by more than 50% of published studies.

CONCLUSIONS

A wide variety of approaches, algorithms, statistical software, and validation strategies were employed in the application of machine learning methods to inform patient-provider decision making. There is a need to ensure that multiple machine learning approaches are used, the model selection strategy is clearly defined, and both internal and external validation are necessary to be sure that decisions for patient care are being made with the highest quality evidence. Future work should routinely employ ensemble methods incorporating multiple machine learning algorithms.

Validation of a Machine Learning Algorithm to Predict 180-Day Mortality for Outpatients With Cancer

Importance

Machine learning (ML) algorithms can identify patients with cancer at risk of short-term mortality to inform treatment and advance care planning. However, no ML mortality risk prediction algorithm has been prospectively validated in oncology or compared with routinely used prognostic indices.

Objective

To validate an electronic health record-embedded ML algorithm that generated real-time predictions of 180-day mortality risk in a general oncology cohort.

Design, Setting, and Participants

This prognostic study comprised a prospective cohort of patients with outpatient oncology encounters between March 1, 2019, and April 30, 2019. An ML algorithm, trained on retrospective data from a subset of practices, predicted 180-day mortality risk between 4 and 8 days before a patient's encounter. Patient encounters took place in 18 medical or gynecologic oncology practices, including 1 tertiary practice and 17 general oncology practices, within a large US academic health care system. Patients aged 18 years or older with outpatient oncology or hematology and oncology encounters were included in the analysis. Patients were excluded if their appointment was scheduled after weekly predictions were generated and if they were only evaluated in benign hematology, palliative care, or rehabilitation practices.

Exposures

Gradient-boosting ML binary classifier.

Main Outcomes and Measures

The primary outcome was the patients' 180-day mortality from the index encounter. The primary performance metric was the area under the receiver operating characteristic curve (AUC).

Results

Among 24 582 patients, 1022 (4.2%) died within 180 days of their index encounter. Their median (interquartile range) age was 64.6 (53.6-73.2) years, 15 319 (62.3%) were women, 18 015 (76.0%) were White, and 10 658 (43.4%) were seen in the tertiary practice. The AUC was 0.89 (95% CI, 0.88-0.90) for the full cohort. The AUC varied across disease-specific groups within the tertiary practice (AUC ranging from 0.74 to 0.96) but was similar between the tertiary and general oncology practices. At a prespecified 40% mortality risk threshold used to differentiate high- vs low-risk patients, observed 180-day mortality was 45.2% (95% CI, 41.3%-49.1%) in the high-risk group vs 3.1% (95% CI, 2.9%-3.3%) in the low-risk group. Integrating the algorithm into the Eastern Cooperative Oncology Group and Elixhauser comorbidity index-based classifiers resulted in favorable reclassification (net reclassification index, 0.09 [95% CI, 0.04-0.14] and 0.23 [95% CI, 0.20-0.27], respectively).

Conclusions and Relevance

In this prognostic study, an ML algorithm was feasibly integrated into the electronic health record to generate real-time, accurate predictions of short-term mortality for patients with cancer and outperformed routinely used prognostic indices. This algorithm may be used to inform behavioral interventions and prompt earlier conversations about goals of care and end-of-life preferences among patients with cancer.

Machine learning provides evidence that stroke risk is not linear: The non-linear Framingham stroke risk score

Current stroke risk assessment tools presume the impact of risk factors is linear and cumulative. However, both novel risk factors and their interplay influencing stroke incidence are difficult to reveal using traditional additive models. The goal of this study was to improve upon the established Revised Framingham Stroke Risk Score and design an interactive Non-Linear Stroke Risk Score. Leveraging machine learning algorithms, our work aimed at increasing the accuracy of event prediction and uncovering new relationships in an interpretable fashion. A two-phase approach was used to create our stroke risk prediction score. First, clinical examinations of the Framingham offspring cohort were utilized as the training dataset for the predictive model. Optimal Classification Trees were used to develop a tree-based model to predict 10-year risk of stroke. Unlike classical methods, this algorithm adaptively changes the splits on the independent variables, introducing non-linear interactions among them. Second, the model was validated with a multi-ethnicity cohort from the Boston Medical Center. Our stroke risk score suggests a key dichotomy between patients with history of cardiovascular disease and the rest of the population. While it agrees with known findings, it also identified 23 unique stroke risk profiles and highlighted new non-linear relationships; such as the role of T-wave abnormality on electrocardiography and hematocrit levels in a patient’s risk profile. Our results suggested that the non-linear approach significantly improves upon the baseline in the c-statistic (training 87.43% (CI 0.85–0.90) vs. 73.74% (CI 0.70–0.76); validation 75.29% (CI 0.74–0.76) vs 65.93% (CI 0.64–0.67), even in multi-ethnicity populations. The clinical implications of the new risk score include prioritization of risk factor modification and personalized care at the patient level with improved targeting of interventions for stroke prevention.

Applying Machine Learning in Liver Disease and Transplantation: A Comprehensive Review

Machine learning (ML) utilizes artificial intelligence to generate predictive models efficiently and more effectively than conventional methods through detection of hidden patterns within large data sets. With this in mind, there are several areas within hepatology where these methods can be applied. In this review, we examine the literature pertaining to machine learning in hepatology and liver transplant medicine. We provide an overview of the strengths and limitations of ML tools and their potential applications to both clinical and molecular data in hepatology. ML has been applied to various types of data in liver disease research, including clinical, demographic, molecular, radiological, and pathological data. We anticipate that use of ML tools to generate predictive algorithms will change the face of clinical practice in hepatology and transplantation. This review will provide readers with the opportunity to learn about the ML tools available and potential applications to questions of interest in hepatology.

Prediction of cervical spine injury in young pediatric patients: an optimal trees artificial intelligence approach

BACKGROUND

Cervical spine injuries (CSI) are a major concern in young pediatric trauma patients. The consequences of missed injuries and difficulties in injury clearance for non-verbal patients have led to a tendency to image young children. Imaging, particularly computed tomography (CT) scans, presents risks including radiation-induced carcinogenesis. In this study we leverage machine learning methods to develop highly accurate clinical decision rules to predict pediatric CSI.

METHODS

The PEDSPINE I registry was used to investigate CSI in blunt trauma patients under the age of three. Predictive models were built using Optimal Classification Trees, a novel machine learning approach offering high accuracy and interpretability, as well as other widely used machine learning methods.

RESULTS

The final Optimal Classification Trees model predicts injury based on overall Glasgow Coma Score (GCS) and patient age. This model has a sensitivity of 93.3% and specificity of 82.3% on the full dataset. It has comparable or superior performance to other machine learning methods as well as existing clinical decision rules.

CONCLUSIONS

This study developed a decision rule that achieves high injury identification while reducing unnecessary imaging. It demonstrates the value of machine learning in improving clinical decision protocols for pediatric trauma.

TYPE OF STUDY

Retrospective Prognosis Study.

LEVEL OF EVIDENCE

II.

Machine Learning Approaches to Predict 6-Month Mortality Among Patients With Cancer

Importance

Machine learning algorithms could identify patients with cancer who are at risk of short-term mortality. However, it is unclear how different machine learning algorithms compare and whether they could prompt clinicians to have timely conversations about treatment and end-of-life preferences.

Objectives

To develop, validate, and compare machine learning algorithms that use structured electronic health record data before a clinic visit to predict mortality among patients with cancer.

Design, Setting, and Participants

Cohort study of 26 525 adult patients who had outpatient oncology or hematology/oncology encounters at a large academic cancer center and 10 affiliated community practices between February 1, 2016, and July 1, 2016. Patients were not required to receive cancer-directed treatment. Patients were observed for up to 500 days after the encounter. Data analysis took place between October 1, 2018, and September 1, 2019.

Exposures

Logistic regression, gradient boosting, and random forest algorithms.

Main Outcomes and Measures

Primary outcome was 180-day mortality from the index encounter; secondary outcome was 500-day mortality from the index encounter.

Results

Among 26 525 patients in the analysis, 1065 (4.0%) died within 180 days of the index encounter. Among those who died, the mean age was 67.3 (95% CI, 66.5-68.0) years, and 500 (47.0%) were women. Among those who were alive at 180 days, the mean age was 61.3 (95% CI, 61.1-61.5) years, and 15 922 (62.5%) were women. The population was randomly partitioned into training (18 567 [70.0%]) and validation (7958 [30.0%]) cohorts at the patient level, and a randomly selected encounter was included in either the training or validation set. At a prespecified alert rate of 0.02, positive predictive values were higher for the random forest (51.3%) and gradient boosting (49.4%) algorithms compared with the logistic regression algorithm (44.7%). There was no significant difference in discrimination among the random forest (area under the receiver operating characteristic curve [AUC], 0.88; 95% CI, 0.86-0.89), gradient boosting (AUC, 0.87; 95% CI, 0.85-0.89), and logistic regression (AUC, 0.86; 95% CI, 0.84-0.88) models (P for comparison = .02). In the random forest model, observed 180-day mortality was 51.3% (95% CI, 43.6%-58.8%) in the high-risk group vs 3.4% (95% CI, 3.0%-3.8%) in the low-risk group; at 500 days, observed mortality was 64.4% (95% CI, 56.7%-71.4%) in the high-risk group and 7.6% (7.0%-8.2%) in the low-risk group. In a survey of 15 oncology clinicians with a 52.1% response rate, 100 of 171 patients (58.8%) who had been flagged as having high risk by the gradient boosting algorithm were deemed appropriate for a conversation about treatment and end-of-life preferences in the upcoming week.

Conclusions and Relevance

In this cohort study, machine learning algorithms based on structured electronic health record data accurately identified patients with cancer at risk of short-term mortality. When the gradient boosting algorithm was applied in real time, clinicians believed that most patients who had been identified as having high risk were appropriate for a timely conversation about treatment and end-of-life preferences.

Predicting the outcome of patients with chronic lymphocytic leukemia: Progress and uncertainty

Because chronic lymphocytic leukemia is a heterogeneous disease, there are considerable efforts underway to develop increasingly accurate and precise analytics with which to estimate the probability of future events such as the need for and probability of response to therapy, progression-free survival, and survival. These analytics typically are constructed from clinical and laboratory variables. These variables often are combined into scores or staging systems, some of which are prognostic (therapy-independent), whereas others are predictive (therapy-dependent). Predictive variables differ with different therapies. Because response to therapy is a necessary condition for the improvement of survival, predictive biomarkers are extremely important. However, despite some progress to identify new predictive biomarkers, del(17p)/TP53 mutation remains the only widely accepted variable used to guide therapy. New laboratory techniques and analytical tools may contribute to improvements in the precision and accuracy of outcome indicators. However, there are inherent limitations when applying cohort-based estimates to individuals within the cohort. The accuracy and precision of prediction also are limited by measurement error and chance. Ultimately, estimating outcomes requires a careful balance between clinical experience, imperfect prediction, and uncertainty.

Pushing the bounds of second stage in term nulliparas with a predictive model

BACKGROUND

Management of the second stage of labor continues to be a clinical challenge with unclear indications for abandoning attempts at spontaneous vaginal delivery. The conflict between diminishing chances of spontaneous vaginal delivery and increasing maternal and neonatal morbidity is difficult to quantify, leading to significant variation in management between providers, and variation in rates of operative vaginal delivery and cesarean birth.

OBJECTIVE

The objective of the study was to develop an hourly prediction model for spontaneous vaginal delivery during the second stage of labor in nulliparous women with epidural anesthesia.

STUDY DESIGN

This was a secondary analysis of the Consortium for Safe Labor database. The Consortium for Safe Labor collected data from 228,652 patients at 19 hospitals in the United State from 2002 through 2008. Primary outcome was delivery type per hour of second stage: spontaneous vaginal delivery vs operative delivery (operative vaginal and cesarean delivery). Inclusion criteria were term nulliparas with singleton gestations, vertex presentation, and attainment of 10 cm cervical dilation with epidural anesthesia. Exclusion criteria were intrauterine fetal demise, planned cesarean delivery, and major congenital anomalies. An optimal decision tree was used to create a prediction model. A test set was withheld from the data set to perform validation. A risk calculator tool was developed for prediction of spontaneous vaginal birth as well as adverse perinatal outcomes per hour. Adverse maternal outcomes were a composite of postpartum hemorrhage, transfusion, endometritis and third-/fourth-degree laceration. Adverse neonatal outcomes were a composite of neonatal intensive care unit admission, hypoxic ischemic encephalopathy, respiratory distress, seizures, apnea, asphyxia, and shoulder dystocia.

RESULTS

The study population included 228,438 deliveries; 26,796 patients met inclusion and exclusion criteria. After removing cases with incomplete data, the study population consisted of 22,299 women, of which 16,593 women had a spontaneous vaginal delivery (74.4%). The number of deliveries at a given hospital per year, fetal position, cervical dilation on admission, chorioamnionitis, augmentation of labor, maternal age, and length of second stage were associated with the odds of spontaneous vaginal delivery. Using the predictors identified, a risk predictor calculator was created, taking into consideration the length of time in the second stage. A receiver-operator characteristic curve was developed to assess the calculator; area under the curve was 0.73. This calculator is available at https://www.pushprescriber.com/.

CONCLUSION

Spontaneous vaginal delivery for women with term, cephalic, singleton gestations with epidural anesthesia was associated with several variables. This calculator tool helps facilitate provider decision making and patient counseling about the value of continuing the second stage of labor based on changing rates of success and risks of maternal and neonatal morbidity with time.