Improving Clinical Translation of Machine Learning Approaches Through Clinician-Tailored Visual Displays of Black Box Algorithms: Development and Validation

Dynamic Prediction of Post-Acute Care Needs for Hospitalized Medicine Patients

OBJECTIVES

Use patient demographic and clinical characteristics at admission and time-varying in-hospital measures of patient mobility to predict patient post-acute care (PAC) discharge.

DESIGN

Retrospective cohort analysis of electronic medical records.

SETTING AND PARTICIPANTS

Patients admitted to the two participating Hospitals from November 2016 through December 2019 with ≥72 hours in a general medicine service.

METHODS

Discharge location (PAC vs home) was the primary outcome, and 2 time-varying measures of patient mobility, Activity Measure for Post-Acute Care (AM-PAC) Mobility "6-clicks" and Johns Hopkins Highest Level of Mobility, were the primary predictors. Other predictors included demographic and clinical characteristics. For each day of hospitalization, we predicted discharge to PAC using the demographic and clinical characteristics and most recent mobility data within a random forest (RF) for survival, longitudinal, and multivariate (RF-SLAM) data. A regression tree for the daily predicted probabilities of discharge to PAC was constructed to represent a global summary of the RF.

RESULTS

There were 23,090 total patients and compared to PAC, those discharged home were younger (64 vs 71), had shorter length of stay (5 vs 8 days), higher AM-PAC at admission (43 vs 32), and average AM-PAC throughout hospitalization (45 vs 35). AM-PAC was the most important predictor, followed by age, and whether the patient lives alone. The area under the hospital day-specific receiver operating characteristic curve ranged from 0.76 to 0.79 during the first 5 days. The global summary tree explained 75% of the variation in predicted probabilities for PAC from the RF. Sensitivity (75%), specificity (70%), and accuracy (72%) were maximized at a PAC probability threshold of 40%.

CONCLUSIONS AND IMPLICATIONS

Daily assessment of patient mobility should be part of routine practice to help inform care planning by hospital teams. Our prediction model could be used as a valuable tool by multidisciplinary teams in the discharge planning process.

Deep learning for Mpox: Advances, challenges, and opportunities

Although deep-learning algorithms in dermatology have shown promise in diagnosing skin cancers, less is known about potential applications for the diagnosis of infectious diseases. In a recent publication in Nature Medicine, Thieme et al. develop a deep-learning algorithm to classify skin lesions from Mpox virus (MPXV) infections.¹.

Application of Radiomics in Predicting Treatment Response to Neoadjuvant Chemoradiotherapy in Locally Advanced Rectal Cancer: Strategies and Challenges

Neoadjuvant chemoradiotherapy (nCRT) followed by total mesorectal excision is the standard treatment for locally advanced rectal cancer (LARC). A noninvasive preoperative prediction method should greatly assist in the evaluation of response to nCRT and for the development of a personalized strategy for patients with LARC. Assessment of nCRT relies on imaging and radiomics can extract valuable quantitative data from medical images. In this review, we examined the status of radiomic application for assessing response to nCRT in patients with LARC and indicated a potential direction for future research.

Ethical Implications of the Use of Language Analysis Technologies for the Diagnosis and Prediction of Psychiatric Disorders

Recent developments in artificial intelligence technologies have come to a point where machine learning algorithms can infer mental status based on someone's photos and texts posted on social media. More than that, these algorithms are able to predict, with a reasonable degree of accuracy, future mental illness. They potentially represent an important advance in mental health care for preventive and early diagnosis initiatives, and for aiding professionals in the follow-up and prognosis of their patients. However, important issues call for major caution in the use of such technologies, namely, privacy and the stigma related to mental disorders. In this paper, we discuss the bioethical implications of using such technologies to diagnose and predict future mental illness, given the current scenario of swiftly growing technologies that analyze human language and the online availability of personal information given by social media. We also suggest future directions to be taken to minimize the misuse of such important technologies.

Explainable medical imaging AI needs human-centered design: guidelines and evidence from a systematic review

Transparency in Machine Learning (ML), often also referred to as interpretability or explainability, attempts to reveal the working mechanisms of complex models. From a human-centered design perspective, transparency is not a property of the ML model but an affordance, i.e., a relationship between algorithm and users. Thus, prototyping and user evaluations are critical to attaining solutions that afford transparency. Following human-centered design principles in highly specialized and high stakes domains, such as medical image analysis, is challenging due to the limited access to end users and the knowledge imbalance between those users and ML designers. To investigate the state of transparent ML in medical image analysis, we conducted a systematic review of the literature from 2012 to 2021 in PubMed, EMBASE, and Compendex databases. We identified 2508 records and 68 articles met the inclusion criteria. Current techniques in transparent ML are dominated by computational feasibility and barely consider end users, e.g. clinical stakeholders. Despite the different roles and knowledge of ML developers and end users, no study reported formative user research to inform the design and development of transparent ML models. Only a few studies validated transparency claims through empirical user evaluations. These shortcomings put contemporary research on transparent ML at risk of being incomprehensible to users, and thus, clinically irrelevant. To alleviate these shortcomings in forthcoming research, we introduce the INTRPRT guideline, a design directive for transparent ML systems in medical image analysis. The INTRPRT guideline suggests human-centered design principles, recommending formative user research as the first step to understand user needs and domain requirements. Following these guidelines increases the likelihood that the algorithms afford transparency and enable stakeholders to capitalize on the benefits of transparent ML.

Machine learning and LACE index for predicting 30-day readmissions after heart failure hospitalization in elderly patients

Machine learning (ML) techniques may improve readmission prediction performance in heart failure (HF) patients. This study aimed to assess the ability of ML algorithms to predict unplanned all-cause 30-day readmissions in HF elderly patients, and to compare them with conventional LACE (Length of hospitalization, Acuity, Comorbidities, Emergency department visits) index. All patients aged ≥ 65 years discharged alive between 2010 and 2019 after a hospitalization for acute HF were included in this retrospective cohort study. We applied MICE (Multivariate Imputation via Chained Equations) method to obtain a balanced, fully valued dataset and LASSO (Least Absolute Shrinkage and Selection Operator) algorithm to get the most significant features. Training (80% of records) and test (20%) cohorts were randomly selected. Study population: 3079 patients, 394 (12.8%) presented at least one readmission within 30 days, and 2685 (87.2%) did not. In the test cohort AUCs (IC95%) of XGBoost, Ada Boost Classifier, Random forest, and Gradient Boosting, and LACE Index were: 0.803 (0.734-0.872), 0.782 (0.711-0.854), 0.776 (0.703-0.848), 0.786 (0.715-0.857), and 0.504 (0.414-0.594), respectively, for predicting readmissions. A SHAP analysis was performed to offer a breakdown of the ML variables associated with readmission. Positive and negative predicting values estimates of the different ML models and LACE index were also provided, for several values of readmission rate prevalence. Among elderly patients, the rate of all-cause unplanned 30-day readmissions after hospitalization due to an acute HF was high. ML models performed better than the conventional LACE index for predicting readmissions. ML models can be proposed as promising tools for the identification of subjects at high risk of hospitalization in this clinical setting, enabling care teams to target interventions for improving overall clinical outcomes.

Integration of artificial intelligence and plasma steroidomics with laboratory information management systems: application to primary aldosteronism

Objectives

Mass spectrometry-based steroidomics combined with machine learning (ML) provides a potentially powerful approach in endocrine diagnostics, but is hampered by limitations in the conveyance of results and interpretations to clinicians. We address this shortcoming by integration of the two technologies with a laboratory information management systems (LIMS) model.

Methods

The approach involves integration of ML algorithm-derived models with commercially available mathematical programming software and a web-based LIMS prototype. To illustrate clinical utility, the process was applied to plasma steroidomics data from 22 patients tested for primary aldosteronism (PA).

Results

Once mass spectrometry data are uploaded into the system, automated processes enable generation of interpretations of steroid profiles from ML models. Generated reports include plasma concentrations of steroids in relation to age- and sex-specific reference intervals along with results of ML models and narrative interpretations that cover probabilities of PA. If PA is predicted, reports include probabilities of unilateral disease and mutations of KCNJ5 known to be associated with successful outcomes of adrenalectomy. Preliminary results, with no overlap in probabilities of disease among four patients with and 18 without PA and correct classification of all four patients with unilateral PA including three of four with KCNJ5 mutations, illustrate potential utility of the approach to guide diagnosis and subtyping of patients with PA.

Conclusions

The outlined process for integrating plasma steroidomics data and ML with LIMS may facilitate improved diagnostic-decision-making when based on higher-dimensional data otherwise difficult to interpret. The approach is relevant to other diagnostic applications involving ML.

Implementation Frameworks for Artificial Intelligence Translation Into Health Care Practice: Scoping Review

Background

Significant efforts have been made to develop artificial intelligence (AI) solutions for health care improvement. Despite the enthusiasm, health care professionals still struggle to implement AI in their daily practice.

Objective

This paper aims to identify the implementation frameworks used to understand the application of AI in health care practice.

Methods

A scoping review was conducted using the Cochrane, Evidence Based Medicine Reviews, Embase, MEDLINE, and PsycINFO databases to identify publications that reported frameworks, models, and theories concerning AI implementation in health care. This review focused on studies published in English and investigating AI implementation in health care since 2000. A total of 2541 unique publications were retrieved from the databases and screened on titles and abstracts by 2 independent reviewers. Selected articles were thematically analyzed against the Nilsen taxonomy of implementation frameworks, and the Greenhalgh framework for the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) of health care technologies.

Results

In total, 7 articles met all eligibility criteria for inclusion in the review, and 2 articles included formal frameworks that directly addressed AI implementation, whereas the other articles provided limited descriptions of elements influencing implementation. Collectively, the 7 articles identified elements that aligned with all the NASSS domains, but no single article comprehensively considered the factors known to influence technology implementation. New domains were identified, including dependency on data input and existing processes, shared decision-making, the role of human oversight, and ethics of population impact and inequality, suggesting that existing frameworks do not fully consider the unique needs of AI implementation.

Conclusions

This literature review demonstrates that understanding how to implement AI in health care practice is still in its early stages of development. Our findings suggest that further research is needed to provide the knowledge necessary to develop implementation frameworks to guide the future implementation of AI in clinical practice and highlight the opportunity to draw on existing knowledge from the field of implementation science.

A Knowledge Distillation Ensemble Framework for Predicting Short- and Long-Term Hospitalization Outcomes From Electronic Health Records Data

The ability to perform accurate prognosis is crucial for proactive clinical decision making, informed resource management and personalised care. Existing outcome prediction models suffer from a low recall of infrequent positive outcomes. We present a highly-scalable and robust machine learning framework to automatically predict adversity represented by mortality and ICU admission and readmission from time-series of vital signs and laboratory results obtained within the first 24 hours of hospital admission. The stacked ensemble platform comprises two components: a) an unsupervised LSTM Autoencoder that learns an optimal representation of the time-series, using it to differentiate the less frequent patterns which conclude with an adverse event from the majority patterns that do not, and b) a gradient boosting model, which relies on the constructed representation to refine prediction by incorporating static features. The model is used to assess a patient's risk of adversity and provides visual justifications of its prediction. Results of three case studies show that the model outperforms existing platforms in ICU and general ward settings, achieving average Precision-Recall Areas Under the Curve (PR-AUCs) of 0.891 (95% CI: 0.878-0.939) for mortality and 0.908 (95% CI: 0.870-0.935) in predicting ICU admission and readmission.

Development of Severe COVID-19 Adaptive Risk Predictor (SCARP), a Calculator to Predict Severe Disease or Death in Hospitalized Patients With COVID-19

BACKGROUND

Predicting the clinical trajectory of individual patients hospitalized with coronavirus disease 2019 (COVID-19) is challenging but necessary to inform clinical care. The majority of COVID-19 prognostic tools use only data present upon admission and do not incorporate changes occurring after admission.

OBJECTIVE

To develop the Severe COVID-19 Adaptive Risk Predictor (SCARP) (https://rsconnect.biostat.jhsph.edu/covid_trajectory/), a novel tool that can provide dynamic risk predictions for progression from moderate disease to severe illness or death in patients with COVID-19 at any time within the first 14 days of their hospitalization.

DESIGN

Retrospective observational cohort study.

SETTINGS

Five hospitals in Maryland and Washington, D.C.

PATIENTS

Patients who were hospitalized between 5 March and 4 December 2020 with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) confirmed by nucleic acid test and symptomatic disease.

MEASUREMENTS

A clinical registry for patients hospitalized with COVID-19 was the primary data source; data included demographic characteristics, admission source, comorbid conditions, time-varying vital signs, laboratory measurements, and clinical severity. Random forest for survival, longitudinal, and multivariate (RF-SLAM) data analysis was applied to predict the 1-day and 7-day risks for progression to severe disease or death for any given day during the first 14 days of hospitalization.

RESULTS

Among 3163 patients admitted with moderate COVID-19, 228 (7%) became severely ill or died in the next 24 hours; an additional 355 (11%) became severely ill or died in the next 7 days. The area under the receiver-operating characteristic curve (AUC) for 1-day risk predictions for progression to severe disease or death was 0.89 (95% CI, 0.88 to 0.90) and 0.89 (CI, 0.87 to 0.91) during the first and second weeks of hospitalization, respectively. The AUC for 7-day risk predictions for progression to severe disease or death was 0.83 (CI, 0.83 to 0.84) and 0.87 (CI, 0.86 to 0.89) during the first and second weeks of hospitalization, respectively.

LIMITATION

The SCARP tool was developed by using data from a single health system.

CONCLUSION

Using the predictive power of RF-SLAM and longitudinal data from more than 3000 patients hospitalized with COVID-19, an interactive tool was developed that rapidly and accurately provides the probability of an individual patient's progression to severe illness or death on the basis of readily available clinical information.

PRIMARY FUNDING SOURCE

Hopkins inHealth and COVID-19 Administrative Supplement for the HHS Region 3 Treatment Center from the Office of the Assistant Secretary for Preparedness and Response.

Decision Tree With Only Two Musculoskeletal Sites to Diagnose Polymyalgia Rheumatica Using [18F]FDG PET-CT

Introduction: The aim of this study was to find the best ordered combination of two FDG positive musculoskeletal sites with a machine learning algorithm to diagnose polymyalgia rheumatica (PMR) vs. other rheumatisms in a cohort of patients with inflammatory rheumatisms.

Methods: This retrospective study included 140 patients who underwent [¹⁸F]FDG PET-CT and whose final diagnosis was inflammatory rheumatism. The cohort was randomized, stratified on the final diagnosis into a training and a validation cohort. FDG uptake of 17 musculoskeletal sites was evaluated visually and set positive if uptake was at least equal to that of the liver. A decision tree classifier was trained and validated to find the best combination of two positives sites to diagnose PMR. Diagnosis performances were measured first, for each musculoskeletal site, secondly for combination of two positive sites and thirdly using the decision tree created with machine learning.

Results: 55 patients with PMR and 85 patients with other inflammatory rheumatisms were included. Musculoskeletal sites, used either individually or in combination of two, were highly imbalanced to diagnose PMR with a high specificity and a low sensitivity. The machine learning algorithm identified an optimal ordered combination of two sites to diagnose PMR. This required a positive interspinous bursa or, if negative, a positive trochanteric bursa. Following the decision tree, sensitivity and specificity to diagnose PMR were respectively 73.2 and 87.5% in the training cohort and 78.6 and 80.1% in the validation cohort.

Conclusion: Ordered combination of two visually positive sites leads to PMR diagnosis with an accurate sensitivity and specificity vs. other rheumatisms in a large cohort of patients with inflammatory rheumatisms.

Baseline and Dynamic Risk Predictors of Appropriate Implantable Cardioverter Defibrillator Therapy

Background Current approaches fail to separate patients at high versus low risk for ventricular arrhythmias owing to overreliance on a snapshot left ventricular ejection fraction measure. We used statistical machine learning to identify important cardiac imaging and time-varying risk predictors. Methods and Results Three hundred eighty-two cardiomyopathy patients (left ventricular ejection fraction ≤35%) underwent cardiac magnetic resonance before primary prevention implantable cardioverter defibrillator insertion. The primary end point was appropriate implantable cardioverter defibrillator discharge or sudden death. Patient characteristics; serum biomarkers of inflammation, neurohormonal status, and injury; and cardiac magnetic resonance-measured left ventricle and left atrial indices and myocardial scar burden were assessed at baseline. Time-varying covariates comprised interval heart failure hospitalizations and left ventricular ejection fractions. A random forest statistical method for survival, longitudinal, and multivariable outcomes incorporating baseline and time-varying variables was compared with (1) Seattle Heart Failure model scores and (2) random forest survival and Cox regression models incorporating baseline characteristics with and without imaging variables. Age averaged 57±13 years with 28% women, 66% white, 51% ischemic, and follow-up time of 5.9±2.3 years. The primary end point (n=75) occurred at 3.3±2.4 years. Random forest statistical method for survival, longitudinal, and multivariable outcomes with baseline and time-varying predictors had the highest area under the receiver operating curve, median 0.88 (95% CI, 0.75-0.96). Top predictors comprised heart failure hospitalization, left ventricle scar, left ventricle and left atrial volumes, left atrial function, and interleukin-6 level; heart failure accounted for 67% of the variation explained by the prediction, imaging 27%, and interleukin-6 2%. Serial left ventricular ejection fraction was not a significant predictor. Conclusions Hospitalization for heart failure and baseline cardiac metrics substantially improve ventricular arrhythmic risk prediction.