The number needed to benefit: estimating the value of predictive analytics in healthcare

Achieving large-scale clinician adoption of AI-enabled decision support

Computerised decision support (CDS) tools enabled by artificial intelligence (AI) seek to enhance accuracy and efficiency of clinician decision-making at the point of care. Statistical models developed using machine learning (ML) underpin most current tools. However, despite thousands of models and hundreds of regulator-approved tools internationally, large-scale uptake into routine clinical practice has proved elusive. While underdeveloped system readiness and investment in AI/ML within Australia and perhaps other countries are impediments, clinician ambivalence towards adopting these tools at scale could be a major inhibitor. We propose a set of principles and several strategic enablers for obtaining broad clinician acceptance of AI/ML-enabled CDS tools.

Predictive analytics and early intervention in healthcare social work: a scoping review

This scoping review investigates the untapped potential of predictive analytics in healthcare social work, specifically targeting early intervention frameworks. Despite the escalating attention predictive analytics has garnered across multiple disciplines, its tailored application in social work remains notably sparse. This study endeavors to fill this lacuna by meticulously reviewing the extant literature and delineating the prospective advantages and inherent constraints of integrating predictive analytics into healthcare social work. The outcomes of this inquiry enrich the prevailing dialogue on the utility of predictive analytics in healthcare, offering indispensable perspectives for practitioners and policymakers in the social work domain.

Synergizing photon-counting CT with deep learning: potential enhancements in medical imaging

This review article highlights the potential of integrating photon-counting computed tomography (CT) and deep learning algorithms in medical imaging to enhance diagnostic accuracy, improve image quality, and reduce radiation exposure. The use of photon-counting CT provides superior image quality, reduced radiation dose, and material decomposition capabilities, while deep learning algorithms excel in automating image analysis and improving diagnostic accuracy. The integration of these technologies can lead to enhanced material decomposition and classification, spectral image analysis, predictive modeling for individualized medicine, workflow optimization, and radiation dose management. However, data requirements, computational resources, and regulatory and ethical concerns remain challenges that need to be addressed to fully realize the potential of this technology. The fusion of photon-counting CT and deep learning algorithms is poised to revolutionize medical imaging and transform patient care.

Automation in health care: the need for an ergonomics-based approach

Healthcare quality and efficiency challenges degrade outcomes and burden multiple stakeholders. Workforce shortage, burnout, and complexity of workflows necessitate effective support for patients and providers. There is interest in employing automation, or the use of 'computer[s] [to] carry out… functions that the human operator would normally perform', in health care to improve delivery of services. However, unique aspects of health care require analysis of workflows across several domains and an understanding of the ways work system factors interact to shape those workflows. Ergonomics has identified key work system issues relevant to effective automation in other industries. Understanding these issues in health care can direct opportunities for the effective use of automation in health care. This article illustrates work system considerations using two example workflows; discusses how those considerations may inform solution design, implementation, and use; and provides future directions to advance the essential role of ergonomics in healthcare automation.

Fog-Based Smart Cardiovascular Disease Prediction System Powered by Modified Gated Recurrent Unit

The ongoing fast-paced technology trend has brought forth ceaseless transformation. In this regard, cloud computing has long proven to be the paramount deliverer of services such as computing power, software, networking, storage, and databases on a pay-per-use basis. The cloud is a big proponent of the internet of things (IoT), furnishing the computation and storage requisite to address internet-of-things applications. With the proliferating IoT devices triggering a continual data upsurge, the cloud-IoT interaction encounters latency, bandwidth, and connectivity restraints. The inclusion of the decentralized and distributed fog computing layer amidst the cloud and IoT layer extends the cloud's processing, storage, and networking services close to end users. This hierarchical edge-fog-cloud model distributes computation and intelligence, yielding optimal solutions while tackling constraints like massive data volume, latency, delay, and security vulnerability. The healthcare domain, warranting time-critical functionalities, can reap benefits from the cloud-fog-IoT interplay. This research paper propounded a fog-assisted smart healthcare system to diagnose heart or cardiovascular disease. It combined a fuzzy inference system (FIS) with the recurrent neural network model's variant of the gated recurrent unit (GRU) for pre-processing and predictive analytics tasks. The proposed system showcases substantially improved performance results, with classification accuracy at 99.125%. With major processing of healthcare data analytics happening at the fog layer, it is observed that the proposed work reveals optimized results concerning delays in terms of latency, response time, and jitter, compared to the cloud. Deep learning models are adept at handling sophisticated tasks, particularly predictive analytics. Time-critical healthcare applications reap benefits from deep learning's exclusive potential to furnish near-perfect results, coupled with the merits of the decentralized fog model, as revealed by the experimental results.

Predictive Modeling to Identify Children With Complex Health Needs At Risk for Hospitalization

BACKGROUND

Identifying children at high risk with complex health needs (CCHN) who have intersecting medical and social needs is challenging. This study's objectives were to (1) develop and evaluate an electronic health record (EHR)-based clinical predictive model ("model") for identifying high-risk CCHN and (2) compare the model's performance as a clinical decision support (CDS) to other CDS tools available for identifying high-risk CCHN.

METHODS

This retrospective cohort study included children aged 0 to 20 years with established care within a single health system. The model development/validation cohort included 33 months (January 1, 2016-September 30, 2018) and the testing cohort included 18 months (October 1, 2018-March 31, 2020) of EHR data. Machine learning methods generated a model that predicted probability (0%-100%) for hospitalization within 6 months. Model performance measures included sensitivity, positive predictive value, area under receiver-operator curve, and area under precision-recall curve. Three CDS rules for identifying high-risk CCHN were compared: (1) hospitalization probability ≥10% (model-predicted); (2) complex chronic disease classification (using Pediatric Medical Complexity Algorithm [PMCA]); and (3) previous high hospital utilization.

RESULTS

Model development and testing cohorts included 116 799 and 27 087 patients, respectively. The model demonstrated area under receiver-operator curve = 0.79 and area under precision-recall curve = 0.13. PMCA had the highest sensitivity (52.4%) and classified the most children as high risk (17.3%). Positive predictive value of the model-based CDS rule (19%) was higher than CDS based on the PMCA (1.9%) and previous hospital utilization (15%).

CONCLUSIONS

A novel EHR-based predictive model was developed and validated as a population-level CDS tool for identifying CCHN at high risk for future hospitalization.

Artificial Intelligence Enabled Classification of Hypertrophic Heart Diseases Using Electrocardiograms

Background

Differentiating among cardiac diseases associated with left ventricular hypertrophy (LVH) informs diagnosis and clinical care.

Objective

To evaluate if artificial intelligence-enabled analysis of the 12-lead electrocardiogram (ECG) facilitates automated detection and classification of LVH.

Methods

We used a pretrained convolutional neural network to derive numerical representations of 12-lead ECG waveforms from patients in a multi-institutional healthcare system who had cardiac diseases associated with LVH (n = 50,709), including cardiac amyloidosis (n = 304), hypertrophic cardiomyopathy (n = 1056), hypertension (n = 20,802), aortic stenosis (n = 446), and other causes (n = 4766). We then regressed LVH etiologies relative to no LVH on age, sex, and the numerical 12-lead representations using logistic regression ("LVH-Net"). To assess deep learning model performance on single-lead data analogous to mobile ECGs, we also developed 2 single-lead deep learning models by training models on lead I ("LVH-Net Lead I") or lead II ("LVH-Net Lead II") from the 12-lead ECG. We compared the performance of the LVH-Net models to alternative models fit on (1) age, sex, and standard ECG measures, and (2) clinical ECG-based rules for diagnosing LVH.

Results

The areas under the receiver operator characteristic curve of LVH-Net by specific LVH etiology were cardiac amyloidosis 0.95 [95% CI, 0.93-0.97], hypertrophic cardiomyopathy 0.92 [95% CI, 0.90-0.94], aortic stenosis LVH 0.90 [95% CI, 0.88-0.92], hypertensive LVH 0.76 [95% CI, 0.76-0.77], and other LVH 0.69 [95% CI 0.68-0.71]. The single-lead models also discriminated LVH etiologies well.

Conclusion

An artificial intelligence-enabled ECG model is favorable for detection and classification of LVH and outperforms clinical ECG-based rules.

Assessing Performance and Clinical Usefulness in Prediction Models With Survival Outcomes: Practical Guidance for Cox Proportional Hazards Models

Risk prediction models need thorough validation to assess their performance. Validation of models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. This article describes measures to evaluate predictions and the potential improvement in decision making from survival models based on Cox proportional hazards regression.

As a motivating case study, the authors consider the prediction of the composite outcome of recurrence or death (the "event") in patients with breast cancer after surgery. They developed a simple Cox regression model with 3 predictors, as in the Nottingham Prognostic Index, in 2982 women (1275 events over 5 years of follow-up) and externally validated this model in 686 women (285 events over 5 years). Improvement in performance was assessed after the addition of progesterone receptor as a prognostic biomarker.

The model predictions can be evaluated across the full range of observed follow-up times or for the event occurring by the end of a fixed time horizon of interest. The authors first discuss recommended statistical measures that evaluate model performance in terms of discrimination, calibration, or overall performance. Further, they evaluate the potential clinical utility of the model to support clinical decision making according to a net benefit measure. They provide SAS and R code to illustrate internal and external validation.

The authors recommend the proposed set of performance measures for transparent reporting of the validity of predictions from survival models.

Conformal Prediction in Clinical Medical Sciences

The use of machine learning (ML) and artificial intelligence (AI) applications in medicine has attracted a great deal of attention in the medical literature, but little is known about how to use Conformal Predictions (CP) to assess the accuracy of individual predictions in clinical applications. We performed a comprehensive search in SCOPUS® to find papers reporting the use of CP in clinical applications. We identified 14 papers reporting the use of CP for clinical applications, and we briefly describe the methods and results reported in these papers. The literature reviewed shows that CP methods can be used in clinical applications to provide important insight into the accuracy of individual predictions. Unfortunately, the review also shows that most of the studies have been performed in isolation, without input from practicing clinicians, not providing comparisons among different approaches and not considering important socio-technical considerations leading to clinical adoption.

A bias evaluation checklist for predictive models and its pilot application for 30-day hospital readmission models

OBJECTIVE

Health care providers increasingly rely upon predictive algorithms when making important treatment decisions, however, evidence indicates that these tools can lead to inequitable outcomes across racial and socio-economic groups. In this study, we introduce a bias evaluation checklist that allows model developers and health care providers a means to systematically appraise a model's potential to introduce bias.

MATERIALS AND METHODS

Our methods include developing a bias evaluation checklist, a scoping literature review to identify 30-day hospital readmission prediction models, and assessing the selected models using the checklist.

RESULTS

We selected 4 models for evaluation: LACE, HOSPITAL, Johns Hopkins ACG, and HATRIX. Our assessment identified critical ways in which these algorithms can perpetuate health care inequalities. We found that LACE and HOSPITAL have the greatest potential for introducing bias, Johns Hopkins ACG has the most areas of uncertainty, and HATRIX has the fewest causes for concern.

DISCUSSION

Our approach gives model developers and health care providers a practical and systematic method for evaluating bias in predictive models. Traditional bias identification methods do not elucidate sources of bias and are thus insufficient for mitigation efforts. With our checklist, bias can be addressed and eliminated before a model is fully developed or deployed.

CONCLUSION

The potential for algorithms to perpetuate biased outcomes is not isolated to readmission prediction models; rather, we believe our results have implications for predictive models across health care. We offer a systematic method for evaluating potential bias with sufficient flexibility to be utilized across models and applications.

"We'd be really motivated to do something about it": a qualitative study of parent and clinician attitudes towards predicting childhood obesity in practice

ISSUE ADDRESSED

In Australia, one-in-four (24.9%) children live with overweight or obesity (OW/OB). Identifying infants at-risk of developing childhood OW/OB is a potential preventive pathway but its acceptability is yet to be investigated in Australia. This study aimed to (1) investigate acceptability of predicting childhood OW/OB with parents of infants (aged 0-2 years) and clinicians and (2) explore key language to address stigma and maximise the acceptability of predicting childhood OW/OB in practice.

METHODS

Cross-sectional and qualitative design, comprising individual semi-structured interviews. Participants were multidisciplinary paediatric clinicians (n=18) and parents (n=13) recruited across public hospitals and health services in Queensland, Australia. Data were analysed under the Framework Method using an inductive, thematic approach.

RESULTS

Five main themes were identified: (1) Optimism for prevention and childhood obesity prediction (2) Parent dedication to child's health (3) Adverse parent response to risk for childhood obesity (4) Language and phrasing for discussing weight and risk (5) Clinical delivery. Most participants were supportive of using a childhood OW/OB prediction tool in practice. Parents expressed dedication to their child's health that superseded potential feelings of judgment or blame. When discussing weight in a clinical setting, the use of sensitive (i.e. 'overweight', 'above average', 'growth' versus 'obesity') and positive, health-focused language was mostly supported.

CONCLUSIONS

Multidisciplinary paediatric clinicians and parents generally accept the concept of predicting childhood OW/OB in practice in Queensland, Australia. SO WHAT?: Clinicians, public health and health promotion professionals and policymakers can act now to implement sensitive communication strategies concerning weight and obesity risk.

Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine

Temporal dataset shift associated with changes in healthcare over time is a barrier to deploying machine learning-based clinical decision support systems. Algorithms that learn robust models by estimating invariant properties across time periods for domain generalization (DG) and unsupervised domain adaptation (UDA) might be suitable to proactively mitigate dataset shift. The objective was to characterize the impact of temporal dataset shift on clinical prediction models and benchmark DG and UDA algorithms on improving model robustness. In this cohort study, intensive care unit patients from the MIMIC-IV database were categorized by year groups (2008-2010, 2011-2013, 2014-2016 and 2017-2019). Tasks were predicting mortality, long length of stay, sepsis and invasive ventilation. Feedforward neural networks were used as prediction models. The baseline experiment trained models using empirical risk minimization (ERM) on 2008-2010 (ERM[08-10]) and evaluated them on subsequent year groups. DG experiment trained models using algorithms that estimated invariant properties using 2008-2016 and evaluated them on 2017-2019. UDA experiment leveraged unlabelled samples from 2017 to 2019 for unsupervised distribution matching. DG and UDA models were compared to ERM[08-16] models trained using 2008-2016. Main performance measures were area-under-the-receiver-operating-characteristic curve (AUROC), area-under-the-precision-recall curve and absolute calibration error. Threshold-based metrics including false-positives and false-negatives were used to assess the clinical impact of temporal dataset shift and its mitigation strategies. In the baseline experiments, dataset shift was most evident for sepsis prediction (maximum AUROC drop, 0.090; 95% confidence interval (CI), 0.080-0.101). Considering a scenario of 100 consecutively admitted patients showed that ERM[08-10] applied to 2017-2019 was associated with one additional false-negative among 11 patients with sepsis, when compared to the model applied to 2008-2010. When compared with ERM[08-16], DG and UDA experiments failed to produce more robust models (range of AUROC difference, - 0.003 to 0.050). In conclusion, DG and UDA failed to produce more robust models compared to ERM in the setting of temporal dataset shift. Alternate approaches are required to preserve model performance over time in clinical medicine.

Predictive Analytics in Clinical Practice: Advantages and Disadvantages

Predictive analytics are increasingly reported by clinicians. These tools aim to improve patient outcomes in terms of quality, safety, and efficiency. However, deploying predictive analytics in clinical practice remains challenging today. We highlight several advantages and disadvantages of the application of predictive analytics in clinical practice. To flourish and reach its potential, predictive analytics need data that is of adequate quantity and quality, ideally tailored to clinical scenarios in equipoise regarding optimal management. Adequate reporting of predictive analytic tools is incumbent for uptake into clinical workflows. At least for now, the clinicians' knowledge, experience, and vigilance remain imperative for applying predictive analytics in clinical practice.

How can we discover the most valuable types of big data and artificial intelligence-based solutions? A methodology for the efficient development of the underlying analytics that improve care

BACKGROUND

Much has been invested in big data and artificial intelligence-based solutions for healthcare. However, few applications have been implemented in clinical practice. Early economic evaluations can help to improve decision-making by developers of analytics underlying these solutions aiming to increase the likelihood of successful implementation, but recommendations about their use are lacking. The aim of this study was to develop and apply a framework that positions best practice methods for economic evaluations alongside development of analytics, thereby enabling developers to identify barriers to success and to select analytics worth further investments.

METHODS

The framework was developed using literature, recommendations for economic evaluations and by applying the framework to use cases (chronic lymphocytic leukaemia (CLL), intensive care, diabetes). First, the feasibility of developing clinically relevant analytics was assessed and critical barriers to successful development and implementation identified. Economic evaluations were then used to determine critical thresholds and guide investment decisions.

RESULTS

When using the framework to assist decision-making of developers of analytics, continuing development was not always feasible or worthwhile. Developing analytics for progressive CLL and diabetes was clinically relevant but not feasible with the data available. Alternatively, developing analytics for newly diagnosed CLL patients was feasible but continuing development was not considered worthwhile because the high drug costs made it economically unattractive for potential users. Alternatively, in the intensive care unit, analytics reduced mortality and per-patient costs when used to identify infections (- 0.5%, - €886) and to improve patient-ventilator interaction (- 3%, - €264). Both analytics have the potential to save money but the potential benefits of analytics that identify infections strongly depend on infection rate; a higher rate implies greater cost-savings.

CONCLUSIONS

We present a framework that stimulates efficiency of development of analytics for big data and artificial intelligence-based solutions by selecting those applications of analytics for which development is feasible and worthwhile. For these applications, results from early economic evaluations can be used to guide investment decisions and identify critical requirements.

Development and Validation of an Obstetric Comorbidity Risk Score for Clinical Use

Background: A comorbidity summary score may support early and systematic identification of women at high risk for adverse obstetric outcomes. The objective of this study was to conduct the initial development and validation of an obstetrics comorbidity risk score for automated implementation in the electronic health record (EHR) for clinical use. Methods: The score was developed and validated using EHR data for a retrospective cohort of pregnancies with delivery between 2010 and 2018 at Kaiser Permanente Northern California, an integrated health care system. The outcome used for model development consisted of adverse obstetric events from delivery hospitalization (e.g., eclampsia, hemorrhage, death). Candidate predictors included maternal age, parity, multiple gestation, and any maternal diagnoses assigned in health care encounters in the 12 months before admission for delivery. We used penalized regression for variable selection, logistic regression to fit the model, and internal validation for model evaluation. We also evaluated prenatal model performance at 18 weeks of pregnancy. Results: The development cohort (n = 227,405 pregnancies) had an outcome rate of 3.8% and the validation cohort (n = 41,683) had an outcome rate of 2.9%. Of 276 candidate predictors, 37 were included in the final model. The final model had a validation c-statistic of 0.72 (95% confidence interval [CI] 0.70-0.73). When evaluated at 18 weeks of pregnancy, discrimination was modestly diminished (c-statistic 0.68 [95% CI 0.67-0.70]). Conclusions: The obstetric comorbidity score demonstrated good discrimination for adverse obstetric outcomes. After additional appropriate validation, the score can be automated in the EHR to support early identification of high-risk women and assist efforts to ensure risk-appropriate maternal care.

Evaluation of Incident 7-Day Infection and Sepsis Hospitalizations in an Integrated Health System

RATIONALE

Pre-hospital opportunities to predict infection and sepsis hospitalization may exist, but little is known about their incidence following common healthcare encounters.

OBJECTIVES

To evaluate the incidence and timing of infection and sepsis hospitalization within 7 days of living hospital discharge, emergency department discharge, and ambulatory visit settings.

METHODS

In each setting, we identified patients in clinical strata based on the presence of infection and severity of illness. We estimated number needed to evaluate values with hypothetical predictive model operating characteristics.

RESULTS

We identified 97,614,228 encounters including 1,117,702 (1.1 %) hospital discharges, 4,635,517 (4.7%) emergency department discharges, and 91,861,009 (94.1 %) ambulatory visits between 2012 and 2017. The incidence of 7-day infection hospitalization varied from 37,140 (3.3%) following inpatient discharge, 50,315 (1.1%) following emergency department discharge, and 277,034 (0.3%) following ambulatory visits. The incidence of 7-day infection hospitalization was increased for inpatient discharges with high readmission risk (10.0%), emergency department discharges with increased acute or chronic severity of illness (3.5% and 4.7%, respectively), and ambulatory visits with acute infection (0.7%). The timing of 7-day infection and sepsis hospitalizations differed across settings with an early rise following ambulatory visits, a later peak following emergency department discharges, and a delayed peak following inpatient discharge. Theoretical number needed to evaluate values varied by strata, but following hospital and emergency department discharge, were as low as 15 to 25.

CONCLUSIONS

Incident 7-day infection and sepsis hospitalizations following encounters in routine healthcare settings were surprisingly common and may be amenable to clinical predictive models.

Expected clinical utility of automatable prediction models for improving palliative and end-of-life care outcomes: Toward routine decision analysis before implementation

Objective

The study sought to evaluate the expected clinical utility of automatable prediction models for increasing goals-of-care discussions (GOCDs) among hospitalized patients at the end of life (EOL).

Materials and Methods

We built a decision model from the perspective of clinicians who aim to increase GOCDs at the EOL using an automated alert system. The alternative strategies were 4 prediction models—3 random forest models and the Modified Hospital One-year Mortality Risk model—to generate alerts for patients at a high risk of 1-year mortality. They were trained on admissions from 2011 to 2016 (70 788 patients) and tested with admissions from 2017-2018 (16 490 patients). GOCDs occurring in usual care were measured with code status orders. We calculated the expected risk difference (beneficial outcomes with alerts minus beneficial outcomes without alerts among those at the EOL), the number needed to benefit (number of alerts needed to increase benefit over usual care by 1 outcome), and the net benefit (benefit minus cost) of each strategy.

Results

Models had a C-statistic between 0.79 and 0.86. A code status order occurred during 2599 of 3773 (69%) hospitalizations at the EOL. At a risk threshold corresponding to an alert prevalence of 10%, the expected risk difference ranged from 5.4% to 10.7% and the number needed to benefit ranged from 5.4 to 10.9 alerts. Using revealed preferences, only 2 models improved net benefit over usual care. A random forest model with diagnostic predictors had the highest expected value, including in sensitivity analyses.

Discussion

Prediction models with acceptable predictive validity differed meaningfully in their ability to improve over usual decision making.

Conclusions

An evaluation of clinical utility, such as by using decision curve analysis, is recommended after validating a prediction model because metrics of model predictiveness, such as the C-statistic, are not informative of clinical value.

Algorithmic prognostication in critical care: a promising but unproven technology for supporting difficult decisions

PURPOSE OF REVIEW

Patients, surrogate decision makers, and clinicians face weighty and urgent decisions under uncertainty in the ICU, which could be aided by risk prediction. Although emerging artificial intelligence/machine learning (AI/ML) algorithms could reduce uncertainty surrounding these life and death decisions, certain criteria must be met to ensure their bedside value.

RECENT FINDINGS

Although ICU severity of illness scores have existed for decades, these tools have not been shown to predict well or to improve outcomes for individual patients. Novel AI/ML tools offer the promise of personalized ICU care but remain untested in clinical trials. Ensuring that these predictive models account for heterogeneity in patient characteristics and treatments, are not only specific to a clinical action but also consider the longitudinal course of critical illness, and address patient-centered outcomes related to equity, transparency, and shared decision-making will increase the likelihood that these tools improve outcomes. Improved clarity around standards and contributions from institutions and critical care departments will be essential.

SUMMARY

Improved ICU prognostication, enabled by advanced ML/AI methods, offer a promising approach to inform difficult and urgent decisions under uncertainty. However, critical knowledge gaps around performance, equity, safety, and effectiveness must be filled and prospective, randomized testing of predictive interventions are still needed.

Rise of the Machines: Artificial Intelligence and the Clinical Laboratory

BACKGROUND

Artificial intelligence (AI) is rapidly being developed and implemented to augment and automate decision-making across healthcare systems. Being an essential part of these systems, laboratories will see significant growth in AI applications for the foreseeable future.

CONTENT

In laboratory medicine, AI can be used for operational decision-making and automating or augmenting human-based workflows. Specific applications include instrument automation, error detection, forecasting, result interpretation, test utilization, genomics, and image analysis. If not doing so today, clinical laboratories will be using AI routinely in the future, therefore, laboratory experts should understand their potential role in this new area and the opportunities for AI technologies. The roles of laboratorians range from passive provision of data to fuel algorithms to developing entirely new algorithms, with subject matter expertise as a perfect fit in the middle. The technical development of algorithms is only a part of the overall picture, where the type, availability, and quality of data are at least as important. Implementation of AI algorithms also offers technical and usability challenges that need to be understood to be successful. Finally, as AI algorithms continue to become available, it is important to understand how to evaluate their validity and utility in the real world.

SUMMARY

This review provides an overview of what AI is, examples of how it is currently being used in laboratory medicine, different ways for laboratorians to get involved in algorithm development, and key considerations for AI algorithm implementation and critical evaluation.

A Simulated Prospective Evaluation of a Deep Learning Model for Real-Time Prediction of Clinical Deterioration Among Ward Patients

OBJECTIVES

The National Early Warning Score, Modified Early Warning Score, and quick Sepsis-related Organ Failure Assessment can predict clinical deterioration. These scores exhibit only moderate performance and are often evaluated using aggregated measures over time. A simulated prospective validation strategy that assesses multiple predictions per patient-day would provide the best pragmatic evaluation. We developed a deep recurrent neural network deterioration model and conducted a simulated prospective evaluation.

DESIGN

Retrospective cohort study.

SETTING

Four hospitals in Pennsylvania.

PATIENTS

Inpatient adults discharged between July 1, 2017, and June 30, 2019.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

We trained a deep recurrent neural network and logistic regression model using data from electronic health records to predict hourly the 24-hour composite outcome of transfer to ICU or death. We analyzed 146,446 hospitalizations with 16.75 million patient-hours. The hourly event rate was 1.6% (12,842 transfers or deaths, corresponding to 260,295 patient-hours within the predictive horizon). On a hold-out dataset, the deep recurrent neural network achieved an area under the precision-recall curve of 0.042 (95% CI, 0.04-0.043), comparable with logistic regression model (0.043; 95% CI 0.041 to 0.045), and outperformed National Early Warning Score (0.034; 95% CI, 0.032-0.035), Modified Early Warning Score (0.028; 95% CI, 0.027- 0.03), and quick Sepsis-related Organ Failure Assessment (0.021; 95% CI, 0.021-0.022). For a fixed sensitivity of 50%, the deep recurrent neural network achieved a positive predictive value of 3.4% (95% CI, 3.4-3.5) and outperformed logistic regression model (3.1%; 95% CI 3.1-3.2), National Early Warning Score (2.0%; 95% CI, 2.0-2.0), Modified Early Warning Score (1.5%; 95% CI, 1.5-1.5), and quick Sepsis-related Organ Failure Assessment (1.5%; 95% CI, 1.5-1.5).

CONCLUSIONS

Commonly used early warning scores for clinical decompensation, along with a logistic regression model and a deep recurrent neural network model, show very poor performance characteristics when assessed using a simulated prospective validation. None of these models may be suitable for real-time deployment.

Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine

OBJECTIVE

The change in performance of machine learning models over time as a result of temporal dataset shift is a barrier to machine learning-derived models facilitating decision-making in clinical practice. Our aim was to describe technical procedures used to preserve the performance of machine learning models in the presence of temporal dataset shifts.

METHODS

Studies were included if they were fully published articles that used machine learning and implemented a procedure to mitigate the effects of temporal dataset shift in a clinical setting. We described how dataset shift was measured, the procedures used to preserve model performance, and their effects.

RESULTS

Of 4,457 potentially relevant publications identified, 15 were included. The impact of temporal dataset shift was primarily quantified using changes, usually deterioration, in calibration or discrimination. Calibration deterioration was more common (n = 11) than discrimination deterioration (n = 3). Mitigation strategies were categorized as model level or feature level. Model-level approaches (n = 15) were more common than feature-level approaches (n = 2), with the most common approaches being model refitting (n = 12), probability calibration (n = 7), model updating (n = 6), and model selection (n = 6). In general, all mitigation strategies were successful at preserving calibration but not uniformly successful in preserving discrimination.

CONCLUSION

There was limited research in preserving the performance of machine learning models in the presence of temporal dataset shift in clinical medicine. Future research could focus on the impact of dataset shift on clinical decision making, benchmark the mitigation strategies on a wider range of datasets and tasks, and identify optimal strategies for specific settings.

A framework for making predictive models useful in practice

OBJECTIVE

To analyze the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality.

MATERIALS AND METHODS

We built a predictive model of 12-month mortality using electronic health record data and evaluated the impact of healthcare delivery factors on the net benefit of triggering an ACP workflow based on the models' predictions. Factors included nonclinical reasons that make ACP inappropriate: limited capacity for ACP, inability to follow up due to patient discharge, and availability of an outpatient workflow to follow up on missed cases. We also quantified the relative benefits of increasing capacity for inpatient ACP versus outpatient ACP.

RESULTS

Work capacity constraints and discharge timing can significantly reduce the net benefit of triggering the ACP workflow based on a model's predictions. However, the reduction can be mitigated by creating an outpatient ACP workflow. Given limited resources to either add capacity for inpatient ACP versus developing outpatient ACP capability, the latter is likely to provide more benefit to patient care.

DISCUSSION

The benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. We provide a framework for quantifying the impact of healthcare delivery factors and work capacity constraints on achieved benefit.

CONCLUSION

An analysis of the sensitivity of the net benefit realized by a predictive model triggered clinical workflow to various healthcare delivery factors is necessary for making predictive models useful in practice.

Improving hospital readmission prediction using individualized utility analysis

OBJECTIVE

Machine learning (ML) models for allocating readmission-mitigating interventions are typically selected according to their discriminative ability, which may not necessarily translate into utility in allocation of resources. Our objective was to determine whether ML models for allocating readmission-mitigating interventions have different usefulness based on their overall utility and discriminative ability.

MATERIALS AND METHODS

We conducted a retrospective utility analysis of ML models using claims data acquired from the Optum Clinformatics Data Mart, including 513,495 commercially-insured inpatients (mean [SD] age 69 [19] years; 294,895 [57%] Female) over the period January 2016 through January 2017 from all 50 states with mean 90 day cost of $11,552. Utility analysis estimates the cost, in dollars, of allocating interventions for lowering readmission risk based on the reduction in the 90-day cost.

RESULTS

Allocating readmission-mitigating interventions based on a GBDT model trained to predict readmissions achieved an estimated utility gain of $104 per patient, and an AUC of 0.76 (95% CI 0.76, 0.77); allocating interventions based on a model trained to predict cost as a proxy achieved a higher utility of $175.94 per patient, and an AUC of 0.62 (95% CI 0.61, 0.62). A hybrid model combining both intervention strategies is comparable with the best models on either metric. Estimated utility varies by intervention cost and efficacy, with each model performing the best under different intervention settings.

CONCLUSION

We demonstrate that machine learning models may be ranked differently based on overall utility and discriminative ability. Machine learning models for allocation of limited health resources should consider directly optimizing for utility.

Rethinking PICO in the Machine Learning Era: ML-PICO

BACKGROUND

Machine learning (ML) has captured the attention of many clinicians who may not have formal training in this area but are otherwise increasingly exposed to ML literature that may be relevant to their clinical specialties. ML papers that follow an outcomes-based research format can be assessed using clinical research appraisal frameworks such as PICO (Population, Intervention, Comparison, Outcome). However, the PICO frameworks strain when applied to ML papers that create new ML models, which are akin to diagnostic tests. There is a need for a new framework to help assess such papers.

OBJECTIVE

We propose a new framework to help clinicians systematically read and evaluate medical ML papers whose aim is to create a new ML model: ML-PICO (Machine Learning, Population, Identification, Crosscheck, Outcomes). We describe how the ML-PICO framework can be applied toward appraising literature describing ML models for health care.

CONCLUSION

The relevance of ML to practitioners of clinical medicine is steadily increasing with a growing body of literature. Therefore, it is increasingly important for clinicians to be familiar with how to assess and best utilize these tools. In this paper we have described a practical framework on how to read ML papers that create a new ML model (or diagnostic test): ML-PICO. We hope that this can be used by clinicians to better evaluate the quality and utility of ML papers.

Predicting preventable hospital readmissions with causal machine learning

OBJECTIVE

To assess both the feasibility and potential impact of predicting preventable hospital readmissions using causal machine learning applied to data from the implementation of a readmissions prevention intervention (the Transitions Program).

DATA SOURCES

Electronic health records maintained by Kaiser Permanente Northern California (KPNC).

STUDY DESIGN

Retrospective causal forest analysis of postdischarge outcomes among KPNC inpatients. Using data from both before and after implementation, we apply causal forests to estimate individual-level treatment effects of the Transitions Program intervention on 30-day readmission. These estimates are used to characterize treatment effect heterogeneity and to assess the notional impacts of alternative targeting strategies in terms of the number of readmissions prevented.

DATA COLLECTION

1 539 285 index hospitalizations meeting the inclusion criteria and occurring between June 2010 and December 2018 at 21 KPNC hospitals.

PRINCIPAL FINDINGS

There appears to be substantial heterogeneity in patients' responses to the intervention (omnibus test for heterogeneity p = 2.23 × 10-7 ), particularly across levels of predicted risk. Notably, predicted treatment effects become more positive as predicted risk increases; patients at somewhat lower risk appear to have the largest predicted effects. Moreover, these estimates appear to be well calibrated, yielding the same estimate of annual readmissions prevented in the actual treatment subgroup (1246, 95% confidence interval [CI] 1110-1381) as did a formal evaluation of the Transitions Program (1210, 95% CI 990-1430). Estimates of the impacts of alternative targeting strategies suggest that as many as 4458 (95% CI 3925-4990) readmissions could be prevented annually, while decreasing the number needed to treat from 33 to 23, by targeting patients with the largest predicted effects rather than those at highest risk.

CONCLUSIONS

Causal machine learning can be used to identify preventable hospital readmissions, if the requisite interventional data are available. Moreover, our results suggest a mismatch between risk and treatment effects.

Review of Clinical Research Informatics

OBJECTIVES

Clinical Research Informatics (CRI) declares its scope in its name, but its content, both in terms of the clinical research it supports-and sometimes initiates-and the methods it has developed over time, reach much further than the name suggests. The goal of this review is to celebrate the extraordinary diversity of activity and of results, not as a prize-giving pageant, but in recognition of the field, the community that both serves and is sustained by it, and of its interdisciplinarity and its international dimension.

METHODS

Beyond personal awareness of a range of work commensurate with the author's own research, it is clear that, even with a thorough literature search, a comprehensive review is impossible. Moreover, the field has grown and subdivided to an extent that makes it very hard for one individual to be familiar with every branch or with more than a few branches in any depth. A literature survey was conducted that focused on informatics-related terms in the general biomedical and healthcare literature, and specific concerns ("artificial intelligence", "data models", "analytics", etc.) in the biomedical informatics (BMI) literature. In addition to a selection from the results from these searches, suggestive references within them were also considered.

RESULTS

The substantive sections of the paper-Artificial Intelligence, Machine Learning, and "Big Data" Analytics; Common Data Models, Data Quality, and Standards; Phenotyping and Cohort Discovery; Privacy: Deidentification, Distributed Computation, Blockchain; Causal Inference and Real-World Evidence-provide broad coverage of these active research areas, with, no doubt, a bias towards this reviewer's interests and preferences, landing on a number of papers that stood out in one way or another, or, alternatively, exemplified a particular line of work.

CONCLUSIONS

CRI is thriving, not only in the familiar major centers of research, but more widely, throughout the world. This is not to pretend that the distribution is uniform, but to highlight the potential for this domain to play a prominent role in supporting progress in medicine, healthcare, and wellbeing everywhere. We conclude with the observation that CRI and its practitioners would make apt stewards of the new medical knowledge that their methods will bring forward.

Comparison of Early Warning Scoring Systems for Hospitalized Patients With and Without Infection at Risk for In-Hospital Mortality and Transfer to the Intensive Care Unit

IMPORTANCE

Risk scores used in early warning systems exist for general inpatients and patients with suspected infection outside the intensive care unit (ICU), but their relative performance is incompletely characterized.

OBJECTIVE

To compare the performance of tools used to determine points-based risk scores among all hospitalized patients, including those with and without suspected infection, for identifying those at risk for death and/or ICU transfer.

DESIGN, SETTING, AND PARTICIPANTS

In a cohort design, a retrospective analysis of prospectively collected data was conducted in 21 California and 7 Illinois hospitals between 2006 and 2018 among adult inpatients outside the ICU using points-based scores from 5 commonly used tools: National Early Warning Score (NEWS), Modified Early Warning Score (MEWS), Between the Flags (BTF), Quick Sequential Sepsis-Related Organ Failure Assessment (qSOFA), and Systemic Inflammatory Response Syndrome (SIRS). Data analysis was conducted from February 2019 to January 2020.

MAIN OUTCOMES AND MEASURES

Risk model discrimination was assessed in each state for predicting in-hospital mortality and the combined outcome of ICU transfer or mortality with area under the receiver operating characteristic curves (AUCs). Stratified analyses were also conducted based on suspected infection.

RESULTS

The study included 773 477 hospitalized patients in California (mean [SD] age, 65.1 [17.6] years; 416 605 women [53.9%]) and 713 786 hospitalized patients in Illinois (mean [SD] age, 61.3 [19.9] years; 384 830 women [53.9%]). The NEWS exhibited the highest discrimination for mortality (AUC, 0.87; 95% CI, 0.87-0.87 in California vs AUC, 0.86; 95% CI, 0.85-0.86 in Illinois), followed by the MEWS (AUC, 0.83; 95% CI, 0.83-0.84 in California vs AUC, 0.84; 95% CI, 0.84-0.85 in Illinois), qSOFA (AUC, 0.78; 95% CI, 0.78-0.79 in California vs AUC, 0.78; 95% CI, 0.77-0.78 in Illinois), SIRS (AUC, 0.76; 95% CI, 0.76-0.76 in California vs AUC, 0.76; 95% CI, 0.75-0.76 in Illinois), and BTF (AUC, 0.73; 95% CI, 0.73-0.73 in California vs AUC, 0.74; 95% CI, 0.73-0.74 in Illinois). At specific decision thresholds, the NEWS outperformed the SIRS and qSOFA at all 28 hospitals either by reducing the percentage of at-risk patients who need to be screened by 5% to 20% or increasing the percentage of adverse outcomes identified by 3% to 25%.

CONCLUSIONS AND RELEVANCE

In all hospitalized patients evaluated in this study, including those meeting criteria for suspected infection, the NEWS appeared to display the highest discrimination. Our results suggest that, among commonly used points-based scoring systems, determining the NEWS for inpatient risk stratification could identify patients with and without infection at high risk of mortality.

Do no harm: a roadmap for responsible machine learning for health care

Interest in machine-learning applications within medicine has been growing, but few studies have progressed to deployment in patient care. We present a framework, context and ultimately guidelines for accelerating the translation of machine-learning-based interventions in health care. To be successful, translation will require a team of engaged stakeholders and a systematic process from beginning (problem formulation) to end (widespread deployment).