Development, implementation, and prospective validation of a model to predict 60-day end-of-life in hospitalized adults upon admission at three sites

Geriatric End-of-Life Screening Tool Prediction of 6-Month Mortality in Older Patients

Importance

Emergency department (ED) visits by older adults with life-limiting illnesses are a critical opportunity to establish patient care end-of-life preferences, but little is known about the optimal screening criteria for resource-constrained EDs.

Objectives

To externally validate the Geriatric End-of-Life Screening Tool (GEST) in an independent population and compare it with commonly used serious illness diagnostic criteria.

Design, Setting, and Participants

This prognostic study assessed a cohort of patients aged 65 years and older who were treated in a tertiary care ED in Boston, Massachusetts, from 2017 to 2021. Patients arriving in cardiac arrest or who died within 1 day of ED arrival were excluded. Data analysis was performed from August 1, 2023, to March 27, 2024.

Exposure

GEST, a logistic regression algorithm that uses commonly available electronic health record (EHR) datapoints and was developed and validated across 9 EDs, was compared with serious illness diagnoses as documented in the EHR. Serious illnesses included stroke/transient ischemic attack, liver disease, cancer, lung disease, and age greater than 80 years, among others.

Main Outcomes and Measures

The primary outcome was 6-month mortality following an ED encounter. Statistical analyses included area under the receiver operating characteristic curve, calibration analyses, Kaplan-Meier survival curves, and decision curves.

Results

This external validation included 82 371 ED encounters by 40 505 unique individuals (mean [SD] age, 76.8 [8.4] years; 54.3% women, 13.8% 6-month mortality rate). GEST had an external validation area under the receiver operating characteristic curve of 0.79 (95% CI, 0.78-0.79) that was stable across years and demographic subgroups. Of included encounters, 53.4% had a serious illness, with a sensitivity of 77.4% (95% CI, 76.6%-78.2%) and specificity of 50.5% (95% CI, 50.1%-50.8%). Varying GEST cutoffs from 5% to 30% increased specificity (5%: 49.1% [95% CI, 48.7%-49.5%]; 30%: 92.2% [95% CI, 92.0%-92.4%]) at the cost of sensitivity (5%: 89.3% [95% CI, 88.8-89.9]; 30%: 36.2% [95% CI, 35.3-37.1]). In a decision curve analysis, GEST outperformed serious illness criteria across all tested thresholds. When comparing patients referred to intervention by GEST with serious illness criteria, GEST reclassified 45.1% of patients with serious illness as having low risk of mortality with an observed mortality rate 8.1% and 2.6% of patients without serious illness as having high mortality risk with an observed mortality rate of 34.3% for a total reclassification rate of 25.3%.

Conclusions and Relevance

The findings of this study suggest that both serious illness criteria and GEST identified older ED patients at risk for 6-month mortality, but GEST offered more useful screening characteristics. Future trials of serious illness interventions for high mortality risk in older adults may consider transitioning from diagnosis code criteria to GEST, an automatable EHR-based algorithm.

Automatable end-of-life screening for older adults in the emergency department using electronic health records

BACKGROUND

Emergency department (ED) visits are common at the end-of-life, but the identification of patients with life-limiting illness remains a key challenge in providing timely and resource-sensitive advance care planning (ACP) and palliative care services. To date, there are no validated, automatable instruments for ED end-of-life screening. Here, we developed a novel electronic health record (EHR) prognostic model to screen older ED patients at high risk for 6-month mortality and compare its performance to validated comorbidity indices.

METHODS

This was a retrospective, observational cohort study of ED visits from adults aged ≥65 years who visited any of 9 EDs across a large regional health system between 2014 and 2019. Multivariable logistic regression that included clinical and demographic variables, vital signs, and laboratory data was used to develop a 6-month mortality predictive model-the Geriatric End-of-life Screening Tool (GEST) using five-fold cross-validation on data from 8 EDs. Performance was compared to the Charlson and Elixhauser comorbidity indices using area under the receiver-operating characteristic curve (AUROC), calibration, and decision curve analyses. Reproducibility was tested against data from the remaining independent ED within the health system. We then used GEST to investigate rates of ACP documentation availability and code status orders in the EHR across risk strata.

RESULTS

A total of 431,179 encounters by 123,128 adults were included in this study with a 6-month mortality rate of 12.2%. Charlson (AUROC (95% CI): 0.65 (0.64-0.69)) and Elixhauser indices (0.69 (0.68-0.70)) were outperformed by GEST (0.82 (0.82-0.83)). GEST displayed robust performance across demographic subgroups and in our independent validation site. Among patients with a greater than 30% mortality risk using GEST, only 5.0% had ACP documentation; 79.0% had a code status previously ordered, of which 70.7% were full code. In decision curve analysis, GEST provided greater net benefit than the Charlson and Elixhauser scores.

CONCLUSIONS

Prognostic models using EHR data robustly identify high mortality risk older adults in the ED for whom code status, ACP, or palliative care interventions may be of benefit. Although all tested methods identified patients approaching the end-of-life, GEST was most performant. These tools may enable resource-sensitive end-of-life screening in the ED.

Frailty, Comorbidity, and Associations With In-Hospital Mortality in Older COVID-19 Patients: Exploratory Study of Administrative Data

BACKGROUND

Older adults have worse outcomes following hospitalization with COVID-19, but within this group there is substantial variation. Although frailty and comorbidity are key determinants of mortality, it is less clear which specific manifestations of frailty and comorbidity are associated with the worst outcomes.

OBJECTIVE

We aimed to identify the key comorbidities and domains of frailty that were associated with in-hospital mortality in older patients with COVID-19 using models developed for machine learning algorithms.

METHODS

This was a retrospective study that used the Hospital Episode Statistics administrative data set from March 1, 2020, to February 28, 2021, for hospitalized patients in England aged 65 years or older. The data set was split into separate training (70%), test (15%), and validation (15%) data sets during model development. Global frailty was assessed using the Hospital Frailty Risk Score (HFRS) and specific domains of frailty were identified using the Global Frailty Scale (GFS). Comorbidity was assessed using the Charlson Comorbidity Index (CCI). Additional features employed in the random forest algorithms included age, sex, deprivation, ethnicity, discharge month and year, geographical region, hospital trust, disease severity, and International Statistical Classification of Disease, 10th Edition codes recorded during the admission. Features were selected, preprocessed, and input into a series of random forest classification algorithms developed to identify factors strongly associated with in-hospital mortality. Two models were developed; the first model included the demographic, hospital-related, and disease-related items described above, as well as individual GFS domains and CCI items. The second model was similar to the first but replaced the GFS domains and CCI items with the HFRS as a global measure of frailty. Model performance was assessed using the area under the receiver operating characteristic (AUROC) curve and measures of model accuracy.

RESULTS

In total, 215,831 patients were included. The model using the individual GFS domains and CCI items had an AUROC curve for in-hospital mortality of 90% and a predictive accuracy of 83%. The model using the HFRS had similar performance (AUROC curve 90%, predictive accuracy 82%). The most important frailty items in the GFS were dementia/delirium, falls/fractures, and pressure ulcers/weight loss. The most important comorbidity items in the CCI were cancer, heart failure, and renal disease.

CONCLUSIONS

The physical manifestations of frailty and comorbidity, particularly a history of cognitive impairment and falls, may be useful in identification of patients who need additional support during hospitalization with COVID-19.

Synchrony of biomarker variability indicates a critical transition: Application to mortality prediction in hemodialysis

Critical transition theory suggests that complex systems should experience increased temporal variability just before abrupt state changes. We tested this hypothesis in 763 patients on long-term hemodialysis, using 11 biomarkers collected every two weeks and all-cause mortality as a proxy for critical transitions. We find that variability-measured by coefficients of variation (CVs)-increases before death for all 11 clinical biomarkers, and is strikingly synchronized across all biomarkers: the first axis of a principal component analysis on all CVs explains 49% of the variance. This axis then generates powerful predictions of mortality (HR95 = 9.7, p < 0.0001, where HR95 is a scale-invariant metric of hazard ratio; AUC up to 0.82) and starts to increase markedly ∼3 months prior to death. Our results provide an early warning sign of physiological collapse and, more broadly, a quantification of joint system dynamics that opens questions of how system modularity may break down before critical transitions.

Development and Structure of an Accurate Machine Learning Algorithm to Predict Inpatient Mortality and Hospice Outcomes in the Coronavirus Disease 2019 Era

BACKGROUND

The coronavirus disease 2019 (COVID-19) pandemic has challenged the accuracy and racial biases present in traditional mortality scores. An accurate prognostic model that can be applied to hospitalized patients irrespective of race or COVID-19 status may benefit patient care.

RESEARCH DESIGN

This cohort study utilized historical and ongoing electronic health record features to develop and validate a deep-learning model applied on the second day of admission predicting a composite outcome of in-hospital mortality, discharge to hospice, or death within 30 days of admission. Model features included patient demographics, diagnoses, procedures, inpatient medications, laboratory values, vital signs, and substance use history. Conventional performance metrics were assessed, and subgroup analysis was performed based on race, COVID-19 status, and intensive care unit admission.

SUBJECTS

A total of 35,521 patients hospitalized between April 2020 and October 2020 at a single health care system including a tertiary academic referral center and 9 community hospitals.

RESULTS

Of 35,521 patients, including 9831 non-White patients and 2020 COVID-19 patients, 2838 (8.0%) met the composite outcome. Patients who experienced the composite outcome were older (73 vs. 61 y old) with similar sex and race distributions between groups. The model achieved an area under the receiver operating characteristic curve of 0.89 (95% confidence interval: 0.88, 0.91) and an average positive predictive value of 0.46 (0.40, 0.52). Model performance did not differ significantly in White (0.89) and non-White (0.90) subgroups or when grouping by COVID-19 status and intensive care unit admission.

CONCLUSION

A deep-learning model using large-volume, structured electronic health record data can effectively predict short-term mortality or hospice outcomes on the second day of admission in the general inpatient population without significant racial bias.

Open-Source Clinical Machine Learning Models: Critical Appraisal of Feasibility, Advantages, and Challenges

Machine learning applications promise to augment clinical capabilities and at least 64 models have already been approved by the US Food and Drug Administration. These tools are developed, shared, and used in an environment in which regulations and market forces remain immature. An important consideration when evaluating this environment is the introduction of open-source solutions in which innovations are freely shared; such solutions have long been a facet of digital culture. We discuss the feasibility and implications of open-source machine learning in a health care infrastructure built upon proprietary information. The decreased cost of development as compared to drugs and devices, a longstanding culture of open-source products in other industries, and the beginnings of machine learning–friendly regulatory pathways together allow for the development and deployment of open-source machine learning models. Such tools have distinct advantages including enhanced product integrity, customizability, and lower cost, leading to increased access. However, significant questions regarding engineering concerns about implementation infrastructure and model safety, a lack of incentives from intellectual property protection, and nebulous liability rules significantly complicate the ability to develop such open-source models. Ultimately, the reconciliation of open-source machine learning and the proprietary information–driven health care environment requires that policymakers, regulators, and health care organizations actively craft a conducive market in which innovative developers will continue to both work and collaborate.

Expected clinical utility of automatable prediction models for improving palliative and end-of-life care outcomes: Toward routine decision analysis before implementation

Objective

The study sought to evaluate the expected clinical utility of automatable prediction models for increasing goals-of-care discussions (GOCDs) among hospitalized patients at the end of life (EOL).

Materials and Methods

We built a decision model from the perspective of clinicians who aim to increase GOCDs at the EOL using an automated alert system. The alternative strategies were 4 prediction models—3 random forest models and the Modified Hospital One-year Mortality Risk model—to generate alerts for patients at a high risk of 1-year mortality. They were trained on admissions from 2011 to 2016 (70 788 patients) and tested with admissions from 2017-2018 (16 490 patients). GOCDs occurring in usual care were measured with code status orders. We calculated the expected risk difference (beneficial outcomes with alerts minus beneficial outcomes without alerts among those at the EOL), the number needed to benefit (number of alerts needed to increase benefit over usual care by 1 outcome), and the net benefit (benefit minus cost) of each strategy.

Results

Models had a C-statistic between 0.79 and 0.86. A code status order occurred during 2599 of 3773 (69%) hospitalizations at the EOL. At a risk threshold corresponding to an alert prevalence of 10%, the expected risk difference ranged from 5.4% to 10.7% and the number needed to benefit ranged from 5.4 to 10.9 alerts. Using revealed preferences, only 2 models improved net benefit over usual care. A random forest model with diagnostic predictors had the highest expected value, including in sensitivity analyses.

Discussion

Prediction models with acceptable predictive validity differed meaningfully in their ability to improve over usual decision making.

Conclusions

An evaluation of clinical utility, such as by using decision curve analysis, is recommended after validating a prediction model because metrics of model predictiveness, such as the C-statistic, are not informative of clinical value.

Magnetic resonance imaging quality control, quality assurance and quality improvement

Quality in MR imaging is a comprehensive process that encompasses scanner performance, clinical processes for efficient scanning and reporting, as well as data-driven improvement involving measurement of key performance indicators. In this paper, the authors review this entire process. This article provides a framework for establishing a successful MR quality program. The collective experiences of the authors across a spectrum of pediatric hospitals is summarized here.