Electronic Health Record Mortality Prediction Model for Targeted Palliative Care Among Hospitalized Medical Patients: a Pilot Quasi-experimental Study

Which medicare advantage enrollees are at highest one-year mortality risk?

BACKGROUND

While a number of tools exist to predict mortality among older adults, less research has described the characteristics of Medicare Advantage (MA) enrollees at higher risk for 1 year mortality.

OBJECTIVES

To describe the characteristics of MA enrollees at higher mortality risk using patient survey data.

RESEARCH DESIGN

Retrospective cohort.

SUBJECTS

MA enrollees completing the 2019 MA Consumer Assessment of Healthcare Providers and Systems (CAHPS) Survey.

MEASURES

Linked demographic, health, and mortality data from a sample of MA enrollees were used to predict 1-year mortality risk and describe enrollee characteristics across levels of predicted mortality risk.

RESULTS

The mortality model had a 0.80 c-statistic. Mortality risks were skewed: 6 % of enrollees had a ≥ 10 % 1-year mortality risk, while 45 % of enrollees had 1 % to < 5 % 1-year mortality risk. Among the high-risk (≥10 %) group, 47 % were age 85+ versus 12 % among those with mortality risk <5 %. 79 % were in fair or poor self-rated health versus 29 % among those with mortality risk of <5 %. 71 % reported needing urgent care in the prior 6 months versus 40 % among those with a mortality risk of 1 to<5 %.

CONCLUSIONS

Relatively few older adults enrolled in MA are at high 1-year mortality risk. Nonetheless, MA enrollees over age 85, in fair or poor health, or with recent urgent care needs are far more likely to be in a high mortality risk group.

Hospital-Onset Sepsis Warrants Expanded Investigation and Consideration as a Unique Clinical Entity

Sepsis causes more than a quarter million deaths among hospitalized adults in the United States each year. Although most cases of sepsis are present on admission, up to one-quarter of patients with sepsis develop this highly morbid and mortal condition while hospitalized. Compared with patients with community-onset sepsis (COS), patients with hospital-onset sepsis (HOS) are twice as likely to require mechanical ventilation and ICU admission, have more than two times longer ICU and hospital length of stay, accrue five times higher hospital costs, and are twice as likely to die. Patients with HOS differ from those with COS with respect to underlying comorbidities, admitting diagnosis, clinical manifestations of infection, and severity of illness. Despite the differences between these patient populations, patients with HOS sepsis are understudied and warrant expanded investigation. Here, we outline important knowledge gaps in the recognition and management of HOS in adults and propose associated research priorities for investigators. Of particular importance are questions regarding standardization of research and clinical case identification, understanding of clinical heterogeneity among patients with HOS, development of tailored management recommendations, identification of impactful prevention strategies, optimization of care delivery and quality metrics, identification and correction of disparities in care and outcomes, and how to ensure goal-concordant care for patients with HOS.

Classification of Documented Goals of Care Among Hospitalized Patients with High Mortality Risk: a Mixed-Methods Feasibility Study

BACKGROUND

The ability to classify patients' goals of care (GOC) from clinical documentation would facilitate serious illness communication quality improvement efforts and pragmatic measurement of goal-concordant care. Feasibility of this approach remains unknown.

OBJECTIVE

To evaluate the feasibility of classifying patients' GOC from clinical documentation in the electronic health record (EHR), describe the frequency and patterns of changes in patients' goals over time, and identify barriers to reliable goal classification.

DESIGN

Retrospective, mixed-methods chart review study.

PARTICIPANTS

Adults with high (50-74%) and very high (≥ 75%) 6-month mortality risk admitted to three urban hospitals.

MAIN MEASURES

Two physician coders independently reviewed EHR notes from 6 months before through 6 months after admission to identify documented GOC discussions and classify GOC. GOC were classified into one of four prespecified categories: (1) comfort-focused, (2) maintain or improve function, (3) life extension, or (4) unclear. Coder interrater reliability was assessed using kappa statistics. Barriers to classifying GOC were assessed using qualitative content analysis.

KEY RESULTS

Among 85 of 109 (78%) patients, 338 GOC discussions were documented. Inter-rater reliability was substantial (75% interrater agreement; Cohen's kappa = 0.67; 95% CI, 0.60-0.73). Patients' initial documented goal was most frequently "life extension" (N = 37, 44%), followed by "maintain or improve function" (N = 28, 33%), "unclear" (N = 17, 20%), and "comfort-focused" (N = 3, 4%). Among the 66 patients whose goals' classification changed over time, most changed to "comfort-focused" goals (N = 49, 74%). Primary reasons for unclear goals were the observation of concurrently held or conditional goals, patient and family uncertainty, and limited documentation.

CONCLUSIONS

Clinical notes in the EHR can be used to reliably classify patients' GOC into discrete, clinically germane categories. This work motivates future research to use natural language models to promote scalability of the approach in clinical care and serious illness research.

Applying Machine Learning Techniques to Implementation Science

Machine learning (ML) approaches could expand the usefulness and application of implementation science methods in clinical medicine and public health settings. The aim of this viewpoint is to introduce a roadmap for applying ML techniques to address implementation science questions, such as predicting what will work best, for whom, under what circumstances, and with what predicted level of support, and what and when adaptation or deimplementation are needed. We describe how ML approaches could be used and discuss challenges that implementation scientists and methodologists will need to consider when using ML throughout the stages of implementation.

Detecting goals of care conversations in clinical notes with active learning

OBJECTIVE

Goals of care (GOC) discussions are an increasingly used quality metric in serious illness care and research. Wide variation in documentation practices within the Electronic Health Record (EHR) presents challenges for reliable measurement of GOC discussions. Novel natural language processing approaches are needed to capture GOC discussions documented in real-world samples of seriously ill hospitalized patients' EHR notes, a corpus with a very low event prevalence.

METHODS

To automatically detect sentences documenting GOC discussions outside of dedicated GOC note types, we proposed an ensemble of classifiers aggregating the predictions of rule-based, feature-based, and three transformers-based classifiers. We trained our classifier on 600 manually annotated EHR notes among patients with serious illnesses. Our corpus exhibited an extremely imbalanced ratio between sentences discussing GOC and sentences that do not. This ratio challenges standard supervision methods to train a classifier. Therefore, we trained our classifier with active learning.

RESULTS

Using active learning, we reduced the annotation cost to fine-tune our ensemble by 70% while improving its performance in our test set of 176 EHR notes, with 0.557 F1-score for sentence classification and 0.629 for note classification.

CONCLUSION

When classifying notes, with a true positive rate of 72% (13/18) and false positive rate of 8% (13/158), our performance may be sufficient for deploying our classifier in the EHR to facilitate bedside clinicians' access to GOC conversations documented outside of dedicated notes types, without overburdening clinicians with false positives. Improvements are needed before using it to enrich trial populations or as an outcome measure.

Estimated mortality risk and use of palliative care services among home care clients during the last 6 months of life: a retrospective cohort study

BACKGROUND

In Canada, only 15% of patients requiring palliative care receive such services in the year before death. We describe health care utilization patterns among home care users in their last 6 months of life to inform care planning for older people with varying mortality risks and evolving care needs as they decline.

METHODS

Using population health administrative data from Ontario, we performed a retrospective cohort study involving home care clients aged 50 years and older who received at least 1 interRAI (Resident Assessment Instrument) Home Care assessment between April 2018 and September 2019. We report the proportion of clients who used acute care, long-term care, and palliative home care services within 6 months of their assessment, stratified by their predicted 6-month mortality risk using a prognostic tool called the Risk Evaluation for Support: Predictions for Elder-life in their Communities Tool (RESPECT) and vital status.

RESULTS

The cohort included 247 377 adults, 11.9% of whom died within 6 months of an assessment. Among decedents, 50.6% of those with a RESPECT-estimated median survival of fewer than 3 months received at least 1 nonphysician palliative home care visit before death. This proportion declined to 38.7% and 29.5% among decedents with an estimated median survival between 3 and 6 months and between 6 and 12 months, respectively.

INTERPRETATION

Many older adults in Ontario do not receive any palliative home care before death. Prognostic tools such as RESPECT may improve recognition of reduced life expectancies and palliative care needs of individuals in their final years of life.

Implementing a Serious Illness Risk Prediction Model: Impact on Goals of Care Documentation

CONTEXT

Goals of care conversations can promote high value care for patients with serious illness, yet documented discussions infrequently occur in hospital settings.

OBJECTIVES

We sought to develop a quality improvement initiative to improve goals of care documentation for hospitalized patients.

METHODS

Implementation occurred at an academic medical center in Pittsburgh, Pennsylvania. Intervention included integration of a 90-day mortality prediction model grouping patients into low, intermediate, and high risk; a centralized goals of care note; and automated notifications and targeted palliative consults. We compared documented goals of care discussions by risk score before and after implementation.

RESULTS

Of the 12,571 patients hospitalized preimplementation and 10,761 postimplementation, 1% were designated high risk and 11% intermediate risk of mortality. Postimplementation, goals of care documentation increased for high (17.6%-70.8%, P< 0.0001) and intermediate risk patients (9.6%-28.0%, P < 0.0001). For intermediate risk patients, the percentage of goals of care documentation performed by palliative medicine specialists increased from pre- to postimplementation (52.3%-71.2%, P = 0.0002). For high-risk patients, the percentage of goals of care documentation completed by the primary service increased from pre-to postimplementation (36.8%-47.1%, P = 0.5898, with documentation performed by palliative medicine specialists slightly decreasing from pre- to postimplementation (63.2%-52.9%, P = 0.5898).

CONCLUSIONS

Implementation of a goals of care initiative using a mortality prediction model significantly increased goals of care documentation especially among high-risk patients. Further study to assess strategies to increase goals of care documentation for intermediate risk patients is needed especially by nonspecialty palliative care.

Evaluating equity in performance of an electronic health record-based 6-month mortality risk model to trigger palliative care consultation: a retrospective model validation analysis

OBJECTIVE

Evaluate predictive performance of an electronic health record (EHR)-based, inpatient 6-month mortality risk model developed to trigger palliative care consultation among patient groups stratified by age, race, ethnicity, insurance and socioeconomic status (SES), which may vary due to social forces (eg, racism) that shape health, healthcare and health data.

DESIGN

Retrospective evaluation of prediction model.

SETTING

Three urban hospitals within a single health system.

PARTICIPANTS

All patients ≥18 years admitted between 1 January and 31 December 2017, excluding observation, obstetric, rehabilitation and hospice (n=58 464 encounters, 41 327 patients).

MAIN OUTCOME MEASURES

General performance metrics (c-statistic, integrated calibration index (ICI), Brier Score) and additional measures relevant to health equity (accuracy, false positive rate (FPR), false negative rate (FNR)).

RESULTS

For black versus non-Hispanic white patients, the model's accuracy was higher (0.051, 95% CI 0.044 to 0.059), FPR lower (-0.060, 95% CI -0.067 to -0.052) and FNR higher (0.049, 95% CI 0.023 to 0.078). A similar pattern was observed among patients who were Hispanic, younger, with Medicaid/missing insurance, or living in low SES zip codes. No consistent differences emerged in c-statistic, ICI or Brier Score. Younger age had the second-largest effect size in the mortality prediction model, and there were large standardised group differences in age (eg, 0.32 for non-Hispanic white versus black patients), suggesting age may contribute to systematic differences in the predicted probabilities between groups.

CONCLUSIONS

An EHR-based mortality risk model was less likely to identify some marginalised patients as potentially benefiting from palliative care, with younger age pinpointed as a possible mechanism. Evaluating predictive performance is a critical preliminary step in addressing algorithmic inequities in healthcare, which must also include evaluating clinical impact, and governance and regulatory structures for oversight, monitoring and accountability.

Effect of an Artificial Intelligence Decision Support Tool on Palliative Care Referral in Hospitalized Patients: A Randomized Clinical Trial

CONTEXT

Palliative care services are commonly provided to hospitalized patients, but accurately predicting who needs them remains a challenge.

OBJECTIVES

To assess the effectiveness on clinical outcomes of an artificial intelligence (AI)/machine learning (ML) decision support tool for predicting patient need for palliative care services in the hospital.

METHODS

The study design was a pragmatic, cluster-randomized, stepped-wedge clinical trial in 12 nursing units at two hospitals over a 15-month period between August 19, 2019, and November 17, 2020. Eligible patients were randomly assigned to either a medical service consultation recommendation triggered by an AI/ML tool predicting the need for palliative care services or usual care. The primary outcome was palliative care consultation note. Secondary outcomes included: hospital readmissions, length of stay, transfer to intensive care and palliative care consultation note by unit.

RESULTS

A total of 3183 patient hospitalizations were enrolled. Of eligible patients, A total of 2544 patients were randomized to the decision support tool (1212; 48%) and usual care (1332; 52%). Of these, 1717 patients (67%) were retained for analyses. Patients randomized to the intervention had a statistically significant higher incidence rate of palliative care consultation compared to the control group (IRR, 1.44 [95% CI, 1.11-1.92]). Exploratory evidence suggested that the decision support tool group reduced 60-day and 90-day hospital readmissions (OR, 0.75 [95% CI, 0.57, 0.97]) and (OR, 0.72 [95% CI, 0.55-0.93]) respectively.

CONCLUSION

A decision support tool integrated into palliative care practice and leveraging AI/ML demonstrated an increased palliative care consultation rate among hospitalized patients and reductions in hospitalizations.

Using the Hospital Frailty Risk Score to assess mortality risk in older medical patients admitted to the intensive care unit

BACKGROUND

Prognostic information at the time of hospital discharge can help guide goals-of-care discussions for future care. We sought to assess the association between the Hospital Frailty Risk Score (HFRS), which may highlight patients' risk of adverse outcomes at the time of hospital discharge, and in-hospital death among patients admitted to the intensive care unit (ICU) within 12 months of a previous hospital discharge.

METHODS

We conducted a multicentre retrospective cohort study that included patients aged 75 years or older admitted at least twice over a 12-month period to the general medicine service at 7 academic centres and large community-based teaching hospitals in Toronto and Mississauga, Ontario, Canada, from Apr. 1, 2010, to Dec. 31, 2019. The HFRS (categorized as low, moderate or high frailty risk) was calculated at the time of discharge from the first hospital admission. Outcomes included ICU admission and death during the second hospital admission.

RESULTS

The cohort included 22 178 patients, of whom 1767 (8.0%) were categorized as having high frailty risk, 9464 (42.7%) as having moderate frailty risk, and 10 947 (49.4%) as having low frailty risk. One hundred patients (5.7%) with high frailty risk were admitted to the ICU, compared to 566 (6.0%) of those with moderate risk and 790 (7.2%) of those with low risk. After adjustment for age, sex, hospital, day of admission, time of admission and Laboratory-based Acute Physiology Score, the odds of ICU admission were not significantly different for patients with high (adjusted odds ratio [OR] 0.99, 95% confidence interval [CI] 0.78 to 1.23) or moderate (adjusted OR 0.97, 95% CI 0.86 to 1.09) frailty risk compared to those with low frailty risk. Among patients admitted to the ICU, 75 (75.0%) of those with high frailty risk died, compared to 317 (56.0%) of those with moderate risk and 416 (52.7%) of those with low risk. After multivariable adjustment, the risk of death after ICU admission was higher for patients with high frailty risk than for those with low frailty risk (adjusted OR 2.86, 95% CI 1.77 to 4.77).

INTERPRETATION

Among patients readmitted to hospital within 12 months, patients with high frailty risk were similarly likely as those with lower frailty risk to be admitted to the ICU but were more likely to die if admitted to ICU. The HFRS at hospital discharge can inform prognosis, which can help guide discussions for preferences for ICU care during future hospital stays.

Improving Access to Specialist Palliative Care for Patients With Catastrophic Strokes Using Best Practice Advisory- a Feasibility Study

Background and Purpose

The American Heart Association and American Stroke Association (AHA/ASA) strongly recommend specialty palliative care (PC) for all patients hospitalized with life-threatening or life-altering strokes to provide expert symptom management, improve communication, promote shared decision-making and relieve suffering. We piloted an intervention to remind physicians about high PC needs of their patients admitted with catastrophic stroke.

Methods

We worked with colleagues from medical informatics to create a "Best Practice Advisory" (BPA) to recommend a goals-of-care conversation and PC consultation for patients with a National Institutes of Health Stroke Scale (NIHSS) score of 20 or more in our electronic medical record (Epic). We evaluated the impact of this BPA, after implementation, on the number and timing of PC consults and reviewed barriers to this system change.

Results

The BPA was operational in Jan 2019. Data analysis showed that it fired for all patients with an entered NIHSS score of ≥20. Though a large portion of the BPAs (more than 90%) were acknowledged without documented reason (after selecting "do not order"), PC consultations per 100 patients with triggered BPA increased from the first year of implementation (11.7 in 2019) to the next 2 years (20.7 in 2020, 15.6 in 2021). Also, the providers learned to manage BPA alerts better resulting in more than 30% reduction in the number of BPA alerts fired for each patient encounter in 2020-2021 compared to 2019.

Automatable end-of-life screening for older adults in the emergency department using electronic health records

BACKGROUND

Emergency department (ED) visits are common at the end-of-life, but the identification of patients with life-limiting illness remains a key challenge in providing timely and resource-sensitive advance care planning (ACP) and palliative care services. To date, there are no validated, automatable instruments for ED end-of-life screening. Here, we developed a novel electronic health record (EHR) prognostic model to screen older ED patients at high risk for 6-month mortality and compare its performance to validated comorbidity indices.

METHODS

This was a retrospective, observational cohort study of ED visits from adults aged ≥65 years who visited any of 9 EDs across a large regional health system between 2014 and 2019. Multivariable logistic regression that included clinical and demographic variables, vital signs, and laboratory data was used to develop a 6-month mortality predictive model-the Geriatric End-of-life Screening Tool (GEST) using five-fold cross-validation on data from 8 EDs. Performance was compared to the Charlson and Elixhauser comorbidity indices using area under the receiver-operating characteristic curve (AUROC), calibration, and decision curve analyses. Reproducibility was tested against data from the remaining independent ED within the health system. We then used GEST to investigate rates of ACP documentation availability and code status orders in the EHR across risk strata.

RESULTS

A total of 431,179 encounters by 123,128 adults were included in this study with a 6-month mortality rate of 12.2%. Charlson (AUROC (95% CI): 0.65 (0.64-0.69)) and Elixhauser indices (0.69 (0.68-0.70)) were outperformed by GEST (0.82 (0.82-0.83)). GEST displayed robust performance across demographic subgroups and in our independent validation site. Among patients with a greater than 30% mortality risk using GEST, only 5.0% had ACP documentation; 79.0% had a code status previously ordered, of which 70.7% were full code. In decision curve analysis, GEST provided greater net benefit than the Charlson and Elixhauser scores.

CONCLUSIONS

Prognostic models using EHR data robustly identify high mortality risk older adults in the ED for whom code status, ACP, or palliative care interventions may be of benefit. Although all tested methods identified patients approaching the end-of-life, GEST was most performant. These tools may enable resource-sensitive end-of-life screening in the ED.

Behavioural economic interventions to embed palliative care in community oncology (BE-EPIC): study protocol for the BE-EPIC randomised controlled trial

INTRODUCTION

Palliative care (PC) is a medical specialty focusing on providing relief from the symptoms and stress of serious illnesses such as cancer. Early outpatient specialty PC concurrent with cancer-directed treatment improves quality of life and symptom burden, decreases aggressive end-of-life care and is an evidence-based practice endorsed by national guidelines. However, nearly half of patients with advanced cancer do not receive specialty PC prior to dying. The objective of this study is to test the impact of an oncologist-directed default PC referral orders on rates of PC utilisation and patient quality of life.

METHODS AND ANALYSIS

This single-centre two-arm pragmatic randomised trial randomises four clinician-led pods, caring for approximately 250 patients who meet guideline-based criteria for PC referral, in a 1:1 fashion into a control or intervention arm. Intervention oncologists receive a nudge consisting of an electronic health record message indicating a patient has a default pended order for PC. Intervention oncologists are given an opportunity to opt out of referral to PC. Oncologists in pods randomised to the control arm will receive no intervention beyond usual practice. The primary outcome is completed PC visits within 12 weeks. Secondary outcomes are change in quality of life and absolute quality of life scores between the two arms.

ETHICS AND DISSEMINATION

This study has been approved by the Institutional Review Board at the University of Pennsylvania. Study results will be disseminated in peer-reviewed journals and scientific conferences using methods that describe the results in ways that key stakeholders can best understand and implement.

TRIAL REGISTRATION NUMBER

NCT05365997.

Impact of a machine learning algorithm on time to palliative care in a primary care population: protocol for a stepped-wedge pragmatic randomized trial

BACKGROUND

As primary care populations age, timely identification of palliative care need is becoming increasingly relevant. Previous studies have targeted particular patient populations with life-limiting disease, but few have focused on patients in a primary care setting. Toward this end, we propose a stepped-wedge pragmatic randomized trial whereby a machine learning algorithm identifies patients empaneled to primary care units at Mayo Clinic (Rochester, Minnesota, United States) with high likelihood of palliative care need.

METHODS

42 care team units in 9 clusters were randomized to 7 wedges, each lasting 42 days. For care teams in treatment wedges, palliative care specialists review identified patients, making recommendations to primary care providers when appropriate. Care teams in control wedges receive palliative care under the standard of care.

DISCUSSION

This pragmatic trial therefore integrates machine learning into clinical decision making, instead of simply reporting theoretical predictive performance. Such integration has the possibility to decrease time to palliative care, improving patient quality of life and symptom burden.

TRIAL REGISTRATION

Clinicaltrials.gov NCT04604457 , restrospectively registered 10/26/2020.

PROTOCOL

v0.5, dated 9/23/2020.

Top Ten Tips Palliative Care Clinicians Should Know About Leveraging the Electronic Health Record for Data Collection and Quality Improvement

As palliative care (PC) programs rapidly grow and expand across settings, the need to measure, improve, and standardize high-quality PC has also grown. The electronic health record (EHR) is a key component of these efforts as a central hub of care delivery and a repository of patient and system data. Deliberate efforts to leverage the EHR for PC quality improvement (QI) can help PC programs and health systems improve care for patients with serious illnesses. This article, written by clinicians with experience in QI, informatics, and clinical program development, provides practical tips and guidance on EHR strategies and tools for QI and quality measurement.

Development of Machine Learning Algorithms Incorporating Electronic Health Record Data, Patient-Reported Outcomes, or Both to Predict Mortality for Outpatients With Cancer

PURPOSE

Machine learning (ML) algorithms that incorporate routinely collected patient-reported outcomes (PROs) alongside electronic health record (EHR) variables may improve prediction of short-term mortality and facilitate earlier supportive and palliative care for patients with cancer.

METHODS

We trained and validated two-phase ML algorithms that incorporated standard PRO assessments alongside approximately 200 routinely collected EHR variables, among patients with medical oncology encounters at a tertiary academic oncology and a community oncology practice.

RESULTS

Among 12,350 patients, 5,870 (47.5%) completed PRO assessments. Compared with EHR- and PRO-only algorithms, the EHR + PRO model improved predictive performance in both tertiary oncology (EHR + PRO v EHR v PRO: area under the curve [AUC] 0.86 [0.85-0.87] v 0.82 [0.81-0.83] v 0.74 [0.74-0.74]) and community oncology (area under the curve 0.89 [0.88-0.90] v 0.86 [0.85-0.88] v 0.77 [0.76-0.79]) practices.

CONCLUSION

Routinely collected PROs contain added prognostic information not captured by an EHR-based ML mortality risk algorithm. Augmenting an EHR-based algorithm with PROs resulted in a more accurate and clinically relevant model, which can facilitate earlier and targeted supportive care for patients with cancer.

Who Gets (and Who Should Get) a Serious Illness Conversation in the Hospital? An Analysis of Readmission Risk Score in an Electronic Health Record

Serious Illness Conversations (SICs) explore patients' prognostic awareness, hopes, and worries, and can help establish priorities for their care during and after hospitalization. While identifying patients who benefit from an SIC remains a challenge, this task may be facilitated by use of validated prediction scores available in most commercial electronic health records (EHRs), such as Epic's Readmission Risk Score (RRS). We identified the RRS on admission for all hospital encounters from October 2018 to August 2019 and measured the area under the receiver operating characteristic (AUROC) curve to determine whether RRS could accurately discriminate post discharge 6-month mortality. For encounters with standardized SIC documentation matched in a 1:3 ratio to controls by sex and age (±5 years), we constructed a multivariable, paired logistic regression model and measured the odds of SIC documentation per every 10% absolute increase in RRS. RRS was predictive of 6-month mortality with acceptable discrimination (AUROC .71) and was significantly associated with SIC documentation (adjusted OR 1.42, 95% CI 1.24-1.63). An RRS >28% used to identify patients with post discharge 6-month mortality had a high specificity (89.0%) and negative predictive value (NPV) (97.0%), but low sensitivity (25.2%) and positive predictive value (PPV) (7.9%). RRS may serve as a practical EHR-based screen to exclude patients not requiring an SIC, thereby leaving a smaller cohort to be further evaluated for SIC needs using other validated tools and clinical assessment.

Palliative Care Exposure Relative to Predicted Risk of Six-Month Mortality in Hospitalized Adults

CONTEXT

The optimal strategy for implementing mortality-predicting algorithms to facilitate clinical care, prognostic discussions, and palliative care interventions remains unknown.

OBJECTIVES

To develop and validate a real-time predictive model for 180 day mortality using routinely available clinical and laboratory admission data and determine if palliative care exposure varies with predicted mortality risk.

METHODS

Adult admissions between October 1, 2013 and October.1, 2017 were included for the model derivation. A separate cohort was collected between January 1, 2018 and July 31, 2020 for validation. Patients were followed for 180 days from discharge, and logistic regression with selected variables was used to estimate patients' risk for mortality.

RESULTS

In the model derivation cohort, 7963 events of 180 day mortality (4.5% event rate) were observed. Median age was 53.0 (IQR 24.0-66.0) with 92,734 females (52.5%). Variables with strongest association with 180 day mortality included: Braden Score (OR 0.83; 95% CI 0.82-0.84); admission Do Not Resuscitate orders (OR 2.61; 95% CI 2.43-2.79); admission service and admission status. The model yielded excellent discriminatory ability in both the derivation (c-statistic 0.873; 95% CI 0.870-0.877; Brier score 0.04) and validation cohorts (c-statistic 0.844; 95% CI 0.840-0.847; Brier score 0.072). Inpatient palliative care consultations increased from 3% of minimal-risk encounters to 41% of high-risk encounters (P < 0.01).

CONCLUSION

We developed and temporally validated a predictive mortality model for adults from a large retrospective cohort, which helps quantify the potential need for palliative care referrals based on risk strata. Machine learning algorithms for mortality require clinical interpretation, and additional studies are needed to design patient-centered and risk-specific interventions.

Development and Structure of an Accurate Machine Learning Algorithm to Predict Inpatient Mortality and Hospice Outcomes in the Coronavirus Disease 2019 Era

BACKGROUND

The coronavirus disease 2019 (COVID-19) pandemic has challenged the accuracy and racial biases present in traditional mortality scores. An accurate prognostic model that can be applied to hospitalized patients irrespective of race or COVID-19 status may benefit patient care.

RESEARCH DESIGN

This cohort study utilized historical and ongoing electronic health record features to develop and validate a deep-learning model applied on the second day of admission predicting a composite outcome of in-hospital mortality, discharge to hospice, or death within 30 days of admission. Model features included patient demographics, diagnoses, procedures, inpatient medications, laboratory values, vital signs, and substance use history. Conventional performance metrics were assessed, and subgroup analysis was performed based on race, COVID-19 status, and intensive care unit admission.

SUBJECTS

A total of 35,521 patients hospitalized between April 2020 and October 2020 at a single health care system including a tertiary academic referral center and 9 community hospitals.

RESULTS

Of 35,521 patients, including 9831 non-White patients and 2020 COVID-19 patients, 2838 (8.0%) met the composite outcome. Patients who experienced the composite outcome were older (73 vs. 61 y old) with similar sex and race distributions between groups. The model achieved an area under the receiver operating characteristic curve of 0.89 (95% confidence interval: 0.88, 0.91) and an average positive predictive value of 0.46 (0.40, 0.52). Model performance did not differ significantly in White (0.89) and non-White (0.90) subgroups or when grouping by COVID-19 status and intensive care unit admission.

CONCLUSION

A deep-learning model using large-volume, structured electronic health record data can effectively predict short-term mortality or hospice outcomes on the second day of admission in the general inpatient population without significant racial bias.

A Comparison of Models Predicting One-Year Mortality at Time of Admission

CONTEXT

Hospitalization provides an opportunity to address end-of-life care (EoLC) preferences if patients at risk of death can be accurately identified while in the hospital. The modified Hospital One-Year Mortality Risk (mHOMR) uses demographic and admission data in a logistic regression algorithm to identify patients at risk of death one year from admission.

OBJECTIVES

This project sought to validate mHOMR and identify superior models.

METHODS

The mHOMR model was validated using historical data from an academic health system. Alternative logistic regression and random forest (RF) models were developed using the same variables. Receiver operating characteristic (ROC) and precision recall curves were developed, and sensitivity, specificity, and positive and negative predictive values were compared over a range of model thresholds.

RESULTS

The RF model demonstrated higher area under the ROC curve (0.950, 95% CI 0.947 - 0.954) as compared to the logistic regression models (0.818 [95% CI 0.812 - 0.825] and 0.841 [95% CI 0.836 - 0.847]). Area under the precision recall curve was higher with the random forest model compared to the logistic regression models (0.863 vs. 0.458 and 0.494, respectively). Across a range of thresholds, the RF model demonstrated superior sensitivity, equivalent specificity, and higher positive and negative predictive values.

CONCLUSION

A machine learning RF model, using common demographic and utilization data available on hospital admission, identified inpatients at risk of death more effectively than logistic regression models using the same variables. Machine learning models have promise for identifying admitted patients with elevated one-year mortality risk, increasing opportunities to prompt discussion of EoLC preferences.

Assessment of Clinical Palliative Care Trigger Status vs Actual Needs Among Critically Ill Patients and Their Family Members

IMPORTANCE

Palliative care consultations in intensive care units (ICUs) are increasingly prompted by clinical characteristics associated with mortality or resource utilization. However, it is not known whether these triggers reflect actual palliative care needs.

OBJECTIVE

To compare unmet needs by clinical palliative care trigger status (present vs absent).

DESIGN, SETTING, AND PARTICIPANTS

This prospective cohort study was conducted in 6 adult medical and surgical ICUs in academic and community hospitals in North Carolina between January 2019 and September 2020. Participants were consecutive patients receiving mechanical ventilation and their family members.

EXPOSURE

Presence of any of 9 common clinical palliative care triggers.

MAIN OUTCOMES AND MEASURES

The primary outcome was the Needs at the End-of-Life Screening Tool (NEST) score (range, 0-130, with higher scores reflecting greater need), which was completed after 3 days of ICU care. Trigger status performance in identifying serious need (NEST score ≥30) was assessed using sensitivity, specificity, positive and negative likelihood ratios, and C statistics.

RESULTS

Surveys were completed by 257 of 360 family members of patients (71.4% of the potentially eligible patient-family member dyads approached) with a median age of 54.0 years (IQR, 44-62 years); 197 family members (76.7%) were female, and 83 (32.3%) were Black. The median age of patients was 58.0 years (IQR, 46-68 years); 126 patients (49.0%) were female, and 88 (33.5%) were Black. There was no difference in median NEST score between participants with a trigger present (45%) and those with a trigger absent (55%) (21.0; IQR, 12.0-37.0 vs 22.5; IQR, 12.0-39.0; P = .52). Trigger presence was associated with poor sensitivity (45%; 95% CI, 34%-55%), specificity (55%; 95% CI, 48%-63%), positive likelihood ratio (1.0; 95% CI, 0.7-1.3), negative likelihood ratio (1.0; 95% CI, 0.8-1.2), and C statistic (0.50; 95% CI, 0.44-0.57).

CONCLUSIONS AND RELEVANCE

In this cohort study, clinical palliative care trigger status was not associated with palliative care needs and no better than chance at identifying the most serious needs, which raises questions about an increasingly common clinical practice. Focusing care delivery on directly measured needs may represent a more person-centered alternative.

Oncologist phenotypes and associations with response to a machine learning-based intervention to increase advance care planning: Secondary analysis of a randomized clinical trial

BACKGROUND

While health systems have implemented multifaceted interventions to improve physician and patient communication in serious illnesses such as cancer, clinicians vary in their response to these initiatives. In this secondary analysis of a randomized trial, we identified phenotypes of oncology clinicians based on practice pattern and demographic data, then evaluated associations between such phenotypes and response to a machine learning (ML)-based intervention to prompt earlier advance care planning (ACP) for patients with cancer.

METHODS AND FINDINGS

Between June and November 2019, we conducted a pragmatic randomized controlled trial testing the impact of text message prompts to 78 oncology clinicians at 9 oncology practices to perform ACP conversations among patients with cancer at high risk of 180-day mortality, identified using a ML prognostic algorithm. All practices began in the pre-intervention group, which received weekly emails about ACP performance only; practices were sequentially randomized to receive the intervention at 4-week intervals in a stepped-wedge design. We used latent profile analysis (LPA) to identify oncologist phenotypes based on 11 baseline demographic and practice pattern variables identified using EHR and internal administrative sources. Difference-in-differences analyses assessed associations between oncologist phenotype and the outcome of change in ACP conversation rate, before and during the intervention period. Primary analyses were adjusted for patients' sex, age, race, insurance status, marital status, and Charlson comorbidity index. The sample consisted of 2695 patients with a mean age of 64.9 years, of whom 72% were White, 20% were Black, and 52% were male. 78 oncology clinicians (42 oncologists, 36 advanced practice providers) were included. Three oncologist phenotypes were identified: Class 1 (n = 9) composed primarily of high-volume generalist oncologists, Class 2 (n = 5) comprised primarily of low-volume specialist oncologists; and 3) Class 3 (n = 28), composed primarily of high-volume specialist oncologists. Compared with class 1 and class 3, class 2 had lower mean clinic days per week (1.6 vs 2.5 [class 3] vs 4.4 [class 1]) a higher percentage of new patients per week (35% vs 21% vs 18%), higher baseline ACP rates (3.9% vs 1.6% vs 0.8%), and lower baseline rates of chemotherapy within 14 days of death (1.4% vs 6.5% vs 7.1%). Overall, ACP rates were 3.6% in the pre-intervention wedges and 15.2% in intervention wedges (11.6 percentage-point difference). Compared to class 3, oncologists in class 1 (adjusted percentage-point difference-in-differences 3.6, 95% CI 1.0 to 6.1, p = 0.006) and class 2 (adjusted percentage-point difference-in-differences 12.3, 95% confidence interval [CI] 4.3 to 20.3, p = 0.003) had greater response to the intervention.

CONCLUSIONS

Patient volume and time availability may be associated with oncologists' response to interventions to increase ACP. Future interventions to prompt ACP should prioritize making time available for such conversations between oncologists and their patients.

Expected clinical utility of automatable prediction models for improving palliative and end-of-life care outcomes: Toward routine decision analysis before implementation

Objective

The study sought to evaluate the expected clinical utility of automatable prediction models for increasing goals-of-care discussions (GOCDs) among hospitalized patients at the end of life (EOL).

Materials and Methods

We built a decision model from the perspective of clinicians who aim to increase GOCDs at the EOL using an automated alert system. The alternative strategies were 4 prediction models—3 random forest models and the Modified Hospital One-year Mortality Risk model—to generate alerts for patients at a high risk of 1-year mortality. They were trained on admissions from 2011 to 2016 (70 788 patients) and tested with admissions from 2017-2018 (16 490 patients). GOCDs occurring in usual care were measured with code status orders. We calculated the expected risk difference (beneficial outcomes with alerts minus beneficial outcomes without alerts among those at the EOL), the number needed to benefit (number of alerts needed to increase benefit over usual care by 1 outcome), and the net benefit (benefit minus cost) of each strategy.

Results

Models had a C-statistic between 0.79 and 0.86. A code status order occurred during 2599 of 3773 (69%) hospitalizations at the EOL. At a risk threshold corresponding to an alert prevalence of 10%, the expected risk difference ranged from 5.4% to 10.7% and the number needed to benefit ranged from 5.4 to 10.9 alerts. Using revealed preferences, only 2 models improved net benefit over usual care. A random forest model with diagnostic predictors had the highest expected value, including in sensitivity analyses.

Discussion

Prediction models with acceptable predictive validity differed meaningfully in their ability to improve over usual decision making.

Conclusions

An evaluation of clinical utility, such as by using decision curve analysis, is recommended after validating a prediction model because metrics of model predictiveness, such as the C-statistic, are not informative of clinical value.

Improving time to palliative care review with predictive modeling in an inpatient adult population: study protocol for a stepped-wedge, pragmatic randomized controlled trial

Background

Palliative care is a medical specialty centered on improving the quality of life (QOL) of patients with complex or life-threatening illnesses. The need for palliative care is increasing and with that the rigorous testing of triage tools that can be used quickly and reliably to identify patients that may benefit from palliative care.

Methods

To that aim, we will conduct a two-armed stepped-wedge cluster randomized trial rolled out to two inpatient hospitals to evaluate whether a machine learning algorithm accurately identifies patients who may benefit from a comprehensive review by a palliative care specialist and decreases time to receiving a palliative care consult in hospital. This is a single-center study which will be conducted from August 2019 to November 2020 at Saint Mary’s Hospital & Methodist Hospital both within Mayo Clinic Rochester in Minnesota. Clusters will be nursing units which will be chosen to be a mix of complex patients from Cardiology, Critical Care, and Oncology and had previously established relationships with palliative medicine. The stepped wedge design will have 12 units allocated to a design matrix of 5 treatment wedges. Each wedge will last 75 days resulting in a study period of 12 months of recruitment unless otherwise specified. Data will be analyzed with Bayesian hierarchical models with credible intervals denoting statistical significance.

Discussion

This intervention offers a pragmatic approach to delivering specialty palliative care to hospital patients in need using machine learning, thereby leading to high value care and improved outcomes. It is not enough for AI to be utilized by simply publishing research showing predictive performance; clinical trials demonstrating better outcomes are critically needed. Furthermore, the deployment of an AI algorithm is a complex process that requires multiple teams with varying skill sets. To evaluate a deployed AI, a pragmatic clinical trial can accommodate the difficulties of clinical practice while retaining scientific rigor.

Trial registration

ClinicalTrials.gov NCT03976297. Registered on 6 June 2019, prior to trial start.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13063-021-05546-5.

Evaluation of automated specialty palliative care in the intensive care unit: A retrospective cohort study

Introduction

Automated specialty palliative care consultation (SPC) has been proposed as an intervention to improve patient-centered care in the intensive care unit (ICU). Existing automated SPC trigger criteria are designed to identify patients at highest risk of in-hospital death. We sought to evaluate common mortality-based SPC triggers and determine whether these triggers reflect actual use of SPC consultation. We additionally aimed to characterize the population of patients who receive SPC without meeting mortality-based triggers.

Methods

We conducted a retrospective cohort study of all adult ICU admissions from 2012–2017 at an academic medical center with five subspecialty ICUs to determine the sensitivity and specificity of the five most common SPC triggers for predicting receipt of SPC. Among ICU admissions receiving SPC, we assessed differences in patients who met any SPC trigger compared to those who met none.

Results

Of 48,744 eligible admissions, 1,965 (4.03%) received SPC; 979 (49.82%) of consultations met at least 1 trigger. The sensitivity and specificity for any trigger predicting SPC was 49.82% and 79.61%, respectively. Patients who met no triggers but received SPC were younger (62.71 years vs 66.58 years, mean difference (MD) 3.87 years (95% confidence interval (CI) 2.44–5.30) p<0.001), had longer ICU length of stay (11.43 days vs 8.42 days, MD -3.01 days (95% CI -4.30 –-1.72) p<0.001), and had a lower rate of in-hospital death (48.68% vs 58.12%, p<0.001).

Conclusion

Mortality-based triggers for specialty palliative care poorly reflect actual use of SPC in the ICU. Reliance on such triggers may unintentionally overlook an important population of patients with clinician-identified palliative care needs.

Development and Validation of a 30-Day In-hospital Mortality Model Among Seriously Ill Transferred Patients: a Retrospective Cohort Study

BACKGROUND

Predicting the risk of in-hospital mortality on admission is challenging but essential for risk stratification of patient outcomes and designing an appropriate plan-of-care, especially among transferred patients.

OBJECTIVE

Develop a model that uses administrative and clinical data within 24 h of transfer to predict 30-day in-hospital mortality at an Academic Health Center (AHC).

DESIGN

Retrospective cohort study. We used 30 putative variables in a multiple logistic regression model in the full data set (n = 10,389) to identify 20 candidate variables obtained from the electronic medical record (EMR) within 24 h of admission that were associated with 30-day in-hospital mortality (p < 0.05). These 20 variables were tested using multiple logistic regression and area under the curve (AUC)-receiver operating characteristics (ROC) analysis to identify an optimal risk threshold score in a randomly split derivation sample (n = 5194) which was then examined in the validation sample (n = 5195).

PARTICIPANTS

Ten thousand three hundred eighty-nine patients greater than 18 years transferred to the Indiana University (IU)-Adult Academic Health Center (AHC) between 1/1/2016 and 12/31/2017.

MAIN MEASURES

Sensitivity, specificity, positive predictive value, C-statistic, and risk threshold score of the model.

KEY RESULTS

The final model was strongly discriminative (C-statistic = 0.90) and had a good fit (Hosmer-Lemeshow goodness-of-fit test [X² (8) =6.26, p = 0.62]). The positive predictive value for 30-day in-hospital death was 68%; AUC-ROC was 0.90 (95% confidence interval 0.89-0.92, p < 0.0001). We identified a risk threshold score of -2.19 that had a maximum sensitivity (79.87%) and specificity (85.24%) in the derivation and validation sample (sensitivity: 75.00%, specificity: 85.71%). In the validation sample, 34.40% (354/1029) of the patients above this threshold died compared to only 2.83% (118/4166) deaths below this threshold.

CONCLUSION

This model can use EMR and administrative data within 24 h of transfer to predict the risk of 30-day in-hospital mortality with reasonable accuracy among seriously ill transferred patients.

Hospital-free Days: A Pragmatic and Patient-centered Outcome for Trials Among Critically and Seriously Ill Patients

Hospital-free days (HFDs), alternatively known as "days alive and outside the hospital," is increasingly used as a primary or secondary outcome in randomized trials among critically and seriously ill patients. This novel outcome measure addresses an existing gap in the availability of patient-centered, reliably obtained outcome measures among patients with acute respiratory failure, advanced lung diseases, lung transplantation, and other serious and critical illnesses. Traditional outcomes such as mortality, organ-failure-free days, and longitudinal patient-reported measures have distinct drawbacks that limit their suitability as endpoints in trials of patients with serious illness, particularly those trials with pragmatic designs. By contrast, HFDs provides a summary measure of important health events and is easily calculated from administrative or electronic health record data, thereby balancing the goals of patient-centeredness and pragmatic measurement. However, before HFDs can be widely adopted as an endpoint in trials of patients with respiratory and critical illnesses, several questions must be addressed regarding the optimal definition, measurement, and analysis of HFDs. In this perspective, we outline important considerations relevant to the use of HFDs as a trial endpoint and suggest directions for further development of the measure.

AI-based clinical decision-making systems in palliative medicine: ethical challenges

BACKGROUND

Improving palliative care (PC) is demanding due to the increase in people with PC needs over the next few years. An early identification of PC needs is fundamental in the care approach: it provides effective patient-centred care and could improve outcomes such as patient quality of life, reduction of the overall length of hospitalisation, survival rate prolongation, the satisfaction of both the patients and caregivers and cost-effectiveness.

METHODS

We reviewed literature with the objective of identifying and discussing the most important ethical challenges related to the implementation of AI-based data processing services in PC and advance care planning.

RESULTS

AI-based mortality predictions can signal the need for patients to obtain access to personalised communication or palliative care consultation, but they should not be used as a unique parameter to activate early PC and initiate an ACP. A number of factors must be included in the ethical decision-making process related to initiation of ACP conversations, among which are autonomy and quality of life, the risk of worsening healthcare status, the commitment by caregivers, the patients' psychosocial and spiritual distress and their wishes to initiate EOL discussions CONCLUSIONS: Despite the integration of artificial intelligence (AI)-based services into routine healthcare practice could have a positive effect of promoting early activation of ACP by means of a timely identification of PC needs, from an ethical point of view, the provision of these automated techniques raises a number of critical issues that deserve further exploration.

Association between Palliative Care and End-of-Life Resource Use for Older Adults Hospitalized with Septic Shock

Rationale: The care of critically ill patients often involves complex discussions surrounding prognosis, goals, and end-of-life decision-making. Yet, physician and hospital practice patterns, rather than patient goals, remain a major determinant of the intensity of end-of-life care. For critically ill patients, palliative care may help promote treatments that are concordant with patients' goals, while minimizing the use of invasive and costly intensive care unit resources that may not be consistent with those goals.Objectives: To determine whether inpatient palliative care, delivered by specialist consultants or a primary medical team, is associated with reduced hospital length of stay and costs for older adults with septic shock at the end of life.Methods: This was a retrospective cohort using the National Inpatient Sample from 2013 to 2014, examining patients aged ≥65 years with septic shock who died during their hospitalization. The exposure of interest was inpatient palliative care encounter, including either generalist- or specialist-delivered palliative care. Outcomes were hospital length of stay, total cost for the hospitalization, and daily hospital cost. Patient and hospital-level confounders were used to derive inverse probability of treatment weights and estimate the association between palliative care and outcomes in a generalized linear model.Results: We studied 45,868 patients who died with a diagnosis of septic shock; 15,370 of these patients had a palliative care encounter. After inverse probability of treatment weighting, there were no appreciable differences between the population characteristics. Palliative care was associated with a shorter adjusted mean hospital length of stay (12.0 vs. 13.1 d; difference, -1.1 d; 95% confidence interval [CI], -1.4 to -0.9; P < 0.001), lower total hospital costs (69,700 vs. 76,800 U.S. dollars [USD]; difference, -7,100 USD; 95% CI, -8.5 to -5.2 thousand USD; P < 0.001), and lower daily hospital cost (5,900 vs. 6,200 USD; difference, -310 USD per day; 95% CI, -420 to -200 USD; P < 0.001) when compared with no palliative care.Conclusions: In a nationally representative sample of adults who died during a hospitalization with septic shock, receipt of palliative care was associated with shorter length of stay and lower total and daily hospital costs. This finding was robust to adjustment for patient- and hospital-level confounders, though unmeasured confounders still could be affecting these findings.

What is needed to mainstream artificial intelligence in health care?

Artificial intelligence (AI) has become a mainstream technology in many industries, but not yet in health care. Although basic research and commercial investment are burgeoning across various clinical disciplines, AI remains relatively non-existent in most healthcare organisations. This is despite hundreds of AI applications having passed proof-of-concept phase, and scores receiving regulatory approval overseas. AI has considerable potential to optimise multiple care processes, maximise workforce capacity, reduce waste and costs, and improve patient outcomes. The current obstacles to wider AI adoption in health care and the pre-requisites for its successful development, evaluation and implementation need to be defined.

mHOMR: a prospective observational study of an automated mortality prediction model to identify patients with unmet palliative needs

OBJECTIVE

Identification of patients with shortened life expectancy is a major obstacle to delivering palliative/end-of-life care. We previously developed the modified Hospitalised-patient One-year Mortality Risk (mHOMR) model for the automated identification of patients with an elevated 1-year mortality risk. Our goal was to investigate whether patients identified by mHOMR at high risk for mortality in the next year also have unmet palliative needs.

METHOD

We conducted a prospective observational study at two quaternary healthcare facilities in Toronto, Canada, with patients admitted to general internal medicine service and identified by mHOMR to have an expected 1-year mortality risk of 10% or more. We measured patients' unmet palliative needs-a severe uncontrolled symptom on the Edmonton Symptom Assessment Scale or readiness to engage in advance care planning (ACP) based on Sudore's ACP Engagement Survey.

RESULTS

Of 518 patients identified by mHOMR, 403 (78%) patients consented to participate; 87% of those had either a severe uncontrolled symptom or readiness to engage in ACP, and 44% had both. Patients represented frailty (38%), cancer (28%) and organ failure (28%) trajectories were admitted for a median of 6 days, and 94% survived to discharge.

CONCLUSIONS

A large majority of hospitalised patients identified by mHOMR have unmet palliative needs, regardless of disease, and are identified early enough in their disease course that they may benefit from a palliative approach to their care. Adoption of such a model could improve the timely introduction of a palliative approach for patients, especially those with non-cancer illness.

Observational study of agreement between attending and trainee physicians on the surprise question: "Would you be surprised if this patient died in the next 12 months?"

BACKGROUND

Optimal end-of-life care requires identifying patients that are near the end of life. The extent to which attending physicians and trainee physicians agree on the prognoses of their patients is unknown. We investigated agreement between attending and trainee physician on the surprise question: "Would you be surprised if this patient died in the next 12 months?", a question intended to assess mortality risk and unmet palliative care needs.

METHODS

This was a multicentre prospective cohort study of general internal medicine patients at 7 tertiary academic hospitals in Ontario, Canada. General internal medicine attending and senior trainee physician dyads were asked the surprise question for each of the patients for whom they were responsible. Surprise question response agreement was quantified by Cohen's kappa using Bayesian multilevel modeling to account for clustering by physician dyad. Mortality was recorded at 12 months.

RESULTS

Surprise question responses encompassed 546 patients from 30 attending-trainee physician dyads on academic general internal medicine teams at 7 tertiary academic hospitals in Ontario, Canada. Patients had median age 75 years (IQR 60-85), 260 (48%) were female, and 138 (25%) were dependent for some or all activities of daily living. Trainee and attending physician responses agreed in 406 (75%) patients with adjusted Cohen's kappa of 0.54 (95% credible interval 0.41 to 0.66). Vital status was confirmed for 417 (76%) patients of whom 160 (38% of 417) had died. Using a response of "No" to predict 12-month mortality had positive likelihood ratios of 1.84 (95% CrI 1.55 to 2.22, trainee physicians) and 1.51 (95% CrI 1.30 to 1.72, attending physicians), and negative likelihood ratios of 0.31 (95% CrI 0.17 to 0.48, trainee physicians) and 0.25 (95% CrI 0.10 to 0.46, attending physicians).

CONCLUSION

Trainee and attending physician responses to the surprise question agreed in 54% of cases after correcting for chance agreement. Physicians had similar discriminative accuracy; both groups had better accuracy predicting which patients would survive as opposed to which patients would die. Different opinions of a patient's prognosis may contribute to confusion for patients and missed opportunities for engagement with palliative care services.

A framework for making predictive models useful in practice

Objective

To analyze the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality.

Materials and Methods

We built a predictive model of 12-month mortality using electronic health record data and evaluated the impact of healthcare delivery factors on the net benefit of triggering an ACP workflow based on the models’ predictions. Factors included nonclinical reasons that make ACP inappropriate: limited capacity for ACP, inability to follow up due to patient discharge, and availability of an outpatient workflow to follow up on missed cases. We also quantified the relative benefits of increasing capacity for inpatient ACP versus outpatient ACP.

Results

Work capacity constraints and discharge timing can significantly reduce the net benefit of triggering the ACP workflow based on a model’s predictions. However, the reduction can be mitigated by creating an outpatient ACP workflow. Given limited resources to either add capacity for inpatient ACP versus developing outpatient ACP capability, the latter is likely to provide more benefit to patient care.

Discussion

The benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. We provide a framework for quantifying the impact of healthcare delivery factors and work capacity constraints on achieved benefit.

Conclusion

An analysis of the sensitivity of the net benefit realized by a predictive model triggered clinical workflow to various healthcare delivery factors is necessary for making predictive models useful in practice.

Impact of palliative care consult service in inpatient hospital setting: a systematic literature review

OBJECTIVES

Despite a number of studies on effectiveness of palliative care, there is a lack of complex updated review of the impact of in-hospital palliative care consult service. The objective is to update information on the impact of palliative care consult service in inpatient hospital setting.

METHODS

This study was a systematic literature review, following the standard protocols (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, Joanna Briggs Institute tools) to ensure the transparent and robust review procedure. The effect of palliative care consult service was classified as being associated with improvement, no difference, deterioration or mixed results in specific outcomes. PubMed, Scopus, Academic Search Ultimate and SocINDEX were systematically searched up to February 2020. Studies were included if they focused on the impact of palliative care consult service caring for adult palliative care patients and their families in inpatient hospital setting.

RESULTS

After removing duplicates, 959 citations were screened of which 49 full-text articles were retained. A total of 28 different outcome variables were extracted. 18 of them showed positive effects within patient, family, staff and healthcare system domains. No difference was observed in patient survival and depression. Inconclusive results represented patient social support and staff satisfaction with care.

CONCLUSIONS

Palliative care consult service has a number of positive effects for patients, families, staff and healthcare system. More research is needed on factors such as patient spiritual well-being, social support, performance, family understanding of patient diagnosis or staff stress.

How, when, and why individuals with stage IV cancer seen in an outpatient setting are referred to palliative care: a mixed methods study

PURPOSE

Early palliative care (PC) for individuals with advanced cancer improves patient and family outcomes and experience. However, it is unknown when, why, and how in an outpatient setting individuals with stage IV cancer are referred to PC.

METHODS

At a large multi-specialty group in the USA with outpatient PC implemented beginning in 2011, clinical records were used to identify adults diagnosed with stage IV cancer after January 1, 2012 and deceased by December 31, 2017 and their PC referrals and hospice use. In-depth interviews were also conducted with 25 members of medical oncology, gynecological oncology, and PC teams and thematically analyzed.

RESULTS

A total of 705 individuals were diagnosed and died between 2012 and 2017: of these, 332 (47%) were referred to PC, with 48.5% referred early (within 60 days of diagnosis). Among referred patients, 79% received hospice care, versus 55% among patients not referred. Oncologists varied dramatically in their rates of referral to PC. Interviews revealed four referral pathways: early referrals, referrals without active anti-cancer treatment, problem-based referrals, and late referrals (when stopping treatment). Participants described PC's benefits as enhancing pain/symptom management, advance care planning, transitions to hospice, end-of-life experiences, a larger team, and more flexible patient care. Challenges reported included variation in oncologist practices, patient fears and misconceptions, and access to PC teams.

CONCLUSION

We found high rates of use and appreciation of PC. However, interviews revealed that exclusively focusing on rates of referrals may obscure how referrals vary in timing, reason for referral, and usefulness to patients, families, and clinical teams.

Integrating machine-generated mortality estimates and behavioral nudges to promote serious illness conversations for cancer patients: Design and methods for a stepped-wedge cluster randomized controlled trial

INTRODUCTION

Patients with cancer often receive care that is not aligned with their personal values and goals. Serious illness conversations (SICs) between clinicians and patients can help increase a patient's understanding of their prognosis, goals and values.

METHODS AND ANALYSIS

In this study, we describe the design of a stepped-wedge cluster randomized trial to evaluate the impact of an intervention that employs machine learning-based prognostic algorithms and behavioral nudges to prompt oncologists to have SICs with patients at high risk of short-term mortality. Data are collected on documented SICs, documented advance care planning discussions, and end-of-life care utilization (emergency room and inpatient admissions, chemotherapy and hospice utilization) for patients of all enrolled clinicians.

CONCLUSION

This trial represents a novel application of machine-generated mortality predictions combined with behavioral nudges in the routine care of outpatients with cancer. Findings from the trial may inform strategies to encourage early serious illness conversations and the application of mortality risk predictions in clinical settings.

TRIAL REGISTRATION

Clinicaltrials.gov Identifier: NCT03984773.

Machine Learning Approaches to Predict 6-Month Mortality Among Patients With Cancer

Importance

Machine learning algorithms could identify patients with cancer who are at risk of short-term mortality. However, it is unclear how different machine learning algorithms compare and whether they could prompt clinicians to have timely conversations about treatment and end-of-life preferences.

Objectives

To develop, validate, and compare machine learning algorithms that use structured electronic health record data before a clinic visit to predict mortality among patients with cancer.

Design, Setting, and Participants

Cohort study of 26 525 adult patients who had outpatient oncology or hematology/oncology encounters at a large academic cancer center and 10 affiliated community practices between February 1, 2016, and July 1, 2016. Patients were not required to receive cancer-directed treatment. Patients were observed for up to 500 days after the encounter. Data analysis took place between October 1, 2018, and September 1, 2019.

Exposures

Logistic regression, gradient boosting, and random forest algorithms.

Main Outcomes and Measures

Primary outcome was 180-day mortality from the index encounter; secondary outcome was 500-day mortality from the index encounter.

Results

Among 26 525 patients in the analysis, 1065 (4.0%) died within 180 days of the index encounter. Among those who died, the mean age was 67.3 (95% CI, 66.5-68.0) years, and 500 (47.0%) were women. Among those who were alive at 180 days, the mean age was 61.3 (95% CI, 61.1-61.5) years, and 15 922 (62.5%) were women. The population was randomly partitioned into training (18 567 [70.0%]) and validation (7958 [30.0%]) cohorts at the patient level, and a randomly selected encounter was included in either the training or validation set. At a prespecified alert rate of 0.02, positive predictive values were higher for the random forest (51.3%) and gradient boosting (49.4%) algorithms compared with the logistic regression algorithm (44.7%). There was no significant difference in discrimination among the random forest (area under the receiver operating characteristic curve [AUC], 0.88; 95% CI, 0.86-0.89), gradient boosting (AUC, 0.87; 95% CI, 0.85-0.89), and logistic regression (AUC, 0.86; 95% CI, 0.84-0.88) models (P for comparison = .02). In the random forest model, observed 180-day mortality was 51.3% (95% CI, 43.6%-58.8%) in the high-risk group vs 3.4% (95% CI, 3.0%-3.8%) in the low-risk group; at 500 days, observed mortality was 64.4% (95% CI, 56.7%-71.4%) in the high-risk group and 7.6% (7.0%-8.2%) in the low-risk group. In a survey of 15 oncology clinicians with a 52.1% response rate, 100 of 171 patients (58.8%) who had been flagged as having high risk by the gradient boosting algorithm were deemed appropriate for a conversation about treatment and end-of-life preferences in the upcoming week.

Conclusions and Relevance

In this cohort study, machine learning algorithms based on structured electronic health record data accurately identified patients with cancer at risk of short-term mortality. When the gradient boosting algorithm was applied in real time, clinicians believed that most patients who had been identified as having high risk were appropriate for a timely conversation about treatment and end-of-life preferences.