Effect of Integrating Machine Learning Mortality Estimates With Behavioral Nudges to Clinicians on Serious Illness Conversations Among Patients With Cancer: A Stepped-Wedge Cluster Randomized Clinical Trial

Applying human-centered design to adapt the Jumpstart Guide for goals-of-care discussions in persons living with dementia

BACKGROUND

Goals-of-care discussions (GOCD) are uncommon in persons living with dementia (PLWD) despite the likelihood of eventual loss of decisional capacity in the population. The Jumpstart Guide, an existing serious illness communication priming tool, can improve GOCD in certain populations, but has not previously been adapted for use among PLWD and their caregivers.

METHODS

Using human-centered design (HCD), we adapted the Jumpstart Guide for use with PLWD and their caregivers. We conducted qualitative interviews with clinicians and caregivers of PLWD. Six team members conducted qualitative rapid analysis of interviews leading to the development of summary templates and integrative matrices. Four iterations of the Jumpstart Guide led to the final version.

RESULTS

Thirteen clinicians and 11 caregivers were interviewed. Interviews provided key insights into the unique barriers PLWD and their caregivers face during GOCD, including discomfort with accepting a dementia diagnosis and concern with using "serious illness" to describe dementia, as is commonly done in palliative care. Clinicians described differences in GOCD with PLWD compared to other serious illnesses, and the challenge of getting patients and families to think about future health states. Interviews led to Jumpstart Guide adaptations in the following domains: (1) format and structure, (2) content, and (3) specific language. Suggested changes included prioritizing naming a decision-maker, changing conversation prompts to improve accessibility and understandability, ensuring the Jumpstart Guide could be used with patients as well as their caregivers, and altering language to avoid references to "serious illness" and "abilities."

CONCLUSION

Using HCD yielded valuable insights from clinicians and caregivers about the unique barriers to conducting GOCD among PLWD and their caregivers. These insights were used to adapt the Jumpstart Guide for use with PLWD and their caregivers, which is currently being tested in a pragmatic randomized controlled trial in outpatient clinics.

Fairness in Predicting Cancer Mortality Across Racial Subgroups

Importance

Machine learning has potential to transform cancer care by helping clinicians prioritize patients for serious illness conversations. However, models need to be evaluated for unequal performance across racial groups (ie, racial bias) so that existing racial disparities are not exacerbated.

Objective

To evaluate whether racial bias exists in a predictive machine learning model that identifies 180-day cancer mortality risk among patients with solid malignant tumors.

Design, Setting, and Participants

In this cohort study, a machine learning model to predict cancer mortality for patients aged 21 years or older diagnosed with cancer between January 2016 and December 2021 was developed with a random forest algorithm using retrospective data from the Mount Sinai Health System cancer registry, Social Security Death Index, and electronic health records up to the date when databases were accessed for cohort extraction (February 2022).

Exposure

Race category.

Main Outcomes and Measures

The primary outcomes were model discriminatory performance (area under the receiver operating characteristic curve [AUROC], F1 score) among each race category (Asian, Black, Native American, White, and other or unknown) and fairness metrics (equal opportunity, equalized odds, and disparate impact) among each pairwise comparison of race categories. True-positive rate ratios represented equal opportunity; both true-positive and false-positive rate ratios, equalized odds; and the percentage of predictive positive rate ratios, disparate impact. All metrics were estimated as a proportion or ratio, with variability captured through 95% CIs. The prespecified criterion for the model's clinical use was a threshold of at least 80% for fairness metrics across different racial groups to ensure the model's prediction would not be biased against any specific race.

Results

The test validation dataset included 43 274 patients with balanced demographics. Mean (SD) age was 64.09 (14.26) years, with 49.6% older than 65 years. A total of 53.3% were female; 9.5%, Asian; 18.9%, Black; 0.1%, Native American; 52.2%, White; and 19.2%, other or unknown race; 0.1% had missing race data. A total of 88.9% of patients were alive, and 11.1% were dead. The AUROCs, F1 scores, and fairness metrics maintained reasonable concordance among the racial subgroups: the AUROCs ranged from 0.75 (95% CI, 0.72-0.78) for Asian patients and 0.75 (95% CI, 0.73-0.77) for Black patients to 0.77 (95% CI, 0.75-0.79) for patients with other or unknown race; F1 scores, from 0.32 (95% CI, 0.32-0.33) for White patients to 0.40 (95% CI, 0.39-0.42) for Black patients; equal opportunity ratios, from 0.96 (95% CI, 0.95-0.98) for Black patients compared with White patients to 1.02 (95% CI, 1.00-1.04) for Black patients compared with patients with other or unknown race; equalized odds ratios, from 0.87 (95% CI, 0.85-0.92) for Black patients compared with White patients to 1.16 (1.10-1.21) for Black patients compared with patients with other or unknown race; and disparate impact ratios, from 0.86 (95% CI, 0.82-0.89) for Black patients compared with White patients to 1.17 (95% CI, 1.12-1.22) for Black patients compared with patients with other or unknown race.

Conclusions and Relevance

In this cohort study, the lack of significant variation in performance or fairness metrics indicated an absence of racial bias, suggesting that the model fairly identified cancer mortality risk across racial groups. It remains essential to consistently review the model's application in clinical settings to ensure equitable patient care.

Clinician- and Patient-Directed Communication Strategies for Patients With Cancer at High Mortality Risk: A Cluster Randomized Trial

Importance

Serious illness conversations (SICs) that elicit patients' values, goals, and care preferences reduce anxiety and depression and improve quality of life, but occur infrequently for patients with cancer. Behavioral economic implementation strategies (nudges) directed at clinicians and/or patients may increase SIC completion.

Objective

To test the independent and combined effects of clinician and patient nudges on SIC completion.

Design, Setting, and Participants

A 2 × 2 factorial, cluster randomized trial was conducted from September 7, 2021, to March 11, 2022, at oncology clinics across 4 hospitals and 6 community sites within a large academic health system in Pennsylvania and New Jersey among 163 medical and gynecologic oncology clinicians and 4450 patients with cancer at high risk of mortality (≥10% risk of 180-day mortality).

Interventions

Clinician clusters and patients were independently randomized to receive usual care vs nudges, resulting in 4 arms: (1) active control, operating for 2 years prior to trial start, consisting of clinician text message reminders to complete SICs for patients at high mortality risk; (2) clinician nudge only, consisting of active control plus weekly peer comparisons of clinician-level SIC completion rates; (3) patient nudge only, consisting of active control plus a preclinic electronic communication designed to prime patients for SICs; and (4) combined clinician and patient nudges.

Main Outcomes and Measures

The primary outcome was a documented SIC in the electronic health record within 6 months of a participant's first clinic visit after randomization. Analysis was performed on an intent-to-treat basis at the patient level.

Results

The study accrued 4450 patients (median age, 67 years [IQR, 59-75 years]; 2352 women [52.9%]) seen by 163 clinicians, randomized to active control (n = 1004), clinician nudge (n = 1179), patient nudge (n = 997), or combined nudges (n = 1270). Overall patient-level rates of 6-month SIC completion were 11.2% for the active control arm (112 of 1004), 11.5% for the clinician nudge arm (136 of 1179), 11.5% for the patient nudge arm (115 of 997), and 14.1% for the combined nudge arm (179 of 1270). Compared with active control, the combined nudges were associated with an increase in SIC rates (ratio of hazard ratios [rHR], 1.55 [95% CI, 1.00-2.40]; P = .049), whereas the clinician nudge (HR, 0.95 [95% CI, 0.64-1.41; P = .79) and patient nudge (HR, 0.99 [95% CI, 0.73-1.33]; P = .93) were not.

Conclusions and Relevance

In this cluster randomized trial, nudges combining clinician peer comparisons with patient priming questionnaires were associated with a marginal increase in documented SICs compared with an active control. Combining clinician- and patient-directed nudges may help to promote SICs in routine cancer care.

Trial Registration

ClinicalTrials.gov Identifier: NCT04867850.

Interventions aiming to improve advance care planning uptake in oncology: a scoping review of recent randomized controlled trials

PURPOSE OF REVIEW

The evidence regarding the effectiveness of interventions aiming to improve ACP uptake appears mixed. The purpose of this scoping review is to summarize findings from recent RCTs assessing the efficacy of interventions designed to increase ACP uptake in cancer care.

RECENT FINDINGS

The PubMed database was searched for studies published between 2021 and 2023. Ten RCTs were identified that evaluated a wide range of interventions aimed at encouraging or facilitating ACP communication among patients with advanced cancer, caregivers, and HCPs. Three RCTs developed ACP facilitation tools for patients and sometimes caregivers. Six RCTs involved integrating trained ACP facilitators into patient care, and one RCT used nudges to encourage oncologists trained in ACP discussions to initiate discussions with patients at highest risk of short-term mortality. Most of the reviewed interventions were based on previously tested intervention approaches or tools.

SUMMARY

The key effects identified in this review are the facilitation of ACP discussions among patients, caregivers, and HCPs, and the documentation of patient preferences in the medical record. The direct involvement of facilitators in ACP document drafting and inclusion in patient files, as well as interventions prompting end-of-life discussions, contributed to this effect.

Coaches Activating, Reaching, and Engaging Patients to Engage in Advance Care Planning: A Randomized Clinical Trial

Importance

Advance care planning (ACP) remains low among patients with advanced cancer. Multilevel interventions compared with clinician-level interventions may be more effective in improving ACP.

Objective

To evaluate whether a multilevel intervention could improve clinician-documented ACP compared with a clinician-level intervention alone.

Design, Setting, and Participants

This randomized clinical trial, performed from September 12, 2019, through May 12, 2021, included adults with advanced genitourinary cancers at an academic, tertiary hospital. Data analysis was performed by intention to treat from May 1 to August 10, 2023.

Intervention

Participants were randomized 1:1 to a 6-month patient-level lay health worker structured ACP education along with a clinician-level intervention composed of 3-hour ACP training and integration of a structured electronic health record documentation template (intervention group) or to the clinician-level intervention alone (control group).

Main Outcome and Measures

The primary outcome was ACP documentation in the electronic health record by the oncology clinician within 12 months after randomization. Secondary, exploratory outcomes included shared decision-making, palliative care use, hospice use, emergency department visits, and hospitalizations within 12 months after randomization.

Results

Among 402 participants enrolled in the study, median age was 71 years (range, 21-102 years); 361 (89.8%) identified as male. More intervention group participants had oncology clinician-documented ACP than control group participants (82 [37.8%] vs 40 [21.6%]; odds ratio [OR], 2.29; 95% CI, 1.44-3.64). At 12-month follow-up, more intervention than control group participants had palliative care (72 [33.2%] vs 25 [13.5%]; OR, 3.18; 95% CI, 1.91-5.28) and hospice use (49 [22.6%] vs 19 [10.3%]; OR, 2.54; 95% CI, 1.44-4.51). There were no differences in the proportion of participants between groups with an emergency department visit (65 [30.0%] vs 61 [33.0%]; OR, 0.87; 95% CI, 0.57-1.33) or hospitalization (89 [41.0%] vs 85 [46.0%]; OR, 0.82; 95% CI, 0.55-1.22). Intervention group participants had fewer hospitalizations than control group participants (mean [SD] number of hospitalizations per year, 0.87 [1.60] vs 1.04 [1.77]) and a lower risk of hospitalization (incidence rate ratio, 0.80; 95% CI, 0.65-0.98).

Conclusions and Relevance

In this randomized clinical trial, a multilevel intervention improved oncology clinician-documented ACP compared with a clinician-level intervention alone for patients with genitourinary cancer. The intervention is one approach to effectively increase ACP among patients with cancer.

Trial Registration

ClinicalTrials.gov Identifier: NCT03856463.

Uses and limitations of artificial intelligence for oncology

Modern artificial intelligence (AI) tools built on high-dimensional patient data are reshaping oncology care, helping to improve goal-concordant care, decrease cancer mortality rates, and increase workflow efficiency and scope of care. However, data-related concerns and human biases that seep into algorithms during development and post-deployment phases affect performance in real-world settings, limiting the utility and safety of AI technology in oncology clinics. To this end, the authors review the current potential and limitations of predictive AI for cancer diagnosis and prognostication as well as of generative AI, specifically modern chatbots, which interfaces with patients and clinicians. They conclude the review with a discussion on ongoing challenges and regulatory opportunities in the field.

Equity in Using Artificial Intelligence Mortality Predictions to Target Goals of Care Documentation

BACKGROUND

Artificial intelligence (AI) algorithms are increasingly used to target patients with elevated mortality risk scores for goals-of-care (GOC) conversations.

OBJECTIVE

To evaluate the association between the presence or absence of AI-generated mortality risk scores with GOC documentation.

DESIGN

Retrospective cross-sectional study at one large academic medical center between July 2021 and December 2022.

PARTICIPANTS

Hospitalized adult patients with AI-defined Serious Illness Risk Indicator (SIRI) scores indicating > 30% 90-day mortality risk (defined as "elevated" SIRI) or no SIRI scores due to insufficient data.

INTERVENTION

A targeted intervention to increase GOC documentation for patients with AI-generated scores predicting elevated risk of mortality.

MAIN MEASURES

Odds ratios comparing GOC documentation for patients with elevated or no SIRI scores with similar severity of illness using propensity score matching and risk-adjusted mixed-effects logistic regression.

KEY RESULTS

Among 13,710 patients with elevated (n = 3643, 27%) or no (n = 10,067, 73%) SIRI scores, the median age was 64 years (SD 18). Twenty-five percent were non-White, 18% had Medicaid, 43% were admitted to an intensive care unit, and 11% died during admission. Patients lacking SIRI scores were more likely to be younger (median 60 vs. 72 years, p < 0.0001), be non-White (29% vs. 13%, p < 0.0001), and have Medicaid (22% vs. 9%, p < 0.0001). Patients with elevated versus no SIRI scores were more likely to have GOC documentation in the unmatched (aOR 2.5, p < 0.0001) and propensity-matched cohorts (aOR 2.1, p < 0.0001).

CONCLUSIONS

Using AI predictions of mortality to target GOC documentation may create differences in documentation prevalence between patients with and without AI mortality prediction scores with similar severity of illness. These finding suggest using AI to target GOC documentation may have the unintended consequence of disadvantaging severely ill patients lacking AI-generated scores from receiving targeted GOC documentation, including patients who are more likely to be non-White and have Medicaid insurance.

Spending Analysis of Machine Learning-Based Communication Nudges in Oncology

BACKGROUND

Serious illness conversations (SICs) in the outpatient setting may improve mood and quality of life among patients with cancer and decrease aggressive end-of-life care. Interventions informed by behavioral economics may increase rates of SICs between oncology clinicians and patients, but the impact of these interventions on end-of-life spending is unknown.

METHODS

This study is a secondary analysis of a stepped-wedge cluster randomized, controlled trial that involved nine medical oncology practices and their high-risk patients at a large academic institution between June 2019 and April 2020. The study included 1187 patients who were identified by a machine-learning algorithm as high risk of 180-day mortality and who died by December 2020. The patients were randomly assigned to standard of care (controls) or to a behavioral intervention designed to increase clinician-initiated SICs. We abstracted spending - defined as inflation-adjusted costs for acute care (inpatient plus emergency room), office/outpatient care, intravenous systemic therapy, other therapy (e.g., radiation), long-term care, and hospice - from the institution's accounting system, and we captured spending at inpatient, outpatient, and pharmacy settings. To evaluate intervention impacts on spending, we used a two-part model, first using logistic regression to model zero versus nonzero spending and second using generalized linear mixed models with gamma distribution and log-link function to model daily mean spending in the last 180days of life. Models were adjusted for clinic and wedge fixed effects, and they were clustered at the oncologist level. For all patients with at least one SIC within 6 months of death, we also calculated their mean daily spending before and after SIC.

RESULTS

Median age at death was 68years (interquartile range, 15.5), 317 patients (27%) were Black or of ethnicities other than white, and 448 patients (38%) had an SIC. The intervention was associated with lower unadjusted mean daily spending in the last 6 months of life for the intervention group versus controls ($377.96 vs. $449.92; adjusted mean difference, -$75.33; 95% confidence interval, -$136.42 to -$14.23; P=0.02), translating to $13,747 total adjusted savings per decedent and $13 million in cumulative savings across all decedents in the intervention group. Compared with controls, patients in the intervention group incurred lower mean daily spending for systemic therapy (adjusted difference, -$44.59; P=0.001), office/outpatient care (-$9.62; P=0.001), and other therapy (-$8.65; P=0.04). The intervention was not associated with differences in end-of-life spending for acute care, long-term care, or hospice. Results were consistent for spending in the last 1 and 3 months of life and after adjusting for age, race, and ethnicity. For patients with SICs, mean daily spending decreased by $37.92 following the first SIC ($329.87 vs. $291.95).

CONCLUSIONS

A machine learning-based, behaviorally informed intervention to prompt SICs led to end-of-life savings among patients with cancer, driven by decreased systemic therapy and outpatient spending. (Funded by the Penn Center for Precision Medicine and the National Institutes of Health; ClinicalTrials.gov number, NCT03984773.).

Let Us Have the Conversation: Serious Illness Communication in Oncology: Definitions, Barriers, and Successful Approaches

Serious illness communications are crucial elements of care delivery for patients with cancer. High-quality serious illness communications are composed of open, honest discussions between patients, caregivers, and clinicians regarding patient's communication preferences, expected illness trajectory, prognosis, and risks and benefits of any recommended care. High-quality communication ideally starts at the time of a patients' cancer diagnosis, allows space for and response to patient emotions, elicits patients' values and care preferences, and is iterative and longitudinal. When integrated into cancer care, such communication can result in improved patient experiences with their care, care that matches patients' goals, and reduced care intensity at the end of life. Despite national recommendations for routine integration of these communication into cancer care, a minority of patients with cancer receive such communication. In this chapter, we describe elements of high-quality serious illness communication, patient-, clinician-, institution-, and payer-level barriers, and successful strategies that can routinely integrate such communication into cancer care delivery.

Why and How to Integrate Early Palliative Care Into Cutting-Edge Personalized Cancer Care

Early palliative care, palliative care integrated with oncology care early in the course of illness, has myriad benefits for patients and their caregivers, including improved quality of life, reduced physical and psychological symptom burden, enhanced prognostic awareness, and reduced health care utilization at the end of life. Although ASCO and others recommend early palliative care for all patients with advanced cancer, widespread implementation of early palliative care has not been realized because of barriers such as insufficient reimbursement and a palliative care workforce shortage. Investigators have recently tested several implementation strategies to overcome these barriers, including triggers for palliative care consultations, telehealth delivery, navigator-delivered interventions, and primary palliative care interventions. More research is needed to identify mechanisms to distribute palliative care optimally and equitably. Simultaneously, the transformation of the oncology treatment landscape has led to shifts in the supportive care needs of patients and caregivers, who may experience longer, uncertain trajectories of cancer. Now, palliative care also plays a clear role in the care of patients with hematologic malignancies and may be beneficial for patients undergoing phase I clinical trials and their caregivers. Further research and clinical guidance regarding how to balance the risks and benefits of opioid therapy and safely manage cancer-related pain across this wide range of settings are urgently needed. The strengths of early palliative care in supporting patients' and caregivers' coping and centering decisions on their goals and values remain valuable in the care of patients receiving cutting-edge personalized cancer care.

Predictors of high-intensity care at the end of life among older adults with solid tumors: A population-based study

INTRODUCTION

High-intensity end-of-life (EoL) care can be burdensome for patients, caregivers, and health systems and does not confer any meaningful clinical benefit. Yet, there are significant knowledge gaps regarding the predictors of high-intensity EoL care. In this study, we identify risk factors associated with high-intensity EoL care among older adults with the four most common malignancies, including breast, prostate, lung, and colorectal cancer.

MATERIALS AND METHODS

Using SEER-Medicare data, we conducted a retrospective analysis of Medicare beneficiaries aged 65 and older who died of breast, prostate, lung, or colorectal cancer between 2011 and 2015. We used multivariable logistic regression to identify clinical, demographic, socioeconomic, and geographic predictors of high-intensity EoL care, which we defined as death in an acute care hospital, receipt of any oral or parenteral chemotherapy within 14 days of death, one or more admissions to the intensive care unit within 30 days of death, two or more emergency department visits within 30 days of death, or two or more inpatient admissions within 30 days of death.

RESULTS

Among 59,355 decedents, factors associated with increased likelihood of receiving high-intensity EoL care were increased comorbidity burden (odds ratio [OR]:1.29; 95% confidence interval [CI]:1.28-1.30), female sex (OR:1.05; 95% CI:1.01-1.09), Black race (OR:1.14; 95% CI:1.07-1.23), Other race/ethnicity (OR:1.20; 95% CI:1.10-1.30), stage III disease (OR:1.11; 95% CI:1.05-1.18), living in a county with >1,000,000 people (OR:1.23; 95% CI:1.16-1.31), living in a census tract with 10%-<20% poverty (OR:1.09; 95% CI:1.03-1.16) or 20%-100% poverty (OR:1.12; 95% CI:1.04-1.19), and having state-subsidized Medicare premiums (OR:1.18; 95% CI:1.12-1.24). The risk of high-intensity EoL care was lower among patients who were older (OR:0.98; 95% CI:0.98-0.99), lived in the Midwest (OR:0.69; 95% CI:0.65-0.75), South (OR:0.70; 95% CI:0.65-0.74), or West (OR:0.81; 95% CI:0.77-0.86), lived in mostly rural areas (OR:0.92; 95% CI:0.86-1.00), and had poor performance status (OR:0.26; 95% CI:0.25-0.28). Results were largely consistent across cancer types.

DISCUSSION

The risk factors identified in our study can inform the development of new interventions for patients with cancer who are likely to receive high-intensity EoL care. Health systems should consider incorporating these risk factors into decision-support tools to assist clinicians in identifying which patients should be referred to hospice and palliative care.

Towards proactive palliative care in oncology: developing an explainable EHR-based machine learning model for mortality risk prediction

BACKGROUND

Ex-ante identification of the last year in life facilitates a proactive palliative approach. Machine learning models trained on electronic health records (EHR) demonstrate promising performance in cancer prognostication. However, gaps in literature include incomplete reporting of model performance, inadequate alignment of model formulation with implementation use-case, and insufficient explainability hindering trust and adoption in clinical settings. Hence, we aim to develop an explainable machine learning EHR-based model that prompts palliative care processes by predicting for 365-day mortality risk among patients with advanced cancer within an outpatient setting.

METHODS

Our cohort consisted of 5,926 adults diagnosed with Stage 3 or 4 solid organ cancer between July 1, 2017, and June 30, 2020 and receiving ambulatory cancer care within a tertiary center. The classification problem was modelled using Extreme Gradient Boosting (XGBoost) and aligned to our envisioned use-case: "Given a prediction point that corresponds to an outpatient cancer encounter, predict for mortality within 365-days from prediction point, using EHR data up to 365-days prior." The model was trained with 75% of the dataset (n = 39,416 outpatient encounters) and validated on a 25% hold-out dataset (n = 13,122 outpatient encounters). To explain model outputs, we used Shapley Additive Explanations (SHAP) values. Clinical characteristics, laboratory tests and treatment data were used to train the model. Performance was evaluated using area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC), while model calibration was assessed using the Brier score.

RESULTS

In total, 17,149 of the 52,538 prediction points (32.6%) had a mortality event within the 365-day prediction window. The model demonstrated an AUROC of 0.861 (95% CI 0.856-0.867) and AUPRC of 0.771. The Brier score was 0.147, indicating slight overestimations of mortality risk. Explanatory diagrams utilizing SHAP values allowed visualization of feature impacts on predictions at both the global and individual levels.

CONCLUSION

Our machine learning model demonstrated good discrimination and precision-recall in predicting 365-day mortality risk among individuals with advanced cancer. It has the potential to provide personalized mortality predictions and facilitate earlier integration of palliative care.

Randomised controlled trials evaluating artificial intelligence in clinical practice: a scoping review

This scoping review of randomised controlled trials on artificial intelligence (AI) in clinical practice reveals an expanding interest in AI across clinical specialties and locations. The USA and China are leading in the number of trials, with a focus on deep learning systems for medical imaging, particularly in gastroenterology and radiology. A majority of trials (70 [81%] of 86) report positive primary endpoints, primarily related to diagnostic yield or performance; however, the predominance of single-centre trials, little demographic reporting, and varying reports of operational efficiency raise concerns about the generalisability and practicality of these results. Despite the promising outcomes, considering the likelihood of publication bias and the need for more comprehensive research including multicentre trials, diverse outcome measures, and improved reporting standards is crucial. Future AI trials should prioritise patient-relevant outcomes to fully understand AI's true effects and limitations in health care.

Comparing Clinician Estimates versus a Statistical Tool for Predicting Risk of Death within 45 Days of Admission for Cancer Patients

OBJECTIVES

 While clinical practice guidelines recommend that oncologists discuss goals of care with patients who have advanced cancer, it is estimated that less than 20% of individuals admitted to the hospital with high-risk cancers have end-of-life discussions with their providers. While there has been interest in developing models for mortality prediction to trigger such discussions, few studies have compared how such models compare with clinical judgment to determine a patient's mortality risk.

METHODS

 This study is a prospective analysis of 1,069 solid tumor medical oncology hospital admissions (n = 911 unique patients) from February 7 to June 7, 2022, at Memorial Sloan Kettering Cancer Center. Electronic surveys were sent to hospitalists, advanced practice providers, and medical oncologists the first afternoon following a hospital admission and they were asked to estimate the probability that the patient would die within 45 days. Provider estimates of mortality were compared with those from a predictive model developed using a supervised machine learning methodology, and incorporated routine laboratory, demographic, biometric, and admission data. Area under the receiver operating characteristic curve (AUC), calibration and decision curves were compared between clinician estimates and the model predictions.

RESULTS

 Within 45 days following hospital admission, 229 (25%) of 911 patients died. The model performed better than the clinician estimates (AUC 0.834 vs. 0.753, p < 0.0001). Integrating clinician predictions with the model's estimates further increased the AUC to 0.853 (p < 0.0001). Clinicians overestimated risk whereas the model was extremely well-calibrated. The model demonstrated net benefit over a wide range of threshold probabilities.

CONCLUSION

 The inpatient prognosis at admission model is a robust tool to assist clinical providers in evaluating mortality risk, and it has recently been implemented in the electronic medical record at our institution to improve end-of-life care planning for hospitalized cancer patients.

Looking Beyond Mortality Prediction: Primary Care Physician Views of Patients' Palliative Care Needs Predicted by a Machine Learning Tool

OBJECTIVES

 To assess primary care physicians' (PCPs) perception of the need for serious illness conversations (SIC) or other palliative care interventions in patients flagged by a machine learning tool for high 1-year mortality risk.

METHODS

 We surveyed PCPs from four Brigham and Women's Hospital primary care practice sites. Multiple mortality prediction algorithms were ensembled to assess adult patients of these PCPs who were either enrolled in the hospital's integrated care management program or had one of several chronic conditions. The patients were classified as high or low risk of 1-year mortality. A blinded survey had PCPs evaluate these patients for palliative care needs. We measured PCP and machine learning tool agreement regarding patients' need for an SIC/elevated risk of mortality.

RESULTS

 Of 66 PCPs, 20 (30.3%) participated in the survey. Out of 312 patients evaluated, 60.6% were female, with a mean (standard deviation [SD]) age of 69.3 (17.5) years, and a mean (SD) Charlson Comorbidity Index of 2.80 (2.89). The machine learning tool identified 162 (51.9%) patients as high risk. Excluding deceased or unfamiliar patients, PCPs felt that an SIC was appropriate for 179 patients; the machine learning tool flagged 123 of these patients as high risk (68.7% concordance). For 105 patients whom PCPs deemed SIC unnecessary, the tool classified 83 as low risk (79.1% concordance). There was substantial agreement between PCPs and the tool (Gwet's agreement coefficient of 0.640).

CONCLUSIONS

 A machine learning mortality prediction tool offers promise as a clinical decision aid, helping clinicians pinpoint patients needing palliative care interventions.

Geriatric End-of-Life Screening Tool Prediction of 6-Month Mortality in Older Patients

Importance

Emergency department (ED) visits by older adults with life-limiting illnesses are a critical opportunity to establish patient care end-of-life preferences, but little is known about the optimal screening criteria for resource-constrained EDs.

Objectives

To externally validate the Geriatric End-of-Life Screening Tool (GEST) in an independent population and compare it with commonly used serious illness diagnostic criteria.

Design, Setting, and Participants

This prognostic study assessed a cohort of patients aged 65 years and older who were treated in a tertiary care ED in Boston, Massachusetts, from 2017 to 2021. Patients arriving in cardiac arrest or who died within 1 day of ED arrival were excluded. Data analysis was performed from August 1, 2023, to March 27, 2024.

Exposure

GEST, a logistic regression algorithm that uses commonly available electronic health record (EHR) datapoints and was developed and validated across 9 EDs, was compared with serious illness diagnoses as documented in the EHR. Serious illnesses included stroke/transient ischemic attack, liver disease, cancer, lung disease, and age greater than 80 years, among others.

Main Outcomes and Measures

The primary outcome was 6-month mortality following an ED encounter. Statistical analyses included area under the receiver operating characteristic curve, calibration analyses, Kaplan-Meier survival curves, and decision curves.

Results

This external validation included 82 371 ED encounters by 40 505 unique individuals (mean [SD] age, 76.8 [8.4] years; 54.3% women, 13.8% 6-month mortality rate). GEST had an external validation area under the receiver operating characteristic curve of 0.79 (95% CI, 0.78-0.79) that was stable across years and demographic subgroups. Of included encounters, 53.4% had a serious illness, with a sensitivity of 77.4% (95% CI, 76.6%-78.2%) and specificity of 50.5% (95% CI, 50.1%-50.8%). Varying GEST cutoffs from 5% to 30% increased specificity (5%: 49.1% [95% CI, 48.7%-49.5%]; 30%: 92.2% [95% CI, 92.0%-92.4%]) at the cost of sensitivity (5%: 89.3% [95% CI, 88.8-89.9]; 30%: 36.2% [95% CI, 35.3-37.1]). In a decision curve analysis, GEST outperformed serious illness criteria across all tested thresholds. When comparing patients referred to intervention by GEST with serious illness criteria, GEST reclassified 45.1% of patients with serious illness as having low risk of mortality with an observed mortality rate 8.1% and 2.6% of patients without serious illness as having high mortality risk with an observed mortality rate of 34.3% for a total reclassification rate of 25.3%.

Conclusions and Relevance

The findings of this study suggest that both serious illness criteria and GEST identified older ED patients at risk for 6-month mortality, but GEST offered more useful screening characteristics. Future trials of serious illness interventions for high mortality risk in older adults may consider transitioning from diagnosis code criteria to GEST, an automatable EHR-based algorithm.

Applying Machine Learning Techniques to Implementation Science

Machine learning (ML) approaches could expand the usefulness and application of implementation science methods in clinical medicine and public health settings. The aim of this viewpoint is to introduce a roadmap for applying ML techniques to address implementation science questions, such as predicting what will work best, for whom, under what circumstances, and with what predicted level of support, and what and when adaptation or deimplementation are needed. We describe how ML approaches could be used and discuss challenges that implementation scientists and methodologists will need to consider when using ML throughout the stages of implementation.

Investigating the Impact of AI on Shared Decision-Making in Post-Kidney Transplant Care (PRIMA-AI): Protocol for a Randomized Controlled Trial

BACKGROUND

Patients after kidney transplantation eventually face the risk of graft loss with the concomitant need for dialysis or retransplantation. Choosing the right kidney replacement therapy after graft loss is an important preference-sensitive decision for kidney transplant recipients. However, the rate of conversations about treatment options after kidney graft loss has been shown to be as low as 13% in previous studies. It is unknown whether the implementation of artificial intelligence (AI)-based risk prediction models can increase the number of conversations about treatment options after graft loss and how this might influence the associated shared decision-making (SDM).

OBJECTIVE

This study aims to explore the impact of AI-based risk prediction for the risk of graft loss on the frequency of conversations about the treatment options after graft loss, as well as the associated SDM process.

METHODS

This is a 2-year, prospective, randomized, 2-armed, parallel-group, single-center trial in a German kidney transplant center. All patients will receive the same routine post-kidney transplant care that usually includes follow-up visits every 3 months at the kidney transplant center. For patients in the intervention arm, physicians will be assisted by a validated and previously published AI-based risk prediction system that estimates the risk for graft loss in the next year, starting from 3 months after randomization until 24 months after randomization. The study population will consist of 122 kidney transplant recipients >12 months after transplantation, who are at least 18 years of age, are able to communicate in German, and have an estimated glomerular filtration rate <30 mL/min/1.73 m2. Patients with multi-organ transplantation, or who are not able to communicate in German, as well as underage patients, cannot participate. For the primary end point, the proportion of patients who have had a conversation about their treatment options after graft loss is compared at 12 months after randomization. Additionally, 2 different assessment tools for SDM, the CollaboRATE mean score and the Control Preference Scale, are compared between the 2 groups at 12 months and 24 months after randomization. Furthermore, recordings of patient-physician conversations, as well as semistructured interviews with patients, support persons, and physicians, are performed to support the quantitative results.

RESULTS

The enrollment for the study is ongoing. The first results are expected to be submitted for publication in 2025.

CONCLUSIONS

This is the first study to examine the influence of AI-based risk prediction on physician-patient interaction in the context of kidney transplantation. We use a mixed methods approach by combining a randomized design with a simple quantitative end point (frequency of conversations), different quantitative measurements for SDM, and several qualitative research methods (eg, records of physician-patient conversations and semistructured interviews) to examine the implementation of AI-based risk prediction in the clinic.

TRIAL REGISTRATION

ClinicalTrials.gov NCT06056518; https://clinicaltrials.gov/study/NCT06056518.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/54857.

Patient and Caregiver Perceptions of an Interface Design to Communicate Artificial Intelligence-Based Prognosis for Patients With Advanced Solid Tumors

PURPOSE

Use of artificial intelligence (AI) in cancer care is increasing. What remains unclear is how best to design patient-facing systems that communicate AI output. With oncologist input, we designed an interface that presents patient-specific, machine learning-based 6-month survival prognosis information designed to aid oncology providers in preparing for and discussing prognosis with patients with advanced solid tumors and their caregivers. The primary purpose of this study was to assess patient and caregiver perceptions and identify enhancements of the interface for communicating 6-month survival and other prognosis information when making treatment decisions concerning anticancer and supportive therapy.

METHODS

This qualitative study included interviews and focus groups conducted between November and December 2022. Purposive sampling was used to recruit former patients with cancer and/or former caregivers of patients with cancer who had participated in cancer treatment decisions from Utah or elsewhere in the United States. Categories and themes related to perceptions of the interface were identified.

RESULTS

We received feedback from 20 participants during eight individual interviews and two focus groups, including four cancer survivors, 13 caregivers, and three representing both. Overall, most participants expressed positive perceptions about the tool and identified its value for supporting decision making, feeling less alone, and supporting communication among oncologists, patients, and their caregivers. Participants identified areas for improvement and implementation considerations, particularly that oncologists should share the tool and guide discussions about prognosis with patients who want to receive the information.

CONCLUSION

This study revealed important patient and caregiver perceptions of and enhancements for the proposed interface. Originally designed with input from oncology providers, patient and caregiver participants identified additional interface design recommendations and implementation considerations to support communication about prognosis.

Thirty-day mortality after palliative radiotherapy in advanced cancer patients: Optimizing end-of-life care in Asia

INTRODUCTION

Evidence-based guidelines recommend hypofractionated palliative radiotherapy (PRT); nonetheless, many patients receive prolonged course of PRT. To identify patients with limited benefits from PRT in end-of-life care, we evaluated the pattern of PRT at an Asian institution and factors associated with 30-day mortality after PRT (30dM).

METHODS

We retrospectively reviewed 228 patients who died after PRT in Yonsei Wonju Severance Christian hospital between October 2014 and March 2022. The associations between clinical factors and survival were assessed using the Cox proportional hazards method. Survival was analysed using the existing models to evaluate their performance in our cohort.

RESULTS

The median PRT duration was 13 (IQR, 7-15) days. Only 11.4% of the patients were treated with hypofractionated radiotherapy. One-third of the patients (32.9%) could not complete PRT and 39 (17.1%) died during PRT. The 30dM was 31.6%. The median time from PRT to death was 17 (IQR, 11-23) days for the patients who died within 30 days. The number of involved organs (≤2 vs. >2; P < 0.001), albumin level (<3.3 vs. ≥3.3; P = 0.016), admission during PRT (P < 0.001), admission 3 months before PRT (P = 0.036) and ICU care during PRT (P < 0.001) were prognostic factors. A comparison of survival based on the existing models yielded unsatisfactory results in our cohort.

CONCLUSION

Almost one-third of the patients received PRT in the last 30 days of life. The use of hypofractionation for PRT was low in this Asian population. Further research is necessary to develop a predictive model of early mortality, allowing tailored end-of-life care for Asian patients.

The association between telemedicine, advance care planning, and unplanned hospitalizations among high-risk patients with cancer

BACKGROUND

Despite the widespread implementation of telemedicine, there are limited data regarding its impact on key components of care for patients with incurable or high-risk cancer. For these patients, high-quality care requires detailed conversations regarding treatment priorities (advance care planning) and clinical care to minimize unnecessary acute care (unplanned hospitalizations). Whether telemedicine affects these outcomes relative to in-person clinic visits was examined among patients with cancer at high risk for 6-month mortality.

METHODS

This retrospective cohort study included adult patients with cancer with any tumor type treated at the University of Pennsylvania who were newly identified between April 1 and December 31, 2020, to be at high risk for 6-month mortality via a validated machine learning algorithm. Separate modified Poisson regressions were used to assess the occurrence of advance care planning and unplanned hospitalizations for telemedicine as compared to in-person visits. Additional analyses were done comparing telemedicine type (video or phone) as compared to in-person clinic visits.

RESULTS

The occurrence of advance care planning was similar between telemedicine and in-person visits (6.8% vs. 6.0%; adjusted risk ratio [aRR], 1.25; 95% CI, 0.92-1.69). In regard to telemedicine subtype, patients exposed to video encounters were modestly more likely to have documented advance care planning in comparison to those seen in person (7.5% vs. 6.0%; aRR, 1.48; 95% CI, 1.03-2.11). The 3-month risk for unplanned hospitalization was comparable for telemedicine compared to in-person clinic encounters (21% vs. 18%; aRR, 1.06; 95% CI, 0.81-1.38).

CONCLUSIONS

In this study, care delivered by telemedicine, compared to in-person clinic visits, produced comparable rates of advance care planning conversations without increasing hospitalizations, which suggests that vulnerable patients can be managed safely by telemedicine.

Characterizing the limitations of using diagnosis codes in the context of machine learning for healthcare

BACKGROUND

Diagnostic codes are commonly used as inputs for clinical prediction models, to create labels for prediction tasks, and to identify cohorts for multicenter network studies. However, the coverage rates of diagnostic codes and their variability across institutions are underexplored. The primary objective was to describe lab- and diagnosis-based labels for 7 selected outcomes at three institutions. Secondary objectives were to describe agreement, sensitivity, and specificity of diagnosis-based labels against lab-based labels.

METHODS

This study included three cohorts: SickKids from The Hospital for Sick Children, and StanfordPeds and StanfordAdults from Stanford Medicine. We included seven clinical outcomes with lab-based definitions: acute kidney injury, hyperkalemia, hypoglycemia, hyponatremia, anemia, neutropenia and thrombocytopenia. For each outcome, we created four lab-based labels (abnormal, mild, moderate and severe) based on test result and one diagnosis-based label. Proportion of admissions with a positive label were presented for each outcome stratified by cohort. Using lab-based labels as the gold standard, agreement using Cohen's Kappa, sensitivity and specificity were calculated for each lab-based severity level.

RESULTS

The number of admissions included were: SickKids (n = 59,298), StanfordPeds (n = 24,639) and StanfordAdults (n = 159,985). The proportion of admissions with a positive diagnosis-based label was significantly higher for StanfordPeds compared to SickKids across all outcomes, with odds ratio (99.9% confidence interval) for abnormal diagnosis-based label ranging from 2.2 (1.7-2.7) for neutropenia to 18.4 (10.1-33.4) for hyperkalemia. Lab-based labels were more similar by institution. When using lab-based labels as the gold standard, Cohen's Kappa and sensitivity were lower at SickKids for all severity levels compared to StanfordPeds.

CONCLUSIONS

Across multiple outcomes, diagnosis codes were consistently different between the two pediatric institutions. This difference was not explained by differences in test results. These results may have implications for machine learning model development and deployment.

Core elements of serious illness conversations: an integrative systematic review

BACKGROUND

Ariadne Labs' Serious Illness Care Program (SICP), inclusive of the Serious Illness Conversation Guide (SICG), has been adapted for use in a variety of settings and among diverse population groups. Explicating the core elements of serious illness conversations could support the inclusion or exclusion of certain components in future iterations of the programme and the guide.

AIM

This integrative systematic review aimed to identify and describe core elements of serious illness conversations in relation to the SICP and/or SICG.

DESIGN

Literature published between 1 January 2014 and 20 March 2023 was searched in MEDLINE, PsycINFO, CINAHL and PubMed. All articles were evaluated using the Joanna Briggs Institute Critical Appraisal Guidelines. Data were analysed with thematic synthesis.

RESULTS

A total of 64 articles met the inclusion criteria. Three themes were revealed: (1) serious illness conversations serve different functions that are reflected in how they are conveyed; (2) serious illness conversations endeavour to discover what matters to patients and (3) serious illness conversations seek to align what patients want in their life and care.

CONCLUSIONS

Core elements of serious illness conversations included explicating the intention, framing, expectations and directions for the conversation. This encompassed discussing current and possible trajectories with a view towards uncovering matters of importance to the patient as a person. Preferences and priorities could be used to inform future preparation and recommendations. Serious illness conversation elements could be adapted and altered depending on the intended purpose of the conversation.

Implementing a Serious Illness Risk Prediction Model: Impact on Goals of Care Documentation

CONTEXT

Goals of care conversations can promote high value care for patients with serious illness, yet documented discussions infrequently occur in hospital settings.

OBJECTIVES

We sought to develop a quality improvement initiative to improve goals of care documentation for hospitalized patients.

METHODS

Implementation occurred at an academic medical center in Pittsburgh, Pennsylvania. Intervention included integration of a 90-day mortality prediction model grouping patients into low, intermediate, and high risk; a centralized goals of care note; and automated notifications and targeted palliative consults. We compared documented goals of care discussions by risk score before and after implementation.

RESULTS

Of the 12,571 patients hospitalized preimplementation and 10,761 postimplementation, 1% were designated high risk and 11% intermediate risk of mortality. Postimplementation, goals of care documentation increased for high (17.6%-70.8%, P< 0.0001) and intermediate risk patients (9.6%-28.0%, P < 0.0001). For intermediate risk patients, the percentage of goals of care documentation performed by palliative medicine specialists increased from pre- to postimplementation (52.3%-71.2%, P = 0.0002). For high-risk patients, the percentage of goals of care documentation completed by the primary service increased from pre-to postimplementation (36.8%-47.1%, P = 0.5898, with documentation performed by palliative medicine specialists slightly decreasing from pre- to postimplementation (63.2%-52.9%, P = 0.5898).

CONCLUSIONS

Implementation of a goals of care initiative using a mortality prediction model significantly increased goals of care documentation especially among high-risk patients. Further study to assess strategies to increase goals of care documentation for intermediate risk patients is needed especially by nonspecialty palliative care.

Remote Patient-Reported Outcomes and Activity Monitoring to Improve Patient-Clinician Communication Regarding Symptoms and Functional Status: A Randomized Controlled Trial

PURPOSE

Routine collection of patient-generated health data (PGHD) may promote earlier recognition of symptomatic and functional decline. This trial assessed the impact of an intervention integrating remote PGHD collection with patient nudges on symptom and functional status understanding between patients with advanced cancer and their oncology team.

METHODS

This three-arm randomized controlled trial was conducted from November 19, 2020, to December 17, 2021, at a large tertiary oncology practice. We enrolled patients with stage IV GI and lung cancers undergoing chemotherapy. Over 6 months, patients in two intervention arms received PROStep-weekly text message-based symptom surveys and passive activity monitoring using a wearable accelerometer. PGHD were summarized in dashboards given to patients' oncology team before appointments. One intervention arm received an additional text-based active choice prompt to discuss worsening symptoms or functional status with their clinician. Control patients did not receive PROStep. The coprimary outcomes patient perceptions of oncology team symptom and functional understanding at 6 months were measured on a 1-5 Likert scale (5 = high understanding).

RESULTS

One hundred eight patients enrolled: 55% male, 81% White, and 77% had GI cancers. Patient-reported clinician understanding did not differ between control and intervention arms for symptoms (4.5 v 4.5; P = .87) or functional status (4.5 v 4.3; P = .31). In the intervention arms, combined patient adherence to weekly symptom reports and daily activity monitoring was 64% and 53%, respectively. Intervention patients in the PROStep versus PROStep + active choice arms reported low burden from wearing the accelerometer (mean burden [standard deviation], 2.7 [1.3] v 2.1 [1.3]; P = .15) and completing surveys (2.1 [1.2] v 1.9 [1.3]; P = .44).

CONCLUSION

Patients receiving PROStep reported high understanding of symptoms and functional status from their oncology team, although this did not differ from controls.

Self-supervised machine learning using adult inpatient data produces effective models for pediatric clinical prediction tasks

OBJECTIVE

Development of electronic health records (EHR)-based machine learning models for pediatric inpatients is challenged by limited training data. Self-supervised learning using adult data may be a promising approach to creating robust pediatric prediction models. The primary objective was to determine whether a self-supervised model trained in adult inpatients was noninferior to logistic regression models trained in pediatric inpatients, for pediatric inpatient clinical prediction tasks.

MATERIALS AND METHODS

This retrospective cohort study used EHR data and included patients with at least one admission to an inpatient unit. One admission per patient was randomly selected. Adult inpatients were 18 years or older while pediatric inpatients were more than 28 days and less than 18 years. Admissions were temporally split into training (January 1, 2008 to December 31, 2019), validation (January 1, 2020 to December 31, 2020), and test (January 1, 2021 to August 1, 2022) sets. Primary comparison was a self-supervised model trained in adult inpatients versus count-based logistic regression models trained in pediatric inpatients. Primary outcome was mean area-under-the-receiver-operating-characteristic-curve (AUROC) for 11 distinct clinical outcomes. Models were evaluated in pediatric inpatients.

RESULTS

When evaluated in pediatric inpatients, mean AUROC of self-supervised model trained in adult inpatients (0.902) was noninferior to count-based logistic regression models trained in pediatric inpatients (0.868) (mean difference = 0.034, 95% CI=0.014-0.057; P < .001 for noninferiority and P = .006 for superiority).

CONCLUSIONS

Self-supervised learning in adult inpatients was noninferior to logistic regression models trained in pediatric inpatients. This finding suggests transferability of self-supervised models trained in adult patients to pediatric patients, without requiring costly model retraining.

Lessons Learned About System-Level Improvement in Serious Illness Communication: A Qualitative Study of Serious Illness Care Program Implementation in Five Health Systems

BACKGROUND

Serious illness communication is a key element of high-quality care, but it is difficult to implement in practice. The Serious Illness Care Program (SICP) is a multifaceted intervention that contributes to more, earlier, and better serious illness conversations and improved patient outcomes. This qualitative study examined the organizational and implementation factors that influenced improvement in real-world contexts.

METHODS

The authors performed semistructured interviews of 30 health professionals at five health systems that adopted SICP as quality improvement initiatives to investigate the organizational and implementation factors that appeared to influence improvement.

RESULTS

After SICP implementation across the organizations studied, approximately 4,661 clinicians have been trained in serious illness communication and 56,712 patients had had an electronic health record (EHR)-documented serious illness conversation. Facilitators included (1) visible support from leaders, who financially invested in an implementation team and champions, expressed the importance of serious illness communication as an institutional priority, and created incentives for training and documenting serious illness conversations; (2) EHR and data infrastructure to foster performance improvement and accountability, including an accessible documentation template, a reporting system, and customized data feedback for clinicians; and (3) communication skills training and sustained support for clinicians to problem-solve communication challenges, reflect on communication experiences, and adapt the intervention. Inhibitors included leadership inaction, competing priorities and incentives, variable clinician acceptance of EHR and data tools, and inadequate support for clinicians after training.

CONCLUSION

Successful implementation appeared to rely on multilevel organizational strategies to prioritize, reward, and reinforce serious illness communication. The insights derived from this research may function as an organizational road map to guide implementation of SICP or related quality initiatives.

Development and validation of the SickKids Enterprise-wide Data in Azure Repository (SEDAR)

Objectives

To describe the processes developed by The Hospital for Sick Children (SickKids) to enable utilization of electronic health record (EHR) data by creating sequentially transformed schemas for use across multiple user types.

Methods

We used Microsoft Azure as the cloud service provider and named this effort the SickKids Enterprise-wide Data in Azure Repository (SEDAR). Epic Clarity data from on-premises was copied to a virtual network in Microsoft Azure. Three sequential schemas were developed. The Filtered Schema added a filter to retain only SickKids and valid patients. The Curated Schema created a data structure that was easier to navigate and query. Each table contained a logical unit such as patients, hospital encounters or laboratory tests. Data validation of randomly sampled observations in the Curated Schema was performed. The SK-OMOP Schema was designed to facilitate research and machine learning. Two individuals mapped medical elements to standard Observational Medical Outcomes Partnership (OMOP) concepts.

Results

A copy of Clarity data was transferred to Microsoft Azure and updated each night using log shipping. The Filtered Schema and Curated Schema were implemented as stored procedures and executed each night with incremental updates or full loads. Data validation required up to 16 iterations for each Curated Schema table. OMOP concept mapping achieved at least 80 % coverage for each SK-OMOP table.

Conclusions

We described our experience in creating three sequential schemas to address different EHR data access requirements. Future work should consider replicating this approach at other institutions to determine whether approaches are generalizable.

Integrating Artificial Intelligence and Machine Learning Into Cancer Clinical Trials

The practice of oncology requires analyzing and synthesizing abundant data. From the patient's workup to determine eligibility to the therapies received to the post-treatment surveillance, practitioners must constantly juggle, evaluate, and weigh decision-making based on their best understanding of information at hand. These complex, multifactorial decisions have a tremendous opportunity to benefit from data-driven machine learning (ML) methods to drive opportunities in artificial intelligence (AI). Within the past 5 years, we have seen AI move from simply a promising opportunity to being used in prospective trials. Here, we review recent efforts of AI in clinical trials that have moved the needle towards improved prediction of actionable outcomes, such as predicting acute care visits, short term mortality, and pathologic extranodal extension. We then pause and reflect on how these AI models ask a different question than traditional statistics models that readers may be more familiar with; how then should readers conceptualize and interpret AI models that they are not as familiar with. We end with what we believe are promising future opportunities for AI in oncology, with an eye towards allowing the data to inform us through unsupervised learning and generative models, rather than asking AI to perform specific functions.

Optimizing Informed Consent in Cancer Clinical Trials

The concept of informed consent has evolved considerably over the course of the 20th century, leading to its establishment as a foundational ethical principle for the conduct of biomedical research in the United States. Even though it is now a highly regulated part of cancer research, the process of obtaining informed consent is often impeded by systemic, clinician, and patient factors that require both small- and large-scale intervention. New challenges and considerations continue to emerge due to innovations in clinical trial design, increases in utilization of genomic sequencing, and advances in genomic editing and artificial intelligence. We present a review of the history, policy, pragmatic challenges, and evolving role of the central ethical tenet of informed consent in clinical trials.

Creating a culture for change: Lessons from behavioral economics and complexity science to increase serious illness conversations for patients with cancer

Patient-centered cancer care requires communication between patients and clinicians about patients' goals, values, and preferences. Serious illness communication improves patient and caregiver outcomes, the value and quality of cancer care, and the well-being of clinicians. Despite these benefits, there are competing factors including time, capacity, bandwidth, and resistance. Health systems and oncology practices have opportunities to invest in pathways that assist patients and clinicians to engage in serious illness conversations. We discuss how applying insights from behavioral economics and complexity science may help clinicians engage in serious illness conversation and improve patient-centered cancer care.

Clinician Trust in Artificial Intelligence: What is Known and How Trust Can Be Facilitated

Predictive analytics based on artificial intelligence (AI) offer clinicians the opportunity to leverage big data available in electronic health records (EHR) to improve clinical decision-making, and thus patient outcomes. Despite this, many barriers exist to facilitating trust between clinicians and AI-based tools, limiting its current impact. Potential solutions are available at both the local and national level. It will take a broad and diverse coalition of stakeholders, from health-care systems, EHR vendors, and clinical educators to regulators, researchers and the patient community, to help facilitate this trust so that the promise of AI in health care can be realized.

Cancer Informatics for Cancer Centers: Sharing Ideas on How to Build an Artificial Intelligence-Ready Informatics Ecosystem for Radiation Oncology

In August 2022, the Cancer Informatics for Cancer Centers brought together cancer informatics leaders for its biannual symposium, Precision Medicine Applications in Radiation Oncology, co-chaired by Quynh-Thu Le, MD (Stanford University), and Walter J. Curran, MD (GenesisCare). Over the course of 3 days, presenters discussed a range of topics relevant to radiation oncology and the cancer informatics community more broadly, including biomarker development, decision support algorithms, novel imaging tools, theranostics, and artificial intelligence (AI) for the radiotherapy workflow. Since the symposium, there has been an impressive shift in the promise and potential for integration of AI in clinical care, accelerated in large part by major advances in generative AI. AI is now poised more than ever to revolutionize cancer care. Radiation oncology is a field that uses and generates a large amount of digital data and is therefore likely to be one of the first fields to be transformed by AI. As experts in the collection, management, and analysis of these data, the informatics community will take a leading role in ensuring that radiation oncology is prepared to take full advantage of these technological advances. In this report, we provide highlights from the symposium, which took place in Santa Barbara, California, from August 29 to 31, 2022. We discuss lessons learned from the symposium for data acquisition, management, representation, and sharing, and put these themes into context to prepare radiation oncology for the successful and safe integration of AI and informatics technologies.

Development of a Multilevel Model to Identify Patients at Risk for Delay in Starting Cancer Treatment

Importance

Delays in starting cancer treatment disproportionately affect vulnerable populations and can influence patients' experience and outcomes. Machine learning algorithms incorporating electronic health record (EHR) data and neighborhood-level social determinants of health (SDOH) measures may identify at-risk patients.

Objective

To develop and validate a machine learning model for estimating the probability of a treatment delay using multilevel data sources.

Design, Setting, and Participants

This cohort study evaluated 4 different machine learning approaches for estimating the likelihood of a treatment delay greater than 60 days (group least absolute shrinkage and selection operator [LASSO], bayesian additive regression tree, gradient boosting, and random forest). Criteria for selecting between approaches were discrimination, calibration, and interpretability/simplicity. The multilevel data set included clinical, demographic, and neighborhood-level census data derived from the EHR, cancer registry, and American Community Survey. Patients with invasive breast, lung, colorectal, bladder, or kidney cancer diagnosed from 2013 to 2019 and treated at a comprehensive cancer center were included. Data analysis was performed from January 2022 to June 2023.

Exposures

Variables included demographics, cancer characteristics, comorbidities, laboratory values, imaging orders, and neighborhood variables.

Main Outcomes and Measures

The outcome estimated by machine learning models was likelihood of a delay greater than 60 days between cancer diagnosis and treatment initiation. The primary metric used to evaluate model performance was area under the receiver operating characteristic curve (AUC-ROC).

Results

A total of 6409 patients were included (mean [SD] age, 62.8 [12.5] years; 4321 [67.4%] female; 2576 [40.2%] with breast cancer, 1738 [27.1%] with lung cancer, and 1059 [16.5%] with kidney cancer). A total of 1621 (25.3%) experienced a delay greater than 60 days. The selected group LASSO model had an AUC-ROC of 0.713 (95% CI, 0.679-0.745). Lower likelihood of delay was seen with diagnosis at the treating institution; first malignant neoplasm; Asian or Pacific Islander or White race; private insurance; and lacking comorbidities. Greater likelihood of delay was seen at the extremes of neighborhood deprivation. Model performance (AUC-ROC) was lower in Black patients, patients with race and ethnicity other than non-Hispanic White, and those living in the most disadvantaged neighborhoods. Though the model selected neighborhood SDOH variables as contributing variables, performance was similar when fit with and without these variables.

Conclusions and Relevance

In this cohort study, a machine learning model incorporating EHR and SDOH data was able to estimate the likelihood of delays in starting cancer therapy. Future work should focus on additional ways to incorporate SDOH data to improve model performance, particularly in vulnerable populations.

External Validation of a Machine Learning Model to Predict 6-Month Mortality for Patients With Advanced Solid Tumors

This prognostic study performed external validation of a machine learning model to predict 6-month mortality among patients with advanced solid tumors.

Intervention to Promote Communication About Goals of Care for Hospitalized Patients With Serious Illness: A Randomized Clinical Trial

Importance

Discussions about goals of care are important for high-quality palliative care yet are often lacking for hospitalized older patients with serious illness.

Objective

To evaluate a communication-priming intervention to promote goals-of-care discussions between clinicians and hospitalized older patients with serious illness.

Design, Setting, and Participants

A pragmatic, randomized clinical trial of a clinician-facing communication-priming intervention vs usual care was conducted at 3 US hospitals within 1 health care system, including a university, county, and community hospital. Eligible hospitalized patients were aged 55 years or older with any of the chronic illnesses used by the Dartmouth Atlas project to study end-of-life care or were aged 80 years or older. Patients with documented goals-of-care discussions or a palliative care consultation between hospital admission and eligibility screening were excluded. Randomization occurred between April 2020 and March 2021 and was stratified by study site and history of dementia.

Intervention

Physicians and advance practice clinicians who were treating the patients randomized to the intervention received a 1-page, patient-specific intervention (Jumpstart Guide) to prompt and guide goals-of-care discussions.

Main Outcomes and Measures

The primary outcome was the proportion of patients with electronic health record-documented goals-of-care discussions within 30 days. There was also an evaluation of whether the effect of the intervention varied by age, sex, history of dementia, minoritized race or ethnicity, or study site.

Results

Of 3918 patients screened, 2512 were enrolled (mean age, 71.7 [SD, 10.8] years and 42% were women) and randomized (1255 to the intervention group and 1257 to the usual care group). The patients were American Indian or Alaska Native (1.8%), Asian (12%), Black (13%), Hispanic (6%), Native Hawaiian or Pacific Islander (0.5%), non-Hispanic (93%), and White (70%). The proportion of patients with electronic health record-documented goals-of-care discussions within 30 days was 34.5% (433 of 1255 patients) in the intervention group vs 30.4% (382 of 1257 patients) in the usual care group (hospital- and dementia-adjusted difference, 4.1% [95% CI, 0.4% to 7.8%]). The analyses of the treatment effect modifiers suggested that the intervention had a larger effect size among patients with minoritized race or ethnicity. Among 803 patients with minoritized race or ethnicity, the hospital- and dementia-adjusted proportion with goals-of-care discussions was 10.2% (95% CI, 4.0% to 16.5%) higher in the intervention group than in the usual care group. Among 1641 non-Hispanic White patients, the adjusted proportion with goals-of-care discussions was 1.6% (95% CI, -3.0% to 6.2%) higher in the intervention group than in the usual care group. There was no evidence of differential treatment effects of the intervention on the primary outcome by age, sex, history of dementia, or study site.

Conclusions and Relevance

Among hospitalized older adults with serious illness, a pragmatic clinician-facing communication-priming intervention significantly improved documentation of goals-of-care discussions in the electronic health record, with a greater effect size in racially or ethnically minoritized patients.

Trial Registration

ClinicalTrials.gov Identifier: NCT04281784.

Automatable end-of-life screening for older adults in the emergency department using electronic health records

BACKGROUND

Emergency department (ED) visits are common at the end-of-life, but the identification of patients with life-limiting illness remains a key challenge in providing timely and resource-sensitive advance care planning (ACP) and palliative care services. To date, there are no validated, automatable instruments for ED end-of-life screening. Here, we developed a novel electronic health record (EHR) prognostic model to screen older ED patients at high risk for 6-month mortality and compare its performance to validated comorbidity indices.

METHODS

This was a retrospective, observational cohort study of ED visits from adults aged ≥65 years who visited any of 9 EDs across a large regional health system between 2014 and 2019. Multivariable logistic regression that included clinical and demographic variables, vital signs, and laboratory data was used to develop a 6-month mortality predictive model-the Geriatric End-of-life Screening Tool (GEST) using five-fold cross-validation on data from 8 EDs. Performance was compared to the Charlson and Elixhauser comorbidity indices using area under the receiver-operating characteristic curve (AUROC), calibration, and decision curve analyses. Reproducibility was tested against data from the remaining independent ED within the health system. We then used GEST to investigate rates of ACP documentation availability and code status orders in the EHR across risk strata.

RESULTS

A total of 431,179 encounters by 123,128 adults were included in this study with a 6-month mortality rate of 12.2%. Charlson (AUROC (95% CI): 0.65 (0.64-0.69)) and Elixhauser indices (0.69 (0.68-0.70)) were outperformed by GEST (0.82 (0.82-0.83)). GEST displayed robust performance across demographic subgroups and in our independent validation site. Among patients with a greater than 30% mortality risk using GEST, only 5.0% had ACP documentation; 79.0% had a code status previously ordered, of which 70.7% were full code. In decision curve analysis, GEST provided greater net benefit than the Charlson and Elixhauser scores.

CONCLUSIONS

Prognostic models using EHR data robustly identify high mortality risk older adults in the ED for whom code status, ACP, or palliative care interventions may be of benefit. Although all tested methods identified patients approaching the end-of-life, GEST was most performant. These tools may enable resource-sensitive end-of-life screening in the ED.

Guidelines for goals of care discussions in patients with gynecologic cancer

This article represents a distillation of literature to provide guidance for goals of care discussions with patients who have gynecologic malignancies. As clinicians who provide surgical care, chemotherapy, and targeted therapeutics, gynecologic oncology clinicians are uniquely positioned to form longitudinal relationships with patients that can enable patient-centered decision making. In this review, we describe optimal timing, components, and best practices for goals of care discussions in gynecologic oncology.

Provider Interaction With an Electronic Health Record Notification to Identify Eligible Patients for a Cluster Randomized Trial of Advance Care Planning in Primary Care: Secondary Analysis

BACKGROUND

Advance care planning (ACP) improves patient-provider communication and aligns care to patient values, preferences, and goals. Within a multisite Meta-network Learning and Research Center ACP study, one health system deployed an electronic health record (EHR) notification and algorithm to alert providers about patients potentially appropriate for ACP and the clinical study.

OBJECTIVE

The aim of the study is to describe the implementation and usage of an EHR notification for referring patients to an ACP study, evaluate the association of notifications with study referrals and engagement in ACP, and assess provider interactions with and perspectives on the notifications.

METHODS

A secondary analysis assessed provider usage and their response to the notification (eg, acknowledge, dismiss, or engage patient in ACP conversation and refer patient to the clinical study). We evaluated all patients identified by the EHR algorithm during the Meta-network Learning and Research Center ACP study. Descriptive statistics compared patients referred to the study to those who were not referred to the study. Health care utilization, hospice referrals, and mortality as well as documentation and billing for ACP and related legal documents are reported. We evaluated associations between notifications with provider actions (ie, referral to study, ACP not documentation, and ACP billing). Provider free-text comments in the notifications were summarized qualitatively. Providers were surveyed on their satisfaction with the notification.

RESULTS

Among the 2877 patients identified by the EHR algorithm over 20 months, 17,047 unique notifications were presented to 45 providers in 6 clinics, who then referred 290 (10%) patients. Providers had a median of 269 (IQR 65-552) total notifications, and patients had a median of 4 (IQR 2-8). Patients with more (over 5) notifications were less likely to be referred to the study than those with fewer notifications (57/1092, 5.2% vs 233/1785, 13.1%; P<.001). The most common free-text comment on the notification was lack of time. Providers who referred patients to the study were more likely to document ACP and submit ACP billing codes (P<.001). In the survey, 11 providers would recommend the notification (n=7, 64%); however, the notification impacted clinical workflow (n=9, 82%) and was difficult to navigate (n=6, 55%).

CONCLUSIONS

An EHR notification can be implemented to remind providers to both perform ACP conversations and refer patients to a clinical study. There were diminishing returns after the fifth EHR notification where additional notifications did not lead to more trial referrals, ACP documentation, or ACP billing. Creation and optimization of EHR notifications for study referrals and ACP should consider the provider user, their workflow, and alert fatigue to improve implementation and adoption.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03577002; https://clinicaltrials.gov/ct2/show/NCT03577002.

Long-term Effect of Machine Learning-Triggered Behavioral Nudges on Serious Illness Conversations and End-of-Life Outcomes Among Patients With Cancer: A Randomized Clinical Trial

Importance

Serious illness conversations (SICs) between oncology clinicians and patients are associated with improved quality of life and may reduce aggressive end-of-life care. However, most patients with cancer die without a documented SIC.

Objective

To test the impact of behavioral nudges to clinicians to prompt SICs on the SIC rate and end-of-life outcomes among patients at high risk of death within 180 days (high-risk patients) as identified by a machine learning algorithm.

Design, Setting, and Participants

This prespecified 40-week analysis of a stepped-wedge randomized clinical trial conducted between June 17, 2019, and April 20, 2020 (including 16 weeks of intervention rollout and 24 weeks of follow-up), included 20 506 patients with cancer representing 41 021 encounters at 9 tertiary or community-based medical oncology clinics in a large academic health system. The current analyses were conducted from June 1, 2021, to May 31, 2022.

Intervention

High-risk patients were identified using a validated electronic health record machine learning algorithm to predict 6-month mortality. The intervention consisted of (1) weekly emails to clinicians comparing their SIC rates for all patients against peers' rates, (2) weekly lists of high-risk patients, and (3) opt-out text messages to prompt SICs before encounters with high-risk patients.

Main Outcomes and Measures

The primary outcome was SIC rates for all and high-risk patient encounters; secondary end-of-life outcomes among decedents included inpatient death, hospice enrollment and length of stay, and intensive care unit admission and systemic therapy close to death. Intention-to-treat analyses were adjusted for clinic and wedge fixed effects and clustered at the oncologist level.

Results

The study included 20 506 patients (mean [SD] age, 60.0 [14.0] years) and 41 021 patient encounters: 22 259 (54%) encounters with female patients, 28 907 (70.5%) with non-Hispanic White patients, and 5520 (13.5%) with high-risk patients; 1417 patients (6.9%) died by the end of follow-up. There were no meaningful differences in demographic characteristics in the control and intervention periods. Among high-risk patient encounters, the unadjusted SIC rates were 3.4% (59 of 1754 encounters) in the control period and 13.5% (510 of 3765 encounters) in the intervention period. In adjusted analyses, the intervention was associated with increased SICs for all patients (adjusted odds ratio, 2.09 [95% CI, 1.53-2.87]; P < .001) and decreased end-of-life systemic therapy (7.5% [72 of 957 patients] vs 10.4% [24 of 231 patients]; adjusted odds ratio, 0.25 [95% CI, 0.11-0.57]; P = .001) relative to controls, but there was no effect on hospice enrollment or length of stay, inpatient death, or end-of-life ICU use.

Conclusions and Relevance

In this randomized clinical trial, a machine learning-based behavioral intervention and behavioral nudges to clinicans led to an increase in SICs and reduction in end-of-life systemic therapy but no changes in other end-of-life outcomes among outpatients with cancer. These results suggest that machine learning and behavioral nudges can lead to long-lasting improvements in cancer care delivery.

Trial Registration

ClinicalTrials.gov Identifier: NCT03984773.

Use of Machine Learning and Lay Care Coaches to Increase Advance Care Planning Conversations for Patients With Metastatic Cancer

PURPOSE

Patients with metastatic cancer benefit from advance care planning (ACP) conversations. We aimed to improve ACP using a computer model to select high-risk patients, with shorter predicted survival, for conversations with providers and lay care coaches. Outcomes included ACP documentation frequency and end-of-life quality measures.

METHODS

In this study of a quality improvement initiative, providers in four medical oncology clinics received Serious Illness Care Program training. Two clinics (thoracic/genitourinary) participated in an intervention, and two (cutaneous/sarcoma) served as controls. ACP conversations were documented in a centralized form in the electronic medical record. In the intervention, providers and care coaches received weekly e-mails highlighting upcoming clinic patients with < 2 year computer-predicted survival and no prior prognosis documentation. Care coaches contacted these patients for an ACP conversation (excluding prognosis). Providers were asked to discuss and document prognosis.

RESULTS

In the four clinics, 4,968 clinic visits by 1,251 patients met inclusion criteria (metastatic cancer with no prognosis previously documented). In their first visit, 28% of patients were high-risk (< 2 year predicted survival). Preintervention, 3% of both intervention and control clinic patients had ACP documentation during a visit. By intervention end (February 2021), 35% of intervention clinic patients had ACP documentation compared with 3% of control clinic patients. Providers' prognosis documentation rate also increased in intervention clinics after the intervention (2%-27% in intervention clinics, P < .0001; 0%-1% in control clinics). End-of-life care intensity was similar in intervention versus control clinics, but patients with ≥ 1 provider ACP edit met fewer high-intensity care measures (P = .04).

CONCLUSION

Combining a computer prognosis model with care coaches increased ACP documentation.

Performance drift in a mortality prediction algorithm among patients with cancer during the SARS-CoV-2 pandemic

Sudden changes in health care utilization during the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic may have impacted the performance of clinical predictive models that were trained prior to the pandemic. In this study, we evaluated the performance over time of a machine learning, electronic health record-based mortality prediction algorithm currently used in clinical practice to identify patients with cancer who may benefit from early advance care planning conversations. We show that during the pandemic period, algorithm identification of high-risk patients had a substantial and sustained decline. Decreases in laboratory utilization during the peak of the pandemic may have contributed to drift. Calibration and overall discrimination did not markedly decline during the pandemic. This argues for careful attention to the performance and retraining of predictive algorithms that use inputs from the pandemic period.

Time of Clinic Appointment and Serious Illness Communication in Oncology

INTRODUCTION

Serious illness communication in oncology increases goal concordant care. Factors associated with the frequency of serious illness conversations are not well understood. Given prior evidence of the association between suboptimal decision-making and clinic time, we aimed to investigate the relationship between appointment time and the likelihood of serious illness conversations in oncology.

METHODS

We conducted a retrospective study of electronic health record data from 55 367 patient encounters between June 2019 to April 2020, using generalized estimating equations to model the likelihood of a serious illness conversation across clinic time.

RESULTS

Documentation rate decreased from 2.1 to 1.5% in the morning clinic session (8am-12pm) and from 1.2% to .9% in the afternoon clinic session (1pm-4pm). Adjusted odds ratios for Serious illness conversations documentation rates were significantly lower for all hours of each session after the earliest hour (adjusted odds ratios .91 [95% CI, .84-.97], P = .006 for overall linear trend).

CONCLUSIONS

Serious illness conversations between oncologists and patients decrease considerably through the clinic day, and proactive strategies to avoid missed conversations should be investigated.

Effect of Nudges to Clinicians, Patients, or Both to Increase Statin Prescribing: A Cluster Randomized Clinical Trial

Importance

Statins reduce the risk of major adverse cardiovascular events, but less than one-half of individuals in America who meet guideline criteria for a statin are actively prescribed this medication.

Objective

To evaluate whether nudges to clinicians, patients, or both increase initiation of statin prescribing during primary care visits.

Design, Setting, and Participants

This cluster randomized clinical trial evaluated statin prescribing of 158 clinicians from 28 primary care practices including 4131 patients. The design included a 12-month preintervention period and a 6-month intervention period between October 19, 2019, and April 18, 2021.

Interventions

The usual care group received no interventions. The clinician nudge combined an active choice prompt in the electronic health record during the patient visit and monthly feedback on prescribing patterns compared with peers. The patient nudge was an interactive text message delivered 4 days before the visit. The combined nudge included the clinician and patient nudges.

Main Outcomes and Measures

The primary outcome was initiation of a statin prescription during the visit.

Results

The sample comprised 4131 patients with a mean (SD) age of 65.5 (10.5) years; 2120 (51.3%) were male; 1210 (29.3%) were Black, 106 (2.6%) were Hispanic, 2732 (66.1%) were White, and 83 (2.0%) were of other race or ethnicity, and 933 (22.6%) had atherosclerotic cardiovascular disease. In unadjusted analyses during the preintervention period, statins were prescribed to 5.6% of patients (105 of 1876) in the usual care group, 4.8% (97 of 2022) in the patient nudge group, 6.0% (104 of 1723) in the clinician nudge group, and 4.7% (82 of 1752) in the combined group. During the intervention, statins were prescribed to 7.3% of patients (75 of 1032) in the usual care group, 8.5% (100 of 1181) in the patient nudge group, 13.0% (128 of 981) in the clinician nudge arm, and 15.5% (145 of 937) in the combined group. In the main adjusted analyses relative to usual care, the clinician nudge significantly increased statin prescribing alone (5.5 percentage points; 95% CI, 3.4 to 7.8 percentage points; P = .01) and when combined with the patient nudge (7.2 percentage points; 95% CI, 5.1 to 9.1 percentage points; P = .001). The patient nudge alone did not change statin prescribing relative to usual care (0.9 percentage points; 95% CI, -0.8 to 2.5 percentage points; P = .32).

Conclusions and Relevance

Nudges to clinicians with and without a patient nudge significantly increased initiation of a statin prescription during primary care visits. The patient nudge alone was not effective.

Trial Registration

ClinicalTrials.gov Identifier: NCT04307472.

Development of Machine Learning Algorithms Incorporating Electronic Health Record Data, Patient-Reported Outcomes, or Both to Predict Mortality for Outpatients With Cancer

PURPOSE

Machine learning (ML) algorithms that incorporate routinely collected patient-reported outcomes (PROs) alongside electronic health record (EHR) variables may improve prediction of short-term mortality and facilitate earlier supportive and palliative care for patients with cancer.

METHODS

We trained and validated two-phase ML algorithms that incorporated standard PRO assessments alongside approximately 200 routinely collected EHR variables, among patients with medical oncology encounters at a tertiary academic oncology and a community oncology practice.

RESULTS

Among 12,350 patients, 5,870 (47.5%) completed PRO assessments. Compared with EHR- and PRO-only algorithms, the EHR + PRO model improved predictive performance in both tertiary oncology (EHR + PRO v EHR v PRO: area under the curve [AUC] 0.86 [0.85-0.87] v 0.82 [0.81-0.83] v 0.74 [0.74-0.74]) and community oncology (area under the curve 0.89 [0.88-0.90] v 0.86 [0.85-0.88] v 0.77 [0.76-0.79]) practices.

CONCLUSION

Routinely collected PROs contain added prognostic information not captured by an EHR-based ML mortality risk algorithm. Augmenting an EHR-based algorithm with PROs resulted in a more accurate and clinically relevant model, which can facilitate earlier and targeted supportive care for patients with cancer.

Oncologist Perceptions of Algorithm-Based Nudges to Prompt Early Serious Illness Communication: A Qualitative Study

Background: Early serious illness conversations (SICs) about goals of care and prognosis improve mood, quality of life, and end-of-life care quality. Algorithm-based behavioral nudges to oncologists increase the frequency and timeliness of such conversations. However, clinicians' perspectives on such nudges are unknown. Design: Qualitative study consisting of semistructured interviews among medical oncology clinicians who participated in a stepped-wedge cluster randomized trial of Conversation Connect, an algorithm-based intervention consisting of behavioral nudges to promote early SICs in the outpatient oncology setting. Results: Of 79 eligible oncology clinicians, 56 (71%) were approached to participate in interviews and 25 (45%) accepted. Key facilitators to algorithm-based nudges included prompting documentation of conversations, peer comparisons, performance reports, and validating norms around early conversations. Barriers included cancer-specific heterogeneity in algorithm performance and the frequency and tone of text messages. Areas of improvement included utilizing different information channels, identifying patients earlier in the disease trajectory, and incorporating patient-targeted messaging that emphasizes the value of early conversations. Conclusions: Oncology clinicians identified key facilitators and barriers to Conversation Connect. These insights inform future algorithm-based supportive care interventions in oncology. Controlled trial (NCT03984773).

Using nudges to promote clinical decision making of healthcare professionals: A scoping review

Nudging has been discussed in the context of policy and public health, but not so much within healthcare. This scoping review aimed to assess the empirical evidence on how nudging techniques can be used to affect the behavior of healthcare professionals (HCPs) in clinical settings. A systematic database search was conducted for the period January 2010-December 2020 using the PRISMA extension for Scoping Review checklist. Two reviewers independently screened each article for inclusion. Included articles were reviewed to extract key information about each intervention, including purpose, target behavior, measured outcomes, key findings, nudging strategies, intervention objectives and their theoretical underpinnings. Two independent dimensions, building on Kahneman's System 1 and System 2, were used to describe nudging strategies according to user action and timing of their implementation. Of the included 51 articles, 40 reported statistically significant results, six were not significant and two reported mixed results. Thirteen different nudging strategies were identified aimed at modifying four types of HPCs' behavior: prescriptions and orders, procedure, hand hygiene, and vaccination. The most common nudging strategy employed were defaults or pre-orders, followed by alerts or reminders, and active choice. Many interventions did not require any deliberate action from users, here termed passive interventions, such as automatically changing prescriptions to their generic equivalent unless indicated by the user. Passive nudges may be successful in changing the target outcome but may go unnoticed by the user. Future work should consider the broader ethical implications of passive nudges.

Variation in the risk of death due to COVID-19: An international multicenter cohort study of hospitalized adults

BACKGROUND

There is wide variation in mortality among patients hospitalized with COVID-19. Whether this is related to patient or hospital factors is unknown.

OBJECTIVE

To compare the risk of mortality for patients hospitalized with COVID-19 and to determine whether the majority of that variation was explained by differences in patient characteristics across sites.

DESIGN, SETTING, AND PARTICIPANTS

An international multicenter cohort study of hospitalized adults with laboratory-confirmed COVID-19 enrolled from 10 hospitals in Ontario, Canada and 8 hospitals in Copenhagen, Denmark between January 1, 2020 and November 11, 2020.

MAIN OUTCOMES AND MEASURES

Inpatient mortality. We used a multivariable multilevel regression model to compare the in-hospital mortality risk across hospitals and quantify the variation attributable to patient-level factors.

RESULTS

There were 1364 adults hospitalized with COVID-19 in Ontario (n = 1149) and in Denmark (n = 215). In Ontario, the absolute risk of in-hospital mortality ranged from 12.0% to 39.8% across hospitals. Ninety-eight percent of the variation in mortality in Ontario was explained by differences in the characteristics of the patients. In Denmark, the absolute risk of inpatients ranged from 13.8% to 20.6%. One hundred percent of the variation in mortality in Denmark was explained by differences in the characteristics of the inpatients.

CONCLUSION

There was wide variation in inpatient COVID-19 mortality across hospitals, which was largely explained by patient-level factors, such as age and severity of presenting illness. However, hospital-level factors that could have affected care, including resource availability and capacity, were not taken into account. These findings highlight potential limitations in comparing crude mortality rates across hospitals for the purposes of reporting on the quality of care.