Predictive Modeling for Adverse Events and Risk Stratification Programs for People Receiving Cancer Treatment

Use of Patient-Reported Outcomes in Risk Prediction Model Development to Support Cancer Care Delivery: A Scoping Review

PURPOSE

The integration of patient-reported outcomes (PROs) into electronic health records (EHRs) has enabled systematic collection of symptom data to manage post-treatment symptoms. The use and integration of PRO data into routine care are associated with overall treatment success, adherence, and satisfaction. Clinical trials have demonstrated the prognostic value of PROs including physical function and global health status in predicting survival. It is unknown to what extent routinely collected PRO data are used in the development of risk prediction models (RPMs) in oncology care. The objective of the scoping review is to assess how PROs are used to train risk RPMs to predict patient outcomes in oncology care.

METHODS

Using the scoping review methodology outlined in the Joanna Briggs Institute Manual for Evidence Synthesis, we searched four databases (MEDLINE, CINAHL, Embase, and Web of Science) to locate peer-reviewed oncology articles that used PROs as predictors to train models. Study characteristics including settings, clinical outcomes, and model training, testing, validation, and performance data were extracted for analyses.

RESULTS

Of the 1,254 studies identified, 18 met inclusion criteria. Most studies performed retrospective analyses of prospectively collected PRO data to build prediction models. Post-treatment survival was the most common outcome predicted. Discriminative performance of models trained using PROs was better than models trained without PROs. Most studies did not report model calibration.

CONCLUSION

Systematic collection of PROs in routine practice provides an opportunity to use patient-reported data to develop RPMs. Model performance improves when PROs are used in combination with other comprehensive data sources.

Development of a Predictive Model for Emergency Department Utilization and Unanticipated Hospital Admission in Patients Receiving Cancer Treatment for Solid Tumor Malignancies

PURPOSE

Unanticipated health care resource utilization, in the form of either emergency department utilization (EDU) or hospital admission (HA), may be an indicator of lower-quality cancer care. The objective of this study was to develop a predictive model for EDU and HAs within 14 days of receipt of systemic therapy for patients with solid tumors.

METHODS

We abstracted electronic health data on oncology encounters from all patients receiving systemic therapy for solid tumors from March 1, 2015, to August 21, 2020, in the Duke University Health System. We defined a primary composite outcome of an EDU or HA within 14 days after the encounter and then developed a predictive model for the primary outcome using least absolute shrinkage and selection operator regression. To evaluate the model, we calculated the area under the receiver operator curve and the calibration slope.

RESULTS

Twelve thousand eight hundred ninety unique patients with 134,641 oncology encounters were included. Five thousand one hundred fifty of these patients (40.0%) had at least one EDU or HA within 14 days of at least one treatment. Forty-six variables were incorporated into the final model. The top predictors, in order of absolute value of the predictive coefficients, were temperature, systolic blood pressure, cancer group, and marital status. The model's AUC was 0.73 (95% CI, 0.722 to 0.732), indicating good sensitivity and specificity to outcome.

CONCLUSION

The model developed in this study demonstrated good sensitivity in identifying patients with solid tumors who are at highest risk for EDU or HA and could be implemented in clinical practice to allow for preventive outpatient interventions.

Predicting the Future by Studying the Past for Patients With Cancer Diagnosed in the Emergency Department

In the article that accompanies this editorial, Kapadia et al. developed a digital quality measure to identify emergency presentations of incident cancers, a measure they found to associated with both antecedent missed opportunities for diagnosis and subsequent 1-year all-cause mortality. Their work highlights the need for a cancer control continuum that includes, not only improved early detection, but also improved symptom recognition, expedited diagnostic work-up, and increased downstream support, including multilevel interventions focused on care continuity and symptom management for these patients with emergency presentations of cancer to improve cancer outcomes.

Predicting Acute Care Events Among Patients Initiating Chemotherapy: A Practice-Based Validation and Adaptation of the PROACCT Model

PURPOSE

Acute care events (ACEs), comprising emergency department visits and hospitalizations, are a priority area for reduction in oncology. Prognostic models are a compelling strategy to identify high-risk patients and target preventive services, but have yet to be broadly implemented, partly because of challenges with electronic health record (EHR) integration. To facilitate EHR integration, we adapted and validated the previously published PRediction Of Acute Care use during Cancer Treatment (PROACCT) model to identify patients at highest risk for ACEs after systemic anticancer treatment.

METHODS

A retrospective cohort of adults with a cancer diagnosis starting systemic therapy at a single center between July and November 2021 was divided into development (70%) and validation (30%) sets. Clinical and demographic variables were extracted, limited to those in structured format in the EHR, including cancer diagnosis, age, drug category, and ACE in prior year. Three logistic regression models of increasing complexity were developed to predict risk of ACEs.

RESULTS

Five thousand one hundred fifty-three patients were evaluated (3,603 development and 1,550 validation). Several factors were predictive of ACEs: age (in decades), receipt of cytotoxic chemotherapy or immunotherapy, thoracic, GI or hematologic malignancy, and ACE in the prior year. We defined high-risk as the top 10% of risk scores; this population had 33.6% ACE rate compared with 8.3% for the remaining 90% in the low-risk group. The simplest Adapted PROACCT model had a C-statistic of 0.79, sensitivity of 0.28, and specificity of 0.93.

CONCLUSION

We present three models designed for EHR integration that effectively identify oncology patients at highest risk for ACE after initiation of systemic anticancer treatment. By limiting predictors to structured data fields and including all cancer types, these models offer broad applicability for cancer care organizations and may offer a safety net to identify and target resources to this high risk.

Data Science Implementation Trends in Nursing Practice: A Review of the 2021 Literature

OBJECTIVES

The goal of this work was to provide a review of the implementation of data science-driven applications focused on structural or outcome-related nurse-sensitive indicators in the literature in 2021. By conducting this review, we aim to inform readers of trends in the nursing indicators being addressed, the patient populations and settings of focus, and lessons and challenges identified during the implementation of these tools.

METHODS

We conducted a rigorous descriptive review of the literature to identify relevant research published in 2021. We extracted data on model development, implementation-related strategies and measures, lessons learned, and challenges and stakeholder involvement. We also assessed whether reports of data science application implementations currently follow the guidelines of the Developmental and Exploratory Clinical Investigations of DEcision support systems driven by AI (DECIDE-AI) framework.

RESULTS

Of 4,943 articles found in PubMed (NLM) and CINAHL (EBSCOhost), 11 were included in the final review and data extraction. Systems leveraging data science were developed for adult patient populations and were primarily deployed in hospital settings. The clinical domains targeted included mortality/deterioration, utilization/resource allocation, and hospital-acquired infections/COVID-19. The composition of development teams and types of stakeholders involved varied. Research teams more frequently reported on implementation methods than implementation results. Most studies provided lessons learned that could help inform future implementations of data science systems in health care.

CONCLUSION

In 2021, very few studies report on the implementation of data science-driven applications focused on structural- or outcome-related nurse-sensitive indicators. This gap in the sharing of implementation strategies needs to be addressed in order for these systems to be successfully adopted in health care settings.

Comparison of Oncologist and Model Estimates of Risk for Hospitalization During Systemic Therapy for Advanced Cancer

PURPOSE

A validated risk model with inputs of pretreatment sodium and albumin can identify patients at risk for hospitalization during cancer treatment. We evaluated how the model compares with risk estimates from treating oncologists.

METHODS

We evaluated the 30-day risk of hospitalization or death in patients starting palliative-intent systemic therapy for solid tumor malignancy. For each patient, we prospectively recorded categorical estimates of 30-day hospitalization risk (bottom third, middle third, top third) generated by a treating oncologist and by the two-variable model; a third hybrid risk estimate represented a composite of the oncologist and model risk assessments. We analyzed the agreement of oncologist and model-based risk estimates and compared discrimination, sensitivity, and specificity of each risk assessment method.

RESULTS

We collected oncologist, model, and hybrid estimates of hospitalization risk for 120 patients. The 30-day rate of hospitalization or death was 20%. There was minimal agreement between oncologist and model risk estimates (weighted kappa = 0.27). The c-statistic (a measure of discrimination) was 0.69 (95% CI, 0.57 to 0.81) for the clinician assessment, 0.77 for the model assessment (CI, 0.67 to 0.86; P = .24 compared with the oncologist assessment), and 0.79 for the hybrid assessment (CI, 0.69 to 0.90; P = .007 compared with the oncologist assessment). Sensitivity and specificity of the high-risk categorization did not differ significantly between the oncologist and model assessments; the hybrid assessment was significantly more sensitive (P = .02) and less specific (P = .03) than the oncologist assessment.

CONCLUSION

A model with inputs of pretreatment sodium and albumin improves oncologists' predictions of hospitalization risk during cancer treatment.

Development and Validation of a Machine Learning Algorithm Predicting Emergency Department Use and Unplanned Hospitalization in Patients With Head and Neck Cancer

Importance

Patient-reported symptom burden was recently found to be associated with emergency department use and unplanned hospitalization (ED/Hosp) in patients with head and neck cancer. It was hypothesized that symptom scores could be combined with administrative health data to accurately risk stratify patients.

Objective

To develop and validate a machine learning approach to predict future ED/Hosp in patients with head and neck cancer.

Design, Setting, and Participants

This was a population-based predictive modeling study of patients in Ontario, Canada, diagnosed with head and neck cancer from January 2007 through March 2018. All outpatient clinical encounters were identified. Edmonton Symptom Assessment System (ESAS) scores and clinical and demographic factors were abstracted. Training and test cohorts were randomly generated in a 4:1 ratio. Various machine learning algorithms were explored, including (1) logistic regression using a least absolute shrinkage and selection operator, (2) random forest, (3) gradient boosting machine, (4) k-nearest neighbors, and (5) an artificial neural network. Data analysis was performed from September 2021 to January 2022.

Main Outcomes and Measures

The main outcome was any 14-day ED/Hosp event following symptom assessment. The performance of each model was assessed on the test cohort using the area under the receiver operator characteristic (AUROC) curve and calibration plots. Shapley values were used to identify the variables with greatest contribution to the model.

Results

The training cohort consisted of 9409 patients (mean [SD] age, 63.3 [10.9] years) undergoing 59 089 symptom assessments (80%). The remaining 2352 patients (mean [SD] age, 63.3 [11] years) and 14 193 symptom assessments were set aside as the test cohort (20%). Several models had high predictive accuracy, particularly the gradient boosting machine (validation AUROC, 0.80 [95% CI, 0.78-0.81]). A Youden-based cutoff corresponded to a validation sensitivity of 0.77 and specificity of 0.66. Patient-reported symptom scores were consistently identified as being the most predictive features within models. A second model built only with symptom severity data had an AUROC of 0.72 (95% CI, 0.70-0.74).

Conclusions and Relevance

In this study, machine learning approaches predicted with a high degree of accuracy ED/Hosp in patients with head and neck cancer. These tools could be used to accurately risk stratify patients and may help direct targeted intervention.