Healthcare Process Modeling to Phenotype Clinician Behaviors for Exploiting the Signal Gain of Clinical Expertise (HPM-ExpertSignals): Development and evaluation of a conceptual framework

Multisite Pragmatic Cluster-Randomized Controlled Trial of the CONCERN Early Warning System

Importance

Late predictions of hospitalized patient deterioration, resulting from early warning systems (EWS) with limited data sources and/or a care team's lack of shared situational awareness, contribute to delays in clinical interventions. The COmmunicating Narrative Concerns Entered by RNs (CONCERN) Early Warning System (EWS) uses real-time nursing surveillance documentation patterns in its machine learning algorithm to identify patients' deterioration risk up to 42 hours earlier than other EWSs.

Objective

To test our a priori hypothesis that patients with care teams informed by the CONCERN EWS intervention have a lower mortality rate and shorter length of stay (LOS) than the patients with teams not informed by CONCERN EWS.

Design

One-year multisite, pragmatic controlled clinical trial with cluster-randomization of acute and intensive care units to intervention or usual-care groups.

Setting

Two large U.S. health systems.

Participants

Adult patients admitted to acute and intensive care units, excluding those on hospice/palliative/comfort care, or with Do Not Resuscitate/Do Not Intubate orders.

Intervention

The CONCERN EWS intervention calculates patient deterioration risk based on nurses' concern levels measured by surveillance documentation patterns, and it displays the categorical risk score (low, increased, high) in the electronic health record (EHR) for care team members.

Main Outcomes and Measures

Primary outcomes: in-hospital mortality, LOS; survival analysis was used. Secondary outcomes: cardiopulmonary arrest, sepsis, unanticipated ICU transfers, 30-day hospital readmission.

Results

A total of 60 893 hospital encounters (33 024 intervention and 27 869 usual-care) were included. Both groups had similar patient age, race, ethnicity, and illness severity distributions. Patients in the intervention group had a 35.6% decreased risk of death (adjusted hazard ratio [HR], 0.644; 95% confidence interval [CI], 0.532-0.778; P<.0001), 11.2% decreased LOS (adjusted incidence rate ratio, 0.914; 95% CI, 0.902-0.926; P<.0001), 7.5% decreased risk of sepsis (adjusted HR, 0.925; 95% CI, 0.861-0.993; P=.0317), and 24.9% increased risk of unanticipated ICU transfer (adjusted HR, 1.249; 95% CI, 1.093-1.426; P=.0011) compared with patients in the usual-care group.

Conclusions and Relevance

A hospital-wide EWS based on nursing surveillance patterns decreased in-hospital mortality, sepsis, and LOS when integrated into the care team's EHR workflow.

Trial Registration

ClinicalTrials.gov Identifier: NCT03911687.

Examining the Generalizability of Pretrained De-identification Transformer Models on Narrative Nursing Notes

BACKGROUND

 Narrative nursing notes are a valuable resource in informatics research with unique predictive signals about patient care. The open sharing of these data, however, is appropriately constrained by rigorous regulations set by the Health Insurance Portability and Accountability Act (HIPAA) for the protection of privacy. Several models have been developed and evaluated on the open-source i2b2 dataset. A focus on the generalizability of these models with respect to nursing notes remains understudied.

OBJECTIVES

 The study aims to understand the generalizability of pretrained transformer models and investigate the variability of personal protected health information (PHI) distribution patterns between discharge summaries and nursing notes with a goal to inform the future design for model evaluation schema.

METHODS

 Two pretrained transformer models (RoBERTa, ClinicalBERT) fine-tuned on i2b2 2014 discharge summaries were evaluated on our data inpatient nursing notes and compared with the baseline performance. Statistical testing was deployed to assess differences in PHI distribution across discharge summaries and nursing notes.

RESULTS

 RoBERTa achieved the optimal performance when tested on an external source of data, with an F1 score of 0.887 across PHI categories and 0.932 in the PHI binary task. Overall, discharge summaries contained a higher number of PHI instances and categories of PHI compared with inpatient nursing notes.

CONCLUSION

 The study investigated the applicability of two pretrained transformers on inpatient nursing notes and examined the distinctions between nursing notes and discharge summaries concerning the utilization of personal PHI. Discharge summaries presented a greater quantity of PHI instances and types when compared with narrative nursing notes, but narrative nursing notes exhibited more diversity in the types of PHI present, with some pertaining to patient's personal life. The insights obtained from the research help improve the design and selection of algorithms, as well as contribute to the development of suitable performance thresholds for PHI.

Variability in Nursing Documentation Patterns across Patients' Hospital Stays

This study explores the variability in nursing documentation patterns in acute care and ICU settings, focusing on vital signs and note documentation, and examines how these patterns vary across patients' hospital stays, documentation types, and comorbidities. In both acute care and critical care settings, there was significant variability in nursing documentation patterns across hospital stays, by documentation type, and by patients' comorbidities. The results suggest that nurses adapt their documentation practices in response to their patients' fluctuating needs and conditions, highlighting the need to facilitate more individualized care and tailored documentation practices. The implications of these findings can inform decisions on nursing workload management, clinical decision support tools, and EHR optimizations.

A methodology of phenotyping ICU patients from EHR data: High-fidelity, personalized, and interpretable phenotypes estimation

OBJECTIVE

Computing phenotypes that provide high-fidelity, time-dependent characterizations and yield personalized interpretations is challenging, especially given the complexity of physiological and healthcare systems and clinical data quality. This paper develops a methodological pipeline to estimate unmeasured physiological parameters and produce high-fidelity, personalized phenotypes anchored to physiological mechanics from electronic health record (EHR).

METHODS

A methodological phenotyping pipeline is developed that computes new phenotypes defined with unmeasurable computational biomarkers quantifying specific physiological properties in real time. Working within the inverse problem framework, this pipeline is applied to the glucose-insulin system for ICU patients using data assimilation to estimate an established mathematical physiological model with stochastic optimization. This produces physiological model parameter vectors of clinically unmeasured endocrine properties, here insulin secretion, clearance, and resistance, estimated for individual patient. These physiological parameter vectors are used as inputs to unsupervised machine learning methods to produce phenotypic labels and discrete physiological phenotypes. These phenotypes are inherently interpretable because they are based on parametric physiological descriptors. To establish potential clinical utility, the computed phenotypes are evaluated with external EHR data for consistency and reliability and with clinician face validation.

RESULTS

The phenotype computation was performed on a cohort of 109 ICU patients who received no or short-acting insulin therapy, rendering continuous and discrete physiological phenotypes as specific computational biomarkers of unmeasured insulin secretion, clearance, and resistance on time windows of three days. Six, six, and five discrete phenotypes were found in the first, middle, and last three-day periods of ICU stays, respectively. Computed phenotypic labels were predictive with an average accuracy of 89%. External validation of discrete phenotypes showed coherence and consistency in clinically observable differences based on laboratory measurements and ICD 9/10 codes and clinical concordance from face validity. A particularly clinically impactful parameter, insulin secretion, had a concordance accuracy of 83%±27%.

CONCLUSION

The new physiological phenotypes computed with individual patient ICU data and defined by estimates of mechanistic model parameters have high physiological fidelity, are continuous, time-specific, personalized, interpretable, and predictive. This methodology is generalizable to other clinical and physiological settings and opens the door for discovering deeper physiological information to personalize medical care.

Predicting emergency department visits and hospitalizations for patients with heart failure in home healthcare using a time series risk model

OBJECTIVES

Little is known about proactive risk assessment concerning emergency department (ED) visits and hospitalizations in patients with heart failure (HF) who receive home healthcare (HHC) services. This study developed a time series risk model for predicting ED visits and hospitalizations in patients with HF using longitudinal electronic health record data. We also explored which data sources yield the best-performing models over various time windows.

MATERIALS AND METHODS

We used data collected from 9362 patients from a large HHC agency. We iteratively developed risk models using both structured (eg, standard assessment tools, vital signs, visit characteristics) and unstructured data (eg, clinical notes). Seven specific sets of variables included: (1) the Outcome and Assessment Information Set, (2) vital signs, (3) visit characteristics, (4) rule-based natural language processing-derived variables, (5) term frequency-inverse document frequency variables, (6) Bio-Clinical Bidirectional Encoder Representations from Transformers variables, and (7) topic modeling. Risk models were developed for 18 time windows (1-15, 30, 45, and 60 days) before an ED visit or hospitalization. Risk prediction performances were compared using recall, precision, accuracy, F1, and area under the receiver operating curve (AUC).

RESULTS

The best-performing model was built using a combination of all 7 sets of variables and the time window of 4 days before an ED visit or hospitalization (AUC = 0.89 and F1 = 0.69).

DISCUSSION AND CONCLUSION

This prediction model suggests that HHC clinicians can identify patients with HF at risk for visiting the ED or hospitalization within 4 days before the event, allowing for earlier targeted interventions.

A methodology of phenotyping ICU patients from EHR data: high-fidelity, personalized, and interpretable phenotypes estimation

Objective

Computing phenotypes that provide high-fidelity, time-dependent characterizations and yield personalized interpretations is challenging, especially given the complexity of physiological and healthcare systems and clinical data quality. This paper develops a methodological pipeline to estimate unmeasured physiological parameters and produce high-fidelity, personalized phenotypes anchored to physiological mechanics from electronic health record (EHR).

Methods

A methodological phenotyping pipeline is developed that computes new phenotypes defined with unmeasurable computational biomarkers quantifying specific physiological properties in real time. Working within the inverse problem framework, this pipeline is applied to the glucose-insulin system for ICU patients using data assimilation to estimate an established mathematical physiological model with stochastic optimization. This produces physiological model parameter vectors of clinically unmeasured endocrine properties, here insulin secretion, clearance, and resistance, estimated for individual patient. These physiological parameter vectors are used as inputs to unsupervised machine learning methods to produce phenotypic labels and discrete physiological phenotypes. These phenotypes are inherently interpretable because they are based on parametric physiological descriptors. To establish potential clinical utility, the computed phenotypes are evaluated with external EHR data for consistency and reliability and with clinician face validation.

Results

The phenotype computation was performed on a cohort of 109 ICU patients who received no or short-acting insulin therapy, rendering continuous and discrete physiological phenotypes as specific computational biomarkers of unmeasured insulin secretion, clearance, and resistance on time windows of three days. Six, six, and five discrete phenotypes were found in the first, middle, and last three-day periods of ICU stays, respectively. Computed phenotypic labels were predictive with an average accuracy of 89%. External validation of discrete phenotypes showed coherence and consistency in clinically observable differences based on laboratory measurements and ICD 9/10 codes and clinical concordance from face validity. A particularly clinically impactful parameter, insulin secretion, had a concordance accuracy of 83% ± 27%.

Conclusion

The new physiological phenotypes computed with individual patient ICU data and defined by estimates of mechanistic model parameters have high physiological fidelity, are continuous, time-specific, personalized, interpretable, and predictive. This methodology is generalizable to other clinical and physiological settings and opens the door for discovering deeper physiological information to personalize medical care.

Informatics and data science perspective on Future of Nursing 2020-2030: Charting a pathway to health equity

The Future of Nursing 2020 to 2030 report explicitly addresses the need for integration of nursing expertise in designing, generating, analyzing, and applying data to support initiatives focused on social determinants of health (SDOH) and health equity. The metrics necessary to enable and evaluate progress on all recommendations require harnessing existing data sources and developing new ones, as well as transforming and integrating data into information systems to facilitate communication, information sharing, and decision making among the key stakeholders. We examine the recommendations of the 2021 report through an interdisciplinary lens that integrates nursing, biomedical informatics, and data science by addressing three critical questions: (a) what data are needed?, (b) what infrastructure and processes are needed to transform data into information?, and (c) what information systems are needed to "level up" nurse-led interventions from the micro-level to the meso- and macro-levels to address social determinants of health and advance health equity?

A Scoping Review of Integrated Medical Devices and Clinical Decision Support in the Acute Care Setting

BACKGROUND

Seamless data integration between point-of-care medical devices and the electronic health record (EHR) can be central to clinical decision support systems (CDSS).

OBJECTIVE

The objective of this scoping review is to (1) examine the existing evidence related to integrated medical devices, primarily medication pump devices, and associated clinical decision support (CDS) in acute care settings and (2) to identify how acute care clinicians may use device CDS in clinical decision-making. The rationale for this review is that integrated devices are ubiquitous in the acute care setting, and they generate data that may help to contribute to the situational awareness of the clinical team necessary to provide individualized patient care.

METHODS

This scoping review was conducted using the Joanna Briggs Institute Manual for Evidence Synthesis and the Preferred Reporting Items for Systematic Reviews and Meta-analyses Extensions for Scoping Review guidelines. PubMed, CINAHL, IEEE Xplore, and Scopus databases were searched for scholarly, peer-reviewed journals indexed between January 1, 2010 and December 31, 2020. A priori inclusion criteria were established.

RESULTS

Of the 1,924 articles screened, 18 were ultimately included for synthesis, and primarily included articles on devices such as intravenous medication pumps and vital signs machines. Clinical alarm burden was mentioned in most of the articles, and despite not including the term "medication" there were many articles about smart pumps being integrated with the EHR. The Revised Technology, Nursing & Patient Safety Conceptual Model provided the organizational framework. Ten articles described patient assessment, monitoring, or surveillance use. Three articles described patient protection from harm. Four articles described direct care use scenarios, all of which described insulin administration. One article described a hybrid situation of patient communication and monitoring. Most of the articles described devices and decision support primarily used by registered nurses (RNs).

CONCLUSION

The articles in this review discussed devices and the associated CDSS that are used by clinicians, primarily RNs, in the daily provision of care for patients. Integrated device data provide insight into user-device interactions and help to illustrate health care processes, especially the activities when providing direct care to patients in an acute care setting. While there are CDSS designed to support the clinician while working with devices, RNs and providers may disregard this guidance, and defer to their own expertise. Additionally, if clinicians perceive CDSS as intrusive, they are at risk for alarm and alert fatigue if CDSS are not tailored to sync with the workflow of the end-user. Areas for future research include refining inclusion criteria to examine the evidence for devices and their CDS that are most likely used by other groups' health care professionals (i.e., doctors and therapists), using integrated device metadata and deep learning analytics to identify patterns in care delivery, and decision support tools for patients using their own personal data.

Factors Influencing Clinician Trust in Predictive Clinical Decision Support Systems for In-Hospital Deterioration: Qualitative Descriptive Study

BACKGROUND

Clinician trust in machine learning-based clinical decision support systems (CDSSs) for predicting in-hospital deterioration (a type of predictive CDSS) is essential for adoption. Evidence shows that clinician trust in predictive CDSSs is influenced by perceived understandability and perceived accuracy.

OBJECTIVE

The aim of this study was to explore the phenomenon of clinician trust in predictive CDSSs for in-hospital deterioration by confirming and characterizing factors known to influence trust (understandability and accuracy), uncovering and describing other influencing factors, and comparing nurses' and prescribing providers' trust in predictive CDSSs.

METHODS

We followed a qualitative descriptive methodology conducting directed deductive and inductive content analysis of interview data. Directed deductive analyses were guided by the human-computer trust conceptual framework. Semistructured interviews were conducted with nurses and prescribing providers (physicians, physician assistants, or nurse practitioners) working with a predictive CDSS at 2 hospitals in Mass General Brigham.

RESULTS

A total of 17 clinicians were interviewed. Concepts from the human-computer trust conceptual framework-perceived understandability and perceived technical competence (ie, perceived accuracy)-were found to influence clinician trust in predictive CDSSs for in-hospital deterioration. The concordance between clinicians' impressions of patients' clinical status and system predictions influenced clinicians' perceptions of system accuracy. Understandability was influenced by system explanations, both global and local, as well as training. In total, 3 additional themes emerged from the inductive analysis. The first, perceived actionability, captured the variation in clinicians' desires for predictive CDSSs to recommend a discrete action. The second, evidence, described the importance of both macro- (scientific) and micro- (anecdotal) evidence for fostering trust. The final theme, equitability, described fairness in system predictions. The findings were largely similar between nurses and prescribing providers.

CONCLUSIONS

Although there is a perceived trade-off between machine learning-based CDSS accuracy and understandability, our findings confirm that both are important for fostering clinician trust in predictive CDSSs for in-hospital deterioration. We found that reliance on the predictive CDSS in the clinical workflow may influence clinicians' requirements for trust. Future research should explore the impact of reliance, the optimal explanation design for enhancing understandability, and the role of perceived actionability in driving trust.

Exploratory analysis of novel electronic health record variables for quantification of healthcare delivery strain, prediction of mortality, and prediction of imminent discharge

OBJECTIVE

To explore the relationship between novel, time-varying predictors for healthcare delivery strain (eg, counts of patient orders per hour) and imminent discharge and in-hospital mortality.

MATERIALS AND METHODS

We conducted a retrospective cohort study using data from adults hospitalized at 21 Kaiser Permanente Northern California hospitals between November 1, 2015 and October 31, 2020 and the nurses caring for them. Patient data extracted included demographics, diagnoses, severity measures, occupancy metrics, and process of care metrics (eg, counts of intravenous drip orders per hour). We linked these data to individual registered nurse records and created multiple dynamic, time-varying predictors (eg, mean acute severity of illness for all patients cared for by a nurse during a given hour). All analyses were stratified by patients' initial hospital unit (ward, stepdown unit, or intensive care unit). We used discrete-time hazard regression to assess the association between each novel time-varying predictor and the outcomes of discharge and mortality, separately.

RESULTS

Our dataset consisted of 84 162 161 hourly records from 954 477 hospitalizations. Many novel time-varying predictors had strong associations with the 2 study outcomes. However, most of the predictors did not merely track patients' severity of illness; instead, many of them only had weak correlations with severity, often with complex relationships over time.

DISCUSSION

Increasing availability of process of care data from automated electronic health records will permit better quantification of healthcare delivery strain. This could result in enhanced prediction of adverse outcomes and service delays.

CONCLUSION

New conceptual models will be needed to use these new data elements.

The Communicating Narrative Concerns Entered by Registered Nurses (CONCERN) Clinical Decision Support Early Warning System: Protocol for a Cluster Randomized Pragmatic Clinical Trial

Background

Every year, hundreds of thousands of inpatients die from cardiac arrest and sepsis, which could be avoided if those patients’ risk for deterioration were detected and timely interventions were initiated. Thus, a system is needed to convert real-time, raw patient data into consumable information that clinicians can utilize to identify patients at risk of deterioration and thus prevent mortality and improve patient health outcomes. The overarching goal of the COmmunicating Narrative Concerns Entered by Registered Nurses (CONCERN) study is to implement and evaluate an early warning score system that provides clinical decision support (CDS) in electronic health record systems. With a combination of machine learning and natural language processing, the CONCERN CDS utilizes nursing documentation patterns as indicators of nurses’ increased surveillance to predict when patients are at the risk of clinical deterioration.

Objective

The objective of this cluster randomized pragmatic clinical trial is to evaluate the effectiveness and usability of the CONCERN CDS system at 2 different study sites. The specific aim is to decrease hospitalized patients’ negative health outcomes (in-hospital mortality, length of stay, cardiac arrest, unanticipated intensive care unit transfers, and 30-day hospital readmission rates).

Methods

A multiple time-series intervention consisting of 3 phases will be performed through a 1-year period during the cluster randomized pragmatic clinical trial. Phase 1 evaluates the adoption of our algorithm through pilot and trial testing, phase 2 activates optimized versions of the CONCERN CDS based on experience from phase 1, and phase 3 will be a silent release mode where no CDS is viewable to the end user. The intervention deals with a series of processes from system release to evaluation. The system release includes CONCERN CDS implementation and user training. Then, a mixed methods approach will be used with end users to assess the system and clinician perspectives.

Results

Data collection and analysis are expected to conclude by August 2022. Based on our previous work on CONCERN, we expect the system to have a positive impact on the mortality rate and length of stay.

Conclusions

The CONCERN CDS will increase team-based situational awareness and shared understanding of patients predicted to be at risk for clinical deterioration in need of intervention to prevent mortality and associated harm.

Trial Registration

ClinicalTrials.gov NCT03911687; https://clinicaltrials.gov/ct2/show/NCT03911687

International Registered Report Identifier (IRRID)

DERR1-10.2196/30238

Predicting next-day discharge via electronic health record access logs

OBJECTIVE

Hospital capacity management depends on accurate real-time estimates of hospital-wide discharges. Estimation by a clinician requires an excessively large amount of effort and, even when attempted, accuracy in forecasting next-day patient-level discharge is poor. This study aims to support next-day discharge predictions with machine learning by incorporating electronic health record (EHR) audit log data, a resource that captures EHR users' granular interactions with patients' records by communicating various semantics and has been neglected in outcome predictions.

MATERIALS AND METHODS

This study focused on the EHR data for all adults admitted to Vanderbilt University Medical Center in 2019. We learned multiple advanced models to assess the value that EHR audit log data adds to the daily prediction of discharge likelihood within 24 h and to compare different representation strategies. We applied Shapley additive explanations to identify the most influential types of user-EHR interactions for discharge prediction.

RESULTS

The data include 26 283 inpatient stays, 133 398 patient-day observations, and 819 types of user-EHR interactions. The model using the count of each type of interaction in the recent 24 h and other commonly used features, including demographics and admission diagnoses, achieved the highest area under the receiver operating characteristics (AUROC) curve of 0.921 (95% CI: 0.919-0.923). By contrast, the model lacking user-EHR interactions achieved a worse AUROC of 0.862 (0.860-0.865). In addition, 10 of the 20 (50%) most influential factors were user-EHR interaction features.

CONCLUSION

EHR audit log data contain rich information such that it can improve hospital-wide discharge predictions.

A Damaged-Informed Lung Ventilator Model for Ventilator Waveforms

Motivated by a desire to understand pulmonary physiology, scientists have developed physiological lung models of varying complexity. However, pathophysiology and interactions between human lungs and ventilators, e.g., ventilator-induced lung injury (VILI), present challenges for modeling efforts. This is because the real-world pressure and volume signals may be too complex for simple models to capture, and while complex models tend not to be estimable with clinical data, limiting clinical utility. To address this gap, in this manuscript we developed a new damaged-informed lung ventilator (DILV) model. This approach relies on mathematizing ventilator pressure and volume waveforms, including lung physiology, mechanical ventilation, and their interaction. The model begins with nominal waveforms and adds limited, clinically relevant, hypothesis-driven features to the waveform corresponding to pulmonary pathophysiology, patient-ventilator interaction, and ventilator settings. The DILV model parameters uniquely and reliably recapitulate these features while having enough flexibility to reproduce commonly observed variability in clinical (human) and laboratory (mouse) waveform data. We evaluate the proof-in-principle capabilities of our modeling approach by estimating 399 breaths collected for differently damaged lungs for tightly controlled measurements in mice and uncontrolled human intensive care unit data in the absence and presence of ventilator dyssynchrony. The cumulative value of mean squares error for the DILV model is, on average, ≈12 times less than the single compartment lung model for all the waveforms considered. Moreover, changes in the estimated parameters correctly correlate with known measures of lung physiology, including lung compliance as a baseline evaluation. Our long-term goal is to use the DILV model for clinical monitoring and research studies by providing high fidelity estimates of lung state and sources of VILI with an end goal of improving management of VILI and acute respiratory distress syndrome.

Utilizing timestamps of longitudinal electronic health record data to classify clinical deterioration events

OBJECTIVE

To propose an algorithm that utilizes only timestamps of longitudinal electronic health record data to classify clinical deterioration events.

MATERIALS AND METHODS

This retrospective study explores the efficacy of machine learning algorithms in classifying clinical deterioration events among patients in intensive care units using sequences of timestamps of vital sign measurements, flowsheets comments, order entries, and nursing notes. We design a data pipeline to partition events into discrete, regular time bins that we refer to as timesteps. Logistic regressions, random forest classifiers, and recurrent neural networks are trained on datasets of different length of timesteps, respectively, against a composite outcome of death, cardiac arrest, and Rapid Response Team calls. Then these models are validated on a holdout dataset.

RESULTS

A total of 6720 intensive care unit encounters meet the criteria and the final dataset includes 830 578 timestamps. The gated recurrent unit model utilizes timestamps of vital signs, order entries, flowsheet comments, and nursing notes to achieve the best performance on the time-to-outcome dataset, with an area under the precision-recall curve of 0.101 (0.06, 0.137), a sensitivity of 0.443, and a positive predictive value of 0. 092 at the threshold of 0.6.

DISCUSSION AND CONCLUSION

This study demonstrates that our recurrent neural network models using only timestamps of longitudinal electronic health record data that reflect healthcare processes achieve well-performing discriminative power.