A Generalizable, Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centers

Coronary Disease Risk Prediction, Risk Reduction, and Postoperative Myocardial Injury

For patients considering surgery, the preoperative evaluation allows physicians to identify and treat acute cardiac conditions before less-urgent surgery, predict the benefits and harms of a proposed surgery, and make temporary management changes to reduce operative risk. Multiple risk prediction tools are reasonable for use in estimating perioperative cardiac risk, but management changes to reduce risk have proven elusive. For all but the most urgent surgical procedures, patients with active coronary syndromes or decompensated heart failure should have surgery postponed.

Deep learning-based prediction of Clostridioides difficile infection caused by antibiotics using longitudinal electronic health records

Clostridioides difficile infection (CDI) is a major cause of antibiotic-associated diarrhea and colitis. It is recognized as one of the most significant hospital-acquired infections. Although CDI can develop severe complications and spores of Clostridioides difficile can be transmitted by the fecal-oral route, CDI is occasionally overlooked in clinical settings. Thus, it is necessary to monitor high CDI risk groups, particularly those undergoing antibiotic treatment, to prevent complications and spread. We developed and validated a deep learning-based model to predict the occurrence of CDI within 28 days after starting antibiotic treatment using longitudinal electronic health records. For each patient, timelines of vital signs and laboratory tests with a 35-day monitoring period and a patient information vector consisting of age, sex, comorbidities, and medications were constructed. Our model achieved the prediction performance with an area under the receiver operating characteristic curve of 0.952 (95% CI: 0.932-0.973) in internal validation and 0.972 (95% CI: 0.968-0.975) in external validation. Platelet count and body temperature emerged as the most important features. The risk score, the output value of the model, exhibited a consistent increase in the CDI group, while the risk score in the non-CDI group either maintained its initial value or decreased. Using our CDI prediction model, high-risk patients requiring symptom monitoring can be identified. This could help reduce the underdiagnosis of CDI, thereby decreasing transmission and preventing complications.

Artificial intelligence applications in the diagnosis and treatment of bacterial infections

The diagnosis and treatment of bacterial infections in the medical and public health field in the 21st century remain significantly challenging. Artificial Intelligence (AI) has emerged as a powerful new tool in diagnosing and treating bacterial infections. AI is rapidly revolutionizing epidemiological studies of infectious diseases, providing effective early warning, prevention, and control of outbreaks. Machine learning models provide a highly flexible way to simulate and predict the complex mechanisms of pathogen-host interactions, which is crucial for a comprehensive understanding of the nature of diseases. Machine learning-based pathogen identification technology and antimicrobial drug susceptibility testing break through the limitations of traditional methods, significantly shorten the time from sample collection to the determination of result, and greatly improve the speed and accuracy of laboratory testing. In addition, AI technology application in treating bacterial infections, particularly in the research and development of drugs and vaccines, and the application of innovative therapies such as bacteriophage, provides new strategies for improving therapy and curbing bacterial resistance. Although AI has a broad application prospect in diagnosing and treating bacterial infections, significant challenges remain in data quality and quantity, model interpretability, clinical integration, and patient privacy protection. To overcome these challenges and, realize widespread application in clinical practice, interdisciplinary cooperation, technology innovation, and policy support are essential components of the joint efforts required. In summary, with continuous advancements and in-depth application of AI technology, AI will enable doctors to more effectivelyaddress the challenge of bacterial infection, promoting the development of medical practice toward precision, efficiency, and personalization; optimizing the best nursing and treatment plans for patients; and providing strong support for public health safety.

Improving Risk Prediction of Methicillin-Resistant Staphylococcus aureus Using Machine Learning Methods With Network Features: Retrospective Development Study

BACKGROUND

Health care-associated infections due to multidrug-resistant organisms (MDROs), such as methicillin-resistant Staphylococcus aureus (MRSA) and Clostridioides difficile (CDI), place a significant burden on our health care infrastructure.

OBJECTIVE

Screening for MDROs is an important mechanism for preventing spread but is resource intensive. The objective of this study was to develop automated tools that can predict colonization or infection risk using electronic health record (EHR) data, provide useful information to aid infection control, and guide empiric antibiotic coverage.

METHODS

We retrospectively developed a machine learning model to detect MRSA colonization and infection in undifferentiated patients at the time of sample collection from hospitalized patients at the University of Virginia Hospital. We used clinical and nonclinical features derived from on-admission and throughout-stay information from the patient's EHR data to build the model. In addition, we used a class of features derived from contact networks in EHR data; these network features can capture patients' contacts with providers and other patients, improving model interpretability and accuracy for predicting the outcome of surveillance tests for MRSA. Finally, we explored heterogeneous models for different patient subpopulations, for example, those admitted to an intensive care unit or emergency department or those with specific testing histories, which perform better.

RESULTS

We found that the penalized logistic regression performs better than other methods, and this model's performance measured in terms of its receiver operating characteristics-area under the curve score improves by nearly 11% when we use polynomial (second-degree) transformation of the features. Some significant features in predicting MDRO risk include antibiotic use, surgery, use of devices, dialysis, patient's comorbidity conditions, and network features. Among these, network features add the most value and improve the model's performance by at least 15%. The penalized logistic regression model with the same transformation of features also performs better than other models for specific patient subpopulations.

CONCLUSIONS

Our study shows that MRSA risk prediction can be conducted quite effectively by machine learning methods using clinical and nonclinical features derived from EHR data. Network features are the most predictive and provide significant improvement over prior methods. Furthermore, heterogeneous prediction models for different patient subpopulations enhance the model's performance.

Generalizability of predictive models for Clostridioides difficile infection, severity and recurrence at an urban safety-net hospital

INTRODUCTION

Predictive models for Clostridioides difficile infection can identify high-risk patients and aid clinicians in preventing infection. Issues of generalizability regarding current predictive models have been acknowledged but, to the authors' knowledge, have never been quantified.

METHODS

C. difficile infection, severity and recurrence predictive models were created using multi-variate logistic regression through case-control sampling from an urban safety-net hospital. Models were validated using five-fold cross-validation, and inverse probability weights (IPW) based on two different catchment area definitions were used to improve external validity. Akaike Information Criterion (AIC), area under the receiver operating characteristic curve (AUROC), and sensitivity and specificity with bootstrapped confidence intervals (CI) were used to assess and compare model fit and performance.

RESULTS

Changes in performance before and after weighting were small across all models, although differences were more apparent after weighting the recurrence model (AUROC values of 0.78, 0.76 and 0.71 for the unweighted and two weighted models, respectively). Overall, the infection model performed the best (AUROC 0.82, 95% CI 0.78-0.85), followed by the recurrence model (AUROC 0.78, 95% CI 0.69-0.86) and then the severity model (AUROC 0.70, 95% CI 0.63-0.78).

CONCLUSIONS

The performance of the models after weighting did not change drastically, suggesting that the models predicting C. difficile infection, severity and recurrence may not be impacted by patient selection factors. However, other researchers may wish to consider addressing these catchment forces using IPW.

Harnessing of Artificial Intelligence for the Diagnosis and Prevention of Hospital-Acquired Infections: A Systematic Review

Healthcare-associated infections (HAIs) are the most common adverse events in healthcare and constitute a major global public health concern. Surveillance represents the foundation for the effective prevention and control of HAIs, yet conventional surveillance is costly and labor intensive. Artificial intelligence (AI) and machine learning (ML) have the potential to support the development of HAI surveillance algorithms for the understanding of HAI risk factors, the improvement of patient risk stratification as well as the prediction and timely detection and prevention of infections. AI-supported systems have so far been explored for clinical laboratory testing and imaging diagnosis, antimicrobial resistance profiling, antibiotic discovery and prediction-based clinical decision support tools in terms of HAIs. This review aims to provide a comprehensive summary of the current literature on AI applications in the field of HAIs and discuss the future potentials of this emerging technology in infection practice. Following the PRISMA guidelines, this study examined the articles in databases including PubMed and Scopus until November 2023, which were screened based on the inclusion and exclusion criteria, resulting in 162 included articles. By elucidating the advancements in the field, we aim to highlight the potential applications of AI in the field, report related issues and shortcomings and discuss the future directions.

Development of a Prediction Model to Identify the Risk of Clostridioides difficile Infection in Hospitalized Patients Receiving at Least One Dose of Antibiotics

OBJECTIVE

This study's objective was to develop a risk-prediction model to identify hospitalized patients at risk of Clostridioides difficile infection (CDI) who had received at least one dose of systemic antibiotics in a large tertiary hospital.

PATIENTS AND METHODS

This was a retrospective case-control study that included patients hospitalized for more than 2 days who received antibiotic therapy during hospitalization. The study included two groups: patients diagnosed with hospital CDI and controls without hospital CDI. Cases were matched 1:3 with assigned controls by age and sex. Descriptive statistics were used to identify the study population by comparing cases with controls. Continuous variables were stated as the means and standard deviations. A multivariate analysis was built to identify the significantly associated covariates between cases and controls for CDI.

RESULTS

A total of 364 patients were included and distributed between the two groups. The control group included 273 patients, and the case group included 91 patients. The risk factors for CDI were investigated, with only significant risks identified and included in the risk assessment model: age older than 70 years (p = 0.034), chronic kidney disease (p = 0.043), solid organ transplantation (p = 0.021), and lymphoma or leukemia (p = 0.019). A risk score of ≥2 showed the best sensitivity, specificity, and accuracy of 78.02%, 45.42%, and 78.02, respectively, with an area under the curve of 0.6172.

CONCLUSION

We identified four associated risk factors in the risk-prediction model. The tool showed good discrimination that might help predict, identify, and evaluate hospitalized patients at risk of developing CDI.

Diagnostic prediction models to identify patients at risk for healthcare-facility-onset Clostridioides difficile: A systematic review of methodology and reporting

OBJECTIVE

To systematically review the methodology, performance, and generalizability of diagnostic models for predicting the risk of healthcare-facility-onset (HO) Clostridioides difficile infection (CDI) in adult hospital inpatients (aged ≥18 years).

BACKGROUND

CDI is the most common cause of healthcare-associated diarrhea. Prediction models that identify inpatients at risk of HO-CDI have been published; however, the quality and utility of these models remain uncertain.

METHODS

Two independent reviewers evaluated articles describing the development and/or validation of multivariable HO-CDI diagnostic models in an inpatient setting. All publication dates, languages, and study designs were considered. Model details (eg, sample size and source, outcome, and performance) were extracted from the selected studies based on the CHARMS checklist. The risk of bias was further assessed using PROBAST.

RESULTS

Of the 3,030 records evaluated, 11 were eligible for final analysis, which described 12 diagnostic models. Most studies clearly identified the predictors and outcomes but did not report how missing data were handled. The most frequent predictors across all models were advanced age, receipt of high-risk antibiotics, history of hospitalization, and history of CDI. All studies reported the area under the receiver operating characteristic curve (AUROC) as a measure of discriminatory ability. However, only 3 studies reported the model calibration results, and only 2 studies were externally validated. All of the studies had a high risk of bias.

CONCLUSION

The studies varied in their ability to predict the risk of HO-CDI. Future models will benefit from the validation on a prospective external cohort to maximize external validity.

Clostridioides difficile infection surveillance in intensive care units and oncology wards using machine learning

OBJECTIVE

Screening individuals admitted to the hospital for Clostridioides difficile presents opportunities to limit transmission and hospital-onset C. difficile infection (HO-CDI). However, detection from rectal swabs is resource intensive. In contrast, machine learning (ML) models may accurately assess patient risk without significant resource usage. In this study, we compared the effectiveness of swab surveillance to daily risk estimates produced by an ML model to identify patients who will likely develop HO-CDI in the intensive care unit (ICU) setting.

DESIGN

A prospective cohort study was conducted with patient carriage of toxigenic C. difficile identified by rectal swabs analyzed by anaerobic culture and polymerase chain reaction (PCR). A previously validated ML model using electronic health record data generated daily risk of HO-CDI for every patient. Swab results and risk predictions were compared to the eventual HO-CDI status.

PATIENTS

Adult inpatient admissions taking place in University of Michigan Hospitals' medical and surgical intensive care units and oncology wards between June 6th and October 8th, 2020.

RESULTS

In total, 2,979 admissions, representing 2,044 patients, were observed over the course of the study period, with 39 admissions developing HO-CDIs. Swab surveillance identified 9 true-positive and 87 false-positive HO-CDIs. The ML model identified 9 true-positive and 226 false-positive HO-CDIs; 8 of the true-positives identified by the model differed from those identified by the swab surveillance.

CONCLUSION

With limited resources, an ML model identified the same number of HO-CDI admissions as swab-based surveillance, though it generated more false-positives. The patients identified by the ML model were not yet colonized with C. difficile. Additionally, the ML model identifies at-risk admissions before disease onset, providing opportunities for prevention.

Integration of clinical demographics and routine laboratory analysis parameters for early prediction of gestational diabetes mellitus in the Chinese population

Gestational diabetes mellitus (GDM) is one of the most common complications in pregnancy, impairing both maternal and fetal health in short and long term. As early interventions are considered desirable to prevent GDM, this study aims to develop a simple-to-use nomogram based on multiple common risk factors from electronic medical health records (EMHRs). A total of 924 pregnant women whose EMHRs were available at Peking University International Hospital from January 2022 to October 2022 were included. Clinical demographics and routine laboratory analysis parameters at 8-12 weeks of gestation were collected. A novel nomogram was established based on the outcomes of multivariate logistic regression. The nomogram demonstrated powerful discrimination (the area under the receiver operating characteristic curve = 0.7542), acceptable agreement (Hosmer-Lemeshow test, P = 0.3214) and favorable clinical utility. The C-statistics of 10-Fold cross validation, Leave one out cross validation and Bootstrap were 0.7411, 0.7357 and 0.7318, respectively, indicating the stability of the nomogram. A novel nomogram based on easily-accessible parameters was developed to predict GDM in early pregnancy, which may provide a paradigm for repurposing clinical data and benefit the clinical management of GDM. There is a need for prospective multi-center studies to validate the nomogram before employing the nomogram in real-world clinical practice.

An integrated pipeline for prediction of Clostridioides difficile infection

With the expansion of electronic health records(EHR)-linked genomic data comes the development of machine learning-enable models. There is a pressing need to develop robust pipelines to evaluate the performance of integrated models and minimize systemic bias. We developed a prediction model of symptomatic Clostridioides difficile infection(CDI) by integrating common EHR-based and genetic risk factors(rs2227306/IL8). Our pipeline includes (1) leveraging phenotyping algorithm to minimize temporal bias, (2) performing simulation studies to determine the predictive power in samples without genetic information, (3) propensity score matching to control for the confoundings, (4) selecting machine learning algorithms to capture complex feature interactions, (5) performing oversampling to address data imbalance, and (6) optimizing models and ensuring proper bias-variance trade-off. We evaluate the performance of prediction models of CDI when including common clinical risk factors and the benefit of incorporating genetic feature(s) into the models. We emphasize the importance of building a robust integrated pipeline to avoid systemic bias and thoroughly evaluating genetic features when integrated into the prediction models in the general population and subgroups.

Enhancing Kidney Transplant Care through the Integration of Chatbot

Kidney transplantation is a critical treatment option for end-stage kidney disease patients, offering improved quality of life and increased survival rates. However, the complexities of kidney transplant care necessitate continuous advancements in decision making, patient communication, and operational efficiency. This article explores the potential integration of a sophisticated chatbot, an AI-powered conversational agent, to enhance kidney transplant practice and potentially improve patient outcomes. Chatbots and generative AI have shown promising applications in various domains, including healthcare, by simulating human-like interactions and generating contextually appropriate responses. Noteworthy AI models like ChatGPT by OpenAI, BingChat by Microsoft, and Bard AI by Google exhibit significant potential in supporting evidence-based research and healthcare decision making. The integration of chatbots in kidney transplant care may offer transformative possibilities. As a clinical decision support tool, it could provide healthcare professionals with real-time access to medical literature and guidelines, potentially enabling informed decision making and improved knowledge dissemination. Additionally, the chatbot has the potential to facilitate patient education by offering personalized and understandable information, addressing queries, and providing guidance on post-transplant care. Furthermore, under clinician or transplant pharmacist supervision, it has the potential to support post-transplant care and medication management by analyzing patient data, which may lead to tailored recommendations on dosages, monitoring schedules, and potential drug interactions. However, to fully ascertain its effectiveness and safety in these roles, further studies and validation are required. Its integration with existing clinical decision support systems may enhance risk stratification and treatment planning, contributing to more informed and efficient decision making in kidney transplant care. Given the importance of ethical considerations and bias mitigation in AI integration, future studies may evaluate long-term patient outcomes, cost-effectiveness, user experience, and the generalizability of chatbot recommendations. By addressing these factors and potentially leveraging AI capabilities, the integration of chatbots in kidney transplant care holds promise for potentially improving patient outcomes, enhancing decision making, and fostering the equitable and responsible use of AI in healthcare.

Prospective evaluation of data-driven models to predict daily risk of Clostridioides difficile infection at 2 large academic health centers

Many data-driven patient risk stratification models have not been evaluated prospectively. We performed and compared the prospective and retrospective evaluations of 2 Clostridioides difficile infection (CDI) risk-prediction models at 2 large academic health centers, and we discuss the models' robustness to data-set shifts.

Graph convolutional network-based fusion model to predict risk of hospital acquired infections

OBJECTIVE

Hospital acquired infections (HAIs) are one of the top 10 leading causes of death within the United States. While current standard of HAI risk prediction utilizes only a narrow set of predefined clinical variables, we propose a graph convolutional neural network (GNN)-based model which incorporates a wide variety of clinical features.

MATERIALS AND METHODS

Our GNN-based model defines patients' similarity based on comprehensive clinical history and demographics and predicts all types of HAI rather than focusing on a single subtype. An HAI model was trained on 38 327 unique hospitalizations while a distinct model for surgical site infection (SSI) prediction was trained on 18 609 hospitalization. Both models were tested internally and externally on a geographically disparate site with varying infection rates.

RESULTS

The proposed approach outperformed all baselines (single-modality models and length-of-stay [LoS]) with achieved area under the receiver operating characteristics of 0.86 [0.84-0.88] and 0.79 [0.75-0.83] (HAI), and 0.79 [0.75-0.83] and 0.76 [0.71-0.76] (SSI) for internal and external testing. Cost-effective analysis shows that the GNN modeling dominated the standard LoS model strategy on the basis of lower mean costs ($1651 vs $1915).

DISCUSSION

The proposed HAI risk prediction model can estimate individualized risk of infection for patient by taking into account not only the patient's clinical features, but also clinical features of similar patients as indicated by edges of the patients' graph.

CONCLUSIONS

The proposed model could allow prevention or earlier detection of HAI, which in turn could decrease hospital LoS and associated mortality, and ultimately reduce the healthcare cost.

Deep Learning Provides a New Magnetic Resonance Imaging-Based Prognostic Biomarker for Recurrence Prediction in High-Grade Serous Ovarian Cancer

This study aims to use a deep learning method to develop a signature extract from preoperative magnetic resonance imaging (MRI) and to evaluate its ability as a non-invasive recurrence risk prognostic marker in patients with advanced high-grade serous ovarian cancer (HGSOC). Our study comprises a total of 185 patients with pathologically confirmed HGSOC. A total of 185 patients were randomly assigned in a 5:3:2 ratio to a training cohort (n = 92), validation cohort 1 (n = 56), and validation cohort 2 (n = 37). We built a new deep learning network from 3839 preoperative MRI images (T2-weighted images and diffusion-weighted images) to extract HGSOC prognostic indicators. Following that, a fusion model including clinical and deep learning features is developed to predict patients' individual recurrence risk and 3-year recurrence likelihood. In the two validation cohorts, the consistency index of the fusion model was higher than both the deep learning model and the clinical feature model (0.752, 0.813 vs. 0.625, 0.600 vs. 0.505, 0.501). Among the three models, the fusion model had a higher AUC than either the deep learning model or the clinical model in validation cohorts 1 or 2 (AUC = was 0.986, 0.961 vs. 0.706, 0.676/0.506, 0.506). Using the DeLong method, the difference between them was statistically significant (p < 0.05). The Kaplan-Meier analysis distinguished two patient groups with high and low recurrence risk (p = 0.0008 and 0.0035, respectively). Deep learning may be a low-cost, non-invasive method for predicting risk for advanced HGSOC recurrence. Deep learning based on multi-sequence MRI serves as a prognostic biomarker for advanced HGSOC, which provides a preoperative model for predicting recurrence in HGSOC. Additionally, using the fusion model as a new prognostic analysis means that can use MRI data can be used without the need to follow-up the prognostic biomarker.

The future of automated infection detection: Innovation to transform practice (Part III/III)

Current methods of emergency-room-based syndromic surveillance were insufficient to detect early community spread of severe acute respiratory coronavirus virus 2 (SARS-CoV-2) in the United States, which slowed the infection prevention and control response to the novel pathogen. Emerging technologies and automated infection surveillance have the potential to improve upon current practice standards and to revolutionize the practice of infection detection, prevention and control both inside and outside of healthcare settings. Genomics, natural language processing, and machine learning can be leveraged to improve identification of transmission events and aid and evaluate outbreak response. In the near future, automated infection detection strategies can be used to advance a true "Learning Healthcare System" that will support near-real-time quality improvement efforts and advance the scientific basis for the practice of infection control.

Dynamic prediction of work status for workers with occupational injuries: assessing the value of longitudinal observations

OBJECTIVE

Occupational injuries (OIs) cause an immense burden on the US population. Prediction models help focus resources on those at greatest risk of a delayed return to work (RTW). RTW depends on factors that develop over time; however, existing methods only utilize information collected at the time of injury. We investigate the performance benefits of dynamically estimating RTW, using longitudinal observations of diagnoses and treatments collected beyond the time of initial injury.

MATERIALS AND METHODS

We characterize the difference in predictive performance between an approach that uses information collected at the time of initial injury (baseline model) and a proposed approach that uses longitudinal information collected over the course of the patient's recovery period (proposed model). To control the comparison, both models use the same deep learning architecture and differ only in the information used. We utilize a large longitudinal observation dataset of OI claims and compare the performance of the two approaches in terms of daily prediction of future work state (working vs not working). The performance of these two approaches was assessed in terms of the area under the receiver operator characteristic curve (AUROC) and expected calibration error (ECE).

RESULTS

After subsampling and applying inclusion criteria, our final dataset covered 294 103 OIs, which were split evenly between train, development, and test datasets (1/3, 1/3, 1/3). In terms of discriminative performance on the test dataset, the proposed model had an AUROC of 0.728 (90% confidence interval: 0.723, 0.734) versus the baseline's 0.591 (0.585, 0.598). The proposed model had an ECE of 0.004 (0.003, 0.005) versus the baseline's 0.016 (0.009, 0.018).

CONCLUSION

The longitudinal approach outperforms current practice and shows potential for leveraging observational data to dynamically update predictions of RTW in the setting of OI. This approach may enable physicians and workers' compensation programs to manage large populations of injured workers more effectively.

A proposal for developing a platform that evaluates algorithmic equity and accuracy

We are at a pivotal moment in the development of healthcare artificial intelligence (AI), a point at which enthusiasm for machine learning has not caught up with the scientific evidence to support the equity and accuracy of diagnostic and therapeutic algorithms. This proposal examines algorithmic biases, including those related to race, gender and socioeconomic status, and accuracy, including the paucity of prospective studies and lack of multisite validation. We then suggest solutions to these problems. We describe the Mayo Clinic, Duke University, Change Healthcare project that is evaluating 35.1 billion healthcare records for bias. And we propose ‘Ingredients’ style labels and an AI evaluation/testing system to help clinicians judge the merits of products and services that include algorithms. Said testing would include input data sources and types, dataset population composition, algorithm validation techniques, bias assessment evaluation and performance metrics.

A comparative analysis of machine learning approaches to predict C. difficile infection in hospitalized patients

BACKGROUND

Interventions to better prevent or manage Clostridioides difficile infection (CDI) may significantly reduce morbidity, mortality, and healthcare spending.

METHODS

We present a retrospective study using electronic health record data from over 700 United States hospitals. A subset of hospitals was used to develop machine learning algorithms (MLAs); the remaining hospitals served as an external test set. Three MLAs were evaluated: gradient-boosted decision trees (XGBoost), Deep Long Short Term Memory neural network, and one-dimensional convolutional neural network. MLA performance was evaluated with area under the receiver operating characteristic curve (AUROC), sensitivity, specificity, diagnostic odds ratios and likelihood ratios.

RESULTS

The development dataset contained 13,664,840 inpatient encounters with 80,046 CDI encounters; the external dataset contained 1,149,088 inpatient encounters with 7,107 CDI encounters. The highest AUROCs were achieved for XGB, Deep Long Short Term Memory neural network, and one-dimensional convolutional neural network via abstaining from use of specialized training techniques, resampling in isolation, and resampling and output bias in combination, respectively. XGBoost achieved the highest AUROC.

CONCLUSIONS

MLAs can predict future CDI in hospitalized patients using just 6 hours of data. In clinical practice, a machine-learning based tool may support prophylactic measures, earlier diagnosis, and more timely implementation of infection control measures.

Augmenting existing deterioration indices with chest radiographs to predict clinical deterioration

IMPORTANCE

When hospitals are at capacity, accurate deterioration indices could help identify low-risk patients as potential candidates for home care programs and alleviate hospital strain. To date, many existing deterioration indices are based entirely on structured data from the electronic health record (EHR) and ignore potentially useful information from other sources.

OBJECTIVE

To improve the accuracy of existing deterioration indices by incorporating unstructured imaging data from chest radiographs.

DESIGN, SETTING, AND PARTICIPANTS

Machine learning models were trained to predict deterioration of patients hospitalized with acute dyspnea using existing deterioration index scores and chest radiographs. Models were trained on hospitalized patients without coronavirus disease 2019 (COVID-19) and then subsequently tested on patients with COVID-19 between January 2020 and December 2020 at a single tertiary care center who had at least one radiograph taken within 48 hours of hospital admission.

MAIN OUTCOMES AND MEASURES

Patient deterioration was defined as the need for invasive or non-invasive mechanical ventilation, heated high flow nasal cannula, IV vasopressor administration or in-hospital mortality at any time following admission. The EPIC deterioration index was augmented with unstructured data from chest radiographs to predict risk of deterioration. We compared discriminative performance of the models with and without incorporating chest radiographs using area under the receiver operating curve (AUROC), focusing on comparing the fraction and total patients identified as low risk at different negative predictive values (NPV).

RESULTS

Data from 6278 hospitalizations were analyzed, including 5562 hospitalizations without COVID-19 (training cohort) and 716 with COVID-19 (216 in validation, 500 in held-out test cohort). At a NPV of 0.95, the best-performing image-augmented deterioration index identified 49 more (9.8%) individuals as low-risk compared to the deterioration index based on clinical data alone in the first 48 hours of admission. At a NPV of 0.9, the EPIC image-augmented deterioration index identified 26 more individuals (5.2%) as low-risk compared to the deterioration index based on clinical data alone in the first 48 hours of admission.

CONCLUSION AND RELEVANCE

Augmenting existing deterioration indices with chest radiographs results in better identification of low-risk patients. The model augmentation strategy could be used in the future to incorporate other forms of unstructured data into existing disease models.

Oral Vancomycin Prophylaxis for Primary and Secondary Prevention of Clostridioides difficile Infection in Patients Treated with Systemic Antibiotic Therapy: A Systematic Review, Meta-Analysis and Trial Sequential Analysis

Background: Clostridioides difficile infection (CDI) is associated with substantial morbidity and mortality as well as high propensity of recurrence. Systemic antibiotic therapy (SAT) represents the top inciting factor of CDI, both primary and recurrent (rCDI). Among the many strategies aimed to prevent CDI in high-risk subjects undergoing SAT, oral vancomycin prophylaxis (OVP) appears promising under a cost-effectiveness perspective. Methods: A systematic review with meta-analysis and trial sequential analysis (TSA) of studies assessing the efficacy and the safety of OVP to prevent primary CDI and rCDI in persons undergoing SAT was carried out. PubMed and EMBASE were searched until 30 September 2021. The protocol was pre-registered on PROSPERO (CRD42019145543). Results: Eleven studies met the inclusion criteria, only one being a randomized controlled trial (RCT). Overall, 929 subjects received OVP and 2011 represented the comparator group (no active prophylaxis). OVP exerted a strong protective effect for CDI occurrence: odds ratio 0.14, 95% confidence interval 0.04–0.38. Moderate heterogeneity was observed: I² 54%. This effect was confirmed throughout several subgroup analyses, including prevention of primary CDI versus rCDI. TSA results pointed at the conclusive nature of the evidence. Results were robust to a variety of sensitivity and quantitative bias analyses, although the underlying evidence was deemed as low quality. No differences between the two groups were highlighted regarding the onset of vancomycin-resistant Enterococcus infections. Conclusions: OVP appears to be an efficacious option for prevention of CDI in high-risk subjects undergoing SAT. Nevertheless, additional data from RCTs are needed to establish OVP as good clinical practice and define optimal dosage and duration.

Can prediction scores be used to identify patients at risk of Clostridioides difficile infection?

PURPOSE OF REVIEW

To describe the current state of literature on modeling risk of incident and recurrent Clostridioides difficile infection (iCDI and rCDI), to underscore limitations, and to propose a path forward for future research.

RECENT FINDINGS

There are many published risk factors and models for both iCDI and rCDI. The approaches include scores with a limited list of variables designed to be used at the bedside, but more recently have also included automated tools that take advantage of the entire electronic health record. Recent attempts to externally validate scores have met with mixed success.

SUMMARY

For iCDI, the performance largely hinges on the incidence, which even for hospitalized patients can be low (often <1%). Most scores fail to achieve high accuracy and/or are not externally validated. A challenge in predicting rCDI is the significant overlap with risk factors for iCDI, reducing the discriminatory ability of models. Automated electronic health record-based tools show promise but portability to other centers is challenging. Future studies should include external validation and consider biomarkers to augment performance.

Prediction across healthcare settings: a case study in predicting emergency department disposition

Several approaches exist today for developing predictive models across multiple clinical sites, yet there is a lack of comparative data on their performance, especially within the context of EHR-based prediction models. We set out to provide a framework for prediction across healthcare settings. As a case study, we examined an ED disposition prediction model across three geographically and demographically diverse sites. We conducted a 1-year retrospective study, including all visits in which the outcome was either discharge-to-home or hospitalization. Four modeling approaches were compared: a ready-made model trained at one site and validated at other sites, a centralized uniform model incorporating data from all sites, multiple site-specific models, and a hybrid approach of a ready-made model re-calibrated using site-specific data. Predictions were performed using XGBoost. The study included 288,962 visits with an overall admission rate of 16.8% (7.9–26.9%). Some risk factors for admission were prominent across all sites (e.g., high-acuity triage emergency severity index score, high prior admissions rate), while others were prominent at only some sites (multiple lab tests ordered at the pediatric sites, early use of ECG at the adult site). The XGBoost model achieved its best performance using the uniform and site-specific approaches (AUC = 0.9–0.93), followed by the calibrated-model approach (AUC = 0.87–0.92), and the ready-made approach (AUC = 0.62–0.85). Our results show that site-specific customization is a key driver of predictive model performance.

Trust in AI: why we should be designing for APPROPRIATE reliance

Use of artificial intelligence in healthcare, such as machine learning-based predictive algorithms, holds promise for advancing outcomes, but few systems are used in routine clinical practice. Trust has been cited as an important challenge to meaningful use of artificial intelligence in clinical practice. Artificial intelligence systems often involve automating cognitively challenging tasks. Therefore, previous literature on trust in automation may hold important lessons for artificial intelligence applications in healthcare. In this perspective, we argue that informatics should take lessons from literature on trust in automation such that the goal should be to foster appropriate trust in artificial intelligence based on the purpose of the tool, its process for making recommendations, and its performance in the given context. We adapt a conceptual model to support this argument and present recommendations for future work.

Prediction of in-hospital mortality of Clostriodiodes difficile infection using critical care database: a big data-driven, machine learning approach

RESEARCH OBJECTIVES

Clostriodiodes difficile infection (CDI) is a major cause of healthcare-associated diarrhoea with high mortality. There is a lack of validated predictors for severe outcomes in CDI. The aim of this study is to derive and validate a clinical prediction tool for CDI in-hospital mortality using a large critical care database.

METHODOLOGY

The demographics, clinical parameters, laboratory results and mortality of CDI were extracted from the Medical Information Mart for Intensive Care-III (MIMIC-III) database. We subsequently trained three machine learning models: logistic regression (LR), random forest (RF) and gradient boosting machine (GBM) to predict in-hospital mortality. The individual performances of the models were compared against current severity scores (Clostridiodes difficile Associated Risk of Death Score (CARDS) and ATLAS (Age, Treatment with systemic antibiotics, leukocyte count, Albumin and Serum creatinine as a measure of renal function) by calculating area under receiver operating curve (AUROC). We identified factors associated with higher mortality risk in each model.

SUMMARY OF RESULTS

From 61 532 intensive care unit stays in the MIMIC-III database, there were 1315 CDI cases. The mortality rate for CDI in the study cohort was 18.33%. AUROC was 0.69 (95% CI, 0.60 to 0.76) for LR, 0.71 (95% CI, 0.62 to 0.77) for RF and 0.72 (95% CI, 0.64 to 0.78) for GBM, while previously AUROC was 0.57 (95% CI, 0.51 to 0.65) for CARDS and 0.63 (95% CI, 0.54 to 0.70) for ATLAS. Albumin, lactate and bicarbonate were significant mortality factors for all the models. Free calcium, potassium, white blood cell, urea, platelet and mean blood pressure were present in at least two of the three models.

CONCLUSION

Our machine learning derived CDI in-hospital mortality prediction model identified pertinent factors that can assist critical care clinicians in identifying patients at high risk of dying from CDI.

An Osmotic Laxative Renders Mice Susceptible to Prolonged Clostridioides difficile Colonization and Hinders Clearance

Antibiotics are a major risk factor for Clostridioides difficile infections (CDIs) because of their impact on the microbiota. However, nonantibiotic medications such as the ubiquitous osmotic laxative polyethylene glycol 3350 (PEG 3350) also alter the microbiota. Clinicians also hypothesize that PEG helps clear C. difficile. But whether PEG impacts CDI susceptibility and clearance is unclear. To examine how PEG impacts susceptibility, we treated C57BL/6 mice with 5-day and 1-day doses of 15% PEG in the drinking water and then challenged the mice with C. difficile 630. We used clindamycin-treated mice as a control because they consistently clear C. difficile within 10 days postchallenge. PEG treatment alone was sufficient to render mice susceptible, and 5-day PEG-treated mice remained colonized for up to 30 days postchallenge. In contrast, 1-day PEG-treated mice were transiently colonized, clearing C. difficile within 7 days postchallenge. To examine how PEG treatment impacts clearance, we administered a 1-day PEG treatment to clindamycin-treated, C. difficile-challenged mice. Administering PEG to mice after C. difficile challenge prolonged colonization up to 30 days postchallenge. When we trained a random forest model with community data from 5 days postchallenge, we were able to predict which mice would exhibit prolonged colonization (area under the receiver operating characteristic curve [AUROC] = 0.90). Examining the dynamics of these bacterial populations during the postchallenge period revealed patterns in the relative abundances of Bacteroides, Enterobacteriaceae, Porphyromonadaceae, Lachnospiraceae, and Akkermansia that were associated with prolonged C. difficile colonization in PEG-treated mice. Thus, the osmotic laxative PEG rendered mice susceptible to C. difficile colonization and hindered clearance.

IMPORTANCE Diarrheal samples from patients taking laxatives are typically rejected for Clostridioides difficile testing. However, there are similarities between the bacterial communities from people with diarrhea and those with C. difficile infections (CDIs), including lower diversity than the communities from healthy patients. This observation led us to hypothesize that diarrhea may be an indicator of C. difficile susceptibility. We explored how osmotic laxatives disrupt the microbiota’s colonization resistance to C. difficile by administering a laxative to mice either before or after C. difficile challenge. Our findings suggest that osmotic laxatives disrupt colonization resistance to C. difficile and prevent clearance among mice already colonized with C. difficile. Considering that most hospitals recommend not performing C. difficile testing on patients taking laxatives, and laxatives are prescribed prior to administering fecal microbiota transplants via colonoscopy to patients with recurrent CDIs, further studies are needed to evaluate if laxatives impact microbiota colonization resistance in humans.

Evaluating a Widely Implemented Proprietary Deterioration Index Model among Hospitalized Patients with COVID-19

Rationale: The Epic Deterioration Index (EDI) is a proprietary prediction model implemented in over 100 U.S. hospitals that was widely used to support medical decision-making during the coronavirus disease (COVID-19) pandemic. The EDI has not been independently evaluated, and other proprietary models have been shown to be biased against vulnerable populations. Objectives: To independently evaluate the EDI in hospitalized patients with COVID-19 overall and in disproportionately affected subgroups. Methods: We studied adult patients admitted with COVID-19 to units other than the intensive care unit at a large academic medical center from March 9 through May 20, 2020. We used the EDI, calculated at 15-minute intervals, to predict a composite outcome of intensive care unit-level care, mechanical ventilation, or in-hospital death. In a subset of patients hospitalized for at least 48 hours, we also evaluated the ability of the EDI to identify patients at low risk of experiencing this composite outcome during their remaining hospitalization. Results: Among 392 COVID-19 hospitalizations meeting inclusion criteria, 103 (26%) met the composite outcome. The median age of the cohort was 64 (interquartile range, 53-75) with 168 (43%) Black patients and 169 (43%) women. The area under the receiver-operating characteristic curve of the EDI was 0.79 (95% confidence interval, 0.74-0.84). EDI predictions did not differ by race or sex. When exploring clinically relevant thresholds of the EDI, we found patients who met or exceeded an EDI of 68.8 made up 14% of the study cohort and had a 74% probability of experiencing the composite outcome during their hospitalization with a sensitivity of 39% and a median lead time of 24 hours from when this threshold was first exceeded. Among the 286 patients hospitalized for at least 48 hours who had not experienced the composite outcome, 14 (13%) never exceeded an EDI of 37.9, with a negative predictive value of 90% and a sensitivity above this threshold of 91%. Conclusions: We found the EDI identifies small subsets of high-risk and low-risk patients with COVID-19 with good discrimination, although its clinical use as an early warning system is limited by low sensitivity. These findings highlight the importance of independent evaluation of proprietary models before widespread operational use among patients with COVID-19.

External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients

IMPORTANCE

The Epic Sepsis Model (ESM), a proprietary sepsis prediction model, is implemented at hundreds of US hospitals. The ESM's ability to identify patients with sepsis has not been adequately evaluated despite widespread use.

OBJECTIVE

To externally validate the ESM in the prediction of sepsis and evaluate its potential clinical value compared with usual care.

DESIGN, SETTING, AND PARTICIPANTS

This retrospective cohort study was conducted among 27 697 patients aged 18 years or older admitted to Michigan Medicine, the academic health system of the University of Michigan, Ann Arbor, with 38 455 hospitalizations between December 6, 2018, and October 20, 2019.

EXPOSURE

The ESM score, calculated every 15 minutes.

MAIN OUTCOMES AND MEASURES

Sepsis, as defined by a composite of (1) the Centers for Disease Control and Prevention surveillance criteria and (2) International Statistical Classification of Diseases and Related Health Problems, Tenth Revision diagnostic codes accompanied by 2 systemic inflammatory response syndrome criteria and 1 organ dysfunction criterion within 6 hours of one another. Model discrimination was assessed using the area under the receiver operating characteristic curve at the hospitalization level and with prediction horizons of 4, 8, 12, and 24 hours. Model calibration was evaluated with calibration plots. The potential clinical benefit associated with the ESM was assessed by evaluating the added benefit of the ESM score compared with contemporary clinical practice (based on timely administration of antibiotics). Alert fatigue was evaluated by comparing the clinical value of different alerting strategies.

RESULTS

We identified 27 697 patients who had 38 455 hospitalizations (21 904 women [57%]; median age, 56 years [interquartile range, 35-69 years]) meeting inclusion criteria, of whom sepsis occurred in 2552 (7%). The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64). The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice. The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.

CONCLUSIONS AND RELEVANCE

This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis. The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.

Predicting optimal treatment regimens for patients with HR+/HER2- breast cancer using machine learning based on electronic health records

Aim: To predict optimal treatments maximizing overall survival (OS) and time to treatment discontinuation (TTD) for patients with metastatic breast cancer (MBC) using machine learning methods on electronic health records. Patients/methods: Adult females with HR+/HER2- MBC on first- or second-line systemic therapy were eligible. Random survival forest (RSF) models were used to predict optimal regimen classes for individual patients and each line of therapy based on baseline characteristics. Results: RSF models suggested greater use of CDK4 & 6 inhibitor-based therapies may maximize OS and TTD. RSF-predicted optimal treatments demonstrated longer OS and TTD compared with nonoptimal treatments across line of therapy (hazard ratios = 0.44∼0.79). Conclusion: RSF may help inform optimal treatment choices and improve outcomes for patients with HR+/HER2- MBC.

Predicting postoperative opioid use with machine learning and insurance claims in opioid-naïve patients

BACKGROUND

The clinical impact of postoperative opioid use requires accurate prediction strategies to identify at-risk patients. We utilize preoperative claims data to predict postoperative opioid refill and new persistent use in opioid-naïve patients.

METHODS

A retrospective study was conducted on 112,898 opioid-naïve adult postoperative patients from Optum's de-identified Clinformatics® Data Mart database. Potential predictors included sociodemographic data, comorbidities, and prescriptions within one year prior to surgery.

RESULTS

Compared to linear models, non-linear models led to modest improvements in predicting refills - area under the receiver operating characteristics curve (AUROC) 0.68 vs. 0.67 (p < 0.05) - and performed identically in predicting new persistent use - AUROC = 0.66. Undergoing major surgery, opioid prescriptions within 30 days prior to surgery, and abdominal pain were useful in predicting refills; back/joint/head pain were the most important features in predicting new persistent use.

CONCLUSIONS

Preoperative patient attributes from insurance claims could potentially be useful in guiding prescription practices for opioid-naïve patients.

Predicting Acute Graft-Versus-Host Disease Using Machine Learning and Longitudinal Vital Sign Data From Electronic Health Records

PURPOSE

Acute graft-versus-host disease (aGVHD) remains a significant complication of allogeneic hematopoietic cell transplantation (HCT) and limits its broader application. The ability to predict grade II to IV aGVHD could potentially mitigate morbidity and mortality. To date, researchers have focused on using snapshots of a patient (eg, biomarkers at a single time point) to predict aGVHD onset. We hypothesized that longitudinal data collected and stored in electronic health records (EHRs) could distinguish patients at high risk of developing aGVHD from those at low risk.

PATIENTS AND METHODS

The study included a cohort of 324 patients undergoing allogeneic HCT at the University of Michigan C.S. Mott Children’s Hospital during 2014 to 2017. Using EHR data, specifically vital sign measurements collected within the first 10 days of transplantation, we built a predictive model using penalized logistic regression for identifying patients at risk for grade II to IV aGVHD. We compared the proposed model with a baseline model trained only on patient and donor characteristics collected at the time of transplantation and performed an analysis of the importance of different input features.

RESULTS

The proposed model outperformed the baseline model, with an area under the receiver operating characteristic curve of 0.659 versus 0.512 (P = .019). The feature importance analysis showed that the learned model relied most on temperature and systolic blood pressure, and temporal trends (eg, increasing or decreasing) were more important than the average values.

CONCLUSION

Leveraging readily available clinical data from EHRs, we developed a machine-learning model for aGVHD prediction in patients undergoing HCT. Continuous monitoring of vital signs, such as temperature, could potentially help clinicians more accurately identify patients at high risk for aGVHD.

A Risk Score to Predict Clostridioides difficile Infection

Background

Clostridioides difficile infection (CDI) is a major cause of severe diarrhea. In this retrospective study, we identified CDI risk factors by comparing demographic and clinical characteristics for Kaiser Permanente Northern California members ≥18 years old with and without laboratory-confirmed incident CDI.

Methods

We included these risk factors in logistic regression models to develop 2 risk scores that predict future CDI after an Index Date for Risk Score Assessment (IDRSA), marking the beginning of a period for which we estimated CDI risk.

Results

During May 2011 to July 2014, we included 9986 CDI cases and 2 230 354 members without CDI. The CDI cases tended to be older, female, white race, and have more hospitalizations, emergency department and office visits, skilled nursing facility stays, antibiotic and proton pump inhibitor use, and specific comorbidities. Using hospital discharge as the IDRSA, our risk score model yielded excellent performance in predicting the likelihood of developing CDI in the subsequent 31–365 days (C-statistic of 0.848). Using a random date as the IDRSA, our model also predicted CDI risk in the subsequent 31–365 days reasonably well (C–statistic 0.722).

Conclusions

These results can be used to identify high-risk populations for enrollment in C difficile vaccine trials and facilitate study feasibility regarding sample size and time to completion.

Systematic review on the definition and predictors of severe Clostridiodes difficile infection

Clostridiodes difficile infection (CDI) is one of the most common hospital-acquired infections with high mortality rates. Optimal management of CDI depends on early recognition of severity. However, currently, there is no acceptable standard of prediction. We reviewed severe CDI predictors in published literature and its definition according to clinical guidelines. We systematically reviewed studies describing clinical predictors for severe CDI in medical databases (Cochrane, EMBASE, Global Health Library, and MEDLINE/PubMed). They were independently evaluated by two reviewers. Six hundred thirty-three titles and abstracts were screened, and 31 studies were included. We excluded studies that were restricted to a specific patient population. There were 16 articles that examined mortality in CDI, as compared with 15 articles investigating non-mortality outcomes of CDI. The commonest risk factors identified were comorbidities, white blood cell count, serum albumin level, age, serum creatinine level and intensive care unit admission. Generally, the studies had small patient populations, were retrospective in nature, and mostly from Western centers. The commonest severe CDI criteria in clinical guidelines were raised white blood cell count, followed by low serum albumin and raised serum creatinine levels. There was no commonly agreed upon definition of severe CDI severity in the literature. Current clinical guidelines' definitions for severe CDI are heterogeneous. Hence, there is a need for prospective multi-center studies using standardized protocol for biospecimen investigation collection and shared data on outcomes of patients in order to devise a universally accepted definition for severe CDI.

Democratizing EHR analyses with FIDDLE: a flexible data-driven preprocessing pipeline for structured clinical data

OBJECTIVE

In applying machine learning (ML) to electronic health record (EHR) data, many decisions must be made before any ML is applied; such preprocessing requires substantial effort and can be labor-intensive. As the role of ML in health care grows, there is an increasing need for systematic and reproducible preprocessing techniques for EHR data. Thus, we developed FIDDLE (Flexible Data-Driven Pipeline), an open-source framework that streamlines the preprocessing of data extracted from the EHR.

MATERIALS AND METHODS

Largely data-driven, FIDDLE systematically transforms structured EHR data into feature vectors, limiting the number of decisions a user must make while incorporating good practices from the literature. To demonstrate its utility and flexibility, we conducted a proof-of-concept experiment in which we applied FIDDLE to 2 publicly available EHR data sets collected from intensive care units: MIMIC-III and the eICU Collaborative Research Database. We trained different ML models to predict 3 clinically important outcomes: in-hospital mortality, acute respiratory failure, and shock. We evaluated models using the area under the receiver operating characteristics curve (AUROC), and compared it to several baselines.

RESULTS

Across tasks, FIDDLE extracted 2,528 to 7,403 features from MIMIC-III and eICU, respectively. On all tasks, FIDDLE-based models achieved good discriminative performance, with AUROCs of 0.757-0.886, comparable to the performance of MIMIC-Extract, a preprocessing pipeline designed specifically for MIMIC-III. Furthermore, our results showed that FIDDLE is generalizable across different prediction times, ML algorithms, and data sets, while being relatively robust to different settings of user-defined arguments.

CONCLUSIONS

FIDDLE, an open-source preprocessing pipeline, facilitates applying ML to structured EHR data. By accelerating and standardizing labor-intensive preprocessing, FIDDLE can help stimulate progress in building clinically useful ML tools for EHR data.

Methodology minute: a machine learning primer for infection prevention and control

•

Use of machine learning (ML) is becoming more common in healthcare.

•

A general understanding of ML is necessary for IPs to make informed decisions about its use.

•

Here, we provide a brief overview of salient concepts in ML for infection prevention.

The use of machine-learning and predictive modeling in infection prevention and control activities is increasing dramatically. In order for infection preventionists to make informed decisions on the performance of any particular model as well as to determine if the output of the model will be useful for their program needs, a suitable understanding of the creation and evaluation of these models is necessary. The purpose of this primer is to introduce the infection preventionist to the most commonly used machine-learning method in infection prevention: supervised learning.

Economic Value of Data and Analytics for Health Care Providers: Hermeneutic Systematic Literature Review

BACKGROUND

The benefits of data and analytics for health care systems and single providers is an increasingly investigated field in digital health literature. Electronic health records (EHR), for example, can improve quality of care. Emerging analytics tools based on artificial intelligence show the potential to assist physicians in day-to-day workflows. Yet, single health care providers also need information regarding the economic impact when deciding on potential adoption of these tools.

OBJECTIVE

This paper examines the question of whether data and analytics provide economic advantages or disadvantages for health care providers. The goal is to provide a comprehensive overview including a variety of technologies beyond computer-based patient records. Ultimately, findings are also intended to determine whether economic barriers for adoption by providers could exist.

METHODS

A systematic literature search of the PubMed and Google Scholar online databases was conducted, following the hermeneutic methodology that encourages iterative search and interpretation cycles. After applying inclusion and exclusion criteria to 165 initially identified studies, 50 were included for qualitative synthesis and topic-based clustering.

RESULTS

The review identified 5 major technology categories, namely EHRs (n=30), computerized clinical decision support (n=8), advanced analytics (n=5), business analytics (n=5), and telemedicine (n=2). Overall, 62% (31/50) of the reviewed studies indicated a positive economic impact for providers either via direct cost or revenue effects or via indirect efficiency or productivity improvements. When differentiating between categories, however, an ambiguous picture emerged for EHR, whereas analytics technologies like computerized clinical decision support and advanced analytics predominantly showed economic benefits.

CONCLUSIONS

The research question of whether data and analytics create economic benefits for health care providers cannot be answered uniformly. The results indicate ambiguous effects for EHRs, here representing data, and mainly positive effects for the significantly less studied analytics field. The mixed results regarding EHRs can create an economic barrier for adoption by providers. This barrier can translate into a bottleneck to positive economic effects of analytics technologies relying on EHR data. Ultimately, more research on economic effects of technologies other than EHRs is needed to generate a more reliable evidence base.

Predicting antibiotic resistance in hospitalized patients by applying machine learning to electronic medical records

BACKGROUND

Computerized decision support systems are becoming increasingly prevalent with advances in data collection and machine learning algorithms. However, they are scarcely used for empiric antibiotic therapy. Here we accurately predict the antibiotic resistance profiles of bacterial infections of hospitalized patients using machine learning algorithms applied to patients' electronic medical records (EMR).

METHODS

The data included antibiotic resistance results of bacterial cultures from hospitalized patients, alongside their electronic medical records. Five antibiotics were examined: Ceftazidime (n=2942), Gentamicin (n=4360), Imipenem (n=2235), Ofloxacin (n=3117) and Sulfamethoxazole-Trimethoprim (n=3544). We applied lasso logistic regression, neural networks, gradient boosted trees, and an ensemble combining all three algorithms, to predict antibiotic resistance. Variable influence was gauged by permutation tests and Shapely Additive Explanations analysis.

RESULTS

The ensemble model outperformed the separate models and produced accurate predictions on a test set data. When no knowledge regarding the infecting bacterial species was assumed, the ensemble model yielded area under the receiver-operating-characteristic (auROC) scores of 0.73-0.79, for different antibiotics. Including information regarding the bacterial species improved the auROCs to 0.8-0.88. The effects of different variables on the predictions were assessed and found consistent with previously identified risk factors for antibiotic resistance.

CONCLUSIONS

Our study demonstrates the potential of machine learning models to accurately predict antibiotic resistance of bacterial infections of hospitalized patients. Moreover, we show that rapid information regarding the infecting bacterial species can improve predictions substantially. The implementation of such systems should be seriously considered by clinicians to aid correct empiric therapy and to potentially reduce antibiotic misuse.

The Promise of Clinical Decision Support Systems Targetting Low-Resource Settings

Low-resource clinical settings are plagued by low physician-to-patient ratios and a shortage of high-quality medical expertise and infrastructure. Together, these phenomena lead to over-burdened healthcare systems that under-serve the needs of the community. Alleviating this burden can be undertaken by the introduction of clinical decision support systems (CDSSs); systems that support stakeholders (ranging from physicians to patients) within the clinical setting in their day-to-day activities. Such systems, which have proven to be effective in the developed world, remain to be under-explored in low-resource settings. This review attempts to summarize the research focused on clinical decision support systems that either target stakeholders within low-resource clinical settings or diseases commonly found in such environments. When categorizing our findings according to disease applications, we find that CDSSs are predominantly focused on dealing with bacterial infections and maternal care, do not leverage deep learning, and have not been evaluated prospectively. Together, these highlight the need for increased research in this domain in order to impact a diverse set of medical conditions and ultimately improve patient outcomes.

Development of a Machine Learning Model for Survival Risk Stratification of Patients With Advanced Oral Cancer

IMPORTANCE

A tool for precisely stratifying postoperative patients with advanced oral cancer is crucial for the treatment plan, such as intensifying or deintensifying the regimen to improve their quality of life and prognosis.

OBJECTIVE

To develop and validate a machine learning-based algorithm that can provide survival risk stratification for patients with advanced oral cancer who have comprehensive clinicopathologic and genetic data.

DESIGN, SETTING, AND PARTICIPANTS

In this prognostic cohort study, the elastic net penalized Cox proportional hazards regression-based risk stratification model was developed and validated using single-center data collected between January 1, 1996, and December 31, 2011. In total, comprehensive clinicopathologic and genetic data (including clinical, pathologic, and 44 cancer-related gene variant profiles) of 334 patients with stage III or IV oral squamous cell carcinoma were used to develop and validate the algorithm in this 15-year cohort study. Data analysis was conducted between February 1, 2018, and May 6, 2020.

MAIN OUTCOMES AND MEASURES

The main outcomes were cancer-specific survival, distant metastasis-free survival, and locoregional recurrence-free survival. Model performance was compared in terms of the Akaike information criterion and the Harrell concordance index (C index).

RESULTS

Complete data were available for 334 patients (315 men; median age at onset, 48 years [interquartile range, 42-56 years]). The predictive models using comprehensive clinicopathologic and genetic data outperformed those using clinicopathologic data alone. In the groups of postoperative patients receiving adjuvant concurrent chemoradiotherapy, the models demonstrated higher classification performance than those using clinicopathologic data alone in cancer-specific survival (mean [SD] C index, 0.689 [0.050] vs 0.673 [0.051]; P = .02) and locoregional recurrence-free survival (mean [SD] C index, 0.693 [0.039] vs 0.678 [0.035]; P = .004). The classification performance in distant metastasis-free survival was not different (mean [SD] C index, 0.702 [0.056] vs 0.688 [0.048]; P = .09).

CONCLUSIONS AND RELEVANCE

A risk stratification model using comprehensive clinicopathologic and genetic data accurately differentiated the high-risk group from the low-risk group in cancer-specific survival and locoregional recurrence-free survival for postoperative patients with advanced oral cancer. This algorithm could be used through an online calculator to provide additional personalized information for postoperative management of patients with advanced oral squamous cell carcinoma.

Advances in Data-Driven Responses to Preventing Spread of Antibiotic Resistance Across Health-Care Settings

Among the most urgent and serious threats to public health are 7 antibiotic-resistant bacterial infections predominately acquired during health-care delivery. There is an emerging field of health-care epidemiology that is focused on preventing health care-associated infections with antibiotic-resistant bacteria and incorporates data from patient transfers or patient movements within and between facilities. This analytic field is being used to help public health professionals identify best opportunities for prevention. Different analytic approaches that draw on uses of big data are being explored to help target the use of limited public health resources, leverage expertise, and enact effective policy to maximize an impact on population-level health. Here, the following recent advances in data-driven responses to preventing spread of antibiotic resistance across health-care settings are summarized: leveraging big data for machine learning, integration or advances in tracking patient movement, and highlighting the value of coordinating response across institutions within a region.

Classification of Current Procedural Terminology Codes from Electronic Health Record Data Using Machine Learning

BACKGROUND

Accurate anesthesiology procedure code data are essential to quality improvement, research, and reimbursement tasks within anesthesiology practices. Advanced data science techniques, including machine learning and natural language processing, offer opportunities to develop classification tools for Current Procedural Terminology codes across anesthesia procedures.

METHODS

Models were created using a Train/Test dataset including 1,164,343 procedures from 16 academic and private hospitals. Five supervised machine learning models were created to classify anesthesiology Current Procedural Terminology codes, with accuracy defined as first choice classification matching the institutional-assigned code existing in the perioperative database. The two best performing models were further refined and tested on a Holdout dataset from a single institution distinct from Train/Test. A tunable confidence parameter was created to identify cases for which models were highly accurate, with the goal of at least 95% accuracy, above the reported 2018 Centers for Medicare and Medicaid Services (Baltimore, Maryland) fee-for-service accuracy. Actual submitted claim data from billing specialists were used as a reference standard.

RESULTS

Support vector machine and neural network label-embedding attentive models were the best performing models, respectively, demonstrating overall accuracies of 87.9% and 84.2% (single best code), and 96.8% and 94.0% (within top three). Classification accuracy was 96.4% in 47.0% of cases using support vector machine and 94.4% in 62.2% of cases using label-embedding attentive model within the Train/Test dataset. In the Holdout dataset, respective classification accuracies were 93.1% in 58.0% of cases and 95.0% among 62.0%. The most important feature in model training was procedure text.

CONCLUSIONS

Through application of machine learning and natural language processing techniques, highly accurate real-time models were created for anesthesiology Current Procedural Terminology code classification. The increased processing speed and a priori targeted accuracy of this classification approach may provide performance optimization and cost reduction for quality improvement, research, and reimbursement tasks reliant on anesthesiology procedure codes.