Effect of deep learning-based assistive technology use on chest radiograph interpretation by emergency department physicians: a prospective interventional simulation-based study

The role of information systems in emergency department decision-making-a literature review

OBJECTIVES

Healthcare providers employ heuristic and analytical decision-making to navigate the high-stakes environment of the emergency department (ED). Despite the increasing integration of information systems (ISs), research on their efficacy is conflicting. Drawing on related fields, we investigate how timing and mode of delivery influence IS effectiveness. Our objective is to reconcile previous contradictory findings, shedding light on optimal IS design in the ED.

MATERIALS AND METHODS

We conducted a systematic review following PRISMA across PubMed, Scopus, and Web of Science. We coded the ISs' timing as heuristic or analytical, their mode of delivery as active for automatic alerts and passive when requiring user-initiated information retrieval, and their effect on process, economic, and clinical outcomes.

RESULTS

Our analysis included 83 studies. During early heuristic decision-making, most active interventions were ineffective, while passive interventions generally improved outcomes. In the analytical phase, the effects were reversed. Passive interventions that facilitate information extraction consistently improved outcomes.

DISCUSSION

Our findings suggest that the effectiveness of active interventions negatively correlates with the amount of information received during delivery. During early heuristic decision-making, when information overload is high, physicians are unresponsive to alerts and proactively consult passive resources. In the later analytical phases, physicians show increased receptivity to alerts due to decreased diagnostic uncertainty and information quantity. Interventions that limit information lead to positive outcomes, supporting our interpretation.

CONCLUSION

We synthesize our findings into an integrated model that reveals the underlying reasons for conflicting findings from previous reviews and can guide practitioners in designing ISs in the ED.

Implementation of Artificial Intelligence in Personalized Prognostic Assessment of Lung Cancer: A Narrative Review

Artificial Intelligence (AI) has revolutionized the management of non-small-cell lung cancer (NSCLC) by enhancing different aspects, including staging, prognosis assessment, treatment prediction, response evaluation, recurrence/prognosis prediction, and personalized prognostic assessment. AI algorithms may accurately classify NSCLC stages using machine learning techniques and deep imaging data analysis. This could potentially improve precision and efficiency in staging, facilitating personalized treatment decisions. Furthermore, there are data suggesting the potential application of AI-based models in predicting prognosis in terms of survival rates and disease progression by integrating clinical, imaging and molecular data. In the present narrative review, we will analyze the preliminary studies reporting on how AI algorithms could predict responses to various treatment modalities, such as surgery, radiotherapy, chemotherapy, targeted therapy, and immunotherapy. There is robust evidence suggesting that AI also plays a crucial role in predicting the likelihood of tumor recurrence after surgery and the pattern of failure, which has significant implications for tailoring adjuvant treatments. The successful implementation of AI in personalized prognostic assessment requires the integration of different data sources, including clinical, molecular, and imaging data. Machine learning (ML) and deep learning (DL) techniques enable AI models to analyze these data and generate personalized prognostic predictions, allowing for a precise and individualized approach to patient care. However, challenges relating to data quality, interpretability, and the ability of AI models to generalize need to be addressed. Collaboration among clinicians, data scientists, and regulators is critical for the responsible implementation of AI and for maximizing its benefits in providing a more personalized prognostic assessment. Continued research, validation, and collaboration are essential to fully exploit the potential of AI in NSCLC management and improve patient outcomes. Herein, we have summarized the state of the art of applications of AI in lung cancer for predicting staging, prognosis, and pattern of recurrence after treatment in order to provide to the readers a large comprehensive overview of this challenging issue.

Designing Chinese hospital emergency departments to leverage artificial intelligence-a systematic literature review on the challenges and opportunities

Artificial intelligence (AI) has witnessed rapid advances in the healthcare domain in recent years, especially in the emergency field, where AI is likely to radically reshape medical service delivery. Although AI has substantial potential to enhance diagnostic accuracy and operational efficiency in hospitals, research on its applications in Emergency Department building design remains relatively scarce. Therefore, this study aims to investigate Emergency Department facility design by identifying the challenges and opportunities of using AI. Two systematic literature reviews are combined, one in AI and the other in sensors, to explore their potential application to support decision-making, resource optimisation and patient monitoring. These reviews have then informed a discussion on integrating AI sensors in contemporary Emergency Department designs for use in China to support the evidence base on resuscitation units, emergency operating rooms and Emergency Department Intensive Care Unit (ED-ICU) design. We hope to inform the strategic implementation of AI sensors and how they might transform Emergency Department design to support medical staff and enhance the patient experience.

Real-world testing of an artificial intelligence algorithm for the analysis of chest X-rays in primary care settings

Interpreting chest X-rays is a complex task, and artificial intelligence algorithms for this purpose are currently being developed. It is important to perform external validations of these algorithms in order to implement them. This study therefore aims to externally validate an AI algorithm's diagnoses in real clinical practice, comparing them to a radiologist's diagnoses. The aim is also to identify diagnoses the algorithm may not have been trained for. A prospective observational study for the external validation of the AI algorithm in a region of Catalonia, comparing the AI algorithm's diagnosis with that of the reference radiologist, considered the gold standard. The external validation was performed with a sample of 278 images and reports, 51.8% of which showed no radiological abnormalities according to the radiologist's report. Analysing the validity of the AI algorithm, the average accuracy was 0.95 (95% CI 0.92; 0.98), the sensitivity was 0.48 (95% CI 0.30; 0.66) and the specificity was 0.98 (95% CI 0.97; 0.99). The conditions where the algorithm was most sensitive were external, upper abdominal and cardiac and/or valvular implants. On the other hand, the conditions where the algorithm was less sensitive were in the mediastinum, vessels and bone. The algorithm has been validated in the primary care setting and has proven to be useful when identifying images with or without conditions. However, in order to be a valuable tool to help and support experts, it requires additional real-world training to enhance its diagnostic capabilities for some of the conditions analysed. Our study emphasizes the need for continuous improvement to ensure the algorithm's effectiveness in primary care.

Effect of multimodal diagnostic approach using deep learning-based automated detection algorithm for active pulmonary tuberculosis

In this study, we developed a model to predict culture test results for pulmonary tuberculosis (PTB) with a customized multimodal approach and evaluated its performance in different clinical settings. Moreover, we investigated potential performance improvements by combining this approach with deep learning-based automated detection algorithms (DLADs). This retrospective observational study enrolled patients over 18 years of age who consecutively visited the level 1 emergency department and underwent chest radiograph and sputum testing. The primary endpoint was positive sputum culture for PTB. We compared the performance of the diagnostic models by replacing radiologists' interpretations of chest radiographs with screening scores calculated through DLAD. The optimal diagnostic model had an area under the receiver operating characteristic curve of 0.924 (95% CI 0.871-0.976) and an area under precision recall curve of 0.403 (95% CI 0.195-0.580) while maintaining a specificity of 81.4% when sensitivity was fixed at 90%. Multicomponent models showed improved performance for detecting PTB when chest radiography interpretation was replaced by DLAD. Multicomponent diagnostic models with DLAD customized for different clinical settings are more practical than traditional methods for detecting patients with PTB. This novel diagnostic approach may help prevent the spread of PTB and optimize healthcare resource utilization in resource-limited clinical settings.

Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings

AIMS AND OBJECTIVES

To examine the nature and use of automation in contemporary clinical information systems by reviewing studies reporting the implementation and evaluation of artificial intelligence (AI) technologies in healthcare settings.

METHOD

PubMed/MEDLINE, Web of Science, EMBASE, the tables of contents of major informatics journals, and the bibliographies of articles were searched for studies reporting evaluation of AI in clinical settings from January 2021 to December 2022. We documented the clinical application areas and tasks supported, and the level of system autonomy. Reported effects on user experience, decision-making, care delivery and outcomes were summarised.

RESULTS

AI technologies are being applied in a wide variety of clinical areas. Most contemporary systems utilise deep learning, use routinely collected data, support diagnosis and triage, are assistive (requiring users to confirm or approve AI provided information or decisions), and are used by doctors in acute care settings in high-income nations. AI systems are integrated and used within existing clinical information systems including electronic medical records. There is limited support for One Health goals. Evaluation is largely based on quantitative methods measuring effects on decision-making.

CONCLUSION

AI systems are being implemented and evaluated in many clinical areas. There remain many opportunities to understand patterns of routine use and evaluate effects on decision-making, care delivery and patient outcomes using mixed-methods. Support for One Health including integrating data about environmental factors and social determinants needs further exploration.

The Application of Design Thinking in Developing a Deep Learning Algorithm for Hip Fracture Detection

(1) Background: Design thinking is a problem-solving approach that has been applied in various sectors, including healthcare and medical education. While deep learning (DL) algorithms can assist in clinical practice, integrating them into clinical scenarios can be challenging. This study aimed to use design thinking steps to develop a DL algorithm that accelerates deployment in clinical practice and improves its performance to meet clinical requirements. (2) Methods: We applied the design thinking process to interview clinical doctors and gain insights to develop and modify the DL algorithm to meet clinical scenarios. We also compared the DL performance of the algorithm before and after the integration of design thinking. (3) Results: After empathizing with clinical doctors and defining their needs, we identified the unmet need of five trauma surgeons as "how to reduce the misdiagnosis of femoral fracture by pelvic plain film (PXR) at initial emergency visiting". We collected 4235 PXRs from our hospital, of which 2146 had a hip fracture (51%) from 2008 to 2016. We developed hip fracture DL detection models based on the Xception convolutional neural network by using these images. By incorporating design thinking, we improved the diagnostic accuracy from 0.91 (0.84-0.96) to 0.95 (0.93-0.97), the sensitivity from 0.97 (0.89-1.00) to 0.97 (0.94-0.99), and the specificity from 0.84 (0.71-0.93) to 0.93(0.990-0.97). (4) Conclusions: In summary, this study demonstrates that design thinking can ensure that DL solutions developed for trauma care are user-centered and meet the needs of patients and healthcare providers.

Multicentre external validation of a commercial artificial intelligence software to analyse chest radiographs in health screening environments with low disease prevalence

OBJECTIVES

To externally validate the performance of a commercial AI software program for interpreting CXRs in a large, consecutive, real-world cohort from primary healthcare centres.

METHODS

A total of 3047 CXRs were collected from two primary healthcare centres, characterised by low disease prevalence, between January and December 2018. All CXRs were labelled as normal or abnormal according to CT findings. Four radiology residents read all CXRs twice with and without AI assistance. The performances of the AI and readers with and without AI assistance were measured in terms of area under the receiver operating characteristic curve (AUROC), sensitivity, and specificity.

RESULTS

The prevalence of clinically significant lesions was 2.2% (68 of 3047). The AUROC, sensitivity, and specificity of the AI were 0.648 (95% confidence interval [CI] 0.630-0.665), 35.3% (CI, 24.7-47.8), and 94.2% (CI, 93.3-95.0), respectively. AI detected 12 of 41 pneumonia, 3 of 5 tuberculosis, and 9 of 22 tumours. AI-undetected lesions tended to be smaller than true-positive lesions. The readers' AUROCs ranged from 0.534-0.676 without AI and 0.571-0.688 with AI (all p values < 0.05). For all readers, the mean reading time was 2.96-10.27 s longer with AI assistance (all p values < 0.05).

CONCLUSIONS

The performance of commercial AI in these high-volume, low-prevalence settings was poorer than expected, although it modestly boosted the performance of less-experienced readers. The technical prowess of AI demonstrated in experimental settings and approved by regulatory bodies may not directly translate to real-world practice, especially where the demand for AI assistance is highest.

KEY POINTS

• This study shows the limited applicability of commercial AI software for detecting abnormalities in CXRs in a health screening population. • When using AI software in a specific clinical setting that differs from the training setting, it is necessary to adjust the threshold or perform additional training with such data that reflects this environment well. • Prospective test accuracy studies, randomised controlled trials, or cohort studies are needed to examine AI software to be implemented in real clinical practice.

Prediction of inappropriate pre-hospital transfer of patients with suspected cardiovascular emergency diseases using machine learning: a retrospective observational study

BACKGROUND

This study aimed to develop a prediction model for transferring patients to an inappropriate hospital for suspected cardiovascular emergency diseases at the pre-hospital stage, using variables obtained from an integrated nationwide dataset, and to assess the performance of this model.

METHODS

We integrated three nationwide datasets and developed a two-step prediction model utilizing a machine learning algorithm. Ninety-eight clinical characteristics of patients identified at the pre-hospital stage and 13 hospital components were used as input data for the model. The primary endpoint of the model was the prediction of transfer to an inappropriate hospital.

RESULTS

A total of 94,256 transferred patients in the public pre-hospital care system matched the National Emergency Department Information System data of patients with a pre-hospital cardiovascular registry created in South Korea between July 2017 and December 2018. Of these, 1,770 (6.26%) patients failed to be transferred to a capable hospital. The area under the receiver operating characteristic curve of the final predictive model was 0.813 (0.800-0.825), and the area under the receiver precision-recall curve was 0.286 (0.265-0.308).

CONCLUSIONS

Our prediction model used machine learning to show favorable performance in transferring patients with suspected cardiovascular disease to a capable hospital. For our results to lead to changes in the pre-hospital care system, a digital platform for sharing real-time information should be developed.

Deep Learning Analysis of Chest Radiographs to Triage Patients with Acute Chest Pain Syndrome

Background Patients presenting to the emergency department (ED) with acute chest pain (ACP) syndrome undergo additional testing to exclude acute coronary syndrome (ACS), pulmonary embolism (PE), or aortic dissection (AD), often yielding negative results. Purpose To assess whether deep learning (DL) analysis of the initial chest radiograph may help triage patients with ACP syndrome more efficiently. Materials and Methods This retrospective study used electronic health records of patients with ACP syndrome at presentation who underwent a combination of chest radiography and additional cardiovascular or pulmonary imaging or stress tests at two hospitals (Massachusetts General Hospital [MGH], Brigham and Women's Hospital [BWH]) between January 2005 and December 2015. A DL model was trained on 23 005 patients from MGH to predict a 30-day composite end point of ACS, PE, AD, and all-cause mortality based on chest radiographs. Area under the receiver operating characteristic curve (AUC) was used to compare performance between models (model 1: age + sex; model 2: model 1 + conventional troponin or d-dimer positivity; model 3: model 2 + DL predictions) in internal and external test sets from MGH and BWH, respectively. Results At MGH, 5750 patients (mean age, 59 years ± 17 [SD]; 3329 men, 2421 women) were evaluated. Model 3, which included DL predictions, significantly improved discrimination of those with the composite outcome compared with models 2 and 1 (AUC, 0.85 [95% CI: 0.84, 0.86] vs 0.76 [95% CI: 0.74, 0.77] vs 0.62 [95% CI: 0.60 0.64], respectively; P < .001 for all). When using a sensitivity threshold of 99%, 14% (813 of 5750) of patients could be deferred from cardiovascular or pulmonary testing for differential diagnosis of ACP syndrome using model 3 compared with 2% (98 of 5750) of patients using model 2 (P < .001). Model 3 maintained its diagnostic performance in different age, sex, race, and ethnicity groups. In external validation at BWH (22 764 patients; mean age, 57 years ± 17; 11 470 women), trends were similar and improved after fine tuning. Conclusion Deep learning analysis of chest radiographs may facilitate more efficient triage of patients with acute chest pain syndrome in the emergency department. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Goo in this issue.

Advantage of Vital Sign Monitoring Using a Wireless Wearable Device for Predicting Septic Shock in Febrile Patients in the Emergency Department: A Machine Learning-Based Analysis

Intermittent manual measurement of vital signs may not rapidly predict sepsis development in febrile patients admitted to the emergency department (ED). We aimed to evaluate the predictive performance of a wireless monitoring device that continuously measures heart rate (HR) and respiratory rate (RR) and a machine learning analysis in febrile but stable patients in the ED. We analysed 468 patients (age, ≥18 years; training set, n = 277; validation set, n = 93; test set, n = 98) having fever (temperature >38 °C) and admitted to the isolation care unit of the ED. The AUROC of the fragmented model with device data was 0.858 (95% confidence interval [CI], 0.809−0.908), and that with manual data was 0.841 (95% CI, 0.789−0.893). The AUROC of the accumulated model with device data was 0.861 (95% CI, 0.811−0.910), and that with manual data was 0.853 (95% CI, 0.803−0.903). Fragmented and accumulated models with device data detected clinical deterioration in febrile patients at risk of septic shock 9 h and 5 h 30 min earlier, respectively, than those with manual data. Continuous vital sign monitoring using a wearable device could accurately predict clinical deterioration and reduce the time to recognise potential clinical deterioration in stable ED patients with fever.

Application of Artificial Intelligence in Lung Cancer

Simple Summary

Lung cancer is the leading cause of malignancy-related mortality worldwide. AI has the potential to help to treat lung cancer from detection, diagnosis and decision making to prognosis prediction. AI could reduce the labor work of LDCT, CXR, and pathology slides reading. AI as a second reader in LDCT and CXR reading reduces the effort of radiologists and increases the accuracy of nodule detection. Introducing AI to WSI in digital pathology increases the Kappa value of the pathologist and help to predict molecular phenotypes with radiomics and H&E staining. By extracting radiomics from image data and WSI from the histopathology field, clinicians could use AI to predict tumor properties such as gene mutation and PD-L1 expression. Furthermore, AI could help clinicians in decision-making by predicting treatment response, side effects, and prognosis prediction in medical treatment, surgery, and radiotherapy. Integrating AI in the future clinical workflow would be promising.

Abstract

Lung cancer is the leading cause of malignancy-related mortality worldwide due to its heterogeneous features and diagnosis at a late stage. Artificial intelligence (AI) is good at handling a large volume of computational and repeated labor work and is suitable for assisting doctors in analyzing image-dominant diseases like lung cancer. Scientists have shown long-standing efforts to apply AI in lung cancer screening via CXR and chest CT since the 1960s. Several grand challenges were held to find the best AI model. Currently, the FDA have approved several AI programs in CXR and chest CT reading, which enables AI systems to take part in lung cancer detection. Following the success of AI application in the radiology field, AI was applied to digitalized whole slide imaging (WSI) annotation. Integrating with more information, like demographics and clinical data, the AI systems could play a role in decision-making by classifying EGFR mutations and PD-L1 expression. AI systems also help clinicians to estimate the patient’s prognosis by predicting drug response, the tumor recurrence rate after surgery, radiotherapy response, and side effects. Though there are still some obstacles, deploying AI systems in the clinical workflow is vital for the foreseeable future.