Does clinical decision support system promote expert consensus for appropriate imaging referrals? Chest-abdominal-pelvis CT as a case study

Computed tomography referral guidelines adherence in Europe: insights from a seven-country audit

BACKGROUND

Ensuring appropriate computed tomography (CT) utilization optimizes patient care while minimizing radiation exposure. Decision support tools show promise for standardizing appropriateness.

OBJECTIVES

In the current study, we aimed to assess CT appropriateness rates using the European Society of Radiology (ESR) iGuide criteria across seven European countries. Additional objectives were to identify factors associated with appropriateness variability and examine recommended alternative exams.

METHODS

As part of the European Commission-funded EU-JUST-CT project, 6734 anonymized CT referrals were audited across 125 centers in Belgium, Denmark, Estonia, Finland, Greece, Hungary, and Slovenia. In each country, two blinded radiologists independently scored each exam's appropriateness using the ESR iGuide and noted any recommended alternatives based on presented indications. Arbitration was used in case auditors disagreed. Associations between appropriateness rate and institution type, patient's age and sex, inpatient/outpatient patient status, anatomical area, and referring physician's specialty were statistically examined within each country.

RESULTS

The average appropriateness rate was 75%, ranging from 58% in Greece to 86% in Denmark. Higher rates were associated with public hospitals, inpatient settings, and referrals from specialists. Variability in appropriateness existed by country and anatomical area, patient age, and gender. Common alternative exam recommendations included magnetic resonance imaging, X-ray, and ultrasound.

CONCLUSION

This multi-country evaluation found that even when using a standardized imaging guideline, significant variations in CT appropriateness persist, ranging from 58% to 86% across the participating countries. The study provided valuable insights into real-world utilization patterns and identified opportunities to optimize practices and reduce clinical and demographic disparities in CT use.

KEY POINTS

Question Largest multinational study (7 EU countries, 6734 CT referrals) assessed real-world CT appropriateness using ESR iGuide, enabling cross-system comparisons. Findings Significant variability in appropriateness rates across institution type, patient status, age, gender, exam area, and physician specialty, highlighted the opportunities to optimize practices based on local factors. Clinical relevance International collaboration on imaging guidelines and decision support can maximize CT benefits while optimizing radiation exposure; ongoing research is crucial for refining evidence-based guidelines globally.

Radiologic Decision-Making for Imaging in Pulmonary Embolism: Accuracy and Reliability of Large Language Models-Bing, Claude, ChatGPT, and Perplexity

Background  Artificial intelligence chatbots have demonstrated potential to enhance clinical decision-making and streamline health care workflows, potentially alleviating administrative burdens. However, the contribution of AI chatbots to radiologic decision-making for clinical scenarios remains insufficiently explored. This study evaluates the accuracy and reliability of four prominent Large Language Models (LLMs)-Microsoft Bing, Claude, ChatGPT 3.5, and Perplexity-in offering clinical decision support for initial imaging for suspected pulmonary embolism (PE). Methods  Open-ended (OE) and select-all-that-apply (SATA) questions were crafted, covering four variants of case scenarios of PE in-line with the American College of Radiology Appropriateness Criteria. These questions were presented to the LLMs by three radiologists from diverse geographical regions and setups. The responses were evaluated based on established scoring criteria, with a maximum achievable score of 2 points for OE responses and 1 point for each correct answer in SATA questions. To enable comparative analysis, scores were normalized (score divided by the maximum achievable score). Result  In OE questions, Perplexity achieved the highest accuracy (0.83), while Claude had the lowest (0.58), with Bing and ChatGPT each scoring 0.75. For SATA questions, Bing led with an accuracy of 0.96, Perplexity was the lowest at 0.56, and both Claude and ChatGPT scored 0.6. Overall, OE questions saw higher scores (0.73) compared to SATA (0.68). There is poor agreement among radiologists' scores for OE (Intraclass Correlation Coefficient [ICC] = -0.067, p  = 0.54), while there is strong agreement for SATA (ICC = 0.875, p  < 0.001). Conclusion  The study revealed variations in accuracy across LLMs for both OE and SATA questions. Perplexity showed superior performance in OE questions, while Bing excelled in SATA questions. OE queries yielded better overall results. The current inconsistencies in LLM accuracy highlight the importance of further refinement before these tools can be reliably integrated into clinical practice, with a need for additional LLM fine-tuning and judicious selection by radiologists to achieve consistent and reliable support for decision-making.

Insights from a web-based questionnaire: examining diagnostic procedures prior to magnetic resonance imaging

The appropriate use of diagnostic imaging, particularly MRI, is a critical concern in modern healthcare. This paper examines the current state of MRI utilization in Israel, drawing on a recent study by Kaim et al. that surveyed 557 Israeli adults who underwent MRI in the public health system. The study revealed that 60% of participants had undergone other imaging tests before their MRI, with 23% having more than one prior examination. While these findings highlight potential inefficiencies in the diagnostic pathway, they also underscore the complexity of medical decision-making in imaging.The paper discusses various factors influencing MRI utilization, including regulatory pressures, healthcare system structure, and the need for evidence-based guidelines. It explores potential strategies for optimizing MRI justification and scheduling, such as implementing clinical decision support systems, enhancing interdisciplinary communication, and leveraging artificial intelligence (AI) for predictive analytics and resource optimization.The need for comprehensive research into MRI justification and scheduling optimization is presented. Key areas for investigation include the effectiveness of decision support tools, patient outcomes, economic analyses, and the application of quality improvement methodologies.

Evaluating the Accuracy and Impact of the ESR-iGuide Decision Support Tool in Optimizing CT Imaging Referral Appropriateness

Radiology referral quality impacts patient care, yet factors influencing quality are poorly understood. This study assessed the quality of computed tomography (CT) referrals, identified associated characteristics, and evaluated the ESR-iGuide clinical decision support tool's ability to optimize referrals. A retrospective review analyzed 300 consecutive CT referrals from an acute care hospital. Referral quality was evaluated on a 5-point scale by three expert reviewers (inter-rater reliability κ = 0.763-0.97). The ESR-iGuide tool provided appropriateness scores and estimated radiation exposure levels for the actual referred exams and recommended exams. Scores were compared between actual and recommended exams. Associations between ESR-iGuide scores and referral characteristics, including the specialty of the ordering physician (surgical vs. non-surgical), were explored. Of the referrals, 67.1% were rated as appropriate. The most common exams were head and abdomen/pelvis CTs. The ESR-iGuide deemed 70% of the actual referrals "usually appropriate" and found that the recommended exams had lower estimated radiation exposure compared to the actual exams. Logistic regression analysis showed that non-surgical physicians were more likely to order inappropriate exams compared to surgical physicians. Over one-third of the referrals showed suboptimal quality in the unstructured system. The ESR-iGuide clinical decision support tool identified opportunities to optimize appropriateness and reduce radiation exposure. Implementation of such a tool warrants consideration to improve communication and maximize patient care quality.

A content-aware chatbot based on GPT 4 provides trustworthy recommendations for Cone-Beam CT guidelines in dental imaging

OBJECTIVES

To develop a content-aware chatbot based on GPT-3.5-Turbo and GPT-4 with specialized knowledge on the German S2 Cone-Beam CT (CBCT) dental imaging guideline and to compare the performance against humans.

METHODS

The LlamaIndex software library was used to integrate the guideline context into the chatbots. Based on the CBCT S2 guideline, 40 questions were posed to content-aware chatbots and early career and senior practitioners with different levels of experience served as reference. The chatbots' performance was compared in terms of recommendation accuracy and explanation quality. Chi-square test and one-tailed Wilcoxon signed rank test evaluated accuracy and explanation quality, respectively.

RESULTS

The GPT-4 based chatbot provided 100% correct recommendations and superior explanation quality compared to the one based on GPT3.5-Turbo (87.5% vs. 57.5% for GPT-3.5-Turbo; P = .003). Moreover, it outperformed early career practitioners in correct answers (P = .002 and P = .032) and earned higher trust than the chatbot using GPT-3.5-Turbo (P = 0.006).

CONCLUSIONS

A content-aware chatbot using GPT-4 reliably provided recommendations according to current consensus guidelines. The responses were deemed trustworthy and transparent, and therefore facilitate the integration of artificial intelligence into clinical decision-making.

Inappropriate CT examinations: how much, who and where? Insights from a clinical decision support system (CDSS) analysis

OBJECTIVE

To assess the appropriateness of Computed Tomography (CT) examinations, using the ESR-iGuide.

MATERIAL AND METHODS

A retrospective study was conducted in 2022 in a medium-sized acute care teaching hospital. A total of 278 consecutive cases of CT referral were included. For each imaging referral, the ESR-iGuide provided an appropriateness score using a scale of 1-9 and the Relative Radiation Level using a scale of 0-5. These were then compared with the appropriateness score and the radiation level of the recommended ESR-iGuide exam.

DATA ANALYSIS

Pearson's chi-square test or Fisher exact test was used to explore the correlation between ESR-iGuide appropriateness level and physician, patients, and shift characteristics. A stepwise logistic regression model was used to capture the contribution of each of these factors.

RESULTS

Most of exams performed were CT head (63.67%) or CT abdominal pelvis (23.74%). Seventy percent of the actual imaging referrals resulted in an ESR-iGuide score corresponding to "usually appropriate." The mean radiation level for actual exam was 3.2 ± 0.45 compared with 2.16 ± 1.56 for the recommended exam. When using a stepwise logistic regression for modeling the probability of non-appropriate score, both physician specialty and status were significant (p = 0.0011, p = 0.0192 respectively). Non-surgical and specialist physicians were more likely to order inappropriate exams than surgical physicians.

CONCLUSIONS

ESR-iGuide software indicates a substantial rate of inappropriate exams of CT head and CT abdominal-pelvis and unnecessary radiation exposure mainly in the ED department. Inappropriate exams were found to be related to physicians' specialty and seniority.

CLINICAL RELEVANCE STATEMENT

These findings underscore the urgent need for improved imaging referral practices to ensure appropriate healthcare delivery and effective resource management. Additionally, they highlight the potential benefits and necessity of integrating CDSS as a standard medical practice. By implementing CDSS, healthcare providers can make more informed decisions, leading to enhanced patient care, optimized resource allocation, and improved overall healthcare outcomes.

KEY POINTS

• The overall mean of appropriateness for the actual exam according to the ESR-iGuide was 6.62 ± 2.69 on a scale of 0-9. • Seventy percent of the actual imaging referrals resulted in an ESR-iGuide score corresponding to "usually appropriate." • Inappropriate examination is related to both the specialty of the physician who requested the exam and the seniority status of the physician.