Clinician's guide to trustworthy and responsible artificial intelligence in cardiovascular imaging

Clinical assessment of an AI tool for measuring biventricular parameters on cardiac MR

Introduction

Cardiac magnetic resonance (CMR) is of diagnostic and prognostic value in a range of cardiopulmonary conditions. Current methods for evaluating CMR studies are laborious and time-consuming, contributing to delays for patients. As the demand for CMR increases, there is a growing need to automate this process. The application of artificial intelligence (AI) to CMR is promising, but the evaluation of these tools in clinical practice has been limited. This study assessed the clinical viability of an automatic tool for measuring cardiac volumes on CMR.

Methods

Consecutive patients who underwent CMR for any indication between January 2022 and October 2022 at a single tertiary centre were included prospectively. For each case, short-axis CMR images were segmented by the AI tool and manually to yield volume, mass and ejection fraction measurements for both ventricles. Automated and manual measurements were compared for agreement and the quality of the automated contours was assessed visually by cardiac radiologists.

Results

462 CMR studies were included. No statistically significant difference was demonstrated between any automated and manual measurements (p > 0.05; independent T-test). Intraclass correlation coefficient and Bland-Altman analysis showed excellent agreement across all metrics (ICC > 0.85). The automated contours were evaluated visually in 251 cases, with agreement or minor disagreement in 229 cases (91.2%) and failed segmentation in only a single case (0.4%). The AI tool was able to provide automated contours in under 90 s.

Conclusions

Automated segmentation of both ventricles on CMR by an automatic tool shows excellent agreement with manual segmentation performed by CMR experts in a retrospective real-world clinical cohort. Implementation of the tool could improve the efficiency of CMR reporting and reduce delays between imaging and diagnosis.

Improved diagnostic performance of insertable cardiac monitors by an artificial intelligence-based algorithm

AIMS

The increasing use of insertable cardiac monitors (ICM) produces a high rate of false positive (FP) diagnoses. Their verification results in a high workload for caregivers. We evaluated the performance of an artificial intelligence (AI)-based ILR-ECG Analyzer™ (ILR-ECG-A). This machine-learning algorithm reclassifies ICM-transmitted events to minimize the rate of FP diagnoses, while preserving device sensitivity.

METHODS AND RESULTS

We selected 546 recipients of ICM followed by the Implicity™ monitoring platform. To avoid clusterization, a single episode per ICM abnormal diagnosis (e.g. asystole, bradycardia, atrial tachycardia (AT)/atrial fibrillation (AF), ventricular tachycardia, artefact) was selected per patient, and analyzed by the ILR-ECG-A, applying the same diagnoses as the ICM. All episodes were reviewed by an adjudication committee (AC) and the results were compared. Among 879 episodes classified as abnormal by the ICM, 80 (9.1%) were adjudicated as 'Artefacts', 283 (32.2%) as FP, and 516 (58.7%) as 'abnormal' by the AC. The algorithm reclassified 215 of the 283 FP as normal (76.0%), and confirmed 509 of the 516 episodes as abnormal (98.6%). Seven undiagnosed false negatives were adjudicated as AT or non-specific abnormality. The overall diagnostic specificity was 76.0% and the sensitivity was 98.6%.

CONCLUSION

The new AI-based ILR-ECG-A lowered the rate of FP ICM diagnoses significantly while retaining a > 98% sensitivity. This will likely alleviate considerably the clinical burden represented by the review of ICM events.

Pilot Testing of a Tool to Standardize the Assessment of the Quality of Health Information Generated by Artificial Intelligence-Based Models

Background Artificial intelligence (AI)-based conversational models, such as Chat Generative Pre-trained Transformer (ChatGPT), Microsoft Bing, and Google Bard, have emerged as valuable sources of health information for lay individuals. However, the accuracy of the information provided by these AI models remains a significant concern. This pilot study aimed to test a new tool with key themes for inclusion as follows: Completeness of content, Lack of false information in the content, Evidence supporting the content, Appropriateness of the content, and Relevance, referred to as "CLEAR", designed to assess the quality of health information delivered by AI-based models. Methods Tool development involved a literature review on health information quality, followed by the initial establishment of the CLEAR tool, which comprised five items that aimed to assess the following: completeness, lack of false information, evidence support, appropriateness, and relevance. Each item was scored on a five-point Likert scale from excellent to poor. Content validity was checked by expert review. Pilot testing involved 32 healthcare professionals using the CLEAR tool to assess content on eight different health topics deliberately designed with varying qualities. The internal consistency was checked with Cronbach's alpha (α). Feedback from the pilot test resulted in language modifications to improve the clarity of the items. The final CLEAR tool was used to assess the quality of health information generated by four distinct AI models on five health topics. The AI models were ChatGPT 3.5, ChatGPT 4, Microsoft Bing, and Google Bard, and the content generated was scored by two independent raters with Cohen's kappa (κ) for inter-rater agreement. Results The final five CLEAR items were: (1) Is the content sufficient?; (2) Is the content accurate?; (3) Is the content evidence-based?; (4) Is the content clear, concise, and easy to understand?; and (5) Is the content free from irrelevant information? Pilot testing on the eight health topics revealed acceptable internal consistency with a Cronbach's α range of 0.669-0.981. The use of the final CLEAR tool yielded the following average scores: Microsoft Bing (mean=24.4±0.42), ChatGPT-4 (mean=23.6±0.96), Google Bard (mean=21.2±1.79), and ChatGPT-3.5 (mean=20.6±5.20). The inter-rater agreement revealed the following Cohen κ values: for ChatGPT-3.5 (κ=0.875, P<.001), ChatGPT-4 (κ=0.780, P<.001), Microsoft Bing (κ=0.348, P=.037), and Google Bard (κ=.749, P<.001). Conclusions The CLEAR tool is a brief yet helpful tool that can aid in standardizing testing of the quality of health information generated by AI-based models. Future studies are recommended to validate the utility of the CLEAR tool in the quality assessment of AI-generated health-related content using a larger sample across various complex health topics.