Methods for Clinical Evaluation of Artificial Intelligence Algorithms for Medical Diagnosis

Improving the efficiency and accuracy of CMR with AI - review of evidence and proposition of a roadmap to clinical translation

Cardiovascular magnetic resonance (CMR) is an important imaging modality for the assessment of heart disease; however, limitations of CMR include long exam times and high complexity compared to other cardiac imaging modalities. Recently advancements in artificial intelligence (AI) technology have shown great potential to address many CMR limitations. While the developments are remarkable, translation of AI-based methods into real-world CMR clinical practice remains at a nascent stage and much work lies ahead to realize the full potential of AI for CMR. Herein we review recent cutting-edge and representative examples demonstrating how AI can advance CMR in areas such as exam planning, accelerated image reconstruction, post-processing, quality control, classification and diagnosis. These advances can be applied to speed up and simplify essentially every application including cine, strain, late gadolinium enhancement, parametric mapping, 3D whole heart, flow, perfusion and others. AI is a unique technology based on training models using data. Beyond reviewing the literature, this paper discusses important AI-specific issues in the context of CMR, including (1) properties and characteristics of datasets for training and validation, (2) previously published guidelines for reporting CMR AI research, (3) considerations around clinical deployment, (4) responsibilities of clinicians and the need for multi-disciplinary teams in the development and deployment of AI in CMR, (5) industry considerations, and (6) regulatory perspectives. Understanding and consideration of all these factors will contribute to the effective and ethical deployment of AI to improve clinical CMR.

Assessing the Impact of an Artificial Intelligence-Based Model for Intracranial Aneurysm Detection in CT Angiography on Patient Diagnosis and Outcomes (IDEAL Study)-a protocol for a multicenter, double-blinded randomized controlled trial

BACKGROUND

This multicenter, double-blinded, randomized controlled trial (RCT) aims to assess the impact of an artificial intelligence (AI)-based model on the efficacy of intracranial aneurysm detection in CT angiography (CTA) and its influence on patients' short-term and long-term outcomes.

METHODS

Study design: Prospective, multicenter, double-blinded RCT.

SETTINGS

The model was designed for the automatic detection of intracranial aneurysms from original CTA images.

PARTICIPANTS

Adult inpatients and outpatients who are scheduled for head CTA scanning. Randomization groups: (1) Experimental Group: Head CTA interpreted by radiologists with the assistance of the True-AI-integrated intracranial aneurysm diagnosis strategy (True-AI arm). (2) Control Group: Head CTA interpreted by radiologists with the assistance of the Sham-AI-integrated intracranial aneurysm diagnosis strategy (Sham-AI arm).

RANDOMIZATION

Block randomization, stratified by center, gender, and age group.

PRIMARY OUTCOMES

Coprimary outcomes of superiority in patient-level sensitivity and noninferiority in specificity for the True-AI arm to the Sham-AI arm in intracranial aneurysms.

SECONDARY OUTCOMES

Diagnostic performance for other intracranial lesions, detection rates, workload of CTA interpretation, resource utilization, treatment-related clinical events, aneurysm-related events, quality of life, and cost-effectiveness analysis.

BLINDING

Study participants and participating radiologists will be blinded to the intervention.

SAMPLE SIZE

Based on our pilot study, the patient-level sensitivity is assumed to be 0.65 for the Sham-AI arm and 0.75 for the True-AI arm, with specificities of 0.90 and 0.88, respectively. The prevalence of intracranial aneurysms for patients undergoing head CTA in the hospital is approximately 12%. To establish superiority in sensitivity and noninferiority in specificity with a margin of 5% using a one-sided α = 0.025 to ensure that the power of coprimary endpoint testing reached 0.80 and a 5% attrition rate, the sample size was determined to be 6450 in a 1:1 allocation to True-AI or Sham-AI arm.

DISCUSSION

The study will determine the precise impact of the AI system on the detection performance for intracranial aneurysms in a double-blinded design and following the real-world effects on patients' short-term and long-term outcomes.

TRIAL REGISTRATION

This trial has been registered with the NIH, U.S. National Library of Medicine at ClinicalTrials.gov, ID: NCT06118840 . Registered 11 November 2023.

Mpox-AISM: AI-mediated super monitoring for mpox and like-mpox

Swift and accurate diagnosis for earlier-stage monkeypox (mpox) patients is crucial to avoiding its spread. However, the similarities between common skin disorders and mpox and the need for professional diagnosis unavoidably impaired the diagnosis of earlier-stage mpox patients and contributed to mpox outbreak. To address the challenge, we proposed "Super Monitoring", a real-time visualization technique employing artificial intelligence (AI) and Internet technology to diagnose earlier-stage mpox cheaply, conveniently, and quickly. Concretely, AI-mediated "Super Monitoring" (mpox-AISM) integrates deep learning models, data augmentation, self-supervised learning, and cloud services. According to publicly accessible datasets, mpox-AISM's Precision, Recall, Specificity, and F1-score in diagnosing mpox reach 99.3%, 94.1%, 99.9%, and 96.6%, respectively, and it achieves 94.51% accuracy in diagnosing mpox, six like-mpox skin disorders, and normal skin. With the Internet and communication terminal, mpox-AISM has the potential to perform real-time and accurate diagnosis for earlier-stage mpox in real-world scenarios, thereby preventing mpox outbreak.

Toward explainable AI in radiology: Ensemble-CAM for effective thoracic disease localization in chest X-ray images using weak supervised learning

Chest X-ray (CXR) imaging is widely employed by radiologists to diagnose thoracic diseases. Recently, many deep learning techniques have been proposed as computer-aided diagnostic (CAD) tools to assist radiologists in minimizing the risk of incorrect diagnosis. From an application perspective, these models have exhibited two major challenges: (1) They require large volumes of annotated data at the training stage and (2) They lack explainable factors to justify their outcomes at the prediction stage. In the present study, we developed a class activation mapping (CAM)-based ensemble model, called Ensemble-CAM, to address both of these challenges via weakly supervised learning by employing explainable AI (XAI) functions. Ensemble-CAM utilizes class labels to predict the location of disease in association with interpretable features. The proposed work leverages ensemble and transfer learning with class activation functions to achieve three objectives: (1) minimizing the dependency on strongly annotated data when locating thoracic diseases, (2) enhancing confidence in predicted outcomes by visualizing their interpretable features, and (3) optimizing cumulative performance via fusion functions. Ensemble-CAM was trained on three CXR image datasets and evaluated through qualitative and quantitative measures via heatmaps and Jaccard indices. The results reflect the enhanced performance and reliability in comparison to existing standalone and ensembled models.

Opportunistic Identification of Vertebral Compression Fractures on CT Scans of the Chest and Abdomen, Using an AI Algorithm, in a Real-Life Setting

This study evaluated the performance of a vertebral fracture detection algorithm (HealthVCF) in a real-life setting and assessed the impact on treatment and diagnostic workflow. HealthVCF was used to identify moderate and severe vertebral compression fractures (VCF) at a Danish hospital. Around 10,000 CT scans were processed by the HealthVCF and CT scans positive for VCF formed both the baseline and 6-months follow-up cohort. To determine performance of the algorithm 1000 CT scans were evaluated by specialized radiographers to determine performance of the algorithm. Sensitivity was 0.68 (CI 0.581-0.776) and specificity 0.91 (CI 0.89-0.928). At 6-months follow-up, 18% of the 538 patients in the retrospective cohort were dead, 78 patients had been referred for a DXA scan, while 25 patients had been diagnosed with osteoporosis. A higher mortality rate was seen in patients not known with osteoporosis at baseline compared to patients known with osteoporosis at baseline, 12.8% versus 22.6% (p = 0.003). Patients receiving bisphosphonates had a lower mortality rate (9.6%) compared to the rest of the population (20.9%) (p = 0.003). HealthVCF demonstrated a poorer performance than expected, and the tested version is not generalizable to the Danish population. Based on its specificity, the HealthVCF can be used as a tool to prioritize resources in opportunistic identification of VCF's. Implementing such a tool on its own only resulted in a small number of new diagnoses of osteoporosis and referrals to DXA scans during a 6-month follow-up period. To increase efficiency, the HealthVCF should be integrated with Fracture Liaison Services (FLS).

Randomised controlled trials evaluating artificial intelligence in clinical practice: a scoping review

This scoping review of randomised controlled trials on artificial intelligence (AI) in clinical practice reveals an expanding interest in AI across clinical specialties and locations. The USA and China are leading in the number of trials, with a focus on deep learning systems for medical imaging, particularly in gastroenterology and radiology. A majority of trials (70 [81%] of 86) report positive primary endpoints, primarily related to diagnostic yield or performance; however, the predominance of single-centre trials, little demographic reporting, and varying reports of operational efficiency raise concerns about the generalisability and practicality of these results. Despite the promising outcomes, considering the likelihood of publication bias and the need for more comprehensive research including multicentre trials, diverse outcome measures, and improved reporting standards is crucial. Future AI trials should prioritise patient-relevant outcomes to fully understand AI's true effects and limitations in health care.

Developing, Purchasing, Implementing and Monitoring AI Tools in Radiology: Practical Considerations. A Multi-Society Statement From the ACR, CAR, ESR, RANZCR & RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever‑growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi‑society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools.

Enhancing the fairness of AI prediction models by Quasi-Pareto improvement among heterogeneous thyroid nodule population

Artificial Intelligence (AI) models for medical diagnosis often face challenges of generalizability and fairness. We highlighted the algorithmic unfairness in a large thyroid ultrasound dataset with significant diagnostic performance disparities across subgroups linked causally to sample size imbalances. To address this, we introduced the Quasi-Pareto Improvement (QPI) approach and a deep learning implementation (QP-Net) combining multi-task learning and domain adaptation to improve model performance among disadvantaged subgroups without compromising overall population performance. On the thyroid ultrasound dataset, our method significantly mitigated the area under curve (AUC) disparity for three less-prevalent subgroups by 0.213, 0.112, and 0.173 while maintaining the AUC for dominant subgroups; we also further confirmed the generalizability of our approach on two public datasets: the ISIC2019 skin disease dataset and the CheXpert chest radiograph dataset. Here we show the QPI approach to be widely applicable in promoting AI for equitable healthcare outcomes.

Empirical data drift detection experiments on real-world medical imaging data

While it is common to monitor deployed clinical artificial intelligence (AI) models for performance degradation, it is less common for the input data to be monitored for data drift - systemic changes to input distributions. However, when real-time evaluation may not be practical (eg., labeling costs) or when gold-labels are automatically generated, we argue that tracking data drift becomes a vital addition for AI deployments. In this work, we perform empirical experiments on real-world medical imaging to evaluate three data drift detection methods' ability to detect data drift caused (a) naturally (emergence of COVID-19 in X-rays) and (b) synthetically. We find that monitoring performance alone is not a good proxy for detecting data drift and that drift-detection heavily depends on sample size and patient features. Our work discusses the need and utility of data drift detection in various scenarios and highlights gaps in knowledge for the practical application of existing methods.

Radiology AI Deployment and Assessment Rubric (RADAR) to bring value-based AI into radiological practice

OBJECTIVE

To provide a comprehensive framework for value assessment of artificial intelligence (AI) in radiology.

METHODS

This paper presents the RADAR framework, which has been adapted from Fryback and Thornbury's imaging efficacy framework to facilitate the valuation of radiology AI from conception to local implementation. Local efficacy has been newly introduced to underscore the importance of appraising an AI technology within its local environment. Furthermore, the RADAR framework is illustrated through a myriad of study designs that help assess value.

RESULTS

RADAR presents a seven-level hierarchy, providing radiologists, researchers, and policymakers with a structured approach to the comprehensive assessment of value in radiology AI. RADAR is designed to be dynamic and meet the different valuation needs throughout the AI's lifecycle. Initial phases like technical and diagnostic efficacy (RADAR-1 and RADAR-2) are assessed pre-clinical deployment via in silico clinical trials and cross-sectional studies. Subsequent stages, spanning from diagnostic thinking to patient outcome efficacy (RADAR-3 to RADAR-5), require clinical integration and are explored via randomized controlled trials and cohort studies. Cost-effectiveness efficacy (RADAR-6) takes a societal perspective on financial feasibility, addressed via health-economic evaluations. The final level, RADAR-7, determines how prior valuations translate locally, evaluated through budget impact analysis, multi-criteria decision analyses, and prospective monitoring.

CONCLUSION

The RADAR framework offers a comprehensive framework for valuing radiology AI. Its layered, hierarchical structure, combined with a focus on local relevance, aligns RADAR seamlessly with the principles of value-based radiology.

CRITICAL RELEVANCE STATEMENT

The RADAR framework advances artificial intelligence in radiology by delineating a much-needed framework for comprehensive valuation.

KEYPOINTS

• Radiology artificial intelligence lacks a comprehensive approach to value assessment. • The RADAR framework provides a dynamic, hierarchical method for thorough valuation of radiology AI. • RADAR advances clinical radiology by bridging the artificial intelligence implementation gap.

Metrics reloaded: recommendations for image analysis validation

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Developing, purchasing, implementing and monitoring AI tools in radiology: Practical considerations. A multi-society statement from the ACR, CAR, ESR, RANZCR & RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever-growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi-society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools.

Interpretable artificial intelligence-based app assists inexperienced radiologists in diagnosing biliary atresia from sonographic gallbladder images

BACKGROUND

A previously trained deep learning-based smartphone app provides an artificial intelligence solution to help diagnose biliary atresia from sonographic gallbladder images, but it might be impractical to launch it in real clinical settings. This study aimed to redevelop a new model using original sonographic images and their derived smartphone photos and then test the new model's performance in assisting radiologists with different experiences to detect biliary atresia in real-world mimic settings.

METHODS

A new model was first trained retrospectively using 3659 original sonographic gallbladder images and their derived 51,226 smartphone photos and tested on 11,410 external validation smartphone photos. Afterward, the new model was tested in 333 prospectively collected sonographic gallbladder videos from 207 infants by 14 inexperienced radiologists (9 juniors and 5 seniors) and 4 experienced pediatric radiologists in real-world mimic settings. Diagnostic performance was expressed as the area under the receiver operating characteristic curve (AUC).

RESULTS

The new model outperformed the previously published model in diagnosing BA on the external validation set (AUC 0.924 vs 0.908, P = 0.004) with higher consistency (kappa value 0.708 vs 0.609). When tested in real-world mimic settings using 333 sonographic gallbladder videos, the new model performed comparable to experienced pediatric radiologists (average AUC 0.860 vs 0.876) and outperformed junior radiologists (average AUC 0.838 vs 0.773) and senior radiologists (average AUC 0.829 vs 0.749). Furthermore, the new model could aid both junior and senior radiologists to improve their diagnostic performances, with the average AUC increasing from 0.773 to 0.835 for junior radiologists and from 0.749 to 0.805 for senior radiologists.

CONCLUSIONS

The interpretable app-based model showed robust and satisfactory performance in diagnosing biliary atresia, and it could aid radiologists with limited experiences to improve their diagnostic performances in real-world mimic settings.

Developing, Purchasing, Implementing and Monitoring AI Tools in Radiology: Practical Considerations. A Multi-Society Statement From the ACR, CAR, ESR, RANZCR & RSNA

Artificial intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever-growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi-society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools. KEY POINTS.

Developing, purchasing, implementing and monitoring AI tools in radiology: practical considerations. A multi-society statement from the ACR, CAR, ESR, RANZCR & RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever-growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones.This multi-society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools.Key points • The incorporation of artificial intelligence (AI) in radiological practice demands increased monitoring of its utility and safety.• Cooperation between developers, clinicians, and regulators will allow all involved to address ethical issues and monitor AI performance.• AI can fulfil its promise to advance patient well-being if all steps from development to integration in healthcare are rigorously evaluated.

Developing, Purchasing, Implementing and Monitoring AI Tools in Radiology: Practical Considerations. A Multi-Society Statement from the ACR, CAR, ESR, RANZCR and RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever-growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi-society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools. This article is simultaneously published in Insights into Imaging (DOI 10.1186/s13244-023-01541-3), Journal of Medical Imaging and Radiation Oncology (DOI 10.1111/1754-9485.13612), Canadian Association of Radiologists Journal (DOI 10.1177/08465371231222229), Journal of the American College of Radiology (DOI 10.1016/j.jacr.2023.12.005), and Radiology: Artificial Intelligence (DOI 10.1148/ryai.230513). Keywords: Artificial Intelligence, Radiology, Automation, Machine Learning Published under a CC BY 4.0 license. ©The Author(s) 2024. Editor's Note: The RSNA Board of Directors has endorsed this article. It has not undergone review or editing by this journal.

Added value of an artificial intelligence algorithm in reducing the number of missed incidental acute pulmonary embolism in routine portal venous phase chest CT

OBJECTIVES

The purpose of this study was to evaluate the incremental value of artificial intelligence (AI) compared to the diagnostic accuracy of radiologists alone in detecting incidental acute pulmonary embolism (PE) on routine portal venous contrast-enhanced chest computed tomography (CT).

METHODS

CTs of 3089 consecutive patients referred to the radiology department for a routine contrast-enhanced chest CT between 27-5-2020 and 31-12-2020, were retrospectively analysed by a CE-certified and FDA-approved AI algorithm. The diagnostic performance of the AI was compared to the initial report. To determine the reference standard, discordant findings were independently evaluated by two readers. In case of disagreement, another experienced cardiothoracic radiologist with knowledge of the initial report and the AI output adjudicated.

RESULTS

The prevalence of acute incidental PE in the reference standard was 2.2% (67 of 3089 patients). In 25 cases, AI detected initially unreported PE. This included three cases concerning central/lobar PE. Sensitivity of the AI algorithm was significantly higher than the outcome of the initial report (respectively 95.5% vs. 62.7%, p < 0.001), whereas specificity was very high for both (respectively 99.6% vs 99.9%, p = 0.012). The AI algorithm only showed a slightly higher amount of false-positive findings (11 vs. 2), resulting in a significantly lower PPV (85.3% vs. 95.5%, p = 0.047).

CONCLUSION

The AI algorithm showed high diagnostic accuracy in diagnosing incidental PE, detecting an additional 25 cases of initially unreported PE, accounting for 37.3% of all positive cases.

CLINICAL RELEVANCE STATEMENT

Radiologist support from AI algorithms in daily practice can prevent missed incidental acute PE on routine chest CT, without a high burden of false-positive cases.

KEY POINTS

• Incidental pulmonary embolism is often missed by radiologists in non-diagnostic scans with suboptimal contrast opacification within the pulmonary trunk. • An artificial intelligence algorithm showed higher sensitivity detecting incidental pulmonary embolism on routine portal venous chest CT compared to the initial report. • Implementation of artificial intelligence support in routine daily practice will reduce the number of missed incidental pulmonary embolism.

Reporting Quality of Research Studies on AI Applications in Medical Images According to the CLAIM Guidelines in a Radiology Journal With a Strong Prominence in Asia

OBJECTIVE

We aimed to evaluate the reporting quality of research articles that applied deep learning to medical imaging. Using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) guidelines and a journal with prominence in Asia as a sample, we intended to provide an insight into reporting quality in the Asian region and establish a journal-specific audit.

MATERIALS AND METHODS

A total of 38 articles published in the Korean Journal of Radiology between June 2018 and January 2023 were analyzed. The analysis included calculating the percentage of studies that adhered to each CLAIM item and identifying items that were met by ≤ 50% of the studies. The article review was initially conducted independently by two reviewers, and the consensus results were used for the final analysis. We also compared adherence rates to CLAIM before and after December 2020.

RESULTS

Of the 42 items in the CLAIM guidelines, 12 items (29%) were satisfied by ≤ 50% of the included articles. None of the studies reported handling missing data (item #13). Only one study respectively presented the use of de-identification methods (#12), intended sample size (#19), robustness or sensitivity analysis (#30), and full study protocol (#41). Of the studies, 35% reported the selection of data subsets (#10), 40% reported registration information (#40), and 50% measured inter and intrarater variability (#18). No significant changes were observed in the rates of adherence to these 12 items before and after December 2020.

CONCLUSION

The reporting quality of artificial intelligence studies according to CLAIM guidelines, in our study sample, showed room for improvement. We recommend that the authors and reviewers have a solid understanding of the relevant reporting guidelines and ensure that the essential elements are adequately reported when writing and reviewing the manuscripts for publication.

Prognostic utility of RECIP 1.0 with manual and AI-based segmentations in biochemically recurrent prostate cancer from [68Ga]Ga-PSMA-11 PET images

PURPOSE

This study aimed to (i) validate the Response Evaluation Criteria in PSMA (RECIP 1.0) criteria in a cohort of biochemically recurrent (BCR) prostate cancer (PCa) patients and (ii) determine if this classification could be performed fully automatically using a trained artificial intelligence (AI) model.

METHODS

One hundred ninety-nine patients were imaged with [68Ga]Ga-PSMA-11 PET/CT once at the time of biochemical recurrence and then a second time a median of 6.0 months later to assess disease progression. Standard-of-care treatments were administered to patients in the interim. Whole-body tumour volume was quantified semi-automatically (TTVman) in all patients and using a novel AI method (TTVAI) in a subset (n = 74, the remainder were used in the training process of the model). Patients were classified as having progressive disease (RECIP-PD), or non-progressive disease (non RECIP-PD). Association of RECIP classifications with patient overall survival (OS) was assessed using the Kaplan-Meier method with the log rank test and univariate Cox regression analysis with derivation of hazard ratios (HRs). Concordance of manual and AI response classifications was evaluated using the Cohen's kappa statistic.

RESULTS

Twenty-six patients (26/199 = 13.1%) presented with RECIP-PD according to semi-automated delineations, which was associated with a significantly lower survival probability (log rank p < 0.005) and higher risk of death (HR = 3.78 (1.96-7.28), p < 0.005). Twelve patients (12/74 = 16.2%) presented with RECIP-PD according to AI-based segmentations, which was also associated with a significantly lower survival (log rank p = 0.013) and higher risk of death (HR = 3.75 (1.23-11.47), p = 0.02). Overall, semi-automated and AI-based RECIP classifications were in fair agreement (Cohen's k = 0.31).

CONCLUSION

RECIP 1.0 was demonstrated to be prognostic in a BCR PCa population and is robust to two different segmentation methods, including a novel AI-based method. RECIP 1.0 can be used to assess disease progression in PCa patients with less advanced disease. This study was registered with the Australian New Zealand Clinical Trials Registry (ACTRN12615000608561) on 11 June 2015.

Overcoming the Challenges in the Development and Implementation of Artificial Intelligence in Radiology: A Comprehensive Review of Solutions Beyond Supervised Learning

Artificial intelligence (AI) in radiology is a rapidly developing field with several prospective clinical studies demonstrating its benefits in clinical practice. In 2022, the Korean Society of Radiology held a forum to discuss the challenges and drawbacks in AI development and implementation. Various barriers hinder the successful application and widespread adoption of AI in radiology, such as limited annotated data, data privacy and security, data heterogeneity, imbalanced data, model interpretability, overfitting, and integration with clinical workflows. In this review, some of the various possible solutions to these challenges are presented and discussed; these include training with longitudinal and multimodal datasets, dense training with multitask learning and multimodal learning, self-supervised contrastive learning, various image modifications and syntheses using generative models, explainable AI, causal learning, federated learning with large data models, and digital twins.

Deep learning system improved detection efficacy of fetal intracranial malformations in a randomized controlled trial

Congenital malformations of the central nervous system are among the most common major congenital malformations. Deep learning systems have come to the fore in prenatal diagnosis of congenital malformation, but the impact of deep learning-assisted detection of congenital intracranial malformations from fetal neurosonographic images has not been evaluated. Here we report a three-way crossover, randomized control trial (Trial Registration: ChiCTR2100048233) that assesses the efficacy of a deep learning system, the Prenatal Ultrasound Diagnosis Artificial Intelligence Conduct System (PAICS), in assisting fetal intracranial malformation detection. A total of 709 fetal neurosonographic images/videos are read interactively by 36 sonologists of different expertise levels in three reading modes: unassisted mode (without PAICS assistance), concurrent mode (using PAICS at the beginning of the assessment) and second mode (using PAICS after a fully unaided interpretation). Aided by PAICS, the average accuracy of the unassisted mode (73%) is increased by the concurrent mode (80%; P < 0.001) and the second mode (82%; P < 0.001). Correspondingly, the AUC is increased from 0.85 to 0.89 and to 0.90, respectively (P < 0.001 for all). The median read time per data is slightly increased in concurrent mode but substantially prolonged in the second mode, from 6 s to 7 s and to 11 s (P < 0.001 for all). In conclusion, PAICS in both concurrent and second modes has the potential to improve sonologists' performance in detecting fetal intracranial malformations from neurosonographic data. PAICS is more efficient when used concurrently for all readers.

Curve Similarity Analysis for Reducing the Temperature Uncertainty of Optical Sensor for Oil-Tank Ground Settlement Monitoring

A nonuniform temperature field can deteriorate the performance of sensors, especially those working in the field, such as an optical sensor for oil-tank ground settlement (GS) monitoring. In this case, the GS monitoring employs hydraulic-level-based sensors (HLBS), which are uniformly installed along with the oil-tank basement perimeter and are all connected by hydraulic tubes. Then, the cylinder structure of the oil tank itself can create a strong temperature difference between the sensors installed in the sunlit front and those in the shadow. Practically, this sunlight-dependent difference can be over 30 °C, by which the thermal expansion of the measuring liquid inside the connecting hydraulic tubes keeps on driving a movement and, thereby, leads to fluctuations in the final result of the oil-tank GS monitoring system. Now, this system can work well at night when the temperature difference becomes negligible. However, temperature uncertainty is generated in the GS sensors due to the large temperature difference between the sensors in the daytime. In this paper, we measured the temperature where the sensor was located. Then, we compared the results of the GS sensors with their corresponding temperatures and fitted them with two separate curves, respectively. After observing the similarity in the tendency of the two curves, we found that there was a qualitative correlative relationship between the change in temperature and the uncertainty in the sensor results. Then, a curve similarity analysis (CSA) principle based on the minimum mean square error (MMSE) criteria was employed to establish an algorithm, by which the temperature uncertainty in the GS sensors was reduced. A practical test proved that the standard deviation was improved by 73.4% by the algorithm. This work could be an example for reducing the temperature uncertainty from in-field sensors through the CSA method.

The Performance of a Deep Learning-Based Automatic Measurement Model for Measuring the Cardiothoracic Ratio on Chest Radiographs

OBJECTIVE

Prior studies on models based on deep learning (DL) and measuring the cardiothoracic ratio (CTR) on chest radiographs have lacked rigorous agreement analyses with radiologists or reader tests. We validated the performance of a commercially available DL-based CTR measurement model with various thoracic pathologies, and performed agreement analyses with thoracic radiologists and reader tests using a probabilistic-based reference.

MATERIALS AND METHODS

This study included 160 posteroanterior view chest radiographs (no lung or pleural abnormalities, pneumothorax, pleural effusion, consolidation, and n = 40 in each category) to externally test a DL-based CTR measurement model. To assess the agreement between the model and experts, intraclass or interclass correlation coefficients (ICCs) were compared between the model and two thoracic radiologists. In the reader tests with a probabilistic-based reference standard (Dawid-Skene consensus), we compared diagnostic measures-including sensitivity and negative predictive value (NPV)-for cardiomegaly between the model and five other radiologists using the non-inferiority test.

RESULTS

For the 160 chest radiographs, the model measured a median CTR of 0.521 (interquartile range, 0.446-0.59) and a mean CTR of 0.522 ± 0.095. The ICC between the two thoracic radiologists and between the model and two thoracic radiologists was not significantly different (0.972 versus 0.959, p = 0.192), even across various pathologies (all p-values > 0.05). The model showed non-inferior diagnostic performance, including sensitivity (96.3% versus 97.8%) and NPV (95.6% versus 97.4%) (p < 0.001 in both), compared with the radiologists for all 160 chest radiographs. However, it showed inferior sensitivity in chest radiographs with consolidation (95.5% versus 99.9%; p = 0.082) and NPV in chest radiographs with pleural effusion (92.9% versus 94.6%; p = 0.079) and consolidation (94.1% versus 98.7%; p = 0.173).

CONCLUSION

While the sensitivity and NPV of this model for diagnosing cardiomegaly in chest radiographs with consolidation or pleural effusion were not as high as those of the radiologists, it demonstrated good agreement with the thoracic radiologists in measuring the CTR across various pathologies.

Radiologist's Guide to Evaluating Publications of Clinical Research on AI: How We Do It

Literacy in research studies of artificial intelligence (AI) has become an important skill for radiologists. It is required to make a proper assessment of the validity, reproducibility, and clinical applicability of AI studies. However, AI studies are generally perceived to be more difficult for clinician readers to evaluate than traditional clinical research studies. This special report-as an effective, concise guide for readers-aims to assist clinical radiologists in critically evaluating different types of clinical research articles involving AI. It does not intend to be a comprehensive checklist or methodological summary for complete clinical evaluation of AI or a reporting guideline. Ten key items for readers to check are described, regarding study purpose, function and clinical context of AI, training data, data preprocessing, AI modeling techniques, test data, AI performance, helpfulness and value of AI, interpretability of AI, and code sharing. The important aspects of each item are explained for readers to consider when reading publications on AI clinical research. Evaluating each item can help radiologists assess the validity, reproducibility, and clinical applicability of clinical research articles involving AI.

Deep learning referral suggestion and tumour discrimination using explainable artificial intelligence applied to multiparametric MRI

OBJECTIVES

An appropriate and fast clinical referral suggestion is important for intra-axial mass-like lesions (IMLLs) in the emergency setting. We aimed to apply an interpretable deep learning (DL) system to multiparametric MRI to obtain clinical referral suggestion for IMLLs, and to validate it in the setting of nontraumatic emergency neuroradiology.

METHODS

A DL system was developed in 747 patients with IMLLs ranging 30 diseases who underwent pre- and post-contrast T1-weighted (T1CE), FLAIR, and diffusion-weighted imaging (DWI). A DL system that segments IMLLs, classifies tumourous conditions, and suggests clinical referral among surgery, systematic work-up, medical treatment, and conservative treatment, was developed. The system was validated in an independent cohort of 130 emergency patients, and performance in referral suggestion and tumour discrimination was compared with that of radiologists using receiver operating characteristics curve, precision-recall curve analysis, and confusion matrices. Multiparametric interpretable visualisation of high-relevance regions from layer-wise relevance propagation overlaid on contrast-enhanced T1WI and DWI was analysed.

RESULTS

The DL system provided correct referral suggestions in 94 of 130 patients (72.3%) and performed comparably to radiologists (accuracy 72.6%, McNemar test; p = .942). For distinguishing tumours from non-tumourous conditions, the DL system (AUC, 0.90 and AUPRC, 0.94) performed similarly to human readers (AUC, 0.81~0.92, and AUPRC, 0.88~0.95). Solid portions of tumours showed a high overlap of relevance, but non-tumours did not (Dice coefficient 0.77 vs. 0.33, p < .001), demonstrating the DL's decision.

CONCLUSIONS

Our DL system could appropriately triage patients using multiparametric MRI and provide interpretability through multiparametric heatmaps, and may thereby aid neuroradiologic diagnoses in emergency settings.

CLINICAL RELEVANCE STATEMENT

Our AI triages patients with raw MRI images to clinical referral pathways in brain intra-axial mass-like lesions. We demonstrate that the decision is based on the relative relevance between contrast-enhanced T1-weighted and diffusion-weighted images, providing explainability across multiparametric MRI data.

KEY POINTS

• A deep learning (DL) system using multiparametric MRI suggested clinical referral to patients with intra-axial mass-like lesions (IMLLs) similar to radiologists (accuracy 72.3% vs. 72.6%). • In the differentiation of tumourous and non-tumourous conditions, the DL system (AUC, 0.90) performed similar with radiologists (AUC, 0.81-0.92). • The DL's decision basis for differentiating tumours from non-tumours can be quantified using multiparametric heatmaps obtained via the layer-wise relevance propagation method.

Clinical Implication and Prognostic Value of Artificial-Intelligence-Based Results of Chest Radiographs for Assessing Clinical Outcomes of COVID-19 Patients

This study aimed to investigate the clinical implications and prognostic value of artificial intelligence (AI)-based results for chest radiographs (CXR) in coronavirus disease 2019 (COVID-19) patients. Patients who were admitted due to COVID-19 from September 2021 to March 2022 were retrospectively included. A commercial AI-based software was used to assess CXR data for consolidation and pleural effusion scores. Clinical data, including laboratory results, were analyzed for possible prognostic factors. Total O₂ supply period, the last SpO₂ result, and deterioration were evaluated as prognostic indicators of treatment outcome. Generalized linear mixed model and regression tests were used to examine the prognostic value of CXR results. Among a total of 228 patients (mean 59.9 ± 18.8 years old), consolidation scores had a significant association with erythrocyte sedimentation rate and C-reactive protein changes, and initial consolidation scores were associated with the last SpO₂ result (estimate -0.018, p = 0.024). All consolidation scores during admission showed significant association with the total O₂ supply period and the last SpO₂ result. Early changing degree of consolidation score showed an association with deterioration (odds ratio 1.017, 95% confidence interval 1.005-1.03). In conclusion, AI-based CXR results for consolidation have potential prognostic value for predicting treatment outcomes in COVID-19 patients.

Retrospective batch analysis to evaluate the diagnostic accuracy of a clinically deployed AI algorithm for the detection of acute pulmonary embolism on CTPA

PURPOSE

To generate and extend the evidence on the clinical validity of an artificial intelligence (AI) algorithm to detect acute pulmonary embolism (PE) on CT pulmonary angiography (CTPA) of patients suspected of PE and to evaluate the possibility of reducing the risk of missed findings in clinical practice with AI-assisted reporting.

METHODS

Consecutive CTPA scan data of 3316 patients referred because of suspected PE between 24-2-2018 and 31-12-2020 were retrospectively analysed by a CE-certified and FDA-approved AI algorithm. The output of the AI was compared with the attending radiologists' report. To define the reference standard, discordant findings were independently evaluated by two readers. In case of disagreement, an experienced cardiothoracic radiologist adjudicated.

RESULTS

According to the reference standard, PE was present in 717 patients (21.6%). PE was missed by the AI in 23 patients, while the attending radiologist missed 60 PE. The AI detected 2 false positives and the attending radiologist 9. The sensitivity for the detection of PE by the AI algorithm was significantly higher compared to the radiology report (96.8% vs. 91.6%, p < 0.001). Specificity of the AI was also significantly higher (99.9% vs. 99.7%, p = 0.035). NPV and PPV of the AI were also significantly higher than the radiology report.

CONCLUSION

The AI algorithm showed a significantly higher diagnostic accuracy for the detection of PE on CTPA compared to the report of the attending radiologist. This finding indicates that missed positive findings could be prevented with the implementation of AI-assisted reporting in daily clinical practice.

CRITICAL RELEVANCE STATEMENT

Missed positive findings on CTPA of patients suspected of pulmonary embolism can be prevented with the implementation of AI-assisted care.

KEY POINTS

The AI algorithm showed excellent diagnostic accuracy detecting PE on CTPA. Accuracy of the AI was significantly higher compared to the attending radiologist. Highest diagnostic accuracy can likely be achieved by radiologists supported by AI. Our results indicate that implementation of AI-assisted reporting could reduce the number of missed positive findings.