Assessment of Radiology Artificial Intelligence Software: A Validation and Evaluation Framework

AI for IMPACTS Framework for Evaluating the Long-Term Real-World Impacts of AI-Powered Clinician Tools: Systematic Review and Narrative Synthesis

BACKGROUND

Artificial intelligence (AI) has the potential to revolutionize health care by enhancing both clinical outcomes and operational efficiency. However, its clinical adoption has been slower than anticipated, largely due to the absence of comprehensive evaluation frameworks. Existing frameworks remain insufficient and tend to emphasize technical metrics such as accuracy and validation, while overlooking critical real-world factors such as clinical impact, integration, and economic sustainability. This narrow focus prevents AI tools from being effectively implemented, limiting their broader impact and long-term viability in clinical practice.

OBJECTIVE

This study aimed to create a framework for assessing AI in health care, extending beyond technical metrics to incorporate social and organizational dimensions. The framework was developed by systematically reviewing, analyzing, and synthesizing the evaluation criteria necessary for successful implementation, focusing on the long-term real-world impact of AI in clinical practice.

METHODS

A search was performed in July 2024 across the PubMed, Cochrane, Scopus, and IEEE Xplore databases to identify relevant studies published in English between January 2019 and mid-July 2024, yielding 3528 results, among which 44 studies met the inclusion criteria. The systematic review followed PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines and the Cochrane Handbook for Systematic Reviews. Data were analyzed using NVivo through thematic analysis and narrative synthesis to identify key emergent themes in the studies.

RESULTS

By synthesizing the included studies, we developed a framework that goes beyond the traditional focus on technical metrics or study-level methodologies. It integrates clinical context and real-world implementation factors, offering a more comprehensive approach to evaluating AI tools. With our focus on assessing the long-term real-world impact of AI technologies in health care, we named the framework AI for IMPACTS. The criteria are organized into seven key clusters, each corresponding to a letter in the acronym: (1) I-integration, interoperability, and workflow; (2) M-monitoring, governance, and accountability; (3) P-performance and quality metrics; (4) A-acceptability, trust, and training; (5) C-cost and economic evaluation; (6) T-technological safety and transparency; and (7) S-scalability and impact. These are further broken down into 28 specific subcriteria.

CONCLUSIONS

The AI for IMPACTS framework offers a holistic approach to evaluate the long-term real-world impact of AI tools in the heterogeneous and challenging health care context and lays the groundwork for further validation through expert consensus and testing of the framework in real-world health care settings. It is important to emphasize that multidisciplinary expertise is essential for assessment, yet many assessors lack the necessary training. In addition, traditional evaluation methods struggle to keep pace with AI's rapid development. To ensure successful AI integration, flexible, fast-tracked assessment processes and proper assessor training are needed to maintain rigorous standards while adapting to AI's dynamic evolution.

TRIAL REGISTRATION

reviewregistry1859; https://tinyurl.com/ysn2d7sh.

Artificial Intelligence-Empowered Radiology-Current Status and Critical Review

Humanity stands at a pivotal moment of technological revolution, with artificial intelligence (AI) reshaping fields traditionally reliant on human cognitive abilities. This transition, driven by advancements in artificial neural networks, has transformed data processing and evaluation, creating opportunities for addressing complex and time-consuming tasks with AI solutions. Convolutional networks (CNNs) and the adoption of GPU technology have already revolutionized image recognition by enhancing computational efficiency and accuracy. In radiology, AI applications are particularly valuable for tasks involving pattern detection and classification; for example, AI tools have enhanced diagnostic accuracy and efficiency in detecting abnormalities across imaging modalities through automated feature extraction. Our analysis reveals that neuroimaging and chest imaging, as well as CT and MRI modalities, are the primary focus areas for AI products, reflecting their high clinical demand and complexity. AI tools are also used to target high-prevalence diseases, such as lung cancer, stroke, and breast cancer, underscoring AI's alignment with impactful diagnostic needs. The regulatory landscape is a critical factor in AI product development, with the majority of products certified under the Medical Device Directive (MDD) and Medical Device Regulation (MDR) in Class IIa or Class I categories, indicating compliance with moderate-risk standards. A rapid increase in AI product development from 2017 to 2020, peaking in 2020 and followed by recent stabilization and saturation, was identified. In this work, the authors review the advancements in AI-based imaging applications, underscoring AI's transformative potential for enhanced diagnostic support and focusing on the critical role of CNNs, regulatory challenges, and potential threats to human labor in the field of diagnostic imaging.

Testing process for artificial intelligence applications in radiology practice

Artificial intelligence (AI) applications are becoming increasingly common in radiology. However, ensuring reliable operation and expected clinical benefits remains a challenge. A systematic testing process aims to facilitate clinical deployment by confirming software applicability to local patient populations, practises, adherence to regulatory and safety requirements, and compatibility with existing systems. In this work, we present our testing process developed based on practical experience. First, a survey and pre-evaluation is conducted, where information requests are sent for potential products, and the specifications are evaluated against predetermined requirements. In the second phase, data collection, testing, and analysis are conducted. In the retrospective stage, the application undergoes testing with a pre selected dataset and is evaluated against specified key performance indicators (KPIs). In the prospective stage, the application is integrated into the clinical workflow and evaluated with additional process-specific KPIs. In the final phase, the results are evaluated in terms of safety, effectiveness, productivity, and integration. The final report summarises the results and includes a procurement/deployment or rejection recommendation. The process allows termination at any phase if the application fails to meet essential criteria. In addition, we present practical remarks from our experiences in AI testing and provide forms to guide and document the testing process. The established AI testing process facilitates a systematic evaluation and documentation of new technologies ensuring that each application undergoes equal and sufficient validation. Testing with local data is crucial for identifying biases and pitfalls of AI algorithms to improve the quality and safety, ultimately benefiting patient care.

The American Society of Radiologic Technologists (ASRT) AI educator survey: A cross-sectional study to explore knowledge, experience, and use of AI within education

INTRODUCTION

Artificial Intelligence (AI) is revolutionizing medical imaging and radiation therapy. AI-powered applications are being deployed to aid Medical Radiation Technologists (MRTs) in clinical workflows, decision-making, dose optimisation, and a wide range of other tasks. Exploring the levels of AI education provided across the United States is crucial to prepare future graduates to deliver the digital future. This study aims to assess educators' levels of AI knowledge, the current state of AI educational provisions, the perceived challenges around AI education, and important factors for future advancements.

METHODS

An online survey was electronically administered to all radiologic technologists in the American Society of Radiologic Technologists (ASRT) database who indicated that they had an educator role in the United States. This was distributed through the membership of the ASRT, from February to April 2023. All quantitative data was analysed using frequency and descriptive statistics. The survey's open-ended questions were analysed using a conceptual content analysis approach.

RESULTS

Out of 5,066 educators in the ASRT database, 373 valid responses were received, resulting in a response rate of 7.4%. Despite 84.5% of educators expressing the importance of teaching AI, 23.7% currently included AI in academic curricula. Of the 76.3% that did not include AI in their curricula, lack of AI knowledge among educators was the top reason for not integrating AI in education (59.1%). Similarly, AI-enabled tools were utilised by only 11.1% of the programs to assist teaching. The levels of trust in AI varied among educators.

CONCLUSION

The study found that although US educators of MRTs have a good baseline knowledge of general concepts regarding AI, they could improve on the teaching and use of AI in their curricula. AI training and guidance, adequate time to develop educational resources, and funding and support from higher education institutions were key priorities as highlighted by educators.

Deep learning for tubes and lines detection in critical illness: Generalizability and comparison with residents

Background

Artificial intelligence (AI) has been proven useful for the assessment of tubes and lines on chest radiographs of general patients. However, validation on intensive care unit (ICU) patients remains imperative.

Methods

This retrospective case-control study evaluated the performance of deep learning (DL) models for tubes and lines classification on both an external public dataset and a local dataset comprising 303 films randomly sampled from the ICU database. The endotracheal tubes (ETTs), central venous catheters (CVCs), and nasogastric tubes (NGTs) were classified into "Normal," "Abnormal," or "Borderline" positions by DL models with and without rule-based modification. Their performance was evaluated using an experienced radiologist as the standard reference.

Results

The algorithm showed decreased performance on the local ICU dataset, compared to that of the external dataset, decreasing from the Area Under the Curve of Receiver (AUC) of 0.967 (95 % CI 0.965-0.973) to the AUC of 0.70 (95 % CI 0.68-0.77). Significant improvement in the ETT classification task was observed after modifications were made to the model to allow the use of the spatial relationship between line tips and reference anatomy with the improvement of the AUC, increasing from 0.71 (95 % CI 0.70 - 0.75) to 0.86 (95 % CI 0.83 - 0.94).

Conclusions

The externally trained model exhibited limited generalizability on the local ICU dataset. Therefore, evaluating the performance of externally trained AI before integrating it into critical care routine is crucial. Rule-based algorithm may be used in combination with DL to improve results.

Leveraging foundation and large language models in medical artificial intelligence

ABSTRACT

Recent advancements in the field of medical artificial intelligence (AI) have led to the widespread adoption of foundational and large language models. This review paper explores their applications within medical AI, introducing a novel classification framework that categorizes them as disease-specific, general-domain, and multi-modal models. The paper also addresses key challenges such as data acquisition and augmentation, including issues related to data volume, annotation, multi-modal fusion, and privacy concerns. Additionally, it discusses the evaluation, validation, limitations, and regulation of medical AI models, emphasizing their transformative potential in healthcare. The importance of continuous improvement, data security, standardized evaluations, and collaborative approaches is highlighted to ensure the responsible and effective integration of AI into clinical applications.

Precision Metrics: A Narrative Review on Unlocking the Power of KPIs in Radiology for Enhanced Precision Medicine

(Background) Over the years, there has been increasing interest in adopting a quality approach in radiology, leading to the strategic pursuit of specific and key performance indicators (KPIs). These indicators in radiology can have significant impacts ranging from radiation protection to integration into digital healthcare. (Purpose) This study aimed to conduct a narrative review on the integration of key performance indicators (KPIs) in radiology with specific key questions. (Methods) This review utilized a standardized checklist for narrative reviews, including the ANDJ Narrative Checklist, to ensure thoroughness and consistency. Searches were performed on PubMed, Scopus, and Google Scholar using a combination of keywords related to radiology and KPIs, with Boolean logic to refine results. From an initial yield of 211 studies, 127 were excluded due to a lack of focus on KPIs. The remaining 84 studies were assessed for clarity, design, and methodology, with 26 ultimately selected for detailed review. The evaluation process involved multiple assessors to minimize bias and ensure a rigorous analysis. (Results and Discussion) This overview highlights the following: KPIs are crucial for advancing radiology by supporting the evolution of imaging technologies (e.g., CT, MRI) and integrating emerging technologies like AI and AR/VR. They ensure high standards in diagnostic accuracy, image quality, and operational efficiency, enhancing diagnostic capabilities and streamlining workflows. KPIs are vital for radiological safety, measuring adherence to protocols that minimize radiation exposure and protect patients. The effective integration of KPIs into healthcare systems requires systematic development, validation, and standardization, supported by national and international initiatives. Addressing challenges like CAD-CAM technology and home-based radiology is essential. Developing specialized KPIs for new technologies will be key to continuous improvement in patient care and radiological practices. (Conclusions) In conclusion, KPIs are essential for advancing radiology, while future research should focus on improving data access and developing specialized KPIs to address emerging challenges. Future research should focus on expanding documentation sources, improving web search methods, and establishing direct connections with scientific associations.

Oncologic Applications of Artificial Intelligence and Deep Learning Methods in CT Spine Imaging-A Systematic Review

In spinal oncology, integrating deep learning with computed tomography (CT) imaging has shown promise in enhancing diagnostic accuracy, treatment planning, and patient outcomes. This systematic review synthesizes evidence on artificial intelligence (AI) applications in CT imaging for spinal tumors. A PRISMA-guided search identified 33 studies: 12 (36.4%) focused on detecting spinal malignancies, 11 (33.3%) on classification, 6 (18.2%) on prognostication, 3 (9.1%) on treatment planning, and 1 (3.0%) on both detection and classification. Of the classification studies, 7 (21.2%) used machine learning to distinguish between benign and malignant lesions, 3 (9.1%) evaluated tumor stage or grade, and 2 (6.1%) employed radiomics for biomarker classification. Prognostic studies included three (9.1%) that predicted complications such as pathological fractures and three (9.1%) that predicted treatment outcomes. AI's potential for improving workflow efficiency, aiding decision-making, and reducing complications is discussed, along with its limitations in generalizability, interpretability, and clinical integration. Future directions for AI in spinal oncology are also explored. In conclusion, while AI technologies in CT imaging are promising, further research is necessary to validate their clinical effectiveness and optimize their integration into routine practice.

An Introductory Guide to Artificial Intelligence in Interventional Radiology: Part 1 Foundational Knowledge

Artificial intelligence (AI) is rapidly evolving and has transformative potential for interventional radiology (IR) clinical practice. However, formal training in AI may be limited for many clinicians and therefore presents a challenge for initial implementation and trust in AI. An understanding of the foundational concepts in AI may help familiarize the interventional radiologist with the field of AI, thus facilitating understanding and participation in the development and deployment of AI. A pragmatic classification system of AI based on the complexity of the model may guide clinicians in the assessment of AI. Finally, the current state of AI in IR and the patterns of implementation are explored (pre-procedural, intra-procedural, and post-procedural).

An Introductory Guide to Artificial Intelligence in Interventional Radiology: Part 2: Implementation Considerations and Harms

The introduction of artificial intelligence (AI) in interventional radiology (IR) will bring about new challenges and opportunities for patients and clinicians. AI may comprise software as a medical device or AI-integrated hardware and will require a rigorous evaluation that should be guided based on the level of risk of the implementation. A hierarchy of risk of harm and possible harms are described herein. A checklist to guide deployment of an AI in a clinical IR environment is provided. As AI continues to evolve, regulation and evaluation of the AI medical devices will need to continue to evolve to keep pace and ensure patient safety.

Clinical, Cultural, Computational, and Regulatory Considerations to Deploy AI in Radiology: Perspectives of RSNA and MICCAI Experts

The Radiological Society of North of America (RSNA) and the Medical Image Computing and Computer Assisted Intervention (MICCAI) Society have led a series of joint panels and seminars focused on the present impact and future directions of artificial intelligence (AI) in radiology. These conversations have collected viewpoints from multidisciplinary experts in radiology, medical imaging, and machine learning on the current clinical penetration of AI technology in radiology and how it is impacted by trust, reproducibility, explainability, and accountability. The collective points-both practical and philosophical-define the cultural changes for radiologists and AI scientists working together and describe the challenges ahead for AI technologies to meet broad approval. This article presents the perspectives of experts from MICCAI and RSNA on the clinical, cultural, computational, and regulatory considerations-coupled with recommended reading materials-essential to adopt AI technology successfully in radiology and, more generally, in clinical practice. The report emphasizes the importance of collaboration to improve clinical deployment, highlights the need to integrate clinical and medical imaging data, and introduces strategies to ensure smooth and incentivized integration. Keywords: Adults and Pediatrics, Computer Applications-General (Informatics), Diagnosis, Prognosis © RSNA, 2024.

Developing, Purchasing, Implementing and Monitoring AI Tools in Radiology: Practical Considerations. A Multi-Society Statement From the ACR, CAR, ESR, RANZCR & RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever‑growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi‑society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools.

Radiology AI Deployment and Assessment Rubric (RADAR) to bring value-based AI into radiological practice

OBJECTIVE

To provide a comprehensive framework for value assessment of artificial intelligence (AI) in radiology.

METHODS

This paper presents the RADAR framework, which has been adapted from Fryback and Thornbury's imaging efficacy framework to facilitate the valuation of radiology AI from conception to local implementation. Local efficacy has been newly introduced to underscore the importance of appraising an AI technology within its local environment. Furthermore, the RADAR framework is illustrated through a myriad of study designs that help assess value.

RESULTS

RADAR presents a seven-level hierarchy, providing radiologists, researchers, and policymakers with a structured approach to the comprehensive assessment of value in radiology AI. RADAR is designed to be dynamic and meet the different valuation needs throughout the AI's lifecycle. Initial phases like technical and diagnostic efficacy (RADAR-1 and RADAR-2) are assessed pre-clinical deployment via in silico clinical trials and cross-sectional studies. Subsequent stages, spanning from diagnostic thinking to patient outcome efficacy (RADAR-3 to RADAR-5), require clinical integration and are explored via randomized controlled trials and cohort studies. Cost-effectiveness efficacy (RADAR-6) takes a societal perspective on financial feasibility, addressed via health-economic evaluations. The final level, RADAR-7, determines how prior valuations translate locally, evaluated through budget impact analysis, multi-criteria decision analyses, and prospective monitoring.

CONCLUSION

The RADAR framework offers a comprehensive framework for valuing radiology AI. Its layered, hierarchical structure, combined with a focus on local relevance, aligns RADAR seamlessly with the principles of value-based radiology.

CRITICAL RELEVANCE STATEMENT

The RADAR framework advances artificial intelligence in radiology by delineating a much-needed framework for comprehensive valuation.

KEYPOINTS

• Radiology artificial intelligence lacks a comprehensive approach to value assessment. • The RADAR framework provides a dynamic, hierarchical method for thorough valuation of radiology AI. • RADAR advances clinical radiology by bridging the artificial intelligence implementation gap.

Artificial intelligence in the risk prediction models of cardiovascular disease and development of an independent validation screening tool: a systematic review

BACKGROUND

A comprehensive overview of artificial intelligence (AI) for cardiovascular disease (CVD) prediction and a screening tool of AI models (AI-Ms) for independent external validation are lacking. This systematic review aims to identify, describe, and appraise AI-Ms of CVD prediction in the general and special populations and develop a new independent validation score (IVS) for AI-Ms replicability evaluation.

METHODS

PubMed, Web of Science, Embase, and IEEE library were searched up to July 2021. Data extraction and analysis were performed for the populations, distribution, predictors, algorithms, etc. The risk of bias was evaluated with the prediction risk of bias assessment tool (PROBAST). Subsequently, we designed IVS for model replicability evaluation with five steps in five items, including transparency of algorithms, performance of models, feasibility of reproduction, risk of reproduction, and clinical implication, respectively. The review is registered in PROSPERO (No. CRD42021271789).

RESULTS

In 20,887 screened references, 79 articles (82.5% in 2017-2021) were included, which contained 114 datasets (67 in Europe and North America, but 0 in Africa). We identified 486 AI-Ms, of which the majority were in development (n = 380), but none of them had undergone independent external validation. A total of 66 idiographic algorithms were found; however, 36.4% were used only once and only 39.4% over three times. A large number of different predictors (range 5-52,000, median 21) and large-span sample size (range 80-3,660,000, median 4466) were observed. All models were at high risk of bias according to PROBAST, primarily due to the incorrect use of statistical methods. IVS analysis confirmed only 10 models as "recommended"; however, 281 and 187 were "not recommended" and "warning," respectively.

CONCLUSION

AI has led the digital revolution in the field of CVD prediction, but is still in the early stage of development as the defects of research design, report, and evaluation systems. The IVS we developed may contribute to independent external validation and the development of this field.

Developing, purchasing, implementing and monitoring AI tools in radiology: Practical considerations. A multi-society statement from the ACR, CAR, ESR, RANZCR & RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever-growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi-society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools.

Developing, purchasing, implementing and monitoring AI tools in radiology: practical considerations. A multi-society statement from the ACR, CAR, ESR, RANZCR & RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever-growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones.This multi-society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools.Key points • The incorporation of artificial intelligence (AI) in radiological practice demands increased monitoring of its utility and safety.• Cooperation between developers, clinicians, and regulators will allow all involved to address ethical issues and monitor AI performance.• AI can fulfil its promise to advance patient well-being if all steps from development to integration in healthcare are rigorously evaluated.

Multimodal Deep Learning for Integrating Chest Radiographs and Clinical Parameters: A Case for Transformers

Background Clinicians consider both imaging and nonimaging data when diagnosing diseases; however, current machine learning approaches primarily consider data from a single modality. Purpose To develop a neural network architecture capable of integrating multimodal patient data and compare its performance to models incorporating a single modality for diagnosing up to 25 pathologic conditions. Materials and Methods In this retrospective study, imaging and nonimaging patient data were extracted from the Medical Information Mart for Intensive Care (MIMIC) database and an internal database comprised of chest radiographs and clinical parameters inpatients in the intensive care unit (ICU) (January 2008 to December 2020). The MIMIC and internal data sets were each split into training (n = 33 893, n = 28 809), validation (n = 740, n = 7203), and test (n = 1909, n = 9004) sets. A novel transformer-based neural network architecture was trained to diagnose up to 25 conditions using nonimaging data alone, imaging data alone, or multimodal data. Diagnostic performance was assessed using area under the receiver operating characteristic curve (AUC) analysis. Results The MIMIC and internal data sets included 36 542 patients (mean age, 63 years ± 17 [SD]; 20 567 male patients) and 45 016 patients (mean age, 66 years ± 16; 27 577 male patients), respectively. The multimodal model showed improved diagnostic performance for all pathologic conditions. For the MIMIC data set, the mean AUC was 0.77 (95% CI: 0.77, 0.78) when both chest radiographs and clinical parameters were used, compared with 0.70 (95% CI: 0.69, 0.71; P < .001) for only chest radiographs and 0.72 (95% CI: 0.72, 0.73; P < .001) for only clinical parameters. These findings were confirmed on the internal data set. Conclusion A model trained on imaging and nonimaging data outperformed models trained on only one type of data for diagnosing multiple diseases in patients in an ICU setting. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Kitamura and Topol in this issue.

Artificial Intelligence for Automated DWI/FLAIR Mismatch Assessment on Magnetic Resonance Imaging in Stroke: A Systematic Review

We conducted this Systematic Review to create an overview of the currently existing Artificial Intelligence (AI) methods for Magnetic Resonance Diffusion-Weighted Imaging (DWI)/Fluid-Attenuated Inversion Recovery (FLAIR)-mismatch assessment and to determine how well DWI/FLAIR mismatch algorithms perform compared to domain experts. We searched PubMed Medline, Ovid Embase, Scopus, Web of Science, Cochrane, and IEEE Xplore literature databases for relevant studies published between 1 January 2017 and 20 November 2022, following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. We assessed the included studies using the Quality Assessment of Diagnostic Accuracy Studies 2 tool. Five studies fit the scope of this review. The area under the curve ranged from 0.74 to 0.90. The sensitivity and specificity ranged from 0.70 to 0.85 and 0.74 to 0.84, respectively. Negative predictive value, positive predictive value, and accuracy ranged from 0.55 to 0.82, 0.74 to 0.91, and 0.73 to 0.83, respectively. In a binary classification of ±4.5 h from stroke onset, the surveyed AI methods performed equivalent to or even better than domain experts. However, using the relation between time since stroke onset (TSS) and increasing visibility of FLAIR hyperintensity lesions is not recommended for the determination of TSS within the first 4.5 h. An AI algorithm on DWI/FLAIR mismatch assessment focused on treatment eligibility, outcome prediction, and consideration of patient-specific data could potentially increase the proportion of stroke patients with unknown onset who could be treated with thrombolysis.

AI vs FRCR: What it means for the future

A recent work by Shelmerdine et al. was published in the Christmas edition of the BMJ. The authors were inspired by George Hinton's statement that artificial intelligence (AI) would supersede radiologists, and ventured to investigate whether the AI software Milvue Suite which had been trained on a few hundred thousand chest and musculoskeletal x-rays, could pass the rapid reporting section of the FRCR - an exam which must be passed in order to practice as a consultant radiologist in the UK. This brief comment sums up the company's opinions and perspective from the practical AI developmental angle and also its translation into a commercially viable and clinically useful tool. Hoping this will provide a fair and balanced view of the role of AI in radiology.