To buy or not to buy-evaluating commercial AI solutions in radiology (the ECLAIR guidelines)

Making sense of artificial intelligence and large language models-including ChatGPT-in pediatric hematology/oncology

ChatGPT and other artificial intelligence (AI) systems have captivated the attention of healthcare providers and researchers for their potential to improve care processes and outcomes. While these technologies hold promise to automate processes, increase efficiency, and reduce cognitive burden, their use also carries risks. In this commentary, we review basic concepts of AI, outline some of the capabilities and limitations of currently available tools, discuss current and future applications in pediatric hematology/oncology, and provide an evaluation and implementation framework that can be used by pediatric hematologist/oncologists considering the use of AI in clinical practice.

Developing, Purchasing, Implementing and Monitoring AI Tools in Radiology: Practical Considerations. A Multi-Society Statement From the ACR, CAR, ESR, RANZCR & RSNA

Artificial intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever-growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi-society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools. KEY POINTS.

Artificial intelligence in endodontics: Data preparation, clinical applications, ethical considerations, limitations, and future directions

Artificial intelligence (AI) is emerging as a transformative technology in healthcare, including endodontics. A gap in knowledge exists in understanding AI's applications and limitations among endodontic experts. This comprehensive review aims to (A) elaborate on technical and ethical aspects of using data to implement AI models in endodontics; (B) elaborate on evaluation metrics; (C) review the current applications of AI in endodontics; and (D) review the limitations and barriers to real-world implementation of AI in the field of endodontics and its future potentials/directions. The article shows that AI techniques have been applied in endodontics for critical tasks such as detection of radiolucent lesions, analysis of root canal morphology, prediction of treatment outcome and post-operative pain and more. Deep learning models like convolutional neural networks demonstrate high accuracy in these applications. However, challenges remain regarding model interpretability, generalizability, and adoption into clinical practice. When thoughtfully implemented, AI has great potential to aid with diagnostics, treatment planning, clinical interventions, and education in the field of endodontics. However, concerted efforts are still needed to address limitations and to facilitate integration into clinical workflows.

Clinical, Cultural, Computational, and Regulatory Considerations to Deploy AI in Radiology: Perspectives of RSNA and MICCAI Experts

The Radiological Society of North of America (RSNA) and the Medical Image Computing and Computer Assisted Intervention (MICCAI) Society have led a series of joint panels and seminars focused on the present impact and future directions of artificial intelligence (AI) in radiology. These conversations have collected viewpoints from multidisciplinary experts in radiology, medical imaging, and machine learning on the current clinical penetration of AI technology in radiology and how it is impacted by trust, reproducibility, explainability, and accountability. The collective points-both practical and philosophical-define the cultural changes for radiologists and AI scientists working together and describe the challenges ahead for AI technologies to meet broad approval. This article presents the perspectives of experts from MICCAI and RSNA on the clinical, cultural, computational, and regulatory considerations-coupled with recommended reading materials-essential to adopt AI technology successfully in radiology and, more generally, in clinical practice. The report emphasizes the importance of collaboration to improve clinical deployment, highlights the need to integrate clinical and medical imaging data, and introduces strategies to ensure smooth and incentivized integration. Keywords: Adults and Pediatrics, Computer Applications-General (Informatics), Diagnosis, Prognosis © RSNA, 2024.

Artificial intelligence (AI) implementation within the National Health Service (NHS): the South West London AI Working Group experience

In the rapidly evolving field of artificial intelligence (AI) for radiology, with a plethora of vendor options and use-cases and evidence claims to sift through, the pressing question is how to effectively implement the right tool for enhanced patient care? This article presents a structured approach to AI deployment, drawing from a comprehensive case study in South West London. We underscore the necessity of forming a dedicated AI team with a clear vision and assertive leadership to navigate such complexities. Central to our discussion is the significance of crafting an AI implementation plan, with an overarching aim to augment patient care, promote operational efficiency, and lay down standardized protocols for seamless AI adoption. By presenting a blueprint for AI implementation within the National Health Service (NHS), we intend to demystify the process for radiology departments across the UK, enabling them to make informed decisions and empowering their staff to embrace and leverage AI responsibly ensuring that patient welfare remains at the heart of innovation. Thus, having a framework to follow when implementing an AI solution that addresses a vision for scalable adoption, core team members with diversity of skillset, staff engagement and education, plan for vendor selection, and change management is crucial for success.

New Frontiers in Breast Cancer Imaging: The Rise of AI

Artificial intelligence (AI) has been implemented in multiple fields of medicine to assist in the diagnosis and treatment of patients. AI implementation in radiology, more specifically for breast imaging, has advanced considerably. Breast cancer is one of the most important causes of cancer mortality among women, and there has been increased attention towards creating more efficacious methods for breast cancer detection utilizing AI to improve radiologist accuracy and efficiency to meet the increasing demand of our patients. AI can be applied to imaging studies to improve image quality, increase interpretation accuracy, and improve time efficiency and cost efficiency. AI applied to mammography, ultrasound, and MRI allows for improved cancer detection and diagnosis while decreasing intra- and interobserver variability. The synergistic effect between a radiologist and AI has the potential to improve patient care in underserved populations with the intention of providing quality and equitable care for all. Additionally, AI has allowed for improved risk stratification. Further, AI application can have treatment implications as well by identifying upstage risk of ductal carcinoma in situ (DCIS) to invasive carcinoma and by better predicting individualized patient response to neoadjuvant chemotherapy. AI has potential for advancement in pre-operative 3-dimensional models of the breast as well as improved viability of reconstructive grafts.

Minimum labelling requirements for dermatology artificial intelligence-based Software as Medical Device (SaMD): A consensus statement

BACKGROUND/OBJECTIVES

Artificial intelligence (AI) holds remarkable potential to improve care delivery in dermatology. End users (health professionals and general public) of AI-based Software as Medical Devices (SaMD) require relevant labelling information to ensure that these devices can be used appropriately. Currently, there are no clear minimum labelling requirements for dermatology AI-based SaMDs.

METHODS

Common labelling recommendations for AI-based SaMD identified in a recent literature review were evaluated by an Australian expert panel in digital health and dermatology via a modified Delphi consensus process. A nine-point Likert scale was used to indicate importance of 10 items, and voting was conducted to determine the specific characteristics to include for some items. Consensus was achieved when more than 75% of the experts agreed that inclusion of information was necessary.

RESULTS

There was robust consensus supporting inclusion of all proposed items as minimum labelling requirements; indication for use, intended user, training and test data sets, algorithm design, image processing techniques, clinical validation, performance metrics, limitations, updates and adverse events. Nearly all suggested characteristics of the labelling items received endorsement, except for some characteristics related to performance metrics. Moreover, there was consensus that uniform labelling criteria should apply across all AI categories and risk classes set out by the Therapeutic Goods Administration.

CONCLUSIONS

This study provides critical evidence for setting labelling standards by the Therapeutic Goods Administration to safeguard patients, health professionals, consumers, industry, and regulatory bodies from AI-based dermatology SaMDs that do not currently provide adequate information about how they were developed and tested.

Towards quality management of artificial intelligence systems for medical applications

The use of artificial intelligence systems in clinical routine is still hampered by the necessity of a medical device certification and/or by the difficulty of implementing these systems in a clinic's quality management system. In this context, the key questions for a user are how to ensure robust model predictions and how to appraise the quality of a model's results on a regular basis. In this paper we discuss some conceptual foundation for a clinical implementation of a machine learning system and argue that both vendors and users should take certain responsibilities, as is already common practice for high-risk medical equipment. We propose the methodology from AAPM Task Group 100 report No. 283 as a conceptual framework for developing risk-driven a quality management program for a clinical process that encompasses a machine learning system. This is illustrated with an example of a clinical workflow. Our analysis shows how the risk evaluation in this framework can accommodate artificial intelligence based systems independently of their robustness evaluation or the user's in-house expertise. In particular, we highlight how the degree of interpretability of a machine learning system can be systematically accounted for within the risk evaluation and in the development of a quality management system.

Examining labelling guidelines for AI-based software as a medical device: A review and analysis of dermatology mobile applications in Australia

In recent years, there has been a surge in the development of AI-based Software as a Medical Device (SaMD), particularly in visual specialties such as dermatology. In Australia, the Therapeutic Goods Administration (TGA) regulates AI-based SaMD to ensure its safe use. Proper labelling of these devices is crucial to ensure that healthcare professionals and the general public understand how to use them and interpret results accurately. However, guidelines for labelling AI-based SaMD in dermatology are lacking, which may result in products failing to provide essential information about algorithm development and performance metrics. This review examines existing labelling guidelines for AI-based SaMD across visual medical specialties, with a specific focus on dermatology. Common recommendations for labelling are identified and applied to currently available dermatology AI-based SaMD mobile applications to determine usage of these labels. Of the 21 AI-based SaMD mobile applications identified, none fully comply with common labelling recommendations. Results highlight the need for standardized labelling guidelines. Ensuring transparency and accessibility of information is essential for the safe integration of AI into health care and preventing potential risks associated with inaccurate clinical decisions.

Survey on Value Elements Provided by Artificial Intelligence and Their Eligibility for Insurance Coverage With an Emphasis on Patient-Centered Outcomes

OBJECTIVE

This study aims to explore the opinions on the insurance coverage of artificial intelligence (AI), as categorized based on the distinct value elements offered by AI, with a specific focus on patient-centered outcomes (PCOs). PCOs are distinguished from traditional clinical outcomes and focus on patient-reported experiences and values such as quality of life, functionality, well-being, physical or emotional status, and convenience.

MATERIALS AND METHODS

We classified the value elements provided by AI into four dimensions: clinical outcomes, economic aspects, organizational aspects, and non-clinical PCOs. The survey comprised three sections: 1) experiences with PCOs in evaluating AI, 2) opinions on the coverage of AI by the National Health Insurance of the Republic of Korea when AI demonstrated benefits across the four value elements, and 3) respondent characteristics. The opinions regarding AI insurance coverage were assessed dichotomously and semi-quantitatively: non-approval (0) vs. approval (on a 1-10 weight scale, with 10 indicating the strongest approval). The survey was conducted from July 4 to 26, 2023, using a web-based method. Responses to PCOs and other value elements were compared.

RESULTS

Among 200 respondents, 44 (22%) were patients/patient representatives, 64 (32%) were industry/developers, 60 (30%) were medical practitioners/doctors, and 32 (16%) were government health personnel. The level of experience with PCOs regarding AI was low, with only 7% (14/200) having direct experience and 10% (20/200) having any experience (either direct or indirect). The approval rate for insurance coverage for PCOs was 74% (148/200), significantly lower than the corresponding rates for other value elements (82.5%-93.5%; P ≤ 0.034). The approval strength was significantly lower for PCOs, with a mean weight ± standard deviation of 5.1 ± 3.5, compared to other value elements (P ≤ 0.036).

CONCLUSION

There is currently limited demand for insurance coverage for AI that demonstrates benefits in terms of non-clinical PCOs.

Implementing Artificial Intelligence for Emergency Radiology Impacts Physicians' Knowledge and Perception: A Prospective Pre- and Post-Analysis

PURPOSE

The aim of this study was to evaluate the impact of implementing an artificial intelligence (AI) solution for emergency radiology into clinical routine on physicians' perception and knowledge.

MATERIALS AND METHODS

A prospective interventional survey was performed pre-implementation and 3 months post-implementation of an AI algorithm for fracture detection on radiographs in late 2022. Radiologists and traumatologists were asked about their knowledge and perception of AI on a 7-point Likert scale (-3, "strongly disagree"; +3, "strongly agree"). Self-generated identification codes allowed matching the same individuals pre-intervention and post-intervention, and using Wilcoxon signed rank test for paired data.

RESULTS

A total of 47/71 matched participants completed both surveys (66% follow-up rate) and were eligible for analysis (34 radiologists [72%], 13 traumatologists [28%], 15 women [32%]; mean age, 34.8 ± 7.8 years). Postintervention, there was an increase that AI "reduced missed findings" (1.28 [pre] vs 1.94 [post], P = 0.003) and made readers "safer" (1.21 vs 1.64, P = 0.048), but not "faster" (0.98 vs 1.21, P = 0.261). There was a rising disagreement that AI could "replace the radiological report" (-2.04 vs -2.34, P = 0.038), as well as an increase in self-reported knowledge about "clinical AI," its "chances," and its "risks" (0.40 vs 1.00, 1.21 vs 1.70, and 0.96 vs 1.34; all P 's ≤ 0.028). Radiologists used AI results more frequently than traumatologists ( P < 0.001) and rated benefits higher (all P 's ≤ 0.038), whereas senior physicians were less likely to use AI or endorse its benefits (negative correlation with age, -0.35 to 0.30; all P 's ≤ 0.046).

CONCLUSIONS

Implementing AI for emergency radiology into clinical routine has an educative aspect and underlines the concept of AI as a "second reader," to support and not replace physicians.

Developing, Purchasing, Implementing and Monitoring AI Tools in Radiology: Practical Considerations. A Multi-Society Statement From the ACR, CAR, ESR, RANZCR & RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever‑growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi‑society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools.

AI Applications to Breast MRI: Today and Tomorrow

In breast imaging, there is an unrelenting increase in the demand for breast imaging services, partly explained by continuous expanding imaging indications in breast diagnosis and treatment. As the human workforce providing these services is not growing at the same rate, the implementation of artificial intelligence (AI) in breast imaging has gained significant momentum to maximize workflow efficiency and increase productivity while concurrently improving diagnostic accuracy and patient outcomes. Thus far, the implementation of AI in breast imaging is at the most advanced stage with mammography and digital breast tomosynthesis techniques, followed by ultrasound, whereas the implementation of AI in breast magnetic resonance imaging (MRI) is not moving along as rapidly due to the complexity of MRI examinations and fewer available dataset. Nevertheless, there is persisting interest in AI-enhanced breast MRI applications, even as the use of and indications of breast MRI continue to expand. This review presents an overview of the basic concepts of AI imaging analysis and subsequently reviews the use cases for AI-enhanced MRI interpretation, that is, breast MRI triaging and lesion detection, lesion classification, prediction of treatment response, risk assessment, and image quality. Finally, it provides an outlook on the barriers and facilitators for the adoption of AI in breast MRI. LEVEL OF EVIDENCE: 5 TECHNICAL EFFICACY: Stage 6.

Artificial Intelligence in Radiology: Opportunities and Challenges

Artificial intelligence's (AI) emergence in radiology elicits both excitement and uncertainty. AI holds promise for improving radiology with regards to clinical practice, education, and research opportunities. Yet, AI systems are trained on select datasets that can contain bias and inaccuracies. Radiologists must understand these limitations and engage with AI developers at every step of the process - from algorithm initiation and design to development and implementation - to maximize benefit and minimize harm that can be enabled by this technology.

Simulation of acquisition shifts in T2 weighted fluid-attenuated inversion recovery magnetic resonance images to stress test artificial intelligence segmentation networks

Purpose

To provide a simulation framework for routine neuroimaging test data, which allows for "stress testing" of deep segmentation networks against acquisition shifts that commonly occur in clinical practice for T2 weighted (T2w) fluid-attenuated inversion recovery magnetic resonance imaging protocols.

Approach

The approach simulates "acquisition shift derivatives" of MR images based on MR signal equations. Experiments comprise the validation of the simulated images by real MR scans and example stress tests on state-of-the-art multiple sclerosis lesion segmentation networks to explore a generic model function to describe the F1 score in dependence of the contrast-affecting sequence parameters echo time (TE) and inversion time (TI).

Results

The differences between real and simulated images range up to 19% in gray and white matter for extreme parameter settings. For the segmentation networks under test, the F1 score dependency on TE and TI can be well described by quadratic model functions (R 2 > 0.9 ). The coefficients of the model functions indicate that changes of TE have more influence on the model performance than TI.

Conclusions

We show that these deviations are in the range of values as may be caused by erroneous or individual differences in relaxation times as described by literature. The coefficients of the F1 model function allow for a quantitative comparison of the influences of TE and TI. Limitations arise mainly from tissues with a low baseline signal (like cerebrospinal fluid) and when the protocol contains contrast-affecting measures that cannot be modeled due to missing information in the DICOM header.

Detailed Image Data Quality and Cleaning Practices for Artificial Intelligence Tools for Breast Cancer

Standardizing image-data preparation practices to improve accuracy/consistency of AI diagnostic tools.

Empirical data drift detection experiments on real-world medical imaging data

While it is common to monitor deployed clinical artificial intelligence (AI) models for performance degradation, it is less common for the input data to be monitored for data drift - systemic changes to input distributions. However, when real-time evaluation may not be practical (eg., labeling costs) or when gold-labels are automatically generated, we argue that tracking data drift becomes a vital addition for AI deployments. In this work, we perform empirical experiments on real-world medical imaging to evaluate three data drift detection methods' ability to detect data drift caused (a) naturally (emergence of COVID-19 in X-rays) and (b) synthetically. We find that monitoring performance alone is not a good proxy for detecting data drift and that drift-detection heavily depends on sample size and patient features. Our work discusses the need and utility of data drift detection in various scenarios and highlights gaps in knowledge for the practical application of existing methods.

Women's perceptions and attitudes towards the use of AI in mammography in Sweden: a qualitative interview study

BACKGROUND

Understanding women's perspectives can help to create an effective and acceptable artificial intelligence (AI) implementation for triaging mammograms, ensuring a high proportion of screening-detected cancer. This study aimed to explore Swedish women's perceptions and attitudes towards the use of AI in mammography.

METHOD

Semistructured interviews were conducted with 16 women recruited in the spring of 2023 at Capio S:t Görans Hospital, Sweden, during an ongoing clinical trial of AI in screening (ScreenTrustCAD, NCT04778670) with Philips equipment. The interview transcripts were analysed using inductive thematic content analysis.

RESULTS

In general, women viewed AI as an excellent complementary tool to help radiologists in their decision-making, rather than a complete replacement of their expertise. To trust the AI, the women requested a thorough evaluation, transparency about AI usage in healthcare, and the involvement of a radiologist in the assessment. They would rather be more worried because of being called in more often for scans than risk having overlooked a sign of cancer. They expressed substantial trust in the healthcare system if the implementation of AI was to become a standard practice.

CONCLUSION

The findings suggest that the interviewed women, in general, hold a positive attitude towards the implementation of AI in mammography; nonetheless, they expect and demand more from an AI than a radiologist. Effective communication regarding the role and limitations of AI is crucial to ensure that patients understand the purpose and potential outcomes of AI-assisted healthcare.

Radiology AI Deployment and Assessment Rubric (RADAR) to bring value-based AI into radiological practice

OBJECTIVE

To provide a comprehensive framework for value assessment of artificial intelligence (AI) in radiology.

METHODS

This paper presents the RADAR framework, which has been adapted from Fryback and Thornbury's imaging efficacy framework to facilitate the valuation of radiology AI from conception to local implementation. Local efficacy has been newly introduced to underscore the importance of appraising an AI technology within its local environment. Furthermore, the RADAR framework is illustrated through a myriad of study designs that help assess value.

RESULTS

RADAR presents a seven-level hierarchy, providing radiologists, researchers, and policymakers with a structured approach to the comprehensive assessment of value in radiology AI. RADAR is designed to be dynamic and meet the different valuation needs throughout the AI's lifecycle. Initial phases like technical and diagnostic efficacy (RADAR-1 and RADAR-2) are assessed pre-clinical deployment via in silico clinical trials and cross-sectional studies. Subsequent stages, spanning from diagnostic thinking to patient outcome efficacy (RADAR-3 to RADAR-5), require clinical integration and are explored via randomized controlled trials and cohort studies. Cost-effectiveness efficacy (RADAR-6) takes a societal perspective on financial feasibility, addressed via health-economic evaluations. The final level, RADAR-7, determines how prior valuations translate locally, evaluated through budget impact analysis, multi-criteria decision analyses, and prospective monitoring.

CONCLUSION

The RADAR framework offers a comprehensive framework for valuing radiology AI. Its layered, hierarchical structure, combined with a focus on local relevance, aligns RADAR seamlessly with the principles of value-based radiology.

CRITICAL RELEVANCE STATEMENT

The RADAR framework advances artificial intelligence in radiology by delineating a much-needed framework for comprehensive valuation.

KEYPOINTS

• Radiology artificial intelligence lacks a comprehensive approach to value assessment. • The RADAR framework provides a dynamic, hierarchical method for thorough valuation of radiology AI. • RADAR advances clinical radiology by bridging the artificial intelligence implementation gap.

French community grid for the evaluation of radiological artificial intelligence solutions (DRIM France Artificial Intelligence Initiative)

PURPOSE

The purpose of this study was to validate a national descriptive and analytical grid for artificial intelligence (AI) solutions in radiology.

MATERIALS AND METHODS

The RAND-UCLA Appropriateness Method was chosen by expert radiologists from the DRIM France IA group for this statement paper. The study, initiated by the radiology community, involved seven steps including literature review, template development, panel selection, pre-panel meeting survey, data extraction and analysis, second and final panel meeting, and data reporting.

RESULTS

The panel consisted of seven software vendors, three for bone fracture detection using conventional radiology and four for breast cancer detection using mammography. A consensus was reached on various aspects, including general target, main objective, certification marking, integration, expression of results, forensic aspects and cybersecurity, performance and scientific validation, description of the company and economic details, possible usage scenarios in the clinical workflow, database, specific objectives and targets of the AI tool.

CONCLUSION

The study validates a descriptive and analytical grid for radiological AI solutions consisting of ten items, using breast cancer and bone fracture as an experimental guide. This grid would assist radiologists in selecting relevant and validated AI solutions. Further developments of the grid are needed to include other organs and tasks.

Growing importance of brain morphometry analysis in the clinical routine: The hidden impact of MR sequence parameters

Volumetric assessment based on structural MRI is increasingly recognized as an auxiliary tool to visual reading, also in examinations acquired in the clinical routine. However, MRI acquisition parameters can significantly influence these measures, which must be considered when interpreting the results on an individual patient level. This Technical Note shall demonstrate the problem. Using data from a dedicated experiment, we show the influence of two crucial sequence parameters on the GM/WM contrast and their impact on the measured volumes. A simulated contrast derived from acquisition parameters TI/TR may serve as surrogate and is highly correlated (r=0.96) with the measured contrast.

[The contribution of artificial intelligence (AI) subsequent to the processing of thoracic imaging]

The contribution of artificial intelligence (AI) to medical imaging is currently the object of widespread experimentation. The development of deep learning (DL) methods, particularly convolution neural networks (CNNs), has led to performance gains often superior to those achieved by conventional methods such as machine learning. Radiomics is an approach aimed at extracting quantitative data not accessible to the human eye from images expressing a disease. The data subsequently feed machine learning models and produce diagnostic or prognostic probabilities. As for the multiple applications of AI methods in thoracic imaging, they are undergoing evaluation. Chest radiography is a practically ideal field for the development of DL algorithms able to automatically interpret X-rays. Current algorithms can detect up to 14 different abnormalities present either in isolation or in combination. Chest CT is another area offering numerous AI applications. Various algorithms have been specifically formed and validated for the detection and characterization of pulmonary nodules and pulmonary embolism, as well as segmentation and quantitative analysis of the extent of diffuse lung diseases (emphysema, infectious pneumonias, interstitial lung disease). In addition, the analysis of medical images can be associated with clinical, biological, and functional data (multi-omics analysis), the objective being to construct predictive approaches regarding disease prognosis and response to treatment.

Environmental Sustainability and AI in Radiology: A Double-Edged Sword

According to the World Health Organization, climate change is the single biggest health threat facing humanity. The global health care system, including medical imaging, must manage the health effects of climate change while at the same time addressing the large amount of greenhouse gas (GHG) emissions generated in the delivery of care. Data centers and computational efforts are increasingly large contributors to GHG emissions in radiology. This is due to the explosive increase in big data and artificial intelligence (AI) applications that have resulted in large energy requirements for developing and deploying AI models. However, AI also has the potential to improve environmental sustainability in medical imaging. For example, use of AI can shorten MRI scan times with accelerated acquisition times, improve the scheduling efficiency of scanners, and optimize the use of decision-support tools to reduce low-value imaging. The purpose of this Radiology in Focus article is to discuss this duality at the intersection of environmental sustainability and AI in radiology. Further discussed are strategies and opportunities to decrease AI-related emissions and to leverage AI to improve sustainability in radiology, with a focus on health equity. Co-benefits of these strategies are explored, including lower cost and improved patient outcomes. Finally, knowledge gaps and areas for future research are highlighted.

Developing, purchasing, implementing and monitoring AI tools in radiology: Practical considerations. A multi-society statement from the ACR, CAR, ESR, RANZCR & RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever-growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi-society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools.

A holistic approach to implementing artificial intelligence in radiology

OBJECTIVE

Despite the widespread recognition of the importance of artificial intelligence (AI) in healthcare, its implementation is often limited. This article aims to address this implementation gap by presenting insights from an in-depth case study of an organisation that approached AI implementation with a holistic approach.

MATERIALS AND METHODS

We conducted a longitudinal, qualitative case study of the implementation of AI in radiology at a large academic medical centre in the Netherlands for three years. Collected data consists of 43 days of work observations, 30 meeting observations, 18 interviews and 41 relevant documents. Abductive reasoning was used for systematic data analysis, which revealed three change initiative themes responding to specific AI implementation challenges.

RESULTS

This study identifies challenges of implementing AI in radiology at different levels and proposes a holistic approach to tackle those challenges. At the technology level, there is the issue of multiple narrow AI applications with no standard use interface; at the workflow level, AI results allow limited interaction with radiologists; at the people and organisational level, there are divergent expectations and limited experience with AI. The case of Southern illustrates that organisations can reap more benefits from AI implementation by investing in long-term initiatives that holistically align both social and technological aspects of clinical practice.

CONCLUSION

This study highlights the importance of a holistic approach to AI implementation that addresses challenges spanning technology, workflow, and organisational levels. Aligning change initiatives between these different levels has proven to be important to facilitate wide-scale implementation of AI in clinical practice.

CRITICAL RELEVANCE STATEMENT

Adoption of artificial intelligence is crucial for future-ready radiological care. This case study highlights the importance of a holistic approach that addresses technological, workflow, and organisational aspects, offering practical insights and solutions to facilitate successful AI adoption in clinical practice.

KEY POINTS

1. Practical and actionable insights into successful AI implementation in radiology are lacking. 2. Aligning technology, workflow, organisational aspects is crucial for a successful AI implementation 3. Holistic approach aids organisations to create sustainable value through AI implementation.

Developing, purchasing, implementing and monitoring AI tools in radiology: practical considerations. A multi-society statement from the ACR, CAR, ESR, RANZCR & RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever-growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones.This multi-society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools.Key points • The incorporation of artificial intelligence (AI) in radiological practice demands increased monitoring of its utility and safety.• Cooperation between developers, clinicians, and regulators will allow all involved to address ethical issues and monitor AI performance.• AI can fulfil its promise to advance patient well-being if all steps from development to integration in healthcare are rigorously evaluated.

Comparison of Commercial AI Software Performance for Radiograph Lung Nodule Detection and Bone Age Prediction

Background Multiple commercial artificial intelligence (AI) products exist for assessing radiographs; however, comparable performance data for these algorithms are limited. Purpose To perform an independent, stand-alone validation of commercially available AI products for bone age prediction based on hand radiographs and lung nodule detection on chest radiographs. Materials and Methods This retrospective study was carried out as part of Project AIR. Nine of 17 eligible AI products were validated on data from seven Dutch hospitals. For bone age prediction, the root mean square error (RMSE) and Pearson correlation coefficient were computed. The reference standard was set by three to five expert readers. For lung nodule detection, the area under the receiver operating characteristic curve (AUC) was computed. The reference standard was set by a chest radiologist based on CT. Randomized subsets of hand (n = 95) and chest (n = 140) radiographs were read by 14 and 17 human readers, respectively, with varying experience. Results Two bone age prediction algorithms were tested on hand radiographs (from January 2017 to January 2022) in 326 patients (mean age, 10 years ± 4 [SD]; 173 female patients) and correlated strongly with the reference standard (r = 0.99; P < .001 for both). No difference in RMSE was observed between algorithms (0.63 years [95% CI: 0.58, 0.69] and 0.57 years [95% CI: 0.52, 0.61]) and readers (0.68 years [95% CI: 0.64, 0.73]). Seven lung nodule detection algorithms were validated on chest radiographs (from January 2012 to May 2022) in 386 patients (mean age, 64 years ± 11; 223 male patients). Compared with readers (mean AUC, 0.81 [95% CI: 0.77, 0.85]), four algorithms performed better (AUC range, 0.86-0.93; P value range, <.001 to .04). Conclusions Compared with human readers, four AI algorithms for detecting lung nodules on chest radiographs showed improved performance, whereas the remaining algorithms tested showed no evidence of a difference in performance. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Omoumi and Richiardi in this issue.

Developing, Purchasing, Implementing and Monitoring AI Tools in Radiology: Practical Considerations. A Multi-Society Statement from the ACR, CAR, ESR, RANZCR and RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever-growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi-society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools. This article is simultaneously published in Insights into Imaging (DOI 10.1186/s13244-023-01541-3), Journal of Medical Imaging and Radiation Oncology (DOI 10.1111/1754-9485.13612), Canadian Association of Radiologists Journal (DOI 10.1177/08465371231222229), Journal of the American College of Radiology (DOI 10.1016/j.jacr.2023.12.005), and Radiology: Artificial Intelligence (DOI 10.1148/ryai.230513). Keywords: Artificial Intelligence, Radiology, Automation, Machine Learning Published under a CC BY 4.0 license. ©The Author(s) 2024. Editor's Note: The RSNA Board of Directors has endorsed this article. It has not undergone review or editing by this journal.

An Interventional Radiologist's Primer of Critical Appraisal of Artificial Intelligence Research

Recent advances in artificial intelligence (AI) are expected to cause a significant paradigm shift in all digital data-driven aspects of information gain, processing, and decision making in both clinical healthcare and medical research. The field of interventional radiology (IR) will be enmeshed in this innovation, yet the collective IR expertise in the field of AI remains rudimentary because of lack of training. This primer provides the clinical interventional radiologist with a simple guide for critically appraising AI research and products by identifying 12 fundamental items that should be considered: (a) need for AI technology to address the clinical problem, (b) type of applied Al algorithm, (c) data quality and degree of annotation, (d) reporting of accuracy, (e) applicability of standardized reporting, (f) reproducibility of methodology and data transparency, (g) algorithm validation, (h) interpretability, (i) concrete impact on IR, (j) pathway toward translation to clinical practice, (k) clinical benefit and cost-effectiveness, and (l) regulatory framework.

How AI May Transform Musculoskeletal Imaging

While musculoskeletal imaging volumes are increasing, there is a relative shortage of subspecialized musculoskeletal radiologists to interpret the studies. Will artificial intelligence (AI) be the solution? For AI to be the solution, the wide implementation of AI-supported data acquisition methods in clinical practice requires establishing trusted and reliable results. This implementation will demand close collaboration between core AI researchers and clinical radiologists. Upon successful clinical implementation, a wide variety of AI-based tools can improve the musculoskeletal radiologist's workflow by triaging imaging examinations, helping with image interpretation, and decreasing the reporting time. Additional AI applications may also be helpful for business, education, and research purposes if successfully integrated into the daily practice of musculoskeletal radiology. The question is not whether AI will replace radiologists, but rather how musculoskeletal radiologists can take advantage of AI to enhance their expert capabilities.

Commercially available artificial intelligence tools for fracture detection: the evidence

Missed fractures are a costly healthcare issue, not only negatively impacting patient lives, leading to potential long-term disability and time off work, but also responsible for high medicolegal disbursements that could otherwise be used to improve other healthcare services. When fractures are overlooked in children, they are particularly concerning as opportunities for safeguarding may be missed. Assistance from artificial intelligence (AI) in interpreting medical images may offer a possible solution for improving patient care, and several commercial AI tools are now available for radiology workflow implementation. However, information regarding their development, evidence for performance and validation as well as the intended target population is not always clear, but vital when evaluating a potential AI solution for implementation. In this article, we review the range of available products utilizing AI for fracture detection (in both adults and children) and summarize the evidence, or lack thereof, behind their performance. This will allow others to make better informed decisions when deciding which product to procure for their specific clinical requirements.

Artificial Intelligence and Radiomics for Endometrial Cancer MRI: Exploring the Whats, Whys and Hows

Endometrial cancer (EC) is intricately linked to obesity and diabetes, which are widespread risk factors. Medical imaging, especially magnetic resonance imaging (MRI), plays a major role in EC assessment, particularly for disease staging. However, the diagnostic performance of MRI exhibits variability in the detection of clinically relevant prognostic factors (e.g., deep myometrial invasion and metastatic lymph nodes assessment). To address these challenges and enhance the value of MRI, radiomics and artificial intelligence (AI) algorithms emerge as promising tools with a potential to impact EC risk assessment, treatment planning, and prognosis prediction. These advanced post-processing techniques allow us to quantitatively analyse medical images, providing novel insights into cancer characteristics beyond conventional qualitative image evaluation. However, despite the growing interest and research efforts, the integration of radiomics and AI to EC management is still far from clinical practice and represents a possible perspective rather than an actual reality. This review focuses on the state of radiomics and AI in EC MRI, emphasizing risk stratification and prognostic factor prediction, aiming to illuminate potential advancements and address existing challenges in the field.

Clinical Evaluation of a Quantitative Imaging Biomarker Supporting Radiological Assessment of Hippocampal Sclerosis

OBJECTIVE

To evaluate the influence of quantitative reports (QReports) on the radiological assessment of hippocampal sclerosis (HS) from MRI of patients with epilepsy in a setting mimicking clinical reality.

METHODS

The study included 40 patients with epilepsy, among them 20 with structural abnormalities in the mesial temporal lobe (13 with HS). Six raters blinded to the diagnosis assessed the 3T MRI in two rounds, first using MRI only and later with both MRI and the QReport. Results were evaluated using inter-rater agreement (Fleiss' kappa [Formula: see text]) and comparison with a consensus of two radiological experts derived from clinical and imaging data, including 7T MRI.

RESULTS

For the primary outcome, diagnosis of HS, the mean accuracy of the raters improved from 77.5% with MRI only to 86.3% with the additional QReport (effect size [Formula: see text]). Inter-rater agreement increased from [Formula: see text] to [Formula: see text]. Five of the six raters reached higher accuracies, and all reported higher confidence when using the QReports.

CONCLUSION

In this pre-use clinical evaluation study, we demonstrated clinical feasibility and usefulness as well as the potential impact of a previously suggested imaging biomarker for radiological assessment of HS.

Enhancing Interoperability and Harmonisation of Nuclear Medicine Image Data and Associated Clinical Data

Nuclear imaging techniques such as positron emission tomography (PET) and single photon emission computed tomography (SPECT) in combination with computed tomography (CT) are established imaging modalities in clinical practice, particularly for oncological problems. Due to a multitude of manufacturers, different measurement protocols, local demographic or clinical workflow variations as well as various available reconstruction and analysis software, very heterogeneous datasets are generated. This review article examines the current state of interoperability and harmonisation of image data and related clinical data in the field of nuclear medicine. Various approaches and standards to improve data compatibility and integration are discussed. These include, for example, structured clinical history, standardisation of image acquisition and reconstruction as well as standardised preparation of image data for evaluation. Approaches to improve data acquisition, storage and analysis will be presented. Furthermore, approaches are presented to prepare the datasets in such a way that they become usable for projects applying artificial intelligence (AI) (machine learning, deep learning, etc.). This review article concludes with an outlook on future developments and trends related to AI in nuclear medicine, including a brief research of commercial solutions.

Guidelines, Consensus Statements, and Standards for the Use of Artificial Intelligence in Medicine: Systematic Review

BACKGROUND

The application of artificial intelligence (AI) in the delivery of health care is a promising area, and guidelines, consensus statements, and standards on AI regarding various topics have been developed.

OBJECTIVE

We performed this study to assess the quality of guidelines, consensus statements, and standards in the field of AI for medicine and to provide a foundation for recommendations about the future development of AI guidelines.

METHODS

We searched 7 electronic databases from database establishment to April 6, 2022, and screened articles involving AI guidelines, consensus statements, and standards for eligibility. The AGREE II (Appraisal of Guidelines for Research & Evaluation II) and RIGHT (Reporting Items for Practice Guidelines in Healthcare) tools were used to assess the methodological and reporting quality of the included articles.

RESULTS

This systematic review included 19 guideline articles, 14 consensus statement articles, and 3 standard articles published between 2019 and 2022. Their content involved disease screening, diagnosis, and treatment; AI intervention trial reporting; AI imaging development and collaboration; AI data application; and AI ethics governance and applications. Our quality assessment revealed that the average overall AGREE II score was 4.0 (range 2.2-5.5; 7-point Likert scale) and the mean overall reporting rate of the RIGHT tool was 49.4% (range 25.7%-77.1%).

CONCLUSIONS

The results indicated important differences in the quality of different AI guidelines, consensus statements, and standards. We made recommendations for improving their methodological and reporting quality.

TRIAL REGISTRATION

PROSPERO International Prospective Register of Systematic Reviews (CRD42022321360); https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=321360.

Morphological assessment of cartilage and osteoarthritis in clinical practice and research: Intermediate-weighted fat-suppressed sequences and beyond

Magnetic resonance imaging (MRI) is widely regarded as the primary modality for the morphological assessment of cartilage and all other joint tissues involved in osteoarthritis. 2D fast spin echo fat-suppressed intermediate-weighted (FSE FS IW) sequences with a TE between 30 and 40ms have stood the test of time and are considered the cornerstone of MRI protocols for clinical practice and trials. These sequences offer a good balance between sensitivity and specificity and provide appropriate contrast and signal within the cartilage as well as between cartilage, articular fluid, and subchondral bone. Additionally, FS IW sequences enable the evaluation of menisci, ligaments, synovitis/effusion, and bone marrow edema-like signal changes. This review article provides a rationale for the use of FSE FS IW sequences in the morphological assessment of cartilage and osteoarthritis, along with a brief overview of other clinically available sequences for this indication. Additionally, the article highlights ongoing research efforts aimed at improving FSE FS IW sequences through 3D acquisitions with enhanced resolution, shortened examination times, and exploring the potential benefits of different magnetic field strengths. While most of the literature on cartilage imaging focuses on the knee, the concepts presented here are applicable to all joints. KEY POINTS: 1. MRI is currently considered the modality of reference for a "whole-joint" morphological assessment of osteoarthritis. 2. Fat-suppressed intermediate-weighted sequences remain the keystone of MRI protocols for the assessment of cartilage morphology, as well as other structures involved in osteoarthritis. 3. Trends for further development in the field of cartilage and joint imaging include 3D FSE imaging, faster acquisition including AI-based acceleration, and synthetic imaging providing multi-contrast sequences.

Artificial intelligence and the work-health interface: A research agenda for a technologically transforming world of work

The labor market is undergoing a rapid artificial intelligence (AI) revolution. There is currently limited empirical scholarship that focuses on how AI adoption affects employment opportunities and work environments in ways that shape worker health, safety, well-being and equity. In this article, we present an agenda to guide research examining the implications of AI on the intersection between work and health. To build the agenda, a full day meeting was organized and attended by 50 participants including researchers from diverse disciplines and applied stakeholders. Facilitated meeting discussions aimed to set research priorities related to workplace AI applications and its impact on the health of workers, including critical research questions, methodological approaches, data needs, and resource requirements. Discussions also aimed to identify groups of workers and working contexts that may benefit from AI adoption as well as those that may be disadvantaged by AI. Discussions were synthesized into four research agenda areas: (1) examining the impact of stronger AI on human workers; (2) advancing responsible and healthy AI; (3) informing AI policy for worker health, safety, well-being, and equitable employment; and (4) understanding and addressing worker and employer knowledge needs regarding AI applications. The agenda provides a roadmap for researchers to build a critical evidence base on the impact of AI on workers and workplaces, and will ensure that worker health, safety, well-being, and equity are at the forefront of workplace AI system design and adoption.

Improving the Computer-Aided Estimation of Ulcerative Colitis Severity According to Mayo Endoscopic Score by Using Regression-Based Deep Learning

BACKGROUND

Assessment of endoscopic activity in ulcerative colitis (UC) is important for treatment decisions and monitoring disease progress. However, substantial inter- and intraobserver variability in grading impairs the assessment. Our aim was to develop a computer-aided diagnosis system using deep learning to reduce subjectivity and improve the reliability of the assessment.

METHODS

The cohort comprises 11 276 images from 564 patients who underwent colonoscopy for UC. We propose a regression-based deep learning approach for the endoscopic evaluation of UC according to the Mayo endoscopic score (MES). Five state-of-the-art convolutional neural network (CNN) architectures were used for the performance measurements and comparisons. Ten-fold cross-validation was used to train the models and objectively benchmark them. Model performances were assessed using quadratic weighted kappa and macro F1 scores for full Mayo score classification and kappa statistics and F1 score for remission classification.

RESULTS

Five classification-based CNNs used in the study were in excellent agreement with the expert annotations for all Mayo subscores and remission classification according to the kappa statistics. When the proposed regression-based approach was used, (1) the performance of most of the models statistically significantly increased and (2) the same model trained on different cross-validation folds produced more robust results on the test set in terms of deviation between different folds.

CONCLUSIONS

Comprehensive experimental evaluations show that commonly used classification-based CNN architectures have successful performance in evaluating endoscopic disease activity of UC. Integration of domain knowledge into these architectures further increases performance and robustness, accelerating their translation into clinical use.

Artificial Intelligence in Neuroradiology: A Review of Current Topics and Competition Challenges

There is an expanding body of literature that describes the application of deep learning and other machine learning and artificial intelligence methods with potential relevance to neuroradiology practice. In this article, we performed a literature review to identify recent developments on the topics of artificial intelligence in neuroradiology, with particular emphasis on large datasets and large-scale algorithm assessments, such as those used in imaging AI competition challenges. Numerous applications relevant to ischemic stroke, intracranial hemorrhage, brain tumors, demyelinating disease, and neurodegenerative/neurocognitive disorders were discussed. The potential applications of these methods to spinal fractures, scoliosis grading, head and neck oncology, and vascular imaging were also reviewed. The AI applications examined perform a variety of tasks, including localization, segmentation, longitudinal monitoring, diagnostic classification, and prognostication. While research on this topic is ongoing, several applications have been cleared for clinical use and have the potential to augment the accuracy or efficiency of neuroradiologists.

Application of a Comprehensive Evaluation Framework to COVID-19 Studies: Systematic Review of Translational Aspects of Artificial Intelligence in Health Care

Background

Despite immense progress in artificial intelligence (AI) models, there has been limited deployment in health care environments. The gap between potential and actual AI applications is likely due to the lack of translatability between controlled research environments (where these models are developed) and clinical environments for which the AI tools are ultimately intended.

Objective

We previously developed the Translational Evaluation of Healthcare AI (TEHAI) framework to assess the translational value of AI models and to support successful transition to health care environments. In this study, we applied the TEHAI framework to the COVID-19 literature in order to assess how well translational topics are covered.

Methods

A systematic literature search for COVID-19 AI studies published between December 2019 and December 2020 resulted in 3830 records. A subset of 102 (2.7%) papers that passed the inclusion criteria was sampled for full review. The papers were assessed for translational value and descriptive data collected by 9 reviewers (each study was assessed by 2 reviewers). Evaluation scores and extracted data were compared by a third reviewer for resolution of discrepancies. The review process was conducted on the Covidence software platform.

Results

We observed a significant trend for studies to attain high scores for technical capability but low scores for the areas essential for clinical translatability. Specific questions regarding external model validation, safety, nonmaleficence, and service adoption received failed scores in most studies.

Conclusions

Using TEHAI, we identified notable gaps in how well translational topics of AI models are covered in the COVID-19 clinical sphere. These gaps in areas crucial for clinical translatability could, and should, be considered already at the model development stage to increase translatability into real COVID-19 health care environments.

An International Non-Inferiority Study for the Benchmarking of AI for Routine Radiology Cases: Chest X-ray, Fluorography and Mammography

An international reader study was conducted to gauge an average diagnostic accuracy of radiologists interpreting chest X-ray images, including those from fluorography and mammography, and establish requirements for stand-alone radiological artificial intelligence (AI) models. The retrospective studies in the datasets were labelled as containing or not containing target pathological findings based on a consensus of two experienced radiologists, and the results of a laboratory test and follow-up examination, where applicable. A total of 204 radiologists from 11 countries with various experience performed an assessment of the dataset with a 5-point Likert scale via a web platform. Eight commercial radiological AI models analyzed the same dataset. The AI AUROC was 0.87 (95% CI:0.83-0.9) versus 0.96 (95% CI 0.94-0.97) for radiologists. The sensitivity and specificity of AI versus radiologists were 0.71 (95% CI 0.64-0.78) versus 0.91 (95% CI 0.86-0.95) and 0.93 (95% CI 0.89-0.96) versus 0.9 (95% CI 0.85-0.94) for AI. The overall diagnostic accuracy of radiologists was superior to AI for chest X-ray and mammography. However, the accuracy of AI was noninferior to the least experienced radiologists for mammography and fluorography, and to all radiologists for chest X-ray. Therefore, an AI-based first reading could be recommended to reduce the workload burden of radiologists for the most common radiological studies such as chest X-ray and mammography.

AI vs FRCR: What it means for the future

A recent work by Shelmerdine et al. was published in the Christmas edition of the BMJ. The authors were inspired by George Hinton's statement that artificial intelligence (AI) would supersede radiologists, and ventured to investigate whether the AI software Milvue Suite which had been trained on a few hundred thousand chest and musculoskeletal x-rays, could pass the rapid reporting section of the FRCR - an exam which must be passed in order to practice as a consultant radiologist in the UK. This brief comment sums up the company's opinions and perspective from the practical AI developmental angle and also its translation into a commercially viable and clinically useful tool. Hoping this will provide a fair and balanced view of the role of AI in radiology.

Assessment of Radiology Artificial Intelligence Software: A Validation and Evaluation Framework

Artificial intelligence (AI) software in radiology is becoming increasingly prevalent and performance is improving rapidly with new applications for given use cases being developed continuously, oftentimes with development and validation occurring in parallel. Several guidelines have provided reporting standards for publications of AI-based research in medicine and radiology. Yet, there is an unmet need for recommendations on the assessment of AI software before adoption and after commercialization. As the radiology AI ecosystem continues to grow and mature, a formalization of system assessment and evaluation is paramount to ensure patient safety, relevance and support to clinical workflows, and optimal allocation of limited AI development and validation resources before broader implementation into clinical practice. To fulfil these needs, we provide a glossary for AI software types, use cases and roles within the clinical workflow; list healthcare needs, key performance indicators and required information about software prior to assessment; and lay out examples of software performance metrics per software category. This conceptual framework is intended to streamline communication with the AI software industry and provide healthcare decision makers and radiologists with tools to assess the potential use of these software. The proposed software evaluation framework lays the foundation for a radiologist-led prospective validation network of radiology AI software. Learning Points: The rapid expansion of AI applications in radiology requires standardization of AI software specification, classification, and evaluation. The Canadian Association of Radiologists' AI Tech & Apps Working Group Proposes an AI Specification document format and supports the implementation of a clinical expert evaluation process for Radiology AI software.

Critical Appraisal of Artificial Intelligence-Enabled Imaging Tools Using the Levels of Evidence System

Clinical adoption of an artificial intelligence-enabled imaging tool requires critical appraisal of its life cycle from development to implementation by using a systematic, standardized, and objective approach that can verify both its technical and clinical efficacy. Toward this concerted effort, the ASFNR/ASNR Artificial Intelligence Workshop Technology Working Group is proposing a hierarchal evaluation system based on the quality, type, and amount of scientific evidence that the artificial intelligence-enabled tool can demonstrate for each component of its life cycle. The current proposal is modeled after the levels of evidence in medicine, with the uppermost level of the hierarchy showing the strongest evidence for potential impact on patient care and health care outcomes. The intended goal of establishing an evidence-based evaluation system is to encourage transparency, foster an understanding of the creation of artificial intelligence tools and the artificial intelligence decision-making process, and to report the relevant data on the efficacy of artificial intelligence tools that are developed. The proposed system is an essential step in working toward a more formalized, clinically validated, and regulated framework for the safe and effective deployment of artificial intelligence imaging applications that will be used in clinical practice.