To buy or not to buy-evaluating commercial AI solutions in radiology (the ECLAIR guidelines)

An International Non-Inferiority Study for the Benchmarking of AI for Routine Radiology Cases: Chest X-ray, Fluorography and Mammography

An international reader study was conducted to gauge an average diagnostic accuracy of radiologists interpreting chest X-ray images, including those from fluorography and mammography, and establish requirements for stand-alone radiological artificial intelligence (AI) models. The retrospective studies in the datasets were labelled as containing or not containing target pathological findings based on a consensus of two experienced radiologists, and the results of a laboratory test and follow-up examination, where applicable. A total of 204 radiologists from 11 countries with various experience performed an assessment of the dataset with a 5-point Likert scale via a web platform. Eight commercial radiological AI models analyzed the same dataset. The AI AUROC was 0.87 (95% CI:0.83-0.9) versus 0.96 (95% CI 0.94-0.97) for radiologists. The sensitivity and specificity of AI versus radiologists were 0.71 (95% CI 0.64-0.78) versus 0.91 (95% CI 0.86-0.95) and 0.93 (95% CI 0.89-0.96) versus 0.9 (95% CI 0.85-0.94) for AI. The overall diagnostic accuracy of radiologists was superior to AI for chest X-ray and mammography. However, the accuracy of AI was noninferior to the least experienced radiologists for mammography and fluorography, and to all radiologists for chest X-ray. Therefore, an AI-based first reading could be recommended to reduce the workload burden of radiologists for the most common radiological studies such as chest X-ray and mammography.

AI vs FRCR: What it means for the future

A recent work by Shelmerdine et al. was published in the Christmas edition of the BMJ. The authors were inspired by George Hinton's statement that artificial intelligence (AI) would supersede radiologists, and ventured to investigate whether the AI software Milvue Suite which had been trained on a few hundred thousand chest and musculoskeletal x-rays, could pass the rapid reporting section of the FRCR - an exam which must be passed in order to practice as a consultant radiologist in the UK. This brief comment sums up the company's opinions and perspective from the practical AI developmental angle and also its translation into a commercially viable and clinically useful tool. Hoping this will provide a fair and balanced view of the role of AI in radiology.

Assessment of Radiology Artificial Intelligence Software: A Validation and Evaluation Framework

Artificial intelligence (AI) software in radiology is becoming increasingly prevalent and performance is improving rapidly with new applications for given use cases being developed continuously, oftentimes with development and validation occurring in parallel. Several guidelines have provided reporting standards for publications of AI-based research in medicine and radiology. Yet, there is an unmet need for recommendations on the assessment of AI software before adoption and after commercialization. As the radiology AI ecosystem continues to grow and mature, a formalization of system assessment and evaluation is paramount to ensure patient safety, relevance and support to clinical workflows, and optimal allocation of limited AI development and validation resources before broader implementation into clinical practice. To fulfil these needs, we provide a glossary for AI software types, use cases and roles within the clinical workflow; list healthcare needs, key performance indicators and required information about software prior to assessment; and lay out examples of software performance metrics per software category. This conceptual framework is intended to streamline communication with the AI software industry and provide healthcare decision makers and radiologists with tools to assess the potential use of these software. The proposed software evaluation framework lays the foundation for a radiologist-led prospective validation network of radiology AI software. Learning Points: The rapid expansion of AI applications in radiology requires standardization of AI software specification, classification, and evaluation. The Canadian Association of Radiologists' AI Tech & Apps Working Group Proposes an AI Specification document format and supports the implementation of a clinical expert evaluation process for Radiology AI software.

Critical Appraisal of Artificial Intelligence-Enabled Imaging Tools Using the Levels of Evidence System

Clinical adoption of an artificial intelligence-enabled imaging tool requires critical appraisal of its life cycle from development to implementation by using a systematic, standardized, and objective approach that can verify both its technical and clinical efficacy. Toward this concerted effort, the ASFNR/ASNR Artificial Intelligence Workshop Technology Working Group is proposing a hierarchal evaluation system based on the quality, type, and amount of scientific evidence that the artificial intelligence-enabled tool can demonstrate for each component of its life cycle. The current proposal is modeled after the levels of evidence in medicine, with the uppermost level of the hierarchy showing the strongest evidence for potential impact on patient care and health care outcomes. The intended goal of establishing an evidence-based evaluation system is to encourage transparency, foster an understanding of the creation of artificial intelligence tools and the artificial intelligence decision-making process, and to report the relevant data on the efficacy of artificial intelligence tools that are developed. The proposed system is an essential step in working toward a more formalized, clinically validated, and regulated framework for the safe and effective deployment of artificial intelligence imaging applications that will be used in clinical practice.

European Society of Paediatric Radiology Artificial Intelligence taskforce: a new taskforce for the digital age

A new task force dedicated to artificial intelligence (AI) with respect to paediatric radiology was created in 2021 at the International Paediatric Radiology (IPR) meeting in Rome, Italy (a joint society meeting by the European Society of Pediatric Radiology [ESPR] and the Society for Pediatric Radiology [SPR]). The concept of a separate task force dedicated to AI was borne from an ESPR-led international survey of health care professionals' opinions, expectations and concerns regarding AI integration within children's imaging departments. In this survey, the majority (> 80%) of ESPR respondents supported the creation of a task force and helped define our key objectives. These include providing educational content about AI relevant for paediatric radiologists, brainstorming ideas for future projects and collaborating on AI-related studies with respect to collating data sets, de-identifying images and engaging in multi-case, multi-reader studies. This manuscript outlines the starting point of the ESPR AI task force and where we wish to go.

Implementation of artificial intelligence in thoracic imaging-a what, how, and why guide from the European Society of Thoracic Imaging (ESTI)

This statement from the European Society of Thoracic imaging (ESTI) explains and summarises the essentials for understanding and implementing Artificial intelligence (AI) in clinical practice in thoracic radiology departments. This document discusses the current AI scientific evidence in thoracic imaging, its potential clinical utility, implementation and costs, training requirements and validation, its' effect on the training of new radiologists, post-implementation issues, and medico-legal and ethical issues. All these issues have to be addressed and overcome, for AI to become implemented clinically in thoracic radiology. KEY POINTS: • Assessing the datasets used for training and validation of the AI system is essential. • A departmental strategy and business plan which includes continuing quality assurance of AI system and a sustainable financial plan is important for successful implementation. • Awareness of the negative effect on training of new radiologists is vital.

Artificial Intelligence in Radiology: A Private Practice Perspective From a Large Health System in Latin America

In the field of radiology, the use of artificial intelligence (AI) is increasing. Even though healthcare facilities are interested in using this technology, having success with an AI project can be challenging. There is a myriad of AI solutions today, and comparing them can be challenging. Moreover, the implementation process involves alignment with many different areas. In our institution, we have been testing, developing, deploying, and monitoring AI solutions for the last four years. This article intends to share our experience and highlight the most important points to ensure a successful project based on our experience in the setting of a large private practice in Latin America.

Artificial Intelligence and Radiology Education

Implementation of artificial intelligence (AI) applications into clinical practice requires AI-savvy radiologists to ensure the safe, ethical, and effective use of these systems for patient care. Increasing demand for AI education reflects recognition of the translation of AI applications from research to clinical practice, with positive trainee attitudes regarding the influence of AI on radiology. However, barriers to AI education, such as limited access to resources, predispose to insufficient preparation for the effective use of AI in practice. In response, national organizations have sponsored formal and self-directed learning courses to provide introductory content on imaging informatics and AI. Foundational courses, such as the National Imaging Informatics Course - Radiology and the Radiological Society of North America Imaging AI Certificate, lay a framework for trainees to explore the creation, deployment, and critical evaluation of AI applications. This report includes additional resources for formal programming courses, video series from leading organizations, and blogs from AI and informatics communities. Furthermore, the scope of "AI and radiology education" includes AI-augmented radiology education, with emphasis on the potential for "precision education" that creates personalized experiences for trainees by accounting for varying learning styles and inconsistent, possibly deficient, clinical case volume. © RSNA, 2022 Keywords: Use of AI in Education, Impact of AI on Education, Artificial Intelligence, Medical Education, Imaging Informatics, Natural Language Processing, Precision Education.

Synthetic Contrasts in Musculoskeletal MRI: A Review

ABSTRACT

This review summarizes the existing techniques and methods used to generate synthetic contrasts from magnetic resonance imaging data focusing on musculoskeletal magnetic resonance imaging. To that end, the different approaches were categorized into 3 different methodological groups: mathematical image transformation, physics-based, and data-driven approaches. Each group is characterized, followed by examples and a brief overview of their clinical validation, if present. Finally, we will discuss the advantages, disadvantages, and caveats of synthetic contrasts, focusing on the preservation of image information, validation, and aspects of the clinical workflow.

Methods for Clinical Evaluation of Artificial Intelligence Algorithms for Medical Diagnosis

Adequate clinical evaluation of artificial intelligence (AI) algorithms before adoption in practice is critical. Clinical evaluation aims to confirm acceptable AI performance through adequate external testing and confirm the benefits of AI-assisted care compared with conventional care through appropriately designed and conducted studies, for which prospective studies are desirable. This article explains some of the fundamental methodological points that should be considered when designing and appraising the clinical evaluation of AI algorithms for medical diagnosis. The specific topics addressed include the following: (a) the importance of external testing of AI algorithms and strategies for conducting the external testing effectively, (b) the various metrics and graphical methods for evaluating the AI performance as well as essential methodological points to note in using and interpreting them, (c) paired study designs primarily for comparative performance evaluation of conventional and AI-assisted diagnoses, (d) parallel study designs primarily for evaluating the effect of AI intervention with an emphasis on randomized clinical trials, and (e) up-to-date guidelines for reporting clinical studies on AI, with an emphasis on guidelines registered in the EQUATOR Network library. Sound methodological knowledge of these topics will aid the design, execution, reporting, and appraisal of clinical evaluation of AI.

[Validation and implementation of artificial intelligence in radiology : Quo vadis in 2022?]

BACKGROUND

The hype around artificial intelligence (AI) in radiology continues and the number of approved AI tools is growing steadily. Despite the great potential, integration into clinical routine in radiology remains limited. In addition, the large number of individual applications poses a challenge for clinical routine, as individual applications have to be selected for different questions and organ systems, which increases the complexity and time required.

OBJECTIVES

This review will discuss the current status of validation and implementation of AI tools in clinical routine, and identify possible approaches for an improved assessment of the generalizability of results of AI tools.

MATERIALS AND METHODS

A literature search in various literature and product databases as well as publications, position papers, and reports from various stakeholders was conducted for this review.

RESULTS

Scientific evidence and independent validation studies are available for only a few commercial AI tools and the generalizability of the results often remains questionable.

CONCLUSIONS

One challenge is the multitude of offerings for individual, specific application areas by a large number of manufacturers, making integration into the existing site-specific IT infrastructure more difficult. Furthermore, remuneration for the use of AI tools in clinical routine by health insurance companies in Germany is lacking. But in order for reimbursement to be granted, the clinical utility of new applications must first be proven. Such proof, however, is lacking for most applications.

Evaluating artificial intelligence algorithms for use in veterinary radiology

Artificial intelligence is increasingly being used for applications in veterinary radiology, including detection of abnormalities and automated measurements. Unlike human radiology, there is no formal regulation or validation of AI algorithms for veterinary medicine and both general practitioner and specialist veterinarians must rely on their own judgment when deciding whether or not to incorporate AI algorithms to aid their clinical decision-making. The benefits and challenges to developing clinically useful and diagnostically accurate AI algorithms are discussed. Considerations for the development of AI research projects are also addressed. A framework is suggested to help veterinarians, in both research and clinical practice contexts, assess AI algorithms for veterinary radiology.

Impact of Artificial Intelligence Assistance on Chest CT Interpretation Times: A Prospective Randomized Study

BACKGROUND. Deep learning-based convolutional neural networks have enabled major advances in development of artificial intelligence (AI) software applications. Modern AI applications offer comprehensive multiorgan evaluation. OBJECTIVE. The purpose of this article was to evaluate the impact of an automated AI platform integrated into clinical workflow for chest CT interpretation on radiologists' interpretation times when evaluated in a real-world clinical setting. METHODS. In this prospective single-center study, a commercial AI software solution was integrated into clinical workflow for chest CT interpretation. The software provided automated analysis of cardiac, pulmonary, and musculoskeletal findings, including labeling, segmenting, and measuring normal structures as well as detecting, labeling, and measuring abnormalities. AI-annotated images and autogenerated summary results were stored in the PACS and available to interpreting radiologists. A total of 390 patients (204 women, 186 men; mean age, 62.8 ± 13.3 [SD] years) who underwent out-patient chest CT between January 19, 2021, and January 28, 2021, were included. Scans were randomized using 1:1 allocation between AI-assisted and non-AI-assisted arms and were clinically interpreted by one of three cardiothoracic radiologists (65 scans per arm per radiologist; total of 195 scans per arm) who recorded interpretation times using a stopwatch. Findings were categorized according to review of report impressions. Interpretation times were compared between arms. RESULTS. Mean interpretation times were significantly shorter in the AI-assisted than in the non-AI-assisted arm for all three readers (289 ± 89 vs 344 ± 129 seconds, p < .001; 449 ± 110 vs 649 ± 82 seconds, p < .001; 281 ± 114 vs 348 ± 93 seconds, p = .01) and for readers combined (328 ± 122 vs 421 ± 175 seconds, p < .001). For readers combined, the mean difference was 93 seconds (95% CI, 63-123 seconds), corresponding with a 22.1% reduction in the AI-assisted arm. Mean interpretation time was also shorter in the AI-assisted arm compared with the non-AI-assisted arm for contrast-enhanced scans (83 seconds), noncontrast scans (104 seconds), negative scans (84 seconds), positive scans without significant new findings (117 seconds), and positive scans with significant new findings (92 seconds). CONCLUSION. Cardiothoracic radiologists exhibited a 22.1% reduction in chest CT interpretations times when they had access to results from an automated AI support platform during real-world clinical practice. CLINICAL IMPACT. Integration of the AI support platform into clinical workflow improved radiologist efficiency.

Artificial Intelligence in Clinical Practice: Implementation Considerations and Barriers

The rapid growth of artificial intelligence (AI) in radiology has led to Food and Drug Administration clearance of more than 20 AI algorithms for breast imaging. The steps involved in the clinical implementation of an AI product include identifying all stakeholders, selecting the appropriate product to purchase, evaluating it with a local data set, integrating it into the workflow, and monitoring its performance over time. Despite the potential benefits of improved quality and increased efficiency with AI, several barriers, such as high costs and liability concerns, may limit its widespread implementation. This article lists currently available AI products for breast imaging, describes the key elements of clinical implementation, and discusses barriers to clinical implementation.

Artificial Intelligence (AI) for Lung Nodules, From the AJR Special Series on AI Applications

Interest in artificial intelligence (AI) applications for lung nodules continues to grow among radiologists, particularly with the expanding eligibility criteria and clinical utilization of lung cancer screening CT. AI has been heavily investigated for detecting and characterizing lung nodules and for guiding prognostic assessment. AI tools have also been used for image postprocessing (e.g., rib suppression on radiography or vessel suppression on CT) and for noninterpretive aspects of reporting and workflow, including management of nodule follow-up. Despite growing interest in and rapid development of AI tools and FDA approval of AI tools for pulmonary nodule evaluation, integration into clinical practice has been limited. Challenges to clinical adoption have included concerns about generalizability, regulatory issues, technical hurdles in implementation, and human skepticism. Further validation of AI tools for clinical use and demonstration of benefit in terms of patient-oriented outcomes also are needed. This article provides an overview of potential applications of AI tools in the imaging evaluation of lung nodules and discusses the challenges faced by practices interested in clinical implementation of such tools.

Value assessment of artificial intelligence in medical imaging: a scoping review

BACKGROUND

Artificial intelligence (AI) is seen as one of the major disrupting forces in the future healthcare system. However, the assessment of the value of these new technologies is still unclear, and no agreed international health technology assessment-based guideline exists. This study provides an overview of the available literature in the value assessment of AI in the field of medical imaging.

METHODS

We performed a systematic scoping review of published studies between January 2016 and September 2020 using 10 databases (Medline, Scopus, ProQuest, Google Scholar, and six related databases of grey literature). Information about the context (country, clinical area, and type of study) and mentioned domains with specific outcomes and items were extracted. An existing domain classification, from a European assessment framework, was used as a point of departure, and extracted data were grouped into domains and content analysis of data was performed covering predetermined themes.

RESULTS

Seventy-nine studies were included out of 5890 identified articles. An additional seven studies were identified by searching reference lists, and the analysis was performed on 86 included studies. Eleven domains were identified: (1) health problem and current use of technology, (2) technology aspects, (3) safety assessment, (4) clinical effectiveness, (5) economics, (6) ethical analysis, (7) organisational aspects, (8) patients and social aspects, (9) legal aspects, (10) development of AI algorithm, performance metrics and validation, and (11) other aspects. The frequency of mentioning a domain varied from 20 to 78% within the included papers. Only 15/86 studies were actual assessments of AI technologies. The majority of data were statements from reviews or papers voicing future needs or challenges of AI research, i.e. not actual outcomes of evaluations.

CONCLUSIONS

This review regarding value assessment of AI in medical imaging yielded 86 studies including 11 identified domains. The domain classification based on European assessment framework proved useful and current analysis added one new domain. Included studies had a broad range of essential domains about addressing AI technologies highlighting the importance of domains related to legal and ethical aspects.

Nuclear Medicine and Artificial Intelligence: Best Practices for Evaluation (the RELAINCE Guidelines)

An important need exists for strategies to perform rigorous objective clinical-task-based evaluation of artificial intelligence (AI) algorithms for nuclear medicine. To address this need, we propose a 4-class framework to evaluate AI algorithms for promise, technical task-specific efficacy, clinical decision making, and postdeployment efficacy. We provide best practices to evaluate AI algorithms for each of these classes. Each class of evaluation yields a claim that provides a descriptive performance of the AI algorithm. Key best practices are tabulated as the RELAINCE (Recommendations for EvaLuation of AI for NuClear medicinE) guidelines. The report was prepared by the Society of Nuclear Medicine and Molecular Imaging AI Task Force Evaluation team, which consisted of nuclear-medicine physicians, physicists, computational imaging scientists, and representatives from industry and regulatory agencies.

Pre-deployment assessment of an AI model to assist radiologists in chest X-ray detection and identification of lead-less implanted electronic devices for pre-MRI safety screening: realized implementation needs and proposed operational solutions

Purpose

Chest X-ray (CXR) use in pre-MRI safety screening, such as for lead-less implanted electronic device (LLIED) recognition, is common. To assist CXR interpretation, we "pre-deployed" an artificial intelligence (AI) model to assess (1) accuracies in LLIED-type (and consequently safety-level) identification, (2) safety implications of LLIED nondetections or misidentifications, (3) infrastructural or workflow requirements, and (4) demands related to model adaptation to real-world conditions.

Approach

A two-tier cascading methodology for LLIED detection/localization and identification on a frontal CXR was applied to evaluate the performance of the original nine-class AI model. With the unexpected early appearance of LLIED types during simulated real-world trialing, retraining of a newer 12-class version preceded retrialing. A zero footprint (ZF) graphical user interface (GUI)/viewer with DICOM-based output was developed for inference-result display and adjudication, supporting end-user engagement and model continuous learning and/or modernization.

Results

During model testing or trialing using both the nine-class and 12-class models, robust detection/localization was consistently 100%, with mAP 0.99 from fivefold cross-validation. Safety-level categorization was high during both testing ( AUC ≥ 0.98 and ≥ 0.99 , respectively) and trialing (accuracy 98% and 97%, respectively). LLIED-type identifications by the two models during testing (1) were 98.9% and 99.5% overall correct and (2) consistently showed AUC ≥ 0.92 (1.00 for 8/9 and 9/12 LLIED-types, respectively). Pre-deployment trialing of both models demonstrated overall type-identification accuracies of 94.5% and 95%, respectively. Of the small number of misidentifications, none involved MRI-stringently conditional or MRI-unsafe types of LLIEDs. Optimized ZF GUI/viewer operations led to greater user-friendliness for radiologist engagement.

Conclusions

Our LLIED-related AI methodology supports (1) 100% detection sensitivity, (2) high identification (including MRI-safety) accuracy, and (3) future model deployment with facilitated inference-result display and adjudication for ongoing model adaptation to future real-world experiences.

The Cases for and against Artificial Intelligence in the Medical School Curriculum

Although artificial intelligence (AI) has immense potential to shape the future of medicine, its place in undergraduate medical education currently is unclear. Numerous arguments exist both for and against including AI in the medical school curriculum. AI likely will affect all medical specialties, perhaps radiology more so than any other. The purpose of this article is to present a balanced perspective on whether AI should be included officially in the medical school curriculum. After presenting the balanced point-counterpoint arguments, the authors provide a compromise. Keywords: Artificial Intelligence, Medical Education, Medical School Curriculum, Medical Students, Radiology, Use of AI in Education © RSNA, 2022.

Longitudinal detection of new MS lesions using deep learning

The detection of new multiple sclerosis (MS) lesions is an important marker of the evolution of the disease. The applicability of learning-based methods could automate this task efficiently. However, the lack of annotated longitudinal data with new-appearing lesions is a limiting factor for the training of robust and generalizing models. In this study, we describe a deep-learning-based pipeline addressing the challenging task of detecting and segmenting new MS lesions. First, we propose to use transfer-learning from a model trained on a segmentation task using single time-points. Therefore, we exploit knowledge from an easier task and for which more annotated datasets are available. Second, we propose a data synthesis strategy to generate realistic longitudinal time-points with new lesions using single time-point scans. In this way, we pretrain our detection model on large synthetic annotated datasets. Finally, we use a data-augmentation technique designed to simulate data diversity in MRI. By doing that, we increase the size of the available small annotated longitudinal datasets. Our ablation study showed that each contribution lead to an enhancement of the segmentation accuracy. Using the proposed pipeline, we obtained the best score for the segmentation and the detection of new MS lesions in the MSSEG2 MICCAI challenge.

Establishing a Hospital Artificial Intelligence Committee to Improve Patient Care

BACKGROUND

The use of artificial intelligence (AI) in health care is increasing and has shown utility in many medical specialties, especially pathology, radiology, and oncology.

OBSERVATIONS

Many barriers exist to successfully implement AI programs in the clinical setting. To address these barriers, a formal governing body, the hospital AI Committee, was created at James A. Haley Veterans' Hospital in Tampa, Florida. The AI committee reviews and assesses AI products based on their success at protecting human autonomy; promoting human well-being and safety and the public interest; ensuring transparency, explainability, and intelligibility; fostering responsibility and accountability; ensuring inclusiveness and equity; and promoting AI that is responsive and sustainable.

CONCLUSIONS

Through the hospital AI Committee, we may overcome many obstacles to successfully implementing AI applications in the clinical setting.

Proceedings from the Society of Interventional Radiology Foundation Research Consensus Panel on Artificial Intelligence in Interventional Radiology: From Code to Bedside

Artificial intelligence (AI)-based technologies are the most rapidly growing field of innovation in healthcare with the promise to achieve substantial improvements in delivery of patient care across all disciplines of medicine. Recent advances in imaging technology along with marked expansion of readily available advanced health information, data offer a unique opportunity for interventional radiology (IR) to reinvent itself as a data-driven specialty. Additionally, the growth of AI-based applications in diagnostic imaging is expected to have downstream effects on all image-guidance modalities. Therefore, the Society of Interventional Radiology Foundation has called upon 13 key opinion leaders in the field of IR to develop research priorities for clinical applications of AI in IR. The objectives of the assembled research consensus panel were to assess the availability and understand the applicability of AI for IR, estimate current needs and clinical use cases, and assemble a list of research priorities for the development of AI in IR. Individual panel members proposed and all participants voted upon consensus statements to rank them according to their overall impact for IR. The results identified the top priorities for the IR research community and provide organizing principles for innovative academic-industrial research collaborations that will leverage both clinical expertise and cutting-edge technology to benefit patient care in IR.

Changes in software as a medical device based on artificial intelligence technologies

Purpose

to develop a procedure for registering changes, notifying users about changes made, unifying software as a medical device based on artificial intelligence technologies (SaMD-AI) changes, as well as requirements for testing and inspections—quality control before and after making changes.

Methods

The main types of changes, divided into two groups—major and minor. Major changes imply a subsequent change of a SaMD-AI version to improve efficiency and safety, to change the functionality, and to ensure the processing of new data types. Minor changes imply those that SaMD-AI developers can make due to errors in the program code. Three types of SaMD-AI testings are proposed to use: functional testing, calibration testing or control, and technical testing.

Results

The presented approaches for validation SaMD-AI changes were introduced. The unified requirements for the request for changes and forms of their submission made this procedure understandable for SaMD-AI developers, and also adjusted the workload for the Experiment experts who checked all the changes made to SaMD-AI.

Conclusion

This article discusses the need to control changes in the module of SaMD-AI, as innovative products influencing medical decision making. It justifies the need to control a module operation of SaMD-AI after making changes. To streamline and optimize the necessary and sufficient control procedures, a systematization of possible changes in SaMD-AI and testing methods was carried out.

Use of artificial intelligence in emergency radiology: An overview of current applications, challenges, and opportunities

The value of artificial intelligence (AI) in healthcare has become evident, especially in the field of medical imaging. The accelerated pace and acuity of care in the Emergency Department (ED) has made it a popular target for artificial intelligence-driven solutions. Software that helps better detect, report, and appropriately guide management can ensure high quality patient care while enabling emergency radiologists to better meet the demands of quick turnaround times. Beyond diagnostic applications, AI-based algorithms also have the potential to optimize other important steps within the ED imaging workflow. This review will highlight the different types of AI-based applications currently available for use in the ED, as well as the challenges and opportunities associated with their implementation.

Meningioma Radiomics: At the Nexus of Imaging, Pathology and Biomolecular Characterization

Simple Summary

Meningiomas are typically benign, common extra-axial tumors of the central nervous system. Routine clinical assessment by radiologists presents some limitations regarding long-term patient outcome prediction and risk stratification. Given the exponential growth of interest in radiomics and artificial intelligence in medical imaging, numerous studies have evaluated the potential of these tools in the setting of meningioma imaging. These were aimed at the development of reliable and reproducible models based on quantitative data. Although several limitations have yet to be overcome for their routine use in clinical practice, their innovative potential is evident. In this review, we present a wide-ranging overview of radiomics and artificial intelligence applications in meningioma imaging.

Abstract

Meningiomas are the most common extra-axial tumors of the central nervous system (CNS). Even though recurrence is uncommon after surgery and most meningiomas are benign, an aggressive behavior may still be exhibited in some cases. Although the diagnosis can be made by radiologists, typically with magnetic resonance imaging, qualitative analysis has some limitations in regard to outcome prediction and risk stratification. The acquisition of this information could help the referring clinician in the decision-making process and selection of the appropriate treatment. Following the increased attention and potential of radiomics and artificial intelligence in the healthcare domain, including oncological imaging, researchers have investigated their use over the years to overcome the current limitations of imaging. The aim of these new tools is the replacement of subjective and, therefore, potentially variable medical image analysis by more objective quantitative data, using computational algorithms. Although radiomics has not yet fully entered clinical practice, its potential for the detection, diagnostic, and prognostic characterization of tumors is evident. In this review, we present a wide-ranging overview of radiomics and artificial intelligence applications in meningioma imaging.

A "Bumper-Car" Curriculum for Teaching Deep Learning to Radiology Residents☆

RATIONALE AND OBJECTIVES

Our goal was to create an artificial intelligence (AI) training curriculum for residents that taught them to create, train, evaluate and refine deep learning (DL) models. Hands-on training of models was emphasized and didactic presentations of the mathematical and programmatic underpinnings of DL were minimized.

MATERIALS AND METHODS

We created a three-session, 6-hour curriculum based on a "no-code" machine learning system called Lobe.ai. This class met weekly in June 2021. Pre-class homework included reading assignments, software installation, dataset downloads, and image-collection and labeling. The class sessions included several short, didactic presentations, but were largely devoted to hands-on training of DL models. After the course, our residents completed a short, anonymous, online survey about the course.

RESULTS

Our residents learned to acquire and label a wide variety of image datasets. They quickly learned to train DL models to classify these datasets, as well as how to evaluate and refine these models. Our survey showed that most residents felt AI to be important and worth learning, but most were not very interested in learning to program. Most felt that the course taught them useful things about DL, and they were now more interested in the topic. Most would recommend the course to other residents, as well as to medical students and to radiology faculty.

CONCLUSION

The course met our objectives of teaching our residents to create, train, evaluate, and refine DL models. We hope that the hands-on experience they gained in this course will enable them to recognize problems in diagnostic AI systems, and to help solve such problems in their own radiology practices.

Commercial AI solutions in detecting COVID-19 pneumonia in chest CT: not yet ready for clinical implementation?

OBJECTIVES

In response to the COVID-19 pandemic, many researchers have developed artificial intelligence (AI) tools to differentiate COVID-19 pneumonia from other conditions in chest CT. However, in many cases, performance has not been clinically validated. The aim of this study was to evaluate the performance of commercial AI solutions in differentiating COVID-19 pneumonia from other lung conditions.

METHODS

Four commercial AI solutions were evaluated on a dual-center clinical dataset consisting of 500 CT studies; COVID-19 pneumonia was microbiologically proven in 50 of these. Sensitivity, specificity, positive and negative predictive values, and AUC were calculated. In a subgroup analysis, the performance of the AI solutions in differentiating COVID-19 pneumonia from other conditions was evaluated in CT studies with ground-glass opacities (GGOs).

RESULTS

Sensitivity and specificity ranges were 62-96% and 31-80%, respectively. Negative and positive predictive values ranged between 82-99% and 19-25%, respectively. AUC was in the range 0.54-0.79. In CT studies with GGO, sensitivity remained unchanged. However, specificity was lower, and ranged between 15 and 53%. AUC for studies with GGO was in the range 0.54-0.69.

CONCLUSIONS

This study highlights the variable specificity and low positive predictive value of AI solutions in diagnosing COVID-19 pneumonia in chest CT. However, one solution yielded acceptable values for sensitivity. Thus, with further improvement, commercial AI solutions currently under development have the potential to be integrated as alert tools in clinical routine workflow. Randomized trials are needed to assess the true benefits and also potential harms of the use of AI in image analysis.

KEY POINTS

• Commercial AI solutions achieved a sensitivity and specificity ranging from 62 to 96% and from 31 to 80%, respectively, in identifying patients suspicious for COVID-19 in a clinical dataset. • Sensitivity remained within the same range, while specificity was even lower in subgroup analysis of CT studies with ground-glass opacities, and interrater agreement between the commercial AI solutions was minimal to nonexistent. • Thus, commercial AI solutions have the potential to be integrated as alert tools for the detection of patients with lung changes suspicious for COVID-19 pneumonia in a clinical routine workflow, if further improvement is made.

The impact of radiomics for human papillomavirus status prediction in oropharyngeal cancer: systematic review and radiomics quality score assessment

Purpose

Human papillomavirus (HPV) status assessment is crucial for decision making in oropharyngeal cancer patients. In last years, several articles have been published investigating the possible role of radiomics in distinguishing HPV-positive from HPV-negative neoplasms. Aim of this review was to perform a systematic quality assessment of radiomic studies published on this topic.

Methods

Radiomics studies on HPV status prediction in oropharyngeal cancer patients were selected. The Radiomic Quality Score (RQS) was assessed by three readers to evaluate their methodological quality. In addition, possible correlations between RQS% and journal type, year of publication, impact factor, and journal rank were investigated.

Results

After the literature search, 19 articles were selected whose RQS median was 33% (range 0–42%). Overall, 16/19 studies included a well-documented imaging protocol, 13/19 demonstrated phenotypic differences, and all were compared with the current gold standard. No study included a public protocol, phantom study, or imaging at multiple time points. More than half (13/19) included feature selection and only 2 were comprehensive of non-radiomic features. Mean RQS was significantly higher in clinical journals.

Conclusion

Radiomics has been proposed for oropharyngeal cancer HPV status assessment, with promising results. However, these are supported by low methodological quality investigations. Further studies with higher methodological quality, appropriate standardization, and greater attention to validation are necessary prior to clinical adoption.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00234-022-02959-0.

Updates in Artificial Intelligence for Breast Imaging

Artificial intelligence (AI) for breast imaging has rapidly moved from the experimental to implementation phase. As of this writing, Food and Drug Administration (FDA)-approved mammographic applications are available for triage, lesion detection and classification, and breast density assessment. For sonography and MRI, FDA-approved applications are available for lesion classification. Numerous other interpretive and noninterpretive AI applications are in the development phase. This article reviews AI applications for mammography, sonography, and MRI that are currently available for clinical use. In addition, clinical implementation and the future of AI for breast imaging are discussed.

Developing, implementing and governing artificial intelligence in medicine: a step-by-step approach to prevent an artificial intelligence winter

Objective

Although the role of artificial intelligence (AI) in medicine is increasingly studied, most patients do not benefit because the majority of AI models remain in the testing and prototyping environment. The development and implementation trajectory of clinical AI models are complex and a structured overview is missing. We therefore propose a step-by-step overview to enhance clinicians’ understanding and to promote quality of medical AI research.

Methods

We summarised key elements (such as current guidelines, challenges, regulatory documents and good practices) that are needed to develop and safely implement AI in medicine.

Conclusion

This overview complements other frameworks in a way that it is accessible to stakeholders without prior AI knowledge and as such provides a step-by-step approach incorporating all the key elements and current guidelines that are essential for implementation, and can thereby help to move AI from bytes to bedside.

Artificial intelligence in liver diseases: improving diagnostics, prognostics and response prediction

Clinical routine in hepatology involves the diagnosis and treatment of a wide spectrum of metabolic, infectious, autoimmune and neoplastic diseases. Clinicians integrate qualitative and quantitative information from multiple data sources to make a diagnosis, prognosticate the disease course, and recommend a treatment. In the last 5 years, advances in artificial intelligence (AI), particularly in deep learning, have made it possible to extract clinically relevant information from complex and diverse clinical datasets. In particular, histopathology and radiology image data contain diagnostic, prognostic and predictive information which AI can extract. Ultimately, such AI systems could be implemented in clinical routine as decision support tools. However, in the context of hepatology, this requires further large-scale clinical validation and regulatory approval. Herein, we summarise the state of the art in AI in hepatology with a particular focus on histopathology and radiology data. We present a roadmap for the further development of novel biomarkers in hepatology and outline critical obstacles which need to be overcome.

Will the EU Medical Device Regulation help to improve the safety and performance of medical AI devices?

Concerns have been raised over the quality of evidence on the performance of medical artificial intelligence devices, including devices that are already on the market in the USA and Europe. Recently, the Medical Device Regulation, which aims to set high standards of safety and quality, has become applicable in the European Union. The aim of this article is to discuss whether, and how, the Medical Device Regulation will help improve the safety and performance of medical artificial intelligence devices entering the market. The Medical Device Regulation introduces new rules for risk classification of the devices, which will result in more devices subjected to a higher degree of scrutiny before entering the market; more stringent requirements on clinical evaluation, including the requirement for appraisal of clinical data; new requirements for post-market surveillance, which may help spot early on any new, unexpected side effects and risks of the devices; and requirements for notified bodies, including for expertise of the personnel and consideration of relevant best practice documents. The guidance of the Medical Device Coordination Group on clinical evaluation of medical device software and the MEDDEV2.7 guideline on clinical evaluation also attend to some of the problems identified in studies on medical artificial intelligence devices. The Medical Device Regulation will likely help improve the safety and performance of the medical artificial intelligence devices on the European market. The impact of the Regulation, however, is also dependent on its adequate enforcement by the European Union member states.

Checklist for Evaluation of Image-Based Artificial Intelligence Reports in Dermatology: CLEAR Derm Consensus Guidelines From the International Skin Imaging Collaboration Artificial Intelligence Working Group

IMPORTANCE

The use of artificial intelligence (AI) is accelerating in all aspects of medicine and has the potential to transform clinical care and dermatology workflows. However, to develop image-based algorithms for dermatology applications, comprehensive criteria establishing development and performance evaluation standards are required to ensure product fairness, reliability, and safety.

OBJECTIVE

To consolidate limited existing literature with expert opinion to guide developers and reviewers of dermatology AI.

EVIDENCE REVIEW

In this consensus statement, the 19 members of the International Skin Imaging Collaboration AI working group volunteered to provide a consensus statement. A systematic PubMed search was performed of English-language articles published between December 1, 2008, and August 24, 2021, for "artificial intelligence" and "reporting guidelines," as well as other pertinent studies identified by the expert panel. Factors that were viewed as critical to AI development and performance evaluation were included and underwent 2 rounds of electronic discussion to achieve consensus.

FINDINGS

A checklist of items was developed that outlines best practices of image-based AI development and assessment in dermatology.

CONCLUSIONS AND RELEVANCE

Clinically effective AI needs to be fair, reliable, and safe; this checklist of best practices will help both developers and reviewers achieve this goal.

Model for ASsessing the value of Artificial Intelligence in medical imaging (MAS-AI)

Objectives

Artificial intelligence (AI) is seen as a major disrupting force in the future healthcare system. However, the assessment of the value of AI technologies is still unclear. Therefore, a multidisciplinary group of experts and patients developed a Model for ASsessing the value of AI (MAS-AI) in medical imaging. Medical imaging is chosen due to the maturity of AI in this area, ensuring a robust evidence-based model.

Methods

MAS-AI was developed in three phases. First, a literature review of existing guides, evaluations, and assessments of the value of AI in the field of medical imaging. Next, we interviewed leading researchers in AI in Denmark. The third phase consisted of two workshops where decision makers, patient organizations, and researchers discussed crucial topics for evaluating AI. The multidisciplinary team revised the model between workshops according to comments.

Results

The MAS-AI guideline consists of two steps covering nine domains and five process factors supporting the assessment. Step 1 contains a description of patients, how the AI model was developed, and initial ethical and legal considerations. In step 2, a multidisciplinary assessment of outcomes of the AI application is done for the five remaining domains: safety, clinical aspects, economics, organizational aspects, and patient aspects.

Conclusions

We have developed an health technology assessment-based framework to support the introduction of AI technologies into healthcare in medical imaging. It is essential to ensure informed and valid decisions regarding the adoption of AI with a structured process and tool. MAS-AI can help support decision making and provide greater transparency for all parties.

DeepLesionBrain: Towards a broader deep-learning generalization for multiple sclerosis lesion segmentation

Recently, segmentation methods based on Convolutional Neural Networks (CNNs) showed promising performance in automatic Multiple Sclerosis (MS) lesions segmentation. These techniques have even outperformed human experts in controlled evaluation conditions such as Longitudinal MS Lesion Segmentation Challenge (ISBI Challenge). However, state-of-the-art approaches trained to perform well on highly-controlled datasets fail to generalize on clinical data from unseen datasets. Instead of proposing another improvement of the segmentation accuracy, we propose a novel method robust to domain shift and performing well on unseen datasets, called DeepLesionBrain (DLB). This generalization property results from three main contributions. First, DLB is based on a large group of compact 3D CNNs. This spatially distributed strategy aims to produce a robust prediction despite the risk of generalization failure of some individual networks. Second, we propose a hierarchical specialization learning (HSL) by pre-training a generic network over the whole brain, before using its weights as initialization to locally specialized networks. By this end, DLB learns both generic features extracted at global image level and specific features extracted at local image level. Finally, DLB includes a new image quality data augmentation to reduce dependency to training data specificity (e.g., acquisition protocol). DLB generalization was validated in cross-dataset experiments on MSSEG'16, ISBI challenge, and in-house datasets. During experiments, DLB showed higher segmentation accuracy, better segmentation consistency and greater generalization performance compared to state-of-the-art methods. Therefore, DLB offers a robust framework well-suited for clinical practice.

Evidence-Based Artificial Intelligence in Medical Imaging

Artificial intelligence (AI) in medical imaging is in its infancy. However, ongoing advances in hardware and software as well as increasing access to ever-expanding datasets for training, validation, and testing purposes are likely to make AI an increasingly prevalent and powerful tool. Of course issues, such as the need to protect the privacy of sensitive health data, remain; nevertheless, it is likely the average imager will need to develop an evidence-based approach to assessing AI in medical imaging. We hope this article will provide insight into just how this can be conducted by applying 5 simple questions, specifically: (1) Who was in the training sample, (2) How was the model trained, (3) How reliable is the algorithm, (4) How was the model validated, and (5) How useable is the algorithm.

External validation of a commercially available deep learning algorithm for fracture detection in children: Fracture detection with a deep learning algorithm

PURPOSE

The purpose of this study was to conduct an external validation of a fracture assessment deep learning algorithm (Rayvolve®) using digital radiographs from a real-life cohort of children presenting routinely to the emergency room.

MATERIALS AND METHODS

This retrospective study was conducted on 2634 radiography sets (5865 images) from 2549 children (1459 boys, 1090 girls; mean age, 8.5 ± 4.5 [SD] years; age range: 0-17 years) referred by the pediatric emergency room for trauma. For each set was recorded whether one or more fractures were found, the number of fractures, and their location found by the senior radiologists and the algorithm. Using the senior radiologist diagnosis as the standard of reference, the diagnostic performance of deep learning algorithm (Rayvolve®) was calculated via three different approaches: a detection approach (presence/absence of a fracture as a binary variable), an enumeration approach (exact number of fractures detected) and a localization approach (focusing on whether the detected fractures were correctly localized). Subgroup analyses were performed according to the presence of a cast or not, age category (0-4 vs. 5-18 years) and anatomical region.

RESULTS

Regarding detection approach, the deep learning algorithm yielded 95.7% sensitivity (95% CI: 94.0-96.9), 91.2% specificity (95% CI: 89.8-92.5) and 92.6% accuracy (95% CI: 91.5-93.6). Regarding enumeration and localization approaches, the deep learning algorithm yielded 94.1% sensitivity (95% CI: 92.1-95.6), 88.8% specificity (95% CI: 87.3-90.2) and 90.4% accuracy (95% CI: 89.2-91.5) for both approaches. Regarding age-related subgroup analyses, the deep learning algorithm yielded greater sensitivity and negative predictive value in the 5-18-years age group than in the 0-4-years age group for the detection approach (P < 0.001 and P = 0.002) and for the enumeration and localization approaches (P = 0.012 and P = 0.028). The high negative predictive value was robust, persisting in all of the subgroup analyses, except for patients with casts (P = 0.001 for the detection approach and P < 0.001 for the enumeration and localization approaches).

CONCLUSION

The Rayvolve® deep learning algorithm is very reliable for detecting fractures in children, especially in those older than 4 years and without cast.

Technical and clinical validation of commercial automated volumetric MRI tools for dementia diagnosis-a systematic review

Developments in neuroradiological MRI analysis offer promise in enhancing objectivity and consistency in dementia diagnosis through the use of quantitative volumetric reporting tools (QReports). Translation into clinical settings should follow a structured framework of development, including technical and clinical validation steps. However, published technical and clinical validation of the available commercial/proprietary tools is not always easy to find and pathways for successful integration into the clinical workflow are varied. The quantitative neuroradiology initiative (QNI) framework highlights six necessary steps for the development, validation and integration of quantitative tools in the clinic. In this paper, we reviewed the published evidence regarding regulatory-approved QReports for use in the memory clinic and to what extent this evidence fulfils the steps of the QNI framework. We summarize unbiased technical details of available products in order to increase the transparency of evidence and present the range of reporting tools on the market. Our intention is to assist neuroradiologists in making informed decisions regarding the adoption of these methods in the clinic. For the 17 products identified, 11 companies have published some form of technical validation on their methods, but only 4 have published clinical validation of their QReports in a dementia population. Upon systematically reviewing the published evidence for regulatory-approved QReports in dementia, we concluded that there is a significant evidence gap in the literature regarding clinical validation, workflow integration and in-use evaluation of these tools in dementia MRI diagnosis.

Evaluation framework to guide implementation of AI systems into healthcare settings

Objectives

To date, many artificial intelligence (AI) systems have been developed in healthcare, but adoption has been limited. This may be due to inappropriate or incomplete evaluation and a lack of internationally recognised AI standards on evaluation. To have confidence in the generalisability of AI systems in healthcare and to enable their integration into workflows, there is a need for a practical yet comprehensive instrument to assess the translational aspects of the available AI systems. Currently available evaluation frameworks for AI in healthcare focus on the reporting and regulatory aspects but have little guidance regarding assessment of the translational aspects of the AI systems like the functional, utility and ethical components.

Methods

To address this gap and create a framework that assesses real-world systems, an international team has developed a translationally focused evaluation framework termed ‘Translational Evaluation of Healthcare AI (TEHAI)’. A critical review of literature assessed existing evaluation and reporting frameworks and gaps. Next, using health technology evaluation and translational principles, reporting components were identified for consideration. These were independently reviewed for consensus inclusion in a final framework by an international panel of eight expert.

Results

TEHAI includes three main components: capability, utility and adoption. The emphasis on translational and ethical features of the model development and deployment distinguishes TEHAI from other evaluation instruments. In specific, the evaluation components can be applied at any stage of the development and deployment of the AI system.

Discussion

One major limitation of existing reporting or evaluation frameworks is their narrow focus. TEHAI, because of its strong foundation in translation research models and an emphasis on safety, translational value and generalisability, not only has a theoretical basis but also practical application to assessing real-world systems.

Conclusion

The translational research theoretic approach used to develop TEHAI should see it having application not just for evaluation of clinical AI in research settings, but more broadly to guide evaluation of working clinical systems.

Machine learning solutions in radiology: does the emperor have no clothes?

KEY POINTS

• Interest in radiomics and machine learning is steadily increasing and is reflected both in research output and number of commercially available solutions.• Currently available commercial products using machine learning are often supported by limited evidence of clinical usefulness and studies are often of low methodological quality.• Ethical and regulatory issues remain open and hinder implementation of machine learning software packages in daily clinical practice.