Lessons learned in transitioning to AI in the medical imaging of COVID-19

Revolutionizing Pulmonary Diagnostics: A Narrative Review of Artificial Intelligence Applications in Lung Imaging

Artificial intelligence (AI) has emerged as a transformative force in healthcare, particularly in pulmonary diagnostics. This comprehensive review explores the impact of AI on revolutionizing lung imaging, focusing on its applications in detecting abnormalities, diagnosing pulmonary conditions, and predicting disease prognosis. We provide an overview of traditional pulmonary diagnostic methods and highlight the importance of accurate and efficient lung imaging for early intervention and improved patient outcomes. Through the lens of AI, we examine machine learning algorithms, deep learning techniques, and natural language processing for analyzing radiology reports. Case studies and examples showcase the successful implementation of AI in pulmonary diagnostics, alongside challenges faced and lessons learned. Finally, we discuss future directions, including integrating AI into clinical workflows, ethical considerations, and the need for further research and collaboration in this rapidly evolving field. This review underscores the transformative potential of AI in enhancing the accuracy, efficiency, and accessibility of pulmonary healthcare.

Homogenization of multi-institutional chest x-ray images in various data transformation schemes

Purpose

Although there are several options for improving the generalizability of learned models, a data instance-based approach is desirable when stable data acquisition conditions cannot be guaranteed. Despite the wide use of data transformation methods to reduce data discrepancies between different data domains, detailed analysis for explaining the performance of data transformation methods is lacking.

Approach

This study compares several data transformation methods in the tuberculosis detection task with multi-institutional chest x-ray (CXR) data. Five different data transformations, including normalization, standardization with and without lung masking, and multi-frequency-based (MFB) standardization with and without lung masking were implemented. A tuberculosis detection network was trained using a reference dataset, and the data from six other sites were used for the network performance comparison. To analyze data harmonization performance, we extracted radiomic features and calculated the Mahalanobis distance. We visualized the features with a dimensionality reduction technique. Through similar methods, deep features of the trained networks were also analyzed to examine the models' responses to the data from various sites.

Results

From various numerical assessments, the MFB standardization with lung masking provided the highest network performance for the non-reference datasets. From the radiomic and deep feature analyses, the features of the multi-site CXRs after MFB with lung masking were found to be well homogenized to the reference data, whereas the others showed limited performance.

Conclusions

Conventional normalization and standardization showed suboptimal performance in minimizing feature differences among various sites. Our study emphasizes the strengths of MFB standardization with lung masking in terms of network performance and feature homogenization.

Sequestration of imaging studies in MIDRC: stratified sampling to balance demographic characteristics of patients in a multi-institutional data commons

Purpose

The Medical Imaging and Data Resource Center (MIDRC) is a multi-institutional effort to accelerate medical imaging machine intelligence research and create a publicly available image repository/commons as well as a sequestered commons for performance evaluation and benchmarking of algorithms. After de-identification, approximately 80% of the medical images and associated metadata become part of the open commons and 20% are sequestered from the open commons. To ensure that both commons are representative of the population available, we introduced a stratified sampling method to balance the demographic characteristics across the two datasets.

Approach

Our method uses multi-dimensional stratified sampling where several demographic variables of interest are sequentially used to separate the data into individual strata, each representing a unique combination of variables. Within each resulting stratum, patients are assigned to the open or sequestered commons. This algorithm was used on an example dataset containing 5000 patients using the variables of race, age, sex at birth, ethnicity, COVID-19 status, and image modality and compared resulting demographic distributions to naïve random sampling of the dataset over 2000 independent trials.

Results

Resulting prevalence of each demographic variable matched the prevalence from the input dataset within one standard deviation. Mann-Whitney U test results supported the hypothesis that sequestration by stratified sampling provided more balanced subsets than naïve randomization, except for demographic subcategories with very low prevalence.

Conclusions

The developed multi-dimensional stratified sampling algorithm can partition a large dataset while maintaining balance across several variables, superior to the balance achieved from naïve randomization.

Translation of AI into oncology clinical practice

Artificial intelligence (AI) is a transformative technology that is capturing popular imagination and can revolutionize biomedicine. AI and machine learning (ML) algorithms have the potential to break through existing barriers in oncology research and practice such as automating workflow processes, personalizing care, and reducing healthcare disparities. Emerging applications of AI/ML in the literature include screening and early detection of cancer, disease diagnosis, response prediction, prognosis, and accelerated drug discovery. Despite this excitement, only few AI/ML models have been properly validated and fewer have become regulated products for routine clinical use. In this review, we highlight the main challenges impeding AI/ML clinical translation. We present different clinical use cases from the domains of radiology, radiation oncology, immunotherapy, and drug discovery in oncology. We dissect the unique challenges and opportunities associated with each of these cases. Finally, we summarize the general requirements for successful AI/ML implementation in the clinic, highlighting specific examples and points of emphasis including the importance of multidisciplinary collaboration of stakeholders, role of domain experts in AI augmentation, transparency of AI/ML models, and the establishment of a comprehensive quality assurance program to mitigate risks of training bias and data drifts, all culminating toward safer and more beneficial AI/ML applications in oncology labs and clinics.

AI in medical imaging grand challenges: translation from competition to research benefit and patient care

Artificial intelligence (AI), in one form or another, has been a part of medical imaging for decades. The recent evolution of AI into approaches such as deep learning has dramatically accelerated the application of AI across a wide range of radiologic settings. Despite the promises of AI, developers and users of AI technology must be fully aware of its potential biases and pitfalls, and this knowledge must be incorporated throughout the AI system development pipeline that involves training, validation, and testing. Grand challenges offer an opportunity to advance the development of AI methods for targeted applications and provide a mechanism for both directing and facilitating the development of AI systems. In the process, a grand challenge centralizes (with the challenge organizers) the burden of providing a valid benchmark test set to assess performance and generalizability of participants' models and the collection and curation of image metadata, clinical/demographic information, and the required reference standard. The most relevant grand challenges are those designed to maximize the open-science nature of the competition, with code and trained models deposited for future public access. The ultimate goal of AI grand challenges is to foster the translation of AI systems from competition to research benefit and patient care. Rather than reference the many medical imaging grand challenges that have been organized by groups such as MICCAI, RSNA, AAPM, and grand-challenge.org, this review assesses the role of grand challenges in promoting AI technologies for research advancement and for eventual clinical implementation, including their promises and limitations.

AI and machine learning ethics, law, diversity, and global impact

Artificial intelligence (AI) and its machine learning (ML) algorithms are offering new promise for personalized biomedicine and more cost-effective healthcare with impressive technical capability to mimic human cognitive capabilities. However, widespread application of this promising technology has been limited in the medical domain and expectations have been tampered by ethical challenges and concerns regarding patient privacy, legal responsibility, trustworthiness, and fairness. To balance technical innovation with ethical applications of AI/ML, developers must demonstrate the AI functions as intended and adopt strategies to minimize the risks for failure or bias. This review describes the new ethical challenges created by AI/ML for clinical care and identifies specific considerations for its practice in medicine. We provide an overview of regulatory and legal issues applicable in Europe and the United States, a description of technical aspects to consider, and present recommendations for trustworthy AI/ML that promote transparency, minimize risks of bias or error, and protect the patient well-being.

MIDRC CRP10 AI interface-an integrated tool for exploring, testing and visualization of AI models

Objective. Developing Machine Learning models (N Gorre et al 2023) for clinical applications from scratch can be a cumbersome task requiring varying levels of expertise. Seasoned developers and researchers may also often face incompatible frameworks and data preparation issues. This is further complicated in the context of diagnostic radiology and oncology applications, given the heterogenous nature of the input data and the specialized task requirements. Our goal is to provide clinicians, researchers, and early AI developers with a modular, flexible, and user-friendly software tool that can effectively meet their needs to explore, train, and test AI algorithms by allowing users to interpret their model results. This latter step involves the incorporation of interpretability and explainability methods that would allow visualizing performance as well as interpreting predictions across the different neural network layers of a deep learning algorithm.Approach. To demonstrate our proposed tool, we have developed the CRP10 AI Application Interface (CRP10AII) as part of the MIDRC consortium. CRP10AII is based on the web service Django framework in Python. CRP10AII/Django/Python in combination with another data manager tool/platform, data commons such as Gen3 can provide a comprehensive while easy to use machine/deep learning analytics tool. The tool allows to test, visualize, interpret how and why the deep learning model is performing. The major highlight of CRP10AII is its capability of visualization and interpretability of otherwise Blackbox AI algorithms.Results. CRP10AII provides many convenient features for model building and evaluation, including: (1) query and acquire data according to the specific application (e.g. classification, segmentation) from the data common platform (Gen3 here); (2) train the AI models from scratch or use pre-trained models (e.g. VGGNet, AlexNet, BERT) for transfer learning and test the model predictions, performance assessment, receiver operating characteristics curve evaluation; (3) interpret the AI model predictions using methods like SHAPLEY, LIME values; and (4) visualize the model learning through heatmaps and activation maps of individual layers of the neural network.Significance. Unexperienced users may have more time to swiftly pre-process, build/train their AI models on their own use-cases, and further visualize and explore these AI models as part of this pipeline, all in an end-to-end manner. CRP10AII will be provided as an open-source tool, and we expect to continue developing it based on users' feedback.

Augmentation of literature review of COVID-19 radiology

We suggest an augmentation of the excellent comprehensive review article titled “Comprehensive literature review on the radiographic findings, imaging modalities, and the role of radiology in the coronavirus disease 2019 (COVID-19) pandemic” under the following categories: (1) “Inclusion of additional radiological features, related to pulmonary infarcts and to COVID-19 pneumonia”; (2) “Amplified discussion of cardiovascular COVID-19 manifestations and the role of cardiac magnetic resonance imaging in monitoring and prognosis”; (3) “Imaging findings related to fluorodeoxyglucose positron emission tomography, optical, thermal and other imaging modalities/devices, including ‘intelligent edge’ and other remote monitoring devices”; (4) “Artificial intelligence in COVID-19 imaging”; (5) “Additional annotations to the radiological images in the manuscript to illustrate the additional signs discussed”; and (6) “A minor correction to a passage on pulmonary destruction”.

The Regulation of Artificial Intelligence in Digital Radiology in the Scientific Literature: A Narrative Review of Reviews

Today, there is growing interest in artificial intelligence (AI) in the field of digital radiology (DR). This is also due to the push that has been applied in this sector due to the pandemic. Many studies are devoted to the challenges of integration in the health domain. One of the most important challenges is that of regulations. This study conducted a narrative review of reviews on the international approach to the regulation of AI in DR. The design of the study was based on: (I) An overview on Scopus and Pubmed (II) A qualification and eligibility process based on a standardized checklist and a scoring system. The results have highlighted an international approach to the regulation of these systems classified as "software as medical devices (SaMD)" arranged into: ethical issues, international regulatory framework, and bottlenecks of the legal issues. Several recommendations emerge from the analysis. They are all based on fundamental pillars: (a) The need to overcome a differentiated approach between countries. (b) The need for greater transparency and publicity of information both for SaMDs as a whole and for the algorithms and test patterns. (c) The need for an interdisciplinary approach that avoids bias (including demographic) in algorithms and test data. (d) The need to reduce some limits/gaps of the scientific literature production that do not cover the international approach.

Towards a guideline for evaluation metrics in medical image segmentation

In the last decade, research on artificial intelligence has seen rapid growth with deep learning models, especially in the field of medical image segmentation. Various studies demonstrated that these models have powerful prediction capabilities and achieved similar results as clinicians. However, recent studies revealed that the evaluation in image segmentation studies lacks reliable model performance assessment and showed statistical bias by incorrect metric implementation or usage. Thus, this work provides an overview and interpretation guide on the following metrics for medical image segmentation evaluation in binary as well as multi-class problems: Dice similarity coefficient, Jaccard, Sensitivity, Specificity, Rand index, ROC curves, Cohen's Kappa, and Hausdorff distance. Furthermore, common issues like class imbalance and statistical as well as interpretation biases in evaluation are discussed. As a summary, we propose a guideline for standardized medical image segmentation evaluation to improve evaluation quality, reproducibility, and comparability in the research field.

The Artificial Intelligence in Digital Radiology: Part 1: The Challenges, Acceptance and Consensus

Artificial intelligence is having important developments in the world of digital radiology also thanks to the boost given to the research sector by the COVID-19 pandemic. In the last two years, there was an important development of studies focused on both challenges and acceptance and consensus in the field of Artificial Intelligence. The challenges and acceptance and consensus are two strategic aspects in the development and integration of technologies in the health domain. The study conducted two narrative reviews by means of two parallel points of view to take stock both on the ongoing challenges and on initiatives conducted to face the acceptance and consensus in this area. The methodology of the review was based on: (I) search of PubMed and Scopus and (II) an eligibility assessment, using parameters with 5 levels of score. The results have: (a) highlighted and categorized the important challenges in place. (b) Illustrated the different types of studies conducted through original questionnaires. The study suggests for future research based on questionnaires a better calibration and inclusion of the challenges in place together with validation and administration paths at an international level.