Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms

An International Non-Inferiority Study for the Benchmarking of AI for Routine Radiology Cases: Chest X-ray, Fluorography and Mammography

An international reader study was conducted to gauge an average diagnostic accuracy of radiologists interpreting chest X-ray images, including those from fluorography and mammography, and establish requirements for stand-alone radiological artificial intelligence (AI) models. The retrospective studies in the datasets were labelled as containing or not containing target pathological findings based on a consensus of two experienced radiologists, and the results of a laboratory test and follow-up examination, where applicable. A total of 204 radiologists from 11 countries with various experience performed an assessment of the dataset with a 5-point Likert scale via a web platform. Eight commercial radiological AI models analyzed the same dataset. The AI AUROC was 0.87 (95% CI:0.83-0.9) versus 0.96 (95% CI 0.94-0.97) for radiologists. The sensitivity and specificity of AI versus radiologists were 0.71 (95% CI 0.64-0.78) versus 0.91 (95% CI 0.86-0.95) and 0.93 (95% CI 0.89-0.96) versus 0.9 (95% CI 0.85-0.94) for AI. The overall diagnostic accuracy of radiologists was superior to AI for chest X-ray and mammography. However, the accuracy of AI was noninferior to the least experienced radiologists for mammography and fluorography, and to all radiologists for chest X-ray. Therefore, an AI-based first reading could be recommended to reduce the workload burden of radiologists for the most common radiological studies such as chest X-ray and mammography.

Comparison of Mammography AI Algorithms with a Clinical Risk Model for 5-year Breast Cancer Risk Prediction: An Observational Study

Background Although several clinical breast cancer risk models are used to guide screening and prevention, they have only moderate discrimination. Purpose To compare selected existing mammography artificial intelligence (AI) algorithms and the Breast Cancer Surveillance Consortium (BCSC) risk model for prediction of 5-year risk. Materials and Methods This retrospective case-cohort study included data in women with a negative screening mammographic examination (no visible evidence of cancer) in 2016, who were followed until 2021 at Kaiser Permanente Northern California. Women with prior breast cancer or a highly penetrant gene mutation were excluded. Of the 324 009 eligible women, a random subcohort was selected, regardless of cancer status, to which all additional patients with breast cancer were added. The index screening mammographic examination was used as input for five AI algorithms to generate continuous scores that were compared with the BCSC clinical risk score. Risk estimates for incident breast cancer 0 to 5 years after the initial mammographic examination were calculated using a time-dependent area under the receiver operating characteristic curve (AUC). Results The subcohort included 13 628 patients, of whom 193 had incident cancer. Incident cancers in eligible patients (additional 4391 of 324 009) were also included. For incident cancers at 0 to 5 years, the time-dependent AUC for BCSC was 0.61 (95% CI: 0.60, 0.62). AI algorithms had higher time-dependent AUCs than did BCSC, ranging from 0.63 to 0.67 (Bonferroni-adjusted P < .0016). Time-dependent AUCs for combined BCSC and AI models were slightly higher than AI alone (AI with BCSC time-dependent AUC range, 0.66-0.68; Bonferroni-adjusted P < .0016). Conclusion When using a negative screening examination, AI algorithms performed better than the BCSC risk model for predicting breast cancer risk at 0 to 5 years. Combined AI and BCSC models further improved prediction. © RSNA, 2023 Supplemental material is available for this article.

A YOLO-based AI system for classifying calcifications on spot magnification mammograms

OBJECTIVES

Use of an AI system based on deep learning to investigate whether the system can aid in distinguishing malignant from benign calcifications on spot magnification mammograms, thus potentially reducing unnecessary biopsies.

METHODS

In this retrospective study, we included public and in-house datasets with annotations for the calcifications on both craniocaudal and mediolateral oblique vies, or both craniocaudal and mediolateral views of each case of mammograms. All the lesions had pathological results for correlation. Our system comprised an algorithm based on You Only Look Once (YOLO) named adaptive multiscale decision fusion module. The algorithm was pre-trained on a public dataset, Curated Breast Imaging Subset of Digital Database for Screening Mammography (CBIS-DDSM), then re-trained and tested on the in-house dataset of spot magnification mammograms. The performance of the system was investigated by receiver operating characteristic (ROC) analysis.

RESULTS

We included 1872 images from 753 calcification cases (414 benign and 339 malignant) from CBIS-DDSM. From the in-house dataset, 636 cases (432 benign and 204 malignant) with 1269 spot magnification mammograms were included, with all lesions being recommended for biopsy by radiologists. The area under the ROC curve for our system on the in-house testing dataset was 0.888 (95% CI 0.868-0.908), with a sensitivity of 88.4% (95% CI 86.9-8.99%), specificity of 80.8% (95% CI 77.6-84%), and an accuracy of 84.6% (95% CI 81.8-87.4%) at the optimal cutoff value. Using the system with two views of spot magnification mammograms, 80.8% benign biopsies could be avoided.

CONCLUSION

The AI system showed good accuracy for classification of calcifications on spot magnification mammograms which were all categorized as suspicious by radiologists, thereby potentially reducing unnecessary biopsies.

Artificial Intelligence as Supporting Reader in Breast Screening: A Novel Workflow to Preserve Quality and Reduce Workload

OBJECTIVE

To evaluate the effectiveness of a new strategy for using artificial intelligence (AI) as supporting reader for the detection of breast cancer in mammography-based double reading screening practice.

METHODS

Large-scale multi-site, multi-vendor data were used to retrospectively evaluate a new paradigm of AI-supported reading. Here, the AI served as the second reader only if it agrees with the recall/no-recall decision of the first human reader. Otherwise, a second human reader made an assessment followed by the standard clinical workflow. The data included 280 594 cases from 180 542 female participants screened for breast cancer at seven screening sites in two countries and using equipment from four hardware vendors. The statistical analysis included non-inferiority and superiority testing of cancer screening performance and evaluation of the reduction in workload, measured as arbitration rate and number of cases requiring second human reading.

RESULTS

Artificial intelligence as a supporting reader was found to be superior or noninferior on all screening metrics compared with human double reading while reducing the number of cases requiring second human reading by up to 87% (245 395/280 594). Compared with AI as an independent reader, the number of cases referred to arbitration was reduced from 13% (35 199/280 594) to 2% (5056/280 594).

CONCLUSION

The simulation indicates that the proposed workflow retains screening performance of human double reading while substantially reducing the workload. Further research should study the impact on the second human reader because they would only assess cases in which the AI prediction and first human reader disagree.

VinDr-Mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography

Mammography, or breast X-ray imaging, is the most widely used imaging modality to detect cancer and other breast diseases. Recent studies have shown that deep learning-based computer-assisted detection and diagnosis (CADe/x) tools have been developed to support physicians and improve the accuracy of interpreting mammography. A number of large-scale mammography datasets from different populations with various associated annotations and clinical data have been introduced to study the potential of learning-based methods in the field of breast radiology. With the aim to develop more robust and more interpretable support systems in breast imaging, we introduce VinDr-Mammo, a Vietnamese dataset of digital mammography with breast-level assessment and extensive lesion-level annotations, enhancing the diversity of the publicly available mammography data. The dataset consists of 5,000 mammography exams, each of which has four standard views and is double read with disagreement (if any) being resolved by arbitration. The purpose of this dataset is to assess Breast Imaging Reporting and Data System (BI-RADS) and breast density at the individual breast level. In addition, the dataset also provides the category, location, and BI-RADS assessment of non-benign findings. We make VinDr-Mammo publicly available as a new imaging resource to promote advances in developing CADe/x tools for mammography interpretation.

Evaluation of an Artificial Intelligence System for Detection of Invasive Lobular Carcinoma on Digital Mammography

Introduction Early breast cancer detection with screening mammography has been shown to reduce mortality and improve breast cancer survival. This study aims to evaluate the ability of an artificial intelligence computer-aided detection (AI CAD) system to detect biopsy-proven invasive lobular carcinoma (ILC) on digital mammography. Methods This retrospective study reviewed mammograms of patients who were diagnosed with biopsy-proved ILC between January 1, 2017, and January 1, 2022. All mammograms were analyzed using cmAssist^® (CureMetrix, San Diego, California, United States), which is an AI CAD for mammography. The AI CAD sensitivity for detecting ILC on mammography was calculated and further subdivided by lesion type, mass shape, and mass margins. To account for the within-subject correlation, generalized linear mixed models were implemented to investigate the association between age, family history, and breast density and whether the AI detected a false positive or true positive. Odds ratios, 95% confidence intervals, and p-values were also calculated. Results A total of 124 patients with 153 biopsy-proven ILC lesions were included. The AI CAD detected ILC on mammography with a sensitivity of 80%. The AI CAD had the highest sensitivity for detecting calcifications (100%), masses with irregular shape (82%), and masses with spiculated margins (86%). However, 88% of mammograms had at least one false positive mark with an average number of 3.9 false positive marks per mammogram. Conclusion The AI CAD system evaluated was successful in marking the malignancy in digital mammography. However, the numerous annotations confounded the ability to determine its overall accuracy and this reduces its potential use in real-life practice.

Microfluidic Technology, Artificial Intelligence, and Biosensors As Advanced Technologies in Cancer Screening: A Review Article

Cancer screening techniques aim to detect premalignant lesions and enable early intervention to delay the onset of cancer while keeping incidence constant. Technology advancements have led to the development of powerful tools such as microfluidic technology, artificial intelligence, machine learning algorithms, and electrochemical biosensors to aid in early cancer detection. Non-invasive cancer screening methods like virtual colonoscopy and endoscopic ultrasonography have also been developed to provide comprehensive pictures of organs and detect cancer early. This review article provides an overview of recent advances in cancer screening in microfluidic technology, artificial intelligence, and biomarkers through a narrative literature search. Microfluidic devices enable easy handling of sub-microliter volumes and have become a promising tool for cancer detection, drug screening, and modeling angiogenesis and metastasis in cancer research. Machine learning and artificial intelligence have shown high accuracy in oncology-related diagnostic imaging, reducing the manual steps in lesion detection and providing standardized and accurate results, with potential for global standardization in areas like colon polyps, breast cancer, and primary and metastatic brain cancer. A biomarker-based cancer diagnosis is promising for early detection and effective therapy, and electrochemical biosensors integrated with nanoparticles offer multiplexing and amplification capabilities. Understanding these advanced technologies' basics, achievements, and challenges is crucial for advancing their use in oncology.

Unassisted Clinicians Versus Deep Learning-Assisted Clinicians in Image-Based Cancer Diagnostics: Systematic Review With Meta-analysis

BACKGROUND

A number of publications have demonstrated that deep learning (DL) algorithms matched or outperformed clinicians in image-based cancer diagnostics, but these algorithms are frequently considered as opponents rather than partners. Despite the clinicians-in-the-loop DL approach having great potential, no study has systematically quantified the diagnostic accuracy of clinicians with and without the assistance of DL in image-based cancer identification.

OBJECTIVE

We systematically quantified the diagnostic accuracy of clinicians with and without the assistance of DL in image-based cancer identification.

METHODS

PubMed, Embase, IEEEXplore, and the Cochrane Library were searched for studies published between January 1, 2012, and December 7, 2021. Any type of study design was permitted that focused on comparing unassisted clinicians and DL-assisted clinicians in cancer identification using medical imaging. Studies using medical waveform-data graphics material and those investigating image segmentation rather than classification were excluded. Studies providing binary diagnostic accuracy data and contingency tables were included for further meta-analysis. Two subgroups were defined and analyzed, including cancer type and imaging modality.

RESULTS

In total, 9796 studies were identified, of which 48 were deemed eligible for systematic review. Twenty-five of these studies made comparisons between unassisted clinicians and DL-assisted clinicians and provided sufficient data for statistical synthesis. We found a pooled sensitivity of 83% (95% CI 80%-86%) for unassisted clinicians and 88% (95% CI 86%-90%) for DL-assisted clinicians. Pooled specificity was 86% (95% CI 83%-88%) for unassisted clinicians and 88% (95% CI 85%-90%) for DL-assisted clinicians. The pooled sensitivity and specificity values for DL-assisted clinicians were higher than for unassisted clinicians, at ratios of 1.07 (95% CI 1.05-1.09) and 1.03 (95% CI 1.02-1.05), respectively. Similar diagnostic performance by DL-assisted clinicians was also observed across the predefined subgroups.

CONCLUSIONS

The diagnostic performance of DL-assisted clinicians appears better than unassisted clinicians in image-based cancer identification. However, caution should be exercised, because the evidence provided in the reviewed studies does not cover all the minutiae involved in real-world clinical practice. Combining qualitative insights from clinical practice with data-science approaches may improve DL-assisted practice, although further research is required.

TRIAL REGISTRATION

PROSPERO CRD42021281372; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=281372.

Lessons learned from independent external validation of an AI tool to detect breast cancer using a representative UK data set

OBJECTIVE

To pilot a process for the independent external validation of an artificial intelligence (AI) tool to detect breast cancer using data from the NHS breast screening programme (NHSBSP).

METHODS

A representative data set of mammography images from 26,000 women attending 2 NHS screening centres, and an enriched data set of 2054 positive cases were used from the OPTIMAM image database. The use case of the AI tool was the replacement of the first or second human reader. The performance of the AI tool was compared to that of human readers in the NHSBSP.

RESULTS

Recommendations for future external validations of AI tools to detect breast cancer are provided. The tool recalled different breast cancers to the human readers. This study showed the importance of testing AI tools on all types of cases (including non-standard) and the clarity of any warning messages. The acceptable difference in sensitivity and specificity between the AI tool and human readers should be determined. Any information vital for the clinical application should be a required output for the AI tool. It is recommended that the interaction of radiologists with the AI tool, and the effect of the AI tool on arbitration be investigated prior to clinical use.

CONCLUSION

This pilot demonstrated several lessons for future independent external validation of AI tools for breast cancer detection.

ADVANCES IN KNOWLEDGE

Knowledge has been gained towards best practice procedures for performing independent external validations of AI tools for the detection of breast cancer using data from the NHS Breast Screening Programme.

Virtual Biopsy by Using Artificial Intelligence-based Multimodal Modeling of Binational Mammography Data

Background Computational models based on artificial intelligence (AI) are increasingly used to diagnose malignant breast lesions. However, assessment from radiologic images of the specific pathologic lesion subtypes, as detailed in the results of biopsy procedures, remains a challenge. Purpose To develop an AI-based model to identify breast lesion subtypes with mammograms and linked electronic health records labeled with histopathologic information. Materials and Methods In this retrospective study, 26 569 images were collected in 9234 women who underwent digital mammography to pretrain the algorithms. The training data included individuals who had at least 1 year of clinical and imaging history followed by biopsy-based histopathologic diagnosis from March 2013 to November 2018. A model that combined convolutional neural networks with supervised learning algorithms was independently trained to make breast lesion predictions with data from 2120 women in Israel and 1642 women in the United States. Results were reported using the area under the receiver operating characteristic curve (AUC) with the 95% DeLong approach to estimate CIs. Significance was tested with bootstrapping. Results The Israeli model was validated in 456 women and tested in 441 women (mean age, 51 years ± 11 [SD]). The U.S. model was validated in 350 women and tested in 344 women (mean age, 60 years ± 12). For predicting malignancy in the test sets (consisting of 220 Israeli patient examinations and 126 U.S. patient examinations with ductal carcinoma in situ or invasive cancer), the algorithms obtained an AUC of 0.88 (95% CI: 0.85, 0.91) and 0.80 (95% CI: 0.74, 0.85) for Israeli and U.S. patients, respectively (P = .006). These results may not hold for other cohorts of patients, and generalizability across populations should be further investigated. Conclusion The results offer supporting evidence that artificial intelligence applied to clinical and mammographic images can identify breast lesion subtypes when the data are sufficiently large, which may help assess diagnostic workflow and reduce biopsy sampling errors. Published under a CC BY 4.0 license. Online supplemental material is available for this article.

Artificial intelligence (AI) for breast cancer screening: BreastScreen population-based cohort study of cancer detection

BACKGROUND

Artificial intelligence (AI) has been proposed to reduce false-positive screens, increase cancer detection rates (CDRs), and address resourcing challenges faced by breast screening programs. We compared the accuracy of AI versus radiologists in real-world population breast cancer screening, and estimated potential impacts on CDR, recall and workload for simulated AI-radiologist reading.

METHODS

External validation of a commercially-available AI algorithm in a retrospective cohort of 108,970 consecutive mammograms from a population-based screening program, with ascertained outcomes (including interval cancers by registry linkage). Area under the ROC curve (AUC), sensitivity and specificity for AI were compared with radiologists who interpreted the screens in practice. CDR and recall were estimated for simulated AI-radiologist reading (with arbitration) and compared with program metrics.

FINDINGS

The AUC for AI was 0.83 compared with 0.93 for radiologists. At a prospective threshold, sensitivity for AI (0.67; 95% CI: 0.64-0.70) was comparable to radiologists (0.68; 95% CI: 0.66-0.71) with lower specificity (0.81 [95% CI: 0.81-0.81] versus 0.97 [95% CI: 0.97-0.97]). Recall rate for AI-radiologist reading (3.14%) was significantly lower than for the BSWA program (3.38%) (-0.25%; 95% CI: -0.31 to -0.18; P < 0.001). CDR was also lower (6.37 versus 6.97 per 1000) (-0.61; 95% CI: -0.77 to -0.44; P < 0.001); however, AI detected interval cancers that were not found by radiologists (0.72 per 1000; 95% CI: 0.57-0.90). AI-radiologist reading increased arbitration but decreased overall screen-reading volume by 41.4% (95% CI: 41.2-41.6).

INTERPRETATION

Replacement of one radiologist by AI (with arbitration) resulted in lower recall and overall screen-reading volume. There was a small reduction in CDR for AI-radiologist reading. AI detected interval cases that were not identified by radiologists, suggesting potentially higher CDR if radiologists were unblinded to AI findings. These results indicate AI's potential role as a screen-reader of mammograms, but prospective trials are required to determine whether CDR could improve if AI detection was actioned in double-reading with arbitration.

FUNDING

National Breast Cancer Foundation (NBCF), National Health and Medical Research Council (NHMRC).

The utilization of artificial intelligence applications to improve breast cancer detection and prognosis

Breast imaging faces challenges with the current increase in medical imaging requests and lesions that breast screening programs can miss. Solutions to improve these challenges are being sought with the recent advancement and adoption of artificial intelligent (AI)-based applications to enhance workflow efficiency as well as patient-healthcare outcomes. rtificial intelligent tools have been proposed and used to analyze different modes of breast imaging, in most of the published studies, mainly for the detection and classification of breast lesions, breast lesion segmentation, breast density evaluation, and breast cancer risk assessment. This article reviews the background of the Conventional Computer-aided Detection system and AI, AI-based applications in breast medical imaging for the identification, segmentation, and categorization of lesions, breast density and cancer risk evaluation. In addition, the challenges, and limitations of AI-based applications in breast imaging are also discussed.

Technology Platforms and Approaches for Building and Evaluating Machine Learning Methods in Healthcare

BACKGROUND

Artificial intelligence (AI) methods are becoming increasingly commonly implemented in healthcare as decision support, business intelligence tools, or, in some cases, Food and Drug Administration-approved clinical decision-makers. Advanced lab-based diagnostic tools are increasingly becoming AI driven. The path from data to machine learning methods is an active area for research and quality improvement, and there are few established best practices. With data being generated at an unprecedented rate, there is a need for processes that enable data science investigation that protect patient privacy and minimize other business risks. New approaches for data sharing are being utilized that lower these risks.

CONTENT

In this short review, clinical and translational AI governance is introduced along with approaches for securely building, sharing, and validating accurate and fair models. This is a constantly evolving field, and there is much interest in collecting data using standards, sharing data, building new models, evaluating models, sharing models, and, of course, implementing models into practice.

SUMMARY

AI is an active area of research and development broadly for healthcare and laboratory testing. Robust data governance and machine learning methodological governance are required. New approaches for data sharing are enabling the development of models and their evaluation. Evaluation of methods is difficult, particularly when the evaluation is performed by the team developing the method, and should ideally be prospective. New technologies have enabled standardization of platforms for moving analytics and data science methods.

The EMory BrEast imaging Dataset (EMBED): A Racially Diverse, Granular Dataset of 3.4 Million Screening and Diagnostic Mammographic Images

Supplemental material is available for this article. Keywords: Mammography, Breast, Machine Learning © RSNA, 2023.

Development and Validation of a Deep Learning Algorithm to Differentiate Colon Carcinoma From Acute Diverticulitis in Computed Tomography Images

IMPORTANCE

Differentiating between malignant and benign etiology in large-bowel wall thickening on computed tomography (CT) images can be a challenging task. Artificial intelligence (AI) support systems can improve the diagnostic accuracy of radiologists, as shown for a variety of imaging tasks. Improvements in diagnostic performance, in particular the reduction of false-negative findings, may be useful in patient care.

OBJECTIVE

To develop and evaluate a deep learning algorithm able to differentiate colon carcinoma (CC) and acute diverticulitis (AD) on CT images and analyze the impact of the AI-support system in a reader study.

DESIGN, SETTING, AND PARTICIPANTS

In this diagnostic study, patients who underwent surgery between July 1, 2005, and October 1, 2020, for CC or AD were included. Three-dimensional (3-D) bounding boxes including the diseased bowel segment and surrounding mesentery were manually delineated and used to develop a 3-D convolutional neural network (CNN). A reader study with 10 observers of different experience levels was conducted. Readers were asked to classify the testing cohort under reading room conditions, first without and then with algorithmic support.

MAIN OUTCOMES AND MEASURES

To evaluate the diagnostic performance, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for all readers and reader groups with and without AI support. Metrics were compared using the McNemar test and relative and absolute predictive value comparisons.

RESULTS

A total of 585 patients (AD: n = 267, CC: n = 318; mean [SD] age, 63.2 [13.4] years; 341 men [58.3%]) were included. The 3-D CNN reached a sensitivity of 83.3% (95% CI, 70.0%-96.6%) and specificity of 86.6% (95% CI, 74.5%-98.8%) for the test set, compared with the mean reader sensitivity of 77.6% (95% CI, 72.9%-82.3%) and specificity of 81.6% (95% CI, 77.2%-86.1%). The combined group of readers improved significantly with AI support from a sensitivity of 77.6% to 85.6% (95% CI, 81.3%-89.3%; P < .001) and a specificity of 81.6% to 91.3% (95% CI, 88.1%-94.5%; P < .001). Artificial intelligence support significantly reduced the number of false-negative and false-positive findings (NPV from 78.5% to 86.4% and PPV from 80.9% to 90.8%; P < .001).

CONCLUSIONS AND RELEVANCE

The findings of this study suggest that a deep learning model able to distinguish CC and AD in CT images as a support system may significantly improve the diagnostic performance of radiologists, which may improve patient care.

Symmetry-based regularization in deep breast cancer screening

Breast cancer is the most common and lethal form of cancer in women. Recent efforts have focused on developing accurate neural network-based computer-aided diagnosis systems for screening to help anticipate this disease. The ultimate goal is to reduce mortality and improve quality of life after treatment. Due to the difficulty in collecting and annotating data in this domain, data scarcity is - and will continue to be - a limiting factor. In this work, we present a unified view of different regularization methods that incorporate domain-known symmetries in the model. Three general strategies were followed: (i) data augmentation, (ii) invariance promotion in the loss function, and (iii) the use of equivariant architectures. Each of these strategies encodes different priors on the functions learned by the model and can be readily introduced in most settings. Empirically we show that the proposed symmetry-based regularization procedures improve generalization to unseen examples. This advantage is verified in different scenarios, datasets and model architectures. We hope that both the principle of symmetry-based regularization and the concrete methods presented can guide development towards more data-efficient methods for breast cancer screening as well as other medical imaging domains.

Artificial intelligence in healthcare: Complementing, not replacing, doctors and healthcare providers

The utilization of artificial intelligence (AI) in clinical practice has increased and is evidently contributing to improved diagnostic accuracy, optimized treatment planning, and improved patient outcomes. The rapid evolution of AI, especially generative AI and large language models (LLMs), have reignited the discussions about their potential impact on the healthcare industry, particularly regarding the role of healthcare providers. Concerning questions, "can AI replace doctors?" and "will doctors who are using AI replace those who are not using it?" have been echoed. To shed light on this debate, this article focuses on emphasizing the augmentative role of AI in healthcare, underlining that AI is aimed to complement, rather than replace, doctors and healthcare providers. The fundamental solution emerges with the human-AI collaboration, which combines the cognitive strengths of healthcare providers with the analytical capabilities of AI. A human-in-the-loop (HITL) approach ensures that the AI systems are guided, communicated, and supervised by human expertise, thereby maintaining safety and quality in healthcare services. Finally, the adoption can be forged further by the organizational process informed by the HITL approach to improve multidisciplinary teams in the loop. AI can create a paradigm shift in healthcare by complementing and enhancing the skills of healthcare providers, ultimately leading to improved service quality, patient outcomes, and a more efficient healthcare system.

A Comparison of Techniques for Class Imbalance in Deep Learning Classification of Breast Cancer

Tools based on deep learning models have been created in recent years to aid radiologists in the diagnosis of breast cancer from mammograms. However, the datasets used to train these models may suffer from class imbalance, i.e., there are often fewer malignant samples than benign or healthy cases, which can bias the model towards the healthy class. In this study, we systematically evaluate several popular techniques to deal with this class imbalance, namely, class weighting, over-sampling, and under-sampling, as well as a synthetic lesion generation approach to increase the number of malignant samples. These techniques are applied when training on three diverse Full-Field Digital Mammography datasets, and tested on in-distribution and out-of-distribution samples. The experiments show that a greater imbalance is associated with a greater bias towards the majority class, which can be counteracted by any of the standard class imbalance techniques. On the other hand, these methods provide no benefit to model performance with respect to Area Under the Curve of the Recall Operating Characteristic (AUC-ROC), and indeed under-sampling leads to a reduction of 0.066 in AUC in the case of a 19:1 benign to malignant imbalance. Our synthetic lesion methodology leads to better performance in most cases, with increases of up to 0.07 in AUC on out-of-distribution test sets over the next best experiment.

Breast cancer screening with digital breast tomosynthesis: comparison of different reading strategies implementing artificial intelligence

Objectives

Digital breast tomosynthesis (DBT) can detect more cancers than the current standard breast screening method, digital mammography (DM); however, it can substantially increase the reading workload and thus hinder implementation in screening. Artificial intelligence (AI) might be a solution. The aim of this study was to retrospectively test different ways of using AI in a screening workflow.

Methods

An AI system was used to analyse 14,772 double-read single-view DBT examinations from a screening trial with paired DM double reading. Three scenarios were studied: if AI can identify normal cases that can be excluded from human reading; if AI can replace the second reader; if AI can replace both readers. The number of detected cancers and false positives was compared with DM or DBT double reading.

Results

By excluding normal cases and only reading 50.5% (7460/14,772) of all examinations, 95% (121/127) of the DBT double reading detected cancers could be detected. Compared to DM screening, 27% (26/95) more cancers could be detected (p < 0.001) while keeping recall rates at the same level. With AI replacing the second reader, 95% (120/127) of the DBT double reading detected cancers could be detected—26% (25/95) more than DM screening (p < 0.001)—while increasing recall rates by 53%. AI alone with DBT has a sensitivity similar to DM double reading (p = 0.689).

Conclusion

AI can open up possibilities for implementing DBT screening and detecting more cancers with the total reading workload unchanged. Considering the potential legal and psychological implications, replacing the second reader with AI would probably be most the feasible approach.

Key Points

• Breast cancer screening with digital breast tomosynthesis and artificial intelligence can detect more cancers than mammography screening without increasing screen-reading workload.

• Artificial intelligence can either exclude low-risk cases from double reading or replace the second reader.

• Retrospective study based on paired mammography and digital breast tomosynthesis screening data.

Artificial Intelligence in Breast Cancer Screening: Evaluation of FDA Device Regulation and Future Recommendations

Importance

Contemporary approaches to artificial intelligence (AI) based on deep learning have generated interest in the application of AI to breast cancer screening (BCS). The US Food and Drug Administration (FDA) has approved several next-generation AI products indicated for BCS in recent years; however, questions regarding their accuracy, appropriate use, and clinical utility remain.

Objectives

To describe the current FDA regulatory process for AI products, summarize the evidence used to support FDA clearance and approval of AI products indicated for BCS, consider the advantages and limitations of current regulatory approaches, and suggest ways to improve the current system.

Evidence Review

Premarket notifications and other publicly available documents used for FDA clearance and approval of AI products indicated for BCS from January 1, 2017, to December 31, 2021.

Findings

Nine AI products indicated for BCS for identification of suggestive lesions and mammogram triage were included. Most of the products had been cleared through the 510(k) pathway, and all clearances were based on previously collected retrospective data; 6 products used multicenter designs; 7 products used enriched data; and 4 lacked details on whether products were externally validated. Test performance measures, including sensitivity, specificity, and area under the curve, were the main outcomes reported. Most of the devices used tissue biopsy as the criterion standard for BCS accuracy evaluation. Other clinical outcome measures, including cancer stage at diagnosis and interval cancer detection, were not reported for any of the devices.

Conclusions and Relevance

The findings of this review suggest important gaps in reporting of data sources, data set type, validation approach, and clinical utility assessment. As AI-assisted reading becomes more widespread in BCS and other radiologic examinations, strengthened FDA evidentiary regulatory standards, development of postmarketing surveillance, a focus on clinically meaningful outcomes, and stakeholder engagement will be critical for ensuring the safety and efficacy of these products.

Using Deep Neural Network Approach for Multiple-Class Assessment of Digital Mammography

According to the Health Promotion Administration in the Ministry of Health and Welfare statistics in Taiwan, over ten thousand women have breast cancer every year. Mammography is widely used to detect breast cancer. However, it is limited by the operator's technique, the cooperation of the subjects, and the subjective interpretation by the physician. It results in inconsistent identification. Therefore, this study explores the use of a deep neural network algorithm for the classification of mammography images. In the experimental design, a retrospective study was used to collect imaging data from actual clinical cases. The mammography images were collected and classified according to the breast image reporting and data-analyzing system (BI-RADS). In terms of model building, a fully convolutional dense connection network (FC-DCN) is used for the network backbone. All the images were obtained through image preprocessing, a data augmentation method, and transfer learning technology to build a mammography image classification model. The research results show the model's accuracy, sensitivity, and specificity were 86.37%, 100%, and 72.73%, respectively. Based on the FC-DCN model framework, it can effectively reduce the number of training parameters and successfully obtain a reasonable image classification model for mammography.

Artificial intelligence assistance for women who had spot compression view: reducing recall rates for digital mammography

Background

Mammography yields inevitable recall for indeterminate findings that need to be confirmed with additional views.

Purpose

To explore whether the artificial intelligence (AI) algorithm for mammography can reduce false-positive recall in patients who undergo the spot compression view.

Material and Methods

From January to December 2017, 236 breasts from 225 women who underwent the spot compression view due to focal asymmetry, mass, or architectural distortion on standard digital mammography were included. Three readers who were blinded to the study purpose, patient information, previous mammograms, following spot compression views, and any clinical or pathologic reports retrospectively reviewed 236 standard mammograms and determined the necessity of patient recall and the probability of malignancy per breast, first without and then with AI assistance. The performances of AI and the readers were evaluated with the recall rate, area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and accuracy.

Results

Among 236 examinations, 8 (3.4%) were cancers and 228 (96.6%) were benign. The recall rates of all three readers significantly decreased with AI assistance ( P < 0.05). The reader-averaged recall rates significantly decreased with AI assistance regardless of breast composition (fatty breasts: 32.7% to 24.1%m P = 0.002; dense breasts: 33.6% to 21.2%, P < 0.001). The reader-averaged AUC increased with AI assistance and was comparable to that of standalone AI (0.835 vs. 0.895; P = 0.234). The reader-averaged specificity (71.2% to 79.8%, P < 0.001) and accuracy (71.3% to 79.7%, P  < 0.001) significantly improved with AI assistance.

Conclusion

AI assistance significantly reduced false-positive recall without compromising cancer detection in women with focal asymmetry, mass, or architectural distortion on standard digital mammography regardless of mammographic breast density.

Depiction of breast cancers on digital mammograms by artificial intelligence-based computer-assisted diagnosis according to cancer characteristics

OBJECTIVE

To evaluate how breast cancers are depicted by artificial intelligence-based computer-assisted diagnosis (AI-CAD) according to clinical, radiological, and pathological factors.

MATERIALS AND METHODS

From January 2017 to December 2017, 896 patients diagnosed with 930 breast cancers were enrolled in this retrospective study. Commercial AI-CAD was applied to digital mammograms and abnormality scores were obtained. We evaluated the abnormality score according to clinical, radiological, and pathological characteristics. False-negative results were defined by abnormality scores less than 10.

RESULTS

The median abnormality score of 930 breasts was 87.4 (range 0-99). The false-negative rate of AI-CAD was 19.4% (180/930). Cancers with an abnormality score of more than 90 showed a high proportion of palpable lesions, BI-RADS 4c and 5 lesions, cancers presenting as mass with or without microcalcifications and invasive cancers compared with low-scored cancers (all p < 0.001). False-negative cancers were more likely to develop in asymptomatic patients and extremely dense breasts and to be diagnosed as occult breast cancers and DCIS compared to detected cancers.

CONCLUSION

Breast cancers depicted with high abnormality scores by AI-CAD are associated with higher BI-RADS category, invasive pathology, and higher cancer stage.

KEY POINTS

• High-scored cancers by AI-CAD included a high proportion of BI-RADS 4c and 5 lesions, masses with or without microcalcifications, and cancers with invasive pathology. • Among invasive cancers, cancers with higher T and N stage and HER2-enriched subtype were depicted with higher abnormality scores by AI-CAD. • Cancers missed by AI-CAD tended to be in asymptomatic patients and extremely dense breasts and to be diagnosed as occult breast cancers by radiologists.

External Validation of an Ensemble Model for Automated Mammography Interpretation by Artificial Intelligence

Importance

With a shortfall in fellowship-trained breast radiologists, mammography screening programs are looking toward artificial intelligence (AI) to increase efficiency and diagnostic accuracy. External validation studies provide an initial assessment of how promising AI algorithms perform in different practice settings.

Objective

To externally validate an ensemble deep-learning model using data from a high-volume, distributed screening program of an academic health system with a diverse patient population.

Design, Setting, and Participants

In this diagnostic study, an ensemble learning method, which reweights outputs of the 11 highest-performing individual AI models from the Digital Mammography Dialogue on Reverse Engineering Assessment and Methods (DREAM) Mammography Challenge, was used to predict the cancer status of an individual using a standard set of screening mammography images. This study was conducted using retrospective patient data collected between 2010 and 2020 from women aged 40 years and older who underwent a routine breast screening examination and participated in the Athena Breast Health Network at the University of California, Los Angeles (UCLA).

Main Outcomes and Measures

Performance of the challenge ensemble method (CEM) and the CEM combined with radiologist assessment (CEM+R) were compared with diagnosed ductal carcinoma in situ and invasive cancers within a year of the screening examination using performance metrics, such as sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC).

Results

Evaluated on 37 317 examinations from 26 817 women (mean [SD] age, 58.4 [11.5] years), individual model AUROC estimates ranged from 0.77 (95% CI, 0.75-0.79) to 0.83 (95% CI, 0.81-0.85). The CEM model achieved an AUROC of 0.85 (95% CI, 0.84-0.87) in the UCLA cohort, lower than the performance achieved in the Kaiser Permanente Washington (AUROC, 0.90) and Karolinska Institute (AUROC, 0.92) cohorts. The CEM+R model achieved a sensitivity (0.813 [95% CI, 0.781-0.843] vs 0.826 [95% CI, 0.795-0.856]; P = .20) and specificity (0.925 [95% CI, 0.916-0.934] vs 0.930 [95% CI, 0.929-0.932]; P = .18) similar to the radiologist performance. The CEM+R model had significantly lower sensitivity (0.596 [95% CI, 0.466-0.717] vs 0.850 [95% CI, 0.766-0.923]; P < .001) and specificity (0.803 [95% CI, 0.734-0.861] vs 0.945 [95% CI, 0.936-0.954]; P < .001) than the radiologist in women with a prior history of breast cancer and Hispanic women (0.894 [95% CI, 0.873-0.910] vs 0.926 [95% CI, 0.919-0.933]; P = .004).

Conclusions and Relevance

This study found that the high performance of an ensemble deep-learning model for automated screening mammography interpretation did not generalize to a more diverse screening cohort, suggesting that the model experienced underspecification. This study suggests the need for model transparency and fine-tuning of AI models for specific target populations prior to their clinical adoption.

Multicenter, Multivendor Validation of an FDA-approved Algorithm for Mammography Triage

OBJECTIVE

Artificial intelligence (AI)-based triage algorithms may improve cancer detection and expedite radiologist workflow. To this end, the performance of a commercial AI-based triage algorithm on screening mammograms was evaluated across breast densities and lesion types.

METHODS

This retrospective, IRB-exempt, multicenter, multivendor study examined 1255 screening 4-view mammograms (400 positive and 855 negative studies). Images were anonymized by providing institutions and analyzed by a commercially available AI algorithm (cmTriage, CureMetrix, La Jolla, CA) that performed retrospective triage at the study level by flagging exams as "suspicious" or not. Sensitivities and specificities with confidence intervals were derived from area under the curve (AUC) calculations.

RESULTS

The algorithm demonstrated an AUC of 0.95 (95% CI: 0.94-0.96) for case identification. Area under the curve held across densities (0.95) and lesion types (masses: 0.94 [95% CI: 0.92-0.96] or microcalcifications: 0.97 [95% CI: 0.96-0.99]). The algorithm has a default sensitivity of 93% (95% CI: 95.6%-90.5%) with specificity of 76.3% (95% CI: 79.2%-73.4%). To evaluate real-world performance, a sensitivity of 86.9% (95% CI: 83.6%-90.2%) was tested, as observed for practicing radiologists by the Breast Cancer Surveillance Consortium (BCSC) study. The resulting specificity was 88.5% (95% CI: 86.4%-90.7%), similar to the BCSC specificity of 88.9%, indicating performance comparable to real-world results.

CONCLUSION

When tested for lesion detection, an AI-based triage software can perform at the level of practicing radiologists. Drawing attention to suspicious exams may improve reader specificity and help streamline radiologist workflow, enabling faster turnaround times and improving care.

Domain generalization in deep learning based mass detection in mammography: A large-scale multi-center study

Computer-aided detection systems based on deep learning have shown great potential in breast cancer detection. However, the lack of domain generalization of artificial neural networks is an important obstacle to their deployment in changing clinical environments. In this study, we explored the domain generalization of deep learning methods for mass detection in digital mammography and analyzed in-depth the sources of domain shift in a large-scale multi-center setting. To this end, we compared the performance of eight state-of-the-art detection methods, including Transformer based models, trained in a single domain and tested in five unseen domains. Moreover, a single-source mass detection training pipeline was designed to improve the domain generalization without requiring images from the new domain. The results show that our workflow generalized better than state-of-the-art transfer learning based approaches in four out of five domains while reducing the domain shift caused by the different acquisition protocols and scanner manufacturers. Subsequently, an extensive analysis was performed to identify the covariate shifts with the greatest effects on detection performance, such as those due to differences in patient age, breast density, mass size, and mass malignancy. Ultimately, this comprehensive study provides key insights and best practices for future research on domain generalization in deep learning based breast cancer detection.

Radiologist Preferences for Artificial Intelligence-Based Decision Support During Screening Mammography Interpretation

BACKGROUND

Artificial intelligence (AI) may improve cancer detection and risk prediction during mammography screening, but radiologists' preferences regarding its characteristics and implementation are unknown.

PURPOSE

To quantify how different attributes of AI-based cancer detection and risk prediction tools affect radiologists' intentions to use AI during screening mammography interpretation.

MATERIALS AND METHODS

Through qualitative interviews with radiologists, we identified five primary attributes for AI-based breast cancer detection and four for breast cancer risk prediction. We developed a discrete choice experiment based on these attributes and invited 150 US-based radiologists to participate. Each respondent made eight choices for each tool between three alternatives: two hypothetical AI-based tools versus screening without AI. We analyzed samplewide preferences using random parameters logit models and identified subgroups with latent class models.

RESULTS

Respondents (n = 66; 44% response rate) were from six diverse practice settings across eight states. Radiologists were more interested in AI for cancer detection when sensitivity and specificity were balanced (94% sensitivity with <25% of examinations marked) and AI markup appeared at the end of the hanging protocol after radiologists complete their independent review. For AI-based risk prediction, radiologists preferred AI models using both mammography images and clinical data. Overall, 46% to 60% intended to adopt any of the AI tools presented in the study; 26% to 33% approached AI enthusiastically but were deterred if the features did not align with their preferences.

CONCLUSION

Although most radiologists want to use AI-based decision support, short-term uptake may be maximized by implementing tools that meet the preferences of dissuadable users.

Evaluation of an artificial intelligence support system for breast cancer screening in Chinese people based on mammogram

BACKGROUND

To evaluate the diagnostic performance of radiologists on breast cancer with or without artificial intelligence (AI) support.

METHODS

A retrospective study was performed. In total, 643 mammograms (average age: 54 years; female: 100%; cancer: 62.05%) were randomly allocated into two groups. Seventy-five percent of mammograms in each group were randomly selected for assessment by two independent radiologists, and the rest were read once. Half of the 71 radiologists could read mammograms with AI support, and the other half could not. Sensitivity, specificity, Youden's index, agreement rate, Kappa value, the area under the receiver operating characteristic curve (AUC) and the reading time of radiologists in each group were analyzed.

RESULTS

The average AUC was higher if the AI support system was used (unaided: 0.84; with AI support: 0.91; p < 0.01). The average sensitivity increased from 84.77% to 95.07% with AI support (p < 0.01), but the average specificity decreased (p = 0.07). Youden's index, agreement rate and Kappa value were larger in the group with AI support, and the average reading time was shorter (p < 0.01).

CONCLUSIONS

The AI support system might contribute to enhancing the diagnostic performance (e.g., higher sensitivity and AUC) of radiologists. In the future, the AI algorithm should be improved, and prospective studies should be conducted.

Applying artificial intelligence technology to assist with breast cancer diagnosis and prognosis prediction

Breast cancer remains the most diagnosed cancer in women. Advances in medical imaging modalities and technologies have greatly aided in the early detection of breast cancer and the decline of patient mortality rates. However, reading and interpreting breast images remains difficult due to the high heterogeneity of breast tumors and fibro-glandular tissue, which results in lower cancer detection sensitivity and specificity and large inter-reader variability. In order to help overcome these clinical challenges, researchers have made great efforts to develop computer-aided detection and/or diagnosis (CAD) schemes of breast images to provide radiologists with decision-making support tools. Recent rapid advances in high throughput data analysis methods and artificial intelligence (AI) technologies, particularly radiomics and deep learning techniques, have led to an exponential increase in the development of new AI-based models of breast images that cover a broad range of application topics. In this review paper, we focus on reviewing recent advances in better understanding the association between radiomics features and tumor microenvironment and the progress in developing new AI-based quantitative image feature analysis models in three realms of breast cancer: predicting breast cancer risk, the likelihood of tumor malignancy, and tumor response to treatment. The outlook and three major challenges of applying new AI-based models of breast images to clinical practice are also discussed. Through this review we conclude that although developing new AI-based models of breast images has achieved significant progress and promising results, several obstacles to applying these new AI-based models to clinical practice remain. Therefore, more research effort is needed in future studies.

Automatic Classification of Simulated Breast Tomosynthesis Whole Images for the Presence of Microcalcification Clusters Using Deep CNNs

Microcalcification clusters (MCs) are among the most important biomarkers for breast cancer, especially in cases of nonpalpable lesions. The vast majority of deep learning studies on digital breast tomosynthesis (DBT) are focused on detecting and classifying lesions, especially soft-tissue lesions, in small regions of interest previously selected. Only about 25% of the studies are specific to MCs, and all of them are based on the classification of small preselected regions. Classifying the whole image according to the presence or absence of MCs is a difficult task due to the size of MCs and all the information present in an entire image. A completely automatic and direct classification, which receives the entire image, without prior identification of any regions, is crucial for the usefulness of these techniques in a real clinical and screening environment. The main purpose of this work is to implement and evaluate the performance of convolutional neural networks (CNNs) regarding an automatic classification of a complete DBT image for the presence or absence of MCs (without any prior identification of regions). In this work, four popular deep CNNs are trained and compared with a new architecture proposed by us. The main task of these trainings was the classification of DBT cases by absence or presence of MCs. A public database of realistic simulated data was used, and the whole DBT image was taken into account as input. DBT data were considered without and with preprocessing (to study the impact of noise reduction and contrast enhancement methods on the evaluation of MCs with CNNs). The area under the receiver operating characteristic curve (AUC) was used to evaluate the performance. Very promising results were achieved with a maximum AUC of 94.19% for the GoogLeNet. The second-best AUC value was obtained with a new implemented network, CNN-a, with 91.17%. This CNN had the particularity of also being the fastest, thus becoming a very interesting model to be considered in other studies. With this work, encouraging outcomes were achieved in this regard, obtaining similar results to other studies for the detection of larger lesions such as masses. Moreover, given the difficulty of visualizing the MCs, which are often spread over several slices, this work may have an important impact on the clinical analysis of DBT images.

Does artificial intelligence aid in the detection of different types of breast cancer?

Background

On mammography many cancers may be missed even in retrospect either due to the breast density, the small size of the tumor or the subtle signs of cancer that are imperceptible. We aimed to compare the sensitivity of artificial intelligence (AI) to that of digital mammography in the detection of different types of breast cancer. Also, the sensitivity of AI in picking up the different breast cancer morphologies namely mass, pathological calcifications, asymmetry, and distortion was assessed. Tissue biopsy and pathology were used as the standard reference. The study included 123 female patients with 134 proved carcinoma. All patients underwent digital mammogram (DM) examination scanned with artificial intelligence algorithm.

Results

AI achieved higher sensitivity than mammography in detecting malignant breast lesions. The sensitivity of AI was 96.6%, and false negative rate was 3.4%, while mammography sensitivity was 87.3% and false negative rate 12.7%. Our study showed AI performed better than mammography in detecting ductal carcinoma in situ and invasive lobular carcinoma with sensitivity (100% and 96.6%) vs (88.9% and 82.2%) respectively. AI was more sensitive to detect cancers presented with suspicious mass 95.2% vs 75%, suspicious calcifications 100% vs 86.5% and asymmetry and distortion 100% vs 84.6%, than mammography.

Conclusions

AI showed potential values to overcome mammographic limitations in the detection of breast cancer even those with challenging morphology as invasive lobular carcinoma, ductal carcinoma in situ, tubular carcinoma and micropapillary carcinoma.

Multimodal Prediction of Five-Year Breast Cancer Recurrence in Women Who Receive Neoadjuvant Chemotherapy

In current clinical practice, it is difficult to predict whether a patient receiving neoadjuvant chemotherapy (NAC) for breast cancer is likely to encounter recurrence after treatment and have the cancer recur locally in the breast or in other areas of the body. We explore the use of clinical history, immunohistochemical markers, and multiparametric magnetic resonance imaging (DCE, ADC, Dixon) to predict the risk of post-treatment recurrence within five years. We performed a retrospective study on a cohort of 1738 patients from Institut Curie and analyzed the data using classical machine learning, image processing, and deep learning. Our results demonstrate the ability to predict recurrence prior to NAC treatment initiation using each modality alone, and the possible improvement achieved by combining the modalities. When evaluated on holdout data, the multimodal model achieved an AUC of 0.75 (CI: 0.70, 0.80) and 0.57 specificity at 0.90 sensitivity. We then stratified the data based on known prognostic biomarkers. We found that our models can provide accurate recurrence predictions (AUC > 0.89) for specific groups of women under 50 years old with poor prognoses. A version of our method won second place at the BMMR2 Challenge, with a very small margin from being first, and was a standout from the other challenge entries.

Inferring pediatric knee skeletal maturity from MRI using deep learning

PURPOSE

Many children who undergo MR of the knee to evaluate traumatic injury may not undergo a separate dedicated evaluation of their skeletal maturity, and we wished to investigate how accurately skeletal maturity could be automatically inferred from knee MRI using deep learning to offer this additional information to clinicians.

MATERIALS AND METHODS

Retrospective data from 894 studies from 783 patients were obtained (mean age 13.1 years, 47% female). Coronal and sagittal sequences that were T1/PD-weighted were included and resized to 224 × 224 pixels. Data were divided into train (n = 673), tune (n = 48), and test (n = 173) sets, and children were separated across sets. The chronologic age was predicted using deep learning approaches based on a long short-term memory (LSTM) model, which took as input DenseNet-121-extracted features from all T1/PD coronal and sagittal slices. Each test case was manually assigned a bone age by two radiology residents using a reference atlas provided by Pennock and Bomar. The patient's age served as ground truth.

RESULTS

The error of the model's predictions for chronological age was not significantly different from that of radiology residents (model M.S.E. 1.30 vs. resident 0.99, paired t-test = 1.47, p = 0.14). Pearson correlation between model and resident prediction of chronologic age was 0.96 (p < 0.001).

CONCLUSION

A deep learning-based approach demonstrated ability to infer skeletal maturity from knee MR sequences that was not significantly different from resident performance and did so in less than 2% of the time required by a human expert. This may offer a method for automatically evaluating lower extremity skeletal maturity automatically as part of every MR examination.

Diagnostic Strategies for Breast Cancer Detection: From Image Generation to Classification Strategies Using Artificial Intelligence Algorithms

Breast cancer is one the main death causes for women worldwide, as 16% of the diagnosed malignant lesions worldwide are its consequence. In this sense, it is of paramount importance to diagnose these lesions in the earliest stage possible, in order to have the highest chances of survival. While there are several works that present selected topics in this area, none of them present a complete panorama, that is, from the image generation to its interpretation. This work presents a comprehensive state-of-the-art review of the image generation and processing techniques to detect Breast Cancer, where potential candidates for the image generation and processing are presented and discussed. Novel methodologies should consider the adroit integration of artificial intelligence-concepts and the categorical data to generate modern alternatives that can have the accuracy, precision and reliability expected to mitigate the misclassifications.

UK National Screening Committee's approach to reviewing evidence on artificial intelligence in breast cancer screening

Artificial intelligence (AI) could have the potential to accurately classify mammograms according to the presence or absence of radiological signs of breast cancer, replacing or supplementing human readers (radiologists). The UK National Screening Committee's assessments of the use of AI systems to examine screening mammograms continues to focus on maximising benefits and minimising harms to women screened, when deciding whether to recommend the implementation of AI into the Breast Screening Programme in the UK. Maintaining or improving programme specificity is important to minimise anxiety from false positive results. When considering cancer detection, AI test sensitivity alone is not sufficiently informative, and additional information on the spectrum of disease detected and interval cancers is crucial to better understand the benefits and harms of screening. Although large retrospective studies might provide useful evidence by directly comparing test accuracy and spectrum of disease detected between different AI systems and by population subgroup, most retrospective studies are biased due to differential verification (ie, the use of different reference standards to verify the target condition among study participants). Enriched, multiple-reader, multiple-case, test set laboratory studies are also biased due to the laboratory effect (ie, radiologists' performance in retrospective, laboratory, observer studies is substantially different to their performance in a clinical environment). Therefore, assessment of the effect of incorporating any AI system into the breast screening pathway in prospective studies is required as it will provide key evidence for the effect of the interaction of medical staff with AI, and the impact on women's outcomes.

Combining the strengths of radiologists and AI for breast cancer screening: a retrospective analysis

BACKGROUND

We propose a decision-referral approach for integrating artificial intelligence (AI) into the breast-cancer screening pathway, whereby the algorithm makes predictions on the basis of its quantification of uncertainty. Algorithmic assessments with high certainty are done automatically, whereas assessments with lower certainty are referred to the radiologist. This two-part AI system can triage normal mammography exams and provide post-hoc cancer detection to maintain a high degree of sensitivity. This study aimed to evaluate the performance of this AI system on sensitivity and specificity when used either as a standalone system or within a decision-referral approach, compared with the original radiologist decision.

METHODS

We used a retrospective dataset consisting of 1 193 197 full-field, digital mammography studies carried out between Jan 1, 2007, and Dec 31, 2020, from eight screening sites participating in the German national breast-cancer screening programme. We derived an internal-test dataset from six screening sites (1670 screen-detected cancers and 19 997 normal mammography exams), and an external-test dataset of breast cancer screening exams (2793 screen-detected cancers and 80 058 normal exams) from two additional screening sites to evaluate the performance of an AI algorithm on sensitivity and specificity when used either as a standalone system or within a decision-referral approach, compared with the original individual radiologist decision at the point-of-screen reading ahead of the consensus conference. Different configurations of the AI algorithm were evaluated. To account for the enrichment of the datasets caused by oversampling cancer cases, weights were applied to reflect the actual distribution of study types in the screening programme. Triaging performance was evaluated as the rate of exams correctly identified as normal. Sensitivity across clinically relevant subgroups, screening sites, and device manufacturers was compared between standalone AI, the radiologist, and decision referral. We present receiver operating characteristic (ROC) curves and area under the ROC (AUROC) to evaluate AI-system performance over its entire operating range. Comparison with radiologists and subgroup analysis was based on sensitivity and specificity at clinically relevant configurations.

FINDINGS

The exemplary configuration of the AI system in standalone mode achieved a sensitivity of 84·2% (95% CI 82·4-85·8) and a specificity of 89·5% (89·0-89·9) on internal-test data, and a sensitivity of 84·6% (83·3-85·9) and a specificity of 91·3% (91·1-91·5) on external-test data, but was less accurate than the average unaided radiologist. By contrast, the simulated decision-referral approach significantly improved upon radiologist sensitivity by 2·6 percentage points and specificity by 1·0 percentage points, corresponding to a triaging performance at 63·0% on the external dataset; the AUROC was 0·982 (95% CI 0·978-0·986) on the subset of studies assessed by AI, surpassing radiologist performance. The decision-referral approach also yielded significant increases in sensitivity for a number of clinically relevant subgroups, including subgroups of small lesion sizes and invasive carcinomas. Sensitivity of the decision-referral approach was consistent across the eight included screening sites and three device manufacturers.

INTERPRETATION

The decision-referral approach leverages the strengths of both the radiologist and AI, demonstrating improvements in sensitivity and specificity surpassing that of the individual radiologist and of the standalone AI system. This approach has the potential to improve the screening accuracy of radiologists, is adaptive to the requirements of screening, and could allow for the reduction of workload ahead of the consensus conference, without discarding the generalised knowledge of radiologists.

FUNDING

Vara.

Impact of artificial intelligence in breast cancer screening with mammography

Objectives

To demonstrate that radiologists, with the help of artificial intelligence (AI), are able to better classify screening mammograms into the correct breast imaging reporting and data system (BI-RADS) category, and as a secondary objective, to explore the impact of AI on cancer detection and mammogram interpretation time.

Methods

A multi-reader, multi-case study with cross-over design, was performed, including 314 mammograms. Twelve radiologists interpreted the examinations in two sessions delayed by a 4 weeks wash-out period with and without AI support. For each breast of each mammogram, they had to mark the most suspicious lesion (if any) and assign it with a forced BI-RADS category and a level of suspicion or “continuous BI-RADS 100”.

Cohen’s kappa correlation coefficient evaluating the inter-observer agreement for BI-RADS category per breast, and the area under the receiver operating characteristic curve (AUC), were used as metrics and analyzed.

Results

On average, the quadratic kappa coefficient increased significantly when using AI for all readers [κ = 0.549, 95% CI (0.528–0.571) without AI and κ = 0.626, 95% CI (0.607–0.6455) with AI]. AUC was significantly improved when using AI (0.74 vs 0.77, p = 0.004). Reading time was not significantly affected for all readers (106 s without AI and vs 102 s with AI; p = 0.754).

Conclusions

When using AI, radiologists were able to better assign mammograms with the correct BI-RADS category without slowing down the interpretation time.

Improved Productivity Using Deep Learning-assisted Reporting for Lumbar Spine MRI

Background Lumbar spine MRI studies are widely used for back pain assessment. Interpretation involves grading lumbar spinal stenosis, which is repetitive and time consuming. Deep learning (DL) could provide faster and more consistent interpretation. Purpose To assess the speed and interobserver agreement of radiologists for reporting lumbar spinal stenosis with and without DL assistance. Materials and Methods In this retrospective study, a DL model designed to assist radiologists in the interpretation of spinal canal, lateral recess, and neural foraminal stenoses on lumbar spine MRI scans was used. Randomly selected lumbar spine MRI studies obtained in patients with back pain who were 18 years and older over a 3-year period, from September 2015 to September 2018, were included in an internal test data set. Studies with instrumentation and scoliosis were excluded. Eight radiologists, each with 2-13 years of experience in spine MRI interpretation, reviewed studies with and without DL model assistance with a 1-month washout period. Time to diagnosis (in seconds) and interobserver agreement (using Gwet κ) were assessed for stenosis grading for each radiologist with and without the DL model and compared with test data set labels provided by an external musculoskeletal radiologist (with 32 years of experience) as the reference standard. Results Overall, 444 images in 25 patients (mean age, 51 years ± 20 [SD]; 14 women) were evaluated in a test data set. DL-assisted radiologists had a reduced interpretation time per spine MRI study, from a mean of 124-274 seconds (SD, 25-88 seconds) to 47-71 seconds (SD, 24-29 seconds) (P < .001). DL-assisted radiologists had either superior or equivalent interobserver agreement for all stenosis gradings compared with unassisted radiologists. DL-assisted general and in-training radiologists improved their interobserver agreement for four-class neural foraminal stenosis, with κ values of 0.71 and 0.70 (with DL) versus 0.39 and 0.39 (without DL), respectively (both P < .001). Conclusion Radiologists who were assisted by deep learning for interpretation of lumbar spinal stenosis on MRI scans showed a marked reduction in reporting time and superior or equivalent interobserver agreement for all stenosis gradings compared with radiologists who were unassisted by deep learning. © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by Hayashi in this issue.

The Economic Impact of AI on Breast Imaging

This article explores the development of computer-aided detection (CAD) and artificial or augmented intelligence (AI) for breast radiology examinations and describes the current applications of AI in breast imaging. Although radiologists in other subspecialties may be less familiar with the use of these technologies in their practices, CAD has been used in breast imaging for more than two decades, as mammography CAD programs have been commercially available in the United States since the late 1990s. Likewise, breast radiologists have seen payment for CAD in mammography and breast MRI evolve over time. With the rapid expansion of AI products in radiology in recent years, many new applications for these technologies in breast imaging have emerged. This article outlines the current state of reimbursement for breast radiology AI algorithms within the traditional fee-for-service model used by Medicare and commercial insurers as well as potential future payment pathways. In addition, the inherent challenges of employing the existing payment framework in the United States to AI programs in radiology are detailed for the reader. This article aims to give breast radiologists a better understanding of how AI will be reimbursed as they seek to further incorporate these emerging technologies into their practices to advance patient care and improve workflow efficiency.

Early detection and classification of abnormality in prior mammograms using image-to-image translation and YOLO techniques

BACKGROUND AND OBJECTIVE

Computer-aided-detection (CAD) systems have been developed to assist radiologists on finding suspicious lesions in mammogram. Deep Learning technology have recently succeeded to increase the chance of recognizing abnormality at an early stage in order to avoid unnecessary biopsies and decrease the mortality rate. In this study, we investigated the effectiveness of an end-to-end fusion model based on You-Only-Look-Once (YOLO) architecture, to simultaneously detect and classify suspicious breast lesions on digital mammograms. Four categories of cases were included: Mass, Calcification, Architectural Distortions, and Normal from a private digital mammographic database including 413 cases. For all cases, Prior mammograms (typically scanned 1 year before) were all reported as Normal, while Current mammograms were diagnosed as cancerous (confirmed by biopsies) or healthy.

METHODS

We propose to apply the YOLO-based fusion model to the Current mammograms for breast lesions detection and classification. Then apply the same model retrospectively to synthetic mammograms for an early cancer prediction, where the synthetic mammograms were generated from the Prior mammograms by using the image-to-image translation models, CycleGAN and Pix2Pix.

RESULTS

Evaluation results showed that our methodology could significantly detect and classify breast lesions on Current mammograms with a highest rate of 93% ± 0.118 for Mass lesions, 88% ± 0.09 for Calcification lesions, and 95% ± 0.06 for Architectural Distortion lesions. In addition, we reported evaluation results on Prior mammograms with a highest rate of 36% ± 0.01 for Mass lesions, 14% ± 0.01 for Calcification lesions, and 50% ± 0.02 for Architectural Distortion lesions. Normal mammograms were accordingly classified with an accuracy rate of 92% ± 0.09 and 90% ± 0.06 respectively on Current and Prior exams.

CONCLUSIONS

Our proposed framework was first developed to help detecting and identifying suspicious breast lesions in X-ray mammograms on their Current screening. The work was also suggested to reduce the temporal changes between pairs of Prior and follow-up screenings for early predicting the location and type of abnormalities in Prior mammogram screening. The paper presented a CAD method to assist doctors and experts to identify the risk of breast cancer presence. Overall, the proposed CAD method incorporates the advances of image processing, deep learning and image-to-image translation for a biomedical application.

Deep learning in breast imaging

Millions of breast imaging exams are performed each year in an effort to reduce the morbidity and mortality of breast cancer. Breast imaging exams are performed for cancer screening, diagnostic work-up of suspicious findings, evaluating extent of disease in recently diagnosed breast cancer patients, and determining treatment response. Yet, the interpretation of breast imaging can be subjective, tedious, time-consuming, and prone to human error. Retrospective and small reader studies suggest that deep learning (DL) has great potential to perform medical imaging tasks at or above human-level performance, and may be used to automate aspects of the breast cancer screening process, improve cancer detection rates, decrease unnecessary callbacks and biopsies, optimize patient risk assessment, and open up new possibilities for disease prognostication. Prospective trials are urgently needed to validate these proposed tools, paving the way for real-world clinical use. New regulatory frameworks must also be developed to address the unique ethical, medicolegal, and quality control issues that DL algorithms present. In this article, we review the basics of DL, describe recent DL breast imaging applications including cancer detection and risk prediction, and discuss the challenges and future directions of artificial intelligence-based systems in the field of breast cancer.

Evaluation of the Combination of Artificial Intelligence and Radiologist Assessments to Interpret Malignant Architectural Distortion on Mammography

Purpose

To compare the mammographic malignant architectural distortion (AD) detection performance of radiologists who read mammographic examinations unaided versus those who read these examinations with the support of artificial intelligence (AI) systems.

Material and Methods

This retrospective case-control study was based on a double-reading of clinical mammograms between January 2011 and December 2016 at a large tertiary academic medical center. The study included 177 malignant and 90 benign architectural distortion (AD) patients. The model was built based on the ResNeXt-50 network. Algorithms used deep learning convolutional neural networks, feature classifiers, image analysis algorithms to depict AD and output a score that translated to malignant. The accuracy for malignant AD detection was evaluated using area under the curve (AUC).

Results

The overall AUC was 0.733 (95% CI, 0.673-0.792) for Reader First-1, 0.652 (95% CI, 0.586-0.717) for Reader First-2, and 0.655 (95% CI, 0.590-0.719) for Reader First-3. and the overall AUCs for Reader Second-1, 2, 3 were 0.875 (95% CI, 0.830-0.919), 0.882 (95% CI, 0.839-0.926), 0.884 (95% CI, 0.841-0.927),respectively. The AUCs for all the reader-second radiologists were significantly higher than those for all the reader-first radiologists (Reader First-1 vs. Reader Second-1, P= 0.004). The overall AUC was 0.792 (95% CI, 0.660-0.925) for AI algorithms. The combination assessment of AI algorithms and Reader First-1 achieved an AUC of 0.880 (95% CI, 0.793-0.968), increased than the Reader First-1 alone and AI algorithms alone. AI algorithms alone achieved a specificity of 61.1% and a sensitivity of 80.6%. The specificity for Reader First-1 was 55.5%, and the sensitivity was 86.1%. The results of the combined assessment of AI and Reader First-1 showed a specificity of 72.7% and sensitivity of 91.7%. The performance showed significant improvements compared with AI alone (p<0.001) as well as the reader first-1 alone (p=0.006).

Conclusion

While the single AI algorithm did not outperform radiologists, an ensemble of AI algorithms combined with junior radiologist assessments were found to improve the overall accuracy. This study underscores the potential of using machine learning methods to enhance mammography interpretation, especially in remote areas and primary hospitals.

Anomaly Detection of Calcifications in Mammography Based on 11,000 Negative Cases

In mammography, calcifications are one of the most common signs of breast cancer. Detection of such lesions is an active area of research for computer-aided diagnosis and machine learning algorithms. Due to limited numbers of positive cases, many supervised detection models suffer from overfitting and fail to generalize. We present a one-class, semi-supervised framework using a deep convolutional autoencoder trained with over 50,000 images from 11,000 negative-only cases. Since the model learned from only normal breast parenchymal features, calcifications produced large signals when comparing the residuals between input and reconstruction output images. As a key advancement, a structural dissimilarity index was used to suppress non-structural noises. Our selected model achieved pixel-based AUROC of 0.959 and AUPRC of 0.676 during validation, where calcification masks were defined in a semi-automated process. Although not trained directly on any cancers, detection performance of calcification lesions on 1,883 testing images (645 malignant and 1238 negative) achieved 75% sensitivity at 2.5 false positives per image. Performance plateaued early when trained with only a fraction of the cases, and greater model complexity or a larger dataset did not improve performance. This study demonstrates the potential of this anomaly detection approach to detect mammographic calcifications in a semi-supervised manner with efficient use of a small number of labeled images, and may facilitate new clinical applications such as computer-aided triage and quality improvement.

External Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review

Purpose

To assess generalizability of published deep learning (DL) algorithms for radiologic diagnosis.

Materials and Methods

In this systematic review, the PubMed database was searched for peer-reviewed studies of DL algorithms for image-based radiologic diagnosis that included external validation, published from January 1, 2015, through April 1, 2021. Studies using nonimaging features or incorporating non-DL methods for feature extraction or classification were excluded. Two reviewers independently evaluated studies for inclusion, and any discrepancies were resolved by consensus. Internal and external performance measures and pertinent study characteristics were extracted, and relationships among these data were examined using nonparametric statistics.

Results

Eighty-three studies reporting 86 algorithms were included. The vast majority (70 of 86, 81%) reported at least some decrease in external performance compared with internal performance, with nearly half (42 of 86, 49%) reporting at least a modest decrease (≥0.05 on the unit scale) and nearly a quarter (21 of 86, 24%) reporting a substantial decrease (≥0.10 on the unit scale). No study characteristics were found to be associated with the difference between internal and external performance.

Conclusion

Among published external validation studies of DL algorithms for image-based radiologic diagnosis, the vast majority demonstrated diminished algorithm performance on the external dataset, with some reporting a substantial performance decrease.Keywords: Meta-Analysis, Computer Applications-Detection/Diagnosis, Neural Networks, Computer Applications-General (Informatics), Epidemiology, Technology Assessment, Diagnosis, Informatics Supplemental material is available for this article. © RSNA, 2022.

Artificial intelligence: Advances and new frontiers in medical imaging

Artificial intelligence (AI) has been entwined with the field of radiology ever since digital imaging began replacing films over half a century ago. These algorithms, ranging from simplistic speech-to-text dictation programs to automated interpretation neural networks, have continuously sought to revolutionize medical imaging. With the number of imaging studies outpacing the amount of trained of readers, AI has been implemented to streamline workflow efficiency and provide quantitative, standardized interpretation. AI relies on massive amounts of data for its algorithms to function, and with the wide-spread adoption of Picture Archiving and Communication Systems (PACS), imaging data is accumulating rapidly. Current AI algorithms using machine-learning technology, or computer aided-detection, have been able to successfully pool this data for clinical use, although the scope of these algorithms remains narrow. Many systems have been developed to assist the workflow of the radiologist through PACS optimization and imaging study triage, however interpretation has generally remained a human responsibility for now. In this review article, we will summarize the current successes and limitations of AI in radiology, and explore the exciting prospects that deep-learning technology offers for the future.