Assessing the stand-alone sensitivity of computer-aided detection with cancer cases from the Digital Mammographic Imaging Screening Trial

A Semiautonomous Deep Learning System to Reduce False Positives in Screening Mammography

Purpose To evaluate the ability of a semiautonomous artificial intelligence (AI) model to identify screening mammograms not suspicious for breast cancer and reduce the number of false-positive examinations. Materials and Methods The deep learning algorithm was trained using 123 248 two-dimensional digital mammograms (6161 cancers) and a retrospective study was performed on three nonoverlapping datasets of 14 831 screening mammography examinations (1026 cancers) from two U.S. institutions and one U.K. institution (2008-2017). The stand-alone performance of humans and AI was compared. Human plus AI performance was simulated to examine reductions in the cancer detection rate, number of examinations, false-positive callbacks, and benign biopsies. Metrics were adjusted to mimic the natural distribution of a screening population, and bootstrapped CIs and P values were calculated. Results Retrospective evaluation on all datasets showed minimal changes to the cancer detection rate with use of the AI device (noninferiority margin of 0.25 cancers per 1000 examinations: U.S. dataset 1, P = .02; U.S. dataset 2, P < .001; U.K. dataset, P < .001). On U.S. dataset 1 (11 592 mammograms; 101 cancers; 3810 female patients; mean age, 57.3 years ± 10.0 [SD]), the device reduced screening examinations requiring radiologist interpretation by 41.6% (95% CI: 40.6%, 42.4%; P < .001), diagnostic examinations callbacks by 31.1% (95% CI: 28.7%, 33.4%; P < .001), and benign needle biopsies by 7.4% (95% CI: 4.1%, 12.4%; P < .001). U.S. dataset 2 (1362 mammograms; 330 cancers; 1293 female patients; mean age, 55.4 years ± 10.5) was reduced by 19.5% (95% CI: 16.9%, 22.1%; P < .001), 11.9% (95% CI: 8.6%, 15.7%; P < .001), and 6.5% (95% CI: 0.0%, 19.0%; P = .08), respectively. The U.K. dataset (1877 mammograms; 595 cancers; 1491 female patients; mean age, 63.5 years ± 7.1) was reduced by 36.8% (95% CI: 34.4%, 39.7%; P < .001), 17.1% (95% CI: 5.9%, 30.1%: P < .001), and 5.9% (95% CI: 2.9%, 11.5%; P < .001), respectively. Conclusion This work demonstrates the potential of a semiautonomous breast cancer screening system to reduce false positives, unnecessary procedures, patient anxiety, and medical expenses. Keywords: Artificial Intelligence, Semiautonomous Deep Learning, Breast Cancer, Screening Mammography Supplemental material is available for this article. Published under a CC BY 4.0 license.

The Systematic Review of Artificial Intelligence Applications in Breast Cancer Diagnosis

Several studies have demonstrated the value of artificial intelligence (AI) applications in breast cancer diagnosis. The systematic review of AI applications in breast cancer diagnosis includes several studies that compare breast cancer diagnosis and AI. However, they lack systematization, and each study appears to be conducted uniquely. The purpose and contributions of this study are to offer elaborative knowledge on the applications of AI in the diagnosis of breast cancer through citation analysis in order to categorize the main area of specialization that attracts the attention of the academic community, as well as thematic issue analysis to identify the species being researched in each category. In this study, a total number of 17,900 studies addressing breast cancer and AI published between 2012 and 2022 were obtained from these databases: IEEE, Embase: Excerpta Medica Database Guide-Ovid, PubMed, Springer, Web of Science, and Google Scholar. We applied inclusion and exclusion criteria to the search; 36 studies were identified. The vast majority of AI applications used classification models for the prediction of breast cancer. Howbeit, accuracy (99%) has the highest number of performance metrics, followed by specificity (98%) and area under the curve (0.95). Additionally, the Convolutional Neural Network (CNN) was the best model of choice in several studies. This study shows that the quantity and caliber of studies that use AI applications in breast cancer diagnosis will continue to rise annually. As a result, AI-based applications are viewed as a supplement to doctors' clinical reasoning, with the ultimate goal of providing quality healthcare that is both affordable and accessible to everyone worldwide.

Are artificial intelligence systems useful in breast cancer screening programs?

Population-based breast cancer screening programs are efficacious in reducing the mortality due to breast cancer. These programs use mammography to screen the women who are invited to participate. Digital mammography makes it possible to develop computer-assisted diagnosis (CAD) systems that promise to reduce the workload of radiologists participating in screening programs. However, various studies have shown that CAD results in a high rate of false positive diagnoses. Systems based on artificial intelligence are being more widely implemented, and studies have shown that these systems have better diagnostic performance than traditional CAD systems. This article explains the fundamentals of artificial intelligence systems and an overview of possible applications of these systems within the framework of breast cancer screening programs.

Palpable Breast Lump Triage by Minimally Trained Operators in Mexico Using Computer-Assisted Diagnosis and Low-Cost Ultrasound

Purpose

In low- to middle-income countries (LMICs), most breast cancers present as palpable lumps; however, most palpable lumps are benign. We have developed artificial intelligence–based computer-assisted diagnosis (CADx) for an existing low-cost portable ultrasound system to triage which lumps need further evaluation and which are clearly benign. This pilot study was conducted to demonstrate that this approach can be successfully used by minimally trained health care workers in an LMIC country.

Patients and Methods

We recruited and trained three nonradiologist health care workers to participate in an institutional review board–approved, Health Insurance Portability and Accountability Act–compliant pilot study in Jalisco, Mexico, to determine whether they could use portable ultrasound (GE Vscan Dual Probe) to acquire images of palpable breast lumps of adequate quality for accurate computer analysis. Images from 32 women with 32 breast masses were then analyzed with a triage-CADx system, generating an output of benign or suspicious (biopsy recommended). Triage-CADx outputs were compared with radiologist readings.

Results

The nonradiologists were able to acquire adequate images. Triage by the CADx software was as accurate as assessment by specialist radiologists, with two (100%) of two cancers considered suspicious and 30 (100%) of 30 benign lesions classified as benign.

Conclusion

A portable ultrasound system with CADx software can be successfully used by first-level health care workers to triage palpable breast lumps. These results open up the possibility of implementing practical, cost-effective triage of palpable breast lumps, ensuring that scarce resources can be dedicated to suspicious lesions requiring further workup.

Artificial Intelligence for Mammography and Digital Breast Tomosynthesis: Current Concepts and Future Perspectives

Although computer-aided diagnosis (CAD) is widely used in mammography, conventional CAD programs that use prompts to indicate potential cancers on the mammograms have not led to an improvement in diagnostic accuracy. Because of the advances in machine learning, especially with use of deep (multilayered) convolutional neural networks, artificial intelligence has undergone a transformation that has improved the quality of the predictions of the models. Recently, such deep learning algorithms have been applied to mammography and digital breast tomosynthesis (DBT). In this review, the authors explain how deep learning works in the context of mammography and DBT and define the important technical challenges. Subsequently, they discuss the current status and future perspectives of artificial intelligence-based clinical applications for mammography, DBT, and radiomics. Available algorithms are advanced and approach the performance of radiologists-especially for cancer detection and risk prediction at mammography. However, clinical validation is largely lacking, and it is not clear how the power of deep learning should be used to optimize practice. Further development of deep learning models is necessary for DBT, and this requires collection of larger databases. It is expected that deep learning will eventually have an important role in DBT, including the generation of synthetic images.

Influence of Monoenergetic Images at Different Energy Levels in Dual-Energy Spectral CT on the Accuracy of Computer-Aided Detection for Pulmonary Embolism

PURPOSE

To investigate the influence of monoenergetic images of different energy levels in spectral computed tomography (CT) on the accuracy of computer aided detection (CAD) for pulmonary embolism (PE).

MATERIALS AND METHODS

CT images of 20 PE patients who underwent spectral CT pulmonary angiography were retrospectively analyzed. Nine sets of monochromatic images from 40 to 80 keV at 5 keV interval were reconstructed and then independently analyzed for detecting PE using a commercially available CAD software. Two experienced radiologists reviewed all images and recorded the number of emboli manually, which was used as the reference standard. The CAD findings for the number of PE at different energies were compared with the reference standard to determine the number of true positives and false positives with CAD and to calculate the sensitivity and false positive rate at different energies.

RESULT

There were 120 true emboli. The total numbers of CAD-detected PE at 40-80 keV were 48, 67, 63, 87, 106, 115, 138, 157, and 226. Images at low energies had low sensitivities and low false positive rates; images at high energies had high sensitivities and high false positive rates. At 60 keV and 65 keV, CAD achieved sensitivity at 81.67% and 84.17%, respectively and false positive rate at 7.55% and 12.17%, respectively to provide the optimum combination of high sensitivity and low false positive rate.

CONCLUSION

Monochromatic images of different energies in dual-energy spectral CT affect the accuracy of CAD for PE. The combination of CAD with images at 60-65 keV provides the optimum combination of high sensitivity and low false positive rate in detecting PE.

A review of computer aided detection in mammography

Breast screening with mammography is widely recognized as the most effective method of detecting early breast cancer and has consistently demonstrated a 20-40% decrease in mortality among screened women. Despite this, the sensitivity of mammography ranges between 70 and 90%. Computer aided detection (CAD) is an artificial intelligence (AI) technique that utilizes pattern recognition to highlight suspicious features on imaging and marks them for the radiologist to review and interpret. It aims to decrease oversights made by interpreting radiologists. Here we review the efficacy of CAD and potential future directions.

Reducing false positives of microcalcification detection systems by removal of breast arterial calcifications

PURPOSE

In the past decades, computer-aided detection (CADe) systems have been developed to aid screening radiologists in the detection of malignant microcalcifications. These systems are useful to avoid perceptual oversights and can increase the radiologists' detection rate. However, due to the high number of false positives marked by these CADe systems, they are not yet suitable as an independent reader. Breast arterial calcifications (BACs) are one of the most frequent false positives marked by CADe systems. In this study, a method is proposed for the elimination of BACs as positive findings. Removal of these false positives will increase the performance of the CADe system in finding malignant microcalcifications.

METHODS

A multistage method is proposed for the removal of BAC findings. The first stage consists of a microcalcification candidate selection, segmentation and grouping of the microcalcifications, and classification to remove obvious false positives. In the second stage, a case-based selection is applied where cases are selected which contain BACs. In the final stage, BACs are removed from the selected cases. The BACs removal stage consists of a GentleBoost classifier trained on microcalcification features describing their shape, topology, and texture. Additionally, novel features are introduced to discriminate BACs from other positive findings.

RESULTS

The CADe system was evaluated with and without BACs removal. Here, both systems were applied on a validation set containing 1088 cases of which 95 cases contained malignant microcalcifications. After bootstrapping, free-response receiver operating characteristics and receiver operating characteristics analyses were carried out. Performance between the two systems was compared at 0.98 and 0.95 specificity. At a specificity of 0.98, the sensitivity increased from 37% to 52% and the sensitivity increased from 62% up to 76% at a specificity of 0.95. Partial areas under the curve in the specificity range of 0.8-1.0 were significantly different between the system without BACs removal and the system with BACs removal, 0.129 ± 0.009 versus 0.144 ± 0.008 (p<0.05), respectively. Additionally, the sensitivity at one false positive per 50 cases and one false positive per 25 cases increased as well, 37% versus 51% (p<0.05) and 58% versus 67% (p<0.05) sensitivity, respectively. Additionally, the CADe system with BACs removal reduces the number of false positives per case by 29% on average. The same sensitivity at one false positive per 50 cases in the CADe system without BACs removal can be achieved at one false positive per 80 cases in the CADe system with BACs removal.

CONCLUSIONS

By using dedicated algorithms to detect and remove breast arterial calcifications, the performance of CADe systems can be improved, in particular, at false positive rates representative for operating points used in screening.

Using breast radiographers' reports as a second opinion for radiologists' readings of microcalcifications in digital mammography

OBJECTIVE

The aim of this study was to investigate a practical method for incorporating radiographers' reports with radiologists' readings of digital mammograms.

METHODS

This simulation study was conducted using data from a free-response receiver operating characteristic observer study obtained with 75 cases (25 malignant, 25 benign and 25 normal cases) of digital mammograms. Each of the rating scores obtained by six breast radiographers was utilized as a second opinion for four radiologists' readings with the radiographers' reports. A logical "OR" operation with various criteria settings was simulated for deciding an appropriate method to select a radiographer's report in all combinations of radiologists and radiographers. The average figure of merit (FOM) of the radiologists' performances was statistically analysed using a jackknife procedure (JAFROC) to verify the clinical utility of using radiographers' reports.

RESULTS

Potential improvement of the average FOM of the radiologists' performances for identifying malignant microcalcifications could be expected when using radiographers' reports as a second opinion. When the threshold value of 2.6 in Breast Imaging-Reporting and Data System (BI-RADS®) assessment was applied to adopt/reject a radiographer's report, FOMs of radiologists' performances were further improved.

CONCLUSION

When using breast radiographers' reports as a second opinion, radiologists' performances potentially improved when reading digital mammograms. It could be anticipated that radiologists' performances were improved further by setting a threshold value on the BI-RADS assessment provided by the radiographers.

ADVANCES IN KNOWLEDGE

For the effective use of a radiographer's report as a second opinion, radiographers' rating scores and its criteria setting for adoption/rejection would be necessary.

Impact of computer-aided detection systems on radiologist accuracy with digital mammography

OBJECTIVE

The purpose of this study was to assess the impact of computer-aided detection (CAD) systems on the performance of radiologists with digital mammograms acquired during the Digital Mammographic Imaging Screening Trial (DMIST).

MATERIALS AND METHODS

Only those DMIST cases with proven cancer status by biopsy or 1-year follow-up that had available digital images were included in this multireader, multicase ROC study. Two commercially available CAD systems for digital mammography were used: iCAD SecondLook, version 1.4; and R2 ImageChecker Cenova, version 1.0. Fourteen radiologists interpreted, without and with CAD, a set of 300 cases (150 cancer, 150 benign or normal) on the iCAD SecondLook system, and 15 radiologists interpreted a different set of 300 cases (150 cancer, 150 benign or normal) on the R2 ImageChecker Cenova system.

RESULTS

The average AUC was 0.71 (95% CI, 0.66-0.76) without and 0.72 (95% CI, 0.67-0.77) with the iCAD system (p = 0.07). Similarly, the average AUC was 0.71 (95% CI, 0.66-0.76) without and 0.72 (95% CI 0.67-0.77) with the R2 system (p = 0.08). Sensitivity and specificity differences without and with CAD for both systems also were not significant.

CONCLUSION

Radiologists in our studies rarely changed their diagnostic decisions after the addition of CAD. The application of CAD had no statistically significant effect on radiologist AUC, sensitivity, or specificity performance with digital mammograms from DMIST.

CADe for early detection of breast cancer-current status and why we need to continue to explore new approaches

The authors describe where we are in terms of using computer-aided detection (CADe) systems during clinical mammographic interpretations, what are the issues that we face, and why they believe that, despite disappointment in terms of verified added value when it comes to detection of soft tissue abnormalities, we need to continue to explore new approaches to improving CADe-alone performance levels and, more important perhaps, new approaches to optimal communication of CADe-generated information.

Effect of radiologists' diagnostic work-up volume on interpretive performance

PURPOSE

To examine radiologists' screening performance in relation to the number of diagnostic work-ups performed after abnormal findings are discovered at screening mammography by the same radiologist or by different radiologists.

MATERIALS AND METHODS

In an institutional review board-approved HIPAA-compliant study, the authors linked 651 671 screening mammograms interpreted from 2002 to 2006 by 96 radiologists in the Breast Cancer Surveillance Consortium to cancer registries (standard of reference) to evaluate the performance of screening mammography (sensitivity, false-positive rate [ FPR false-positive rate ], and cancer detection rate [ CDR cancer detection rate ]). Logistic regression was used to assess the association between the volume of recalled screening mammograms ("own" mammograms, where the radiologist who interpreted the diagnostic image was the same radiologist who had interpreted the screening image, and "any" mammograms, where the radiologist who interpreted the diagnostic image may or may not have been the radiologist who interpreted the screening image) and screening performance and whether the association between total annual volume and performance differed according to the volume of diagnostic work-up.

RESULTS

Annually, 38% of radiologists performed the diagnostic work-up for 25 or fewer of their own recalled screening mammograms, 24% performed the work-up for 0-50, and 39% performed the work-up for more than 50. For the work-up of recalled screening mammograms from any radiologist, 24% of radiologists performed the work-up for 0-50 mammograms, 32% performed the work-up for 51-125, and 44% performed the work-up for more than 125. With increasing numbers of radiologist work-ups for their own recalled mammograms, the sensitivity (P = .039), FPR false-positive rate (P = .004), and CDR cancer detection rate (P < .001) of screening mammography increased, yielding a stepped increase in women recalled per cancer detected from 17.4 for 25 or fewer mammograms to 24.6 for more than 50 mammograms. Increases in work-ups for any radiologist yielded significant increases in FPR false-positive rate (P = .011) and CDR cancer detection rate (P = .001) and a nonsignificant increase in sensitivity (P = .15). Radiologists with a lower annual volume of any work-ups had consistently lower FPR false-positive rate , sensitivity, and CDR cancer detection rate at all annual interpretive volumes.

CONCLUSION

These findings support the hypothesis that radiologists may improve their screening performance by performing the diagnostic work-up for their own recalled screening mammograms and directly receiving feedback afforded by means of the outcomes associated with their initial decision to recall. Arranging for radiologists to work up a minimum number of their own recalled cases could improve screening performance but would need systems to facilitate this workflow.