Lessons learned from independent external validation of an AI tool to detect breast cancer using a representative UK data set

OBJECTIVE

To pilot a process for the independent external validation of an artificial intelligence (AI) tool to detect breast cancer using data from the NHS breast screening programme (NHSBSP).

METHODS

A representative data set of mammography images from 26,000 women attending 2 NHS screening centres, and an enriched data set of 2054 positive cases were used from the OPTIMAM image database. The use case of the AI tool was the replacement of the first or second human reader. The performance of the AI tool was compared to that of human readers in the NHSBSP.

RESULTS

Recommendations for future external validations of AI tools to detect breast cancer are provided. The tool recalled different breast cancers to the human readers. This study showed the importance of testing AI tools on all types of cases (including non-standard) and the clarity of any warning messages. The acceptable difference in sensitivity and specificity between the AI tool and human readers should be determined. Any information vital for the clinical application should be a required output for the AI tool. It is recommended that the interaction of radiologists with the AI tool, and the effect of the AI tool on arbitration be investigated prior to clinical use.

CONCLUSION

This pilot demonstrated several lessons for future independent external validation of AI tools for breast cancer detection.

ADVANCES IN KNOWLEDGE

Knowledge has been gained towards best practice procedures for performing independent external validations of AI tools for the detection of breast cancer using data from the NHS Breast Screening Programme.

Virtual clinical trial to compare cancer detection using combinations of 2D mammography, digital breast tomosynthesis and synthetic 2D imaging

OBJECTIVES

This study was designed to compare the detection of subtle lesions (calcification clusters or masses) when using the combination of digital breast tomosynthesis (DBT) and synthetic mammography (SM) with digital mammography (DM) alone or combined with DBT.

METHODS

A set of 166 cases without cancer was acquired on a DBT mammography system. Realistic subtle calcification clusters and masses in the DM images and DBT planes were digitally inserted into 104 of the acquired cases. Three study arms were created: DM alone, DM with DBT and SM with DBT. Five mammographic readers located the centre of any lesion within the images that should be recalled for further investigation and graded their suspiciousness. A JAFROC figure of merit (FoM) and lesion detection fraction (LDF) were calculated for each study arm. The visibility of the lesions in the DBT images was compared with SM and DM images.

RESULTS

For calcification clusters, there were no significant differences (p > 0.075) in FoM or LDF. For masses, the FoM and LDF were significantly improved in the arms using DBT compared to DM alone (p < 0.001). On average, both calcification clusters and masses were more visible on DBT than on DM and SM images.

CONCLUSIONS

This study demonstrated that masses were detected better with DBT than with DM alone and there was no significant difference (p = 0.075) in LDF between DM&DBT and SM&DBT for calcifications clusters. Our results support previous studies that it may be acceptable to not acquire digital mammography alongside tomosynthesis for subtle calcification clusters and ill-defined masses.

KEY POINTS

• The detection of masses was significantly better using DBT than with digital mammography alone. • The detection of calcification clusters was not significantly different between digital mammography and synthetic 2D images combined with tomosynthesis. • Our results support previous studies that it may be acceptable to not acquire digital mammography alongside tomosynthesis for subtle calcification clusters and ill-defined masses for the imaging technology used.

Effect of glandularity on the detection of simulated cancers in planar, tomosynthesis, and synthetic 2D imaging of the breast using a hybrid virtual clinical trial

PURPOSE

The purpose of this study was to measure the threshold diameter of calcifications and masses for 2D imaging, digital breast tomosynthesis (DBT), and synthetic 2D images, for a range of breast glandularities. This study shows the limits of detection for each of the technologies and the strengths and weaknesses of each in terms of visualizing the radiological features of small cancers.

METHODS

Mathematical voxel breast phantoms with glandularities by volume of 9%, 18%, and 30% with a thickness of 53 mm were created. Simulated ill-defined masses and calcification clusters with a range of diameters were inserted into some of these breast models. The imaging characteristics of a Siemens Inspiration X-ray system were measured for a 29 kV, tungsten/rhodium anode/filter combination. Ray tracing through the breast models was undertaken to create simulated 2D and DBT projection images. These were then modified to adjust the image sharpness, and to add scatter and noise. The mean glandular doses for the images were 1.43, 1.47, and 1.47 mGy for 2D and 1.92, 1.97, and 1.98 mGy for DBT for the three glandularities. The resultant images were processed to create 2D, DBT planes and synthetic 2D images. Patches of the images with or without a simulated lesion were extracted, and used in a four-alternative forced choice study to measure the threshold diameters for each imaging mode, lesion type, and glandularity. The study was undertaken by six physicists.

RESULTS

The threshold diameters of the lesions were 6.2, 4.9, and 6.7 mm (masses) and 225, 370, and 399 μm, (calcifications) for 2D, DBT, and synthetic 2D, respectively, for a breast glandularity of 18%. The threshold diameter of ill-defined masses is significantly smaller for DBT than for both 2D (p≤0.006) and synthetic 2D (p≤0.012) for all glandularities. Glandularity has a significant effect on the threshold diameter of masses, even for DBT where there is reduced background structure in the images. The calcification threshold diameters for 2D images were significantly smaller than for DBT and synthetic 2D for all glandularities. There were few significant differences for the threshold diameter of calcifications between glandularities, indicating that the background structure has little effect on the detection of calcifications. We measured larger but nonsignificant differences in the threshold diameters for synthetic 2D imaging than for 2D imaging for masses in the 9% (p = 0.059) and 18% (p = 0.19) glandularities. The threshold diameters for synthetic 2D imaging were larger than for 2D imaging for calcifications (p < 0.001) for all glandularities.

CONCLUSIONS

We have shown that glandularity has only a small effect on the detection of calcifications, but the threshold diameter of masses was significantly larger for higher glandularity for all of the modalities tested. We measured nonsignificantly larger threshold diameters for synthetic 2D imaging than for 2D imaging for masses at the 9% (p = 0.059) and 18% (p = 0.19) glandularities and significantly larger diameters for calcifications (p < 0.001) for all glandularities. The lesions simulated were very subtle and further work is required to examine the clinical effect of not seeing the smallest calcifications in clusters.

Quantitative breast density analysis to predict interval and node-positive cancers in pursuit of improved screening protocols: a case-control study

BACKGROUND

This study investigates whether quantitative breast density (BD) serves as an imaging biomarker for more intensive breast cancer screening by predicting interval, and node-positive cancers.

METHODS

This case-control study of 1204 women aged 47-73 includes 599 cancer cases (302 screen-detected, 297 interval; 239 node-positive, 360 node-negative) and 605 controls. Automated BD software calculated fibroglandular volume (FGV), volumetric breast density (VBD) and density grade (DG). A radiologist assessed BD using a visual analogue scale (VAS) from 0 to 100. Logistic regression and area under the receiver operating characteristic curves (AUC) determined whether BD could predict mode of detection (screen-detected or interval); node-negative cancers; node-positive cancers, and all cancers vs. controls.

RESULTS

FGV, VBD, VAS, and DG all discriminated interval cancers (all p < 0.01) from controls. Only FGV-quartile discriminated screen-detected cancers (p < 0.01). Based on AUC, FGV discriminated all cancer types better than VBD or VAS. FGV showed a significantly greater discrimination of interval cancers, AUC = 0.65, than of screen-detected cancers, AUC = 0.61 (p < 0.01) as did VBD (0.63 and 0.53, respectively, p < 0.001).

CONCLUSION

FGV, VBD, VAS and DG discriminate interval cancers from controls, reflecting some masking risk. Only FGV discriminates screen-detected cancers perhaps adding a unique component of breast cancer risk.

OPTIMAM Mammography Image Database: A Large-Scale Resource of Mammography Images and Clinical Data

Supplemental material is available for this article.

Mammography cancer detection: comparison of single 8MP and pair of 5MP reporting monitors

OBJECTIVE:

To compare breast cancer detection using a single 8MP display with using a standard pair of 5MP monitors.

METHODS:

An observer study was carried out in which mammograms were read using full field views only, and again with the additional use of magnified quadrant views. Seven observers read 300 cases, one view per breast, using each display type. Cases comprised 100 normal cases and 200 cases with cancers of subtle or very subtle appearance: 100 with malignant calcification clusters and 100 with non-calcified lesions. JAFROC software was used to analyse the results.

RESULTS:

When mammograms were viewed full field only, observers performed better (p = 0.050) in detecting malignant calcification clusters when using the pair of 5MP monitors compared with a single 8MP monitor. This result became non-significant when results were generalised to a population of readers. Performance in detecting calcification clusters was improved by using quadrant view in addition to full field view when using either the pair of 5MP monitors or the 8MP monitor. There was no significant difference in detection of all types of cancer between the pair of 5MP monitors and the 8MP monitor when quadrant zoom was used.

CONCLUSION:

Providing quadrant view is used in addition to full field view, there is no significant difference in cancer detection between the 8MP monitor and the pair of 5MP monitors.

ADVANCES IN KNOWLEDGE:

Effect of magnification on the detectability of subtle malignant calcification clusters in breast screening.

Image processing can cause some malignant soft-tissue lesions to be missed in digital mammography images

AIM

To investigate the effect of image processing on cancer detection in mammography.

METHODS AND MATERIALS

An observer study was performed using 349 digital mammography images of women with normal breasts, calcification clusters, or soft-tissue lesions including 191 subtle cancers. Images underwent two types of processing: FlavourA (standard) and FlavourB (added enhancement). Six observers located features in the breast they suspected to be cancerous (4,188 observations). Data were analysed using jackknife alternative free-response receiver operating characteristic (JAFROC) analysis. Characteristics of the cancers detected with each image processing type were investigated.

RESULTS

For calcifications, the JAFROC figure of merit (FOM) was equal to 0.86 for both types of image processing. For soft-tissue lesions, the JAFROC FOM were better for FlavourA (0.81) than FlavourB (0.78); this difference was significant (p=0.001). Using FlavourA a greater number of cancers of all grades and sizes were detected than with FlavourB. FlavourA improved soft-tissue lesion detection in denser breasts (p=0.04 when volumetric density was over 7.5%) CONCLUSIONS: The detection of malignant soft-tissue lesions (which were primarily invasive) was significantly better with FlavourA than FlavourB image processing. This is despite FlavourB having a higher contrast appearance often preferred by radiologists. It is important that clinical choice of image processing is based on objective measures.

Radiation risk of breast screening in England with digital mammography

OBJECTIVE

To estimate the risks and benefits of breast screening in terms of number of deaths due to radiation-induced cancers and the number of lives saved owing to modern screening in the National Health Service Breast Screening Programme (NHSBSP) in England.

METHODS

Radiation risk model, patient dose data and data from national screening statistics were used to estimate the number of deaths due to radiation-induced breast cancers in the NHSBSP in England. Dose and dose effectiveness factors (DDREFs) equal to one and two were assumed. The breast cancer mortality reduction in the invited population due to screening and the percentage of females diagnosed with symptomatic breast cancer, who die from breast cancer, were collated from the literature. The number of lives saved owing to screening was calculated.

RESULTS

Assuming, a total of 1,770,436 females between the ages of 50-70 years were screened each year, and a breast cancer mortality reduction of 20% due to screening in the invited population, the number of screen-detected cancers were 14,872 annually, resulting in 1071 lives saved. Conversely, for the same mortality reduction, the number of radiation-induced cancers was 36 and 18 for DDREFs of 1 and 2, respectively. This resulted in seven and three deaths due to radiation-induced cancers annually for DDREFs of 1 and 2, respectively. The ratios of lives saved owing to screening to radiation-induced cancers were 30 : 1 and 60 : 1 for DDREFs of 1 and 2. The ratios of lives saved owing to screening to deaths due to radiation-induced cancers were 156 : 1 and 312 : 1 for DDREFs of 1 and 2. For the 1.8% of the screening population with very thick breasts, the latter ratios decrease to 94 : 1 and 187 : 1 for DDREFs of 1 and 2.

CONCLUSION

The breast cancer mortality reduction due to screening greatly outweighs the risk of death due to radiation-induced cancers. Advances in knowledge: Estimation of the radiation risk for modern breast screening in England using digital mammography.

VALIDATION OF A SIMULATION PROCEDURE FOR GENERATING BREAST TOMOSYNTHESIS PROJECTION IMAGES

In order to achieve optimal diagnostic performance in breast tomosynthesis (BT) imaging, the parameters of the imaging chain should be evaluated. For the purpose of such evaluations, a simulation procedure based on the Monte Carlo code system Penelope and the geometry of a Siemens BT system has been developed to generate BT projection images. In this work, the simulation procedure is validated by comparing contrast and sharpness in simulated images with contrast and sharpness in real images acquired with the BT system. The results of the study showed a good agreement of sharpness in real and simulated reconstructed image planes, but the contrast was shown to be higher in the simulated compared with the real projection images. The developed simulation procedure could be used to generate BT images, but it is of interest to further investigate how the procedure could be modified to generate more realistic image noise and contrast.

The relationship between cancer detection in mammography and image quality measurements

PURPOSE

To investigate the relationship between image quality measurements and the clinical performance of digital mammographic systems.

METHODS

Mammograms containing subtle malignant non-calcification lesions and simulated malignant calcification clusters were adapted to appear as if acquired by four types of detector. Observers searched for suspicious lesions and gave these a malignancy score. Analysis was undertaken using jackknife alternative free-response receiver operating characteristics weighted figure of merit (FoM). Images of a CDMAM contrast-detail phantom were adapted to appear as if acquired using the same four detectors as the clinical images. The resultant threshold gold thicknesses were compared to the FoMs using a linear regression model and an F-test was used to find if the gradient of the relationship was significantly non-zero.

RESULTS

The detectors with the best image quality measurement also had the highest FoM values. The gradient of the inverse relationship between FoMs and threshold gold thickness for the 0.25mm diameter disk was significantly different from zero for calcification clusters (p=0.027), but not for non-calcification lesions (p=0.11). Systems performing just above the minimum image quality level set in the European Guidelines for Quality Assurance in Breast Cancer Screening and Diagnosis resulted in reduced cancer detection rates compared to systems performing at the achievable level.

CONCLUSIONS

The clinical effectiveness of mammography for the task of detecting calcification clusters was found to be linked to image quality assessment using the CDMAM phantom. The European Guidelines should be reviewed as the current minimum image quality standards may be too low.

Breast cancer detection rates using four different types of mammography detectors

OBJECTIVE

To compare the performance of different types of detectors in breast cancer detection.

METHODS

A mammography image set containing subtle malignant non-calcification lesions, biopsy-proven benign lesions, simulated malignant calcification clusters and normals was acquired using amorphous-selenium (a-Se) detectors. The images were adapted to simulate four types of detectors at the same radiation dose: digital radiography (DR) detectors with a-Se and caesium iodide (CsI) convertors, and computed radiography (CR) detectors with a powder phosphor (PIP) and a needle phosphor (NIP). Seven observers marked suspicious and benign lesions. Analysis was undertaken using jackknife alternative free-response receiver operating characteristics weighted figure of merit (FoM). The cancer detection fraction (CDF) was estimated for a representative image set from screening.

RESULTS

No significant differences in the FoMs between the DR detectors were measured. For calcification clusters and non-calcification lesions, both CR detectors' FoMs were significantly lower than for DR detectors. The calcification cluster's FoM for CR NIP was significantly better than for CR PIP. The estimated CDFs with CR PIP and CR NIP detectors were up to 15% and 22% lower, respectively, than for DR detectors.

CONCLUSION

Cancer detection is affected by detector type, and the use of CR in mammography should be reconsidered.

KEY POINTS

The type of mammography detector can affect the cancer detection rates. CR detectors performed worse than DR detectors in mammography. Needle phosphor CR performed better than powder phosphor CR. Calcification clusters detection is more sensitive to detector type than other cancers.

Development and validation of a modelling framework for simulating 2D-mammography and breast tomosynthesis images

Planar 2D x-ray mammography is generally accepted as the preferred screening technique used for breast cancer detection. Recently, digital breast tomosynthesis (DBT) has been introduced to overcome some of the inherent limitations of conventional planar imaging, and future technological enhancements are expected to result in the introduction of further innovative modalities. However, it is crucial to understand the impact of any new imaging technology or methodology on cancer detection rates and patient recall. Any such assessment conventionally requires large scale clinical trials demanding significant investment in time and resources. The concept of virtual clinical trials and virtual performance assessment may offer a viable alternative to this approach. However, virtual approaches require a collection of specialized modelling tools which can be used to emulate the image acquisition process and simulate images of a quality indistinguishable from their real clinical counterparts. In this paper, we present two image simulation chains constructed using modelling tools that can be used for the evaluation of 2D-mammography and DBT systems. We validate both approaches by comparing simulated images with real images acquired using the system being simulated. A comparison of the contrast-to-noise ratios and image blurring for real and simulated images of test objects shows good agreement ( < 9% error). This suggests that our simulation approach is a promising alternative to conventional physical performance assessment followed by large scale clinical trials.

The effect of image processing on the detection of cancers in digital mammography

OBJECTIVE. The objective of our study was to investigate the effect of image processing on the detection of cancers in digital mammography images. MATERIALS AND METHODS. Two hundred seventy pairs of breast images (both breasts, one view) were collected from eight systems using Hologic amorphous selenium detectors: 80 image pairs showed breasts containing subtle malignant masses; 30 image pairs, biopsy-proven benign lesions; 80 image pairs, simulated calcification clusters; and 80 image pairs, no cancer (normal). The 270 image pairs were processed with three types of image processing: standard (full enhancement), low contrast (intermediate enhancement), and pseudo-film-screen (no enhancement). Seven experienced observers inspected the images, locating and rating regions they suspected to be cancer for likelihood of malignancy. The results were analyzed using a jackknife-alternative free-response receiver operating characteristic (JAFROC) analysis. RESULTS. The detection of calcification clusters was significantly affected by the type of image processing: The JAFROC figure of merit (FOM) decreased from 0.65 with standard image processing to 0.63 with low-contrast image processing (p = 0.04) and from 0.65 with standard image processing to 0.61 with film-screen image processing (p = 0.0005). The detection of noncalcification cancers was not significantly different among the image-processing types investigated (p > 0.40). CONCLUSION. These results suggest that image processing has a significant impact on the detection of calcification clusters in digital mammography. For the three image-processing versions and the system investigated, standard image processing was optimal for the detection of calcification clusters. The effect on cancer detection should be considered when selecting the type of image processing in the future.

Validation of simulation of calcifications for observer studies in digital mammography

Studies using simulated calcifications can be performed to measure the effect of different imaging factors on calcification detection in digital mammography. The simulated calcifications must be inserted into clinical images with realistic contrast and sharpness. MoCa is a program which modifies the contrast and sharpness of simulated calcification clusters extracted from images of mastectomy specimens acquired on a digital specimen cabinet at high magnification for insertion into clinical mammography images. This work determines whether the use of MoCa results in simulated calcifications with the correct contrast and sharpness. Aluminium foils (thickness 0.1-0.4 mm) and 1.60 µm thick gold discs (diameter 0.13-0.8 mm) on 0.5 mm aluminium were imaged with a range of thicknesses of polymethyl methacrylate (PMMA) using an amorphous selenium direct digital (DR) system and a powder phosphor computed radiography (CR) system (real images). Simulated images of the tests objects were also generated using MoCa. The contrast of the aluminium squares and the degradation of the contrast of the gold discs as a function of disc diameter were compared in the real and simulated images. The average ratios of the simulated-to-real aluminium contrasts over all aluminium and PMMA thicknesses were 1.03 ± 0.04 (two standard errors in the mean) and 0.99 ± 0.03 for the DR and CR systems, respectively. The ratio of the simulated-to-real degradations of contrast averaged over all disc diameters and PMMA thicknesses were 1.007 ± 0.008 and 1.002 ± 0.013 for DR and CR, respectively. The use of MoCa was accurate within the experimental errors.

Comparison of the x-ray attenuation properties of breast calcifications, aluminium, hydroxyapatite and calcium oxalate

Aluminium is often used as a substitute material for calcifications in phantom measurements in mammography. Additionally, calcium oxalate, hydroxyapatite and aluminium are used in simulation studies. This assumes that these materials have similar attenuation properties to calcification, and this assumption is examined in this work. Sliced mastectomy samples containing calcification were imaged at ×5 magnification using a digital specimen cabinet. Images of the individual calcifications were extracted, and the diameter and contrast of each calculated. The thicknesses of aluminium required to achieve the same contrast as each calcification when imaged under the same conditions were calculated using measurements of the contrast of aluminium foils. As hydroxyapatite and calcium oxalate are also used to simulate calcifications, the equivalent aluminium thicknesses of these materials were also calculated using tabulated attenuation coefficients. On average the equivalent aluminium thickness was 0.85 times the calcification diameter. For calcium oxalate and hydroxyapatite, the equivalent aluminium thicknesses were 1.01 and 2.19 times the thickness of these materials respectively. Aluminium and calcium oxalate are suitable substitute materials for calcifications. Hydroxyapatite is much more attenuating than the calcifications and aluminium. Using solid hydroxyapatite as a substitute for calcification of the same size would lead to excessive contrast in the mammographic image.

Effect of image quality on calcification detection in digital mammography

PURPOSE

This study aims to investigate if microcalcification detection varies significantly when mammographic images are acquired using different image qualities, including: different detectors, dose levels, and different image processing algorithms. An additional aim was to determine how the standard European method of measuring image quality using threshold gold thickness measured with a CDMAM phantom and the associated limits in current EU guidelines relate to calcification detection.

METHODS

One hundred and sixty two normal breast images were acquired on an amorphous selenium direct digital (DR) system. Microcalcification clusters extracted from magnified images of slices of mastectomies were electronically inserted into half of the images. The calcification clusters had a subtle appearance. All images were adjusted using a validated mathematical method to simulate the appearance of images from a computed radiography (CR) imaging system at the same dose, from both systems at half this dose, and from the DR system at quarter this dose. The original 162 images were processed with both Hologic and Agfa (Musica-2) image processing. All other image qualities were processed with Agfa (Musica-2) image processing only. Seven experienced observers marked and rated any identified suspicious regions. Free response operating characteristic (FROC) and ROC analyses were performed on the data. The lesion sensitivity at a nonlesion localization fraction (NLF) of 0.1 was also calculated. Images of the CDMAM mammographic test phantom were acquired using the automatic setting on the DR system. These images were modified to the additional image qualities used in the observer study. The images were analyzed using automated software. In order to assess the relationship between threshold gold thickness and calcification detection a power law was fitted to the data.

RESULTS

There was a significant reduction in calcification detection using CR compared with DR: the alternative FROC (AFROC) area decreased from 0.84 to 0.63 and the ROC area decreased from 0.91 to 0.79 (p < 0.0001). This corresponded to a 30% drop in lesion sensitivity at a NLF equal to 0.1. Detection was also sensitive to the dose used. There was no significant difference in detection between the two image processing algorithms used (p > 0.05). It was additionally found that lower threshold gold thickness from CDMAM analysis implied better cluster detection. The measured threshold gold thickness passed the acceptable limit set in the EU standards for all image qualities except half dose CR. However, calcification detection varied significantly between image qualities. This suggests that the current EU guidelines may need revising.

CONCLUSIONS

Microcalcification detection was found to be sensitive to detector and dose used. Standard measurements of image quality were a good predictor of microcalcification cluster detection.

Effects of BHV-1 on PMN adhesion to bovine lung endothelial cells

Bovine herpes virus-1 (BHV-1) infection appears to decrease the rate of polymorphonuclear leukocyte (PMN) influx into the lung in response to the secondary invader, Pasteurella haemolytica. It was postulated that BHV-1 may affect the rate of cellular infiltration by altering the function of the endothelium, thereby preventing PMN movement across the blood-tissue barrier. Therefore, we decided to investigate the effect of BHV-1 on the ability of PMN to adhere to lung endothelial cells (LEC). LEC were isolated from fetal bovine fetal tissue and were shown to function in PMN adhesion assays. Furthermore, enhanced PMN adhesion was observed after exposure of LEC to recombinant bovine TNF-alpha (rBoTNF-alpha) for 4, 8, 12, and 24 h. LEC infected with BHV-1 were shown to be less responsive to rBoTNF-alpha. However, infection of LEC with BHV-1 at an multiplicity of infection (MOI) of 1.0 or 10 did not affect basal levels of PMN adhesion to these cells. Decreased PMN binding to BHV-1-infected LEC, simultaneously treated with rBoTNF-alpha, was observed at 10-12 h post-infection. The data suggest that BHV-1 may prevent cytokine-induced PMN infiltration of the lung through the modification of EC responses to cytokines.