Quantitative versus subjective evaluation of mammography accreditation phantom images

Automated image quality assessment of mammography phantoms: a systematic review

BACKGROUND

Computerized image analysis is a viable technique for evaluating image quality as a complement to human observers.

PURPOSE

To systematically review the image analysis software used in the assessment of 2D image quality using mammography phantoms.

MATERIAL AND METHODS

A systematic search of multiple databases was performed from inception to July 2020 for articles that incorporated computerized analysis of 2D images of physical mammography phantoms to determine image quality.

RESULTS

A total of 26 studies were included, 12 were carried out using direct digital imaging and 14 using screen film mammography. The ACR phantom (model-156) was the most frequently evaluated phantom, possibly due to the lack of accepted standard software. In comparison to the inter-observer variations, the computerized image analysis was more consistent in scoring test objects. The template matching method was found to be one of the most reliable algorithms, especially for high-contrast test objects, while several algorithms found low-contrast test objects to be harder to distinguish due to the smaller contrast variations between test objects and their backgrounds. This was particularly true for small object sizes.

CONCLUSION

Image analysis software was in agreement with human observers but demonstrated higher consistency and reproducibility of quality evaluation. Additionally, using computerized analysis, several quantitative metrics such as contrast-to-noise ratio (CNR) and the signal-to-noise ratio (SNR) could be used to complement the conventional scoring method. Implementing a computerized approach for monitoring image quality over time would be crucial to detect any deteriorating mammography system before clinical images are impacted.

Assessment of task-based performance from five clinical DBT systems using an anthropomorphic breast phantom

PURPOSE

Digital breast tomosynthesis (DBT) is a limited-angle tomographic breast imaging modality that can be used for breast cancer screening in conjunction with full-field digital mammography (FFDM) or synthetic mammography (SM). Currently, there are five commercial DBT systems that have been approved by the U.S. FDA for breast cancer screening, all varying greatly in design and imaging protocol. Because the systems are different in technical specifications, there is a need for a quantitative approach for assessing them. In this study, the DBT systems are assessed using a novel methodology with an inkjet-printed anthropomorphic phantom and four alternative forced choice (4AFC) study scheme.

METHOD

A breast phantom was fabricated using inkjet printing and parchment paper. The phantom contained 5-mm spiculated masses fabricated with potassium iodide (KI)-doped ink and microcalcifications (MCs) made with calcium hydroxyapatite. Images of the phantom were acquired on all five systems with DBT, FFDM, and SM modalities where available using beam settings under automatic exposure control. A 4AFC study was conducted to assess reader performance with each signal under each modality. Statistical analysis was performed on the data to determine proportion correct (PC), standard deviations, and levels of significance.

RESULTS

For masses, overall detection was highest with DBT. The difference in PC was statistically significant between DBT and SM for most systems. A relationship was observed between increasing PC and greater gantry span. For MCs, performance was highest with DBT and FFDM compared to SM. The difference between PC of DBT and PC of SM was statistically significant for all manufacturers.

CONCLUSIONS

This methodology represents a novel approach for evaluating systems. This study is the first of its kind to use an inkjet-printed anthropomorphic phantom with realistic signals to assess performance of clinical DBT imaging systems.

Computerized quantitative evaluation of mammographic accreditation phantom images

PURPOSE

The objective was to develop and investigate an automated scoring scheme of the American College of Radiology (ACR) mammographic accreditation phantom (RMI 156, Middleton, WI) images.

METHODS

The developed method consisted of background subtraction, determination of region of interest, classification of fiber and mass objects by Mahalanobis distance, detection of specks by template matching, and rule-based scoring. Fifty-one phantom images were collected from 51 facilities for this study (one facility provided one image). A medical physicist and two radiologic technologists also scored the images. The human and computerized scores were compared.

RESULTS

In terms of meeting the ACR's criteria, the accuracies of the developed method for computerized evaluation of fiber, mass, and speck were 90%, 80%, and 98%, respectively. Contingency table analysis revealed significant association between observer and computer scores for microcalcifications (p<5%) but not for masses and fibers.

CONCLUSIONS

The developed method may achieve a stable assessment of visibility for test objects in mammographic accreditation phantom image in whether the phantom image meets the ACR's criteria in the evaluation test, although there is room left for improvement in the approach for fiber and mass objects.

Modeling, validation and application of a mathematical tissue-equivalent breast phantom for linear slot-scanning digital mammography

This paper presents a mathematical tissue-equivalent breast phantom for linear slot-scanning digital mammography. A recently developed prototype linear slot-scanning digital mammography system was used for model validation; image quality metrics such as image contrast and contrast-to-noise ratio were calculated. The results were in good agreement with values measured using a physical breast-equivalent phantom designed for mammography. The estimated pixel intensity of the mathematical phantom, the analogue-to-digital conversion gain and the detector additive noise showed good agreement with measured values with correlation of nearly 1. An application of the model, to examine the feasibility of using a monochromatic filter for dose reduction and improvement of image quality in slot-scanning digital mammography, is presented.

Detection of simulated microcalcifications in a phantom with digital mammography: effect of pixel size

PURPOSE

To evaluate the effect of pixel size on the detection of simulated microcalcifications in a phantom with digital mammography.

MATERIALS AND METHODS

A high-spatial-resolution prototype imager that yields variable pixel size (39 and 78 microm) and a clinical full-field digital mammography (FFDM) system that yields a 100-microm pixel size were used. Radiographic images of a contrast-detail (CD) phantom were obtained to perform four-alternative forced-choice observer experiments. Polymethylmethacrylate was added to obtain phantom thicknesses of 45 and 58 mm, which are typical breast thicknesses encountered in mammography. Phantom images were acquired with both systems under nearly identical exposure conditions by using an antiscatter grid. Twelve images were acquired for each phantom thickness and pixel size (for a total of 72 images), and six observers participated in this study. Observer responses were used to compute the fraction of correctly detected disks. A signal detection model was used to fit the recorded data from which CD characteristics were obtained. Repeated-measures analyses with mixed-effects linear models were performed for each of the six observers. All statistical tests were two sided and unadjusted for multiple comparisons. A P value of .05 or less was considered to indicate a significant difference.

RESULTS

Statistical analysis revealed significantly better CD characteristics with 39- and 78-microm pixel sizes compared with 100-microm pixel size for all disk diameters and phantom thicknesses (P<.001). Increase in phantom thickness degraded CD characteristics regardless of pixel size (P<.001).

CONCLUSION

On the basis of the conditions of this study, reducing pixel size below 100 mum with low imaging system noise enhances the visual perception of small objects that correspond to typical microcalcifications.

Automated analysis of phantom images for the evaluation of long-term reproducibility in digital mammography

The performance of an automatic software package was evaluated with phantom images acquired by a full-field digital mammography unit. After the validation, the software was used, together with a Leeds TORMAS test object, to model the image acquisition process. Process modelling results were used to evaluate the sensitivity of the method in detecting changes of exposure parameters from routine image quality measurements in digital mammography, which is the ultimate purpose of long-term reproducibility tests. Image quality indices measured by the software included the mean pixel value and standard deviation of circular details and surrounding background, contrast-to-noise ratio and relative contrast; detail counts were also collected. The validation procedure demonstrated that the software localizes the phantom details correctly and the difference between automatic and manual measurements was within few grey levels. Quantitative analysis showed sufficient sensitivity to relate fluctuations in exposure parameters (kV(p) or mAs) to variations in image quality indices. In comparison, detail counts were found less sensitive in detecting image quality changes, even when limitations due to observer subjectivity were overcome by automatic analysis. In conclusion, long-term reproducibility tests provided by the Leeds TORMAS phantom with quantitative analysis of multiple IQ indices have been demonstrated to be effective in predicting causes of deviation from standard operating conditions and can be used to monitor stability in full-field digital mammography.

An alternate method for using a visual discrimination model (VDM) to optimize soft-copy display image quality

Researchers have developed visual discrimination models (VDMs) that can predict a human observer's ability to detect a target object superposed on an image. These models incorporate sophisticated knowledge of the properties of the human visual system. In the predictive approach, termed conventional VDM usage, two input images with and without a target are analyzed by an algorithm that calculates a just-noticeable-difference (JND) index, which is a taken as a measure of the detectability of the target. A new method of using the VDM is described, termed channelized VDM, which involves finding the linear combination of the VDM-generated channels (which are not used in conventional VDM analysis) that has optimal classification ability between normal and abnormal images. The classification ability can be measured using receiver operating characteristic (ROC) or two alternative forced choice (2AFC) experiments, and in special cases they can also be predicted by signal detection theory (SDT) based model-observer methods. In this study simulated background and nodule containing regions were used to validate the new method. It was found that the channelized VDM predictions were in excellent qualitative agreement with human-observer validated SDT predictions. Either VDM method (conventional or channelized) has potential applicability to soft-copy display optimization. An advantage of any VDM-based approach is that complex effects, such as visual masking, are automatically accounted for, which effects are usually not included in SDT-based methods.

Are phantoms useful for predicting the potential of dose reduction in full-field digital mammography?

A phantom study was performed in full-field digital mammography to investigate the opportunity and the magnitude of a possible dose reduction that would leave the image quality above the accepted thresholds associated with some classical phantoms. This preliminary work is intended to lay the groundwork for a future clinical study on the impact of dose reduction on clinical results. Three different mammography phantoms (ACR RMI 156, CIRS 11A and CDMAM 3.4) were imaged by a full-field digital mammography unit (GE Senographe 2000D) at different dose levels. Images were rated by three observers with softcopy reading and scoring methods specific to each phantom. Different types of data analysis were applied to the ACR (American College of Radiology) and the other two phantoms, respectively. With reference to the minimum acceptance score in screen/film accreditation programmes, the ACR phantom showed that about 45% dose reduction could be applied, while keeping the phantom scores above that threshold. A relative comparison was done for CIRS and CDMAM, for which no threshold is defined. CIRS scoring remained close to the reference level down to 40% dose reduction, the inter- and intra-observer variability being the main source of uncertainty. Contrast-detail curves provided by CDMAM overlapped down to 50% dose reduction, at least for object contrast values ranging between 30% and 3%. This multi-phantom study shows the potential of further reducing the dose in full-field digital mammography beyond the current values. A common dose reduction factor around 50% seems acceptable for all phantoms. However, caution is required before extrapolating the results for clinical use, given the limitations of these widely used phantoms, mainly related to their limited dynamic range and uniform background.

A new test phantom with different breast tissue compositions for image quality assessment in conventional and digital mammography

Our objective is to describe a new test phantom that permits the objective assessment of image quality in conventional and digital mammography for different types of breast tissue. A test phantom, designed to represent a compressed breast, was made from tissue equivalent materials. Three separate regions, with different breast tissue compositions, are used to evaluate low and high contrast resolution, spatial resolution and image noise. The phantom was imaged over a range of kV using a Contour 2000 (Bennett) mammography unit with a Kodak MinR 2190-MinR L screen-film combination and a Senograph 2000D (General Electric) digital mammography unit. Objective image quality assessments for different breast tissue compositions were performed using the phantom for conventional and digital mammography. For a similar mean glandular dose (MGD), the digital system gives a significantly higher contrast-to-noise ratio (CNR) than the screen-film system for 100% glandular tissue. In conclusion, in mammography, a range of exposure conditions is used for imaging because of the different breast tissue compositions encountered clinically. Ideally, the patient dose-image quality relationship should be optimized over the range of exposure conditions. The test phantom presented in this work permits image quality parameters to be evaluated objectively for three different types of breast tissue. Thus, it is a useful tool for optimizing the patient dose-image quality relationship.

A perceptual evaluation of JPEG 2000 image compression for digital mammography: contrast-detail characteristics

In this investigation the effect of JPEG 2000 compression on the contrast-detail (CD) characteristics of digital mammography images was studied using an alternative forced choice (AFC) technique. Images of a contrast-detail phantom, acquired using a clinical full-field digital mammography system, were compressed using a commercially available software product (JPEG 2000). Data compression was achieved at ratios of 1:1, 10:1, 20:1, and 30:1 and the images were reviewed by seven observers on a high-resolution display. Psychophysical detection characteristics were first computed by fitting perception data using a maximum-likelihood technique from which CD curves were derived at 50%, 62.5%, and 75% threshold levels. Statistical analysis indicated no significant difference in the perception of mean disk thickness up to 20:1 compression except for disk diameter of 1 mm. All other compression combinations exhibited significant degradation in CD characteristics.

Automatic quantitative low contrast analysis of digital chest phantom radiographs

Up to the present, the majority of low contrast object evaluations performed on phantom images have been accomplished in a subjective fashion. This is mainly due to the time and effort required to manually measure each radiograph with a densitometer to obtain quantitative results. However, with the development of digital radiographic systems, it has become feasible to automate the detection and computation processes. In this work, a method that can automatically detect and compute the subject-to-noise ratio of the low contrast disks inside a geometric chest phantom is examined. This algorithm has the ability to locate the low contrast objects to an accuracy of less than one pixel, and provides results that are consistent with the understanding of subject-to-noise ratio. This algorithm should simplify the task of quantitative evaluation of contrast detail phantoms.

Flat-panel digital mammography system: contrast-detail comparison between screen-film radiographs and hard-copy images

PURPOSE

To compare the contrast-detail (CD) characteristics of screen-film (SF) and postprocessed digital images by using a phantom-based method.

MATERIALS AND METHODS

Images of a CD phantom with polymerized methyl methacrylate were acquired with SF and full-field digital mammography systems at matched exposure conditions. A four-alternative forced-choice experiment was conducted with seven observers participating in the study. Each observer was required to identify randomly located disks in phantom images from which detection curves were computed. The CD diagrams for the SF and digital systems were estimated from the detection curves and compared at 50% and 62.5% threshold levels. Furthermore, a theoretic model was used to estimate the CD performance of the SF and digital systems.

RESULTS

Analysis of covariance for mixed models was used with the natural logarithm of disk thickness as the dependent variable, the natural logarithm of disk diameter as the covariate, and the observer as a random factor. The results of statistical analysis indicated significant differences between the CD characteristics of SF and digital mammographic images at both 50% (P <.001) and 62.5% (P <.001) detection thresholds.

CONCLUSION

The authors conclude that digital CD curves, on average, exhibit threshold contrast characteristics that are lower (better) than those of SF mammography.

Factors affecting phantom scores at annual mammography facility inspections by the U.S. Food and Drug Administration

RATIONALE AND OBJECTIVES

The authors performed this study to evaluate the factors affecting phantom image score at the annual inspection of mammography facilities.

MATERIALS AND METHODS

In 1997, three U.S. Food and Drug Administration (FDA)-trained inspectors performed inspections of all mammography facilities in North Carolina. All federal and state inspection data were collected and evaluated by using linear regression analysis. Factors affecting the American College of Radiology phantom scores were assessed.

RESULTS

Phantom score was affected by inspector identity, view box luminance, and optical density. All of these factors had a statistically significant effect on mass score (P < .05). Inspector identity yielded a statistically significant effect on speck group score, fibril score, and total score. Luminance yielded a statistically significant effect on both speck group score and total score.

CONCLUSION

Phantom scoring should be automated to allow for more consistent interobserver scoring. In addition, radiology facilities can improve the likelihood of receiving a passing phantom score by reducing the ambient light and increasing the view box luminance in the location where the images are evaluated and the phantom is scored routinely. Radiologists should also consider increasing phantom and clinical image optical density to allow for improved phantom testing outcomes.

Image quality and dose in film-screen magnification mammography

Extended exposure times in magnification mammography are a result of the reduced X-ray tube currents required for a small focal spot. The consequences of this are the potential for reduced image quality through motion blur during exposure as well as the onset of film reciprocity law failure. Previous investigators have suggested increasing the X-ray tube potential as a practical mechanism for reducing exposure times in magnification mammography and have demonstrated negligible image quality degradation at least up to 32 kVp. This paper describes a film-screen magnification mammography study that expands upon this previous work to investigate the magnitude of the reduction of breast mean glandular dose and exposure time and the changes in subjective image quality (visibility of low contrast details in an RMI 152 phantom) with increases in tube potential between 28 kVp and 35 kVp. Measures of changes in the radiographic contrast and in the scatter-to-primary ratio (SPR) in magnification geometry as a function of tube potential were also obtained. Evidence for reciprocity law failure was also assessed. For a constant film optical density, increasing the X-ray tube potential from 28 kVp to 35 kVp reduced the mean glandular dose from 3.9 mGy to 2.7 mGy and reduced the exposure time from 3.2 s to 1.0 s. Over this range, the detection rate of fibrils and microcalcification-mimicking specks did not vary with tube potential at the 0.05 level of significance. It was found that only the low contrast mass detail detection rate at 35 kVp was significantly less than that at 28 kVp. The measured radiographic contrast decreased with tube potential and the SPR increased with tube potential. However, both changes were weak, and linear regressions determined that the 95% confidence intervals of the slopes relating both contrast and SPR with tube potential encompassed zero. It is concluded that magnification mammography performed at 34 kVp yields significant reductions in exposure time and mean glandular dose, with a detail detection capability similar to that at 28 kVp.

A comparison of digital and screen-film mammography using quality control phantoms

AIM

To compare the performance of a direct digital mammography system with normal-view and magnified-view conventional screen-film methods using quality control phantoms.

MATERIALS AND METHODS

Using a Siemens Mammomat((R))3000 and an Opdima((R))digital spot imaging and biopsy attachment, film and direct digital images of two phantoms [DuPont and TOR (MAM)] were obtained under normal operating conditions. These were assessed by three groups of observers with differing expertise - radiologists, radiographers and medical physicists. Each observer was asked to compare the direct digital image with films taken in standard view and magnified view, providing scores for object visibility and confidence. For the digital images, observers were allowed to vary the image presentation parameters.

RESULTS

Both phantoms showed that overall the direct digital view and the magnified view film performed significantly better (P < 0.05) than standard view film. For certain small or low contrast objects the differences became very highly significant (P < 0.001).

CONCLUSION

Only the TOR (MAM) phantom showed any significant difference between digital and magnified modalities, with magnified views performing better for fine, faint filaments and digital acquisition better for low contrast objects. Almost no difference existed between the three observer groups. Undrill, P. E. (2000). Clinical Radiology53, 782-790.

The effect of the antiscatter grid on full-field digital mammography phantom images

Computer Analysis of Mammography Phantom Images (CAMPI) is a method for making quantitative measurements of image quality. This article reports on a recent application of this method to a prototype full-field digital mammography (FFDM) machine. Images of a modified ACR phantom were acquired on the General Electric Diagnostic Molybdenum Rhodium (GE-DMR) FFDM machine at a number of x-ray techniques, both with and without the scatter reduction grid. The techniques were chosen so that one had sets of grid and non-grid images with matched doses (200 mrads) and matched gray-scale values (1500). A third set was acquired at constant 26 kVp and varying mAs for both grid conditions. Analyses of the images yielded signal-to-noise-ratio (SNR), contrast and noise corresponding to each target object, and a non-uniformity measure. The results showed that under conditions of equal gray-scale value the grid images were markedly superior, albeit at higher doses than the non-grid images. Under constant dose conditions, the non-grid images were slightly superior in SNR (7%) but markedly less uniform (60%). Overall, the grid images had substantially greater contrast and superior image uniformity. These conclusions applied to the whole kVp range studied for the Mo-Mo target filter combination and 4 cm of breast equivalent material of average composition. These results suggest that use of the non-grid technique in digital mammography with the GE-DMR-FFDM unit, is presently not warranted. With improved uniformity correction procedure, this conclusion would change and one should be able to realize a 14% reduction in patient dose at the same SNR by using a non-grid technique.

Computerized evaluation of mammographic image quality using phantom images

A simple, quick and computerized method for quantitatively evaluating the image quality of mammography phantom images has been developed. Images of the American College of Radiology (ACR) mammographic accreditation phantoms were acquired under different X-ray techniques, scored and ranked subjectively by five expert readers, and digitized for quantitative analysis. The contrast and signal-to-noise (contrast-to-noise) ratios of the main nodule and microcalcification group were obtained accurately and reproducibly using an image processing protocol. The contrast values were successful at discriminating differences in image quality due to variations in scatter conditions (as a result of different kVp's, and the presence or absence of an acrylic scatterer and/or a moving Bucky grid). They were more precise, reproducible and sensitive than the ACR score. In particular, the contrast of the main nodule was highly correlated (r(s) = 0.988: p<0.001) with the ranking of image quality by our panel of expert readers.

Objective assessment of phantom image quality in mammography: a feasibility study

The need for test objects in mammography quality control programmes to provide an objective measure of image quality pertinent to clinical problems is well documented. However, interobserver variations may be greater than the fluctuations in image quality that the quality control programme is seeking to detect. We have developed a computer algorithm to score a number of features in the Leeds TOR(MAX) mammography phantom. Threshold scoring techniques have been applied in the first instance; scoring schemes which utilize measures such as signal-to-noise ratio and modulation have also been formulated. This fully automatic algorithm has been applied to a set of 10 films which have been digitized at 25 microns resolution using a Joyce-Loebl scanning microdensitometer. The films were chosen retrospectively from quality control test films to demonstrate: (a) a range of optimized imaging systems, and (b) variation from the optimum. The performance of the algorithm has been compared with that of five experienced observers, and has been shown to be as consistent as individual observers, but more consistent than a pool of observers. Problems have been encountered with the detection of small details, indicating that a more sophisticated localization technique is desirable. The computer performs more successfully with the scoring scheme which utilizes the full imaging information available, rather than with the threshold-determined one. However, both the observers and the computer algorithm failed to identify the non-optimum films, suggesting that the sensitivity of the TOR(MAX) test object may not be adequate for modern mammography imaging systems.

Computer analysis of mammography phantom images (CAMPI): an application to the measurement of microcalcification image quality of directly acquired digital images

The purpose of this investigation was to apply the recently developed CAMPI (computer analysis of mammography phantom images) method to a Fischer Mammotest Stereotactic Digital Biopsy machine. Another aim was to further elucidate the nature of the empirically introduced CAMPI measures. Images of an American College of Radiology (ACR) accreditation phantom centered on the largest two speck groups were obtained on this machine under a variety of x-ray conditions. An additional measure, alternative SNR (ASNR) is introduced which is complementary to the SNR measure. Analyses of the Mammotest images revealed that the mAs and kVp dependencies of the CAMPI measures could be understood from basic imaging physics principles. It is shown that: (1) the measures reflect the expected linearity of the digital detector and Poisson photon statistics; (2) under automatic exposure control (AEC) conditions the signal (SIG) measure is proportional to subject contrast; and (3) under AEC conditions the noise (NOI) measure is proportional to the square root of the average absorbed photon energy. Correspondence with basic imaging physics principles shows that the measures are significantly free of artifacts. Precision of the CAMPI measures exceeds that of human observers by orders of magnitude. CAMPI measures are expected to be more relevant to clinical mammography than Fourier metrics as the measurements are done on objects of arbitrary shape and size that were designed by the manufacturer to resemble various detection tasks in mammography. It is concluded that CAMPI can perform objective and highly precise evaluations of phantom image quality in mammography. It could be used as a sophisticated quality control tool, as a replacement for the current ACR/MQSA phantom evaluation program, and to evaluate the rapidly evolving digital mammography technology.

Automated analysis of the American College of Radiology mammographic accreditation phantom images

A significant metric in federal mammography quality standards is the phantom image quality assessment. The present work seeks to demonstrate that automated image analyses for American College of Radiology (ACR) mammographic accreditation phantom (MAP) images may be performed by a computer with objectivity, once a human acceptance level has been established. Twelve MAP images were generated with different x-ray techniques and digitized. Nineteen medical physicists in diagnostic roles (five of which were specially trained in mammography) viewed the original film images under similar conditions and provided individual scores for each test object (fibrils, microcalcifications, and nodules). Fourier domain template matching, used for low-level processing, combined with derivative filters, for intermediate-level processing, provided translation and rotation-independent localization of the test objects in the MAP images. The visibility classification decision was modeled by a Bayesian classifer using threshold contrast. The 50% visibility contrast threshold established by the trained observers' responses were: fibrils 1.010, microcalcifications 1.156, and nodules 1.016. Using these values as an estimate of human observer performance and given the automated localization of test objects, six images were graded with the computer algorithm. In all but one instance, the algorithm scored the images the same as the diagnostic physicists. In the case where it did not, the margin of disagreement was 10% due to the fact that the human scoring did not allow for half-visible fibrils (agreement occurred for the other test objects). The implication from this is that an operator-independent, machine-based scoring of MAP images is feasible and could be used as a tool to help eliminate the effect of observer variability within the current system, given proper, consistent digitization is performed.