Potential effect of different radiologist reporting methods on studies showing benefit of CAD

A brief history of free-response receiver operating characteristic paradigm data analysis

In the receiver operating characteristic paradigm the observer assigns a single rating to each image and the location of the perceived abnormality, if any, is ignored. In the free-response receiver operating characteristic paradigm the observer is free to mark and rate as many suspicious regions as are considered clinically reportable. Credit for a correct localization is given only if a mark is sufficiently close to an actual lesion; otherwise, the observer's mark is scored as a location-level false positive. Until fairly recently there existed no accepted method for analyzing the resulting relatively unstructured data containing random numbers of mark-rating pairs per image. This report reviews the history of work in this field, which has now spanned more than five decades. It introduces terminology used to describe the paradigm, proposed measures of performance (figures of merit), ways of visualizing the data (operating characteristics), and software for analyzing free-response receiver operating characteristic studies.

Comparison of Breast Cancer Screening Results in Korean Middle-Aged Women: A Hospital-based Prospective Cohort Study

Objectives

The aim of this hospital-based prospective study was to evaluate the diagnostic ability of breast cancer screening in Korean middle-aged women using age, ultrasonography, mammography, and magnification mammography, which are commonly used in most hospitals.

Methods

A total of 21 patents were examined using ultrasonography, mammography, and magnification mammography, and their data were prospectively analyzed from August 2011 to March 2013. All patients were divided into benign and malignant groups and the screening results were classified using the American College of Radiology Breast Imaging Reporting and Data System (BI-RADS). The final pathology report was used as the reference standard and the sensitivity and specificity of ultrasonography, mammography, and magnification mammography were evaluated using receiver-operating characteristics (ROC) analysis.

Results

The analysis included 21 patients who underwent biopsy. Among them, three (14.3%) were positive and 18 (85.7%) negative for breast cancer. The average age was 50.5 years (range = 38–61 years). The sensitivity was the same for ultrasonography and magnification mammography and the specificity of magnification mammography was higher than that of ultrasonography. The highest area under the ROC curve (AUC) was observed in the combination of age and magnification mammography (1.000) and the decreasing order of AUC in others was magnification mammography (0.833), ultrasonography (0.787), mammography (0.667), and age (0.648).

Conclusions

In Korean women, the diagnostic accuracy of magnification mammography was better than that of ultrasonography and mammography. The combination of age and magnification mammography increased the sensitivity and diagnostic accuracy.

Breast image analysis for risk assessment, detection, diagnosis, and treatment of cancer

The role of breast image analysis in radiologists' interpretation tasks in cancer risk assessment, detection, diagnosis, and treatment continues to expand. Breast image analysis methods include segmentation, feature extraction techniques, classifier design, biomechanical modeling, image registration, motion correction, and rigorous methods of evaluation. We present a review of the current status of these task-based image analysis methods, which are being developed for the various image acquisition modalities of mammography, tomosynthesis, computed tomography, ultrasound, and magnetic resonance imaging. Depending on the task, image-based biomarkers from such quantitative image analysis may include morphological, textural, and kinetic characteristics and may depend on accurate modeling and registration of the breast images. We conclude with a discussion of future directions.

Evaluating imaging and computer-aided detection and diagnosis devices at the FDA

This report summarizes the Joint FDA-MIPS Workshop on Methods for the Evaluation of Imaging and Computer-Assist Devices. The purpose of the workshop was to gather information on the current state of the science and facilitate consensus development on statistical methods and study designs for the evaluation of imaging devices to support US Food and Drug Administration submissions. Additionally, participants expected to identify gaps in knowledge and unmet needs that should be addressed in future research. This summary is intended to document the topics that were discussed at the meeting and disseminate the lessons that have been learned through past studies of imaging and computer-aided detection and diagnosis device performance.

Evaluation of the accuracy of a computer-aided diagnosis (CAD) system in breast ultrasound according to the radiologist's experience

RATIONALE AND OBJECTIVES

The aim of this study was to evaluate the performance of a computer-aided diagnosis (CAD) system for breast ultrasound to improve the characterization of breast lesions detected on ultrasound by junior and senior radiologists.

MATERIALS AND METHODS

One hundred sixty ultrasound breast lesions were randomly reviewed blindly by four radiologists with different levels of expertise (from 20 years [radiologist A] to 4 months [radiologist D]), with and without the help of an ultrasound CAD system (B-CAD version 2). All lesions had been biopsied. Sensitivity and specificity with and without CAD were calculated for each radiologist for the following evaluation criteria: Breast Imaging Reporting and Data System category and the final diagnosis (benign or malignant). Intrinsic sensitivity, specificity, and predictive values of CAD alone were also calculated.

RESULTS

CAD detected all cancers, and its use increased radiologists' sensitivity scores when this was possible (with vs without CAD: radiologist A, 99% vs 99%; radiologist B, 96% vs 87%; radiologist C, 95% vs 88%; radiologist D, 91% vs 88%). Seven additional cancers were diagnosed. However, the low specificity of CAD (48%) decreased the specificity of radiologists, especially of the more experienced among them (with vs without CAD: radiologist A, 46% vs 70%; radiologist B, 58% vs 80%; radiologist C, 57% vs 69%; radiologist D, 71% vs 71%).

CONCLUSIONS

CAD for breast ultrasound appears to be a useful tool for improving the diagnosis of malignant lesions for junior radiologists. Nevertheless, its low specificity must be taken into account to limit biopsies of benign lesions.

Computer-aided classification of breast masses: performance and interobserver variability of expert radiologists versus residents

PURPOSE

To evaluate the interobserver variability in descriptions of breast masses by dedicated breast imagers and radiology residents and determine how any differences in lesion description affect the performance of a computer-aided diagnosis (CAD) computer classification system.

MATERIALS AND METHODS

Institutional review board approval was obtained for this HIPAA-compliant study, and the requirement to obtain informed consent was waived. Images of 50 breast lesions were individually interpreted by seven dedicated breast imagers and 10 radiology residents, yielding 850 lesion interpretations. Lesions were described with use of 11 descriptors from the Breast Imaging Reporting and Data System, and interobserver variability was calculated with the Cohen κ statistic. Those 11 features were selected, along with patient age, and merged together by a linear discriminant analysis (LDA) classification model trained by using 1005 previously existing cases. Variability in the recommendations of the computer model for different observers was also calculated with the Cohen κ statistic.

RESULTS

A significant difference was observed for six lesion features, and radiology residents had greater interobserver variability in their selection of five of the six features than did dedicated breast imagers. The LDA model accurately classified lesions for both sets of observers (area under the receiver operating characteristic curve = 0.94 for residents and 0.96 for dedicated imagers). Sensitivity was maintained at 100% for residents and improved from 98% to 100% for dedicated breast imagers. For residents, the computer model could potentially improve the specificity from 20% to 40% (P < .01) and the κ value from 0.09 to 0.53 (P < .001). For dedicated breast imagers, the computer model could increase the specificity from 34% to 43% (P = .16) and the κ value from 0.21 to 0.61 (P < .001).

CONCLUSION

Among findings showing a significant difference, there was greater interobserver variability in lesion descriptions among residents; however, an LDA model using data from either dedicated breast imagers or residents yielded a consistently high performance in the differentiation of benign from malignant breast lesions, demonstrating potential for improving specificity and decreasing interobserver variability in biopsy recommendations.

Multimodality computer-aided breast cancer diagnosis with FFDM and DCE-MRI

RATIONALE AND OBJECTIVES

To investigate a multimodality computer-aided diagnosis (CAD) scheme that combines image information from full-field digital mammography (FFDM) and dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) for computerized breast cancer classification.

MATERIALS AND METHODS

From a retrospective FFDM database with 432 lesions (255 malignant, 177 benign) and a retrospective DCE-MRI database including 476 lesions (347 malignant, 129 benign), we constructed a multimodality dataset of 213 lesions (168 malignant, 45 benign). Each lesion was present on both FFDM and DCE-MRI images and deemed to be a difficult case given the necessity of having both clinical imaging exams. Using a manually indicated lesion location (ie, a seed point on FFDM images or a region of interest on DCE-MRI images, the computer automatically segmented the mass lesions and extracted lesion features). A subset of features was selected using linear stepwise feature selection and merged by a Bayesian artificial neural network to yield an estimate of the probability of malignancy. Receiver operating characteristic (ROC) analysis was used to evaluate the performance of the selected features in distinguishing between malignant and benign lesions.

RESULTS

With leave-one-lesion-out cross-validation on the multimodality dataset, the mammography-only features yielded an area under the ROC curve (AUC) of 0.74 +/- 0.04, and the DCE-MRI-only features yielded an AUC of 0.78 +/- 0.04. The combination of these two modalities, which included a spiculation feature from mammography and two kinetic features from DCE-MRI, yielded an AUC of 0.87 +/- 0.03. The improvement of combining multimodality information was statistically significant as compared to the use of single modality information alone.

CONCLUSIONS

A CAD scheme that combines features extracted from FFDM and DCE-MRI images may be advantageous to single-modality CAD in the task of differentiating between malignant and benign lesions.

Repeatability in computer-aided diagnosis: application to breast cancer diagnosis on sonography

PURPOSE

The aim of this study was to investigate the concept of repeatability in a case-based performance evaluation of two classifiers commonly used in computer-aided diagnosis in the task of distinguishing benign from malignant lesions.

METHODS

The authors performed .632+ bootstrap analyses using a data set of 1251 sonographic lesions of which 212 were malignant. Several analyses were performed investigating the impact of sample size and number of bootstrap iterations. The classifiers investigated were a Bayesian neural net (BNN) with five hidden units and linear discriminant analysis (LDA). Both used the same four input lesion features. While the authors did evaluate classifier performance using receiver operating characteristic (ROC) analysis, the main focus was to investigate case-based performance based on the classifier output for individual cases, i.e., the classifier outputs for each test case measured over the bootstrap iterations. In this case-based analysis, the authors examined the classifier output variability and linked it to the concept of repeatability. Repeatability was assessed on the level of individual cases, overall for all cases in the data set, and regarding its dependence on the case-based classifier output. The impact of repeatability was studied when aiming to operate at a constant sensitivity or specificity and when aiming to operate at a constant threshold value for the classifier output.

RESULTS

The BNN slightly outperformed the LDA with an area under the ROC curve of 0.88 versus 0.85 (p < 0.05). In the repeatability analysis on an individual case basis, it was evident that different cases posed different degrees of difficulty to each classifier as measured by the by-case output variability. When considering the entire data set, however, the overall repeatability of the BNN classifier was lower than for the LDA classifier, i.e., the by-case variability for the BNN was higher. The dependence of the by-case variability on the average by-case classifier output was markedly different for the classifiers. The BNN achieved the lowest variability (best repeatability) when operating at high sensitivity (> 90%) and low specificity (< 66%), while the LDA achieved this at moderate sensitivity (approximately 74%) and specificity (approximately 84%). When operating at constant 90% sensitivity or constant 90% specificity, the width of the 95% confidence intervals for the corresponding classifier output was considerable for both classifiers and increased for smaller sample sizes. When operating at a constant threshold value for the classifier output, the width of the 95% confidence intervals for the corresponding sensitivity and specificity ranged from 9 percentage points (pp) to 30 pp.

CONCLUSIONS

The repeatability of the classifier output can have a substantial effect on the obtained sensitivity and specificity. Knowledge of classifier repeatability, in addition to overall performance level, is important for successful translation and implementation of computer-aided diagnosis in clinical decision making.

Anniversary paper: History and status of CAD and quantitative image analysis: the role of Medical Physics and AAPM

The roles of physicists in medical imaging have expanded over the years, from the study of imaging systems (sources and detectors) and dose to the assessment of image quality and perception, the development of image processing techniques, and the development of image analysis methods to assist in detection and diagnosis. The latter is a natural extension of medical physicists' goals in developing imaging techniques to help physicians acquire diagnostic information and improve clinical decisions. Studies indicate that radiologists do not detect all abnormalities on images that are visible on retrospective review, and they do not always correctly characterize abnormalities that are found. Since the 1950s, the potential use of computers had been considered for analysis of radiographic abnormalities. In the mid-1980s, however, medical physicists and radiologists began major research efforts for computer-aided detection or computer-aided diagnosis (CAD), that is, using the computer output as an aid to radiologists-as opposed to a completely automatic computer interpretation-focusing initially on methods for the detection of lesions on chest radiographs and mammograms. Since then, extensive investigations of computerized image analysis for detection or diagnosis of abnormalities in a variety of 2D and 3D medical images have been conducted. The growth of CAD over the past 20 years has been tremendous-from the early days of time-consuming film digitization and CPU-intensive computations on a limited number of cases to its current status in which developed CAD approaches are evaluated rigorously on large clinically relevant databases. CAD research by medical physicists includes many aspects-collecting relevant normal and pathological cases; developing computer algorithms appropriate for the medical interpretation task including those for segmentation, feature extraction, and classifier design; developing methodology for assessing CAD performance; validating the algorithms using appropriate cases to measure performance and robustness; conducting observer studies with which to evaluate radiologists in the diagnostic task without and with the use of the computer aid; and ultimately assessing performance with a clinical trial. Medical physicists also have an important role in quantitative imaging, by validating the quantitative integrity of scanners and developing imaging techniques, and image analysis tools that extract quantitative data in a more accurate and automated fashion. As imaging systems become more complex and the need for better quantitative information from images grows, the future includes the combined research efforts from physicists working in CAD with those working on quantitative imaging systems to readily yield information on morphology, function, molecular structure, and more-from animal imaging research to clinical patient care. A historical review of CAD and a discussion of challenges for the future are presented here, along with the extension to quantitative image analysis.