Application of likelihood ratio to classification of mammographic masses; performance comparison to case-based reasoning

Observer study of a prototype clinical decision support system for breast cancer diagnosis using dynamic contrast-enhanced MRI

OBJECTIVE

The purpose of this article is to evaluate the performance of radiologists using a prototype clinical decision support system to diagnose and manage patients with breast cancer based on dynamic contrast-enhanced MRI studies.

MATERIALS AND METHODS

The study was conducted with three breast radiologists and two breast imaging fellows who gave patient treatment recommendations and confidence ratings, both without and with computer aid. The computer aid presented similar cases from a retrieval database of 192 lesions (96 malignant and 96 benign) for a test set of 97 mass lesions (46 malignant and 51 benign). The performance of each observer was quantified by receiver operating characteristic analysis. The radiologists' confidence in their recommendations was analyzed with respect to the query case pathologic diagnosis, perceived usefulness of the similar cases, and the accuracy of the computer in retrieving cases of the correct diagnosis. The statistical significance in the performance measure differences was determined by using a two-tailed Student t test for paired data.

RESULTS

For each observer, the area under the receiver operating characteristic curve did not change significantly with the use of the computer aid (from a mean of 0.8 to a mean of 0.8; p = 0.61). The average confidence of three of the five observers increased significantly with the computer aid (from 5.9 to 6.3 [p < 0.001], from 7.0 to 7.2 [p = 0.04], and from 4.4 to 5.4 [p < 0.001], respectively). The confidence change of the radiologists was more frequent and larger for malignant lesions where the computer was correct. However, for benign lesions, even when the computer was correct, the confidence of the radiologists did not necessarily change.

CONCLUSION

The presentation of similar cases reinforced radiologists' confidence rating in the diagnosis of malignant lesions; however, it did not change their confidence rating for benign lesions or reduce the number of unnecessary biopsies in managing patients with breast cancer using dynamic contrast-enhanced MRI under the limited study conditions.

Multi-modality CADx: ROC study of the effect on radiologists' accuracy in characterizing breast masses on mammograms and 3D ultrasound images

RATIONALE AND OBJECTIVES

To investigate the effect of a computer-aided diagnosis (CADx) system on radiologists' performance in discriminating malignant and benign masses on mammograms and three-dimensional (3D) ultrasound (US) images.

MATERIALS AND METHODS

Our dataset contained mammograms and 3D US volumes from 67 women (median age, 51; range: 27-86) with 67 biopsy-proven breast masses (32 benign and 35 malignant). A CADx system was designed to automatically delineate the mass boundaries on mammograms and the US volumes, extract features, and merge the extracted features into a multi-modality malignancy score. Ten experienced readers (subspecialty academic breast imaging radiologists) first viewed the mammograms alone, and provided likelihood of malignancy (LM) ratings and Breast Imaging and Reporting System assessments. Subsequently, the reader viewed the US images with the mammograms, and provided LM and action category ratings. Finally, the CADx score was shown and the reader had the opportunity to revise the ratings. The LM ratings were analyzed using receiver-operating characteristic (ROC) methodology, and the action category ratings were used to determine the sensitivity and specificity of cancer diagnosis.

RESULTS

Without CADx, readers' average area under the ROC curve, A(z), was 0.93 (range, 0.86-0.96) for combined assessment of the mass on both the US volume and mammograms. With CADx, their average A(z) increased to 0.95 (range, 0.91-0.98), which was borderline significant (P = .05). The average sensitivity of the readers increased from 98% to 99% with CADx, while the average specificity increased from 27% to 29%. The change in sensitivity with CADx did not achieve statistical significance for the individual radiologists, and the change in specificity was statistically significant for one of the radiologists.

CONCLUSIONS

A well-trained CADx system that combines features extracted from mammograms and US images may have the potential to improve radiologists' performance in distinguishing malignant from benign breast masses and making decisions about biopsies.

Optimized approach to decision fusion of heterogeneous data for breast cancer diagnosis

As more diagnostic testing options become available to physicians, it becomes more difficult to combine various types of medical information together in order to optimize the overall diagnosis. To improve diagnostic performance, here we introduce an approach to optimize a decision-fusion technique to combine heterogeneous information, such as from different modalities, feature categories, or institutions. For classifier comparison we used two performance metrics: The receiving operator characteristic (ROC) area under the curve [area under the ROC curve (AUC)] and the normalized partial area under the curve (pAUC). This study used four classifiers: Linear discriminant analysis (LDA), artificial neural network (ANN), and two variants of our decision-fusion technique, AUC-optimized (DF-A) and pAUC-optimized (DF-P) decision fusion. We applied each of these classifiers with 100-fold cross-validation to two heterogeneous breast cancer data sets: One of mass lesion features and a much more challenging one of microcalcification lesion features. For the calcification data set, DF-A outperformed the other classifiers in terms of AUC (p < 0.02) and achieved AUC=0.85 +/- 0.01. The DF-P surpassed the other classifiers in terms of pAUC (p < 0.01) and reached pAUC=0.38 +/- 0.02. For the mass data set, DF-A outperformed both the ANN and the LDA (p < 0.04) and achieved AUC=0.94 +/- 0.01. Although for this data set there were no statistically significant differences among the classifiers' pAUC values (pAUC=0.57 +/- 0.07 to 0.67 +/- 0.05, p > 0.10), the DF-P did significantly improve specificity versus the LDA at both 98% and 100% sensitivity (p < 0.04). In conclusion, decision fusion directly optimized clinically significant performance measures, such as AUC and pAUC, and sometimes outperformed two well-known machine-learning techniques when applied to two different breast cancer data sets.

Computer aid for decision to biopsy breast masses on mammography: validation on new cases

RATIONALE AND OBJECTIVES

The purpose of this study was to validate the performance of a previously developed computer aid for breast mass classification for mammography on a new, independent database of cases not used for algorithm development.

MATERIALS AND METHODS

A computer aid (classifier) based on the likelihood ratio (LRb) was previously developed on a database of 670 mass cases. The 670 cases (245 malignant) from one medical institution were described using 16 features from the American College of Radiology Breast Imaging-Reporting and Data System lexicon and patient history findings. A separate database of 151 (43 malignant) validation cases were collected that were previously unseen by the classifier. These new validation cases were evaluated by the classifier without retraining. Performance evaluation methods included Receiver Operating Characteristic (ROC), round-robin, and leave-one-out bootstrap sampling.

RESULTS

The performance of the classifier on the training data yielded an average ROC area of 0.90 +/- 0.02 and partial ROC area (0.90AUC) of 0.60 +/- 0.06. The exact nonparametric performance on the validation set of 151 cases yielded a ROC area of 0.88 and 0.90AUC of 0.57. Using a 100% sensitivity cutoff threshold established on the training data (100% negative predictive value), the classifier correctly identified 100% of the malignant masses in the validation test set, while potentially obviating 26% of the biopsies performed on benign masses.

CONCLUSION

The LRb classifier performed consistently on new data that was not used for classifier development. The LRb classifier shows promise as a potential aid in reducing the number of biopsies performed on benign masses.