Evaluating computer-aided detection algorithms

Computerized detection of breast lesions using deformable part models in ultrasound images

Ultrasound imaging is considered an important complementary technique for the screening of dense breasts. Detection of lesions at an early stage is a key step in which computerized lesion detection systems could play an important role in the analysis of US images. In this article, we propose adaptation of a generic object detection technique, deformable part models, to detect lesions in breast US images. The data set used in this study included 326 images, all from different patients (54 malignant lesions, 109 benign lesions and 163 healthy breasts). In terms of lesion detection, our proposal outperformed some of the most relevant approaches described in the literature; we obtained a sensitivity of 86% with 0.28 false-positive detection per image and an Az value of 0.975. In the detection of malignant lesions, our proposed approached had an Az value of 0.93 and a sensitivity of 78% at a 1.15 false-positive detections per image.

Evaluation of computer-aided detection and diagnosis systems

Computer-aided detection and diagnosis (CAD) systems are increasingly being used as an aid by clinicians for detection and interpretation of diseases. Computer-aided detection systems mark regions of an image that may reveal specific abnormalities and are used to alert clinicians to these regions during image interpretation. Computer-aided diagnosis systems provide an assessment of a disease using image-based information alone or in combination with other relevant diagnostic data and are used by clinicians as a decision support in developing their diagnoses. While CAD systems are commercially available, standardized approaches for evaluating and reporting their performance have not yet been fully formalized in the literature or in a standardization effort. This deficiency has led to difficulty in the comparison of CAD devices and in understanding how the reported performance might translate into clinical practice. To address these important issues, the American Association of Physicists in Medicine (AAPM) formed the Computer Aided Detection in Diagnostic Imaging Subcommittee (CADSC), in part, to develop recommendations on approaches for assessing CAD system performance. The purpose of this paper is to convey the opinions of the AAPM CADSC members and to stimulate the development of consensus approaches and "best practices" for evaluating CAD systems. Both the assessment of a standalone CAD system and the evaluation of the impact of CAD on end-users are discussed. It is hoped that awareness of these important evaluation elements and the CADSC recommendations will lead to further development of structured guidelines for CAD performance assessment. Proper assessment of CAD system performance is expected to increase the understanding of a CAD system's effectiveness and limitations, which is expected to stimulate further research and development efforts on CAD technologies, reduce problems due to improper use, and eventually improve the utility and efficacy of CAD in clinical practice.

Mammographic image denoising and enhancement using the Anscombe transformation, adaptive wiener filtering, and the modulation transfer function

A new restoration methodology is proposed to enhance mammographic images through the improvement of contrast features and the simultaneous suppression of noise. Denoising is performed in the first step using the Anscombe transformation to convert the signal-dependent quantum noise into an approximately signal-independent Gaussian additive noise. In the Anscombe domain, noise is filtered through an adaptive Wiener filter, whose parameters are obtained by considering local image statistics. In the second step, a filter based on the modulation transfer function of the imaging system in the whole radiation field is applied for image enhancement. This methodology can be used as a preprocessing module for computer-aided detection (CAD) systems to improve the performance of breast cancer screening. A preliminary assessment of the restoration algorithm was performed using synthetic images with different levels of quantum noise. Afterward, we evaluated the effect of the preprocessing on the performance of a previously developed CAD system for clustered microcalcification detection in mammographic images. The results from the synthetic images showed an increase of up to 11.5 dB (p = 0.002) in the peak signal-to-noise ratio. Moreover, the mean structural similarity index increased up to 8.3 % (p < 0.001). Regarding CAD performance, the results suggested that the preprocessing increased the detectability of microcalcifications in mammographic images without increasing the false-positive rates. Receiver operating characteristic analysis revealed an average increase of 14.1 % (p = 0.01) in overall CAD performance when restored image sets were used.

Application of threshold-bias independent analysis to eye-tracking and FROC data

RATIONALE AND OBJECTIVES

Studies of medical image interpretation have focused on either assessing radiologists' performance using, for example, the receiver operating characteristic (ROC) paradigm, or assessing the interpretive process by analyzing their eye-tracking (ET) data. Analysis of ET data has not benefited from threshold-bias independent figures of merit (FOMs) analogous to the area under the receiver operating characteristic (ROC) curve. The aim was to demonstrate the feasibility of such FOMs and to measure the agreement between FOMs derived from free-response ROC (FROC) and ET data.

METHODS

Eight expert breast radiologists interpreted a case set of 120 two-view mammograms while eye-position data and FROC data were continuously collected during the interpretation interval. Regions that attract prolonged (>800 ms) visual attention were considered to be virtual marks, and ratings based on the dwell and approach-rate (inverse of time-to-hit) were assigned to them. The virtual ratings were used to define threshold-bias independent FOMs in a manner analogous to the area under the trapezoidal alternative FROC (AFROC) curve (0 = worst, 1 = best). Agreement at the case level (0.5 = chance, 1 = perfect) was measured using the jackknife and 95% confidence intervals (CI) for the FOMs and agreement were estimated using the bootstrap.

RESULTS

The AFROC mark-ratings' FOM was largest at 0.734 (CI 0.65-0.81) followed by the dwell at 0.460 (0.34-0.59) and then by the approach-rate FOM 0.336 (0.25-0.46). The differences between the FROC mark-ratings' FOM and the perceptual FOMs were significant (P < .05). All pairwise agreements were significantly better then chance: ratings vs. dwell 0.707 (0.63-0.88), dwell vs. approach-rate 0.703 (0.60-0.79) and rating vs. approach-rate 0.606 (0.53-0.68). The ratings vs. approach-rate agreement was significantly smaller than the dwell vs. approach-rate agreement (P = .008).

CONCLUSIONS

Leveraging current methods developed for analyzing observer performance data could complement current ways of analyzing ET data and lead to new insights.

Improving performance of computer-aided detection of masses by incorporating bilateral mammographic density asymmetry: an assessment

RATIONALE AND OBJECTIVES

Bilateral mammographic density asymmetry is a promising indicator in assessing risk of having or developing breast cancer. This study aims to assess the performance improvement of a computer-aided detection (CAD) scheme in detecting masses by incorporating bilateral mammographic density asymmetrical information.

MATERIALS AND METHODS

A testing dataset containing 2400 full-field digital mammograms (FFDM) acquired from 600 examination cases was established. Among them, 300 were positive cases with verified cancer associated with malignant masses and 300 were negative cases. Two computerized schemes were applied to process images of each case. The first single-image based CAD scheme detected suspicious mass regions and the second scheme computed average and difference of mammographic tissue density depicted between the left and right breast. A fusion method based on rotation of the CAD scoring projection reference axis was then applied to combine CAD-generated mass detection scores and either the computed average or difference (asymmetry) of bilateral mammographic density scores. The CAD performance levels with and without incorporating mammographic density information were evaluated and compared using a free-response receiver operating characteristic type data analysis method.

RESULTS

CAD achieved a case-based mass detection sensitivity of 0.74 and a region-based sensitivity of 0.56 at a false-positive rate of 0.25 per image. By fusing the CAD and bilateral mammographic density asymmetry scores, the case-based and region-based sensitivity levels of the CAD scheme were increased to 0.84 and 0.69, respectively, at the same false-positive rate. Fusion with average mammographic density only slightly increased CAD sensitivity to 0.75 (case-based) and 0.57 (region-based).

CONCLUSIONS

This study indicated that 1) bilateral mammographic density asymmetry was a stronger indicator of the case depicting suspicious masses than the average density computed from two breasts and 2) fusion between the conventional CAD scores and bilateral mammographic density asymmetry information could substantially increase CAD performance in mass detection.

Improving the performance of computer-aided detection of subtle breast masses using an adaptive cueing method

Current computer-aided detection (CAD) schemes for detecting mammographic masses have several limitations including high correlation with radiologists' detection and cueing most subtle masses only on one view. To increase CAD sensitivity in cueing more subtle masses that are likely missed and/or overlooked by radiologists without increasing false-positive rates, we investigated a new case-dependent cueing method by combining the original CAD-generated detection scores with a computed bilateral mammographic density asymmetry index. Using the new method, we adaptively raise the CAD-generated scores of the regions detected on 'high-risk' cases to cue more subtle mass regions and reduce the CAD scores of the regions detected on 'low-risk' cases to discard more false-positive regions. A testing dataset involving 78 positive and 338 negative cases was used to test this adaptive cueing method. Each positive case involves two sequential examinations in which the mass was detected in 'current' examination and missed in 'prior' examination but detected in a retrospective review by radiologists. Applying to this dataset, a pre-optimized CAD scheme yielded 75% case-based and 55% region-based sensitivity on 'current' examinations at a false-positive rate of 0.25 per image. CAD sensitivity was reduced to 42% (case based) and 27% (region based) on 'prior' examinations. Using the new cueing method, case-based and region-based sensitivity could maximally increase 9% and 33% on the 'prior' examinations, respectively. The percentages of the masses cued on two views also increased from 27% to 65%. The study demonstrated that using this adaptive cueing method enabled us to help CAD cue more subtle cancers without increasing the false-positive cueing rate.

Computer-assisted detection of infectious lung diseases: a review

Respiratory tract infections are a leading cause of death and disability worldwide. Although radiology serves as a primary diagnostic method for assessing respiratory tract infections, visual analysis of chest radiographs and computed tomography (CT) scans is restricted by low specificity for causal infectious organisms and a limited capacity to assess severity and predict patient outcomes. These limitations suggest that computer-assisted detection (CAD) could make a valuable contribution to the management of respiratory tract infections by assisting in the early recognition of pulmonary parenchymal lesions, providing quantitative measures of disease severity and assessing the response to therapy. In this paper, we review the most common radiographic and CT features of respiratory tract infections, discuss the challenges of defining and measuring these disorders with CAD, and propose some strategies to address these challenges.

A software framework for building biomedical machine learning classifiers through grid computing resources

This paper describes the BiomedTK software framework, created to perform massive explorations of machine learning classifiers configurations for biomedical data analysis over distributed Grid computing resources. BiomedTK integrates ROC analysis throughout the complete classifier construction process and enables explorations of large parameter sweeps for training third party classifiers such as artificial neural networks and support vector machines, offering the capability to harness the vast amount of computing power serviced by Grid infrastructures. In addition, it includes classifiers modified by the authors for ROC optimization and functionality to build ensemble classifiers and manipulate datasets (import/export, extract and transform data, etc.). BiomedTK was experimentally validated by training thousands of classifier configurations for representative biomedical UCI datasets reaching in little time classification levels comparable to those reported in existing literature. The comprehensive method herewith presented represents an improvement to biomedical data analysis in both methodology and potential reach of machine learning based experimentation.

Improving a CAD system using bilateral information

Computer Aided Detection (CAD) mammographic systems are used in medicine to assist radiologists in the evaluation of mammographic images. The aim of this work is to compare the results of a developed single-image CAD system with a new one, dual-image CAD, that adds registration information of bilateral mammographic images in the training step of the former system. The evaluation of the different registration methods is performed using similarity measures. Receiver Operating Characteristic (ROC) analysis and Free Receiver Operating Characteristics (FROC) analysis are used to compare the results of both CAD systems. At a sensitivity of 80%, the false positives per image was 1.68 for the single-image CAD system and 0.90 for the dual-image CAD system. The results shows the benefits of integrating bilateral information into the CAD system.

Computer-aided detection; the effect of training databases on detection of subtle breast masses

RATIONALE AND OBJECTIVES

Lesion conspicuity is typically highly correlated with visual difficulty for lesion detection, and computer-aided detection (CAD) has been widely used as a "second reader" in mammography. Hence, increasing CAD sensitivity in detecting subtle cancers without increasing false-positive rates is important. The aim of this study was to investigate the effect of training database case selection on CAD performance in detecting low-conspicuity breast masses.

MATERIALS AND METHODS

A full-field digital mammographic image database that included 525 cases depicting malignant masses was randomly partitioned into three subsets. A CAD scheme was applied to detect all initially suspected mass regions and compute region conspicuity. Training samples were iteratively selected from two of the subsets. Four types of training data sets-(1) one including all available true-positive mass regions in the two subsets ("all"), (2) one including 350 randomly selected mass regions ("diverse"), (3) one including 350 high-conspicuity mass regions ("easy"), and (4) one including 350 low-conspicuity mass regions ("difficult")-were assembled. In each training data set, the same number of randomly selected false-positive regions as the true-positives were also included. Two classifiers, an artificial neural network (ANN) and a k-nearest neighbor (KNN) algorithm, were trained using each of the four training data sets and tested on all suspected regions in the remaining data set. Using a threefold cross-validation method, the performance changes of the CAD schemes trained using one of the four training data sets were computed and compared.

RESULTS

CAD initially detected 1025 true-positive mass regions depicted on 507 cases (97% case-based sensitivity) and 9569 false-positive regions (3.5 per image) in the entire database. Using the all training data set, CAD achieved the highest overall performance on the entire testing database. However, CAD detected the highest number of low-conspicuity masses when the difficult training data set was used for training. Results did agree for both ANN-based and KNN-based classifiers in all tests. Compared to the use of the all training data set, the sensitivity of the schemes trained using the difficult data set decreased by 8.6% and 8.4% for the ANN and KNN algorithm on the entire database, respectively, but the detection of low-conspicuity masses increased by 7.1% and 15.1% for the ANN and KNN algorithm at a false-positive rate of 0.3 per image.

CONCLUSIONS

CAD performance depends on the size, diversity, and difficulty level of the training database. To increase CAD sensitivity in detecting subtle cancer, one should increase the fraction of difficult cases in the training database rather than simply increasing the training data set size.

Consistency of methods for analysing location-specific data

Although the receiver operating characteristic (ROC) method is the acknowledged gold-standard for imaging system assessment, it ignores localisation information and differentiation between multiple abnormalities per case. As the free-response ROC (FROC) method uses localisation information and more closely resembles the clinical reporting process, it is being increasingly used. A number of methods have been proposed to analyse the data that result from an FROC study: jackknife alternative FROC (JAFROC) and a variant termed JAFROC1, initial detection and candidate analysis (IDCA) and ROC analysis via the reduction of the multiple ratings on a case to a single rating. The focus of this paper was to compare JAFROC1, IDCA and the ROC analysis methods using a clinical FROC human data set. All methods agreed on the ordering of the modalities and all yielded statistically significant differences of the figures-of-merit, i.e. p < 0.05. Both IDCA and JAFROC1 yielded much smaller p-values than ROC. The results are consistent with a recent simulation-based validation study comparing these and other methods. In conclusion, IDCA or JAFROC1 analysis of FROC human data may be superior at detecting modality differences than ROC analysis.

Grid infrastructures for developing mammography CAD systems

This paper presents a set of technologies developed to exploit Grid infrastructures for breast cancer CAD, that include (1) federated repositories of mammography images and clinical data over Grid storage, (2) a workstation for mammography image analysis and diagnosis and (3) a framework for data analysis and training machine learning classifiers over Grid computing power specially tuned for medical image based data. An experimental mammography digital repository of approximately 300 mammograms from the MIAS database was created and classifiers were built achieving a 0.85 average area under the ROC curve in a dataset of 100 selected mammograms with representative pathological lesions and normal cases. Similar results were achieved with classifiers built for the UCI Breast Cancer Wisconsin dataset (699 features vectors). Now these technologies are being validated in a real medical environment at the Faculty of Medicine in Porto University after a process of integrating the tools within the clinicians workflows and IT systems.

Computer-aided detection (CAD) as a second reader using perspective filet view at CT colonography: effect on performance of inexperienced readers

AIM

To evaluate whether computer-aided detection (CAD) as a second reader using perspective filet view [three-dimensional (3D) filet] improves the performance of inexperienced readers at computed tomography colonography (CTC) compared with unassisted 3D filet and unassisted two-dimensional (2D) CTC.

MATERIAL AND METHODS

Fifty symptomatic patients underwent CTC and same-day colonoscopy with segmental unblinding. Two inexperienced readers read the CTC studies on 3D filet and 2D several weeks apart. Four months later, readers re-read the cases only evaluating CAD marks using 3D filet. Suspicious CAD marks not previously described on 3D filet were recorded. Jackknife free-response receiver operating characteristic (JAFROC-1) analysis was used to compare the observers' performances in detecting lesions with 3D filet, 2D and 3D filet with CAD.

RESULTS

One hundred and three lesions > or =3mm were detected at colonoscopy with segmental unblinding. CAD alone had a sensitivity of 73% (75/103) at a mean false-positive rate per patient of 12.8 in supine and 11.4 in prone. For inexperienced readers sensitivities with 3D filet with CAD were 58% (60/103) and 48% (50/103) with an improvement of 14-16 percentage points (p<0.05) compared with 2D and of 10-11 percentage points (p<0.05) compared with 3D filet. For inexperienced readers, the false-positive rate was 25-41% and 71-200% higher with 3D filet with CAD compared with 3D filet and 2D, respectively. JAFROC-1 analysis showed no significant differences in per-lesion overall performance among reading modes (p=0.8).

CONCLUSION

CAD applied as a second reader using 3D filet increased both sensitivity and the number of false positives by inexperienced readers compared with 3D filet and 2D, thus not improving overall performance, i.e., the ability to distinguish between lesions and non-lesions.

Improving performance of computer-aided detection scheme by combining results from two machine learning classifiers

RATIONALE AND OBJECTIVES

Global data-based and local instance-based machine-learning methods and classifiers have been widely used to optimize computer-aided detection and diagnosis (CAD) schemes to classify between true-positive and false-positive detections. In this study, the correlation between these two types of classifiers was investigated using a new independent testing data set, and the potential improvement of a CAD scheme's performance by combining the results of the two classifiers in detecting breast masses was assessed.

MATERIALS AND METHODS

The CAD scheme first used image filtering and a multilayer topographic region growth algorithm to detect and segment suspicious mass regions. The scheme then used an image feature-based classifier to classify these regions into true-positive and false-positive regions. Two classifiers were used in this study. One was a global data-based machine-learning classifier, an artificial neural network (ANN), and the other was a local instance-based machine-learning classifier, a k-nearest neighbor (KNN) algorithm. An independent image database including 400 mammographic examinations was used in this study. Of these, 200 were cancer cases and 200 were negative cases. The preoptimized CAD scheme was applied twice to the database using the two different classifiers. The correlation between the two sets of classification results was analyzed. Three sets of CAD performance results using the ANN, KNN, and average detection scores from both classifiers were assessed and compared using the free-response receiver-operating characteristic method.

RESULTS

The results showed that the ANN achieved higher performance than the KNN algorithm, with a normalized area under the performance curve (AUC) of 0.891 versus 0.845. The correlation coefficients between the detection scores generated by the two classifiers were 0.436 and 0.161 for the true-positive and false-positive detections, respectively. The average detection scores of the two classifiers improved CAD performance and reliability by increasing the AUC to 0.912 and reducing the standard error of the estimated AUC by 14.4%. The detection sensitivity was also increased from 75.8% (ANN) and 65.9% (KNN) to 80.3% at a false-positive detection rate of 0.3 per image.

CONCLUSIONS

This study demonstrates that two global data-based and local data-based machine-learning classifiers (ANN and KNN) generated low correlated detection results and that combining the detection scores of these two classifiers significantly improved overall CAD performance (P < .01) and reduced standard error in CAD performance assessment.

Validation and statistical power comparison of methods for analyzing free-response observer performance studies

RATIONALE AND OBJECTIVES

The aim of this work was to validate and compare the statistical powers of proposed methods for analyzing free-response data using a search-model-based simulator.

MATERIALS AND METHODS

A free-response data simulator is described that can model a single reader interpreting the same cases in two modalities, or two computer-aided detection (CAD) algorithms, or two human observers, interpreting the same cases in one modality. A variance components model, analogous to the Roe and Metz receiver-operating characteristic (ROC) data simulator, is described; it models intracase and intermodality correlations in free-response studies. Two generic observers were simulated: a quasi-human observer and a quasi-CAD algorithm. Null hypothesis (NH) validity and statistical powers of ROC, jackknife alternative free-response operating characteristic (JAFROC), a variant of JAFROC termed JAFROC-1, initial detection and candidate analysis (IDCA), and a nonparametric (NP) approach were investigated.

RESULTS

All methods had valid NH behavior over a wide range of simulator parameters. For equal numbers of normal and abnormal cases, for the human observer, the statistical power ranking of the methods was JAFROC-1 > JAFROC > (IDCA approximately NP) > ROC. For the CAD algorithm, the ranking was (NP approximately IDCA) > (JAFROC-1 approximately JAFROC) > ROC. In either case, the statistical power of the highest ranked method exceeded that of the lowest ranked method by about a factor of two. Dependence of statistical power on simulator parameters followed expected trends. For data sets with more abnormal cases than normal cases, JAFROC-1 power significantly exceeded JAFROC power.

CONCLUSION

Based on this work, the recommendation is to use JAFROC-1 for human observers (including human observers with CAD assist) and the NP method for evaluating CAD algorithms.

Performance assessments of diagnostic systems under the FROC paradigm: experimental, analytical, and results interpretation issues

As use of free response receiver-operating characteristic (FROC) curves gains more acceptance for quantitatively assessing the performance of diagnostic systems, it is important that the experimentalist understands the possible role of this approach as one of the experimental design paradigms that are available to him or her among all other approaches as well as some of the issues associated with FROC type studies. In a number of experimental scenarios, the FROC paradigm and associated analytical tools have theoretical and practical advantages over both the binary and the ROC approaches to performance assessments of diagnostic systems, but it also has some limitations related to experimental design, data analyses, clinical relevance, and complexity in the interpretation of the results. These issues are rarely discussed and are the focus of this work.

Operating characteristics predicted by models for diagnostic tasks involving lesion localization

In 1996 Swensson published an observer model that predicted receiver operating characteristic (ROC), localization ROC (LROC), free-response ROC (FROC) and alternative FROC (AFROC) curves, thereby achieving "unification" of different observer performance paradigms. More recently a model termed initial detection and candidate analysis (IDCA) has been proposed for fitting computer aided detection (CAD) generated FROC data, and recently a search model for human observer FROC data has been proposed. The purpose of this study was to derive IDCA and the search model based expressions for operating characteristics, and to compare the predictions to the Swensson model. For three out of four mammography CAD data sets all models yielded good fits in the high-confidence region, i.e., near the lower end of the plots. The search model and IDCA tended to better fit the data in the low-confidence region, i.e., near the upper end of the plots, particularly for FROC curves for which the Swensson model predictions departed markedly from the data. For one data set none of the models yielded satisfactory fits. A unique characteristic of search model and IDCA predicted operating characteristics is that the operating point is not allowed to move continuously to the lowest confidence limit of the corresponding Swensson model curves. This prediction is actually observed in the CAD raw data and it is the primary reason for the poor FROC fits of the Swensson model in the low-confidence region.