Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data

Surface normal overlap: a computer-aided detection algorithm with application to colonic polyps and lung nodules in helical CT

We developed a novel computer-aided detection (CAD) algorithm called the surface normal overlap method that we applied to colonic polyp detection and lung nodule detection in helical computed tomography (CT) images. We demonstrate some of the theoretical aspects of this algorithm using a statistical shape model. The algorithm was then optimized on simulated CT data and evaluated using a per-lesion cross-validation on 8 CT colonography datasets and on 8 chest CT datasets. It is able to achieve 100% sensitivity for colonic polyps 10 mm and larger at 7.0 false positives (FPs)/dataset and 90% sensitivity for solid lung nodules 6 mm and larger at 5.6 FP/dataset.

Ipsilateral-mammogram computer-aided detection of breast cancer

In this paper, an ipsilateral multi-view computer-aided detection (CAD) scheme is presented for mass detection in digital mammograms by exploiting correlative information of suspicious lesions between mammograms of the same breast. After nonlinear tree-structured filtering for image noise suppression, two wavelet-based methods, directional wavelet transform and tree-structured wavelet transform for image enhancement, and adaptive fuzzy C-means algorithm for segmentation are employed on each mammograms of the same breast, respectively, concurrent analysis is developed for iterative analysis of ipsilateral multi-view mammograms by inter-projective feature matching analysis. A supervised artificial neural network is developed as a classifier, in which the back-propagation algorithm combined with Kalman filtering is used as training algorithm, and free-response receiver operating characteristic analysis is used to test the performance of the developed unilateral CAD system. Performance comparison has been conducted between the final ipsilateral multi-view CAD system and our previously developed single-mammogram-based CAD system. The study results demonstrate the advantages of ipsilateral multi-view CAD method combined with concurrent analysis over current single-view CAD system on false positive reduction.

Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model

We have developed a model for FROC curve fitting that relates the observer's FROC performance not to the ROC performance that would be obtained if the observer's responses were scored on a per image basis, but rather to a hypothesized ROC performance that the observer would obtain in the task of classifying a set of "candidate detections" as positive or negative. We adopt the assumptions of the Bunch FROC model, namely that the observer's detections are all mutually independent, as well as assumptions qualitatively similar to, but different in nature from, those made by Chakraborty in his AFROC scoring methodology. Under the assumptions of our model, we show that the observer's FROC performance is a linearly scaled version of the candidate analysis ROC curve, where the scaling factors are just given by the FROC operating point coordinates for detecting initial candidates. Further, we show that the likelihood function of the model parameters given observational data takes on a simple form, and we develop a maximum likelihood method for fitting a FROC curve to this data. FROC and AFROC curves are produced for computer vision observer datasets and compared with the results of the AFROC scoring method. Although developed primarily with computer vision schemes in mind, we hope that the methodology presented here will prove worthy of further study in other applications as well.

Statistical power in observer-performance studies: comparison of the receiver operating characteristic and free-response methods in tasks involving localization

RATIONALE AND OBJECTIVES

Statistical power, defined as the probability of detecting real differences between imaging modalities, determines the cost in terms of readers and cases of conducting receiver operating characteristic (ROC) studies. Neglect of location information in lesion-detection studies analyzed with the ROC method can compromise power. Use of the alternative free-response ROC (AFROC) method, which considers location information, has been discouraged, because it neglects intraimage correlations. The relative statistical powers of the two methods, however, have not been tested. The purpose of this study was to compare the statistical power of ROC and AFROC methods using simulations.

MATERIALS AND METHODS

A new model including intraimage correlations was developed to describe the decision variable sampling and to simulate data for ROC and AFROC analyses. Five readers and 200 cases (half of which contained one signal) were simulated for each trial. Two hundred trials, equally split between the null hypothesis and alternative hypothesis, were run. Ratings were analyzed with the Dorfman-Berbaum-Metz method, and separation of the null hypothesis and alternative hypothesis distributions was calculated.

RESULTS

The AFROC method yielded higher power than the ROC method. Separation of the null hypothesis and alternative hypothesis distributions was larger by a factor of 1.6 regardless of the presence or absence of intraimage correlations. The effect of the incorrect localizations during ROC analysis of localization data is believed to be the major reason for the enhanced power of the AFROC method.

CONCLUSION

The AFROC method can yield higher power than the ROC method for studies involving lesion localization. Greater consideration of this methodology is warranted.

A constrained formulation for the receiver operating characteristic (ROC) curve based on probability summation

We propose a principled formulation of the ROC curve that is constrained in a realistic way by the mechanism of probability summation. The constrained and conventional ROC formulations were fitted to 150 separate sets of rating data taken from previous observer studies of 250 or 529 chest radiographs. A total of 20 different readers had used either discrete or continuous rating scales to evaluate those chest cases for likelihood of separate specified abnormalities: interstitial disease, pulmonary nodule, pneumothorax, alveolar infiltrate, or rib fracture. Both ROC formulations were fitted separately to every set of rating data using maximum-likelihood statistical procedures that specified each ROC curve by normally distributed latent variables with two scaling parameters, and estimated the area below the ROC curve (Az) with its standard error. The conventional and constrained binormal formulations usually fitted ROC curves that were nearly indistinguishable in form and in Az. But when fitted to asymmetric rating data that contained few false-positive cases, the conventional ROC curves often rose steeply, then flattened and extrapolated into an unrealistic upward "hook" at the higher false-positive rates. For those sets of rating data, the constrained ROC curves (without hooks) estimated larger values for Az with smaller standard errors. The constrained ROC formulation describes observers' ratings of cases at least as well as the conventional ROC, and always guarantees a realistic fitted curve for observer performance. Its estimated parameters are easy to interpret, and may also be used to predict observer accuracy in localizing the image abnormalities.

Comparison of full-field digital mammography with screen-film mammography for cancer detection: results of 4,945 paired examinations

PURPOSE

To prospectively compare full-field digital mammography (FFDM) with screen-film mammography (SFM) for cancer detection in a screening population.

MATERIALS AND METHODS

At two institutions, 4,945 FFDM examinations were performed in women aged 40 years and older presenting for SFM. Two views of each breast were acquired with each modality. SFM and FFDM images were interpreted independently. Findings detected with either SFM or FFDM were evaluated with additional imaging and, if warranted, biopsy.

RESULTS

Patients in the study underwent 152 biopsies, which resulted in the diagnosis of 35 breast cancers. Twenty-two cancers were detected with SFM and 21 with FFDM. Four were interval cancers that became palpable within 1 year of screening and were considered false-negative findings with both modalities. The difference in cancer detection rate was not significant. FFDM had a significantly lower recall rate (11.5%; 568 of 4,945) than SFM (13.8%; 685 of 4,945) (P <.001, McNemar chi(2) model; P <.03, generalized estimating equations model). The positive biopsy rate for findings detected with FFDM (30%; 21 of 69) was higher than that for findings detected with SFM (19%; 22 of 114), but this difference was not significant.

CONCLUSION

No difference in cancer detection rate has yet been observed between FFDM and SFM. FFDM has so far led to fewer recalls than SFM.

Digital storage phosphor chest radiography: an ROC study of the effect of 2K versus 4K matrix size on observer performance

PURPOSE

To compare observer performance in the detection of abnormalities on 1,760 x 2,140 matrix (2K) and 3,520 x 4,280 matrix (4K) digital storage phosphor chest radiographs.

MATERIALS AND METHODS

One hundred sixty patients who underwent dedicated computed tomography (CT) of the thorax were prospectively recruited into the study. Posteroanterior and lateral computed radiographs of the chest were acquired in each patient and printed in 2K and 4K formats. Six radiologists independently analyzed the hard-copy images and scored the presence of parenchymal (opacities </=2 cm, opacities >2 cm, and subtle interstitial), mediastinal, and pleural abnormalities on a five-point confidence scale. With CT as the reference standard, observer performance tests were carried out by using receiver operating characteristic (ROC) analysis.

RESULTS

Analysis of averaged observer performance showed 2K and 4K images were equally effective in detection of all three groups of abnormalities. In the detection of the three subtypes of parenchymal abnormalities, there were no significant differences in averaged performance between the 2K and 4K formats (area below ROC curve [A(z)] values: opacities </=2 cm, 0.62 +/- 0.056 [standard error] and 0.59 +/- 0.045; opacities >2 cm, 0.86 +/-.025 and 0.85 +/- 0.030; subtle interstitial abnormalities, 0.73 +/- 0.041 and 0.72 +/- 0.041). Averaged performance in detection of mediastinal and pleural abnormalities was equivalent (A(z) values: mediastinal, 0.70 +/- 0.046 and 0.73 +/- 0.033; pleural, 0.85 +/- 0.032 and 0.86 +/- 0.033).

CONCLUSION

Observer performance in detection of parenchymal, mediastinal, and pleural abnormalities was not significantly different on 2K and 4K storage phosphor chest radiographs.

Data analysis for detection and localization of multiple abnormalities with application to mammography

RATIONALE AND OBJECTIVES

In assessing diagnostic accuracy it is often essential to determine the reader's ability both to detect and to correctly locate multiple abnormalities per patient. The authors developed a new approach for the detection and localization of multiple abnormalities and compared it with other approaches.

MATERIALS AND METHODS

The new approach involves partitioning the image into multiple regions of interest (ROIs). The reader assigns a confidence score to each ROI. Statistical methods for clustered data are used to assess and compare reader accuracy. The authors applied this new method to a reader-performance study of conventional film images and digitized images used to detect and locate malignant breast cancer lesions.

RESULTS

The ROI-based approach, the free-response receiver operating characteristic (FROC) curve, and the patient-based approach handle the estimation of the false-positive rate (FPR) quite differently. These differences affect the measures of the respective areas under the curves. In the ROI-based approach the denominator is the number of ROIs without a malignant lesion. In the FROC approach the average number of false-positive findings per patient is plotted on the x axis of the curve. In contrast, the patient-based approach mishandles the FPR by ignoring multiple detection and/or localization errors in the same patient. The FROC approach does not lend itself easily to statistical evaluations.

CONCLUSION

The ROI-based approach appropriately captures both the detection and localization tasks. The interpretation of the ROI-based accuracy measures is simple and clinically relevant. There are statistical methods for estimating and comparing ROI-based estimates of accuracy.

1K versus 2K monitor: a clinical alternative free-response receiver operating characteristic study of observer performance using pulmonary nodules

OBJECTIVE

The aim of this study was to investigate whether and how observer performance in detecting pulmonary nodules is influenced by the use of 1K and 2K monitors with and without voluntary postprocessing.

MATERIALS AND METHODS

The study was conducted with clinical digital chest radiographs of 48 patients. CT images of the same patient group served as the gold standard. Data on four different monitor conditions (1K overview, 2K overview, 1K with postprocessing, and 2K with postprocessing) were collected using a 6-point confidence-rating scale and interpreted with an alternative free-response receiver operating characteristic.

RESULTS

When magnification and window settings were applied on the 1K monitor at the expense of an increased interpretation time, observer performance with the 1K monitor was not significantly different from that with the 2K monitor. A significant difference only occurred between the 1K monitor postprocessing condition and the 1K monitor overview condition.

CONCLUSION

Considering diagnostic accuracy, the 1K monitor is sufficient for the detection of pulmonary nodules, provided that postprocessing options--especially magnification--are applied. Further comparative monitor studies on the detectability of other abnormalities (e.g., fine interstitial structures) need to be performed.

Digital mammography: comparison of adaptive and nonadaptive CAD methods for mass detection

RATIONALE AND OBJECTIVES

The authors compared the performance of adaptive and nonadaptive computer-aided diagnostic (CAD) methods for breast mass detection with digital mammography.

MATERIALS AND METHODS

Both adaptive and nonadaptive modular CAD methods employed recent advances in multiresolution and mutiorientation wavelet transforms for improved feature extraction. The nonadaptive method uses fixed parameters for the image preprocessing modules. The adaptive method, a new class of algorithms, adapts to image content by selecting parameters for the image preprocessing modules within a parameter range. Comparison of the two methods was performed for each individual CAD module with a region-of-interest (ROI) database containing all mass types and normal tissue.

RESULTS

Receiver operating characteristic (ROC) analysis clearly demonstrated an improvement in performance for the three adaptive modules and a significant overall difference between the two methods. The average ROC area index (Az) values were 0.86 and 0.95 for the nonadaptive and adaptive methods, respectively. The corresponding P value is .0145. For a previously reported database of full mammographic images containing 50 abnormal cases with all mass types and 50 normal images, the adaptive CAD method had a sensitivity of 96% (1.71 false-positive results per image) compared with 89% (1.91 false-positive results per image) for the nonadaptive CAD method.

CONCLUSION

The adaptive CAD method demonstrated better performance. A study is in progress to determine the generalizability of the adaptive CAD method by applying it to larger retrospective image databases with different film digitizers.

Combined adaptive enhancement and region-growing segmentation of breast masses on digitized mammograms

As an ongoing effort to develop a computer aid for detection of masses on mammograms, we recently designed an object-based region-growing technique to improve mass segmentation. This segmentation method utilizes the density-weighted contrast enhancement (DWCE) filter as a pre-processing step. The DWCE filter adaptively enhances the contrast between the breast structures and the background. Object-based region growing was then applied to each of the identified structures. The region-growing technique uses gray-scale and gradient information to adjust the initial object borders and to reduce merging between adjacent or overlapping structures. Each object is then classified as a breast mass or normal tissue based on extracted morphological and texture features. In this study we evaluated the sensitivity of this combined segmentation scheme and its ability to reduce false positive (FP) detections on a data set of 253 digitized mammograms, each of which contained a biopsy-proven breast mass. It was found that the segmentation scheme detected 98% of the 253 biopsy-proven breast masses in our data set. After final FP reduction, the detection resulted in 4.2 FP per image at a 90% true positive (TP) fraction and 2.0 FPs per image at an 80% TP fraction. The combined DWCE and object-based region growing technique increased the initial detection sensitivity, reduced merging between neighboring structures, and reduced the number of FP detections in our automated breast mass detection scheme.

Prediction of human observer performance by numerical observers: an experimental study

Numerical observers are investigated for predicting the outcome of a free-response human observer study involving the detection of simulated pulmonary nodules in images reconstructed from low-dose computed tomography projection data by use of several reconstruction algorithms. A new way of calculating the figure of merit of a numerical observer is proposed wherein the detectability of signals in a particular image depends on the noise properties associated with that image and not the other images in the data set. The resulting variants of numerical observers are found to perform better than their traditional counterparts. In particular, the imagewise variant of the region-of-interest observer is found to predict best the rank ordering of algorithms by human observers for the free-response task.

Computer-aided diagnosis: automatic detection of malignant masses in digitized mammograms

A computerized method to automatically detect malignant masses on digital mammograms based on bilateral subtraction to identify asymmetries between left and right breast images was developed. After the digitization, in order to align left and right mammograms the breast border and nipple were automatically detected. Images were corrected to avoid differences in brightness due to the recording procedure. Left and right mammograms were subtracted and a threshold was applied to obtain a binary image with the information of suspicious areas. The suspicious regions or asymmetries were delimited by a region growing algorithm. Size and eccentricity tests were used to eliminate false-positive responses and texture features were extracted from suspicious regions to reject normal tissue regions. The scheme, tested in 70 pairs of digital mammograms, achieved a true-positive rate of 71% with an average number of 0.67 false positives per image. Computerized detection was evaluated by using free-response operating characteristic analysis (FROC). An area under the AFROC (A1) of 0.667 was obtained. Our results show that the scheme may be helpful to the radiologists by serving as a second reader in mammographic screening. The low number of false positives indicates that our scheme would not confuse the radiologist by suggesting normal regions as suspicious.

Development of an improved CAD scheme for automated detection of lung nodules in digital chest images

Lung cancer is the leading cause of cancer deaths in men and women in the United States, with a 5-year survival rate of only about 13%. However, this survival rate can be improved to 47% if the disease is diagnosed and treated at an early stage. In this study, we developed an improved computer-aided diagnosis (CAD) scheme for the automated detection of lung nodules in digital chest images to assist radiologists, who could miss up to 30% of the actually positive cases in their daily practice. Two hundred PA chest radiographs, 100 normals and 100 abnormals, were used as the database for our study. The presence of nodules in the 100 abnormal cases was confirmed by two experienced radiologists on the basis of CT scans or radiographic follow-up. In our CAD scheme, nodule candidates were selected initially by multiple gray-level thresholding of the difference image (which corresponds to the subtraction of a signal-enhanced image and a signal-suppressed image) and then classified into six groups. A large number of false positives were eliminated by adaptive rule-based tests and an artificial neural network (ANN). The CAD scheme achieved, on average, a sensitivity of 70% with 1.7 false positives per chest image, a performance which was substantially better as compared with other studies. The CPU time for the processing of one chest image was about 20 seconds on an IBM RISC/6000 Powerstation 590. We believe that the CAD scheme with the current performance is ready for initial clinical evaluation.

Performance of Gd-EOB-DTPA and superparamagnetic iron oxide particles in the detection of primary liver cancer: a comparative study by alternative free-response receiver operating characteristic analysis

The performance of gadolinium-ethoxybenzyl-diethylenetriaminepentaacetic acid (Gd-EOB-DTPA) and superparamagnetic iron oxide (SPIO) particles in detecting liver cancer was compared using alternative free-response receiver operating characteristic (AFROC) analysis, which allowed observers to indicate both the confidence level and the locations of all perceived abnormalities. Axial T1-weighted MR images (1.5 T) pre/post Gd-EOB-DTPA (25 mumol/kg) injection were obtained for 12 rats with chemically induced liver tumors (64 tumors). T2-weighted images (T2WI) were obtained pre/post SPIO (10 mumol/kg) injection for the same animal. Liver signal-to-noise ratio (SNR), tumor-liver contrast-to-noise ratio (CNR), and histopathologic sections corresponding to MR images were obtained. In AFROC, the location and the confidence level for each tumor were indicated independently on MR images by four radiologists. By plotting true-positive fraction and probability of false-positive per image, the area under the AFROC curve (A1) was estimated and statistically analyzed between each sequence. Either drug significantly improved tumor-liver CNR (P < .001) and tumor detection (diameter < or = 6 mm; P < .05). Gd-EOB-DTPA significantly (P < .05) improved the A1 in T1WI. There was no A1 difference between T2WI + SPIO and T1WI + Gd-EOB-DTPA. Gd-EOB-DTPA-enhanced T1WI showed the same performance as SPIO-enhanced T2WI in detecting liver tumors.

ROC methods for evaluation of fMRI techniques

Receiver operating characteristic (ROC) methods provide a standardized and statistically meaningful means for comparing signal-detection accuracy. A brief overview of ROC methods is presented. Example applications include a comparison of four different postprocessing algorithms operating on simulated fMRI time-course data sets and on human null data sets to which a simulated fMR response had been added. ROC methods also were used to reanalyze one data set from a previously published work. Additional ROC methods that also may be useful for fMRI comparisons are described.

On the generalization of the receiver operating characteristic analysis to the population of readers and cases with the jackknife method: an assessment

RATIONALE AND OBJECTIVES

A new methodology that analyzes receiver operating characteristic (ROC) data sets based on jackknifing and that considers both case and reader variability has been proposed. The purpose of this investigation was to compare results using this method to those using commonly reported methodology.

METHODS

ROC data sets using discrete and continuous rating scales were analyzed using the proposed jackknifing method, and results were compared to analysis of the same data sets using the paired t test.

RESULTS

The two methodologies did not result in the same significance levels, and in some cases, the difference was sufficient to affect conclusions regarding comparisons of diagnostic modalities. The probability value for the jackknifing procedure is based on large sample distribution theory, and its appropriateness is unknown for sample sizes used in practice. Also, the jackknifing technique was found to be sensitive to outliers resulting when data from the computer programs used to estimate area under the ROC curve failed to converge.

CONCLUSION

Although the proposed methodology yields reasonable results, several fundamental and practical issues must be addressed before it can be used widely as the analytic method of choice in ROC studies comparing different imaging techniques or reading environments.

An index for diagnostic accuracy in the multiple disease setting

RATIONALE AND OBJECTIVES

Evaluation of diagnostic accuracy in the clinical environment should entail some assessment of performance in patients with multiple abnormalities. Although receiver operating characteristic (ROC) curves often are used to assess the diagnostic accuracy of imaging systems, the concept is not easily generalizable to patients with multiple abnormalities. I propose a measure of diagnostic accuracy that is a generalization of the area under the ROC curve for a single disease.

METHODS

The proposed measure of diagnostic accuracy is a weighted average of the area under individual ROC curves for the single disease setting and of components representing areas under ROC curves constructed for patients with multiple diseases. Several options are discussed for scoring the presence of abnormality for patients who have two or more abnormalities.

RESULTS

Methods of estimating diagnostic accuracy are demonstrated on a set of data in which more than one third of the abnormal cases included multiple abnormalities of chest disease.

CONCLUSION

An easy-to-use method is given to estimate diagnostic accuracy in the multiple abnormality setting. This should make it easier to incorporate cases with multiple abnormalities when assessing the diagnostic accuracy of imaging systems.

Computerized detection of masses in digital mammograms: investigation of feature-analysis techniques

Mammographic screening of asymptomatic women has shown effectiveness in the reduction of breast cancer mortality. We are developing a computerized scheme for the detection of mammographic masses as an aid to radiologists in mammographic screening programs. Possible masses on digitized screen/film mammograms are initially identified using a nonlinear bilateral-subtraction technique, which is based on asymmetric density patterns occurring in corresponding portions of right and left mammograms. In this study, we analyze the characteristics of actual masses and nonmass detections to develop feature-analysis techniques with which to reduce the number of nonmass (ie, false-positive) detections. These feature-analysis techniques involve (1) the extraction of various features (such as area, contrast, circularity and border-distance based on the density and geometric information of masses in both processed, and original breast images), and (2) tests of the extracted features to reduce nonmass detections. Cumulative histograms of both actual-mass detections and nonmass detections are used to characterize extracted features and to determine the cutoff values used in the feature tests. The effectiveness of the feature-analysis techniques is evaluated in combination with the computerized detection scheme that uses the nonlinear bilateral-subtraction technique using free-response receiver operating characteristic analysis and 77 patient cases (308 mammograms). Results show that the feature-analysis techniques effectively improve the performance of the computerized detection scheme: about 35% false-positive detections were eliminated without loss in sensitivity when the feature-analysis techniques were used.

Perceptual enhancement of tumor targets in chest X-ray images

Under tachistoscopic viewing conditions, precuing the location of potential lung tumor targets in chest X-ray images was less effective than precuing followed by bounding the region of interest (ROI) with a circle directly on the image. Detection performance increased as the image was systematically masked so that its size approximated that of the circled ROI. When viewing time was extended to allow shifts in eye position, circling the ROI was found to restrict the dispersion of fixations and increase the accuracy of fixating the target tumor. When targets were placed outside the ROI, the circle inhibited their detection relative to detection of targets inside the circled region. These findings suggest that cuing by circling restricts target detection to the ROI, and by doing so reduces the interfering effects of outside distractors that complete with the target for attention.

Advances in statistical methodology for diagnostic medicine in the 1980's

Research on diagnostic medicine has been directed at a number of topics in the past decade. Issues which have received a lot of attention are ROC analysis and the identification and correction of various analytic biases. Other topics of widespread interest include the use of expert systems, the relationship of such systems to statistical data-based systems, and the evaluation of tests using cost-effectiveness analysis. Increasingly there is a sentiment that well-designed, prospective trials are required to provide credible information on the accuracy of diagnostic technologies, and so a consensus on methodological standards is needed, paralleling the earlier development of such standards in clinical trials and epidemiology.

ROC analysis of observer-response subjective rating data--application to periodontal radiograph assessment

Many physical anthropological studies require that an observer or device discriminate between states that can be easily confused. Receiver operating characteristic (ROC) analysis currently offers the best method for determining the accuracy of such choices, particularly for small sample sizes. Although ROC analysis is widely accepted in psychophysical and biomedical testing, its use in anthropological studies has not been reported. ROC analysis is used here to determine the usefulness of enhanced dental radiographs to assess vertical alveolar bone defects for quantitative studies of human variation with regard to periodontal disease. The presence or absence of vertical-bony defects (truth) for 75 human skulls was established by the consensus of two trained observers. Dental bitewing-radiographs were taken of the alveolar processes, the radiographs digitized, and the brightness and contrast of the digital images enhanced. The two observers who established truth then rated 1) plain bitewing radiographs, 2) unenhanced digital images of bitewings, and 3) enhanced digital images of bitewings for vertical bony defects. The rating scale varied from 1 (vertical defect definitely or almost definitely present) to 5 (definitely or almost definitely absent). ROC analysis was used to compared the diagnostic value of the 3 imaging modalities. All modalities had nearly identical diagnostic performance, measured as Az values (areas beneath ROC curves) that were less than 0.80, which indicates only moderate usefulness. It is concluded that enhancement does not increase success in vertical-bony-defect diagnosis from digital dental radiographs processed in this manner. Moreover, it is suggested that conventional bitewing radiographs may be unsuitable for accurate quantification of such defects.

Some methodological questions concerning receiver operating characteristic (ROC) analysis as a method for assessing image quality in radiology

This paper raises five methodological questions concerning receiver operating characteristic (ROC) analysis: (1) can the ROC "confidence criterion" be applied in a valid, reliable way?; (2) can ROC deal with ambiguous findings?; (3) can ROC deal effectively with false-negative findings?; (4) are ROC curves susceptible to valid statistical testing?; and (5) are ROC results useful in choosing among alternative imaging modalities? A review of the evidence leads to six conclusions. First, using ROC, all radiological findings must be unambiguously scored as true-positive, true-negative, false-positive, or false-negative, often forcing arbitrary, procrustean choices on readers and evaluators. Second, ROC requires radiologists to report findings by confidence level on a consistent, reliable basis throughout a ROC experiment; something that seems unrealistic, given what is known about human performance in almost all perceptual tasks of comparable complexity. Third, as gathered during the typical experiment, ROC data are probably nominal, but treated as if ordinal (or even interval) data, leading to distorted results. Fourth, ROC does not deal effectively with false-negatives, despite their importance. Fifth, there is no satisfactory method for statistically testing the significance of observed differences between two ROC curves if they are based on nominal data. Finally, the artificial tasks required of radiologists in a ROC evaluation limit the usefulness of ROC results in choosing among the imaging modalities.