Model for the detection of signals in images with multiple suspicious locations

Evaluation of diagnostic accuracy in free-response detection-localization tasks using ROC tools

Diagnostic systems designed to detect possibly multiple lesions per patient (e.g. multiple polyps during CT colonoscopy) are often evaluated in “free-response” studies that allow for diagnostic responses unconstrained in their number and locations. Analysis of free-response studies requires extensions of the traditional receiver operating characteristic (ROC) analysis, which are termed free-response ROC (FROC) methodology. Despite substantial developments in this area, FROC tools and approaches are much more cumbersome than traditional ROC methods. Alternative approaches that use well-known ROC tools (e.g. ROI-ROC) require defining and physically delineating regions of interest (ROI) and combine FROC data within ROIs. We propose an approach that allows analyzing FROC data using conventional ROC tools without delineating the actual ROIs or reducing data. The design parameters of FROC study are used to make FROC data analyzable using ROC tools and to calibrate the corresponding FROC and ROC curves on both conceptual and numerical levels. Differences in the performance indices of the nonparametric FROC and the new approach are shown to be asymptotically negligible and typically rather small in practice. Data from a large multi-reader study of colon cancer detection are used to illustrate the new approach.

A new ROC analysis method considering the correlation between neighboring pixels

In this paper, we introduce a novel receiver operating characteristic (ROC) analysis method that considers spatial correlation between pixels to evaluate classification algorithms. ROC analysis is one of the most important tools in the evaluation of medical images and computer aided diagnosis (CAD) systems. It provides a comprehensive description of the detection accuracy of the test image. To evaluate the localization performance, operating points of ROC curves are obtained based on the classification results of individual pixels. To this date, the confidence level or intensity value of each pixel is assumed to be independent within the image. However, this assumption is not satisfied in real problems. In this paper, a new ROC analysis algorithm that considers the correlation between neighboring pixels is proposed. Our results show that the new ROC curves provide a more accurate evaluation of the test image.

Nonparametric signal detectability evaluation using an exponential transformation of the FROC curve

PURPOSE

To develop an efficient nonparametric method for evaluation the detectability of signals at unknown locations in images, as a mean for image quality assessment.

METHODS

We use the free-response methodology that allows the image observer to mark and score all locations found as suspicious in an image, summarizing these results in a free-response operating characteristic (FROC) curve. However, unlike the relative (or receiver) operating characteristic (ROC), or the localization ROC (LROC), the FROC curve has an undefined, theoretically infinite, right side limit. Therefore area under the FROC curves cannot be directly used as an overall performance index, as the area under the curve is for ROC or LROC. We circumvent this drawback by using a transformation of the abscissa that leads to a finite integration range. By applying an exponential transformation we derive a nonparametric estimator for such a metric, and we study its properties by deriving analytical expressions for the mean and standard deviation in conditions of scores independence.

RESULTS

A comparative study with other related nonparametric estimators for ROC, LROC, and alternative FROC (AFROC) method is presented.

CONCLUSIONS

The new nonparametric estimator has sensitivity and scalability properties that make it particularly advantageous for signal detectability evaluation in phantom experiments using model observers.

Ideal AFROC and FROC observers

Detection of multiple lesions in images is a medically important task and free-response receiver operating characteristic (FROC) analyses and its variants, such as alternative FROC (AFROC) analyses, are commonly used to quantify performance in such tasks. However, ideal observers that optimize FROC or AFROC performance metrics have not yet been formulated in the general case. If available, such ideal observers may turn out to be valuable for imaging system optimization and in the design of computer aided diagnosis techniques for lesion detection in medical images. In this paper, we derive ideal AFROC and FROC observers. They are ideal in that they maximize, amongst all decision strategies, the area, or any partial area, under the associated AFROC or FROC curve. Calculation of observer performance for these ideal observers is computationally quite complex. We can reduce this complexity by considering forms of these observers that use false positive reports derived from signal-absent images only. We also consider a Bayes risk analysis for the multiple-signal detection task with an appropriate definition of costs. A general decision strategy that minimizes Bayes risk is derived. With particular cost constraints, this general decision strategy reduces to the decision strategy associated with the ideal AFROC or FROC observer.