Location Bias in ROC Studies

Multireader Diagnostic Accuracy Imaging Studies: Fundamentals of Design and Analysis

The design and analysis of multireader multicase (MRMC) studies are quite challenging. These studies differ from most medical studies because they need a reference standard and sampling from two populations (ie, reader and patient populations). They are quite expensive to conduct, requiring a good deal of readers' time for image interpretation. One common problem is the use of imperfect reference standards, often correlated with the test or tests being evaluated. Another common issue is oversimplification of the multidimensional MRMC data. In this study, the fundamentals of MRMC study design and analysis are reviewed. The goal is to provide investigators with a guide to the fundamentals of MRMC design and analysis, with references to more detailed discussions. In addition, readers are updated on newer areas of research, including correction for studies with multiple diagnostic accuracy end points and adjustment for location bias.

An Exploratory Multi-reader, Multi-case Study Comparing Transmission Ultrasound to Mammography on Recall Rates and Detection Rates for Breast Cancer Lesions

BACKGROUND

Three-dimensional Quantitative Transmission (QT) ultrasound imaging is an emerging modality for improving the detection and diagnosis of breast cancer. QT ultrasound has high resolution and high contrast to noise ratio, making it effective in evaluating breast tissue. This study compares radiologists' performance of noncancer recall rates and lesion detection rates using QT Ultrasound versus full-field digital mammography (FFDM) in a cross section of female subjects.

MATERIALS AND METHODS

In this multi-reader multi-case (MRMC) study, we examined retrospective data from two clinical trials conducted at five sites. All subjects received FFDM and QT scans within 90 days. Data were analyzed in a reader study with full factorial design involving 22 radiologists and 108 breast cases (42 normal, 39 pathology-confirmed benign, and 27 pathology-confirmed cancer cases). The main results used a random-reader random-case analysis adjusted for location bias performed after a primary predefined random-reader fixed-case analysis.

RESULTS

The readers' mean rate of detecting lesions of any type was 4% higher (p-value > 0.05) with QT imaging. The mean non-cancer recall rate improved significantly, showing a decrease of 16% with QT (p-value = 0.03), at the expense of a 2% decrease in the mean cancer recall rate (p-value >0.05) in comparison to FFDM. Combining performance on cancer and noncancer recall rates, the mean area under the receiver operator curve of confidence scores improved significantly by 10% with QT (p-value = 0.01).

CONCLUSION

This MRMC study indicates that QT improves non-cancer recall rates without substantially affecting cancer recall rates. The main limitation is the small number of cases from retrospective data. A larger prospective MRMC study is warranted for further assessment.

Receiver Operating Characteristic (ROC) Analysis of Image Search-and-Localize Tasks

RATIONALE AND OBJECTIVES

Receiver operating characteristic (ROC) analysis for the common image search-and-localize task, in which readers search an image for lesion or lesions not knowing a priori any exists, has been studied for over four decades. However, a satisfactory solution seems elusive.

MATERIALS AND METHODS

We show that the ROC curve predictive of clinical outcomes where readers are penalized appropriately for not correctly localizing known lesions cannot be obtained because it is a missing data problem. Further, this ROC curve is between the case-based ROC curve where readers are not penalized and the lesion-based ROC curve where penalty applies. Moreover, the lesion-based ROC curve is the LROC curve proposed by Starr et al. We show maximum-likelihood (ML) estimation of the LROC curve, validation of this procedure with Monte Carlo simulations, and its application to reader ROC datasets.

RESULTS

Monte Carlo simulations validated ML estimation of area under the LROC curve (AUC) and its variance. Example applications showed that ML estimate of LROC curve fits experimental datasets.

CONCLUSION

The ROC curve predictive of clinical performance cannot be estimated from reader ROC data alone because it is a missing data problem, and is between the case-based ROC curve where readers are not penalized for not correctly identifying known lesions and the lesion-based ROC curve where penalty applies. The lesion-based ROC curve is the LROC curve proposed by Starr et al. and can be estimated via ML estimation.