Validation and statistical power comparison of methods for analyzing free-response observer performance studies

Effect of Artificial Intelligence as a Second Reader on the Lung Nodule Detection and Localization Accuracy of Radiologists and Non-radiology Physicians in Chest Radiographs: A Multicenter Reader Study

RATIONALE AND OBJECTIVES

Missed nodules in chest radiographs (CXRs) are common occurrences. We assessed the effect of artificial intelligence (AI) as a second reader on the accuracy of radiologists and non-radiology physicians in lung nodule detection and localization in CXRs.

MATERIALS AND METHODS

This retrospective study using the multi-reader multi-case design included 300 CXRs acquired from 40 hospitals across the US. All CXRs had a paired follow-up image (chest CT or CXR) to augment the ground truth establishment for the presence and location of nodules on CXRs by five independent thoracic radiologists. 15 readers (nine radiologists and six non-radiology physicians) read each CXR twice in a second-reader paradigm, once without AI and then immediately with AI assistance. The primary analysis assessed the difference in area-under-the-alternative-free-response-receiver-operating-characteristic-curve (AFROC) of readers with and without AI. Case-level area-under-the-receiver-operating-characteristic-curve (AUROC), sensitivity, and specificity were assessed in secondary analyses.

RESULTS

A total of 300 CXRs (147 with nodules, 153 without nodules) from 300 patients (mean age, 64 years ± 15 [standard deviation]; 174 women) were included. The mean AFROC of readers was 0.73 without AI and 0.81 with AI (95% CI of difference, 0.05-0.10). Case-level AUROC was 0.77 without AI and 0.84 with AI (95% CI of difference, 0.04-0.09). Case-level sensitivity was 72.8% and 83.5% (95% CI of difference, 6.8-14.6) and specificity was 71.1% and 72.0% (95% CI of difference, -0.8-2.6) without and with AI, respectively.

CONCLUSION

Using AI, readers detected and localized more nodules without any significant difference in false positive interpretations.

An initial exploration of factors that may impact radiographer performance in reporting mammograms

OBJECTIVES

In the United Kingdom, radiographers with a qualification in image interpretation have interpreted mammograms since 1995. These radiographers work under the title of radiography advanced practitioners (RAP) or Consultant Radiographer. This study extends upon what has been very recently published by exploring further clinical, non-clinical and experiential factors that may impact the reporting performance of RAPs.

METHODS

Fifteen RAPs interpreted an image test set of 60 2D mammograms of known truth using the Detected-X software platform. Unknown to the reader, twenty cases contained a malignancy. Sensitivity, specificity, lesion sensitivity, receiver operating characteristic (ROC) and jack-knife free response operating characteristic (AFROC) values were established for each RAP. Specific features that had significant impact on accuracy were identified using Student's-T and Mann Whitney tests.

RESULTS

RAPs with more than 10 years' experience in image interpretation, compared to those with less than 10 years' experience, demonstrated lower specificity (51.3% vs 84.8%, p = 0.0264), ROC (0.83 vs 0.91, p = 0.0264) and AFROC (0.75 vs 0.87, p = 0.0037) values. Further, higher sensitivity values of 90.7% were seen in those RAPs who had an eye test in the last year compared to those who had not, 82% (p = 0.021). Other changes are presented in the paper.

CONCLUSION

These data reveal previously unidentified factors that impact the diagnostic efficacy of RAPs when interpreting mammographic images. Highlighting such findings will empower screening authorities to better examine ways of standardising performance and offer a baseline for performance benchmarks.

IMPLICATIONS FOR PRACTICE

This study for the first time performs an initial exploration of the factors that may be associated with RAP performance when interpreting screening mammograms.

GravityNet for end-to-end small lesion detection

This paper introduces a novel one-stage end-to-end detector specifically designed to detect small lesions in medical images. Precise localization of small lesions presents challenges due to their appearance and the diverse contextual backgrounds in which they are found. To address this, our approach introduces a new type of pixel-based anchor that dynamically moves towards the targeted lesion for detection. We refer to this new architecture as GravityNet, and the novel anchors as gravity points since they appear to be "attracted" by the lesions. We conducted experiments on two well-established medical problems involving small lesions to evaluate the performance of the proposed approach: microcalcifications detection in digital mammograms and microaneurysms detection in digital fundus images. Our method demonstrates promising results in effectively detecting small lesions in these medical imaging tasks.

Learnable DoG convolutional filters for microcalcification detection

Difference of Gaussians (DoG) convolutional filters are one of the earliest image processing methods employed for detecting microcalcifications on mammogram images before machine and deep learning methods became widespread. DoG is a blob enhancement filter that consists in subtracting one Gaussian-smoothed version of an image from another less Gaussian-smoothed version of the same image. Smoothing with a Gaussian kernel suppresses high-frequency spatial information, thus DoG can be regarded as a band-pass filter. However, due to their small size and overimposed breast tissue, microcalcifications vary greatly in contrast-to-noise ratio and sharpness. This makes it difficult to find a single DoG configuration that enhances all microcalcifications. In this work, we propose a convolutional network, named DoG-MCNet, where the first layer automatically learns a bank of DoG filters parameterized by their associated standard deviations. We experimentally show that when employed for microcalcification detection, our DoG layer acts as a learnable bank of band-pass preprocessing filters and improves detection performance by 4.86% AUFROC over baseline MCNet and 1.53% AUFROC over state-of-the-art multicontext ensemble of CNNs.

Image Quality and Lesion Detectability of Pancreatic Phase Thin-Slice Computed Tomography Images With a Deep Learning-Based Reconstruction Algorithm

OBJECTIVE

To evaluate the image quality and lesion detectability of pancreatic phase thin-slice computed tomography (CT) images reconstructed with a deep learning-based reconstruction (DLR) algorithm compared with filtered-back projection (FBP) and hybrid iterative reconstruction (IR) algorithms.

METHODS

Fifty-three patients who underwent dynamic contrast-enhanced CT including pancreatic phase were enrolled in this retrospective study. Pancreatic phase thin-slice (0.625 mm) images were reconstructed with each FBP, hybrid IR, and DLR. Objective image quality and signal-to-noise ratio of the pancreatic parenchyma, and contrast-to-noise ratio of pancreatic lesions were compared between the 3 reconstruction algorithms. Two radiologists independently assessed the image quality of all images. The diagnostic performance for the detection of pancreatic lesions was compared among the reconstruction algorithms using jackknife alternative free-response receiver operating characteristic analysis.

RESULTS

Deep learning-based reconstruction resulted in significantly lower image noise and higher signal-to-noise ratio and contrast-to-noise ratio than hybrid IR and FBP ( P < 0.001). Deep learning-based reconstruction also yielded significantly higher visual scores than hybrid IR and FBP ( P < 0.01). The diagnostic performance of DLR for detecting pancreatic lesions was highest for both readers, although a significant difference was found only between DLR and FBP in one reader ( P = 0.02).

CONCLUSIONS

Deep learning-based reconstruction showed improved objective and subjective image quality of pancreatic phase thin-slice CT relative to other reconstruction algorithms and has potential for improving lesion detectability.

Computer-aided diagnosis systems: a comparative study of classical machine learning versus deep learning-based approaches

The diagnostic phase of the treatment process is essential for patient guidance and follow-up. The accuracy and effectiveness of this phase can determine the life or death of a patient. For the same symptoms, different doctors may come up with different diagnoses whose treatments may, instead of curing a patient, be fatal. Machine learning (ML) brings new solutions to healthcare professionals to save time and optimize the appropriate diagnosis. ML is a data analysis method that automates the creation of analytical models and promotes predictive data. There are several ML models and algorithms that rely on features extracted from, for example, a patient's medical images to indicate whether a tumor is benign or malignant. The models differ in the way they operate and the method used to extract the discriminative features of the tumor. In this article, we review different ML models for tumor classification and COVID-19 infection to evaluate the different works. The computer-aided diagnosis (CAD) systems, which we referred to as classical, are based on accurate feature identification, usually performed manually or with other ML techniques that are not involved in classification. The deep learning-based CAD systems automatically perform the identification and extraction of discriminative features. The results show that the two types of DAC have quite close performances but the use of one or the other type depends on the datasets. Indeed, manual feature extraction is necessary when the size of the dataset is small; otherwise, deep learning is used.

Added value of contrast enhancement boost images in routine multiphasic contrast-enhanced CT for the diagnosis of small (<20 mm) hypervascular hepatocellular carcinoma

PURPOSE

To investigate the added value of contrast enhancement boost (CE-boost) images in multiphasic contrast-enhanced CT (CE-CT) for diagnosing small (<20 mm) hypervascular hepatocellular carcinoma (HCC).

MATERIALS AND METHODS

This retrospective study included 69 patients (age, 74 ± 8 years; 52 men) with 70 hypervascular HCCs (<20 mm) who underwent multiphasic CE-CT (pre-contrast, late arterial phase [LAP], portal venous phase [PVP], and equilibrium phase). Two types of CE-boost images were generated by subtracting PVP from LAP (LA-PV) images and LAP from PVP (PV-LA) images to enhance the contrast effect of hepatic arterial and portal venous perfusion more selectively. Tumor-to-liver contrast-to-noise ratios (CNRs) in CE-boost images were compared with those in CE-CT images using the Wilcoxon signed-rank test. Two independent readers reviewed the imaging datasets: CE-CT alone and CE-CT with CE-boost images. The diagnostic performance of each dataset was compared using jackknife alternative free-response receiver operating characteristics (JAFROC-1).

RESULTS

The tumor-to-liver CNRs in the LA-PV (6.4 ± 3.0) and PV-LA (-3.3 ± 2.1) images were greater than those in the LAP (3.2 ± 1.7) and PVP images (-1.1 ± 1.4) (p <.001 for both). The reader-averaged figures of merit were 0.751 for CE-CT alone and 0.807 for CE-CT with CE-boost images (p <.001). Sensitivities increased by adding CE-boost images for both readers (p <.001 and = 0.03), while positive predictive values were equivalent (p >.99).

CONCLUSION

Adding CE-boost images to multiphasic CE-CT can improve the diagnostic accuracy and sensitivity for small hypervascular HCC by increasing the tumor-to-liver CNR.

Understanding Reader Variability: A 25-Radiologist Study on Liver Metastasis Detection at CT

Background Substantial interreader variability exists for common tasks in CT imaging, such as detection of hepatic metastases. This variability can undermine patient care by leading to misdiagnosis. Purpose To determine the impact of interreader variability associated with (a) reader experience, (b) image navigation patterns (eg, eye movements, workstation interactions), and (c) eye gaze time at missed liver metastases on contrast-enhanced abdominal CT images. Materials and Methods In a single-center prospective observational trial at an academic institution between December 2020 and February 2021, readers were recruited to examine 40 contrast-enhanced abdominal CT studies (eight normal, 32 containing 91 liver metastases). Readers circumscribed hepatic metastases and reported confidence. The workstation tracked image navigation and eye movements. Performance was quantified by using the area under the jackknife alternative free-response receiver operator characteristic (JAFROC-1) curve and per-metastasis sensitivity and was associated with reader experience and image navigation variables. Differences in area under JAFROC curve were assessed with the Kruskal-Wallis test followed by the Dunn test, and effects of image navigation were assessed by using the Wilcoxon signed-rank test. Results Twenty-five readers (median age, 38 years; IQR, 31-45 years; 19 men) were recruited and included nine subspecialized abdominal radiologists, five nonabdominal staff radiologists, and 11 senior residents or fellows. Reader experience explained differences in area under the JAFROC curve, with abdominal radiologists demonstrating greater area under the JAFROC curve (mean, 0.77; 95% CI: 0.75, 0.79) than trainees (mean, 0.71; 95% CI: 0.69, 0.73) (P = .02) or nonabdominal subspecialists (mean, 0.69; 95% CI: 0.60, 0.78) (P = .03). Sensitivity was similar within the reader experience groups (P = .96). Image navigation variables that were associated with higher sensitivity included longer interpretation time (P = .003) and greater use of coronal images (P < .001). The eye gaze time was at least 0.5 and 2.0 seconds for 71% (266 of 377) and 40% (149 of 377) of missed metastases, respectively. Conclusion Abdominal radiologists demonstrated better discrimination for the detection of liver metastases on abdominal contrast-enhanced CT images. Missed metastases frequently received at least a brief eye gaze. Higher sensitivity was associated with longer interpretation time and greater use of liver display windows and coronal images. © RSNA, 2022 Online supplemental material is available for this article.

Comparison of two-view versus single-view digital breast tomosynthesis and 2D-mammography in breast cancer surveillance imaging

PURPOSE

Limited work has been performed for the implementation of digital breast tomosynthesis (DBT) in breast cancer surveillance imaging. The aim of this study was to investigate the differences between two different DBT implementations in breast cancer surveillance imaging, for patients with a personal history of breast cancer.

METHOD

The DBT implementations investigated were: (1) 2-view 2D digital mammography and 2-view DBT (2vDM&2vDBT) (2) 1-view (cranial-caudal) DM and 1-view (mediolateral-oblique) DBT (1vDM&1vDBT). Clinical performance of these two implementations was assessed retrospectively using observer studies with 118 sets of real patient images, from a single imaging centre, and six observers. Sensitivity, specificity and area under the curve (AUC) using the Jack-knife alternative free-response receiver operating characteristics (JAFROC) analysis were evaluated.

RESULTS

Results suggest that the two DBT implementations are not significantly different in terms of sensitivity, specificity and AUC. When looking at the two main different lesion types, non-calcifications and calcifications, and two different density levels, no difference in the performance of the two DBT implementations was found.

CONCLUSIONS

Since 1vDM&1vDBT exposes the patient to half the dose of 2vDM&2vDBT, it might be worth considering 1vDM&1vDBT in breast cancer surveillance imaging. However, larger studies are required to conclude on this matter.

The role of digital breast tomosynthesis in breast cancer screening: a manufacturer- and metrics-specific analysis

Aim

Digital Breast Tomosynthesis (DBT), with or without Digital Mammography (DM) or Synthetic Mammography (SM), has been introduced or is under consideration for its introduction in breast cancer screening in several countries, as it has been shown that it has advantages over DM. Despite this there is no agreement on how to implement DBT in screening, and in many cases there is a lack of official guidance on the optimum usage of each commercially available system. The aim of this review is to carry out a manufacturer-specific summary of studies on the implementation of DBT in breast cancer screening.

Methods

An exhaustive literature review was undertaken to identify clinical observer studies that evaluated at least one of five common metrics: sensitivity, specificity, area under the curve (AUC) of the receiver-operating characteristics (ROC) analysis, recall rate and cancer detection rate. Four common DBT implementation methods were discussed in this review: (1) DBT, (2) DM with DBT, (3) 1-view DBT with or without 1-view DM or 2-view DM and (4) DBT with SM.

Results

A summary of 89 studies, selected from a database of 677 studies, on the assessment of the implementation of DBT in breast cancer screening is presented in tables and discussed in a manufacturer- and metric-specific approach. Much more studies were carried out using some DBT systems than others. For one implementation method of DBT by one manufacturer there is a shortage of studies, for another implementation there are conflicting results. In some cases, there is a strong agreement between studies, making the advantages and disadvantages of each system clear.

Conclusion

The optimum implementation method of DBT in breast screening, in terms of diagnostic benefit and patient radiation dose, for one manufacturer does not necessarily apply to other manufacturers.

Radiographers' perspectives' on Visual Grading Analysis as a scientific method to evaluate image quality

INTRODUCTION

Radiographers routinely undertake many initiatives to balance image quality with radiation dose (optimisation). For optimisation studies to be successful image quality needs to be carefully evaluated. Purpose was to 1) discuss the strengths and limitations of a Visual Grading Analysis (VGA) method for image quality evaluation and 2) to outline the method from a radiographer's perspective.

METHODS

A possible method for investigating and discussing the relationship between radiographic image quality parameters and the interpretation and perception of X-ray images is the VGA method. VGA has a number of advantages such as being low cost and a detailed image quality assessment, although it is limited to ensure the images convey the relevant clinical information and relate the task based radiography.

RESULTS

Comparing the experience of using VGA and Receiver Operating Characteristic (ROC) it is obviously that less papers are published on VGA (Pubmed n=1.384) compared to ROC (Pubmed n=122.686). Hereby the scientific experience of the VGA method is limited compared to the use of ROC. VGA is, however, a much newer method and it is slowly gaining more and more attention.

CONCLUSION

The success of VGA requires a number of steps to be completed, such as defining the VGA criteria, choosing the VGA method (absolute or relative), including observers, finding the best image display platforms, training observers and selecting the best statistical method for the study purpose should be thoroughly considered.

IMPLICATION FOR PRACTICE

Detailed evaluation of image quality for optimisation studies related to technical definition of image quality.

Reduction of the radiation dose and the amount of contrast material in hepatic dynamic CT using low tube voltage and adaptive iterative dose reduction 3-dimensional

The purpose of this study was to prospectively evaluate the image quality and the diagnostic ability of low tube voltage and reduced contrast material dose hepatic dynamic computed tomography (CT) reconstructed with adaptive iterative dose reduction 3-dimensional (AIDR 3D).Eighty-nine patients underwent hepatic dynamic CT using one of the 2 protocols: tube voltage of 120 kVp, contrast dose of 600 mgI/kg, and filtered back projection in Protocol A (n = 46), and tube voltage of 100 kVp, contrast dose of 500 mgI/kg, and AIDR 3D in Protocol B (n = 43). The volume CT dose index (CTDIvol) and size-specific dose estimates (SSDEs) were compared between the 2 groups. Objective image noise and tumor to liver contrast-to-noise ratio (CNR) were also compared. Three radiologists independently reviewed image quality. The jackknife alternative free-response receiver-operating characteristic (JAFROC) analysis was performed to compare diagnostic performance.The mean CTDIvol and SSDE of Protocol B (14.3 and 20.2, respectively) were significantly lower than those of Protocol A (22.1 and 31.4, P < .001). There were no significant differences in either objective image noise or CNR. In the qualitative analysis, 2 readers assigned significant lower scores to images of Protocol B for at least one of the 3 phases regarding overall image quality (P < .05). There was no significant difference in the JAFROC1 figure of merit between protocols.Low tube voltage CT with AIDR 3D yielded a reduction in radiation dose and in the amount of contrast material while maintaining diagnostic performance.

Evaluation of Transient Motion During Gadoxetic Acid-Enhanced Multiphasic Liver Magnetic Resonance Imaging Using Free-Breathing Golden-Angle Radial Sparse Parallel Magnetic Resonance Imaging

OBJECTIVES

The aims of this study were to observe the pattern of transient motion after gadoxetic acid administration including incidence, onset, and duration, and to evaluate the clinical feasibility of free-breathing gadoxetic acid-enhanced liver magnetic resonance imaging using golden-angle radial sparse parallel (GRASP) imaging with respiratory gating.

MATERIALS AND METHODS

In this institutional review board-approved prospective study, 59 patients who provided informed consents were analyzed. Free-breathing dynamic T1-weighted images (T1WIs) were obtained using GRASP at 3 T after a standard dose of gadoxetic acid (0.025 mmol/kg) administration at a rate of 1 mL/s, and development of transient motion was monitored, which is defined as a distinctive respiratory frequency alteration of the self-gating MR signals. Early arterial, late arterial, and portal venous phases retrospectively reconstructed with and without respiratory gating and with different temporal resolutions (nongated 13.3-second, gated 13.3-second, gated 6-second T1WI) were evaluated for image quality and motion artifacts. Diagnostic performance in detecting focal liver lesions was compared among the 3 data sets.

RESULTS

Transient motion (mean duration, 21.5 ± 13.0 seconds) was observed in 40.0% (23/59) of patients, 73.9% (17/23) of which developed within 15 seconds after gadoxetic acid administration. On late arterial phase, motion artifacts were significantly reduced on gated 13.3-second and 6-second T1WI (3.64 ± 0.34, 3.61 ± 0.36, respectively), compared with nongated 13.3-second T1WI (3.12 ± 0.51, P < 0.0001). Overall, image quality was the highest on gated 13.3-second T1WI (3.76 ± 0.39) followed by gated 6-second and nongated 13.3-second T1WI (3.39 ± 0.55, 2.57 ± 0.57, P < 0.0001). Only gated 6-second T1WI showed significantly higher detection performance than nongated 13.3-second T1WI (figure of merit, 0.69 [0.63-0.76]) vs 0.60 [0.56-0.65], P = 0.004).

CONCLUSIONS

Transient motion developed in 40% (23/59) of patients shortly after gadoxetic acid administration, and gated free-breathing T1WI using GRASP was able to consistently provide acceptable arterial phase imaging in patients who exhibited transient motion.

Detection and Characterization of Solid Pulmonary Nodules at Digital Chest Tomosynthesis: Data from a Cohort of the Pilot Swedish Cardiopulmonary Bioimage Study

Purpose To investigate the performance of digital tomosynthesis (DTS) for detection and characterization of incidental solid lung nodules. Materials and Methods This prospective study was based on a population study with 1111 randomly selected participants (age range, 50-64 years) who underwent a medical evaluation that included chest computed tomography (CT). Among these, 125 participants with incidental nodules 5 mm or larger were included in this study, which added DTS in conjunction with the follow-up CT and was performed between March 2012 and October 2014. DTS images were assessed by four thoracic radiologists blinded to the true number of nodules in two separate sessions according to the 5-mm (125 participants) and 6-mm (55 participants) cut-off for follow-up of incidental nodules. Pulmonary nodules were directly marked on the images by the readers and graded regarding confidence of presence and recommendation for follow-up. Statistical analyses included jackknife free-response receiver operating characteristic, receiver operating characteristic, and Cohen κ coefficient. Results Overall detection rate ranges of CT-proven nodules 5 mm or larger and 6 mm or larger were, respectively, 49%-58% and 48%-62%. Jackknife free-response receiver operating characteristics figure of merit for detection of CT-proven nodules 5 mm or larger and 6 mm or larger was 0.47 and 0.51, respectively, and area under the receiver operating characteristic curve regarding recommendation for follow-up was 0.62 and 0.65, respectively. Conclusion Routine use of DTS would result in lower detection rates and reduced number of small nodules recommended for follow-up. ^© RSNA, 2018.

One-view digital breast tomosynthesis as a stand-alone modality for breast cancer detection: do we need more?

Purpose

To compare the performance of one-view digital breast tomosynthesis (1v-DBT) to that of three other protocols combining DBT and mammography (DM) for breast cancer detection.

Materials and methods

Six radiologists, three experienced with 1v-DBT in screening, retrospectively reviewed 181 cases (76 malignant, 50 benign, 55 normal) in two sessions. First, they scored sequentially: 1v-DBT (medio-lateral oblique, MLO), 1v-DBT (MLO) + 1v-DM (cranio-caudal, CC) and two-view DM + DBT (2v-DM+2v-DBT). The second session involved only 2v-DM. Lesions were scored using BI-RADS® and level of suspiciousness (1–10). Sensitivity, specificity, receiver operating characteristic (ROC) and jack-knife alternative free-response ROC (JAFROC) were computed.

Results

On average, 1v-DBT was non-inferior to any of the other protocols in terms of JAFROC figure-of-merit, area under ROC curve, sensitivity or specificity (p>0.391). While readers inexperienced with 1v-DBT screening improved their sensitivity when adding more images (69–79 %, p=0.019), experienced readers showed similar sensitivity (76 %) and specificity (70 %) between 1v-DBT and 2v-DM+2v-DBT (p=0.482). Subanalysis by lesion type and breast density showed no difference among modalities.

Conclusion

Detection performance with 1v-DBT is not statistically inferior to 2v-DM or to 2v-DM+2v-DBT; its use as a stand-alone modality might be sufficient for readers experienced with this protocol.

Key points

• One-view breast tomosynthesis is not inferior to two-view digital mammography.

• One-view DBT is not inferior to 2-view DM plus 2-view DBT.

• Training may lead to 1v-DBT being sufficient for screening.

The Reproducibility of Changes in Diagnostic Figures of Merit Across Laboratory and Clinical Imaging Reader Studies

RATIONALE AND OBJECTIVES

In this paper we examine which comparisons of reading performance between diagnostic imaging systems made in controlled retrospective laboratory studies may be representative of what we observe in later clinical studies. The change in a meaningful diagnostic figure of merit between two diagnostic modalities should be qualitatively or quantitatively comparable across all kinds of studies.

MATERIALS AND METHODS

In this meta-study we examine the reproducibility of relative measures of sensitivity, false positive fraction (FPF), area under the receiver operating characteristic (ROC) curve, and expected utility across laboratory and observational clinical studies for several different breast imaging modalities, including screen film mammography, digital mammography, breast tomosynthesis, and ultrasound.

RESULTS

Across studies of all types, the changes in the FPFs yielded very small probabilities of having a common mean value. The probabilities of relative sensitivity being the same across ultrasound and tomosynthesis studies were low. No evidence was found for different mean values of relative area under the ROC curve or relative expected utility within any of the study sets.

CONCLUSION

The comparison demonstrates that the ratios of areas under the ROC curve and expected utilities are reproducible across laboratory and clinical studies, whereas sensitivity and FPF are not.

On the meaning of the weighted alternative free-response operating characteristic figure of merit

PURPOSE

The free-response receiver operating characteristic (FROC) method is being increasingly used to evaluate observer performance in search tasks. Data analysis requires definition of a figure of merit (FOM) quantifying performance. While a number of FOMs have been proposed, the recommended one, namely, the weighted alternative FROC (wAFROC) FOM, is not well understood. The aim of this work is to clarify the meaning of this FOM by relating it to the empirical area under a proposed wAFROC curve.

METHODS

The weighted wAFROC FOM is defined in terms of a quasi-Wilcoxon statistic that involves weights, coding the clinical importance, assigned to each lesion. A new wAFROC curve is proposed, the y-axis of which incorporates the weights, giving more credit for marking clinically important lesions, while the x-axis is identical to that of the AFROC curve. An expression is derived relating the area under the empirical wAFROC curve to the wAFROC FOM. Examples are presented with small numbers of cases showing how AFROC and wAFROC curves are affected by correct and incorrect decisions and how the corresponding FOMs credit or penalize these decisions. The wAFROC, AFROC, and inferred ROC FOMs were applied to three clinical data sets involving multiple reader FROC interpretations in different modalities.

RESULTS

It is shown analytically that the area under the empirical wAFROC curve equals the wAFROC FOM. This theorem is the FROC analog of a well-known theorem developed in 1975 for ROC analysis, which gave meaning to a Wilcoxon statistic based ROC FOM. A similar equivalence applies between the area under the empirical AFROC curve and the AFROC FOM. The examples show explicitly that the wAFROC FOM gives equal importance to all diseased cases, regardless of the number of lesions, a desirable statistical property not shared by the AFROC FOM. Applications to the clinical data sets show that the wAFROC FOM yields results comparable to that using the AFROC FOM.

CONCLUSIONS

The equivalence theorem gives meaning to the weighted AFROC FOM, namely, it is identical to the empirical area under weighted AFROC curve.

Analysis of computer-aided detection techniques and signal characteristics for clustered microcalcifications on digital mammography and digital breast tomosynthesis

With IRB approval, digital breast tomosynthesis (DBT) images of human subjects were collected using a GE GEN2 DBT prototype system. Corresponding digital mammograms (DMs) of the same subjects were collected retrospectively from patient files. The data set contained a total of 237 views of DBT and equal number of DM views from 120 human subjects, each included 163 views with microcalcification clusters (MCs) and 74 views without MCs. The data set was separated into training and independent test sets. The pre-processing, object prescreening and segmentation, false positive reduction and clustering strategies for MC detection by three computer-aided detection (CADe) systems designed for DM, DBT, and a planar projection image generated from DBT were analyzed. Receiver operating characteristic (ROC) curves based on features extracted from microcalcifications and free-response ROC (FROC) curves based on scores from MCs were used to quantify the performance of the systems. Jackknife FROC (JAFROC) and non-parametric analysis methods were used to determine the statistical difference between the FROC curves. The difference between the CAD_DM and CAD_DBT systems when the false positive rate was estimated from cases without MCs did not reach statistical significance. The study indicates that the large search space in DBT may not be a limiting factor for CADe to achieve similar performance as that observed in DM.

Multi-Institutional Evaluation of Digital Tomosynthesis, Dual-Energy Radiography, and Conventional Chest Radiography for the Detection and Management of Pulmonary Nodules

Purpose To conduct a multi-institutional, multireader study to compare the performance of digital tomosynthesis, dual-energy (DE) imaging, and conventional chest radiography for pulmonary nodule detection and management. Materials and Methods In this binational, institutional review board-approved, HIPAA-compliant prospective study, 158 subjects (43 subjects with normal findings) were enrolled at four institutions. Informed consent was obtained prior to enrollment. Subjects underwent chest computed tomography (CT) and imaging with conventional chest radiography (posteroanterior and lateral), DE imaging, and tomosynthesis with a flat-panel imaging device. Three experienced thoracic radiologists identified true locations of nodules (n = 516, 3-20-mm diameters) with CT and recommended case management by using Fleischner Society guidelines. Five other radiologists marked nodules and indicated case management by using images from conventional chest radiography, conventional chest radiography plus DE imaging, tomosynthesis, and tomosynthesis plus DE imaging. Sensitivity, specificity, and overall accuracy were measured by using the free-response receiver operating characteristic method and the receiver operating characteristic method for nodule detection and case management, respectively. Results were further analyzed according to nodule diameter categories (3-4 mm, >4 mm to 6 mm, >6 mm to 8 mm, and >8 mm to 20 mm). Results Maximum lesion localization fraction was higher for tomosynthesis than for conventional chest radiography in all nodule size categories (3.55-fold for all nodules, P < .001; 95% confidence interval [CI]: 2.96, 4.15). Case-level sensitivity was higher with tomosynthesis than with conventional chest radiography for all nodules (1.49-fold, P < .001; 95% CI: 1.25, 1.73). Case management decisions showed better overall accuracy with tomosynthesis than with conventional chest radiography, as given by the area under the receiver operating characteristic curve (1.23-fold, P < .001; 95% CI: 1.15, 1.32). There were no differences in any specificity measures. DE imaging did not significantly affect nodule detection when paired with either conventional chest radiography or tomosynthesis. Conclusion Tomosynthesis outperformed conventional chest radiography for lung nodule detection and determination of case management; DE imaging did not show significant differences over conventional chest radiography or tomosynthesis alone. These findings indicate performance likely achievable with a range of reader expertise. ^© RSNA, 2016 Online supplemental material is available for this article.

Computerized breast mass detection using multi-scale Hessian-based analysis for dynamic contrast-enhanced MRI

This study aimed to investigate a computer-aided system for detecting breast masses using dynamic contrast-enhanced magnetic resonance imaging for clinical use. Detection performance of the system was analyzed on 61 biopsy-confirmed lesions (21 benign and 40 malignant lesions) in 34 women. The breast region was determined using the demons deformable algorithm. After the suspicious tissues were identified by kinetic feature (area under the curve) and the fuzzy c-means clustering method, all breast masses were detected based on the rotation-invariant and multi-scale blob characteristics. Subsequently, the masses were further distinguished from other detected non-tumor regions (false positives). Free-response operating characteristics (FROC) curve and detection rate were used to evaluate the detection performance. Using the combined features, including blob, enhancement, morphologic, and texture features with 10-fold cross validation, the mass detection rate was 100 % (61/61) with 15.15 false positives per case and 91.80 % (56/61) with 4.56 false positives per case. In conclusion, the proposed computer-aided detection system can help radiologists reduce inter-observer variability and the cost associated with detection of suspicious lesions from a large number of images. Our results illustrated that breast masses can be efficiently detected and that enhancement and morphologic characteristics were useful for reducing non-tumor regions.

Using breast radiographers' reports as a second opinion for radiologists' readings of microcalcifications in digital mammography

OBJECTIVE

The aim of this study was to investigate a practical method for incorporating radiographers' reports with radiologists' readings of digital mammograms.

METHODS

This simulation study was conducted using data from a free-response receiver operating characteristic observer study obtained with 75 cases (25 malignant, 25 benign and 25 normal cases) of digital mammograms. Each of the rating scores obtained by six breast radiographers was utilized as a second opinion for four radiologists' readings with the radiographers' reports. A logical "OR" operation with various criteria settings was simulated for deciding an appropriate method to select a radiographer's report in all combinations of radiologists and radiographers. The average figure of merit (FOM) of the radiologists' performances was statistically analysed using a jackknife procedure (JAFROC) to verify the clinical utility of using radiographers' reports.

RESULTS

Potential improvement of the average FOM of the radiologists' performances for identifying malignant microcalcifications could be expected when using radiographers' reports as a second opinion. When the threshold value of 2.6 in Breast Imaging-Reporting and Data System (BI-RADS®) assessment was applied to adopt/reject a radiographer's report, FOMs of radiologists' performances were further improved.

CONCLUSION

When using breast radiographers' reports as a second opinion, radiologists' performances potentially improved when reading digital mammograms. It could be anticipated that radiologists' performances were improved further by setting a threshold value on the BI-RADS assessment provided by the radiographers.

ADVANCES IN KNOWLEDGE

For the effective use of a radiographer's report as a second opinion, radiographers' rating scores and its criteria setting for adoption/rejection would be necessary.

Digital breast tomosynthesis: computer-aided detection of clustered microcalcifications on planar projection images

This paper describes a new approach to detect microcalcification clusters (MCs) in digital breast tomosynthesis (DBT) via its planar projection (PPJ) image. With IRB approval, two-view (cranio-caudal and mediolateral oblique views) DBTs of human subject breasts were obtained with a GE GEN2 prototype DBT system that acquires 21 projection angles spanning 60° in 3° increments. A data set of 307 volumes (154 human subjects) was divided by case into independent training (127 with MCs) and test sets (104 with MCs and 76 free of MCs). A simultaneous algebraic reconstruction technique with multiscale bilateral filtering (MSBF) regularization was used to enhance microcalcifications and suppress noise. During the MSBF regularized reconstruction, the DBT volume was separated into high frequency (HF) and low frequency components representing microcalcifications and larger structures. At the final iteration, maximum intensity projection was applied to the regularized HF volume to generate a PPJ image that contained MCs with increased contrast-to-noise ratio (CNR) and reduced search space. High CNR objects in the PPJ image were extracted and labeled as microcalcification candidates. Convolution neural network trained to recognize the image pattern of microcalcifications was used to classify the candidates into true calcifications and tissue structures and artifacts. The remaining microcalcification candidates were grouped into MCs by dynamic conditional clustering based on adaptive CNR threshold and radial distance criteria. False positive (FP) clusters were further reduced using the number of candidates in a cluster, CNR and size of microcalcification candidates. At 85% sensitivity an FP rate of 0.71 and 0.54 was achieved for view- and case-based sensitivity, respectively, compared to 2.16 and 0.85 achieved in DBT. The improvement was significant (p-value = 0.003) by JAFROC analysis.

Computer-aided diagnosis improves detection of small intracranial aneurysms on MRA in a clinical setting

BACKGROUND AND PURPOSE

MRA is widely accepted as a noninvasive diagnostic tool for the detection of intracranial aneurysms, but detection is still a challenging task with rather low detection rates. Our aim was to examine the performance of a computer-aided diagnosis algorithm for detecting intracranial aneurysms on MRA in a clinical setting.

MATERIALS AND METHODS

Aneurysm detectability was evaluated retrospectively in 48 subjects with and without computer-aided diagnosis by 6 readers using a clinical 3D viewing system. Aneurysms ranged from 1.1 to 6.0 mm (mean = 3.12 mm, median = 2.50 mm). We conducted a multireader, multicase, double-crossover design, free-response, observer-performance study on sets of images from different MRA scanners by using DSA as the reference standard. Jackknife alternative free-response operating characteristic curve analysis with the figure of merit was used.

RESULTS

For all readers combined, the mean figure of merit improved from 0.655 to 0.759, indicating a change in the figure of merit attributable to computer-aided diagnosis of 0.10 (95% CI, 0.03-0.18), which was statistically significant (F(1,47) = 7.00, P = .011). Five of the 6 radiologists had improved performance with computer-aided diagnosis, primarily due to increased sensitivity.

CONCLUSIONS

In conditions similar to clinical practice, using computer-aided diagnosis significantly improved radiologists' detection of intracranial DSA-confirmed aneurysms of ≤6 mm.

Evaluation of computer-aided detection and diagnosis systems

Computer-aided detection and diagnosis (CAD) systems are increasingly being used as an aid by clinicians for detection and interpretation of diseases. Computer-aided detection systems mark regions of an image that may reveal specific abnormalities and are used to alert clinicians to these regions during image interpretation. Computer-aided diagnosis systems provide an assessment of a disease using image-based information alone or in combination with other relevant diagnostic data and are used by clinicians as a decision support in developing their diagnoses. While CAD systems are commercially available, standardized approaches for evaluating and reporting their performance have not yet been fully formalized in the literature or in a standardization effort. This deficiency has led to difficulty in the comparison of CAD devices and in understanding how the reported performance might translate into clinical practice. To address these important issues, the American Association of Physicists in Medicine (AAPM) formed the Computer Aided Detection in Diagnostic Imaging Subcommittee (CADSC), in part, to develop recommendations on approaches for assessing CAD system performance. The purpose of this paper is to convey the opinions of the AAPM CADSC members and to stimulate the development of consensus approaches and "best practices" for evaluating CAD systems. Both the assessment of a standalone CAD system and the evaluation of the impact of CAD on end-users are discussed. It is hoped that awareness of these important evaluation elements and the CADSC recommendations will lead to further development of structured guidelines for CAD performance assessment. Proper assessment of CAD system performance is expected to increase the understanding of a CAD system's effectiveness and limitations, which is expected to stimulate further research and development efforts on CAD technologies, reduce problems due to improper use, and eventually improve the utility and efficacy of CAD in clinical practice.

Computer-aided detection of clustered microcalcifications in multiscale bilateral filtering regularized reconstructed digital breast tomosynthesis volume

PURPOSE

Develop a computer-aided detection (CADe) system for clustered microcalcifications in digital breast tomosynthesis (DBT) volume enhanced with multiscale bilateral filtering (MSBF) regularization.

METHODS

With Institutional Review Board approval and written informed consent, two-view DBT of 154 breasts, of which 116 had biopsy-proven microcalcification (MC) clusters and 38 were free of MCs, was imaged with a General Electric GEN2 prototype DBT system. The DBT volumes were reconstructed with MSBF-regularized simultaneous algebraic reconstruction technique (SART) that was designed to enhance MCs and reduce background noise while preserving the quality of other tissue structures. The contrast-to-noise ratio (CNR) of MCs was further improved with enhancement-modulated calcification response (EMCR) preprocessing, which combined multiscale Hessian response to enhance MCs by shape and bandpass filtering to remove the low-frequency structured background. MC candidates were then located in the EMCR volume using iterative thresholding and segmented by adaptive region growing. Two sets of potential MC objects, cluster centroid objects and MC seed objects, were generated and the CNR of each object was calculated. The number of candidates in each set was controlled based on the breast volume. Dynamic clustering around the centroid objects grouped the MC candidates to form clusters. Adaptive criteria were designed to reduce false positive (FP) clusters based on the size, CNR values and the number of MCs in the cluster, cluster shape, and cluster based maximum intensity projection. Free-response receiver operating characteristic (FROC) and jackknife alternative FROC (JAFROC) analyses were used to assess the performance and compare with that of a previous study.

RESULTS

Unpaired two-tailed t-test showed a significant increase (p < 0.0001) in the ratio of CNRs for MCs with and without MSBF regularization compared to similar ratios for FPs. For view-based detection, a sensitivity of 85% was achieved at an FP rate of 2.16 per DBT volume. For case-based detection, a sensitivity of 85% was achieved at an FP rate of 0.85 per DBT volume. JAFROC analysis showed a significant improvement in the performance of the current CADe system compared to that of our previous system (p = 0.003).

CONCLUSIONS

MBSF regularized SART reconstruction enhances MCs. The enhancement in the signals, in combination with properly designed adaptive threshold criteria, effective MC feature analysis, and false positive reduction techniques, leads to a significant improvement in the detection of clustered MCs in DBT.

A brief history of free-response receiver operating characteristic paradigm data analysis

In the receiver operating characteristic paradigm the observer assigns a single rating to each image and the location of the perceived abnormality, if any, is ignored. In the free-response receiver operating characteristic paradigm the observer is free to mark and rate as many suspicious regions as are considered clinically reportable. Credit for a correct localization is given only if a mark is sufficiently close to an actual lesion; otherwise, the observer's mark is scored as a location-level false positive. Until fairly recently there existed no accepted method for analyzing the resulting relatively unstructured data containing random numbers of mark-rating pairs per image. This report reviews the history of work in this field, which has now spanned more than five decades. It introduces terminology used to describe the paradigm, proposed measures of performance (figures of merit), ways of visualizing the data (operating characteristics), and software for analyzing free-response receiver operating characteristic studies.

Computer-aided tumor detection based on multi-scale blob detection algorithm in automated breast ultrasound images

Automated whole breast ultrasound (ABUS) is an emerging screening tool for detecting breast abnormalities. In this study, a computer-aided detection (CADe) system based on multi-scale blob detection was developed for analyzing ABUS images. The performance of the proposed CADe system was tested using a database composed of 136 breast lesions (58 benign lesions and 78 malignant lesions) and 37 normal cases. After speckle noise reduction, Hessian analysis with multi-scale blob detection was applied for the detection of tumors. This method detected every tumor, but some nontumors were also detected. The tumor like lihoods for the remaining candidates were estimated using a logistic regression model based on blobness, internal echo, and morphology features. The tumor candidates with tumor likelihoods higher than a specific threshold (0.4) were considered tumors. By using the combination of blobness, internal echo, and morphology features with 10-fold cross-validation, the proposed CAD system showed sensitivities of 100%, 90%, and 70% with false positives per pass of 17.4, 8.8, and 2.7, respectively. Our results suggest that CADe systems based on multi-scale blob detection can be used to detect breast tumors in ABUS images.

Comparison of dual-energy subtraction and electronic bone suppression combined with computer-aided detection on chest radiographs: effect on human observers' performance in nodule detection

OBJECTIVE

The objective of our study was to compare the effect of dual-energy subtraction and bone suppression software alone and in combination with computer-aided detection (CAD) on the performance of human observers in lung nodule detection.

MATERIALS AND METHODS

One hundred one patients with from one to five lung nodules measuring 5-29 mm and 42 subjects with no nodules were retrospectively selected and randomized. Three independent radiologists marked suspicious-appearing lesions on the original chest radiographs, dual-energy subtraction images, and bone-suppressed images before and after postprocessing with CAD. Marks of the observers and CAD marks were compared with CT as the reference standard. Data were analyzed using nonparametric tests and the jackknife alternative free-response receiver operating characteristic (JAFROC) method.

RESULTS

Using dual-energy subtraction alone (p = 0.0198) or CAD alone (p = 0.0095) improved the detection rate compared with using the original conventional chest radiograph. The combination of bone suppression and CAD provided the highest sensitivity (51.6%) and the original nonenhanced conventional chest radiograph alone provided the lowest (46.9%; p = 0.0049). Dual-energy subtraction and bone suppression provided the same false-positive (p = 0.2702) and true-positive (p = 0.8451) rates. Up to 22.9% of lesions were found only by the CAD program and were missed by the readers. JAFROC showed no difference in the performance between modalities (p = 0.2742-0.5442).

CONCLUSION

Dual-energy subtraction and the electronic bone suppression program used in this study provided similar detection rates for pulmonary nodules. Additionally, CAD alone or combined with bone suppression can significantly improve the sensitivity of human observers for pulmonary nodule detection.

The effect of JPEG2000 compression on detection of skull fractures

RATIONAL AND OBJECTIVES

To investigate the effect of the Joint Photographic Experts Group (JPEG2000) 30:1 and 60:1 lossy compression on the detection of cranial vault fractures when compared to JPEG2000 lossless compression.

MATERIALS AND METHODS

Fifty cranial computed tomography (CT) images were processed with three different level of JPEG2000 compression (lossless, 30:1 lossy, and 60:1 lossy) creating three sets of images. These were presented to five musculoskeletal specialists and five neuroradiologists. Each reader read at two of the three compression levels. Twenty-two cases contained a single fracture; the remaining 28 cases contained no fractures. Observers were asked to identify the presence or absence of a fracture, to locate its site, and rate their degree of confidence. Receiver operating characteristic (ROC), jackknife free-response receiver operating characteristic (JAFROC) and the Dorfman-Berbaum-Metz multiple reader multiple case (DBM-MRMC) analyses were used to explore differences between the lossless and lossy compressed images.

RESULTS

JPEG2000 lossless and 30:1 lossy compression demonstrated no significant difference in their performance with JAFROC and DBM-MRMC analysis (P < .416); however, JPEG2000 30:1 lossy compression demonstrated significantly better performance than 60:1 lossy compression (P < .016). A significant increase in misplaced confidence ratings was also seen with 60:1 (P < .037) over 30:1 lossy and lossless compression.

CONCLUSION

JPEG2000 60:1 compression degrades the detection of skull fractures significantly while increasing the confidence with which readers rate fractures compared with 30:1 lossy and lossless compression. JPEG2000 30:1 lossy compression does not significantly change performance when compared to JPEG2000 lossless for the detection of skull fractures on CT.

Noise-reducing algorithms do not necessarily provide superior dose optimisation for hepatic lesion detection with multidetector CT

OBJECTIVE

To compare the dose-optimisation potential of a smoothing filtered backprojection (FBP) and a hybrid FBP/iterative algorithm to that of a standard FBP algorithm at three slice thicknesses for hepatic lesion detection with multidetector CT.

METHODS

A liver phantom containing a 9.5-mm opacity with a density of 10 HU below background was scanned at 125, 100, 75, 50 and 25 mAs. Data were reconstructed with standard FBP (B), smoothing FBP (A) and hybrid FBP/iterative (iDose(4)) algorithms at 5-, 3- and 1-mm collimation. 10 observers marked opacities using a four-point confidence scale. Jackknife alternative free-response receiver operating characteristic figure of merit (FOM), sensitivity and noise were calculated.

RESULTS

Compared with the 125-mAs/5-mm setting for each algorithm, significant reductions in FOM (p<0.05) and sensitivity (p<0.05) were found for all three algorithms for all exposures at 1-mm thickness and for all slice thicknesses at 25 mAs, with the exception of the 25-mAs/5-mm setting for the B algorithm. Sensitivity was also significantly reduced for all exposures at 3-mm thickness for the A algorithm (p<0.05). Noise for the A and iDose(4) algorithms was approximately 13% and 21% lower, respectively, than for the B algorithm.

CONCLUSION

Superior performance for hepatic lesion detection was not shown with either a smoothing FBP algorithm or a hybrid FBP/iterative algorithm compared with a standard FBP technique, even though noise reduction with thinner slices was demonstrated with the alternative approaches.

ADVANCES IN KNOWLEDGE

Reductions in image noise with non-standard CT algorithms do not necessarily translate to an improvement in low-contrast object detection.

Estimating the relative utility of screening mammography

BACKGROUND

The concept of diagnostic utility is a fundamental component of signal detection theory, going back to some of its earliest works. Attaching utility values to the various possible outcomes of a diagnostic test should, in principle, lead to meaningful approaches to evaluating and comparing such systems. However, in many areas of medical imaging, utility is not used because it is presumed to be unknown.

METHODS

In this work, we estimate relative utility (the utility benefit of a detection relative to that of a correct rejection) for screening mammography using its known relation to the slope of a receiver operating characteristic (ROC) curve at the optimal operating point. The approach assumes that the clinical operating point is optimal for the goal of maximizing expected utility and therefore the slope at this point implies a value of relative utility for the diagnostic task, for known disease prevalence. We examine utility estimation in the context of screening mammography using the Digital Mammographic Imaging Screening Trials (DMIST) data.

RESULTS

We show how various conditions can influence the estimated relative utility, including characteristics of the rating scale, verification time, probability model, and scope of the ROC curve fit. Relative utility estimates range from 66 to 227.

CONCLUSIONS

We argue for one particular set of conditions that results in a relative utility estimate of 162 (±14%). This is broadly consistent with values in screening mammography determined previously by other means. At the disease prevalence found in the DMIST study (0.59% at 365-day verification), optimal ROC slopes are near unity, suggesting that utility-based assessments of screening mammography will be similar to those found using Youden's index.

A perceptually relevant channelized joint observer (PCJO) for the detection-localization of parametric signals

Many numerical observers have been proposed in the framework of task-based approach for medical image quality assessment. However, the existing numerical observers are still limited in diagnostic tasks: the detection task has been largely studied, while the localization task concerning one signal has been little studied and the localization of multiple signals has not been studied yet. In addition, most existing numerical observers need a priori knowledge about all the parameters of the underdetection signals, while only a few of them need at least two signal parameters. In this paper, we propose a novel numerical observer called the perceptually relevant channelized joint observer (PCJO), which cannot only detect but also localize multiple signals with unknown amplitude, orientation, size and location. We validated the PCJO for predicting human observer task performance by conducting a clinically relevant free-response subjective experiment in which six radiologists (including two experts) had to detect and localize multiple sclerosis (MS) lesions on magnetic resonance (MR) images. By using the jackknife alternative free-response operating characteristic (JAFROC) as the figure of merit (FOM), the detection-localization task performance of the PCJO was evaluated and then compared to that of the radiologists and two other numerical observers--channelized hotelling observer (CHO) and Goossenss CHO for detecting asymmetrical signals with random orientations. Overall, the results show that the PCJO performance was closer to that of the experts than to that of the other radiologists. The JAFROC1 FOMs of the PCJO (around 0.75) are not significantly different from those of the two experts (0.7672 and 0.7110), while the JAFROC1 FOMs of the numerical observers mentioned above (always over 0.84) outperform those of the experts. This indicates that the PCJO is a promising method for predicting radiologists' performance in the joint detection-localization task.

Measures of divergence of oriented patterns for the detection of architectural distortion in prior mammograms

PURPOSE

We propose a method for the detection of architectural distortion in prior mammograms of interval-cancer cases based on the expected orientation of breast tissue patterns in mammograms.

METHODS

The expected orientation of the breast tissue at each pixel was derived by using automatically detected landmarks including the breast boundary, the nipple, and the pectoral muscle (in mediolateral-oblique views). We hypothesize that the presence of architectural distortion changes the normal expected orientation of breast tissue patterns in a mammographic image. The angular deviation of the oriented structures in a given mammogram as compared to the expected orientation was analyzed to detect potential sites of architectural distortion using a measure of divergence of oriented patterns. Each potential site of architectural distortion was then characterized using measures of spicularity and angular dispersion specifically designed to represent spiculating patterns. The novel features for the characterization of spiculating patterns include an index of divergence of spicules computed from the intensity image and Gabor magnitude response using the Gabor angle response; radially weighted difference and angle-weighted difference (AWD) measures of the intensity, Gabor magnitude, and Gabor angle response; and AWD in the entropy of spicules computed from the intensity, Gabor magnitude, and Gabor angle response.

RESULTS

Using the newly proposed features with a database of 106 prior mammograms of 56 interval-cancer cases and 52 mammograms of 13 normal cases, through feature selection and pattern classification with an artificial neural network, an area under the receiver operating characteristic curve of 0.75 was obtained. Free-response receiver operating characteristic analysis indicated a sensitivity of 0.80 at 5.3 false positives (FPs) per patient. Combining the features proposed in the present paper with others described in our previous works led to significant improvement with a sensitivity of 0.80 at 3.7 FPs per patient.

CONCLUSION

The proposed methods can detect architectural distortion in prior mammograms taken 15 months (on the average) before clinical diagnosis of breast cancer, but the FP rate needs to be reduced.

ROCView: prototype software for data collection in jackknife alternative free-response receiver operating characteristic analysis

ROCView has been developed as an image display and response capture (IDRC) solution to image display and consistent recording of reader responses in relation to the free-response receiver operating characteristic paradigm. A web-based solution to IDRC for observer response studies allows observations to be completed from any location, assuming that display performance and viewing conditions are consistent with the study being completed. The simplistic functionality of the software allows observations to be completed without supervision. ROCView can display images from multiple modalities, in a randomised order if required. Following registration, observers are prompted to begin their image evaluation. All data are recorded via mouse clicks, one to localise (mark) and one to score confidence (rate) using either an ordinal or continuous rating scale. Up to nine "mark-rating" pairs can be made per image. Unmarked images are given a default score of zero. Upon completion of the study, both true-positive and false-positive reports can be downloaded and adapted for analysis. ROCView has the potential to be a useful tool in the assessment of modality performance difference for a range of imaging methods.

Quantifying the clinical relevance of a laboratory observer performance paradigm

OBJECTIVE

Laboratory observer performance measurements, receiver operating characteristic (ROC) and free-response ROC (FROC) differ from actual clinical interpretations in several respects, which could compromise their clinical relevance. The objective of this study was to develop a method for quantifying the clinical relevance of a laboratory paradigm and apply it to compare the ROC and FROC paradigms in a nodule detection task.

METHODS

The original prospective interpretations of 80 digital chest radiographs were classified by the truth panel as correct (C=1) or incorrect (C=0), depending on correlation with additional imaging, and the average of C was interpreted as the clinical figure of merit. FROC data were acquired for 21 radiologists and ROC data were inferred using the highest ratings. The areas under the ROC and alternative FROC curves were used as laboratory figures of merit. Bootstrap analysis was conducted to estimate conventional agreement measures between laboratory and clinical figures of merit. Also computed was a pseudovalue-based image-level correctness measure of the laboratory interpretations, whose association with C as measured by the area (rAUC) under an appropriately defined relevance ROC curve, is as a measure of the clinical relevance of a laboratory paradigm.

RESULTS

Low correlations (e.g. κ=0.244) and near chance level rAUC values (e.g. 0.598), attributable to differences between the clinical and laboratory paradigms, were observed. The absolute width of the confidence interval was 0.38 for the interparadigm differences of the conventional measures and 0.14 for the difference of the rAUCs.

CONCLUSION

The rAUC measure was consistent with the traditional measures but was more sensitive to the differences in clinical relevance. A new relevance ROC method for quantifying the clinical relevance of a laboratory paradigm is proposed.

Evaluating imaging and computer-aided detection and diagnosis devices at the FDA

This report summarizes the Joint FDA-MIPS Workshop on Methods for the Evaluation of Imaging and Computer-Assist Devices. The purpose of the workshop was to gather information on the current state of the science and facilitate consensus development on statistical methods and study designs for the evaluation of imaging devices to support US Food and Drug Administration submissions. Additionally, participants expected to identify gaps in knowledge and unmet needs that should be addressed in future research. This summary is intended to document the topics that were discussed at the meeting and disseminate the lessons that have been learned through past studies of imaging and computer-aided detection and diagnosis device performance.

Measures of angular spread and entropy for the detection of architectural distortion in prior mammograms

PURPOSE

Architectural distortion is an important sign of early breast cancer. We present methods for computer-aided detection of architectural distortion in mammograms acquired prior to the diagnosis of breast cancer in the interval between scheduled screening sessions.

METHODS

Potential sites of architectural distortion were detected using node maps obtained through the application of a bank of Gabor filters and linear phase portrait modeling. A total of 4,224 regions of interest (ROIs) were automatically obtained from 106 prior mammograms of 56 interval-cancer cases, including 301 true-positive ROIs, and from 52 mammograms of 13 normal cases. Each ROI was represented by three types of entropy measures of angular histograms composed with the Gabor magnitude response, angle, coherence, orientation strength, and the angular spread of power in the Fourier spectrum, including Shannon's entropy, Tsallis entropy for nonextensive systems, and Rényi entropy for extensive systems.

RESULTS

Using the entropy measures with stepwise logistic regression and the leave-one-patient-out method for feature selection and cross-validation, an artificial neural network resulted in an area under the receiver operating characteristic curve of 0.75. Free-response receiver operating characteristics indicated a sensitivity of 0.80 at 5.2 false positives (FPs) per patient.

CONCLUSION

The proposed methods can detect architectural distortion in prior mammograms taken 15 months (on the average) before clinical diagnosis of breast cancer, with a high sensitivity and a moderate number of FPs per patient. The results are promising and may be improved with additional features to characterize subtle abnormalities and larger databases including prior mammograms.

New developments in observer performance methodology in medical imaging

A common task in medical imaging is assessing whether a new imaging system, or a variant of an existing one, is an improvement over an existing imaging technology. Imaging systems are generally quite complex, consisting of several components-for example, image acquisition hardware, image processing and display hardware and software, and image interpretation by radiologists- each of which can affect performance. Although it may appear odd to include the radiologist as a "component" of the imaging chain, because the radiologist's decision determines subsequent patient care, the effect of the human interpretation has to be included. Physical measurements such as modulation transfer function, signal-to-noise ratio, are useful for characterizing the nonhuman parts of the imaging chain under idealized and often unrealistic conditions, such as uniform background phantoms and target objects with sharp edges. Measuring the performance of the entire imaging chain, including the radiologist, and using real clinical images requires different methods that fall under the rubric of observer performance methods or "ROC" analysis, that involve collecting rating data on images. The purpose of this work is to review recent developments in this field, particularly with respect to the free-response method, where location information is also collected.

Distributed human intelligence for colonic polyp classification in computer-aided detection for CT colonography

PURPOSE

To assess the diagnostic performance of distributed human intelligence for the classification of polyp candidates identified with computer-aided detection (CAD) for computed tomographic (CT) colonography.

MATERIALS AND METHODS

This study was approved by the institutional Office of Human Subjects Research. The requirement for informed consent was waived for this HIPAA-compliant study. CT images from 24 patients, each with at least one polyp of 6 mm or larger, were analyzed by using CAD software to identify 268 polyp candidates. Twenty knowledge workers (KWs) from a crowdsourcing platform labeled each polyp candidate as a true or false polyp. Two trials involving 228 KWs were conducted to assess reproducibility. Performance was assessed by comparing the area under the receiver operating characteristic curve (AUC) of KWs with the AUC of CAD for polyp classification.

RESULTS

The detection-level AUC for KWs was 0.845 ± 0.045 (standard error) in trial 1 and 0.855 ± 0.044 in trial 2. These were not significantly different from the AUC for CAD, which was 0.859 ± 0.043. When polyp candidates were stratified by difficulty, KWs performed better than CAD on easy detections; AUCs were 0.951 ± 0.032 in trial 1, 0.966 ± 0.027 in trial 2, and 0.877 ± 0.048 for CAD (P = .039 for trial 2). KWs who participated in both trials showed a significant improvement in performance going from trial 1 to trial 2; AUCs were 0.759 ± 0.052 in trial 1 and 0.839 ± 0.046 in trial 2 (P = .041).

CONCLUSION

The performance of distributed human intelligence is not significantly different from that of CAD for colonic polyp classification.

Recent developments in imaging system assessment methodology, FROC analysis and the search model

A frequent problem in imaging is assessing whether a new imaging system is an improvement over an existing standard. Observer performance methods, in particular the receiver operating characteristic (ROC) paradigm, are widely used in this context. In ROC analysis lesion location information is not used and consequently scoring ambiguities can arise in tasks, such as nodule detection, involving finding localized lesions. This paper reviews progress in the free-response ROC (FROC) paradigm in which the observer marks and rates suspicious regions and the location information is used to determine whether lesions were correctly localized. Reviewed are FROC data analysis, a search-model for simulating FROC data, predictions of the model and a method for estimating the parameters. The search model parameters are physically meaningful quantities that can guide system optimization.

Computer-aided detection scheme for sentinel lymph nodes in lymphoscintigrams using symmetrical property around mapped injection point

It is difficult to detect sentinel lymph nodes (SLNs) around an injection point of radiopharmaceuticals mapped in lymphoscintigrams. The purpose of this study was to develop a computer-aided detection (CAD) scheme for SLNs by a subtraction technique using the symmetrical property in the mapped injection point. Our database consisted of 78 lymphoscintigrams with 86 SLNs. In our CAD scheme, the mapped injection point of radiopharmaceuticals was first segmented from the lymphoscintigram using a gray-level thresholding technique. Lymphoscintigram was then divided into four regions by vertical and horizontal straight lines through the center of the segmented injection point. One of the four divided regions was defined as the target region. The correlation coefficients based on pixel values were calculated between the target region and each of the other three regions. The region with the highest correlation coefficient among three regions was selected as the similar region to the target region. The values of pixels on the target region were subtracted by the values of the corresponding pixels on the similar region. This procedure was repeated until every divided region had been used as target region. SLNs were segmented by applying a gray-level thresholding technique to the subtracted image. With our CAD scheme, sensitivity and the number of false positives were 95.3% (82/86) and 2.51 per image, respectively. Our CAD scheme achieved a high level of detection accuracy, and would have a great potential in assisting physicians to detect SLNs in lymphoscintigrams.