On the choice of acceptance radius in free-response observer performance studies

Does mammographic density remain a radiological challenge in the digital era?

BACKGROUND

The low subject contrast between cancerous and fibroglandular tissue could obscure breast abnormalities.

PURPOSE

To investigate radiologists' performance for detection of breast cancer in low and high mammographic density (MD) when cases are digitally acquired.

MATERIAL AND METHODS

A test set of 60 digital mammography cases, of which 20 were cancerous, were examined by 17 radiologists. Mammograms were categorized as low (≤50%) or high (>50%) MD and rated for suspicion of malignancy using the Royal Australian and New Zealand College of Radiology (RANZCR) classification system. Radiologist demographics including cases read per year, age, subspecialty, and years of reporting were recorded. Radiologist performance was analyzed by the following metrics: sensitivity; specificity; area under the receiver operating characteristic (ROC) curve (AUC), location sensitivity, and jackknife free-response ROC (JAFROC) figure of merit (FOM).

RESULTS

Comparing high to low MD cases, radiologists showed a significantly higher sensitivity (P = 0.015), AUC (P = 0.003), location sensitivity (P = 0.002), and JAFROC FOM (P = 0.001). In high compared to low MD cases, radiologists with <1000 annual reads and radiologists with no mammographic subspecialty had significantly higher AUC, location sensitivity, and JAFROC FOM. Radiologists with ≥1000 annual reads and radiologists with mammography subspecialty demonstrated a significant increase in location sensitivity in high compared to low MD cases.

CONCLUSION

In this experimental situation, radiologists' performance was higher when reading cases with high compared to low MD. Experienced radiologists were able to precisely localize lesions in breasts with higher MD. Further studies in unselected screening materials are needed to verify the results.

The pop-up research centre - Challenges and opportunities

OBJECTIVES

This article sets out to describe the concept of the "pop-up" research centre as a means to promote and develop radiography research locally, nationally and internationally, and to empower professional colleagues to set up similar initiatives in the future.

KEY FINDINGS

A detailed overview of the development and management of "pop-up" research is provided based on the experiences of the authors, including specific examples. Matters such as study design, approvals, equipment and software, environment, participant recruitment and management, research teams and activity costs are discussed. Quantifiable benefits of "pop-up" research such as resultant peer reviewed publications, development of researchers' skills and potential collaborations are described. A number of "soft skill" benefits are also apparent and include enhanced organisational profiles, team building and the development of leadership skills.

CONCLUSIONS

"Pop-up" research centres are a valuable option for conducting research and offer the radiography profession an achievable mechanism to increase and enhance research activity. However, careful planning and execution are essential.

Chest X-ray Interpretation by Radiographers Is Not Inferior to Radiologists: A Multireader, Multicase Comparison Using JAFROC (Jack-knife Alternative Free-response Receiver Operating Characteristics) Analysis

RATIONALE AND OBJECTIVES

Chest X-rays (CXR) are one of the most frequently requested imaging examinations and are fundamental to many patient pathways. The aim of this study was to investigate the diagnostic accuracy of CXR interpretation by reporting radiographers (technologists).

METHODS

A cohort of consultant radiologists (n = 10) and reporting radiographers (technologists; n = 11) interpreted a bank (n = 106) of adult CXRs that contained a range of pathologies. Jack-knife alternate free-response receiver operating characteristic (JAFROC) methodology was used to determine the performance of the observers (JAFROC v4.2). A noninferiority approach was used, with a predefined margin of clinical insignificance of 10% of average consultant radiologist diagnostic accuracy.

RESULTS

The diagnostic accuracy of the reporting radiographers (figure of merit = 0.828, 95% confidence interval 0.808-0.847) was noninferior to the consultant radiologists (figure of merit = 0.788, 95% confidence interval 0.766-0.811), P < .0001.

CONCLUSIONS

With appropriate postgraduate education, reporting radiographers are able to interpret CXRs at a level comparable to consultant radiologists.

Finding cancer in mammograms: if you know it's there, do you know where?

Humans can extract considerable information from scenes, even when these are presented extremely quickly. The ability of an experienced radiologist to rapidly detect an abnormality on a mammogram may build upon this general capacity. Although radiologists have been shown to be able to detect an abnormality 'above chance' at short durations, the extent to which abnormalities can be localised at brief presentations is less clear. Extending previous work, we presented radiologists with unilateral mammograms, 50% containing a mass, for 250 or 1000 ms. As the female breast varies with respect to the level of normal fibroglandular tissue, the images were categorised into high and low density (50% of each), resulting in difficult and easy searches, respectively. Participants were asked to decide whether there was an abnormality (detection) and then to locate the mass on a blank outline of the mammogram (localisation). We found both detection and localisation information for all conditions. Although there may be a dissociation between detection and localisation on a small proportion of trials, we find a number of factors that lead to the underestimation of localisation including stimulus variability, response imprecision and participant guesses. We emphasise the importance of taking these factors into account when interpreting results. The effect of density on detection and localisation highlights the importance of considering breast density in medical screening.

Classification images for localization performance in ramp-spectrum noise

PURPOSE

This study investigates forced localization of targets in simulated images with statistical properties similar to trans-axial sections of x-ray computed tomography (CT) volumes. A total of 24 imaging conditions are considered, comprising two target sizes, three levels of background variability, and four levels of frequency apodization. The goal of the study is to better understand how human observers perform forced-localization tasks in images with CT-like statistical properties.

METHODS

The transfer properties of CT systems are modeled by a shift-invariant transfer function in addition to apodization filters that modulate high spatial frequencies. The images contain noise that is the combination of a ramp-spectrum component, simulating the effect of acquisition noise in CT, and a power-law component, simulating the effect of normal anatomy in the background, which are modulated by the apodization filter as well. Observer performance is characterized using two psychophysical techniques: efficiency analysis and classification image analysis. Observer efficiency quantifies how much diagnostic information is being used by observers to perform a task, and classification images show how that information is being accessed in the form of a perceptual filter.

RESULTS

Psychophysical studies from five subjects form the basis of the results. Observer efficiency ranges from 29% to 77% across the different conditions. The lowest efficiency is observed in conditions with uniform backgrounds, where significant effects of apodization are found. The classification images, estimated using smoothing windows, suggest that human observers use center-surround filters to perform the task, and these are subjected to a number of subsequent analyses. When implemented as a scanning linear filter, the classification images appear to capture most of the observer variability in efficiency (r² = 0.86). The frequency spectra of the classification images show that frequency weights generally appear bandpass in nature, with peak frequency and bandwidth that vary with statistical properties of the images.

CONCLUSIONS

In these experiments, the classification images appear to capture important features of human-observer performance. Frequency apodization only appears to have a significant effect on performance in the absence of anatomical variability, where the observers appear to underweight low spatial frequencies that have relatively little noise. Frequency weights derived from the classification images generally have a bandpass structure, with adaptation to different conditions seen in the peak frequency and bandwidth. The classification image spectra show relatively modest changes in response to different levels of apodization, with some evidence that observers are attempting to rebalance the apodized spectrum presented to them.

Impact of compressed breast thickness and dose on lesion detectability in digital mammography: FROC study with simulated lesions in real mammograms

PURPOSE

The aim of this work was twofold: (1) to examine whether, with standard automatic exposure control (AEC) settings that maintain pixel values in the detector constant, lesion detectability in clinical images decreases as a function of breast thickness and (2) to verify whether a new AEC setup can increase lesion detectability at larger breast thicknesses.

METHODS

Screening patient images, acquired on two identical digital mammography systems, were collected over a period of 2 yr. Mammograms were acquired under standard AEC conditions (part 1) and subsequently with a new AEC setup (part 2), programmed to use the standard AEC settings for compressed breast thicknesses ≤49 mm, while a relative dose increase was applied above this thickness. The images were divided into four thickness groups: T1 ≤ 29 mm, T2 = 30-49 mm, T3 = 50-69 mm, and T4 ≥ 70 mm, with each thickness group containing 130 randomly selected craniocaudal lesion-free images. Two measures of density were obtained for every image: a BI-RADS score and a map of volumetric breast density created with a software application (VolparaDensity, Matakina, NZ). This information was used to select subsets of four images, containing one image from each thickness group, matched to a (global) BI-RADS score and containing a region with the same (local) volpara volumetric density value. One selected lesion (a microcalcification cluster or a mass) was simulated into each of the four images. This process was repeated so that, for a given thickness group, half the images contained a single lesion and half were lesion-free. The lesion templates created and inserted in groups T3 and T4 for the first part of the study were then inserted into the images of thickness groups T3 and T4 acquired with higher dose settings. Finally, all images were visualized using the ViewDEX software and scored by four radiologists performing a free search study. A statistical jackknife-alternative free-response receiver operating characteristic analysis was applied.

RESULTS

For part 1, the alternative free-response receiver operating characteristic curves for the four readers were 0.80, 0.65, 0.55 and 0.56 in going from T1 to T4, indicating a decrease in detectability with increasing breast thickness. P-values and the 95% confidence interval showed no significant difference for the T3-T4 comparison (p = 0.78) while all the other differences were significant (p < 0.05). Separate analysis of microcalcification clusters presented the same results while for mass detection, the only significant difference came when comparing T1 to the other thickness groups. Comparing the scores of part 1 and part 2, results for the T3 group acquired with the new AEC setup and T3 group at standard AEC doses were significantly different (p = 0.0004), indicating improved detection. For this group a subanalysis for microcalcification detection gave the same results while no significant difference was found for mass detection.

CONCLUSIONS

These data using clinical images confirm results found in simple QA tests for many mammography systems that detectability falls as breast thickness increases. Results obtained with the AEC setup for constant detectability above 49 mm showed an increase in lesion detection with compressed breast thickness, bringing detectability of lesions to the same level.

Does Expectation of Abnormality Affect the Search Pattern of Radiologists When Looking for Pulmonary Nodules?

This experiment investigated whether there might be an effect on the visual search strategy of radiologists during image interpretation of the same adult chest radiographs when given different clinical information. Each of 17 experienced radiologists was asked to interpret a set of 57 (10 abnormal) posteroanterior chest images to identify the presence of pulmonary lesions using differing clinical information (leading to unknown, low and high expectations of prevalence). Eye position metrics (search time, dwell time and time to first fixation) were compared for normal and abnormal images, as well as between conditions. For all images, there was a significantly longer search time at high prevalence expectation compared to low prevalence expectation (W = 75.19, P = <0.0001). Mann-Whitney analysis of the abnormal images demonstrated that the dwell time on correctly identified lesions was significantly shorter at low prevalence expectation compared to both unknown (U = 364.5, P = 0.02) and high prevalence expectation (U = 397.0, P = 0.0002). Visual search patterns of radiologists appear to be affected by changing a priori information where such information fosters an expectation of abnormality.

The impact of greyscale inversion for nodule detection in an anthropomorphic chest phantom: a free-response observer study

Objective:

The aim of this work was to assess the impact of greyscale inversion on nodule detection on posteroanterior chest X-ray images. Previous work has attempted this, with no consensus opinion formed. We assessed the value of “fast-flicking” between standard and inverted display modes for nodule detection.

Methods:

Six consultant radiologists (with 5–32 years' reporting experience) completed an observer task under the free-response paradigm. An anthropomorphic chest phantom was loaded with 50 different configurations of simulated nodules (1–4 nodules per case) measuring 5, 8, 10 and 12 mm in spherical diameter; each configuration represented a single case. In addition, 25 cases contained no nodules. Images were displayed in three modes: (i) standard, (ii) inverted and (iii) fast-flicking between standard and inverted display modes. Each observer completed the study in a different order of display (i, ii, iii) using a calibrated 5-megapixel monitor. Nodules were localized with mouse clicks and ratings assigned using a 1–10 discrete slider-bar confidence scale. Rjafroc (Pittsburgh, PA) was used for data analysis; differences in nodule detection performance were considered significant at 0.05.

Results:

The observer-averaged weighted jackknife alternative free-response receiver-operating characteristic figures of merit were 0.715 (standard), 0.684 (inverted) and 0.717 (fast-flicking). Random-reader fixed-case analysis revealed no statistically significant difference between any treatment pair [F(2,8) = 1.22; p = 0.345].

Conclusion:

No statistically significant difference in nodule detection was found for the three display conditions.

Advances in knowledge:

We have investigated the impact of fast-flicking between standard and inverted display modes for the detection of nodules. We found no benefit.

Increasing Prevalence Expectation in Thoracic Radiology Leads to Overcall

RATIONALE AND OBJECTIVES

The aim of this study was to measure the effect of prevalence expectation as determined by clinical history on the diagnostic performance of radiologists during pulmonary nodule detection on adult chest radiographs.

MATERIALS AND METHODS

A multi-observer, counter-balanced study (having half the readers in each group read a different condition initially) was performed to assess the effect of abnormality expectation on experienced radiologists' performance. A total of 33 board-certified radiologists were divided into three groups and searched for evidence of malignancy on a single set of 47 postero-anterior (PA) chest radiographs, 10 of which contained a single pulmonary nodule. The radiologists were unaware of disease prevalence. Before each viewing of the same dataset, the radiologists were allocated to two of three conditions based on the differing clinical information (previous cancer, no history, visa applicant). Location sensitivity, specificity, and jack-knife free-response receiver operator characteristics figure of merit were used to compare radiologist performance between conditions.

RESULTS

A significant reduction in specificity was shown for the cancer compared to that for the visa condition (W = -41 P = 0.02). No other significant findings were demonstrated for this or the other condition comparisons. No significant difference in the performance of radiologists was noted when viewing images under the same conditions.

CONCLUSIONS

This study suggested that there is a reduction in specificity with high compared to low prevalence expectation following specific radiological contexts. A reduction in specificity can have important clinical consequences leading to unnecessary interventions. The results and their implications emphasize the caution that should be placed on providing accurate referral criteria.

In the digital era, architectural distortion remains a challenging radiological task

AIM

To compare readers' performance in detecting architectural distortion (AD) compared with other breast cancer types using digital mammography.

MATERIALS AND METHODS

Forty-one experienced breast screen readers (20 US and 21 Australian) were asked to read a single test set of 30 digitally acquired mammographic cases. Twenty cases had abnormal findings (10 with AD, 10 non-AD) and 10 cases were normal. Each reader was asked to locate and rate any abnormalities. Lesion and case-based performance was assessed. For each collection of readers (US; Australian; combined), jackknife free-response receiver operating characteristic (JAFROC), figure of merit (FOM), and inferred receiver operating characteristic (ROC), area under curve (Az) were calculated using JAFROC v.4.1 software. Readers' sensitivity, location sensitivity, JAFROC, FOM, ROC, Az scores were compared between cases groups using Wilcoxon's signed ranked test statistics.

RESULTS

For lesion-based analysis, significantly lower location sensitivity (p=0.001) was shown on AD cases compared with non-AD cases for all reader collections. The case-based analysis demonstrated significantly lower ROC Az values (p=0.02) for the first collection of readers, and lower sensitivity for the second collection of readers (p=0.04) and all-readers collection (p=0.008), for AD compared with non-AD cases.

CONCLUSIONS

The current work demonstrates that AD remains a challenging task for readers, even in the digital era.

Number of mammography cases read per year is a strong predictor of sensitivity

Early detection of breast cancers affects the 5-year recurrence rates and treatment options for diagnosed patients, and consequently, many countries have instituted nationwide screening programs. This study compared the performance of expert radiologists from Australia and the United States in detection of breast cancer. Forty-one radiologists, 21 from Australia and 20 from the United States, reviewed 30 mammographic cases containing two-view mammograms. Twenty cases had abnormal findings and 10 cases had normal findings. Radiologists were asked to locate malignancies and assign a level of confidence. A jackknife free-response receiver operating characteristic, figure of merit (JAFROC, FOM), inferred receiver operating characteristic, area under curve (ROC, AUC), specificity, sensitivity, and location sensitivity were calculated using Ziltron software and JAFROC v4.1. A Mann-Whitney U test was used to compare the performance of Australian and U.S. radiologists. The results showed that when experience and the number of mammograms read per year were taken into account, the Australian radiologists sampled showed significantly higher sensitivity and location sensitivity ([Formula: see text]). JAFROC (FOM) and inferred ROC (AUC) analysis showed no difference between the overall performance of the two countries. ROC (AUC) and location sensitivity were higher for the Australian radiologists who read the most cases per year.

Memory for previously viewed radiographs and the effect of prior knowledge of memory task

RATIONALE AND OBJECTIVES

To investigate the effect of being forewarned that they would be asked to identify repeated images on radiologists' recognition of previously interpreted versus new chest radiographs.

MATERIALS AND METHODS

Thirteen radiologists viewed 60 posterior-anterior chest radiographs, 31 with and 29 without nodules, in two sets of 40 images each. Eight radiologists were forewarned and five radiologists were not forewarned of the memory task. Twenty images in each of the two sets were unique to each set and 20 images occurred in both sets. The readers indicated the presence or absence of any nodules during both readings, and in the second reading session they also indicated whether they thought each image had also occurred in the first reading.

RESULTS

There was no significant difference in recognition memory performance between forewarned and not-forewarned readers. Overall accuracy in distinguishing previously-viewed from new images was 60.7%.

CONCLUSIONS

Being forewarned of the memory task did not improve recognition memory.

A brief history of free-response receiver operating characteristic paradigm data analysis

In the receiver operating characteristic paradigm the observer assigns a single rating to each image and the location of the perceived abnormality, if any, is ignored. In the free-response receiver operating characteristic paradigm the observer is free to mark and rate as many suspicious regions as are considered clinically reportable. Credit for a correct localization is given only if a mark is sufficiently close to an actual lesion; otherwise, the observer's mark is scored as a location-level false positive. Until fairly recently there existed no accepted method for analyzing the resulting relatively unstructured data containing random numbers of mark-rating pairs per image. This report reviews the history of work in this field, which has now spanned more than five decades. It introduces terminology used to describe the paradigm, proposed measures of performance (figures of merit), ways of visualizing the data (operating characteristics), and software for analyzing free-response receiver operating characteristic studies.