Mammography test sets: reading location and prior images do not affect group performance

Does the availability of prior mammograms improve radiologists' observer performance?-a scoping review

Objective

The objective of this review was to examine the impact of previous mammogram availability on radiologists' performance from screening populations and experimental studies.

Materials and Methods

A search of the literature was conducted using five databases: MEDLINE, PubMed, Web of Science, ScienceDirect, and CINAHL as well as Google and reference lists of articles. Keywords were combined with "AND" or "OR" or "WITH" and included "prior mammograms, diagnostic performance, initial images, diagnostic efficacy, subsequent images, previous imaging, and radiologist's performance". Studies that assessed the impact of previous mammogram availability on radiologists' performance were reviewed. The Standard for Reporting Diagnostic Accuracy guidelines was used to critically appraise individual sources of evidence.

Results

A total of 15 articles were reviewed. The sample of mammogram cases used across these studies varied from 36 to 1,208,051. Prior mammograms did not affect sensitivity [with priors: 62-86% (mean = 73.3%); without priors: 69.4-87.4% (mean = 75.8%)] and cancer detection rate, but increased specificity [with priors: 72-96% (mean = 87.5%); without priors: 63-87% (mean = 80.5%)] and reduced false-positive rates [with priors: 3.7 to 36% (mean = 19.9%); without priors 13.3-49% (mean = 31.4%)], recall rates [with priors: 3.8-57% (mean = 26.6%); without priors: [4.9%-67.5% (mean = 37.9%)], and abnormal interpretation rate decreased by 4% with priors. Evidence for the associations between the availability of prior mammograms and positive-predictive value, area under the curve (AUC) from the receiver operating characteristic curve (ROC) and localisation ROC AUC, and positive-predictive value of recall is limited and unclear.

Conclusion

Availability of prior mammograms reduces recall rates, false-positive rates, abnormal interpretation rates, and increases specificity without affecting sensitivity and cancer detection rate.

The Reliability of Expert Diagnosis of Childhood Apraxia of Speech

PURPOSE

The current standard for clinical diagnosis of childhood apraxia of speech (CAS) is expert clinician judgment. The psychometric properties of this standard are not well understood; however, they are important for improving clinical diagnosis. The purpose of this study is to determine the extent to which experts agree on the clinical diagnosis of CAS using two cohorts of children with mixed speech sound disorders (SSDs).

METHOD

Speech samples of children with SSDs were obtained from previous and ongoing research from video recordings of children aged 3-8 years (n = 36) and audio recordings of children aged 8-17 years (n = 56). A total of 23 expert, English-speaking clinicians were recruited internationally. Three of these experts rated each speech sample to provide a description of the observed features and a diagnosis. Intrarater reliability was acceptable at 85% agreement.

RESULTS

Interrater reliability on the presence or absence of CAS among experts was poor both as a categorical diagnosis (κ = .187, 95% confidence interval [CI] [0.089, 0.286]) and on a continuous "likelihood of CAS" scale (0-100; intraclass correlation = .183, 95% CI [.037, .347]). Reliability was similar across the video-recorded and audio-only samples. There was greater agreement on other diagnoses (such as articulation disorder) than on the diagnosis of CAS, although these too did not meet the predetermined standard. Likelihood of CAS was greater in children who presented with more American Speech-Language-Hearing Association CAS consensus features.

CONCLUSIONS

Different expert raters had different thresholds for applying the diagnosis of CAS. If expert clinician judgment is to be used for diagnosis of CAS or other SSDs, further standardization and calibration is needed to increase interrater reliability. Diagnosis may require operationalized checklists or reliable measures that operate along a diagnostic continuum.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.23949105.

Identification of factors associated with diagnostic performance variation in reporting of mammograms: A review

OBJECTIVES

This narrative review aims to identify what factors are linked to diagnostic performance variation for those who interpret mammograms. Identification of influential factors has potential to contribute to the optimisation of breast cancer diagnosis. PubMed, ScienceDirect and Google Scholar databases were searched using the following terms: 'Radiology', 'Radiologist', 'Radiographer', 'Radiography', 'Mammography', 'Interpret', 'read', 'observe' 'report', 'screen', 'image', 'performance' and 'characteristics.' Exclusion criteria included articles published prior to 2000 as digital mammography was introduced at this time. Non-English articles language were also excluded. 38 of 2542 studies identified were analysed.

KEY FINDINGS

Influencing factors included, new technology, volume of reads, experience and training, availability of prior images, social networking, fatigue and time-of-day of interpretation. Advancements in breast imaging such as digital breast tomosynthesis and volume of mammograms are primary factors that affect performance as well as tiredness, time-of-day when images are interpreted, stages of training and years of experience. Recent studies emphasised the importance of social networking and knowledge sharing if breast cancer diagnosis is to be optimised.

CONCLUSION

It was demonstrated that data on radiologist performance variability is widely available but there is a paucity of data on radiographers who interpret mammographic images.

IMPLICATIONS FOR PRACTICE

This scarcity of research needs to be addressed in order to optimise radiography-led reporting and set baseline values for diagnostic efficacy.

Periapical Radiography versus Cone Beam Computed Tomography in Endodontic Disease Detection: A Free-response, Factorial Study

AIM

To assess and compare reader performance in interpreting digital periapical (PA) radiography and cone beam computed tomography (CBCT) in endodontic disease detection, using a free-response, factorial model.

MATERIALS AND METHODS

A reader performance study of 2 image test sets was undertaken using a factorial, free-response design, accounting for the independent variables: case type, case severity, reader type, and imaging modality. Twenty-two readers interpreted 60 PA and 60 CBCT images divided into 5 categories: diseased-subtle, diseased-moderate, diseased-obvious, nondiseased-subtle, and nondiseased-obvious. Lesion localization fraction, specificity, false positive (FP) marks, and the weighted alternative free-response receiver operating characteristic figure of merit were calculated.

RESULTS

CBCT had greater specificity than PA in the obvious nondiseased cases (P = .01) and no significant difference in the subtle nondiseased category. Weighted alternative free-response receiver operating characteristic values were higher for PA than CBCT in the subtle diseased (P = .02) and moderate diseased (P = .01) groups with no significant difference between in the obvious diseased groups. CBCT had higher mean FPs than PA (P < .05) in subtle diseased cases. Mean lesion localization fraction in the moderate diseased group was higher in PA than CBCT (P = .003). No relationships were found between clinical experience and all diagnostic performance measures, except for in the obvious diseased CBCT group, where increasing experience was associated mean FP marks (P = .04).

CONCLUSIONS

Reader performance in the detection of endodontic disease is better with PA radiography than CBCT. Clinical experience does not impact upon the accuracy of interpretation of both PA radiography and CBCT.

The effect of clinical history on diagnostic performance of endodontic cone-beam CT interpretation

AIM

To assess the effect of clinical history on the interpretation of endodontic disease in dental cone-beam computed tomography (CBCT).

MATERIALS AND METHODS

A reader performance study of an image test-set was undertaken using a factorial, free-response, crossover design, accounting for the independent variables: case type, case severity, reader type, and reading modality. Twenty-three readers interpreted 60 CBCT images twice over two reading sessions using a balanced design, once with access to clinical history and once without, where 30 in each session included history. Lesion localisations, specificity, false-positive marks and the weighted alternative free-response receiver operating characteristic (wAFROC1) figure of merit were calculated.

RESULTS

Clinical history had no significant effect on specificity and false-positive rates in non-diseased cases (p>0.05), but improved lesion localisation in subtle and obvious diseased cases (p<0.01). wAFROC1 values were higher with clinical history for subtle (0.58 versus 0.48; p<0.001) and obvious (0.77 versus 0.71; p=0.006) diseased categories. No associations were observed between clinical history and both readers' years of experience and reading volume in the non-diseased categories. Readers with fewer (p=0.03) and moderate (p=0.008) years of experience and low (p=0.002) CBCT reading volume demonstrated better lesion localisation in subtle diseased cases when clinical history was available.

CONCLUSIONS

Clinical history improved the interpretation of CBCT images with disease without affecting the interpretation of images without disease. Less and moderately experienced readers and low-volume readers benefitted more from availability of clinical history.

The Effect of Clinical History on Diagnostic Imaging Interpretation - A Systematic Review

RATIONALE AND OBJECTIVES

To provide updated information on the effect of clinical history on diagnostic image interpretation and to provide study methodology and design recommendations for future studies assessing the effect of clinical history on diagnostic image performance.

MATERIALS AND METHODS

A literature search of Medline, Embase, Scopus, Web of Science, and the Cochrane Central Register of Controlled Trials (CENTRAL) databases was conducted from database inception to July 21, 2020. Studies comparing diagnostic imaging performance with and without clinical history, using observers reading images under both conditions that used an independent reference standard were included.

RESULTS

Twenty-two studies met the inclusion criteria, with 15 showing clinical history improved diagnostic performance. One study reported a decrease in diagnostic performance with clinical history and the remaining six studies found no significant change in performance. Two studies used the free response paradigm with both reporting clinical history increased location sensitivity, decreased specificity and had no overall change in diagnostic performance. The disease spectrum of included cases was largely unreported and a balanced reading design was not used in 19 studies.

CONCLUSION

Most published studies found that clinical history improved diagnostic performance. More recent studies accounting for abnormality location and multiple abnormalities showed an increase in false positives and no significant change in overall diagnostic performance with clinical history.

Mammography self-evaluation online test for screening readers: an Italian Society of Medical Radiology (SIRM) initiative

OBJECTIVES

To report and analyse the characteristics and performance of the first cohort of Italian radiologists completing the national mammography self-evaluation online test established by the Italian Society of Medical Radiology (SIRM).

METHODS

A specifically-built dataset of 132 mammograms (24 with screen-detected cancers and 108 negative cases) was preliminarily tested on 48 radiologists to define pass thresholds (62% sensitivity and 86% specificity) and subsequently made available online to SIRM members during a 13-month timeframe between 2018 and 2019. Associations between participants' characteristics, pass rates, and diagnostic accuracy were then investigated with descriptive statistics and univariate and multivariable regression analyses.

RESULTS

A total of 342 radiologists completed the test, 151/342 (44.2%) with success. All individual variables, except gender, showed a significant correlation with pass rates and diagnostic sensitivity, confirmed by univariate logistic regression, while only involvement in organised screening programs and number of mammograms read per year showed a positive association with specificity at univariate logistic regression. In the multivariable regression analysis, fewer variables remained significant: > 3000 mammograms read per year for success rate; female gender, public practice setting, and higher experience self-judgement for sensitivity; no variables were significantly associated with specificity.

CONCLUSIONS

This national self-evaluation test effectively differentiated multiple aspects of mammographic reading experience, but specific breast imaging experience was shown not to strictly guarantee good diagnostic accuracy. Due to its easy use and the validity of obtained results, this test could be extended to all Italian breast radiologists, regardless of their experience, also as a Breast Unit accreditation criterion.

KEY POINTS

• This self-evaluation test was found to be able to differentiate various degrees of mammographic interpretation experience. • Breast cancer screening readers should undergo a self-assessment test, since experience parameters alone do not guarantee diagnostic ability.