When radiologists perform best: the learning curve in screening mammogram interpretation

Low-contrast lesion detection in neck CT: a multireader study comparing deep learning, iterative, and filtered back projection reconstructions using realistic phantoms

BACKGROUND

Computed tomography (CT) reconstruction algorithms can improve image quality, especially deep learning reconstruction (DLR). We compared DLR, iterative reconstruction (IR), and filtered back projection (FBP) for lesion detection in neck CT.

METHODS

Nine patient-mimicking neck phantoms were examined with a 320-slice scanner at six doses: 0.5, 1, 1.6, 2.1, 3.1, and 5.2 mGy. Each of eight phantoms contained one circular lesion (diameter 1 cm; contrast -30 HU to the background) in the parapharyngeal space; one phantom had no lesions. Reconstruction was made using FBP, IR, and DLR. Thirteen readers were tasked with identifying and localizing lesions in 32 images with a lesion and 20 without lesions for each dose and reconstruction algorithm. Receiver operating characteristic (ROC) and localization ROC (LROC) analysis were performed.

RESULTS

DLR improved lesion detection with ROC area under the curve (AUC) 0.724 ± 0.023 (mean ± standard error of the mean) using DLR versus 0.696 ± 0.021 using IR (p = 0.037) and 0.671 ± 0.023 using FBP (p < 0.001). Likewise, DLR improved lesion localization, with LROC AUC 0.407 ± 0.039 versus 0.338 ± 0.041 using IR (p = 0.002) and 0.313 ± 0.044 using FBP (p < 0.001). Dose reduction to 0.5 mGy compromised lesion detection in FBP-reconstructed images compared to doses ≥ 2.1 mGy (p ≤ 0.024), while no effect was observed with DLR or IR (p ≥ 0.058).

CONCLUSION

DLR improved the detectability of lesions in neck CT imaging. Dose reduction to 0.5 mGy maintained lesion detectability when denoising reconstruction was used.

RELEVANCE STATEMENT

Deep learning enhances lesion detection in neck CT imaging compared to iterative reconstruction and filtered back projection, offering improved diagnostic performance and potential for x-ray dose reduction.

KEY POINTS

Low-contrast lesion detectability was assessed in anatomically realistic neck CT phantoms. Deep learning reconstruction (DLR) outperformed filtered back projection and iterative reconstruction. Dose has little impact on lesion detectability against anatomical background structures.

Improving mammography interpretation for both novice and experienced readers: a comparative study of two commercial artificial intelligence software

OBJECTIVES

To evaluate the improvement of mammography interpretation for novice and experienced radiologists assisted by two commercial AI software.

METHODS

We compared the performance of two AI software (AI-1 and AI-2) in two experienced and two novice readers for 200 mammographic examinations (80 cancer cases). Two reading sessions were conducted within 4 weeks. The readers rated the likelihood of malignancy (range, 1-7) and the percentage probability of malignancy (range, 0-100%), with and without AI assistance. Differences in AUROC, sensitivity, and specificity were analyzed.

RESULTS

Mean AUROC increased in both novice (0.86 to 0.90 with AI-1 [p = 0.005]; 0.91 with AI-2 [p < 0.001]) and experienced readers (0.87 to 0.92 with AI-1 [p < 0.001]; 0.90 with AI-2 [p = 0.004]). Sensitivities increased from 81.3 to 88.8% with AI-1 (p = 0.027) and to 91.3% with AI-2 (p = 0.005) in novice readers, and from 81.9 to 90.6% with AI-1 (p = 0.001) and to 87.5% with AI-2 (p = 0.016) in experienced readers. Specificity did not decrease significantly in both novice (p > 0.999, both) and experienced readers (p > 0.999 with AI-1 and 0.282 with AI-2). There was no significant difference in the performance change depending on the type of AI software (p > 0.999).

CONCLUSION

Commercial AI software improved the diagnostic performance of both novice and experienced readers. The type of AI software used did not significantly impact performance changes. Further validation with a larger number of cases and readers is needed.

CLINICAL RELEVANCE STATEMENT

Commercial AI software effectively aided mammography interpretation irrespective of the experience level of human readers.

KEY POINTS

• Mammography interpretation remains challenging and is subject to a wide range of interobserver variability. • In this multi-reader study, two commercial AI software improved the sensitivity of mammography interpretation by both novice and experienced readers. The type of AI software used did not significantly impact performance changes. • Commercial AI software may effectively support mammography interpretation irrespective of the experience level of human readers.

Screening chest wall ultrasound in the mastectomy patient

BACKGROUND

While there are clear guidelines regarding chest wall ultrasound in the symptomatic patient, there is conflicting evidence regarding the use of ultrasound in the screening of women post-mastectomy.

OBJECTIVE

To assess the utility of screening chest wall ultrasound after mastectomy and to assess features of detected malignancies.

METHODS

This IRB approved, retrospective study evaluates screening US examinations of the chest wall after mastectomy. Asymptomatic women presenting for screening chest wall ultrasound from January 2016 through May 2017 were included. Cases of known active malignancy were excluded. All patients had at least one year of clinical or imaging follow-up. 43 exams (8.5 %) were performed with a history of contralateral malignancy, 465 exams (91.3 %) were performed with a history of ipsilateral malignancy, and one exam (0.2 %) was performed in a patient with bilateral prophylactic mastectomy.

RESULTS

During the 17-month period, there were 509 screening US in 389 mastectomy patients. 504 (99.0 %) exams were negative/benign. Five exams (1.0 %) were considered suspicious, with recommendation for biopsy, which was performed. Out of 509 exams, 3 (0.6 %) yielded benign results, while 2 (0.39 %) revealed recurrent malignancy, with a 95 % confidence interval (exact binomial) of 0.05 % to 1.41 % for screening ultrasound. Both patients who recurred had previously recurred, and both had initial cancer of lobular histology.

CONCLUSION

Of 509 chest wall screening US exams performed in mastectomy, 2 malignancies were detected, and each patient had history of invasive lobular carcinoma and at least one prior recurrence prior to this study, suggesting benefit of screening ultrasound in these populations.

Use of artificial intelligence in breast surgery: a narrative review

Background and Objective

We have witnessed tremendous advances in artificial intelligence (AI) technologies. Breast surgery, a subspecialty of general surgery, has notably benefited from AI technologies. This review aims to evaluate how AI has been integrated into breast surgery practices, to assess its effectiveness in improving surgical outcomes and operational efficiency, and to identify potential areas for future research and application.

Methods

Two authors independently conducted a comprehensive search of PubMed, Google Scholar, EMBASE, and Cochrane CENTRAL databases from January 1, 1950, to September 4, 2023, employing keywords pertinent to AI in conjunction with breast surgery or cancer. The search focused on English language publications, where relevance was determined through meticulous screening of titles, abstracts, and full-texts, followed by an additional review of references within these articles. The review covered a range of studies illustrating the applications of AI in breast surgery encompassing lesion diagnosis to postoperative follow-up. Publications focusing specifically on breast reconstruction were excluded.

Key Content and Findings

AI models have preoperative, intraoperative, and postoperative applications in the field of breast surgery. Using breast imaging scans and patient data, AI models have been designed to predict the risk of breast cancer and determine the need for breast cancer surgery. In addition, using breast imaging scans and histopathological slides, models were used for detecting, classifying, segmenting, grading, and staging breast tumors. Preoperative applications included patient education and the display of expected aesthetic outcomes. Models were also designed to provide intraoperative assistance for precise tumor resection and margin status assessment. As well, AI was used to predict postoperative complications, survival, and cancer recurrence.

Conclusions

Extra research is required to move AI models from the experimental stage to actual implementation in healthcare. With the rapid evolution of AI, further applications are expected in the coming years including direct performance of breast surgery. Breast surgeons should be updated with the advances in AI applications in breast surgery to provide the best care for their patients.

Specialist learning curves and clinical feasibility of introducing a new MRI grading system for skeletal maturity

Objective

MRI is an emerging imaging modality to assess skeletal maturity. This study aimed to chart the learning curves of paediatric radiologists when using an unfamiliar MRI grading system of skeletal maturity and to assess the clinical feasibility of implementing said system.

Methods

958 healthy paediatric volunteers were prospectively included in a dual-facility study. Each subject underwent a conventional MRI scan at 1.5 T. To perform the image reading, the participants were grouped into five subsets (subsets 1-5) of equal size (n∼192) in chronological order for scan acquisition. Two paediatric radiologists (R1-2) with different levels of MRI experience, both of whom were previously unfamiliar with the study's MRI grading system, independently evaluated the subsets to assess skeletal maturity in five different growth plate locations. Congruent cases at blinded reading established the consensus reading. For discrepant cases, the consensus reading was obtained through an unblinded reading by a third paediatric radiologist (R3), also unfamiliar with the MRI grading system. Further, R1 performed a second blinded image reading for all included subjects with a memory wash-out of 180 days. Weighted Cohen kappa was used to assess interreader reliability (R1 vs consensus; R2 vs consensus) at non-cumulative and cumulative time points, as well as interreader (R1 vs R2) and intrareader (R1 vs R1) reliability at non-cumulative time points.

Results

Mean weighted Cohen kappa values for each pair of blinded readers compared to consensus reading (interreader reliability, R1-2 vs consensus) were ≥0.85, showing a strong to almost perfect interreader agreement at both non-cumulative and cumulative time points and in all growth plate locations. Weighted Cohen kappa values for interreader (R1 vs R2) and intrareader reliability (R1 vs R1) were ≥0.72 at non-cumulative time points, with values ≥0.82 at subset 5.

Conclusions

Paediatric radiologists' clinical confidence when introduced to a new MRI grading system for skeletal maturity was high from the outset of their learning curve, despite the radiologists' varying levels of work experience with MRI assessment. The MRI grading system for skeletal maturity investigated in this study is a robust clinical method when used by paediatric radiologists and can be used in clinical practice.

Advances in knowledge

Radiologists with fellowship training in paediatric radiology experienced no learning curve progress when introduced to a new MRI grading system for skeletal maturity and achieved desirable agreement from the first time point of the learning curve. The robustness of the investigated MRI grading system was not affected by the earlier different levels of MRI experience among the readers.

Unenhanced Breast MRI With Diffusion-Weighted Imaging for Breast Cancer Detection: Effects of Training on Performance and Agreement of Subspecialty Radiologists

OBJECTIVE

To investigate whether reader training improves the performance and agreement of radiologists in interpreting unenhanced breast magnetic resonance imaging (MRI) scans using diffusion-weighted imaging (DWI).

MATERIALS AND METHODS

A study of 96 breasts (35 cancers, 24 benign, and 37 negative) in 48 asymptomatic women was performed between June 2019 and October 2020. High-resolution DWI with b-values of 0, 800, and 1200 sec/mm² was performed using a 3.0-T system. Sixteen breast radiologists independently reviewed the DWI, apparent diffusion coefficient maps, and T1-weighted MRI scans and recorded the Breast Imaging Reporting and Data System (BI-RADS) category for each breast. After a 2-h training session and a 5-month washout period, they re-evaluated the BI-RADS categories. A BI-RADS category of 4 (lesions with at least two suspicious criteria) or 5 (more than two suspicious criteria) was considered positive. The per-breast diagnostic performance of each reader was compared between the first and second reviews. Inter-reader agreement was evaluated using a multi-rater κ analysis and intraclass correlation coefficient (ICC).

RESULTS

Before training, the mean sensitivity, specificity, and accuracy of the 16 readers were 70.7% (95% confidence interval [CI]: 59.4-79.9), 90.8% (95% CI: 85.6-94.2), and 83.5% (95% CI: 78.6-87.4), respectively. After training, significant improvements in specificity (95.2%; 95% CI: 90.8-97.5; P = 0.001) and accuracy (85.9%; 95% CI: 80.9-89.8; P = 0.01) were observed, but no difference in sensitivity (69.8%; 95% CI: 58.1-79.4; P = 0.58) was observed. Regarding inter-reader agreement, the κ values were 0.57 (95% CI: 0.52-0.63) before training and 0.68 (95% CI: 0.62-0.74) after training, with a difference of 0.11 (95% CI: 0.02-0.18; P = 0.01). The ICC was 0.73 (95% CI: 0.69-0.74) before training and 0.79 (95% CI: 0.76-0.80) after training (P = 0.002).

CONCLUSION

Brief reader training improved the performance and agreement of interpretations by breast radiologists using unenhanced MRI with DWI.

Artificial intelligence-based computer-assisted detection/diagnosis (AI-CAD) for screening mammography: Outcomes of AI-CAD in the mammographic interpretation workflow

Purpose

To evaluate the stand-alone diagnostic performances of AI-CAD and outcomes of AI-CAD detected abnormalities when applied to the mammographic interpretation workflow.

Methods

From January 2016 to December 2017, 6499 screening mammograms of 5228 women were collected from a single screening facility. Historic reads of three radiologists were used as radiologist interpretation. A commercially-available AI-CAD was used for analysis. One radiologist not involved in interpretation had retrospectively reviewed the abnormality features and assessed the significance (negligible vs. need recall) of the AI-CAD marks. Ground truth in terms of cancer, benign or absence of abnormality was confirmed according to histopathologic diagnosis or negative results on the next-round screen.

Results

Of the 6499 mammograms, 6282 (96.7%) were in the negative, 189 (2.9%) were in the benign, and 28 (0.4%) were in the cancer group. AI-CAD detected 5 (17.9%, 5 of 28) of the 9 cancers that were intially interpreted as negative. Of the 648 AI-CAD recalls, 89.0% (577 of 648) were marks seen on examinations in the negative group, and 267 (41.2%) of the AI-CAD marks were considered to be negligible. Stand-alone AI-CAD has significantly higher recall rates (10.0% vs. 3.4%, P < 0.001) with comparable sensitivity and cancer detection rates (P = 0.086 and 0.102, respectively) when compared to the radiologists' interpretation.

Conclusion

AI-CAD detected 17.9% additional cancers on screening mammography that were initially overlooked by the radiologists. In spite of the additional cancer detection, AI-CAD had significantly higher recall rates in the clinical workflow, in which 89.0% of AI-CAD marks are on negative mammograms.

Effect of artificial intelligence-based computer-aided diagnosis on the screening outcomes of digital mammography: a matched cohort study

OBJECTIVE

To investigate whether artificial intelligence-based computer-aided diagnosis (AI-CAD) can improve radiologists' performance when used to support radiologists' interpretation of digital mammography (DM) in breast cancer screening.

METHODS

A retrospective database search identified 3158 asymptomatic Korean women who consecutively underwent screening DM between January and December 2019 without AI-CAD support, and screening DM between February and July 2020 with image interpretation aided by AI-CAD in a tertiary referral hospital using single reading. Propensity score matching was used to match the DM with AI-CAD group in a 1:1 ratio with the DM without AI-CAD group according to age, breast density, experience level of the interpreting radiologist, and screening round. Performance measures were compared with the McNemar test and generalized estimating equations.

RESULTS

A total of 1579 women who underwent DM with AI-CAD were matched with 1579 women who underwent DM without AI-CAD. Radiologists showed higher specificity (96% [1500 of 1563] vs 91.6% [1430 of 1561]; p < 0.001) and lower abnormal interpretation rates (AIR) (4.9% [77 of 1579] vs 9.2% [145 of 1579]; p < 0.001) with AI-CAD than without. There was no significant difference in the cancer detection rate (CDR) (AI-CAD vs no AI-CAD, 8.9 vs 8.9 per 1000 examinations; p = 0.999), sensitivity (87.5% vs 77.8%; p = 0.999), and positive predictive value for biopsy (PPV3) (35.0% vs 35.0%; p = 0.999) according to AI-CAD support.

CONCLUSIONS

AI-CAD increases the specificity for radiologists without decreasing sensitivity as a supportive tool in the single reading of DM for breast cancer screening.

CLINICAL RELEVANCE STATEMENT

This study shows that AI-CAD could improve the specificity of radiologists' DM interpretation in the single reading system without decreasing sensitivity, suggesting that it can benefit patients by reducing false positive and recall rates.

KEY POINTS

• In this retrospective-matched cohort study (DM without AI-CAD vs DM with AI-CAD), radiologists showed higher specificity and lower AIR when AI-CAD was used to support decision-making in DM screening. • CDR, sensitivity, and PPV for biopsy did not differ with and without AI-CAD support.

Simulating complex patient populations with hierarchical learning effects to support methods development for post-market surveillance

BACKGROUND

Validating new algorithms, such as methods to disentangle intrinsic treatment risk from risk associated with experiential learning of novel treatments, often requires knowing the ground truth for data characteristics under investigation. Since the ground truth is inaccessible in real world data, simulation studies using synthetic datasets that mimic complex clinical environments are essential. We describe and evaluate a generalizable framework for injecting hierarchical learning effects within a robust data generation process that incorporates the magnitude of intrinsic risk and accounts for known critical elements in clinical data relationships.

METHODS

We present a multi-step data generating process with customizable options and flexible modules to support a variety of simulation requirements. Synthetic patients with nonlinear and correlated features are assigned to provider and institution case series. The probability of treatment and outcome assignment are associated with patient features based on user definitions. Risk due to experiential learning by providers and/or institutions when novel treatments are introduced is injected at various speeds and magnitudes. To further reflect real-world complexity, users can request missing values and omitted variables. We illustrate an implementation of our method in a case study using MIMIC-III data for reference patient feature distributions.

RESULTS

Realized data characteristics in the simulated data reflected specified values. Apparent deviations in treatment effects and feature distributions, though not statistically significant, were most common in small datasets (n < 3000) and attributable to random noise and variability in estimating realized values in small samples. When learning effects were specified, synthetic datasets exhibited changes in the probability of an adverse outcomes as cases accrued for the treatment group impacted by learning and stable probabilities as cases accrued for the treatment group not affected by learning.

CONCLUSIONS

Our framework extends clinical data simulation techniques beyond generation of patient features to incorporate hierarchical learning effects. This enables the complex simulation studies required to develop and rigorously test algorithms developed to disentangle treatment safety signals from the effects of experiential learning. By supporting such efforts, this work can help identify training opportunities, avoid unwarranted restriction of access to medical advances, and hasten treatment improvements.

Do Reader Characteristics Affect Diagnostic Efficacy in Screening Mammography? A Systematic Review

To examine reader characteristics associated with diagnostic efficacy in the interpretation of screening mammograms. A systematic search of the literature was conducted using databases such as Cochrane, Scopus, Medline, Embase, Web of Science, and PubMed. Search terms were combined with "AND" or "OR" and included: "Radiologist's characteristics AND performance"; "radiologist experience AND screening mammography"; "annual volume read AND diagnostic efficacy"; "screening mammography performance OR diagnostic efficacy". Studies were included if they assessed reader performance in screening mammography interpretation, breast readers, used a reference standard to assess the performance, and were published in the English language. Twenty-eight studies were reviewed. Increasing reader's age was associated with lower false positive rates. No association was found between gender and performance. Half of the studies showed no association between years of reading mammograms and performance. Most studies showed that high reading volume was more likely to be associated with increased sensitivity, cancer detection rates (CDR), lower recall rate, and lower false positive rates. Inconsistent associations were found between fellowship training in breast imaging and reader performance. Specialization in breast imaging was associated with better CDR, sensitivity, and specificity. Limited studies were available to establish the association between performance and factors such as time spent in breast imaging (n = 2), screening focus (n = 1), formal rotation in mammography (n = 1), owner of practice (n = 1), and practice type (n = 1). No individual characteristics is associated with versatility in diagnostic efficacy, albeit reading volume and specialization in breast imaging appear to be associated with with increased sensitivity and CDR without significantly affecting other performance metrics.

Identification of factors associated with diagnostic performance variation in reporting of mammograms: A review

OBJECTIVES

This narrative review aims to identify what factors are linked to diagnostic performance variation for those who interpret mammograms. Identification of influential factors has potential to contribute to the optimisation of breast cancer diagnosis. PubMed, ScienceDirect and Google Scholar databases were searched using the following terms: 'Radiology', 'Radiologist', 'Radiographer', 'Radiography', 'Mammography', 'Interpret', 'read', 'observe' 'report', 'screen', 'image', 'performance' and 'characteristics.' Exclusion criteria included articles published prior to 2000 as digital mammography was introduced at this time. Non-English articles language were also excluded. 38 of 2542 studies identified were analysed.

KEY FINDINGS

Influencing factors included, new technology, volume of reads, experience and training, availability of prior images, social networking, fatigue and time-of-day of interpretation. Advancements in breast imaging such as digital breast tomosynthesis and volume of mammograms are primary factors that affect performance as well as tiredness, time-of-day when images are interpreted, stages of training and years of experience. Recent studies emphasised the importance of social networking and knowledge sharing if breast cancer diagnosis is to be optimised.

CONCLUSION

It was demonstrated that data on radiologist performance variability is widely available but there is a paucity of data on radiographers who interpret mammographic images.

IMPLICATIONS FOR PRACTICE

This scarcity of research needs to be addressed in order to optimise radiography-led reporting and set baseline values for diagnostic efficacy.

Inter- and Intra-Observer Agreement When Using a Diagnostic Labeling Scheme for Annotating Findings on Chest X-rays-An Early Step in the Development of a Deep Learning-Based Decision Support System

Consistent annotation of data is a prerequisite for the successful training and testing of artificial intelligence-based decision support systems in radiology. This can be obtained by standardizing terminology when annotating diagnostic images. The purpose of this study was to evaluate the annotation consistency among radiologists when using a novel diagnostic labeling scheme for chest X-rays. Six radiologists with experience ranging from one to sixteen years, annotated a set of 100 fully anonymized chest X-rays. The blinded radiologists annotated on two separate occasions. Statistical analyses were done using Randolph's kappa and PABAK, and the proportions of specific agreements were calculated. Fair-to-excellent agreement was found for all labels among the annotators (Randolph's Kappa, 0.40-0.99). The PABAK ranged from 0.12 to 1 for the two-reader inter-rater agreement and 0.26 to 1 for the intra-rater agreement. Descriptive and broad labels achieved the highest proportion of positive agreement in both the inter- and intra-reader analyses. Annotating findings with specific, interpretive labels were found to be difficult for less experienced radiologists. Annotating images with descriptive labels may increase agreement between radiologists with different experience levels compared to annotation with interpretive labels.

Small lesion classification on abbreviated breast MRI: training can improve diagnostic performance and inter-reader agreement

OBJECTIVE

To determine whether the diagnostic performance and inter-reader agreement for small lesion classification on abbreviated breast MRI (AB-MRI) can be improved by training, and can achieve the level of full diagnostic protocol MRI (FDP-MRI).

METHODS

This retrospective study enrolled 1165 breast lesions (≤ 2 cm; 409 malignant and 756 benign) from 1165 MRI examinations for reading test. Twelve radiologists were assigned into a trained group and a non-trained group. They interpreted each AB-MRI twice, which was extracted from FDP-MRI. After the first read, the trained group received a structured training for AB-MRI interpretation while the non-trained group did not. FDP-MRIs were interpreted by the trained group after the second read. BI-RADS category for each lesion was compared to the standard of reference (histopathological examination or follow-up) to calculate diagnostic accuracy. Inter-reader agreement was assessed using multirater k analysis. Diagnostic accuracy and inter-reader agreement were compared between the trained and non-trained groups, between the first and second reads, and between AB-MRI and FDP-MRI.

RESULTS

After training, the diagnostic accuracy of AB-MRI increased from 77.6 to 84.4%, and inter-reader agreement improved from 0.410 to 0.579 (both p < 0.001), which were higher than those of the non-trained group (accuracy, 84.4% vs 78.0%; weighted k, 0.579 vs 0.461; both p < 0.001). The post-training accuracy and inter-reader agreement of AB-MRI were lower than those of FDP-MRI (accuracy, 84.4% vs 92.8%; weighted k, 0.579 vs 0.602; both p < 0.001).

CONCLUSIONS

Training can improve the diagnostic performance and inter-reader agreement for small lesion classification on AB-MRI; however, it remains inferior to those of FDP-MRI.

KEY POINTS

• Training can improve the diagnostic performance for small breast lesions on AB-MRI. • Training can reduce inter-observer variation for breast lesion classification on AB-MRI, especially among junior radiologists. • The post-training diagnostic performance and inter-reader agreement of AB-MRI remained inferior to those of FDP-MRI.

Breast cancer detection across dense and non-dense breasts: Markers of diagnostic confidence and efficacy

Background

The impact of radiologists’ characteristics has become a major focus of recent research. However, the markers of diagnostic efficacy and confidence in dense and non-dense breasts are poorly understood.

Purpose

This study aims to assess the relationship between radiologists’ characteristics and diagnostic performance across dense and non-dense breasts.

Materials and methods

Radiologists specialising in breast imaging (n = 128) who had 0.5–40 (13±10.6) years of experience reading mammograms were recruited. Participants independently interpreted a test set containing 60 digital mammograms (40 normal and 20 abnormal) with similarly distributed breast densities. Diagnostic performance measures were analysed via Jamovi software (version 1.6.22).

Results

In dense breasts, breast-imaging fellowship completion significantly improved specificity (p = 0.004), location sensitivity (p = 0.01) and the area under the curve (AUC) of the receiver operating characteristic (p = 0.03). Only participation in BreastScreen reading significantly improved all performance metrics: specificity (p = 0.04), sensitivity (p = 0.005), location sensitivity (p < 0.001) and AUC (p < 0.001). Reading > 100 mammograms weekly significantly improved sensitivity (p = 0.03), location sensitivity (p = 0.001), and AUC (p = 0.03).In non-dense breasts, breast fellowship completion significantly improved sensitivity (p = 0.02), location sensitivity (p = 0.04) and AUC (p = 0.002). Participation in BreastScreen reading and reading > 100 mammograms weekly significantly improved only sensitivity (p = 0.002 and p = 0.003, respectively) and location sensitivity (p < 0.001 and p < 0.001, respectively).

Conclusion

Participating in screening programs, breast fellowships and reading > 100 mammograms weekly are important indicators of the diagnostic performance of radiologists across dense and non-dense breasts. In dense breasts, optimal performance resulted from participation in a breast screening program.

Empowering study of breast cancer data with application of artificial intelligence technology: promises, challenges, and use cases

In healthcare, artificial intelligence (AI) technologies have the potential to create significant value by improving time-sensitive outcomes while lowering error rates for each patient. Diagnostic images, clinical notes, and reports are increasingly generated and stored in electronic medical records. This heterogeneous data presenting us with challenges in data analytics and reusability that is by nature has high complexity, thereby necessitating novel ways to store, manage and process, and reuse big data. This presents an urgent need to develop new, scalable, and expandable AI infrastructure and analytical methods that can enable healthcare providers to access knowledge for individual patients, yielding better decisions and outcomes. In this review article, we briefly discuss the nature of data in breast cancer study and the role of AI for generating "smart data" which offer actionable information that supports the better decision for personalized medicine for individual patients. In our view, the biggest challenge is to create a system that makes data robust and smart for healthcare providers and patients that can lead to more effective clinical decision-making, improved health outcomes, and ultimately, managing the healthcare outcomes and costs. We highlight some of the challenges in using breast cancer data and propose the need for an AI-driven environment to address them. We illustrate our vision with practical use cases and discuss a path for empowering the study of breast cancer databases with the application of AI and future directions.

Perspective: in pursuit of a learning culture

Transitioning from peer review to peer learning is an important step forward in developing a learning culture. Additional measures are going to be required to meet this goal. Ideas toward establishing a learning culture are detailed in this perspective.

Transitioning to Practice: Getting up to Speed in Efficiency and Accuracy

The transition from trainee to breast radiologist is challenging. The many new responsibilities that breast radiologists acquire while establishing themselves as clinicians may increase stress and anxiety. Taking inventory of existing knowledge and skills and addressing deficits toward the end of one's training can be beneficial. New breast radiologists should expect to be slower and gain proficiency in the first several years out of training. Having realistic expectations for oneself with respect to screening mammography interpretation and following up on the subsequent diagnostic imaging workup of screening callback examinations can increase competence and confidence. Familiarity with the available literature to guide management in the diagnostic setting can increase efficiency. Planning ahead for localizations and biopsies also allows for efficiency while alleviating anxiety. Ultimately, adapting to a new work environment using a collaborative approach with primary healthcare providers, pathologists, and surgeons while remembering to have mentors within and beyond the field of radiology allows for a more successful transition.

Perceptions Regarding Optimal Breast Imaging Education for Radiology Residents: Results of a National Survey

OBJECTIVES

To assess perceptions among breast radiologists regarding optimal breast imaging rotation organization and educational activities for radiology residents.

METHODS

An IRB-exempt anonymous questionnaire was developed in REDCap and distributed as a member survey email by the Society of Breast imaging to 2188 members June 2019. A form with 33 questions asked questions about opinions related to resident rotation composition, read-out strategies, study/procedure volume, simulation and educational materials specific to breast imaging.

RESULTS

A total of 109/2,188 (4.98%) complete survey responses were received. Of the responders, 69/109 (62%) work in academic practice, 16/109 (15%) work in private practice with residents, and 24/109 (22%) work in private practice without residents. There was no significant variation in opinion between those who have >10 years of experience teaching breast imaging 49/109(42.2%) to those with less <10 years' experience 60/109 (55%). A range of opinions is demonstrated regarding the multiple questions asked with more support for diagnostics and procedures on the second and thirds rotations compared to first rotations. There was strong support of in-person staffing, checklists, simulation for ultrasound-guided procedures (91%) and formal training on delivering bad news (90%). Radiology-pathology conferences and faculty-developed teaching files were highest-rated for effective educational tools.

CONCLUSIONS

The results from this survey show varied opinions regarding perceived best practices for resident breast radiology rotations . Further research is needed to determine training outcomes related to rotation organization. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Radiologist Characteristics Associated with Interpretive Performance of Screening Mammography: A National Mammography Database (NMD) Study

Background Factors affecting radiologists' performance in screening mammography interpretation remain poorly understood. Purpose To identify radiologists characteristics that affect screening mammography interpretation performance. Materials and Methods This retrospective study included 1223 radiologists in the National Mammography Database (NMD) from 2008 to 2019 who could be linked to Centers for Medicare & Medicaid Services (CMS) datasets. NMD screening performance metrics were extracted. Acceptable ranges were defined as follows: recall rate (RR) between 5% and 12%; cancer detection rate (CDR) of at least 2.5 per 1000 screening examinations; positive predictive value of recall (PPV1) between 3% and 8%; positive predictive value of biopsies recommended (PPV2) between 20% and 40%; positive predictive value of biopsies performed (PPV3) between the 25th and 75th percentile of study sample; invasive CDR of at least the 25th percentile of the study sample; and percentage of ductal carcinoma in situ (DCIS) of at least the 25th percentile of the study sample. Radiologist characteristics extracted from CMS datasets included demographics, subspecialization, and clinical practice patterns. Multivariable stepwise logistic regression models were performed to identify characteristics independently associated with acceptable performance for the seven metrics. The most influential characteristics were defined as those independently associated with the majority of the metrics (at least four). Results Relative to radiologists practicing in the Northeast, those in the Midwest were more likely to achieve acceptable RR, PPV1, PPV2, and CDR (odds ratio [OR], 1.4-2.5); those practicing in the West were more likely to achieve acceptable RR, PPV2, and PPV3 (OR, 1.7-2.1) but less likely to achieve acceptable invasive CDR (OR, 0.6). Relative to general radiologists, breast imagers were more likely to achieve acceptable PPV1, invasive CDR, percentage DCIS, and CDR (OR, 1.4-4.4). Those performing diagnostic mammography were more likely to achieve acceptable PPV1, PPV2, PPV3, invasive CDR, and CDR (OR, 1.9-2.9). Those performing breast US were less likely to achieve acceptable PPV1, PPV2, percentage DCIS, and CDR (OR, 0.5-0.7). Conclusion The geographic location of the radiology practice, subspecialization in breast imaging, and performance of diagnostic mammography are associated with better screening mammography performance; performance of breast US is associated with lower performance. ^©RSNA, 2021 Online supplemental material is available for this article.

Evaluation of diagnostic ultrasound use in a breast cancer detection strategy in Northern Peru

To evaluate the diagnostic impact of point-of-care breast ultrasound by trained primary care physicians (PCPs) as part of a breast cancer detection program using clinical breast exam in an underserved region of Peru. Medical records and breast ultrasound images of symptomatic women presenting to the Breast Cancer Detection Model (BCDM) in Trujillo, Peru were collected from 2017–2018. Performance was measured against final outcomes derived from regional cancer center medical records, fine needle aspiration results, patient follow-up (sensitivity, specificity, positive, and negative predictive values), and by percent agreement with the retrospective, blinded interpretation of images by a fellowship-trained breast radiologist, and a Peruvian breast surgeon. The diagnostic impact of ultrasound, compared to clinical breast exam (CBE), was calculated for actual practice and for potential impact of two alternative reporting systems. Of the 171 women presenting for breast ultrasound, 23 had breast cancer (13.5%). Breast ultrasound used as a triage test (current practice) detected all cancer cases (including four cancers missed on confirmatory CBE). PCPs showed strong agreement with radiologist and surgeon readings regarding the final management of masses (85.4% and 80.4%, respectively). While the triage system yielded a similar number of biopsies as CBE alone, using the condensed and full BI-RADS systems would have reduced biopsies by 60% while identifying 87% of cancers immediately and deferring 13% to six-month follow-up. Point-of-care ultrasound performed by trained PCPs improves diagnostic accuracy for managing symptomatic women over CBE alone and enhances access. Greater use of BI-RADS to guide management would reduce the diagnostic burden substantially.

Improving radiologist's ability in identifying particular abnormal lesions on mammograms through training test set with immediate feedback

It has been shown that there are differences in diagnostic accuracy of cancer detection on mammograms, from below 50% in developing countries to over 80% in developed world. One previous study reported that radiologists from a population in Asia displayed a low mammographic cancer detection of 48% compared with over 80% in developed countries, and more importantly, that most lesions missed by these radiologists were spiculated masses or stellate lesions. The aim of this study was to explore the performance of radiologists after undertaking a training test set which had been designed to improve the capability in detecting a specific type of cancers on mammograms. Twenty-five radiologists read two sets of 60 mammograms in a standardized mammogram reading room. The first test set focused on stellate or spiculated masses. When radiologists completed the first set, the system displayed immediate feedback to the readers comparing their performances in each case with the truth of cancer cases and cancer types so that the readers could identify individual-based errors. Later radiologists were asked to read the second set of mammograms which contained different types of cancers including stellate/spiculated masses, asymmetric density, calcification, discrete mass and architectural distortion. Case sensitivity, lesion sensitivity, specificity, receiver operating characteristics (ROC) and Jackknife alternative free-response receiver operating characteristics (JAFROC) were calculated for each participant and their diagnostic accuracy was compared between two sessions. Results showed significant improvement among radiologists in case sensitivity (+ 11.4%; P < 0.05), lesion sensitivity (+ 18.7%; P < 0.01) and JAFROC (+ 11%; P < 0.01) in the second set compared with the first set. The increase in diagnostic accuracy was also recorded in the detection of stellate/spiculated mass (+ 20.6%; P < 0.05). This indicated that the performance of radiologists in detecting malignant lesions on mammograms can be improved if an appropriate training intervention is applied after the readers' weakness and strength are identified.

Radiologists and Clinical Trials: Part 1 The Truth About Reader Disagreements

The debate over human visual perception and how medical images should be interpreted have persisted since X-rays were the only imaging technique available. Concerns over rates of disagreement between expert image readers are associated with much of the clinical research and at times driven by the belief that any image endpoint variability is problematic. The deeper understanding of the reasons, value, and risk of disagreement are somewhat siloed, leading, at times, to costly and risky approaches, especially in clinical trials. Although artificial intelligence promises some relief from mistakes, its routine application for assessing tumors within cancer trials is still an aspiration. Our consortium of international experts in medical imaging for drug development research, the Pharma Imaging Network for Therapeutics and Diagnostics (PINTAD), tapped the collective knowledge of its members to ground expectations, summarize common reasons for reader discordance, identify what factors can be controlled and which actions are likely to be effective in reducing discordance. Reinforced by an exhaustive literature review, our work defines the forces that shape reader variability. This review article aims to produce a singular authoritative resource outlining reader performance's practical realities within cancer trials, whether they occur within a clinical or an independent central review.

Assessing the Positive Predictive Value of Architectural Distortion Detected with Digital Breast Tomosynthesis in BI-RADS 4 Cases

OBJECTIVE

The purpose of this study was to evaluate the positive predictive value of biopsy (PPV3) of architectural distortion (AD) detected on digital breast tomosynthesis (DBT) in BI-RADS 4 cases, where suspicion for malignancy remains broad.

METHODS

This Institutional Review Board-approved, retrospective study included screening and diagnostic mammograms performed from August 2015 to December 2017 with DBT and digital mammography (DM) revealing suspicious AD with a BI-RADS 4 assessment. Medical records were reviewed for clinical data, imaging, and pathology results. Malignancy rate was assessed by lesion visibility on DM and DBT. Multivariate analysis was performed to assess the odds ratio (OR) of malignancy.

RESULTS

A total of 63/179 cases were malignant, yielding a PPV3 of 35%. No significant difference in PPV3 was found by race, personal or family history of breast cancer, presence of microcalcifications, or mammogram type. Architectural distortion was more likely to be malignant when an US correlate was present (PPV3 49% vs 19%; P < 0.0001). Multivariate analysis demonstrated a 3-fold increased OR for malignancy with an US correlate present (P = 0.005). Lesion visibility analysis revealed a higher PPV3 for AD visible on DM-DBT compared with DBT alone (44% vs 26%; P = 0.01) and when an US correlate was present (DM-DBT 54% vs 30%, P = 0.02; DBT-only 43% vs 11%, P < 0.001).

CONCLUSIONS

Tomosynthesis-detected BI-RADS 4 AD are malignant in 35% of cases and are more likely to be malignant if an US correlate is present and if visible on both DM and DBT.

Expert radiologist review at a hepatobiliary multidisciplinary tumor board: impact on patient management

PURPOSE

To identify the frequency, source, and management impact of discrepancies between the initial radiology report and expert reinterpretation occurring in the context of a hepatobiliary multidisciplinary tumor board (MTB).

METHODS

This retrospective study included 974 consecutive patients discussed at a weekly MTB at a large tertiary care academic medical center over a 2-year period. A single radiologist with dedicated hepatobiliary imaging expertise attended all conferences to review and discuss the relevant liver imaging and rated the concordance between original and re-reads based on RADPEER scoring criteria. Impact on management was based on the conference discussion and reflected changes in follow-up imaging, recommendations for biopsy/surgery, or liver transplant eligibility.

RESULTS

Image reinterpretation was discordant with the initial report in 19.9% (194/974) of cases (59.8%, 34.5%, 5.7% RADPEER 2/3/4 discrepancies, respectively). A change in LI-RADS category occurred in 59.8% of discrepancies. Most common causes of discordance included re-classification of a lesion as benign rather than malignant (16.0%) and missed tumor recurrence (13.9%). Impact on management occurred in 99.0% of discordant cases and included loco-regional therapy instead of follow-up imaging (19.1%), follow-up imaging instead of treatment (17.5%), and avoidance of biopsy (12.4%). 11.3% received OPTN exception scores due to the revised interpretation, and 8.8% were excluded from listing for orthotopic liver transplant.

CONCLUSION

Even in a sub-specialized abdominal imaging academic practice, expert radiologist review in the MTB setting identified discordant interpretations and impacted management in a substantial fraction of patients, potentially impacting transplant allocation. The findings may impact how abdominal imaging sections best staff advanced MTBs.

Lumbar MRI Reporting Efficiency for Trainees Over the Academic Year: An Opportunity for Improving Clinical Workflows in Academic Medical Centers

PURPOSE

Reporting efficiency is commonly used to measure performance and quality in diagnostic imaging. For academic centers, balancing the clinical demand for efficient reporting and educational obligation to trainees remains a major challenge. The objective of this study was to quantify the effect of trainee education on reporting efficiency over the academic year (July to June) for a single diagnostic imaging examination type.

METHODS

The authors reviewed a 10-year data set of lumbar spinal MRI reports and time-stamp data and compared change in mean reporting time for trainee versus attending radiologist-only reports. Odds ratios, linear regression, and correlation analysis were performed to evaluate relationships of mean and cumulative reporting times, volume, and study month.

RESULTS

Mean reporting time for the trainee group peaked in July (287.2 ± 4.9 min). The largest month-to-month increase was from June to July (+46.3 min, P < .01) for the trainees, and July showed the largest deviation from annualized mean reporting time (odds ratio, 1.31; 95% confidence interval, 1.16-1.47; P < .001). Mean reporting time improved linearly over the course of the academic year from July to June for trainees (R² = 0.59, P = .002), but this effect was absent in the attending radiologist-only group (P = .52).

CONCLUSIONS

This study quantifies the effect of trainee education on reporting efficiency and models the operational "learning curve" of improved performance over the academic year. These data may inform staffing and workflow improvement efforts in academic radiology departments.

Pull back the curtain: External data validation is an essential element of quality improvement benchmark reporting

BACKGROUND

Accurate and reliable data are pivotal to credible risk-adjusted modeling and hospital benchmarking. Evidence assessing the reliability and accuracy of data elements considered as variables in risk-adjustment modeling and measurement of outcomes is lacking. This deficiency holds the potential to compromise benchmarking integrity. We detail the findings of a longitudinal program to evaluate the impact of external data validation on data validity and reliability for variables utilized in benchmarking of trauma centers.

METHODS

A collaborative quality initiative-based study was conducted of 29 trauma centers from March 2010 through December 2018. Case selection criteria were applied to identify high-yield cases that were likely to challenge data abstractors. There were 127,238 total variables validated (i.e., reabstracted, compared, and reported to trauma centers). Study endpoints included data accuracy (agreement between registry data and contemporaneous documentation) and reliability (consistency of accuracy within and between hospitals). Data accuracy was assessed by mean error rate and type (under capture, inaccurate capture, or over capture). Cohen's kappa estimates were calculated to evaluate reliability.

RESULTS

There were 185,120 patients that met the collaborative inclusion criteria. There were 1,243 submissions reabstracted. The initial validation visit demonstrated the highest mean error rate at 6.2% ± 4.7%, and subsequent validation visits demonstrated a statistically significant decrease in error rate compared with the first visit (p < 0.05). The mean hospital error rate within the collaborative steadily improved over time (2010, 8.0%; 2018, 3.2%) compared with the first year (p < 0.05). Reliability of substantial or higher (kappa ≥0.61) was demonstrated in 90% of the 20 comorbid conditions considered in the benchmark risk-adjustment modeling, 39% of these variables exhibited a statistically significant (p < 0.05) interval decrease in error rate from the initial visit.

CONCLUSION

Implementation of an external data validation program is correlated with increased data accuracy and reliability. Improved data reliability both within and between trauma centers improved risk-adjustment model validity and quality improvement program feedback.

Between Always and Never: Evaluating Uncertainty in Radiology Reports Using Natural Language Processing

The ideal radiology report reduces diagnostic uncertainty, while avoiding ambiguity whenever possible. The purpose of this study was to characterize the use of uncertainty terms in radiology reports at a single institution and compare the use of these terms across imaging modalities, anatomic sections, patient characteristics, and radiologist characteristics. We hypothesized that there would be variability among radiologists and between subspecialities within radiology regarding the use of uncertainty terms and that the length of the impression of a report would be a predictor of use of uncertainty terms. Finally, we hypothesized that use of uncertainty terms would often be interpreted by human readers as "hedging." To test these hypotheses, we applied a natural language processing (NLP) algorithm to assess and count the number of uncertainty terms within radiology reports. An algorithm was created to detect usage of a published set of uncertainty terms. All 642,569 radiology report impressions from 171 reporting radiologists were collected from 2011 through 2015. For validation, two radiologists without knowledge of the software algorithm reviewed report impressions and were asked to determine whether the report was "uncertain" or "hedging." The relationship between the presence of 1 or more uncertainty terms and the human readers' assessment was compared. There were significant differences in the proportion of reports containing uncertainty terms across patient admission status and across anatomic imaging subsections. Reports with uncertainty were significantly longer than those without, although report length was not significantly different between subspecialities or modalities. There were no significant differences in rates of uncertainty when comparing the experience of the attending radiologist. When compared with reader 1 as a gold standard, accuracy was 0.91, sensitivity was 0.92, specificity was 0.9, and precision was 0.88, with an F1-score of 0.9. When compared with reader 2, accuracy was 0.84, sensitivity was 0.88, specificity was 0.82, and precision was 0.68, with an F1-score of 0.77. Substantial variability exists among radiologists and subspecialities regarding the use of uncertainty terms, and this variability cannot be explained by years of radiologist experience or differences in proportions of specific modalities. Furthermore, detection of uncertainty terms demonstrates good test characteristics for predicting human readers' assessment of uncertainty.

Supervised Learning Occurs in Visual Perceptual Learning of Complex Natural Images

There have been long-standing debates regarding whether supervised or unsupervised learning mechanisms are involved in visual perceptual learning (VPL) [1-14]. However, these debates have been based on the effects of simple feedback only about response accuracy in detection or discrimination tasks of low-level visual features such as orientation [15-22]. Here, we examined whether the content of response feedback plays a critical role for the acquisition and long-term retention of VPL of complex natural images. We trained three groups of human subjects (n = 72 in total) to better detect "grouped microcalcifications" or "architectural distortion" lesions (referred to as calcification and distortion in the following) in mammograms either with no trial-by-trial feedback, partial trial-by-trial feedback (response correctness only), or detailed trial-by-trial feedback (response correctness and target location). Distortion lesions consist of more complex visual structures than calcification lesions [23-26]. We found that partial feedback is necessary for VPL of calcifications, whereas detailed feedback is required for VPL of distortions. Furthermore, detailed feedback during training is necessary for VPL of distortion and calcification lesions to be retained for 6 months. These results show that although supervised learning is heavily involved in VPL of complex natural images, the extent of supervision for VPL varies across different types of complex natural images. Such differential requirements for VPL to improve the detectability of lesions in mammograms are potentially informative for the professional training of radiologists.

Rates of positive lung cancer screening examinations in academic versus community practice

The benefits and harms of lung cancer screening reported in the National Lung Screening Trial (NLST) likely differ from those observed in academic and community settings. High rates of positive screening findings in the NLST led to the development of the Lung CT Screening Reporting and Data System (Lung-RADS) to standardize interpretation and reporting. We conducted a prospective observational study of lung cancer screening data from four lung cancer screening sites in North Carolina to compare prospective use of Lung-RADS in a real-world screened population versus Lung-RADS retrospectively applied to the NLST, and to determine if Lung-RADS assessment use differs in academic versus community settings. We included 4,037 screening examinations from 11/2014 to 12/2018 in academic and community sites and 75,126 NLST LDCT screening exams. On baseline screening exams, the proportion of positive LDCT exams was higher in community versus academic sites or the NLST (17.7% vs. 11.4% and 13.6%, P value <0.01). On subsequent screens, the proportion of positive exams was lowest in the NLST and higher in community and academic sites (5.9% vs. 12.7% and 11.6%, P value <0.01). After adjusting for age, race, sex, and smoking status, patients screened at academic versus community sites were 34% less likely to have a positive screen at baseline [adjusted odds ratio (aOR) =0.66; 95% confidence interval (95% CI): 0.51-0.86] but on subsequent examinations, there was no difference in academic versus community sites (aOR =0.91; 95% CI: 0.58-1.43). Our findings may be due to differences in radiologists' training or experiences or the availability of prior images for comparison.

Comparison between execution and reading time of 3D ABUS versus HHUS

BACKGROUND

Breast density is an independent risk factor for breast cancer. Mammography is supplemented with handheld ultrasound (HHUS) to increase sensitivity. Automatic breast ultrasound (ABUS) is an alternative to HHUS. Our study wanted to assess the difference in execution and reading time between ABUS and HHUS.

METHODS AND MATERIALS

N = 221 women were evaluated consecutively between January 2019 and June 2019 (average age 53 years; range 24-89). The execution and reading time of ABUS and HHUS was calculated with an available stopwatch. Time started for both procedures when the patient was ready on the examination table to be examined to the end of image acquisition and interpretation.

RESULTS

No patients interrupted the exam due to pain or discomfort. N = 221 women underwent ABUS and HHUS; N = 11 patients refused to undergo both procedures due to time constraints and refused ABUS; therefore, 210 patients were enrolled with both ABUS and HHUS available. The average time to perform and read the exam was 5 min for HHUS (DS ± 1.5) with a maximum time of 11 min and a minimum of 2 min. The average time with ABUS was 17 min (DS ± 3.8, with a maximum time of 31 min and a minimum time of 9 min). The ABUS technique took longer to be performed in all patients, with an average difference of 11 min (range 3-23 min) per patient, P < 0,001. Separating ABUS execution from reading time we highlighted as ABUS execution is more time-consuming respect HHUS. In addition, we can underline that time required by radiologists is longer for ABUS even only considering the interpretation time of the exam.

CONCLUSION

A significant difference was observed in the execution and reading time of the two exams, where the HHUS method was more rapid and tolerated.

Telemammography: Technical Advances Improve Patient Access in Breast Care

Screening mammography's efficacy in reducing breast cancer deaths depends on patient compliance with screening recommendations and the radiologist's interpretative skills. Reasons for suboptimal screening compliance may be multifactorial, including possible limitations in access. Additionally, while studies show experienced breast radiologists are more accurate in their mammographic interpretation, only a minority of the nation's mammograms are interpreted by breast imaging specialists. To simultaneously optimize the benefit of early breast cancer detection while minimizing the harms associated with a false positive interpretation, delivery models that help improve access to breast expertise should be considered. Telemammography is one such delivery model that may be underutilized in current practice. While radiologists and other stakeholders of healthcare have accepted teleradiology interpretation of non-mammography studies as routine, telemammography use and acceptance is less well known. In this article, we review the operational components of a telemammography practice in today's information- and technology-dependent society. Current use of telemammography and remaining potential challenges are discussed. Telemammography can improve healthcare delivery and access by bringing together patients and breast expertise. If accepted, use of telemammography can help meet Centers for Disease Control's Healthy People 2020 goals related to breast cancer.

Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study

BACKGROUND

Mammography is the current standard for breast cancer screening. This study aimed to develop an artificial intelligence (AI) algorithm for diagnosis of breast cancer in mammography, and explore whether it could benefit radiologists by improving accuracy of diagnosis.

METHODS

In this retrospective study, an AI algorithm was developed and validated with 170 230 mammography examinations collected from five institutions in South Korea, the USA, and the UK, including 36 468 cancer positive confirmed by biopsy, 59 544 benign confirmed by biopsy (8827 mammograms) or follow-up imaging (50 717 mammograms), and 74 218 normal. For the multicentre, observer-blinded, reader study, 320 mammograms (160 cancer positive, 64 benign, 96 normal) were independently obtained from two institutions. 14 radiologists participated as readers and assessed each mammogram in terms of likelihood of malignancy (LOM), location of malignancy, and necessity to recall the patient, first without and then with assistance of the AI algorithm. The performance of AI and radiologists was evaluated in terms of LOM-based area under the receiver operating characteristic curve (AUROC) and recall-based sensitivity and specificity.

FINDINGS

The AI standalone performance was AUROC 0·959 (95% CI 0·952-0·966) overall, and 0·970 (0·963-0·978) in the South Korea dataset, 0·953 (0·938-0·968) in the USA dataset, and 0·938 (0·918-0·958) in the UK dataset. In the reader study, the performance level of AI was 0·940 (0·915-0·965), significantly higher than that of the radiologists without AI assistance (0·810, 95% CI 0·770-0·850; p<0·0001). With the assistance of AI, radiologists' performance was improved to 0·881 (0·850-0·911; p<0·0001). AI was more sensitive to detect cancers with mass (53 [90%] vs 46 [78%] of 59 cancers detected; p=0·044) or distortion or asymmetry (18 [90%] vs ten [50%] of 20 cancers detected; p=0·023) than radiologists. AI was better in detection of T1 cancers (73 [91%] vs 59 [74%] of 80; p=0·0039) or node-negative cancers (104 [87%] vs 88 [74%] of 119; p=0·0025) than radiologists.

INTERPRETATION

The AI algorithm developed with large-scale mammography data showed better diagnostic performance in breast cancer detection compared with radiologists. The significant improvement in radiologists' performance when aided by AI supports application of AI to mammograms as a diagnostic support tool.

FUNDING

Lunit.

Building and Exploitation of Learning Curves to Train Radiographer Students in X-Ray CT Image Postprocessing

INTRODUCTION

This study aims to construct learning curves related to the realization of standardized postprocessing by radiographer students and to discuss their exploitation and interest.

MATERIALS AND METHODS

This study was carried out in 21 French students in their 3rd year of training. Two postprocessing protocols in CT (#1 traumatic shoulder; #2 petrous bone) were repeated 15 times by each student. Each achievement was timed to obtain overall learning curves. The realization accuracy was also assessed for each student at each repetition.

RESULTS

The learning rates for the two protocols are 63% and 56%, respectively. The number of repetitions to reach the reference time for each protocol is 11 and 12, respectively. In both protocols, the standard deviations are significantly reduced and stabilized during repetitions. The mean accuracy progresses more quickly in protocol #1.

DISCUSSION

The measured learning rates reflect a rapid learning process for each protocol. The analysis of the standard deviations shows that students have reached a homogeneous level. The average times and accuracies measured during the last repetitions show that the group has reached a high level of performance. Building learning curves helps students measure their progress and motivates them.

CONCLUSION

Obtaining learning curves allows trainers/supervisors to qualify the learning difficulty of a task while motivating students/radiographers. The use of learning curves is inline with the competency-based training paradigm.

Utility of Supplemental Training to Improve Radiologist Performance in Breast Cancer Screening: A Literature Review

PURPOSE

The authors evaluate whether supplemental training for radiologists improves their breast screening performance and how this is measured.

METHODS

A systematic search was conducted in PubMed on August 3, 2017. Articles were included if they described supplemental training for radiologists reading mammograms to improve their breast screening performance and at least one outcome measure was reported. Study quality was assessed using the Medical Education Research Study Quality Instrument.

RESULTS

Of 2,199 identified articles, 18 were included, of which 17 showed improvement on at least one of the outcome measures, for at least one training activity or subgroup. Two measurement approaches were found. For the first approach, measuring performance on test sets, sensitivity, and specificity were the most reported outcomes (8 of 11 studies). Recall rate is the most reported outcome (6 of 7 studies) for the second approach, which measures performance in actual screening practice. The studies were mainly of moderate quality (Medical Education Research Study Quality Instrument score 11.7 ± 1.7), caused by small sample sizes and the lack of a control group.

CONCLUSIONS

Supplemental training helps radiologists improve their screening performance, despite the mainly moderate quality of the studies. There is a need for better designed studies. Future studies should focus on performance in actual screening practice and should look for methods to isolate the training effect. If test sets are used, focus should be on knowledge about correlation between performance on test sets and actual screening practice.

MRI evaluation of extramural vascular invasion by inexperienced radiologists

OBJECTIVE

We proposed to determine whether the performance of inexperienced radiologists in determining extramural vascular invasion (EMVI) in rectal cancer on MRI can be promoted by means of targeted training.

METHODS

230 rectal cancer patients who underwent pre-operative chemoradiotherapy were included. Pre-therapy and post-therapy MR images and pathology EMVI evaluation were available for cases. 230 cases were randomly divided into 150 training cases and 80 testing cases, including 40 testing case A and 40 testing case B. Four radiologists were included for MRI EMVI evaluation, who were divided into targeted training group and non-targeted training group. The two groups evaluated testing case A at baseline, 3 month and 6 month, evaluated testing case B at 6 month. The main outcome was agreement with expert-reference for pre-therapy and post-therapy evaluation, the other outcome was accuracy with pathology for post-therapy evaluation.

RESULTS

After 6 months of training, targeted training group showed statistically higher agreement with expert-reference than non-targeted training group for both pre-therapy and post-therapy MRI EMVI evaluation of testing case A and testing case B, all p < 0.05. Targeted training group also showed significantly higher accuracy with pathology than non-targeted training group for post-therapy evaluation of testing case A and testing case B after 6 months of training, all p < 0.05.

CONCLUSION

The diagnostic performance for MRI EMVI evaluation could be promoted by targeted training for inexperienced radiologist.

ADVANCES IN KNOWLEDGE

This study provided the first evidence that after 6 month targeted training, inexperienced radiologists demonstrated improved diagnostic performance, with a 20% increase in agreement with expert-reference for both pre-therapy and post-therapy MRI EMVI evaluation and also a 20% increase in or accuracy with pathology for post-therapy evaluation, while inexperienced radiologists could not gain obvious improvement in MRI EMVI evaluation through the same period of regular clinical practice. It indicated that targeted training may be necessary for helping inexperienced radiologist to acquire adequate experience for the MRI EMVI evaluation of rectal cancer, especially for radiologist who works in a medical unit where MRI EMVI diagnosis is uncommon.

Does learning from mistakes have to be painful? Analysis of 5 years' experience from the Leeds radiology educational cases meetings identifies common repetitive reporting errors and suggests acknowledging and celebrating excellence (ACE) as a more positive way of teaching the same lessons

BACKGROUND

The Royal College of Radiologists (RCR) and General Medical Council (GMC) encourage learning from mistakes. But negative feedback can be a demoralising process with adverse implications for staff morale, clinical engagement, team working and perhaps even patient outcomes. We first reviewed the literature regarding positive feedback and teamworking. We wanted to see if we could reconcile our guidance to review and learn from mistakes with evidence that positive interactions had a better effect on teamworking and outcomes than negative interactions. We then aimed to review and categorise the over 600 (mainly discrepancy) cases discussed in our educational cases meeting into educational 'themes'. Finally, we explored whether we could use these educational themes to deliver the same teaching points in a more positive way.

METHODS AND RESULTS

The attendance records, programmes and educational cases from 30 consecutive bimonthly meetings between 2011 and 2017 were prospectively collated and retrospectively analysed. Six hundred and thirty-two cases were collated over the study period where 76% of the cases submitted were discrepancies, or perceived errors. Eight percent were 'good spots' where examples of good calls, excellent reporting, exemplary practice or subtle findings that were successfully reported. Eight percent were educational cases in which no mistake had been made. The remaining 7% included procedural complications or system errors.

CONCLUSION

By analysing the pattern of discrepancies in a department and delivering the teaching in a less negative way, the 'lead' of clinical errors can be turned in to the 'gold' of useful educational tools. Interrogating the whole database periodically can enable a more constructive, wider view of the meeting itself, highlight recurrent deficiencies in practice, and point to where the need for continuing medical training is greatest. Three ways in which our department have utilised this material are outlined: the use of 'good spots', arrangement of targeted teaching and production of specialist educational material. These techniques can all contribute to a more positive learning experience with the emphasis on acknowledging and celebrating excellence (ACE).

Characterizing Diagnostic Search Patterns in Digital Breast Pathology: Scanners and Drillers

Following a baseline demographic survey, 87 pathologists interpreted 240 digital whole slide images of breast biopsy specimens representing a range of diagnostic categories from benign to atypia, ductal carcinoma in situ, and invasive cancer. A web-based viewer recorded pathologists' behaviors while interpreting a subset of 60 randomly selected and randomly ordered slides. To characterize diagnostic search patterns, we used the viewport location, time stamp, and zoom level data to calculate four variables: average zoom level, maximum zoom level, zoom level variance, and scanning percentage. Two distinct search strategies were confirmed: scanning is characterized by panning at a constant zoom level, while drilling involves zooming in and out at various locations. Statistical analysis was applied to examine the associations of different visual interpretive strategies with pathologist characteristics, diagnostic accuracy, and efficiency. We found that females scanned more than males, and age was positively correlated with scanning percentage, while the facility size was negatively correlated. Throughout 60 cases, the scanning percentage and total interpretation time per slide decreased, and these two variables were positively correlated. The scanning percentage was not predictive of diagnostic accuracy. Increasing average zoom level, maximum zoom level, and zoom variance were correlated with over-interpretation.

Implementation of lung cancer screening at the national level: Polish example

In Poland the national demonstration lung cancer screening program is about to be started in 2019. We share our concerns and discussing most important topics to be resolved while preparing such a program. The decisions made are virtually based on available scientific data and the results of two randomized controlled trials but also on the personal experience gained during the lung cancer screening studies performed in Poland. The most important and comprehensive guidelines and statements, both European and American, have been searched to find an optimal solution adjusted to the Polish national circumstances-as we assume that should be done in each country implementing such a program.

Chest X-ray evaluation training: impact of normal and abnormal image ratio and instructional sequence

CONTEXT

Medical image perception training generally focuses on abnormalities, whereas normal images are more prevalent in medical practice. Furthermore, instructional sequences that let students practice prior to expert instruction (inductive) may lead to improved performance compared with methods that give students expert instruction before practice (deductive). This study investigates the effects of the proportion of normal images and practice-instruction order on learning to interpret medical images. It is hypothesised that manipulation of the proportion of normal images will lead to a sensitivity-specificity trade-off and that students in practice-first (inductive) conditons need more time per practice case but will correctly identify more test cases.

METHODS

Third-year medical students (n = 103) learned radiograph interpretation by practising cases with, respectively, 30% or 70% normal radiographs prior to expert instruction (practice-first order) or after expert instruction (instruction-first order). After training, students performed a test (60% normal) and sensitivity (% of correctly identified abnormal radiographs), specificity (% of correctly identified normal radiographs), diagnostic performance (% of correct diagnoses) and case duration were measured.

RESULTS

The conditions with 30% of normal images scored higher on sensitivity but the conditions with 70% of normal images scored higher on specificity, indicating a sensitivity and specificity trade-off. Those who participated in inductive conditions took less time per practice case but more per test case. They had similar test sensitivity, but scored lower on test specificity.

CONCLUSIONS

The proportion of normal images impacted the sensitivity-specificity trade-off. This trade-off should be an important consideration for the alignment of training with future practice. Furthermore, the deductive conditions unexpectedly scored higher on specificity when participants took less time per case. An inductive approach did not lead to higher diagnostic performance, possibly because participants might already have relevant prior knowledge. Deductive approaches are therefore advised for the training of advanced learners.

Survival analysis of hierarchical learning curves in assessment of cardiac device and procedural safety

Many Americans rely on cardiac surgical procedures and devices such as pacemakers and thrombolytic catheters to treat or manage their cardiovascular diseases. However, the failure of these cardiac devices and procedures could have grave consequences. One reason cardiac devices tended to fail was due to physician error; there is a learning effect for the physician or operator to come up to speed in skillfully implanting devices and conducting procedures. In order to better understand these learning effects, we had previously modeled the resulting learning curve effects in simulations a hierarchical setting with physicians clustered within institutions using our unique methodology (see the work of Govindarajulu et al 2017). Previously, we had employed these in hierarchical linear modeling and also in generalized estimating equations. In this setting, we have demonstrated how to apply similar methodology but revised in a survival analytic framework or time-to-event analyses. Through simulations and real dataset applications, we found that, out of the three shapes modeled to fit the learning curve, the logarithmic shape tended to have the best fit, similar to previous work (see the work of Govindarajulu et al 2017). However, as seen before, modeling the learning rate can be dataset specific and one shape may be better than another. We learned that modeling the learning rate could also be applied in the survival analysis setting through this new methodology. The goal of this paper is to model cardiac device and procedure learning curve effects in a time-to-event setting so that this knowledge may allow for the improvement of both short and long-term patient survival.

Automated Breast Ultrasound Interpretation Times: A Reader Performance Study

RATIONALE AND OBJECTIVES

This study aimed to determine the average time for breast radiologists of varied experience to interpret automated breast ultrasound (ABUS) examinations.

MATERIALS AND METHODS

A reader performance study was conducted on female patients, with ACR BI-RADS 4 breast density classifications of C or D, who received both an ABUS screening examination and a digital mammogram from 2013 to 2014 at an academic institution. Three faculty breast radiologists with varied levels of ABUS experience (advanced, intermediate, novice) read all ABUS examinations, with interpretation times and final impressions (categorized as "normal" or "abnormal") recorded for each examination.

RESULTS

Ninety-nine patients were included, with all readers demonstrating an average ABUS interpretation time of less than 3 minutes. Compared to the other two readers, the intermediate reader had a significantly longer mean interpretation time at 2.6 minutes (95% confidence interval 2.4-2.8; P < .001). In addition to having the shortest mean interpretation time, the novice reader also demonstrated reduced times in subsequent interpretations, with a significant decrease in interpretation times of 3.1 seconds (95% confidence interval 0.4-5.8) for every 10 ABUS examinations interpreted (P < .05).

CONCLUSIONS

Overall, mean ABUS interpretation time by radiologists of all experience levels was short, at less than 3 minutes per examination, which should not deter radiologists from incorporating ABUS examinations into a busy clinical environment.

Mammographic detection of breast cancer in a non-screening country

OBJECTIVE:

To compare the diagnostic accuracy between radiologists' from a country with and without breast cancer screening.

METHODS:

All participating radiologists gave informed consent. A test-set involving 60 mammographic cases (20 cancer and 40 non-cancer) were read by 11 radiologists from a non-screening (NS) country during a workshop in July 2016. 52 radiologists from a screening country read the same test-set at the Royal Australian and New Zealand College of Radiologists' meetings in July 2015. The screening radiologists were classified into two groups: those with less than or equal to 5 years of experience; those with more than 5 years of experience, and each group was compared to the group of NS radiologists. A Kruskal-Wallis test followed by post-hoc multiple comparisons test were used to compare measures of diagnostic accuracy among the reader groups.

RESULTS:

The diagnostic accuracy of the NS radiologists was significantly lower in terms of sensitivity [mean = 54.0; 95% confidence interval (CI) (40.0-67.0)], location sensitivity [mean = 26.0; 95% CI (16.0-37.0)], receive roperating characteristic area under curve [mean = 73.0; 95% CI (66.5-81.0)] and Jackknifefree-response receiver operating characteristics figure-of-merit [mean = 45.0; 95% CI (40.0-50.0)] when compared with the less and more experienced screening radiologists, whilst no difference in specificity [mean = 75.0; 95% CI (70.0- 81.0)] was found. No significant differences in all measured diagnostic accuracy were found between the two groups of screening radiologists.

CONCLUSION:

The mammographic performance of a group of radiologists from a country without screening program was suboptimal compared with radiologists from Australia.

ADVANCES IN KNOWLEDGE:

Identifying mammographic performance in developing countries is required to optimize breast cancer diagnosis.

Screening for breast cancer in 2018-what should we be doing today?

Although screening mammography has delivered many benefits since its introduction in Canada in 1988, questions about perceived harms warrant an up-to-date review. To help oncologists and physicians provide optimal patient recommendations, the literature was reviewed to find the latest guidelines for screening mammography, including benefits and perceived harms of overdiagnosis, false positives, false negatives, and technologic advances. For women 40-74 years of age who actually participate in screening every 1-2 years, breast cancer mortality is reduced by 40%. With appropriate corrections, overdiagnosis accounts for 10% or fewer breast cancers. False positives occur in about 10% of screened women, 80% of which are resolved with additional imaging, and 10%, with breast biopsy. An important limitation of screening is the false negatives (15%-20%). The technologic advances of digital breast tomosynthesis, breast ultrasonography, and magnetic resonance imaging counter the false negatives of screening mammography, particularly in women with dense breast tissue.

Assessing Resident Performance in Screening Mammography: Development of a Quantitative Algorithm

RATIONALE AND OBJECTIVES

This study aims to provide objective performance data and feedback, including examination volumes, recall rates, and concordance with faculty interpretations, for residents performing independent interpretation of screening mammography examinations.

METHOD AND MATERIALS

Residents (r) and faculty (f) interpret screening mammograms separately and identify non-callbacks (NCBs) and callbacks (CBs). Residents review all discordant results. The number of concordant interpretations (fCB-rCB and fNCB-rNCB) and discordant interpretations (fCB-rNCB and fNCB-rCB) are entered into a macro-driven spreadsheet. These macros weigh the data dependent on the perceived clinical impact of the resident's decision. Weighted outcomes are combined with volumes to generate a weighted mammography performance score. Rotation-specific goals are assigned for the weighted score, screening volumes, recall rate relative to faculty, and concordance rates. Residents receive one point for achieving each goal.

RESULTS

Between July 2013 and May 2017, 18,747 mammography examinations were reviewed by 31 residents, in 71 resident rotations, over 246 resident weeks. Mean resident recall rate was 9.9% and significantly decreased with resident level (R), R2 = 11.3% vs R3 = 9.4%, R4 = 9.2%. Mean resident-faculty discordance rate was 10% and significantly decreased from R2 = 12% to R4 = 9.6%. Weighted performance scores ranged from 1.1 to 2.0 (mean 1.6, standard deviation 0.17), but did not change with rotation experience. Residents had a mean goal achievement score of 2.6 (standard deviation 0.47).

CONCLUSIONS

This method provides residents with easily accessible case-by-case individualized screening outcome data over the longitudinal period of their residency, and provides an objective method of assessing resident screening mammography performance.

Fellowship training: a qualitative study of scope and purpose across one department of medicine

BACKGROUND

Fellowship training follows certification in a primary specialty or subspecialty and focusses on distinct and advanced clinical and/or academic skills. This phase of medical education is growing in prevalence, but has been an "invisible phase of postgraduate training" lacking standards for education and accreditation, as well as funding. We aimed to explore fellowship programs and examine the reasons to host and participate in fellowship training, seeking to inform the future development of fellowship education.

METHODS

During the 2013-14 academic year, we conducted interviews and focus groups to examine the current status of fellowship training from the perspectives of division heads, fellowship directors and current fellows at the Department of Medicine, University of Ottawa, Canada. Descriptive statistics were used to depict the prevailing status of fellowship training. A process of data reduction, data analysis and conclusions/verifications was performed to analyse the quantitative data.

RESULTS

We interviewed 16 division heads (94%), 15 fellowship directors (63%) and 8 fellows (21%). We identified three distinct types of fellowships. Individualized fellowships focus on the career goals of the trainee and/or the recruitment goals of the division. Clinical fellowships focus on the attainment of clinical expertise over and above the competencies of residency. Research fellowships focus on research productivity. Participants identified a variety of reasons to offer fellowships: improve academic productivity; improve clinical productivity; share/develop enhanced clinical expertise; recruit future faculty members/attain an academic position; enhance the reputation of the division/department/trainee; and enhance the scholarly environment.

CONCLUSIONS

Fellowships serve a variety of purposes which benefit both individual trainees as well as the academic enterprise. Fellowships can be categorized within a distinct taxonomy: individualized; clinical; and research. Each type of fellowship may serve a variety of purposes, and each may need distinct support and resources. Further research is needed to catalogue the operational requirements for hosting and undertaking fellowship training, and establish recommendations for educational and administrative policy and processes in this new phase of postgraduate education.

Breast Density Legislation in New England: A Survey Study of Practicing Radiologists

RATIONALE AND OBJECTIVES

This study aimed to assess radiologists' knowledge about breast density legislation as well as perceived practice changes resulting from the enactment of breast density legislation.

MATERIALS AND METHODS

This is an institutional review board-exempt anonymous email survey of 523 members of the New England Roentgen Ray Society. In addition to radiologist demographics, survey questions addressed radiologist knowledge of breast density legislation, knowledge of breast density as a risk factor for breast cancer, recommendations for supplemental screening, and perceived practice changes resulting from density notification legislation.

RESULTS

Of the 523 members, 96 responded, yielding an 18% response rate. Seventy-three percent of respondents practiced in a state with breast density legislation. Sixty-nine percent felt that breast density notification increased patient anxiety about breast cancer, but also increased patient (74%) and provider (66%) understanding of the effect of breast density on mammographic sensitivity. Radiologist knowledge of the relative risk of breast cancer when comparing breasts of different density was variable.

CONCLUSIONS

Considerable confusion and controversy regarding breast density persists, even among practicing radiologists.