Assessment of the Incremental Benefit of Computer-Aided Detection (CAD) for Interpretation of CT Colonography by Experienced and Inexperienced Readers

Prospective effects of an artificial intelligence-based computer-aided detection system for prostate imaging on routine workflow and radiologists' outcomes

OBJECTIVES

Artificial intelligence (AI) is expected to alleviate the negative consequences of rising case numbers for radiologists. Currently, systematic evaluations of the impact of AI solutions in real-world radiological practice are missing. Our study addresses this gap by investigating the impact of the clinical implementation of an AI-based computer-aided detection system (CAD) for prostate MRI reading on clinicians' workflow, workflow throughput times, workload, and stress.

MATERIALS AND METHODS

CAD was newly implemented into radiology workflow and accompanied by a prospective pre-post study design. We assessed prostate MRI case readings using standardized work observations and questionnaires. The observation period was three months each in a single department. Workflow throughput times, PI-RADS score, CAD usage and radiologists' self-reported workload and stress were recorded. Linear mixed models were employed for effect identification.

RESULTS

In data analyses, 91 observed case readings (pre: 50, post: 41) were included. Variation of routine workflow was observed following CAD implementation. A non-significant increase in overall workflow throughput time was associated with CAD implementation (mean 16.99 ± 6.21 vs 18.77 ± 9.69 min, p = .51), along with an increase in diagnostic reading time for high suspicion cases (mean 15.73 ± 4.99 vs 23.07 ± 8.75 min, p = .02). Changes in radiologists' self-reported workload or stress were not found.

CONCLUSION

Implementation of an AI-based detection aid was associated with lower standardization and no effects over time on radiologists' workload or stress. Expectations of AI decreasing the workload of radiologists were not confirmed by our real-world study.

PRE-REGISTRATION

German register for clinical trials https://drks.de/; DRKS00027391.

Development and validation of an educational software based in artificial neural networks for training in radiology (JORCAD) through an interactive learning activity

The use of Computer Aided Detection (CAD) software has been previously documented as a valuable tool to improve specialist training in Radiology. This research assesses the utility of an educational software tool aimed to train residents in Radiology and other related medical specialties and students from Medicine degree. This in-house developed software, called JORCAD, integrates a CAD system based in Convolutional Neural Networks (CNNs) with annotated cases from radiological image databases. The methodology followed for software validation was expert judgement after completing an interactive learning activity. Participants received a theoretical session and a software usage tutorial and afterwards utilized the application in a dedicated workstation to analyze a series of proposed cases of thorax computed tomography (CT) and mammography. A total of 26 expert participants from the Radiology Department at Salamanca University Hospital (15 specialists and 11 residents) fulfilled the activity and evaluated different aspects through a series of surveys: software usability, case navigation tools, CAD module utility for learning and JORCAD educational capabilities. Participants also graded imaging cases to establish JORCAD usefulness for training radiology residents. According to the statistical analysis of survey results and expert cases scoring, along with their opinions, it can be concluded that JORCAD software is a useful tool for training future specialists. The combination of CAD with annotated cases from validated databases enhances learning, offering a second opinion and changing the usual training paradigm. Including software as JORCAD in residency training programs of Radiology and other medical specialties would have a positive effect on trainees' background knowledge.

Computer-aided detection improves brain metastasis identification on non-enhanced CT in less experienced radiologists

Background

Brain metastases (BMs) are the most common intracranial tumors causing neurological complications associated with significant morbidity and mortality.

Purpose

To evaluate the effect of computer-aided detection (CAD) on the performance of observers in detecting BMs on non-enhanced computed tomography (NECT).

Material and Methods

Three less experienced and three experienced radiologists interpreted 30 NECT scans with 89 BMs in 25 cases to detect BMs with and without the assistance of CAD. The observers’ sensitivity, number of false positives (FPs), positive predictive value (PPV), and reading time with and without CAD were compared using paired t-tests. The sensitivity of CAD and the observers were compared using a one-sample t-test

Results

With CAD, less experienced radiologists’ sensitivity significantly increased from 27.7% ± 4.6% to 32.6% ± 4.8% ( P = 0.007), while the experienced radiologists’ sensitivity did not show a significant difference (from 33.3% ± 3.5% to 31.9% ± 3.7%; P = 0.54). There was no significant difference between conditions with CAD and without CAD for FPs (less experienced radiologists: 23.0 ± 10.4 and 25.0 ± 9.3; P = 0.32; experienced radiologists: 18.3 ± 7.4 and 17.3 ± 6.7; P = 0.76) and PPVs (less experienced radiologists: 57.9% ± 8.3% and 50.9% ± 7.0%; P = 0.14; experienced radiologists: 61.8% ± 12.7% and 64.0% ± 12.1%; P = 0.69). There were no significant differences in reading time with and without CAD (85.0 ± 45.6 s and 73.7 ± 36.7 s; P = 0.09). The sensitivity of CAD was 47.2% (with a PPV of 8.9%), which was significantly higher than that of any radiologist ( P < 0.001).

Conclusion

CAD improved BM detection sensitivity on NECT without increasing FPs or reading time among less experienced radiologists, but this was not the case among experienced radiologists.

Artificial intelligence in radiography: Where are we now and what does the future hold?

OBJECTIVES

This paper will outline the status and basic principles of artificial intelligence (AI) in radiography along with some thoughts and suggestions on what the future might hold. While the authors are not always able to separate the current status from future developments in this field, given the speed of innovation in AI, every effort has been made to give a view to the present with projections to the future.

KEY FINDINGS

AI is increasingly being integrated within radiography and radiographers will increasingly be working with AI based tools in the future. As new AI tools are developed it is essential that robust validation is undertaken in unseen data, supported by more prospective interdisciplinary research. A framework of stronger, more comprehensive approvals are recommended and the involvement of service users, including practitioners, patients and their carers in the design and implementation of AI tools is essential. Clearer accountability and medicolegal frameworks are required in cases of erroneous results from the use of AI-powered software and hardware. Clearer career pathways and role extension provision for healthcare practitioners, including radiographers, are required along with education in this field where AI will be central.

CONCLUSION

With the current growth rate of AI tools it is expected that many of the applications in medical imaging will continue to develop to more accurate, less expensive and more readily available versions moving from the bench to the bedside. The hope is that, alongside efficiency and increased patient throughput, patient centred care and precision medicine will find their way in, so we will not only deliver a faster, safer, seamless clinical service but also one that will have the patients at its heart.

IMPACT FOR PRACTICE

AI is already reaching clinical practice in many forms and its presence will continue to increase over the short and long-term future. Radiographers must learn to work with AI, embracing it and maximising the positive outcomes from this new technology.

Computer-Aided Diagnosis Improves the Detection of Clinically Significant Prostate Cancer on Multiparametric-MRI: A Multi-Observer Performance Study Involving Inexperienced Readers

Recently, Computer Aided Diagnosis (CAD) systems have been proposed to help radiologists in detecting and characterizing Prostate Cancer (PCa). However, few studies evaluated the performances of these systems in a clinical setting, especially when used by non-experienced readers. The main aim of this study is to assess the diagnostic performance of non-experienced readers when reporting assisted by the likelihood map generated by a CAD system, and to compare the results with the unassisted interpretation. Three resident radiologists were asked to review multiparametric-MRI of patients with and without PCa, both unassisted and assisted by a CAD system. In both reading sessions, residents recorded all positive cases, and sensitivity, specificity, negative and positive predictive values were computed and compared. The dataset comprised 90 patients (45 with at least one clinically significant biopsy-confirmed PCa). Sensitivity significantly increased in the CAD assisted mode for patients with at least one clinically significant lesion (GS > 6) (68.7% vs. 78.1%, p = 0.018). Overall specificity was not statistically different between unassisted and assisted sessions (94.8% vs. 89.6, p = 0.072). The use of the CAD system significantly increases the per-patient sensitivity of inexperienced readers in the detection of clinically significant PCa, without negatively affecting specificity, while significantly reducing overall reporting time.

CT colonography: implementation, indications, and technical performance - a follow-up national survey

BACKGROUND

Computed tomography colonography (CTC) is an accepted complement or alternative to optical colonoscopy (OC) but its implementation is incompletely analyzed, and technical performance varies between centers.

PURPOSE

To evaluate implementation, indications, and technical performance of CTC in Sweden and to evaluate compliance to international guidelines.

MATERIAL AND METHODS

A structured, self-assessed questionnaire regarding implementation and technical performance of CTC was sent to all eligible radiology departments in Sweden. Eighty-six out of 89 departments replied. Comparisons were made with similar national surveys from 2004 and 2009.

RESULTS

The number of centers performing CTC gradually increased from 23 in 2004 to 77 in 2016. In parallel, centers performing barium enema (BE) examinations have decreased from 89 in 2004 to 13 in 2016. Main reasons stated for still performing BE were lack of resources regarding CTC/OC. Main reasons for not performing CTC were lack of suitable software, lack of machine/reading time, and lack of experience. The majority of centers follow international CTC guidelines. An important exception is fecal tagging, which was implemented in only 63% of the centers. Incomplete OC remains a major indication for CTC, while preoperative CTC in colorectal cancer and follow-up after diverticulitis have emerged as new indications.

CONCLUSION

CTC today is well implemented in routine healthcare but still lacking in capacity. Indications have expanded over time, and most departments perform "state of the art" CTC, although fecal tagging is incompletely implemented.

Interobserver Variation of Colonic Polyp Measurement at Computed Tomography Colonography

BACKGROUND

The concept of "advanced polyps" is well accepted and is defined as polyps ≥10 mm and/or those having a villous component and/or demonstrating areas of dysplasia. Of these parameters, computed tomography colonography (CTC) can only document size. The accepted management of CTC-detected "advanced polyps" is to recommend excision if feasible, whereas the management of "intermediate" (6-9 mm) polyps is more controversial, and interval surveillance may be acceptable. Therefore, distinction between 6-9 mm and ≥10 mm is important.

METHODS

Datasets containing 26 polyps originally reported as between 8-12 mm in diameter were reviewed independently by 4 CTC-accredited radiologists. Observers tabulated the largest measurement for each polyp on axial, coronal, sagittal, and endoluminal views at lung-window settings. These measurements were also compared to those determined by the computer-aided detection (CAD) software.

RESULTS

The interobserver reliability intra-class correlation coefficient (ICC) for sagittal projection was 0.80 ("excellent" category of Hosmer and Lemeshow [2004]), 0.71 for axial ("acceptable"), 0.69 for coronal, and 0.41 for endoluminal ("unacceptable"). The largest of sagittal/axial/coronal measurement gave the best reliability with the smallest variance (ICC = 0.80; 95% CI 0.67-0.89). For 8 of 26 polyps, at least one radiologist's measurement placed the polyp in a different category compared to a colleague. For the majority of the polyps, the CAD significantly overestimated the readings compared to the largest of the manual measurements with an average difference of 1.6 mm (P < .0001 for sagittal/axial/coronal). This resulted in 33% of polyps falling into a different category-10% were lower and 23% were higher (P < .034).

CONCLUSION

It is apparent that around the cutoff point of 10 mm between "advanced" and "intermediate" polyps, interobserver performance is variable.

Computer-based self-training for CT colonography with and without CAD

OBJECTIVES

To determine whether (1) computer-based self-training for CT colonography (CTC) improves interpretation performance of novice readers; (2) computer-aided detection (CAD) use during training affects learning.

METHODS

Institutional review board approval and patients' informed consent were obtained for all cases included in this study. Twenty readers (17 radiology residents, 3 radiologists) with no experience in CTC interpretation were recruited in three centres. After an introductory course, readers performed a baseline assessment test (37 cases) using CAD as second reader. Then they were randomized (1:1) to perform either a computer-based self-training (150 cases verified at colonoscopy) with CAD as second reader or the same training without CAD. The same assessment test was repeated after completion of the training programs. Main outcome was per lesion sensitivity (≥ 6 mm). A generalized estimating equation model was applied to evaluate readers' performance and the impact of CAD use during training.

RESULTS

After training, there was a significant improvement in average per lesion sensitivity in the unassisted phase, from 74% (356/480) to 83% (396/480) (p < 0.001), and in the CAD-assisted phase, from 83% (399/480) to 87% (417/480) (p = 0.021), but not in average per patient sensitivity, from 93% (390/420) to 94% (395/420) (p = 0.41), and specificity, from 81% (260/320) to 86% (276/320) (p = 0.15). No significant effect of CAD use during training was observed on per patient sensitivity and specificity, nor on per lesion sensitivity.

CONCLUSIONS

A computer-based self-training program for CTC improves readers' per lesion sensitivity. CAD as second reader does not have a significant impact on learning if used during training.

KEY POINTS

• Computer-based self-training for CT colonography improves per lesion sensitivity of novice readers. • Self-training program does not increase per patient specificity of novice readers. • CAD used during training does not have significant impact on learning.

CT colonography interpretation: how to maximize polyp detection and minimize overcalling

This article outlines how to achieve maximum accuracy in interpreting CT colonography (CTC) regarding colonic findings. Interpreting extracolonic findings seen on CTC is a separate diagnostic task and will not be addressed in this article. While many interpretive pitfalls are in fact related to CTC techniques, this article focuses on issues that are related to interpretive knowledge and skills, avoiding in-depth discussions on CTC techniques. Principal methods and further tips for detecting possible polyp candidates and for confirming true soft-tissue polyps will be discussed. Specific points about optimizing interpretation strategies for difficult flat polyps including sessile serrated polyp will be raised. There are numerous interpretive pitfalls regarding the colonic interpretation of CTC. Knowledge of these pitfalls will shorten the learning curve and help achieve accurate reads.

Progress in Fully Automated Abdominal CT Interpretation

OBJECTIVE

Automated analysis of abdominal CT has advanced markedly over just the last few years. Fully automated assessment of organs, lymph nodes, adipose tissue, muscle, bowel, spine, and tumors are some examples where tremendous progress has been made. Computer-aided detection of lesions has also improved dramatically.

CONCLUSION

This article reviews the progress and provides insights into what is in store in the near future for automated analysis for abdominal CT, ultimately leading to fully automated interpretation.