Added value of double reading in diagnostic radiology,a systematic review

Assessing the impact of trained Radiologist Assistants in a busy emergency teleradiology practice: a comprehensive evaluation

PURPOSE

This study aims to study the feasibility and usefulness of trained Radiologist Assistants in a busy emergency teleradiology practice.

METHOD

This is a retrospective study over a 21-month period (January 2021 to September 2022). The study analysed archived data from 247118 peer review studies performed by Radiologist Assistants (RAs) out of a total case volume of 828526 and evaluated the rate of discrepancies, the study types commonly noted to have discrepancies, and the severity of errors. These missed findings were brought to the attention of the radiologists for approval and further decision-making.

RESULTS

Peer review by RAs was performed on 30% (n = 247118) of the total volume 828526 studies reported, and yielded additional findings including but not limited to fractures (218; 23%), hemorrhage,(94; 10%) pulmonary thromboembolism, (n = 104; 11%), Calculus (n = 75; 8%) lesion (n = 66; 5%), appendicitis(n = 50; 4%) and others. These were brought to the attention of the radiologist, who agreed in 97% (1279 out of 1318) of cases, and communicated the same to the referring facility, with an addended report.

CONCLUSION

Trained RAs can provide value to the peer review program of a busy teleradiology practice and decrease errors. This may be useful to meet the ongoing radiologist shortages.

Evaluation of Image-Defined Risk Factor (IDRF) Assessment in Patients With Intermediate-risk Neuroblastoma: A Report From the Children's Oncology Group Study ANBL0531

BACKGROUND

The International Neuroblastoma Risk Group (INRG) classifier utilizes a staging system based on pretreatment imaging criteria in which image-defined risk factors (IDRFs) are used to evaluate the extent of locoregional disease. Children's Oncology Group (COG) study ANBL0531 prospectively examined institutional determination of IDRF status and compared that to a standardized central review.

METHODS

Between 9/2009-6/2011, patients with intermediate-risk neuroblastoma were enrolled on ANBL0531 and had IDRF assessment at treating institutions. Paired COG pediatric surgeons and radiologists performed blinded central review of diagnostic imaging for the presence or absence of IDRFs. Second blinded review was performed in cases of discordance. Comparison of local and central review was performed using the Kappa coefficient to determine concordance in IDRF assessment.

RESULTS

211 patients enrolled in ANBL0531 underwent IDRF assessment; 3 patients were excluded due to poor image quality. Central reviewer pairs agreed on the presence or absence of any IDRF in 170/208 (81.7%; κ = 0.48) cases. Thirteen (6.3%) cases could not be adjudicated after second blinded review. Radiologists were more likely to identify IRDFs as present than surgeons (p < 0.001). Local and central reviewers agreed on the presence or absence of any IDRF in only108/208 (51.9%; κ = 0.06) cases.

CONCLUSIONS

Among experienced pediatric surgeons and radiologists participating in central review, concordance was moderate, with agreement in 81.7% of cases. On comparison of local and central assessment of IDRFs, concordance was poor. These data indicate that greater standardization, education, technology, and training are needed to improve the assessment of IDRFs in children with neuroblastoma.

LEVEL OF EVIDENCE

Treatment Study, Level III.

Artificial intelligence-assisted double reading of chest radiographs to detect clinically relevant missed findings: a two-centre evaluation

OBJECTIVES

To evaluate an artificial intelligence (AI)-assisted double reading system for detecting clinically relevant missed findings on routinely reported chest radiographs.

METHODS

A retrospective study was performed in two institutions, a secondary care hospital and tertiary referral oncology centre. Commercially available AI software performed a comparative analysis of chest radiographs and radiologists' authorised reports using a deep learning and natural language processing algorithm, respectively. The AI-detected discrepant findings between images and reports were assessed for clinical relevance by an external radiologist, as part of the commercial service provided by the AI vendor. The selected missed findings were subsequently returned to the institution's radiologist for final review.

RESULTS

In total, 25,104 chest radiographs of 21,039 patients (mean age 61.1 years ± 16.2 [SD]; 10,436 men) were included. The AI software detected discrepancies between imaging and reports in 21.1% (5289 of 25,104). After review by the external radiologist, 0.9% (47 of 5289) of cases were deemed to contain clinically relevant missed findings. The institution's radiologists confirmed 35 of 47 missed findings (74.5%) as clinically relevant (0.1% of all cases). Missed findings consisted of lung nodules (71.4%, 25 of 35), pneumothoraces (17.1%, 6 of 35) and consolidations (11.4%, 4 of 35).

CONCLUSION

The AI-assisted double reading system was able to identify missed findings on chest radiographs after report authorisation. The approach required an external radiologist to review the AI-detected discrepancies. The number of clinically relevant missed findings by radiologists was very low.

CLINICAL RELEVANCE STATEMENT

The AI-assisted double reader workflow was shown to detect diagnostic errors and could be applied as a quality assurance tool. Although clinically relevant missed findings were rare, there is potential impact given the common use of chest radiography.

KEY POINTS

• A commercially available double reading system supported by artificial intelligence was evaluated to detect reporting errors in chest radiographs (n=25,104) from two institutions. • Clinically relevant missed findings were found in 0.1% of chest radiographs and consisted of unreported lung nodules, pneumothoraces and consolidations. • Applying AI software as a secondary reader after report authorisation can assist in reducing diagnostic errors without interrupting the radiologist's reading workflow. However, the number of AI-detected discrepancies was considerable and required review by a radiologist to assess their relevance.

Quantitative Approach to Quality Review of Prenatal Ultrasound Examinations: Fetal Biometry

Background/Objectives: To evaluate the quality of an ultrasound practice, both large-scale and focused audits are recommended by professional organizations, but such audits can be time-consuming, inefficient, and expensive. Our objective was to develop a time-efficient, quantitative, objective, large-scale method to evaluate fetal biometry measurements for an entire practice, combined with a process for focused image review for personnel whose measurements are outliers. Methods: Ultrasound exam data for a full year are exported from commercial ultrasound reporting software to a statistical package. Fetal biometry measurements are converted to z-scores to standardize across gestational ages. For a large-scale audit, sonographer mean z-scores are compared using analysis of variance (ANOVA) with Scheffe multiple comparisons test. A focused image review is performed on a random sample of exams for sonographers whose mean z-scores differ significantly from the practice mean. A similar large-scale audit is performed, comparing physician mean z-scores. Results: Using fetal abdominal circumference measurements as an example, significant differences between sonographer mean z-scores are readily identified by the ANOVA and Scheffe test. A method is described for the blinded image audit of sonographers with outlier mean z-scores. Examples are also given for the identification and interpretation of several types of systematic errors that are unlikely to be detectable by image review, including z-scores with large or small standard deviations and physicians with outlier mean z-scores. Conclusions: The large-scale quantitative analysis provides an overview of the biometry measurements of all the sonographers and physicians in a practice, so that image audits can be focused on those whose measurements are outliers. The analysis takes little time to perform after initial development and avoids the time, complexity, and expense of auditing providers whose measurements fall within the expected range. We encourage commercial software developers to include tools in their ultrasound reporting software to facilitate such quantitative reviews.

Evaluation of multiple deep neural networks for detection of intracranial dural arteriovenous fistula on susceptibility weighted angiography imaging

BACKGROUND

The natural history of intracranial dural arteriovenous fistula (DAVF) is variable and early diagnosis is crucial in order to positively impact the clinical course of aggressive DAVF. Artificial intelligence (AI) based techniques can be promising in this regard, and in this study, we used various deep neural network (DNN) architectures to determine whether DAVF could be reliably identified on susceptibility-weighted angiography images (SWAN).

MATERIALS AND METHODS

A total of 3965 SWAN image slices from 30 digital subtraction angiographically proven DAVF patients and 4380 SWAN image slices from 40 age-matched patients with normal MRI findings as control group were included. The images were categorized as either DAVF or normal and the data was trained using various DNN such as VGG-16, EfficientNet-B0, and ResNet-50.

RESULTS

Various DNN architectures showed the accuracy of 95.96% (VGG-16), 91.75% (EfficientNet-B0), and 86.23% (ResNet-50) on the SWAN image dataset. ROC analysis yielded an area under the curve of 0.796 (p < .001), best for VGG-16 model. Criterion of seven consecutive positive slices for DAVF diagnosis yielded a sensitivity of 74.68% with a specificity of 69.15%, while setting eight slices improved the sensitivity to above 80.38%, with a decrease of specificity up to 56.38%. Based on seven consecutive positive slices criteria, EfficientNet-B0 yielded a sensitivity of 73.21% with a specificity of 45.92% and ResNet-50 yielded a sensitivity of 72.39% with a specificity of 67.42%.

CONCLUSION

This study shows that DNN can extract discriminative features of SWAN for the classification of DAVF from normal with good accuracy, reasonably good sensitivity and specificity.

Assessment of Claimant, Clinical, and Financial Characteristics of Teleradiology Medical Malpractice Cases

Background The increasing use of teleradiology has been accompanied by concerns relating to risk management and patient safety. Purpose To compare characteristics of teleradiology and nonteleradiology radiology malpractice cases and identify contributing factors underlying these cases. Materials and Methods In this retrospective analysis, a national database of medical malpractice cases was queried to identify cases involving telemedicine that closed between January 2010 and March 2022. Teleradiology malpractice cases were identified based on manual review of cases in which telemedicine was coded as one of the contributing factors. These cases were compared with nonteleradiology cases that closed during the same time period in which radiology had been determined to be the primary responsible clinical service. Claimant, clinical, and financial characteristics of the cases were recorded, and continuous or categorical data were compared using the Wilcoxon rank-sum test or Fisher exact test, respectively. Results This study included 135 teleradiology and 3474 radiology malpractices cases. The death of a patient occurred more frequently in teleradiology cases (48 of 135 [35.6%]) than in radiology cases (685 of 3474 [19.7%]; P < .001). Cerebrovascular disease was a more common final diagnosis in the teleradiology cases (13 of 135 [9.6%]) compared with the radiology cases (124 of 3474 [3.6%]; P = .002). Problems with communication among providers was a more frequent contributing factor in the teleradiology cases (35 of 135 [25.9%]) than in the radiology cases (439 of 3474 [12.6%]; P < .001). Teleradiology cases were more likely to close with indemnity payment (79 of 135 [58.5%]) than the radiology cases (1416 of 3474 [40.8%]; P < .001) and had a higher median indemnity payment than the radiology cases ($339 230 [IQR, $120 790-$731 615] vs $214 063 [IQR, $66 620-$585 424]; P = .01). Conclusion Compared with radiology cases, teleradiology cases had higher clinical and financial severity and were more likely to involve issues with communication. © RSNA, 2024 See also the editorial by Mezrich in this issue.

Enhancing Radiology Education With a Case-Based Intro to Radiology on the UF WIDI e-Learning Platform

RATIONALE AND OBJECTIVES

This study explores the implementation and efficacy of an online, interactive, case-based radiology education tool, Wisdom in Diagnostic Imaging (WIDI) Case-Based Intro to Radiology (CBIR). We hypothesize that the WIDI CBIR platform would enhance radiology teaching, foster critical thinking, and provide a comprehensive curriculum in imaging interpretation and utilization.

MATERIALS AND METHODS

A focus group consisting of 1 undergraduate, 7 medical students, 9 physician assistant students, and 3 PhD students participated in this study. We tested 3 different teaching methods: a didactic approach without WIDI, a proctored didactic approach using WIDI, and a flipped classroom approach using WIDI. An online survey was conducted to assess student preference and feedback on these methods and the use of WIDI in their curriculum.

RESULTS

Most students preferred the proctored didactic approach with WIDI. They reported that the platform complemented their curriculum and encouraged critical thinking. The modules covered adequate clinical and imaging details and enhanced their skills in imaging interpretation. Despite the limitations of a small sample size and reliance on self-reported outcomes, this study indicates that the WIDI platform could be integrated into PA and medical school curricula throughout the US, offering a standardized radiology curriculum.

CONCLUSION

The UF WIDI appears to be a promising tool for modernizing radiology education, improving imaging interpretation skills, and enhancing appropriate imaging selection among nonradiologist medical learners. WIDI offers case-based education in imaging use, workflow, search-pattern selection, and interpretation of common radiological findings, potentially bridging the gap in radiology education. Further research and larger studies are required to assess the long-term impact on performance and clinical practice.

Improving traumatic fracture detection on radiographs with artificial intelligence support: a multi-reader study

Objectives

The aim of this study was to evaluate the diagnostic performance of nonspecialist readers with and without the use of an artificial intelligence (AI) support tool to detect traumatic fractures on radiographs of the appendicular skeleton.

Methods

The design was a retrospective, fully crossed multi-reader, multi-case study on a balanced dataset of patients (≥2 years of age) with an AI tool as a diagnostic intervention. Fifteen readers assessed 340 radiographic exams, with and without the AI tool in 2 different sessions and the time spent was automatically recorded. Reference standard was established by 3 consultant radiologists. Sensitivity, specificity, and false positives per patient were calculated.

Results

Patient-wise sensitivity increased from 72% to 80% (P < .05) and patient-wise specificity increased from 81% to 85% (P < .05) in exams aided by the AI tool compared to the unaided exams. The increase in sensitivity resulted in a relative reduction of missed fractures of 29%. The average rate of false positives per patient decreased from 0.16 to 0.14, corresponding to a relative reduction of 21%. There was no significant difference in average reading time spent per exam. The largest gain in fracture detection performance, with AI support, across all readers, was on nonobvious fractures with a significant increase in sensitivity of 11 percentage points (pp) (60%-71%).

Conclusions

The diagnostic performance for detection of traumatic fractures on radiographs of the appendicular skeleton improved among nonspecialist readers tested AI fracture detection support tool showed an overall reader improvement in sensitivity and specificity when supported by an AI tool. Improvement was seen in both sensitivity and specificity without negatively affecting the interpretation time.

Advances in knowledge

The division and analysis of obvious and nonobvious fractures are novel in AI reader comparison studies like this.

A clinically applicable AI system for diagnosis of congenital heart diseases based on computed tomography images

Congenital heart disease (CHD) is the most common type of birth defect. Without timely detection and treatment, approximately one-third of children with CHD would die in the infant period. However, due to the complicated heart structures, early diagnosis of CHD and its types is quite challenging, even for experienced radiologists. Here, we present an artificial intelligence (AI) system that achieves a comparable performance of human experts in the critical task of classifying 17 categories of CHD types. We collected the first-large CT dataset from three different CT machines, including more than 3750 CHD patients over 14 years. Experimental results demonstrate that it can achieve diagnosis accuracy (86.03%) comparable with junior cardiovascular radiologists (86.27%) in a World Health Organization appointed research and cooperation center in China on most types of CHD, and obtains a higher sensitivity (82.91%) than junior cardiovascular radiologists (76.18%). The accuracy of the combination of our AI system (97.20%) and senior radiologists achieves comparable results to that of junior radiologists and senior radiologists (97.16%) which is the current clinical routine. Our AI system can further provide 3D visualization of hearts to senior radiologists for interpretation and flexible review, surgeons for precise intuition of heart structures, and clinicians for more precise outcome prediction. We demonstrate the potential of our model to be integrated into current clinic practice to improve the diagnosis of CHD globally, especially in regions where experienced radiologists can be scarce.

Abdominopelvic CT in COVID-19 patients with abdominal complaints including both waves and controls: reader agreement and overcalls after consensus review

BACKGROUND

Since many COVID-19 publications lack consensus reviews or controls, interpretive accuracy is unclear; abdominal processes unique or infrequent during the pandemic remain unknown. The incidence and nature of CT findings accounting for abdominal complaints in COVID patients, reader agreement and overcalling will be determined.

METHODS

A retrospective study was performed on COVID patients with abdominal complaints from 3/15/2020-5/31/2020 and 11/1/2020-4/15/2021 including matched controls. Reviewers blinded to initial reads interpreted abdominopelvic CT exams, with discordant cases resolved in consensus. Reader agreement was measured by Cohen's Kappa, differences between cohorts by permutation tests and factors affecting false positive/negative rates by Fisher's Exact Test and logistic regression.

RESULTS

116 first wave (average age 65 years [±15.3], 63 [54%] women) and 194 second wave COVID cases (average age 64 years [±16.3], 103 [53%] women) including 116 wave 1 and 194 wave 2 prepandemic controls were included. Concordance was lower among COVID cases than controls (Cohen's Kappa of 0.58 vs. 0.82 [p ≤ 0.001]) and among wave 1 than wave 2 cases (Cohen's Kappa of 0.45 vs. 0.66 [p = 0.052]). With true positives defined as consensus between the initial reader and study reader, false positive rates were higher among COVID cases than controls (OR = 0.42, p = 0.003) and for initial than study reader (OR = 0.36, p ≤ 0.001), but lower in wave 2 than 1 (OR = 0.5, p = 0.028).

CONCLUSION

Greater reader disagreement occurred during COVID than prepandemic with no reader bias as both initial and study readers called more false positives among COVID cases than controls. More overcalling occurred during COVID with colitis and cystitis most common.

Discrepancies between Radiology Specialists and Residents in Fracture Detection from Musculoskeletal Radiographs

(1) Background: The aim of this study was to compare the competence in appendicular trauma radiograph image interpretation between radiology specialists and residents. (2) Methods: In this multicenter retrospective cohort study, we collected radiology reports from radiology specialists (N = 506) and residents (N = 500) during 2018-2021. As a reference standard, we used the consensus of two subspecialty-level musculoskeletal (MSK) radiologists, who reviewed all original reports. (3) Results: A total of 1006 radiograph reports were reviewed by the two subspecialty-level MSK radiologists. Out of the 1006 radiographs, 41% were abnormal. In total, 67 radiographic findings were missed (6.7%) and 31 findings were overcalled (3.1%) in the original reports. Sensitivity, specificity, positive predictive value, and negative predictive value were 0.86, 0.92, 0.91 and 0.88 respectively. There were no statistically significant differences between radiology specialists' and residents' competence in interpretation (p = 0.44). However, radiology specialists reported more subtle cases than residents did (p = 0.04). There were no statistically significant differences between errors made in the morning, evening, or night shifts (p = 0.57). (4) Conclusions: This study found a lack of major discrepancies between radiology specialists and residents in radiograph interpretation, although there were differences between MSK regions and in subtle or obvious radiographic findings. In addition, missed findings found in this study often affected patient treatment. Finally, there are MSK regions where the sensitivity or specificity is below 90%, and these should raise concerns and highlight the need for double reading and should be taken into consideration in radiology education.

Can we predict discordant RECIST 1.1 evaluations in double read clinical trials?

Background

In lung clinical trials with imaging, blinded independent central review with double reads is recommended to reduce evaluation bias and the Response Evaluation Criteria In Solid Tumor (RECIST) is still widely used. We retrospectively analyzed the inter-reader discrepancies rate over time, the risk factors for discrepancies related to baseline evaluations, and the potential of machine learning to predict inter-reader discrepancies.

Materials and methods

We retrospectively analyzed five BICR clinical trials for patients on immunotherapy or targeted therapy for lung cancer. Double reads of 1724 patients involving 17 radiologists were performed using RECIST 1.1. We evaluated the rate of discrepancies over time according to four endpoints: progressive disease declared (PDD), date of progressive disease (DOPD), best overall response (BOR), and date of the first response (DOFR). Risk factors associated with discrepancies were analyzed, two predictive models were evaluated.

Results

At the end of trials, the discrepancy rates between trials were not different. On average, the discrepancy rates were 21.0%, 41.0%, 28.8%, and 48.8% for PDD, DOPD, BOR, and DOFR, respectively. Over time, the discrepancy rate was higher for DOFR than DOPD, and the rates increased as the trial progressed, even after accrual was completed. It was rare for readers to not find any disease, for less than 7% of patients, at least one reader selected non-measurable disease only (NTL). Often the readers selected some of their target lesions (TLs) and NTLs in different organs, with ranges of 36.0-57.9% and 60.5-73.5% of patients, respectively. Rarely (4-8.1%) two readers selected all their TLs in different locations. Significant risk factors were different depending on the endpoint and the trial being considered. Prediction had a poor performance but the positive predictive value was higher than 80%. The best classification was obtained with BOR.

Conclusion

Predicting discordance rates necessitates having knowledge of patient accrual, patient survival, and the probability of discordances over time. In lung cancer trials, although risk factors for inter-reader discrepancies are known, they are weakly significant, the ability to predict discrepancies from baseline data is limited. To boost prediction accuracy, it would be necessary to enhance baseline-derived features or create new ones, considering other risk factors and looking into optimal reader associations.

Generative Artificial Intelligence for Chest Radiograph Interpretation in the Emergency Department

Importance

Multimodal generative artificial intelligence (AI) methodologies have the potential to optimize emergency department care by producing draft radiology reports from input images.

Objective

To evaluate the accuracy and quality of AI-generated chest radiograph interpretations in the emergency department setting.

Design, Setting, and Participants

This was a retrospective diagnostic study of 500 randomly sampled emergency department encounters at a tertiary care institution including chest radiographs interpreted by both a teleradiology service and on-site attending radiologist from January 2022 to January 2023. An AI interpretation was generated for each radiograph. The 3 radiograph interpretations were each rated in duplicate by 6 emergency department physicians using a 5-point Likert scale.

Main Outcomes and Measures

The primary outcome was any difference in Likert scores between radiologist, AI, and teleradiology reports, using a cumulative link mixed model. Secondary analyses compared the probability of each report type containing no clinically significant discrepancy with further stratification by finding presence, using a logistic mixed-effects model. Physician comments on discrepancies were recorded.

Results

A total of 500 ED studies were included from 500 unique patients with a mean (SD) age of 53.3 (21.6) years; 282 patients (56.4%) were female. There was a significant association of report type with ratings, with post hoc tests revealing significantly greater scores for AI (mean [SE] score, 3.22 [0.34]; P < .001) and radiologist (mean [SE] score, 3.34 [0.34]; P < .001) reports compared with teleradiology (mean [SE] score, 2.74 [0.34]) reports. AI and radiologist reports were not significantly different. On secondary analysis, there was no difference in the probability of no clinically significant discrepancy between the 3 report types. Further stratification of reports by presence of cardiomegaly, pulmonary edema, pleural effusion, infiltrate, pneumothorax, and support devices also yielded no difference in the probability of containing no clinically significant discrepancy between the report types.

Conclusions and Relevance

In a representative sample of emergency department chest radiographs, results suggest that the generative AI model produced reports of similar clinical accuracy and textual quality to radiologist reports while providing higher textual quality than teleradiologist reports. Implementation of the model in the clinical workflow could enable timely alerts to life-threatening pathology while aiding imaging interpretation and documentation.

Radiologist age and diagnostic errors

PURPOSE

Previous investigations into the causes of error by radiologists have addressed work schedule, volume, shift length, and sub-specialization. Studies regarding possible associations between radiologist errors and radiologist age and timing of residency training are lacking in the literature, to our knowledge. The aim of our study was to determine if radiologist age and residency graduation date is associated with diagnostic errors.

METHODS

Our retrospective analysis included 1.9 million preliminary interpretations (out of a total of 5.2 million preliminary and final interpretations) of imaging examinations by 361 radiologists in a US-based national teleradiology practice between 1/1/2019 and 1/1/2020. Quality assurance data regarding the number of radiologist errors was generated through client facility feedback to the teleradiology practice. With input from both the client radiologist and the teleradiologist, the final determination of the presence, absence, and severity of a teleradiologist error was determined by the quality assurance committee of radiologists within the teleradiology company using standardized criteria. Excluded were 3.2 million final examination interpretations and 93,963 (1.8%) of total examinations from facilities reporting less than one discrepancy in examination interpretation in 2019. Logistic regression with covariates radiologist age and residency graduation date was performed for calculation of relative risk of overall error rates and by major imaging modality. Major errors were separated from minor errors as those with a greater likelihood of affecting patient care. Logistic regression with covariates radiologist age, residency graduation date, and log total examinations interpreted was used to calculate odds of making a major error to that of making a minor error.

RESULTS

Mean age of the 361 radiologists was 51.1 years, with a mean residency graduation date of 2001. Mean error rate for all examinations was 0.5%. Radiologist age at any residency graduation date was positively associated with major errors (p < 0.05), with a relative risk 1.021 for each 1-year increase in age and relative risk 1.235 for each decade as well as for minor errors (p < 0.05, relative risk 1.007 for each year, relative risk 1.082 for each decade). By major imaging modality, radiologist age at any residency graduation date was positively associated with computed tomography (CT) and X-ray (XR) major and minor error, magnetic resonance imaging (MRI) major error, and ultrasound (US) minor error (p < 0.05). Radiologist age was positively associated with odds of making a major vs. minor error (p < 0.05).

CONCLUSIONS

The mean error rate for all radiologists was low. We observed that increasing age at any residency graduation date was associated with increasing relative risk of major and minor errors as well as increasing odds of a major vs. minor error among providers. Further study is needed to corroborate these results, determine clinical relevance, and highlight strategies to address these findings.

Why don't we inform patients about the risk of diagnostic errors?

The principles of autonomy and informed consent dictate that patients who undergo a radiological examination should actually be informed about the risk of diagnostic errors. Implementing such a policy could potentially increase the quality of care. However, due to the vast number of radiological examinations that are performed in each hospital each day, financial constraints, and the risk of losing trust, patients, and income if the requirement for informed consent is not imposed by law on a state or national level, it may be challenging to inform patients about the risk of diagnostic errors. Future research is necessary to determine if and how an informed consent procedure for diagnostic errors can be implemented in clinical practice.

Medical imaging and multimodal artificial intelligence models for streamlining and enhancing cancer care: opportunities and challenges

INTRODUCTION

Artificial intelligence (AI) has the potential to transform oncologic care. There have been significant developments in AI applications in medical imaging and increasing interest in multimodal models. These are likely to enable improved oncologic care through more precise diagnosis, increasingly in a more personalized and less invasive manner. In this review, we provide an overview of the current state and challenges that clinicians, administrative personnel and policy makers need to be aware of and mitigate for the technology to reach its full potential.

AREAS COVERED

The article provides a brief targeted overview of AI, a high-level review of the current state and future potential AI applications in diagnostic radiology and to a lesser extent digital pathology, focusing on oncologic applications. This is followed by a discussion of emerging approaches, including multimodal models. The article concludes with a discussion of technical, regulatory challenges and infrastructure needs for AI to realize its full potential.

EXPERT OPINION

There is a large volume of promising research, and steadily increasing commercially available tools using AI. For the most advanced and promising precision diagnostic applications of AI to be used clinically, robust and comprehensive quality monitoring systems and informatics platforms will likely be required.

Mammography Datasets for Neural Networks-Survey

Deep neural networks have gained popularity in the field of mammography. Data play an integral role in training these models, as training algorithms requires a large amount of data to capture the general relationship between the model's input and output. Open-access databases are the most accessible source of mammography data for training neural networks. Our work focuses on conducting a comprehensive survey of mammography databases that contain images with defined abnormal areas of interest. The survey includes databases such as INbreast, the Curated Breast Imaging Subset of Digital Database for Screening Mammography (CBIS-DDSM), the OPTIMAM Medical Image Database (OMI-DB), and The Mammographic Image Analysis Society Digital Mammogram Database (MIAS). Additionally, we surveyed recent studies that have utilized these databases in conjunction with neural networks and the results they have achieved. From these databases, it is possible to obtain at least 3801 unique images with 4125 described findings from approximately 1842 patients. The number of patients with important findings can be increased to approximately 14,474, depending on the type of agreement with the OPTIMAM team. Furthermore, we provide a description of the annotation process for mammography images to enhance the understanding of the information gained from these datasets.

Measuring appropriateness of diagnostic imaging: a scoping review

In radiology, the justification of diagnostic imaging is a key performance indicator. To date, specific recommendations on the measurement of appropriateness in diagnostic imaging are missing. To map the study literature concerning the definition, measures, methods and data used for analyses of appropriateness in research of diagnostic imaging. We conducted a scoping review in Medline, EMBASE, Scopus and the Cochrane Central Register of Controlled Trials. Two independent reviewers undertook screening and data extraction. After screening 6021 records, we included 50 studies. National guidelines (n = 22/50) or American College of Radiology Appropriateness Criteria (n = 23/50) were used to define and rate appropriateness. 22/50 studies did not provide methodological details about the appropriateness assessment. The included studies varied concerning modality, amount of reviewed examinations (88-13,941) and body regions. Computed tomography (27 studies, 27,168 examinations) was the most frequently analyzed modality, followed by magnetic resonance imaging (17 studies, 6559 examinations) and radiography (10 studies, 7095 examinations). Heterogeneous appropriateness rates throughout single studies (0-100%), modalities, and body regions (17-95%) were found. Research on pediatric and outpatient imaging was sparse. Multicentric, methodologically robust and indication-oriented studies would strengthen appropriateness research in diagnostic imaging and help to develop reliable key performance indicators.

Breaking down the RECIST 1.1 double read variability in lung trials: What do baseline assessments tell us?

Background

In clinical trials with imaging, Blinded Independent Central Review (BICR) with double reads ensures data blinding and reduces bias in drug evaluations. As double reads can cause discrepancies, evaluations require close monitoring which substantially increases clinical trial costs. We sought to document the variability of double reads at baseline, and variabilities across individual readers and lung trials.

Material and methods

We retrospectively analyzed data from five BICR clinical trials evaluating 1720 lung cancer patients treated with immunotherapy or targeted therapy. Fifteen radiologists were involved. The variability was analyzed using a set of 71 features derived from tumor selection, measurements, and disease location. We selected a subset of readers that evaluated ≥50 patients in ≥two trials, to compare individual reader’s selections. Finally, we evaluated inter-trial homogeneity using a subset of patients for whom both readers assessed the exact same disease locations. Significance level was 0.05. Multiple pair-wise comparisons of continuous variables and proportions were performed using one-way ANOVA and Marascuilo procedure, respectively.

Results

Across trials, on average per patient, target lesion (TL) number ranged 1.9 to 3.0, sum of tumor diameter (SOD) 57.1 to 91.9 mm. MeanSOD=83.7 mm. In four trials, MeanSOD of double reads was significantly different. Less than 10% of patients had TLs selected in completely different organs and 43.5% had at least one selected in different organs. Discrepancies in disease locations happened mainly in lymph nodes (20.1%) and bones (12.2%). Discrepancies in measurable disease happened mainly in lung (19.6%). Between individual readers, the MeanSOD and disease selection were significantly different (p&lt;0.001). In inter-trials comparisons, on average per patient, the number of selected TLs ranged 2.1 to 2.8, MeanSOD 61.0 to 92.4 mm. Trials were significantly different in MeanSOD (p&lt;0.0001) and average number of selected TLs (p=0.007). The proportion of patients having one of the top diseases was significantly different only between two trials for lung. Significant differences were observed for all other disease locations (p&lt;0.05).

Conclusions

We found significant double read variabilities at baseline, evidence of reading patterns and a means to compare trials. Clinical trial reliability is influenced by the interplay of readers, patients and trial design.

Utility of dual read in the setting of prostate MRI interpretation

PURPOSE

The purpose of this study is to assess the utility of dual reader interpretation of prostate MRI in the evaluation/detection of prostate cancer, using the PI-RADS v2.1 scoring system.

METHODS

We performed a retrospective study to assess the utility of dual reader interpretation for prostate MRI. All MRI cases compiled for analysis were accompanied with prostate biopsy pathology reports that included Gleason scores to correlate to the MRI PI-RADS v2.1 score, tissue findings and location of pathology within the prostate gland. To assess for dual reader utility, two fellowship trained abdominal imagers (each with > 5 years of experience) provided independent and concurrent PI-RADS v2.1 scores on all included MRI examinations, which were then compared to the biopsy proven Gleason scores.

RESULTS

After application of inclusion criteria, 131 cases were used for analysis. The mean age of the cohort was 63.6 years. Sensitivity, specificity and positive/negative predictive values were calculated for each reader and concurrent scores. Reader 1 demonstrated 71.43% sensitivity, 85.39% specificity, 69.77% PPV and 86.36% NPV. Reader 2 demonstrated 83.33% sensitivity, 78.65% specificity, 64.81% PPV and 90.91% NPV. Concurrent reads demonstrated 78.57% sensitivity, 80.9% specificity, 66% PPV and 88.89% NPV. There was no statistically significant difference between the individual readers or concurrent reads (p = 0.79).

CONCLUSION

Our results highlight that dual reader interpretation in prostate MRI is not needed to detect clinically relevant tumor and that radiologists with experience and training in prostate MRI interpretation establish acceptable sensitivity and specificity marks on PI-RADS v2.1 assessment.

Frequency and Clinical Utility of Alerts for Intra-Institutional Radiologist Discrepant Opinions

OBJECTIVE

Determine the rate of documented notification, via an alert, for intra-institutional discrepant radiologist opinions and addended reports and resulting clinical management changes.

METHODS

This institutional review board-exempt, retrospective study was performed at a large academic medical center. We defined an intra-institutional discrepant opinion as when a consultant radiologist provides a different interpretation from that formally rendered by a colleague at our institution. We implemented a discrepant opinion policy requiring closed-loop notification of the consulting radiologist's second opinion to the original radiologist, who must acknowledge this alert within 30 days. This study included all discrepant opinion alerts created December 1, 2019, to December 31, 2021, of which two radiologists and an internal medicine physician performed consensus review. Primary outcomes were degree of discrepancy and percent of discrepant opinions leading to change in clinical management. Secondary outcome was report addendum rate compared with an existing peer learning program using Fisher's exact test.

RESULTS

Of 114 discrepant opinion alerts among 1,888,147 reports generated during the study period (0.006%), 58 alerts were categorized as major (50.9%), 41 as moderate (36.0%), and 15 as minor discrepancies (13.1%). Clinical management change occurred in 64 of 114 cases (56.1%). Report addendum rate for discrepant opinion alerts was 4-fold higher than for peer learning alerts at our institution (66 of 315 = 21% versus 432 of 8,273 =5.2%; P < .0001).

DISCUSSION

Although discrepant intra-institutional radiologist second opinions were rare, they frequently led to changes in clinical management. Capturing these discrepancies by encouraging alert use may help optimize patient care and document what was communicated to the referring or consulting care team by consulting radiologists.

Non-task expert physicians benefit from correct explainable AI advice when reviewing X-rays

Artificial intelligence (AI)-generated clinical advice is becoming more prevalent in healthcare. However, the impact of AI-generated advice on physicians' decision-making is underexplored. In this study, physicians received X-rays with correct diagnostic advice and were asked to make a diagnosis, rate the advice's quality, and judge their own confidence. We manipulated whether the advice came with or without a visual annotation on the X-rays, and whether it was labeled as coming from an AI or a human radiologist. Overall, receiving annotated advice from an AI resulted in the highest diagnostic accuracy. Physicians rated the quality of AI advice higher than human advice. We did not find a strong effect of either manipulation on participants' confidence. The magnitude of the effects varied between task experts and non-task experts, with the latter benefiting considerably from correct explainable AI advice. These findings raise important considerations for the deployment of diagnostic advice in healthcare.

Transfer Patient Imaging: Assessment of the Impact of Discrepancies Identified by Emergency Radiologists

PURPOSE

Advanced imaging examinations of emergently transferred patients (ETPs) are overread to various degrees by receiving institutions. The practical clinical impact of these second opinions has not been studied in the past. The purpose of this study is to determine if emergency radiology overreads change emergency medicine decision making on ETPs in the emergency department (ED).

METHODS

All CT and MRI examinations on patients transferred to a level I trauma center during calendar year 2018 were routinely overread by emergency radiologists and discrepancies with the outside report electronically flagged. All discrepant reports compared with the outside interpretations were reviewed by one of four emergency medicine physicians. Comparing the original and final reports, reviewers identified changes in patient management that could be attributed to the additional information contained in the final report. Changes in patient care were categorized as affecting ED management, disposition, follow-up, or consulting services.

RESULTS

Over a 12-month period, 5,834 patients were accepted in transfer. Among 5,631 CT or MRI examinations with outside reports available, 669 examinations (12%) had at least one discrepancy in the corresponding outside report. In 219 examinations (33%), ED management was changed by discrepancies noted on the final report; patient disposition was affected in 84 (13%), outpatient follow-up in 54 (8%), and selection of consulting services in 411 (61%), and ED stay was extended in 544 (81%). Discrepant findings affected decision making in 613 of 669 of examinations (92%).

CONCLUSION

Emergency radiology overreading of transferred patients' advanced imaging examinations provided actionable additional information to emergency medicine physicians in the care of 613 of 669 (92%) examinations with discrepant findings. This added value is worth the effort to design workflows to routinely overread CT and MRI examinations of ETPs.

Paediatric gonad shielding in pelvic radiography: A systematic review and meta-analysis

INTRODUCTION

The British Institute of Radiology (BIR) and American Association of Physicists in Medicine (AAPM) have recommended that gonad shielding is no longer used during pelvic X-ray examinations. The BIR guidance states that shielding may still be considered for use on males, but should not be used on females. This paper aimed to evaluate if this decision was supported by evidence from practice, by comparing the accuracy of gonad shield placement in paediatric males and females.

METHODS

A systematic review of databases including EMBASE, MEDLINE and PubMed was performed in February 2021. Studies were considered eligible if they provided data on the use of gonad shielding during pelvic X-ray examinations on male and female patients under the age of 18. Nine studies met the inclusion criteria and data extraction was performed. Quality appraisal was undertaken, and a meta-analysis of shielding accuracy was performed on seven studies.

RESULTS

The results from the meta-analysis (2187 total radiographs) demonstrated that female patients were significantly more likely (OR 1.38, 95% CI 0.88-1.87) than males to have gonad shields placed inaccurately (p value < 0.001).

CONCLUSION

Gonad shield placement on paediatric female patients is significantly less accurate than on males, and so the results support the AAPM and BIR guidance to stop the practice for females. Shield application may also be frequently inaccurate for males, but the review does not provide clear evidence for or against continuing the practice for males.

IMPLICATIONS FOR PRACTICE

Discontinuing the use of gonad shields in paediatric pelvic radiography on female patients is supported. Any continued use on male patients, or for reasons such as psychological reassurance, should be subject to enhanced training and audit to ensure benefits outweigh any risks.

Real-world staging computed tomography scanning technique and important reporting discrepancies in pancreatic ductal adenocarcinoma

BACKGROUND

Computed tomography (CT) is the first-line staging imaging modality for pancreatic ductal adenocarcinoma (PDAC) which determines resectability and treatment pathways.

METHODS

Between January 2016 and December 2019, prospectively collated data from two Australian cancer centres was extracted from the PURPLE Pancreatic Cancer registry. Real-world staging CTs and corresponding reports were blindly reviewed by a sub-specialist radiologist and compared to initial reports.

RESULTS

Of 131 patients assessed, 117 (89.3%) presented with symptoms, 74 (56.5%) CTs included slices ≤3 mm thickness and CT pancreas protocol was applied in 69 (52.7%) patients. Initial reports lacked synoptic reporting in 131 (100%), tumour identification in 2 (1.6%) and tumour measurement in 13 (9.9%) cases. Tumour-vascular relationship reporting was missing in 69-109 (52.7-83.2%) for regarding the key arterial and venous structures that is required to assess resectability. Initial reports had no comment on venous thrombus or venous collaterals in 80 (61.1%) and 109 (83.2%) and lacked locoregional lymphadenopathy interpretation in 13 (9.9%) cases. Complete initial staging report was present in 72 (55.0%) patients. Sub-specialist radiological review resulted in down-staging in 16 (22.2%) and up-staging in 1 (1.4%) patient. Staging discrepancies were mainly regarding metastatic disease (12, 70.6%) and tumour-vascular relationship (5, 29.4%).

CONCLUSION

Real-world staging imaging in PDAC patients show low proportion of dedicated CT pancreas protocol, high proportion of incomplete staging reports and no synoptic reporting. The most common discrepancy between initial and sub-specialist reporting was regarding metastases and tumour-vascular relationship assessment resulting in sub-specialist down-staging in almost every fifth case.

Mandating Limits on Workload, Duty, and Speed in Radiology

Research has not yet quantified the effects of workload or duty hours on the accuracy of radiologists. With the exception of a brief reduction in imaging studies during the 2020 peak of the COVID-19 pandemic, the workload of radiologists in the United States has seen relentless growth in recent years. One concern is that this increased demand could lead to reduced accuracy. Behavioral studies in species ranging from insects to humans have shown that decision speed is inversely correlated to decision accuracy. A potential solution is to institute workload and duty limits to optimize radiologist performance and patient safety. The concern, however, is that any prescribed mandated limits would be arbitrary and thus no more advantageous than allowing radiologists to self-regulate. Specific studies have been proposed to determine whether limits reduce error, and if so, to provide a principled basis for such limits. This could determine the precise susceptibility of individual radiologists to medical error as a function of speed during image viewing, the maximum number of studies that could be read during a work shift, and the appropriate shift duration as a function of time of day. Before principled recommendations for restrictions are made, however, it is important to understand how radiologists function both optimally and at the margins of adequate performance. This study examines the relationship between interpretation speed and error rates in radiology, the potential influence of artificial intelligence on reading speed and error rates, and the possible outcomes of imposed limits on both caseload and duty hours. This review concludes that the scientific evidence needed to make meaningful rules is lacking and notes that regulating workloads without scientific principles can be more harmful than not regulating at all.

A narrative review of deep learning applications in lung cancer research: from screening to prognostication

Background and Objective

Deep learning (DL) algorithms have been developed for various tasks, including lung nodule detection on chest radiographs or lung cancer computed tomography screening, potential candidate selection in lung cancer screening, malignancy prediction for indeterminate pulmonary nodules, lung cancer staging, treatment response prediction, prognostication, and prediction of genetic mutations in lung cancer. Furthermore, these DL algorithms have been applied in various clinical settings in order for them to be generalized in real-world clinical practice. Multiple DL algorithms have been corroborated to be on par with experts or current clinical prediction models for several specific tasks. However, no article has yet comprehensively reviewed DL algorithms dedicated to lung cancer research. This narrative review presents an overview of the literature dealing with DL techniques applied in lung cancer research and briefly summarizes the results according to the DL algorithms’ clinical use cases.

Methods

we performed a narrative review by searching the Embase and OVID-MEDLINE databases for articles published in English from October, 2016 until September, 2021 and reviewing the bibliographies of key references to identify important literature related to DL in lung cancer research. The background, development, results, and clinical implications of each DL algorithm are briefly discussed. Lastly, we end this review article by highlighting future directions in lung cancer research using DL techniques.

Key Content and Findings

DL algorithms have been introduced to show comparable or higher performance than human experts in various clinical settings. Specifically, they have been actively applied to detect lung nodules in chest radiographs or computed tomography (CT) examinations, optimize candidate selection for lung cancer screening (LCS), predict the malignancy of lung nodules, stage lung cancer, and predict treatment response, patients’ prognoses, and genetic mutations in lung cancers.

Conclusions

DL algorithms have corroborated their potential value for various tasks, ranging from lung cancer screening to prognostication of lung cancer patients. Future research is warranted for the clinical application of these algorithms in daily clinical practice and verification of their real-world clinical usefulness.

Diagnostic Accuracy and Performance of Artificial Intelligence in Detecting Lung Nodules in Patients With Complex Lung Disease: A Noninferiority Study

OBJECTIVES

The aim of the study is to investigate the performance of artificial intelligence (AI) convolutional neural networks (CNN) in detecting lung nodules on chest computed tomography of patients with complex lung disease, and demonstrate its noninferiority when compared against an experienced radiologist through clinically relevant assessments.

METHODS

A CNN prototype was used to retrospectively evaluate 103 complex lung disease cases and 40 control cases without reported nodules. Computed tomography scans were blindly evaluated by an expert thoracic radiologist; a month after initial analyses, 20 positive cases were re-evaluated with the assistance of AI. For clinically relevant applications: (1) AI was asked to classify each patient into nodules present or absent and (2) AI results were compared against standard radiology reports. Standard statistics were performed to determine detection performance.

RESULTS

AI was, on average, 27 seconds faster than the expert and detected 8.4% of nodules that would have been missed. AI had a sensitivity of 67.7%, similar to an accuracy reported for experienced radiologists. AI correctly classified each patient (nodules present/absent) with a sensitivity of 96.1%. When matched against radiology reports, AI performed with a sensitivity of 89.4%. Control group assessment demonstrated an overall specificity of 82.5%. When aided by AI, the expert decreased the average assessment time per case from 2:44 minutes to 35.7 seconds, while reporting an overall increase in confidence.

CONCLUSION

In a group of patients with complex lung disease, the sensitivity of AI is similar to an experienced radiologist and the tool helps detect previously missed nodules. AI also helps experts analyze for lung nodules faster and more confidently, a feature that is beneficial to patients and favorable to hospitals due to increased patient load and need for shorter turnaround times.

Clinical Value of Surveillance 18F-fluorodeoxyglucose PET/CT for Detecting Unsuspected Recurrence or Second Primary Cancer in Non-Small Cell Lung Cancer after Curative Therapy

Simple Summary

Non-small cell lung cancer (NSCLC) patients are at considerable risk of recurrence or second primary cancer (SPC) after curative therapy. The utility of ¹⁸F-fluorodeoxyglucose (FDG) positron emission tomography/computed tomography (PET/CT) surveillance to detect recurrent lesions in NSCLC patients without suspicion of recurrence has not been established. The aim of our retrospective study was to evaluate the diagnostic value of surveillance FDG PET/CT for detecting clinically unsuspected recurrence or SPC in patients with NSCLC after curative therapy. In a cohort of 2684 NSCLC patients after curative therapy, surveillance FDG PET/CT showed good diagnostic efficacy for detecting clinically unexpected recurrence or SPC. Furthermore, the diagnostic performance was improved in subgroups of patients with advanced stage prior to curative therapy, PET/CT scans performed within 3 years after curative-intent therapy, and curative surgery. Surveillance PET/CT can be more useful when performed soon after therapy in curative surgery recipients and those with an advanced disease stage considering its diagnostic efficacy and yield.

Abstract

We evaluated the diagnostic value of ¹⁸F-fluorodeoxyglucose (FDG) positron emission tomography (PET)/CT surveillance for detecting clinically unsuspected recurrence or second primary cancer (SPC) in patients with non-small cell lung cancer (NSCLC) after curative therapy. A total of 4478 surveillance FDG PET/CT scans from 2864 NSCLC patients without suspicion of recurrence after curative therapy were reviewed retrospectively. In 274 of 2864 (9.6%) patients, recurrent NSCLC or SPC was found by surveillance PET/CT during clinical follow-up. Surveillance PET/CT scans showed sensitivity of 98.9% (274/277), specificity of 98.1% (4122/4201), accuracy of 98.2% (4396/4478), positive predictive value (PPV) of 77.6% (274/353), and negative predictive value of 99.9% (4122/4125). The specificity and accuracy in the curative surgery group were significantly higher than those in the curative radiotherapy group. PPV was significantly improved in subgroups of patients with advanced stage prior to curative therapy, PET/CT scans performed within 3 years after curative-intent therapy, and curative surgery. FDG PET/CT surveillance showed good diagnostic efficacy for detecting clinically unexpected recurrence or SPC in NSCLC patients after curative therapy. It can be more useful when performed soon after therapy in curative surgery recipients and those with an advanced disease stage considering its diagnostic efficacy and yield.

Agreement on grading of normal clivus using magnetic resonance imaging among radiologists

Purpose

The present study was aimed to evaluate the agreement on grading normal clivus on MRI among radiologists.

Methods

A retrospective study included patients who underwent MRI brain during January 1, 2015 to October 31, 2019. Two hundred forty-four patients who had no marrow pathology on MRI were included and divided into 8 age groups by decades. Three radiologists independently reviewed the signal intensity of clivus in mid sagittal T1-weighted image. The signal intensity was classified into three grades (Grade I-III). Fleiss’ kappa coefficients (k) were calculated to assess interrater agreement.

Results

Of 244 patients, there were 123 (50.4%) males and 121 (49.6%) females. Age ranged from 1 to 79 years old. Clivus Grade II was more frequently reported (> 50%) by radiologists. The agreement (kappa) among all three radiologists on evaluation of clivus irrespective of the grading equals to 0.67 (95%CI: 0.60–0.74). In stratified analyses by the grade of clivus, the kappa values for Grade I to III and were 0.73, 0.62, and 0.69 respectively.

Conclusion

Interrater agreement of MRI evaluation of normal clivus among radiologists was good. The visual grading criteria to classify the clivus is sufficient to distinguish the marrow maturation. However, the consensus reading should be made whenever normal clivus Grade II is read.

Impact of double reading on NI-RADS diagnostic accuracy in reporting oral squamous cell carcinoma surveillance imaging - a single-center study

OBJECTIVES

The Neck Imaging Reporting and Data System (NI-RADS) is an increasingly utilized risk stratification tool for imaging surveillance after treatment for head and neck cancer. This study aims to measure the impact of supervision by subspecialized radiologists on diagnostic accuracy of NI-RADS when initial reading is performed by residents.

METHODS

150 CT and MRI datasets were initially read by two trained residents, and then supervised by two subspecialized radiologists. Recurrence rates by NI-RADS category were calculated, and receiver operating characteristic (ROC) curves were plotted. After dichotomization of the NI-RADS system (category 1 vs categories 2 + 3+4 and categories 1 + 2 vs 3 + 4), sensitivity, specificity, positive and negative predictive value were calculated.

RESULTS

26% of the reports were modified by the supervising radiologists. Area under the curve of ROC plots values of the supervision session were higher than those of the initial reading session for both the primary site (0.89 vs 0.86) and the neck (0.94 vs 0.91), but the difference was not statistically significant. For dichotomized NI-RADS category assignments, differences between the initial reading and the supervision session were statistically significant regarding specificity and PPV for the primary site (1 + 2 vs 3 + 4 and 1 vs 2 + 3+4) or even for both sites combined (1 vs 2 + 3+4).

CONCLUSION

NI-RADS enables trained resident radiologists to report surveillance imaging in patients with treated oral squamous cell carcinoma with high discriminatory power. Additional supervision by a subspecialized head and neck radiologist particularly improves specificity of radiological reports.

Applications of artificial intelligence in the thorax: a narrative review focusing on thoracic radiology

OBJECTIVE

This review will focus on how AI-and, specifically, deep learning-can be applied to complement aspects of the current healthcare system. We describe how AI-based tools can augment existing clinical workflows by discussing the applications of AI to worklist prioritization and patient triage, the performance-boosting effects of AI as a second reader, and the use of AI to facilitate complex quantifications. We also introduce prominent examples of recent AI applications, such as tuberculosis screening in resource-constrained environments, the detection of lung cancer with screening CT, and the diagnosis of COVID-19. We also provide examples of prognostic predictions and new discoveries beyond existing clinical practices.

BACKGROUND

Artificial intelligence (AI) has shown promising performance for thoracic diseases, particularly in the field of thoracic radiology. However, it has not yet been established how AI-based image analysis systems can help physicians in clinical practice.

METHODS

This review included peer-reviewed research articles on AI in the thorax published in English between 2015 and 2021.

CONCLUSIONS

With advances in technology and appropriate preparation of physicians, AI could address various clinical problems that have not been solved due to a lack of clinical resources or technological limitations.

KEYWORDS

Artificial intelligence (AI); deep learning (DL); computer aided diagnosis (CAD); thoracic radiology; pulmonary medicine.

Are We Overdoing It? Changes in Diagnostic Imaging Workload during the Years 2010-2020 including the Impact of the SARS-CoV-2 Pandemic

Since the 1990s, there has been a significant increase in the number of imaging examinations as well as a related increase in the healthcare expenditure and the exposure of the population to X-rays. This study aimed to analyze the workload trends in radiology during the last decade, including the impact of COVID-19 in a single university hospital in Poland and to identify possible solutions to the challenges that radiology could face in the future. We compared the annual amount of computed tomography (CT), radiography (X-ray), and ultrasound (US) examinations performed between the years 2010 and 2020 and analyzed the changes in the number of practicing radiologists in Poland. The mean number of patients treated in our hospital was 60,727 per year. During the last decade, the number of CT and US examinations nearly doubled (from 87.4 to 155.7 and from 52.1 to 86.5 per 1000 patients in 2010 and 2020 respectively), while X-ray examinations decreased from 115.1 to 96.9 per 1000 patients. The SARS-CoV-2 pandemic did not change the workload trends as more chest examinations were performed. AI, which contributed to the COVID-19 diagnosis, could aid radiologists in the future with the growing workload by increasing the efficiency of radiology departments as well as by potentially minimizing the related costs.

A deep learning algorithm for automatic detection and classification of acute intracranial hemorrhages in head CT scans

Acute Intracranial hemorrhage (ICH) is a life-threatening disease that requires emergency medical attention, which is routinely diagnosed using non-contrast head CT imaging. The diagnostic accuracy of acute ICH on CT varies greatly among radiologists due to the difficulty of interpreting subtle findings and the time pressure associated with the ever-increasing workload. The use of artificial intelligence technology may help automate the process and assist radiologists for more prompt and better decision-making. In this work, we design a deep learning approach that mimics the interpretation process of radiologists, and combines a 2D CNN model and two sequence models to achieve accurate acute ICH detection and subtype classification. Being developed using the extensive 2019-RSNA Brain CT Hemorrhage Challenge dataset with over 25000 CT scans, our deep learning algorithm can accurately classify the acute ICH and its five subtypes with AUCs of 0.988 (ICH), 0.984 (EDH), 0.992 (IPH), 0.996 (IVH), 0.985 (SAH), and 0.983 (SDH), respectively, reaching the accuracy level of expert radiologists. Our method won 1st place among 1345 teams from 75 countries in the RSNA challenge. We have further evaluated our algorithm on two independent external validation datasets with 75 and 491 CT scans, respectively, and our method maintained high AUCs of 0.964 and 0.949 for acute ICH detection. These results have demonstrated the high performance and robust generalization ability of our proposed method, which makes it a useful second-read or triage tool that can facilitate routine clinical applications.

Secondary Interpretations of Diagnostic Imaging Examinations: Patient Liabilities and Out-of-Pocket Costs

BACKGROUND

Secondary interpretations of diagnostic imaging examinations are increasingly performed to improve care for complex patients. We sought to determine associated patient-billed liabilities and out-of-pocket payments and to identify patient and imaging study characteristics that correlate with higher patient bills and out-of-pocket payments.

METHODS

Data extracted for 7,740 secondary imaging interpretations performed across our large metropolitan health system over 25 months included total professional charges, insurance payments, patient-billed liabilities, and patient out-of-pocket payments. Multivariable linear regression analyses were performed to identify patient and imaging factors associated with higher patient bills and out-of-pocket payments.

RESULTS

Mean secondary interpretation professional charges, insurance payments, patient-billed liabilities, and patient out-of-pocket payments were $306.50, $108.02, $27.80, and $14.55, respectively. Patients received bills for 47.5% of services and made out-of-pocket payments for 17.1%. Patient-billed liabilities and out-of-pocket payments were higher for patients who were younger and uninsured and for secondary interpretations requested for patients seen in outpatient (versus inpatient) settings. Patient-billed liabilities and out-of-pocket payments were lower for patients who were Black (versus White) and had government-sponsored (versus commercial) insurance and for secondary interpretations performed during the second, third, or fourth (versus first) quarter of each calendar year.

CONCLUSION

Observed differences between patient-billed liabilities and out-of-pocket payments suggest that secondary interpretations of diagnostic imaging examinations can result in small but real patient financial burdens. Improved price transparency and enhanced patient communication about the value of secondary interpretations could reduce potential surprises when patients receive these bills.

A simplified cluster model and a tool adapted for collaborative labeling of lung cancer CT scans

BACKGROUND AND OBJECTIVE

Lung cancer is the most common type of cancer with a high mortality rate. Early detection using medical imaging is critically important for the long-term survival of the patients. Computer-aided diagnosis (CAD) tools can potentially reduce the number of incorrect interpretations of medical image data by radiologists. Datasets with adequate sample size, annotation, and truth are the dominant factors in developing and training effective CAD algorithms. The objective of this study was to produce a practical approach and a tool for the creation of medical image datasets.

METHODS

The proposed model uses the modified maximum transverse diameter approach to mark a putative lung nodule. The modification involves the possibility to use a set of overlapping spheres of appropriate size to approximate the shape of the nodule. The algorithm embedded in the model also groups the marks made by different readers for the same lesion. We used the data of 536 randomly selected patients of Moscow outpatient clinics to create a dataset of standard-dose chest computed tomography (CT) scans utilizing the double-reading approach with arbitration. Six volunteer radiologists independently produced a report for each scan using the proposed model with the main focus on the detection of lesions with sizes ranging from 3 to 30 mm. After this, an arbitrator reviewed their marks and annotations.

RESULTS

The maximum transverse diameter approach outperformed the alternative methods (3D box, ellipsoid, and complete outline construction) in a study of 10,000 computer-generated tumor models of different shapes in terms of accuracy and speed of nodule shape approximation. The markup and annotation of the CTLungCa-500 dataset revealed 72 studies containing no lung nodules. The remaining 464 CT scans contained 3151 lesions marked by at least one radiologist: 56%, 14%, and 29% of the lesions were malignant, benign, and non-nodular, respectively. 2887 lesions have the target size of 3-30 mm. Only 70 nodules were uniformly identified by all the six readers. An increase in the number of independent readers providing CT scans interpretations led to an accuracy increase associated with a decrease in agreement. The dataset markup process took three working weeks.

CONCLUSIONS

The developed cluster model simplifies the collaborative and crowdsourced creation of image repositories and makes it time-efficient. Our proof-of-concept dataset provides a valuable source of annotated medical imaging data for training CAD algorithms aimed at early detection of lung nodules. The tool and the dataset are publicly available at https://github.com/Center-of-Diagnostics-and-Telemedicine/FAnTom.git and https://mosmed.ai/en/datasets/ct_lungcancer_500/, respectively.

Limited added value of Doppler ultrasound of the liver after recent contrast-enhanced computed tomography

PURPOSE

The aim of this study is to assess the added diagnostic value of Doppler ultrasound of the liver (DUL) performed within 3 days of contrast-enhanced CT (CECT) for the diagnosis of portal vein (PV) or hepatic vein (HV) thrombosis.

METHODS

Adult patients were included if they underwent DUL within three days after a CECT of the abdomen in the emergency or inpatient setting. Retrospective review of clinical data and imaging reports was performed. In patients with discrepant or positive findings on CECT and/or DUL with respect to PV or HV thrombosis, image review was performed by three fellowship-trained abdominal radiologists in consensus.

RESULTS

The final cohort consisted of 468 patients. Of these, 26 (5.6%) patients had equivocal findings for thrombosis on CECT, and DUL could make a confident diagnosis of positive or negative in 18 (69%) patients. Additionally, there were 2 (0.4%) patients with PV or HV thrombosis on DUL following a limited CECT, and 2 (0.4%) patients who developed interval PV thrombosis between CECT and DUL.

CONCLUSION

DUL after CECT added diagnostic value for PV and/or HV thrombosis in less than 5% of patients. The patency of PV and HV is often not explicitly mentioned in CECT reports at our institution, which may lead to uncertainty for the referring provider as to whether the PV and HV were adequately evaluated. Few CECT have false positive or missed or underreported findings, and a careful review of the original CECT should be performed if DUL is requested.

Inter and intra-observer errors for postoperative total hip radiographic assessment using computer aided design

Plain radiographic assessment of primary total hip arthroplasty following surgery remains to be the commonest radiological assessment. The current paper, studies the accuracy and concordance between observers reviewing these radiographs. A prospective radiographic and medical note review of ten patients who underwent total hip replacement for primary osteoarthritis, with a mean age of 69 years. Early and 6 weeks postoperative x-rays were assessed for hip profile and version profile using computer aided design (CAD) by two observers on two different occasions. The observers were Orthopaedic surgeons who perform arthroplasty of the hip. The results were analyzed statistically. Dimensions, including Femoral offset, medial offset and ilioischial offset showed a high degree of inter- film and intra-film correlation, with inter-class correlation (ICC) over 0.8. Except of the intra-film correlation of ilioischial offset measured on the post- operative films (p=0.067) by the first rater, all the intra and inter film correlation were significantly over the benchmark of 0.6. In terms of stem alignment, cup inclination and cup version, the intra-film correlation by rater n°2 ranges from 0.574 to 0.975 and were significantly over the benchmark of 0.6, except in the case of cup inclination measured on the 6 th? week follow-up ; meanwhile the intra-film correlation by rater n°1 ranges from 0.581 to 0.819 and none were significantly over the benchmark of 0.6. The inter-rater reliability and inter-film correlation showed a dichotomy of results among different dimensions of the measurement. Dimensions of femo- ral offset, medial offset and ilioischial offset showed a substantial degree of reliability in terms of inter-rater reliability, inter-film correlation, and intra-rater/film reliability.

Fractures of child abuse

The objective of this paper is to review common challenges when evaluating fractures in the setting of possible child abuse and approaches to navigate them. This paper reviews the variety of imaging modalities available for evaluating child abuse and the advantages/disadvantages of each. Additionally, the authors discuss management of equivocal fractures, including the impact of double-reading skeletal surveys. The complexity of dating the acuity of fractures in young children is discussed. Utilizing the knowledge of fracture type, fracture patterns and patient history, as well as in the setting of cardiopulmonary resuscitation, the authors provide methods for determining the likelihood of abuse.

An Overview of Artificial Intelligence Applications in Liver and Pancreatic Imaging

Artificial intelligence (AI) is one of the most promising fields of research in medical imaging so far. By means of specific algorithms, it can be used to help radiologists in their routine workflow. There are several papers that describe AI approaches to solve different problems in liver and pancreatic imaging. These problems may be summarized in four different categories: segmentation, quantification, characterization and image quality improvement. Segmentation is usually the first step of successive elaborations. If done manually, it is a time-consuming process. Therefore, the semi-automatic and automatic creation of a liver or a pancreatic mask may save time for other evaluations, such as quantification of various parameters, from organs volume to their textural features. The alterations of normal liver and pancreas structure may give a clue to the presence of a diffuse or focal pathology. AI can be trained to recognize these alterations and propose a diagnosis, which may then be confirmed or not by radiologists. Finally, AI may be applied in medical image reconstruction in order to increase image quality, decrease dose administration (referring to computed tomography) and reduce scan times. In this article, we report the state of the art of AI applications in these four main categories.

Prevalence of axial spondyloarthritis in patients with inflammatory bowel disease using cross-sectional imaging: a systematic literature review

BACKGROUND

Patients with inflammatory bowel disease (IBD) have an excess burden of axial spondyloarthritis (axSpA), which, if left untreated, may significantly impact on clinical outcomes. We aimed to estimate the prevalence of axSpA, including previously undiagnosed cases, in IBD patients from studies involving cross-sectional imaging and identify the IBD features potentially associated with axSpA.

METHODS

PubMed, Embase and Cochrane databases were searched systematically between 1990 and 2018. Article reference lists and key conference abstract lists from 2012 to 2018 were also reviewed. All abstracts were reviewed by two authors to determine eligibility for inclusion. The study inclusion criteria were (a) adults aged 18 years or above, (b) a clinical diagnosis of IBD and (c) reporting identification of sacroiliitis using cross-sectional imaging.

RESULTS

A total of 20 observational studies were identified: 12 used CT, 6 used MR and 2 utilised both computed tomography (CT) and magnetic resonance (MR) imaging. Sample sizes ranged from 25 to 1247 (a total of 4096 patients); 31 studies were considered to have low selection bias, 13 included two or more radiology readers, and 3 included rheumatological assessments. The prevalence of sacroiliitis, the most commonly reported axSpA feature, ranged from 2.2% to 68.0% with a pooled prevalence of 21.0% [95% confidence interval (CI) 17-26%]. Associated IBD features include increasing IBD duration, increasing age, male sex, IBD location, inflammatory back pain and peripheral arthritis. No significant difference in the prevalence of sacroiliitis between Crohn's disease and ulcerative colitis was identified. Study limitations include variability in the individual study sample sizes and patient demographics.

CONCLUSION

This review highlights the need for larger, well-designed studies using more sensitive imaging modalities and multivariable modelling to better estimate the prevalence of axSpA in IBD. An improved knowledge of the IBD phenotype(s) associated with axSpA and use of cross-sectional imaging intended for IBD assessment to screen for axSpA may help clinicians identify those patients most at risk.

Under-exploration of Three-Dimensional Images Leads to Search Errors for Small Salient Targets

Advances in 3D imaging technology are transforming how radiologists search for cancer1,2 and how security officers scrutinize baggage for dangerous objects.3 These new 3D technologies often improve search over 2D images4,5 but vastly increase the image data. Here, we investigate 3D search for targets of various sizes in filtered noise and digital breast phantoms. For a Bayesian ideal observer optimally processing the filtered noise and a convolutional neural network processing the digital breast phantoms, search with 3D image stacks increases target information and improves accuracy over search with 2D images. In contrast, 3D search by humans leads to high miss rates for small targets easily detected in 2D search, but not for larger targets more visible in the visual periphery. Analyses of human eye movements, perceptual judgments, and a computational model with a foveated visual system suggest that human errors can be explained by interaction among a target's peripheral visibility, eye movement under-exploration of the 3D images, and a perceived overestimation of the explored area. Instructing observers to extend the search reduces 75% of the small target misses without increasing false positives. Results with twelve radiologists confirm that even medical professionals reading realistic breast phantoms have high miss rates for small targets in 3D search. Thus, under-exploration represents a fundamental limitation to the efficacy with which humans search in 3D image stacks and miss targets with these prevalent image technologies.

Studying human-AI collaboration protocols: the case of the Kasparov's law in radiological double reading

Purpose

The integration of Artificial Intelligence into medical practices has recently been advocated for the promise to bring increased efficiency and effectiveness to these practices. Nonetheless, little research has so far been aimed at understanding the best human-AI interaction protocols in collaborative tasks, even in currently more viable settings, like independent double-reading screening tasks.

Methods

To this aim, we report about a retrospective case–control study, involving 12 board-certified radiologists, in the detection of knee lesions by means of Magnetic Resonance Imaging, in which we simulated the serial combination of two Deep Learning models with humans in eight double-reading protocols. Inspired by the so-called Kasparov’s Laws, we investigate whether the combination of humans and AI models could achieve better performance than AI models alone, and whether weak reader, when supported by fit-for-use interaction protocols, could out-perform stronger readers.

Results

We discuss two main findings: groups of humans who perform significantly worse than a state-of-the-art AI can significantly outperform it if their judgements are aggregated by majority voting (in concordance with the first part of the Kasparov’s law); small ensembles of significantly weaker readers can significantly outperform teams of stronger readers, supported by the same computational tool, when the judgments of the former ones are combined within “fit-for-use” protocols (in concordance with the second part of the Kasparov’s law).

Conclusion

Our study shows that good interaction protocols can guarantee improved decision performance that easily surpasses the performance of individual agents, even of realistic super-human AI systems. This finding highlights the importance of focusing on how to guarantee better co-operation within human-AI teams, so to enable safer and more human sustainable care practices.

Utility of Artificial Intelligence Tool as a Prospective Radiology Peer Reviewer - Detection of Unreported Intracranial Hemorrhage

RATIONALE AND OBJECTIVES

Misdiagnosis of intracranial hemorrhage (ICH) can adversely impact patient outcomes. The increasing workload on the radiologists may increase the chance of error and compromise the quality of care provided by the radiologists.

MATERIALS AND METHODS

We used an FDA approved artificial intelligence (AI) solution based on a convolutional neural network to assess the prevalence of ICH in scans, which were reported as negative for ICH. We retrospectively applied the AI solution to all consecutive noncontrast computed tomography (CT) head scans performed at eight imaging sites affiliated to our institution.

RESULTS

In the 6565 noncontrast CT head scans, which met the inclusion criteria, 5585 scans were reported to have no ICH ("negative-by-report" cases). We applied AI solution to these "negative-by-report" cases. AI solution suggested there were ICH in 28 of these scans ("negative-by-report" and "positive-by-AI solution"). After consensus review by three neuroradiologists, 16 of these scans were found to have ICH, which was not reported (missed diagnosis by radiologists), with a false-negative rate of radiologists for ICH detection at 1.6%. Most commonly missed ICH was overlying the cerebral convexity and in the parafalcine regions.

CONCLUSION

Our study demonstrates that an AI solution can help radiologists to diagnose ICH and thus decrease the error rate. AI solution can serve as a prospective peer review tool for non-contrast head CT scans to identify ICH and thus minimize false negatives.

Expert radiologist review at a hepatobiliary multidisciplinary tumor board: impact on patient management

PURPOSE

To identify the frequency, source, and management impact of discrepancies between the initial radiology report and expert reinterpretation occurring in the context of a hepatobiliary multidisciplinary tumor board (MTB).

METHODS

This retrospective study included 974 consecutive patients discussed at a weekly MTB at a large tertiary care academic medical center over a 2-year period. A single radiologist with dedicated hepatobiliary imaging expertise attended all conferences to review and discuss the relevant liver imaging and rated the concordance between original and re-reads based on RADPEER scoring criteria. Impact on management was based on the conference discussion and reflected changes in follow-up imaging, recommendations for biopsy/surgery, or liver transplant eligibility.

RESULTS

Image reinterpretation was discordant with the initial report in 19.9% (194/974) of cases (59.8%, 34.5%, 5.7% RADPEER 2/3/4 discrepancies, respectively). A change in LI-RADS category occurred in 59.8% of discrepancies. Most common causes of discordance included re-classification of a lesion as benign rather than malignant (16.0%) and missed tumor recurrence (13.9%). Impact on management occurred in 99.0% of discordant cases and included loco-regional therapy instead of follow-up imaging (19.1%), follow-up imaging instead of treatment (17.5%), and avoidance of biopsy (12.4%). 11.3% received OPTN exception scores due to the revised interpretation, and 8.8% were excluded from listing for orthotopic liver transplant.

CONCLUSION

Even in a sub-specialized abdominal imaging academic practice, expert radiologist review in the MTB setting identified discordant interpretations and impacted management in a substantial fraction of patients, potentially impacting transplant allocation. The findings may impact how abdominal imaging sections best staff advanced MTBs.

The use of imaging to identify immunocompromised children requiring biopsy for invasive fungal rhinosinusitis

BACKGROUND AND PURPOSE

Children with severe immunocompromise due to cancer therapy or hematopoietic cell transplant are at risk both for potentially lethal invasive fungal rhinosinusitis (IFRS), and for complications associated with gold-standard biopsy diagnosis. We investigated whether early imaging could reliably identify or exclude IFRS in this population, thereby reducing unnecessary biopsy.

METHODS

We reviewed clinical/laboratory data and cross-sectional imaging from 31 pediatric patients evaluated for suspicion of IFRS, 19 without (age 11.8 ± 5.4 years) and 12 with proven IFRS (age 11.9 ± 4.6 years). Imaging examinations were graded for mucosal thickening (Lund score), for fungal-specific signs (FSS) of bone destruction, extra-sinus inflammation, and nasal mucosal ulceration. Loss of contrast enhancement (LoCE) was assessed separately where possible. Clinical and imaging findings were compared with parametric or nonparametric tests as appropriate. Diagnostic accuracy was assessed by receiver operating characteristic (ROC) analysis. Positive (+LR) and negative likelihood ratios (-LR) and probabilities were calculated.

RESULTS

Ten of 12 patients with IFRS and one of 19 without IFRS had at least one FSS on early imaging (83% sensitive, 95% specific, +LR = 15.83, -LR = 0.18; P < .001). Absolute neutrophil count (ANC) ≤ 200/mm³ was 100% sensitive and 58% specific for IFRS (+LR = 2.38, -LR = 0; P = .001). Facial pain was the only discriminating symptom of IFRS (P < .001). In a symptomatic child with ANC ≤ 200/m³ , the presence of at least one FSS indicated high (79%) probability of IFRS; absence of FSS suggested low (<4%) probability.

CONCLUSION

In symptomatic, severely immunocompromised children, the presence or absence of fungal-specific imaging findings may effectively rule in or rule out early IFRS, potentially sparing some patients the risks associated with biopsy.

Body MRI Subspecialty Reinterpretations at a Tertiary Care Center: Discrepancy Rates and Error Types

OBJECTIVE. Radiology departments in tertiary care centers are frequently asked to perform secondary interpretations of imaging studies, particularly when a patient is transferred from a community hospital. Discrepancy rates in radiology vary widely, with low rates reported for preliminary resident reports that are overread by attending radiologists (2-6%) and higher rates (up to 56%) for secondary interpretations. Abdominal and pelvic imaging and cross-sectional imaging have the highest discrepancy rates. The purpose of our study was to determine the discrepancy rate and the most common reasons for discrepancies between abdominal and pelvic MRI reports obtained from outside institutions and secondary interpretations of these reports by a fellowship-trained radiologist at a tertiary care center. MATERIALS AND METHODS. We retrospectively identified 395 secondary MRI reports from January 2015 to December 2018 that were labeled as body MRI examinations at a tertiary care center. Thirty-eight cases were excluded for various reasons, including incorrect categorization or lack of outside report. We reviewed the outside reports, compared them with the secondary interpretations, and categorized the cases as discrepancy or no discrepancy. The discrepancies were subdivided into the most likely reason for the error using previously published categories; these categories were also divided into perceptive and cognitive errors. RESULTS. Of the 357 included cases, 246 (68.9%) had at least one discrepancy. The most common reason for error was faulty reasoning (34.3%), which is a cognitive error characterized by misidentifying an abnormality. Satisfaction of search, which is a perceptive error, was the most common reason for second discrepancies (15.0%). CONCLUSION. Secondary interpretations of body MR images at a tertiary care center identify a high rate of discrepancies, with cognitive error types predominating.

WITHDRAWN: Radiology Extenders: Impact on Throughput and Accuracy for Routine Chest Radiographs

This article has been withdrawn at the request of the author(s) and/or editor. The Publisher apologizes for any inconvenience this may cause. The full Elsevier Policy on Article Withdrawal can be found at https://www.elsevier.com/about/our-business/policies/article-withdrawal.