Discrepancy Rates and Clinical Impact of Imaging Secondary Interpretations: A Systematic Review and Meta-Analysis

Reinterpretation of Hepatopancreaticobiliary Imaging Exams: Assessment of Clinical Impact, Peer Learning, and Physician Satisfaction

OBJECTIVES

To assess the impact on clinical management, potential for peer learning, and referring physician satisfaction with subspecialist reinterpretations of hepatopancreaticobiliary (HPB) imaging examinations.

MATERIALS AND METHODS

HPB CTs and MRIs from outside hospitals were reinterpreted by two subspecialty radiologists between March 2021 and August 2022. Reinterpretation reports were mailed to radiologists that issued primary reports. The electronic record was reviewed to assess for changes in clinical management based on the reinterpretations (yes/no/unavailable). To assess the potential for peer learning, a survey using a 5-point Likert scale was sent to radiologists who issued primary reports. A separate survey was sent to referring physicians to assess satisfaction with reinterpretations.

RESULTS

Two hundred fifty imaging examinations (122 CT, 128 MRI) were reinterpreted at the request of 19 referring physicians. Ninety-six radiologists issued primary reports. RADPEER scores 1-3 were assigned to 131/250 (52%), 86/250 (34%), and 33/250 (13%) examinations, respectively. Of 213 reinterpretations with adequate records for assessment, 75/213 (35%) were associated with a change in management; of these, 71/75 (95%) were classified as RADPEER 2 or 3. Most radiologists agreed or strongly agreed with the following: prefer to receive reinterpretations (34/36, 94%); reinterpretations changed practice of reporting HPB imaging examinations (23/36, 64%); and reinterpretations offer opportunities for peer learning (34/36, 94%). Referring physicians agreed or strongly agreed (7/7, 100%) that reinterpretations are valuable and often change or clarify management of patients with complex HPB disease, and offer an opportunity for peer learning.

CONCLUSION

Radiologists and referring physicians strongly agree that HPB imaging reinterpretations help support peer learning and patient management, respectively.

Shift Volume Directly Impacts Neuroradiology Error Rate at a Large Academic Medical Center: The Case for Volume Limits

BACKGROUND AND PURPOSE

Unlike in Europe and Japan, guidelines or recommendations from specialized radiological societies on workflow management and adaptive intervention to reduce error rates are currently lacking in the United States. This study of neuroradiologic reads at a large US academic medical center, which may hopefully contribute to this discussion, found a direct relationship between error rate and shift volume.

MATERIALS AND METHODS

CT and MR imaging reports from our institution's Neuroradiology Quality Assurance database (years 2014-2020) were searched for attending physician errors. Data were collected on shift volume specific error rates per 1000 interpreted studies and RADPEER scores. Optimal cutoff points for 2, 3 and 4 groups of shift volumes were computed along with subgroups' error rates.

RESULTS

A total of 643 errors were found, 91.7% of which were clinically significant (RADPEER 2b, 3b). The overall error rate (errors/1000 examinations) was 2.36. The best single shift volume cutoff point generated 2 groups: ≤ 26 studies (error rate 1.59) and > 26 studies (2.58; OR: 1.63, P < .001). The best 2 shift volume cutoff points generated 3 shift volume groups: ≤ 19 (1.34), 20-28 (1.88; OR: 1.4, P = .1) and ≥ 29 (2.6; OR: 1.94, P < .001). The best 3 shift volume cutoff points generated 4 groups: ≤ 24 (1.59), 25-66 (2.44; OR: 1.54, P < .001), 67-90 (3.03; OR: 1.91, P < .001), and ≥ 91 (2.07; OR: 1.30, P = .25). The group with shift volume ≥ 91 had a limited sample size.

CONCLUSIONS

Lower shift volumes yielded significantly lower error rates. The lowest error rates were observed with shift volumes that were limited to 19-26 studies. Error rates at shift volumes between 67-90 studies were 226% higher, compared with the error rate at shift volumes of ≤ 19 studies.

How to prepare for a bright future of radiology in Europe

Because artificial intelligence (AI)-powered algorithms allow automated image analysis in a growing number of diagnostic scenarios, some healthcare stakeholders have raised doubts about the future of the entire radiologic profession. Their view disregards not only the role of radiologists in the diagnostic service chain beyond reporting, but also the many multidisciplinary and patient-related consulting tasks for which radiologists are solicited. The time commitment for these non-reporting tasks is considerable but difficult to quantify and often impossible to fulfil considering the current mismatch between workload and workforce in many countries. Nonetheless, multidisciplinary, and patient-centred consulting activities could move up on radiologists' agendas as soon as AI-based tools can save time in daily routine. Although there are many reasons why AI will assist and not replace radiologists as imaging experts in the future, it is important to position the next generation of European radiologists in view of this expected trend. To ensure radiologists' personal professional recognition and fulfilment in multidisciplinary environments, the focus of training should go beyond diagnostic reporting, concentrating on clinical backgrounds, specific communication skills with referrers and patients, and integration of imaging findings with those of other disciplines. Close collaboration between the European Society of Radiology (ESR) and European national radiologic societies can help to achieve these goals. Although each adequate treatment begins with a correct diagnosis, many health politicians see radiologic procedures mainly as a cost factor. Radiologic research should, therefore, increasingly investigate the imaging impact on treatment and outcome rather than focusing mainly on technical improvements and diagnostic accuracy alone.Critical relevance statement Strategies are presented to prepare for a successful future of the radiologic profession in Europe, if AI-powered tools can alleviate the current reporting overload: engaging in multidisciplinary activities (clinical and integrative diagnostics), enhancing the value and recognition of radiologists' role through clinical expertise, focusing radiological research on the impact on diagnosis and outcome, and promoting patient-centred radiology by enhancing communication skills.Key points • AI-powered tools will not replace radiologists but hold promise to reduce the current reporting burden, enabling them to reinvest liberated time in multidisciplinary clinical and patient-related tasks.• The skills and resources for these tasks should be considered when recruiting and teaching the next generation of radiologists, when organising departments and planning staffing.• Communication skills will play an increasing role in both multidisciplinary activities and patient-centred radiology.• The value and importance of a correct and integrative diagnosis and the cost of an incorrect imaging diagnosis should be emphasised when discussing with non-medical stakeholders in healthcare.• The radiologic community in Europe should start now to prepare for a bright future of the profession for the benefit of patients and medical colleagues alike.

Can we predict discordant RECIST 1.1 evaluations in double read clinical trials?

Background

In lung clinical trials with imaging, blinded independent central review with double reads is recommended to reduce evaluation bias and the Response Evaluation Criteria In Solid Tumor (RECIST) is still widely used. We retrospectively analyzed the inter-reader discrepancies rate over time, the risk factors for discrepancies related to baseline evaluations, and the potential of machine learning to predict inter-reader discrepancies.

Materials and methods

We retrospectively analyzed five BICR clinical trials for patients on immunotherapy or targeted therapy for lung cancer. Double reads of 1724 patients involving 17 radiologists were performed using RECIST 1.1. We evaluated the rate of discrepancies over time according to four endpoints: progressive disease declared (PDD), date of progressive disease (DOPD), best overall response (BOR), and date of the first response (DOFR). Risk factors associated with discrepancies were analyzed, two predictive models were evaluated.

Results

At the end of trials, the discrepancy rates between trials were not different. On average, the discrepancy rates were 21.0%, 41.0%, 28.8%, and 48.8% for PDD, DOPD, BOR, and DOFR, respectively. Over time, the discrepancy rate was higher for DOFR than DOPD, and the rates increased as the trial progressed, even after accrual was completed. It was rare for readers to not find any disease, for less than 7% of patients, at least one reader selected non-measurable disease only (NTL). Often the readers selected some of their target lesions (TLs) and NTLs in different organs, with ranges of 36.0-57.9% and 60.5-73.5% of patients, respectively. Rarely (4-8.1%) two readers selected all their TLs in different locations. Significant risk factors were different depending on the endpoint and the trial being considered. Prediction had a poor performance but the positive predictive value was higher than 80%. The best classification was obtained with BOR.

Conclusion

Predicting discordance rates necessitates having knowledge of patient accrual, patient survival, and the probability of discordances over time. In lung cancer trials, although risk factors for inter-reader discrepancies are known, they are weakly significant, the ability to predict discrepancies from baseline data is limited. To boost prediction accuracy, it would be necessary to enhance baseline-derived features or create new ones, considering other risk factors and looking into optimal reader associations.

Factors Associated With Neuroradiologic Diagnostic Errors at a Large Tertiary-Care Academic Medical Center: A Case-Control Study

BACKGROUND. Numerous studies have explored factors associated with diagnostic errors in neuroradiology; however, large-scale multivariable analyses are lacking. OBJECTIVE. The purpose of this study was to evaluate associations of interpretation time, shift volume, care setting, day of week, and trainee participation with diagnostic errors by neuroradiologists at a large academic medical center. METHODS. This retrospective case-control study using a large tertiary-care academic medical center's neuroradiology quality assurance database evaluated CT and MRI examinations for which neuroradiologists had assigned RADPEER scores. The database was searched from January 2014 through March 2020 for examinations without (RADPEER score of 1) or with (RADPEER scores of 2a, 2b, 3a, 3b, or 4) diagnostic error. For each examination with error, two examinations without error were randomly selected (unless only one examination could be identified) and matched by interpreting radiologist and examination type to form case and control groups. Marginal mixed-effects logistic regression models were used to assess associations of diagnostic error with interpretation time (number of minutes since the immediately preceding report's completion), shift volume (number of examinations interpreted during the shift), emergency/inpatient setting, weekend interpretation, and trainee participation in interpretation. RESULTS. The case group included 564 examinations in 564 patients (mean age, 50.0 ± 25.0 [SD] years; 309 men, 255 women); the control group included 1019 examinations in 1019 patients (mean age, 52.5 ± 23.2 years; 540 men, 479 women). In the case versus control group, mean interpretation time was 16.3 ± 17.2 [SD] minutes versus 14.8 ± 16.7 minutes; mean shift volume was 50.0 ± 22.1 [SD] examinations versus 45.4 ± 22.9 examinations. In univariable models, diagnostic error was associated with shift volume (OR = 1.22, p < .001) and weekend interpretation (OR = 1.60, p < .001) but not interpretation time, emergency/inpatient setting, or trainee participation (p > .05). However, in multivariable models, diagnostic error was independently associated with interpretation time (OR = 1.18, p = .003), shift volume (OR = 1.27, p < .001), and weekend interpretation (OR = 1.69, p = .02). In subanalysis, diagnostic error showed independent associations on weekdays with interpretation time (OR = 1.18, p = .003) and shift volume (OR = 1.27, p < .001); such associations were not observed on weekends (interpretation time: p = .62; shift volume: p = .58). CONCLUSION. Diagnostic errors in neuroradiology were associated with longer interpretation times, higher shift volumes, and weekend interpretation. CLINICAL IMPACT. These findings should be considered when designing work-flow-related interventions seeking to reduce neuroradiology interpretation errors.

Second-opinion interpretation of outside facility general ultrasound studies: rate of discrepancies and management change

BACKGROUND

Second-opinion reads on imaging studies are common for CT and MRI, but many institutions are hesitant to implement a workflow for second read of ultrasound studies performed at other facilities due to quality considerations.

OBJECTIVE

The purpose of this study was to assess discrepancy rates between initial and second-opinion general ultrasound reports METHODS: We reviewed all requests of second-opinion US studies referred to our tertiary care center between 02/01/2020 and 06/23/2022. We evaluated percentage of exams that were interpreted versus archived. Whenever the original report was available (n = 196 studies), we evaluated any discrepancy in findings, interpretation, and potential management change based on second report compared to the initial report as evaluated by consensus agreement of 3 subspecialized radiologists.

RESULTS

A total of 586 ultrasound studies for 533 patients were nominated for consult. After excluding 58 studies for technical reasons (e.g., duplicate nomination, images for procedure guidance, modality is not ultrasound) and 282 studies that were archived by the reading radiologist due to various objective (e.g., studies such as echocardiography not interpreted by the abdominal imagers or more recent study available obviating need for consultation) and subjective (e.g., suboptimal image quality, lack of cine clips) reasons, a total of 246 studies were reinterpreted and were further analyzed. Only 21/246 patients (8.5%) got repeat ultrasound of the same body part within 3 months of original study date. The original (first-read) report was available for 196/246 studies, with discrepancy present between the first and second reads in 74/196 (37.8%) studies, with potential management change in 51/196 (26.0%) studies.

CONCLUSION

Second-opinion interpretation of outside ultrasound examinations by subspecialized radiologists can result in recommended management change in 26% of studies indicating potential for added value to reinterpreting ultrasound studies despite the concerns for quality control.

Impact of Shift Volume on Neuroradiology Diagnostic Errors at a Large Tertiary Academic Center

BACKGROUND AND PURPOSE

Medical errors can result in significant morbidity and mortality. The goal of our study is to evaluate correlation between shift volume and errors made by attending neuroradiologists at an academic medical center, using a large data set.

MATERIALS AND METHODS

CT and MRI reports from our Neuroradiology Quality Assurance database (years 2014 - 2020) were searched for attending physician errors. Data were collected on shift volume, category of missed findings, error type, interpretation setting, exam type, clinical significance.

RESULTS

654 reports contained diagnostic error. There was a significant difference between mean volume of interpreted studies on shifts when an error was made compared with shifts in which no error was documented (46.58 (SD=22.37) vs 34.09 (SD=18.60), p<0.00001); and between shifts when perceptual error was made compared with shifts when interpretive errors were made (49.50 (SD=21.9) vs 43.26 (SD=21.75), p=0.0094). 59.6% of errors occurred in the emergency/inpatient setting, 84% were perceptual and 91.1% clinically significant. Categorical distribution of errors was: vascular 25.8%, brain 23.4%, skull base 13.8%, spine 12.4%, head/neck 11.3%, fractures 10.2%, other 3.1%. Errors were detected most often on brain MRI (25.4%), head CT (18.7%), head/neck CTA (13.8%), spine MRI (13.7%).

CONCLUSION

Errors were associated with higher volume shifts, were primarily perceptual and clinically significant. We need National guidelines establishing a range of what is a safe number of interpreted cross-sectional studies per day.

Circulating Cells with Macrophage-like Characteristics in Cancer: The Importance of Circulating Neoplastic-Immune Hybrid Cells in Cancer

Simple Summary

In cancer, disseminated neoplastic cells circulating in blood are a source of tumor DNA, RNA, and protein, which can be harnessed to diagnose, monitor, and better understand the biology of the tumor from which they are derived. Historically, circulating tumor cells (CTCs) have dominated this field of study. While CTCs are shed directly into circulation from a primary tumor, they remain relatively rare, particularly in early stages of disease, and thus are difficult to utilize as a reliable cancer biomarker. Neoplastic-immune hybrid cells represent a novel subpopulation of circulating cells that are more reliably attainable as compared to their CTC counterparts. Here, we review two recently identified circulating cell populations in cancer—cancer-associated macrophage-like cells and circulating hybrid cells—and discuss the future impact for the exciting area of disseminated hybrid cells.

Abstract

Cancer remains a significant cause of mortality in developed countries, due in part to difficulties in early detection, understanding disease biology, and assessing treatment response. If effectively harnessed, circulating biomarkers promise to fulfill these needs through non-invasive “liquid” biopsy. While tumors disseminate genetic material and cellular debris into circulation, identifying clinically relevant information from these analytes has proven difficult. In contrast, cell-based circulating biomarkers have multiple advantages, including a source for tumor DNA and protein, and as a cellular reflection of the evolving tumor. While circulating tumor cells (CTCs) have dominated the circulating cell biomarker field, their clinical utility beyond that of prognostication has remained elusive, due to their rarity. Recently, two novel populations of circulating tumor-immune hybrid cells in cancer have been characterized: cancer-associated macrophage-like cells (CAMLs) and circulating hybrid cells (CHCs). CAMLs are macrophage-like cells containing phagocytosed tumor material, while CHCs can result from cell fusion between cancer and immune cells and play a role in the metastatic cascade. Both are detected in higher numbers than CTCs in peripheral blood and demonstrate utility in prognostication and assessing treatment response. Additionally, both cell populations are heterogeneous in their genetic, transcriptomic, and proteomic signatures, and thus have the potential to inform on heterogeneity within tumors. Herein, we review the advances in this exciting field.

Neuroradiology diagnostic errors at a tertiary academic centre: effect of participation in tumour boards and physician experience

AIM

To quantify and correlate the diagnostic error rates in radiological interpretation with the experience of the attending neuroradiologist at a tertiary academic medical centre.

MATERIALS AND METHODS

The institution's Neuroradiology Quality Assurance Database of diagnostic errors was searched for misses from 2014-2020. Attendance at Head and Neck (H&N), Brain, and Paediatric Neuroradiology (PN) tumour boards (TB) as the presenting radiologist was recorded. Number of post-fellowship years of clinical practice (CPY) and frequency of TB attendance were considered separate metrics of a radiologist's experience. Radiological errors were categorised as Total, H&N, Skull Base (SKB), Brain, or PN diagnostic errors. Diagnostic error rates per attending neuroradiologist within each category were correlated with the frequency of TB participation and CPY using Spearman's rank correlation coefficients.

RESULTS

A total 607 examinations contained a diagnostic error. Spearman's rank correlation coefficients between Total TB participation and Total, H&N, SKB, Brain error rates were: -0.89 (p=0.0002); -0.81 (p=0.002); -0.66 (p=0.03); -0.82 (p=0.002); respectively. Spearman's rank correlation coefficients between CPY and Total, H&N, SKB, Brain and PN error rates were: 0.05 (p=0.88); 0.08 (p=0.82); 0.28 (p=0.41); -0.10 (p=0.77); -0.16 (p=0.63), respectively. Spearman's rank correlation coefficients between H&N TB and H&N, SKB error rates; and between Brain TB attendance and Brain error rates were statistically significant (p<0.05).

CONCLUSION

The present study shows a strong correlation between high TB participation rates and low diagnostic error rates. The number of years in practice did not appear to influence error rate.

Prevalence, features, and explanations of missed and misinterpreted pancreatic cancer on imaging: a matched case-control study

PURPOSE

To characterize the prevalence of missed pancreatic masses and pancreatic ductal adenocarcinoma (PDAC)-related findings on CT and MRI between pre-diagnostic patients and healthy individuals.

MATERIALS AND METHODS

Patients diagnosed with PDAC (2010-2016) were retrospectively reviewed for abdominal CT- or MRI-examinations 1 month-3 years prior to their diagnosis, and subsequently matched to controls in a 1:4 ratio. Two blinded radiologists scored each imaging exam on the presence of a pancreatic mass and secondary features of PDAC. Additionally, original radiology reports were graded based on the revised RADPEER criteria.

RESULTS

The cohort of 595 PDAC patients contained 60 patients with a pre-diagnostic CT and 27 with an MRI. A pancreatic mass was suspected in hindsight on CT in 51.7% and 50% of cases and in 1.3% and 0.9% of controls by reviewer 1 (p < .001) and reviewer 2 (p < .001), respectively. On MRI, a mass was suspected in 70.4% and 55.6% of cases and 2.9% and 0% of the controls by reviewer 1 (p < .001) and reviewer 2 (p < .001), respectively. Pancreatic duct dilation, duct interruption, focal atrophy, and features of acute pancreatitis is strongly associated with PDAC (p < .001). In cases, a RADPEER-score of 2 or 3 was assigned to 56.3% of the CT-reports and 71.4% of MRI-reports.

CONCLUSION

Radiological features as pancreatic duct dilation and interruption, and focal atrophy are common first signs of PDAC and are often missed or unrecognized. Further investigation with dedicated pancreas imaging is warranted in patients with PDAC-related radiological findings.

Health Economic Impact of Software-Assisted Brain MRI on Therapeutic Decision-Making and Outcomes of Relapsing-Remitting Multiple Sclerosis Patients-A Microsimulation Study

Aim: To develop a microsimulation model to assess the potential health economic impact of software-assisted MRI in detecting disease activity or progression in relapsing-remitting multiple sclerosis (RRMS) patients. Methods: We develop a simulated decision analytical model based on a hypothetical cohort of RRMS patients to compare a baseline decision-making strategy in which only clinical evolution (relapses and disability progression) factors are used for therapy decisions in MS follow-up, with decision-making strategies involving MRI. In this context, we include comparisons with a visual radiologic assessment of lesion evolution, software-assisted lesion detection, and software-assisted brain volume loss estimation. The model simulates clinical (EDSS transitions, number of relapses) and subclinical (new lesions and brain volume loss) disease progression and activity, modulated by the efficacy profiles of different disease-modifying therapies (DMTs). The simulated decision-making process includes the possibility to escalate from a low efficacy DMT to a high efficacy DMT or to switch between high efficacy DMTs when disease activity is detected. We also consider potential error factors that may occur during decision making, such as incomplete detection of new lesions, or inexact computation of brain volume loss. Finally, differences between strategies in terms of the time spent on treatment while having undetected disease progression/activity, the impact on the patient’s quality of life, and costs associated with health status from a US perspective, are reported. Results: The average time with undetected disease progression while on low efficacy treatment is shortened significantly when using MRI, from around 3 years based on clinical criteria alone, to 2 when adding visual examination of MRI, and down to only 1 year with assistive software. Hence, faster escalation to a high efficacy DMT can be performed when MRI software is added to the radiological reading, which has positive effects in terms of health outcomes. The incremental utility shows average gains of 0.23 to 0.37 QALYs over 10 and 15 years, respectively, when using software-assisted MRI compared to clinical parameters only. Due to long-term health benefits, the average annual costs associated with health status are lower by $1500–$2200 per patient when employing MRI and assistive software. Conclusions: The health economic burden of MS is high. Using assistive MRI software to detect and quantify lesions and/or brain atrophy has a significant impact on the detection of disease activity, treatment decisions, health outcomes, utilities, and costs in patients with MS.

Accuracy and Clinical Utility of Reports from Outside Hospitals for CT of the Cervical Spine in Blunt Trauma

BACKGROUND AND PURPOSE

Multidetector CT is the workhorse for detecting blunt cervical spine injury. There is no standard of care for re-interpretation of radiology images for patients with blunt trauma transferred to a higher level of care. The clinical impact of discrepancies of cervical spine CT reads remains unclear. We evaluated the discordance between primary (from referring hospitals) and secondary radiology interpretations (from a receiving level I tertiary trauma center) of cervical spine CT scans in patients with blunt trauma and assessed the clinical implications of missed cervical spine fractures.

MATERIALS AND METHODS

Medical records of patients with blunt trauma transferred to our institution between 2008 and 2015 were reviewed. Primary and secondary interpretations were compared and categorized as concordant and discordant. Two senior neuroradiologists adjudicated discordant reports. The benefit of re-interpretation was determined. For discordant cases, outcomes at discharge, injury severity pattern, treatment, and arrival in a cervical collar were assessed.

RESULTS

Six hundred fifty patients were included; 608 (94%) presented with concordant reports: 401 (61.7%) with fractures and 207 (31.8%) with no fractures. There were 42 (6.5%) discordant reports; 18 (2.8%) were cervical spine injuries undetected on the primary interpretation. Following adjudication, the secondary interpretation improved the sensitivity (99.3% versus 95.7%) and specificity (99.1% versus 91.7%) in detecting cervical spine fractures compared with the primary interpretation alone (P < .001).

CONCLUSIONS

There was an overall 6.5% discordance rate between primary and secondary interpretations of cervical spine CT scans. The secondary interpretation of the cervical spine CT increased the sensitivity and specificity of detecting cervical spine fractures in patients with blunt trauma transferred to higher-level care.

Interpretations of Examinations Outside of Radiologists' Fellowship Training: Assessment of Discrepancy Rates Among 5.9 Million Examinations From a National Teleradiology Databank

Background: In community settings, radiologists commonly function as multispecialty radiologists, interpreting examinations outside of their fellowship training. Objective: To compare discrepancy rates for preliminary interpretations of acute community-setting examinations concordant versus discordant with interpreting radiologists' fellowship training. Methods: This retrospective study used the databank of a U.S. teleradiology company that provides preliminary interpretations for client community hospitals. The analysis included 5,883,980 acute examinations performed from 2012 to 2016 that were preliminarily interpreted by 269 teleradiologists with a fellowship of neuroradiology, abdominal radiology, or musculoskeletal radiology. When providing final interpretations, client on-site radiologists voluntarily submitted quality assurance (QA) requests if preliminary and final interpretations were discrepant; the teleradiology company's QA committee categorized discrepancies as major (n=8,444) or minor (n=17,208). Associations among examination type (common vs advanced), relationship between examination subspecialty and the teleradiologist's fellowship (concordant vs discordant), and major and minor discrepancies were assessed using three-way conditional analyses with generalized estimating equations. Results: For examinations with concordant subspecialty, major discrepancy rate was lower for common than advanced examinations [0.13% vs 0.26%; relative risk (RR) 0.50, 95% CI: 0.42, 0.60; p < .001]. For examinations with discordant subspecialty, major discrepancy rate was lower for common than advanced examinations (0.14% vs 0.18%; RR 0.81, 95% CI: 0.72, 0.90; p < .001). For common examinations, major discrepancy rate was not different between examinations with concordant versus discordant subspecialty (0.13% vs 0.14%; RR 0.90, 95% CI: 0.81, 1.01; p = .07). For advanced examinations, major discrepancy rate was higher for examinations with concordant versus discordant subspecialty (0.26% vs 0.18%; RR 1.45, 95% CI: 1.18, 1.79; p < .001). Minor discrepancy rate was higher among advanced examinations for those with concordant versus discordant subspecialty (0.34% vs 0.29%; RR 1.17, 95% CI: 1.001, 1.36; p = .04), but not different for other comparisons (p > .05). Conclusion: Major and minor discrepancy rates were not higher for acute community-setting examinations outside of interpreting radiologists' fellowship training. Discrepancy rates increased for advanced examinations. Clinical Impact: The findings support multispecialty radiologist practice in acute community settings. Efforts to match examination and interpreting radiologist subspecialty may not reduce diagnostic discrepancies.

Key Technology Considerations in Developing and Deploying Machine Learning Models in Clinical Radiology Practice

The use of machine learning to develop intelligent software tools for the interpretation of radiology images has gained widespread attention in recent years. The development, deployment, and eventual adoption of these models in clinical practice, however, remains fraught with challenges. In this paper, we propose a list of key considerations that machine learning researchers must recognize and address to make their models accurate, robust, and usable in practice. We discuss insufficient training data, decentralized data sets, high cost of annotations, ambiguous ground truth, imbalance in class representation, asymmetric misclassification costs, relevant performance metrics, generalization of models to unseen data sets, model decay, adversarial attacks, explainability, fairness and bias, and clinical validation. We describe each consideration and identify the techniques used to address it. Although these techniques have been discussed in prior research, by freshly examining them in the context of medical imaging and compiling them in the form of a laundry list, we hope to make them more accessible to researchers, software developers, radiologists, and other stakeholders.

A Novel Digital Care Management Platform to Monitor Clinical and Subclinical Disease Activity in Multiple Sclerosis

In multiple sclerosis (MS), the early detection of disease activity or progression is key to inform treatment changes and could be supported by digital tools. We present a novel CE-marked and FDA-cleared digital care management platform consisting of (1) a patient phone/web application and healthcare professional portal (icompanion) including validated symptom, disability, cognition, and fatigue patient-reported outcomes; and (2) clinical brain magnetic resonance imaging (MRI) quantifications (icobrain ms). We validate both tools using their ability to detect (sub)clinical disease activity (known-groups validity) and real-world data insights. Surveys showed that 95.6% of people with MS (PwMS) were interested in using an MS app, and 98.2% were interested in knowing about MRI changes. The icompanion measures of disability (p < 0.001) and symptoms (p = 0.005) and icobrain ms MRI parameters were sensitive to (sub)clinical differences between MS subtypes. icobrain ms also decreased intra- and inter-rater lesion count variability and increased sensitivity for detecting disease activity/progression from 24% to 76% compared to standard radiological reading. This evidence shows PwMS’ interest, the digital care platform’s potential to improve the detection of (sub)clinical disease activity and care management, and the feasibility of linking different digital tools into one overarching MS care pathway.

Analysis of misses in imaging of head and neck pathology by attending neuroradiologists at a single tertiary academic medical centre

AIM

To analyse errors in head and neck (H&N) pathology made by attending neuroradiologists at a single tertiary-care centre.

MATERIALS AND METHODS

A neuroradiology quality assurance (QA) database of radiological errors was searched for attending physician errors in H&N pathology from 2014-2020. Data were limited to computed tomography (CT) and magnetic resonance imaging (MRI) reports. Data were collected on missed pathologies and study types. Misses were grouped into three categories: central neck (thyroid gland, aerodigestive tract), lateral neck (salivary glands, lymph nodes, soft tissues), and face/orbits (orbits, sinuses, masticator space).

RESULTS

During the study period, a total of 283,248 CT and MRI neuroradiology examinations were interpreted (all indications). Seventy-four H&N misses were identified comprising 85.1% perceptual and 14.9% interpretive errors. The distribution of errors was face/orbits (37.8%), central neck (36.5%), and lateral neck (25.7%). Clinically significant errors were found most commonly in the aerodigestive tract (21%), orbits (17.7%), masticator space, and parotid glands (14.5% each). The majority (67.6%) of the misses were detected on examinations that were not performed for a primary H&N indication; MRI brain was the most common examination (27%). Clearly malignant or potentially malignant masses accounted for 48.6% of all misses.

CONCLUSION

The majority of H&N misses were perceptual and were detected on examinations not performed for a H&N indication. Clearly malignant or potentially malignant masses represented half of all misses.

Secondary Interpretations of Diagnostic Imaging Examinations: Patient Liabilities and Out-of-Pocket Costs

BACKGROUND

Secondary interpretations of diagnostic imaging examinations are increasingly performed to improve care for complex patients. We sought to determine associated patient-billed liabilities and out-of-pocket payments and to identify patient and imaging study characteristics that correlate with higher patient bills and out-of-pocket payments.

METHODS

Data extracted for 7,740 secondary imaging interpretations performed across our large metropolitan health system over 25 months included total professional charges, insurance payments, patient-billed liabilities, and patient out-of-pocket payments. Multivariable linear regression analyses were performed to identify patient and imaging factors associated with higher patient bills and out-of-pocket payments.

RESULTS

Mean secondary interpretation professional charges, insurance payments, patient-billed liabilities, and patient out-of-pocket payments were $306.50, $108.02, $27.80, and $14.55, respectively. Patients received bills for 47.5% of services and made out-of-pocket payments for 17.1%. Patient-billed liabilities and out-of-pocket payments were higher for patients who were younger and uninsured and for secondary interpretations requested for patients seen in outpatient (versus inpatient) settings. Patient-billed liabilities and out-of-pocket payments were lower for patients who were Black (versus White) and had government-sponsored (versus commercial) insurance and for secondary interpretations performed during the second, third, or fourth (versus first) quarter of each calendar year.

CONCLUSION

Observed differences between patient-billed liabilities and out-of-pocket payments suggest that secondary interpretations of diagnostic imaging examinations can result in small but real patient financial burdens. Improved price transparency and enhanced patient communication about the value of secondary interpretations could reduce potential surprises when patients receive these bills.

Availability of a final abdominopelvic CT report before emergency department disposition: risk-adjusted outcomes in patients with abdominal pain

OBJECTIVE

To determine whether availability of a final radiologist report versus an experienced senior resident preliminary report prior to disposition affects major care outcomes in emergency department (ED) patient presenting with abdominal pain undergoing abdominopelvic CT.

MATERIALS AND METHODS

This single-institution, IRB-approved, HIPAA-compliant retrospective cohort study included 5019 ED patients with abdominal pain undergoing abdominopelvic CT from October 2015 to April 2019. Patients were categorized as being dispositioned after either an experienced senior resident preliminary report (i.e., overnight model) or the final attending radiologist interpretation (i.e., daytime model) of the CT was available. Multivariable regression models were built accounting for demographic data, clinical factors (vital signs, ED triage score, laboratory data), and disposition timing to analyze the impact on four important patient outcomes: inpatient admission (primary outcome), readmission (within 30 days), second operation within 30 days, and death.

RESULTS

In the setting of an available experienced senior resident preliminary report, timing of the final radiologist report (before vs. after disposition) was not a significant multivariable predictor of inpatient admission (p = 0.63), readmission within 30 days (p = 0.66), second operation within 30 days (p = 0.09), or death (p = 0.63). Unadjusted event rates for overnight vs daytime reports, respectively, were 37.2% vs. 38.0% (inpatient admission), 15.9% vs. 16.5% (30-day readmission), 0.65% vs. 0.3% (second operation within 30 days), and 0.85% vs. 1.3% (death).

CONCLUSION

Given the presence of an experienced senior resident preliminary report, availability of a final radiology report prior to ED disposition did not affect four major clinical care outcomes of patients with abdominal pain undergoing abdominopelvic CT.

Expert radiologist review at a hepatobiliary multidisciplinary tumor board: impact on patient management

PURPOSE

To identify the frequency, source, and management impact of discrepancies between the initial radiology report and expert reinterpretation occurring in the context of a hepatobiliary multidisciplinary tumor board (MTB).

METHODS

This retrospective study included 974 consecutive patients discussed at a weekly MTB at a large tertiary care academic medical center over a 2-year period. A single radiologist with dedicated hepatobiliary imaging expertise attended all conferences to review and discuss the relevant liver imaging and rated the concordance between original and re-reads based on RADPEER scoring criteria. Impact on management was based on the conference discussion and reflected changes in follow-up imaging, recommendations for biopsy/surgery, or liver transplant eligibility.

RESULTS

Image reinterpretation was discordant with the initial report in 19.9% (194/974) of cases (59.8%, 34.5%, 5.7% RADPEER 2/3/4 discrepancies, respectively). A change in LI-RADS category occurred in 59.8% of discrepancies. Most common causes of discordance included re-classification of a lesion as benign rather than malignant (16.0%) and missed tumor recurrence (13.9%). Impact on management occurred in 99.0% of discordant cases and included loco-regional therapy instead of follow-up imaging (19.1%), follow-up imaging instead of treatment (17.5%), and avoidance of biopsy (12.4%). 11.3% received OPTN exception scores due to the revised interpretation, and 8.8% were excluded from listing for orthotopic liver transplant.

CONCLUSION

Even in a sub-specialized abdominal imaging academic practice, expert radiologist review in the MTB setting identified discordant interpretations and impacted management in a substantial fraction of patients, potentially impacting transplant allocation. The findings may impact how abdominal imaging sections best staff advanced MTBs.

Body MRI Subspecialty Reinterpretations at a Tertiary Care Center: Discrepancy Rates and Error Types

OBJECTIVE. Radiology departments in tertiary care centers are frequently asked to perform secondary interpretations of imaging studies, particularly when a patient is transferred from a community hospital. Discrepancy rates in radiology vary widely, with low rates reported for preliminary resident reports that are overread by attending radiologists (2-6%) and higher rates (up to 56%) for secondary interpretations. Abdominal and pelvic imaging and cross-sectional imaging have the highest discrepancy rates. The purpose of our study was to determine the discrepancy rate and the most common reasons for discrepancies between abdominal and pelvic MRI reports obtained from outside institutions and secondary interpretations of these reports by a fellowship-trained radiologist at a tertiary care center. MATERIALS AND METHODS. We retrospectively identified 395 secondary MRI reports from January 2015 to December 2018 that were labeled as body MRI examinations at a tertiary care center. Thirty-eight cases were excluded for various reasons, including incorrect categorization or lack of outside report. We reviewed the outside reports, compared them with the secondary interpretations, and categorized the cases as discrepancy or no discrepancy. The discrepancies were subdivided into the most likely reason for the error using previously published categories; these categories were also divided into perceptive and cognitive errors. RESULTS. Of the 357 included cases, 246 (68.9%) had at least one discrepancy. The most common reason for error was faulty reasoning (34.3%), which is a cognitive error characterized by misidentifying an abnormality. Satisfaction of search, which is a perceptive error, was the most common reason for second discrepancies (15.0%). CONCLUSION. Secondary interpretations of body MR images at a tertiary care center identify a high rate of discrepancies, with cognitive error types predominating.

National Trends in Oncologic Diagnostic Imaging

OBJECTIVE

To characterize national trends in oncologic imaging (OI) utilization.

METHODS

This retrospective cross-sectional study used 2004 and 2016 CMS 5% Carrier Claims Research Identifiable Files. Radiologist-performed, primary noninvasive diagnostic imaging examinations were identified from billed Current Procedural Terminology codes; CT, MRI, and PET/CT examinations were categorized as "advanced" imaging. OI examinations were identified from imaging claims' primary International Classification of Diseases-9 and International Classification of Diseases-10 codes. Imaging services were stratified by academic practice status and place of service. State-level correlations of oncologic advanced imaging utilization (examinations per 1,000 beneficiaries) with cancer prevalence and radiologist supply were assessed by Spearman correlation coefficient.

RESULTS

The national Medicare sample included 5,051,095 diagnostic imaging examinations (1,220,224 of them advanced) in 2004 and 5,023,115 diagnostic imaging examinations (1,504,608 of them advanced) in 2016. In 2004 and 2016, OI represented 4.3% and 3.9%, respectively, of all imaging versus 10.8% and 9.5%, respectively, of advanced imaging. The percentage of advanced OI done in academic practices rose from 18.8% in 2004 to 34.1% in 2016, leaving 65.9% outside academia. In 2016, 58.0% of advanced OI was performed in the hospital outpatient setting and 23.9% in the physician office setting. In 2016, state-level oncologic advanced imaging utilization correlated with state-level radiologist supply (r = +0.489, P < .001) but not with state-level cancer prevalence (r = -0.139, P = .329).

DISCUSSION

OI usage varied between practice settings. Although the percentage of advanced OI done in academic settings nearly doubled from 2004 to 2016, the majority remained in nonacademic practices. State-level oncologic advanced imaging utilization correlated with radiologist supply but not cancer prevalence.

Second thoughts: Emergency clinicians see value in secondary interpretations

OBJECTIVE

Secondary interpretations of imaging studies performed at another facility are increasingly common in radiology, particularly emergency radiology. While data suggests there are often discrepancies found between original and secondary reports, the benefit from the clinician perspective is unclear.

METHODS AND MATERIALS

An anonymous electronic survey on secondary interpretations was circulated to 58 attending adult emergency physicians and trauma surgeons at a Level I trauma center from March 2018 to April 2018. Chi-squared testing was used for statistical analysis.

RESULTS

80.8% of respondents requested secondary interpretations either "always" or "most of the time." Over half of the respondents cited trust in the house radiologist interpretation as the primary reason for secondary interpretation requests. 92.3% and 84.6% of respondents felt that the ability to obtain second interpretations improves patient care and facilitates disposition, respectively. 88.5% of respondents reported reduced imaging utilization due to secondary reads. When presented with conflicting interpretations, all trauma surgeons would rely on the in-house interpretation, whereas 50% of the emergency physicians would pursue further imaging (p < 0.05). 96.2% of respondents were uncertain about insurance coverage of secondary interpretations, but 73.1% would continue to order them, regardless.

CONCLUSION

Secondary reads were heavily utilized, felt to influence patient care, reduced additional imaging and aided in disposition, suggesting clinical benefit. When presented with conflicting reports, trauma surgeons would rely on the in-house interpretation whereas emergency physicians more often opted to pursue additional imaging. Most respondents would still request secondary interpretations despite being unaware of insurance coverage for these interpretations.

Siamese neural networks for continuous disease severity evaluation and change detection in medical imaging

Using medical images to evaluate disease severity and change over time is a routine and important task in clinical decision making. Grading systems are often used, but are unreliable as domain experts disagree on disease severity category thresholds. These discrete categories also do not reflect the underlying continuous spectrum of disease severity. To address these issues, we developed a convolutional Siamese neural network approach to evaluate disease severity at single time points and change between longitudinal patient visits on a continuous spectrum. We demonstrate this in two medical imaging domains: retinopathy of prematurity (ROP) in retinal photographs and osteoarthritis in knee radiographs. Our patient cohorts consist of 4861 images from 870 patients in the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) cohort study and 10,012 images from 3021 patients in the Multicenter Osteoarthritis Study (MOST), both of which feature longitudinal imaging data. Multiple expert clinician raters ranked 100 retinal images and 100 knee radiographs from excluded test sets for severity of ROP and osteoarthritis, respectively. The Siamese neural network output for each image in comparison to a pool of normal reference images correlates with disease severity rank (ρ = 0.87 for ROP and ρ = 0.89 for osteoarthritis), both within and between the clinical grading categories. Thus, this output can represent the continuous spectrum of disease severity at any single time point. The difference in these outputs can be used to show change over time. Alternatively, paired images from the same patient at two time points can be directly compared using the Siamese neural network, resulting in an additional continuous measure of change between images. Importantly, our approach does not require manual localization of the pathology of interest and requires only a binary label for training (same versus different). The location of disease and site of change detected by the algorithm can be visualized using an occlusion sensitivity map-based approach. For a longitudinal binary change detection task, our Siamese neural networks achieve test set receiving operator characteristic area under the curves (AUCs) of up to 0.90 in evaluating ROP or knee osteoarthritis change, depending on the change detection strategy. The overall performance on this binary task is similar compared to a conventional convolutional deep-neural network trained for multi-class classification. Our results demonstrate that convolutional Siamese neural networks can be a powerful tool for evaluating the continuous spectrum of disease severity and change in medical imaging.

Prevalence and clinical significance of discordant LI-RADS® observations on multiphase contrast-enhanced MRI in patients with cirrhosis

PURPOSE

To determine the prevalence and clinical significance of discordant LI-RADS^® (Liver Imaging Reporting and Data System) liver observations on multiphase contrast-enhanced (MCE) magnetic resonance imaging (MRI) in patients with cirrhosis.

METHODS

This cross-sectional study included 93 cirrhosis patients who underwent 1.5 or 3 T MCE MRI for evaluation of hepatocellular carcinoma (HCC). Two abdominal radiologists independently reviewed T1-, T2-, diffusion-weighted unenhanced images as well as MCE T1-weighted fat-suppressed images and reported liver observations using LI-RADS^®. Concordance were recorded for detection (co-detected by both radiologists or not), size category (< 10; 10-19; ≥ 20 mm), and LI-RADS^® category assignment as reportable (LR-3/4/5/M) and actionable (LR-4/5/M). The overall concordance (i.e., concordant in detection, size, and LR-category) was calculated with 95% confidence interval [CI], and separately for detection, size, and LR-category. Clinical significance of discordance was assessed as impact on follow-up imaging, referral for biopsy, liver transplant eligibility, or treatment modality.

RESULTS

Reportable and actionable observations were overall concordant between two radiologists only in 32.3% [24.6, 41.0] and 40.1% [29.5, 51.5] of cases, respectively. Poor overall concordance was related to detection concordance of 52.0% [44.3, 59.5] and 62.5% [52.3, 71.8], as well as LR-category concordance of 73.7% [61.6, 83.1] and 70.9% [57.3, 81.6], for reportable and actionable observations, respectively. Discordant LI-RADS^® observations would have impacted clinical management in 30 subjects (43.5%), most (66.7%) of whom were due to discordant detection.

CONCLUSION

Discordant MRI LI-RADS^® observations are common in patients with cirrhosis and may have potential implications for patient management.

Chest Radiograph Interpretation with Deep Learning Models: Assessment with Radiologist-adjudicated Reference Standards and Population-adjusted Evaluation

BackgroundDeep learning has the potential to augment the use of chest radiography in clinical radiology, but challenges include poor generalizability, spectrum bias, and difficulty comparing across studies.PurposeTo develop and evaluate deep learning models for chest radiograph interpretation by using radiologist-adjudicated reference standards.Materials and MethodsDeep learning models were developed to detect four findings (pneumothorax, opacity, nodule or mass, and fracture) on frontal chest radiographs. This retrospective study used two data sets. Data set 1 (DS1) consisted of 759 611 images from a multicity hospital network and ChestX-ray14 is a publicly available data set with 112 120 images. Natural language processing and expert review of a subset of images provided labels for 657 954 training images. Test sets consisted of 1818 and 1962 images from DS1 and ChestX-ray14, respectively. Reference standards were defined by radiologist-adjudicated image review. Performance was evaluated by area under the receiver operating characteristic curve analysis, sensitivity, specificity, and positive predictive value. Four radiologists reviewed test set images for performance comparison. Inverse probability weighting was applied to DS1 to account for positive radiograph enrichment and estimate population-level performance.ResultsIn DS1, population-adjusted areas under the receiver operating characteristic curve for pneumothorax, nodule or mass, airspace opacity, and fracture were, respectively, 0.95 (95% confidence interval [CI]: 0.91, 0.99), 0.72 (95% CI: 0.66, 0.77), 0.91 (95% CI: 0.88, 0.93), and 0.86 (95% CI: 0.79, 0.92). With ChestX-ray14, areas under the receiver operating characteristic curve were 0.94 (95% CI: 0.93, 0.96), 0.91 (95% CI: 0.89, 0.93), 0.94 (95% CI: 0.93, 0.95), and 0.81 (95% CI: 0.75, 0.86), respectively.ConclusionExpert-level models for detecting clinically relevant chest radiograph findings were developed for this study by using adjudicated reference standards and with population-level performance estimation. Radiologist-adjudicated labels for 2412 ChestX-ray14 validation set images and 1962 test set images are provided.© RSNA, 2019Online supplemental material is available for this article.See also the editorial by Chang in this issue.

Ensuring Appropriateness of Pediatric Second Opinion Consultations

PURPOSE

We sought to evaluate discrepancy rates between outside interpretations, radiology trainee preliminary reports, and subspecialist attending final interpretations for pediatric second opinion consultations on plain film and computed tomography imaging and to evaluate the impact of a process improvement for second opinion consultations.

METHODS

Of a total of 572 requests for second opinion consultations during 1-year preintervention period, we utilized RADPEER to score concurrence of 158 requests which occurred overnight and included outside radiologist interpretations and resident preliminary reports. In consultation with clinician committees, we developed new guidelines for requesting second opinion consultations. We evaluated the impact on the number of consultations for the 1-year period following implementation of this process improvement.

RESULTS

There was concurrence between the outside interpretation and pediatric subspecialist second opinion in 146 of 158 cases (92%). There was concurrence between the radiology resident and pediatric subspecialist second opinion in 145 of 158 cases (92%). During the 1-year period following our process improvement implementation, the total number of second opinion consultations decreased to 185 (from 572, a decrease of 68%) and the number of overnight requests for resident preliminary reports decreased to 11 (from 158, a decrease of 93%).

CONCLUSIONS

There was a high degree of concurrence between interpretations provided by outside radiologists, overnight radiology residents, and attending pediatric radiologists at our institution. Analyzing institutional-specific discrepancy rates is a valuable first step in partnering with clinicians to develop appropriate guidelines for second opinion consultations.