Double reading rates and quality assurance practices in Norwegian hospital radiology departments: two parallel national surveys

Artificial intelligence-assisted double reading of chest radiographs to detect clinically relevant missed findings: a two-centre evaluation

OBJECTIVES

To evaluate an artificial intelligence (AI)-assisted double reading system for detecting clinically relevant missed findings on routinely reported chest radiographs.

METHODS

A retrospective study was performed in two institutions, a secondary care hospital and tertiary referral oncology centre. Commercially available AI software performed a comparative analysis of chest radiographs and radiologists' authorised reports using a deep learning and natural language processing algorithm, respectively. The AI-detected discrepant findings between images and reports were assessed for clinical relevance by an external radiologist, as part of the commercial service provided by the AI vendor. The selected missed findings were subsequently returned to the institution's radiologist for final review.

RESULTS

In total, 25,104 chest radiographs of 21,039 patients (mean age 61.1 years ± 16.2 [SD]; 10,436 men) were included. The AI software detected discrepancies between imaging and reports in 21.1% (5289 of 25,104). After review by the external radiologist, 0.9% (47 of 5289) of cases were deemed to contain clinically relevant missed findings. The institution's radiologists confirmed 35 of 47 missed findings (74.5%) as clinically relevant (0.1% of all cases). Missed findings consisted of lung nodules (71.4%, 25 of 35), pneumothoraces (17.1%, 6 of 35) and consolidations (11.4%, 4 of 35).

CONCLUSION

The AI-assisted double reading system was able to identify missed findings on chest radiographs after report authorisation. The approach required an external radiologist to review the AI-detected discrepancies. The number of clinically relevant missed findings by radiologists was very low.

CLINICAL RELEVANCE STATEMENT

The AI-assisted double reader workflow was shown to detect diagnostic errors and could be applied as a quality assurance tool. Although clinically relevant missed findings were rare, there is potential impact given the common use of chest radiography.

KEY POINTS

• A commercially available double reading system supported by artificial intelligence was evaluated to detect reporting errors in chest radiographs (n=25,104) from two institutions. • Clinically relevant missed findings were found in 0.1% of chest radiographs and consisted of unreported lung nodules, pneumothoraces and consolidations. • Applying AI software as a secondary reader after report authorisation can assist in reducing diagnostic errors without interrupting the radiologist's reading workflow. However, the number of AI-detected discrepancies was considerable and required review by a radiologist to assess their relevance.

Neuroradiology diagnostic errors at a tertiary academic centre: effect of participation in tumour boards and physician experience

AIM

To quantify and correlate the diagnostic error rates in radiological interpretation with the experience of the attending neuroradiologist at a tertiary academic medical centre.

MATERIALS AND METHODS

The institution's Neuroradiology Quality Assurance Database of diagnostic errors was searched for misses from 2014-2020. Attendance at Head and Neck (H&N), Brain, and Paediatric Neuroradiology (PN) tumour boards (TB) as the presenting radiologist was recorded. Number of post-fellowship years of clinical practice (CPY) and frequency of TB attendance were considered separate metrics of a radiologist's experience. Radiological errors were categorised as Total, H&N, Skull Base (SKB), Brain, or PN diagnostic errors. Diagnostic error rates per attending neuroradiologist within each category were correlated with the frequency of TB participation and CPY using Spearman's rank correlation coefficients.

RESULTS

A total 607 examinations contained a diagnostic error. Spearman's rank correlation coefficients between Total TB participation and Total, H&N, SKB, Brain error rates were: -0.89 (p=0.0002); -0.81 (p=0.002); -0.66 (p=0.03); -0.82 (p=0.002); respectively. Spearman's rank correlation coefficients between CPY and Total, H&N, SKB, Brain and PN error rates were: 0.05 (p=0.88); 0.08 (p=0.82); 0.28 (p=0.41); -0.10 (p=0.77); -0.16 (p=0.63), respectively. Spearman's rank correlation coefficients between H&N TB and H&N, SKB error rates; and between Brain TB attendance and Brain error rates were statistically significant (p<0.05).

CONCLUSION

The present study shows a strong correlation between high TB participation rates and low diagnostic error rates. The number of years in practice did not appear to influence error rate.

Analysis of misses in imaging of head and neck pathology by attending neuroradiologists at a single tertiary academic medical centre

AIM

To analyse errors in head and neck (H&N) pathology made by attending neuroradiologists at a single tertiary-care centre.

MATERIALS AND METHODS

A neuroradiology quality assurance (QA) database of radiological errors was searched for attending physician errors in H&N pathology from 2014-2020. Data were limited to computed tomography (CT) and magnetic resonance imaging (MRI) reports. Data were collected on missed pathologies and study types. Misses were grouped into three categories: central neck (thyroid gland, aerodigestive tract), lateral neck (salivary glands, lymph nodes, soft tissues), and face/orbits (orbits, sinuses, masticator space).

RESULTS

During the study period, a total of 283,248 CT and MRI neuroradiology examinations were interpreted (all indications). Seventy-four H&N misses were identified comprising 85.1% perceptual and 14.9% interpretive errors. The distribution of errors was face/orbits (37.8%), central neck (36.5%), and lateral neck (25.7%). Clinically significant errors were found most commonly in the aerodigestive tract (21%), orbits (17.7%), masticator space, and parotid glands (14.5% each). The majority (67.6%) of the misses were detected on examinations that were not performed for a primary H&N indication; MRI brain was the most common examination (27%). Clearly malignant or potentially malignant masses accounted for 48.6% of all misses.

CONCLUSION

The majority of H&N misses were perceptual and were detected on examinations not performed for a H&N indication. Clearly malignant or potentially malignant masses represented half of all misses.

An International Survey of Quality and Safety Programs in Radiology

PURPOSE

The aim of this study was to determine the status of radiology quality improvement programs in a variety of selected nations worldwide.

METHODS

A survey was developed by select members of the International Economics Committee of the American College of Radiology on quality programs and was distributed to committee members. Members responded on behalf of their country. The 51-question survey asked about 12 different quality initiatives which were grouped into 4 themes: departments, users, equipment, and outcomes. Respondents reported whether a designated type of quality initiative was used in their country and answered subsequent questions further characterizing it.

RESULTS

The response rate was 100% and represented Australia, Canada, China, England, France, Germany, India, Israel, Japan, the Netherlands, Russia, and the United States. The most frequently reported quality initiatives were imaging appropriateness (91.7%) and disease registries (91.7%), followed by key performance indicators (83.3%) and morbidity and mortality rounds (83.3%). Peer review, equipment accreditation, radiation dose monitoring, and structured reporting were reported by 75.0% of respondents, followed by 58.3% of respondents for quality audits and critical incident reporting. The least frequently reported initiatives included Lean/Kaizen exercises and physician performance assessments, implemented by 25.0% of respondents.

CONCLUSION

There is considerable diversity in the quality programs used throughout the world, despite some influence by national and international organizations, from whom further guidance could increase uniformity and optimize patient care in radiology.

Utility of Artificial Intelligence Tool as a Prospective Radiology Peer Reviewer - Detection of Unreported Intracranial Hemorrhage

RATIONALE AND OBJECTIVES

Misdiagnosis of intracranial hemorrhage (ICH) can adversely impact patient outcomes. The increasing workload on the radiologists may increase the chance of error and compromise the quality of care provided by the radiologists.

MATERIALS AND METHODS

We used an FDA approved artificial intelligence (AI) solution based on a convolutional neural network to assess the prevalence of ICH in scans, which were reported as negative for ICH. We retrospectively applied the AI solution to all consecutive noncontrast computed tomography (CT) head scans performed at eight imaging sites affiliated to our institution.

RESULTS

In the 6565 noncontrast CT head scans, which met the inclusion criteria, 5585 scans were reported to have no ICH ("negative-by-report" cases). We applied AI solution to these "negative-by-report" cases. AI solution suggested there were ICH in 28 of these scans ("negative-by-report" and "positive-by-AI solution"). After consensus review by three neuroradiologists, 16 of these scans were found to have ICH, which was not reported (missed diagnosis by radiologists), with a false-negative rate of radiologists for ICH detection at 1.6%. Most commonly missed ICH was overlying the cerebral convexity and in the parafalcine regions.

CONCLUSION

Our study demonstrates that an AI solution can help radiologists to diagnose ICH and thus decrease the error rate. AI solution can serve as a prospective peer review tool for non-contrast head CT scans to identify ICH and thus minimize false negatives.

Time Requirement and Feasibility of a Systematic Quality Peer Review of Reporting in Radiology

OBJECTIVE

 To estimate the human resources required for a retrospective quality review of different percentages of all routine diagnostic procedures in the Department of Radiology at Bern University Hospital, Switzerland.

MATERIALS AND METHODS

 Three board-certified radiologists retrospectively evaluated the quality of the radiological reports of a total of 150 examinations (5 different examination types: abdominal CT, chest CT, mammography, conventional X-ray images and abdominal MRI). Each report was assigned a RADPEER score of 1 to 3 (score 1: concur with previous interpretation; score 2: discrepancy in interpretation/not ordinarily expected to be made; score 3: discrepancy in interpretation/should be made most of the time). The time (in seconds, s) required for each review was documented and compared. A sensitivity analysis was conducted to calculate the total workload for reviewing different percentages of the total annual reporting volume of the clinic.

RESULTS

 Among the total of 450 reviews analyzed, 91.1 % (410/450) were assigned a score of 1 and 8.9 % (40/450) were assigned scores of 2 or 3. The average time (in seconds) required for a peer review was 60.4 s (min. 5 s, max. 245 s). The reviewer with the greatest clinical experience needed significantly less time for reviewing the reports than the two reviewers with less clinical expertise (p < 0.05). Average review times were longer for discrepant ratings with a score of 2 or 3 (p < 0.05). The total time requirement calculated for reviewing all 5 types of examination for one year would be more than 1200 working hours.

CONCLUSION

 A retrospective peer review of reports of radiological examinations using the RADPEER system requires considerable human resources. However, to improve quality, it seems feasible to peer review at least a portion of the total yearly reporting volume.

KEY POINTS

  · A systematic retrospective assessment of the content of radiological reports using the RADPEER system involves high personnel costs.. · The retrospective assessment of all reports of a clinic or practice seems unrealistic due to the lack of highly specialized personnel.. · At least part of all reports should be reviewed with the aim of improving the quality of reports..

CITATION FORMAT

· Maurer MH, Brönnimann M, Schroeder C et al. Time Requirement and Feasibility of a Systematic Quality Peer Review of Reporting in Radiology. Fortschr Röntgenstr 2021; 193: 160 - 167.

Added value of double reading in diagnostic radiology,a systematic review

Objectives

Double reading in diagnostic radiology can find discrepancies in the original report, but a systematic program of double reading is resource consuming. There are conflicting opinions on the value of double reading. The purpose of the current study was to perform a systematic review on the value of double reading.

Methods

A systematic review was performed to find studies calculating the rate of misses and overcalls with the aim of establishing the added value of double reading by human observers.

Results

The literature search resulted in 1610 hits. After abstract and full-text reading, 46 articles were selected for analysis. The rate of discrepancy varied from 0.4 to 22% depending on study setting. Double reading by a sub-specialist, in general, led to high rates of changed reports.

Conclusions

The systematic review found rather low discrepancy rates. The benefit of double reading must be balanced by the considerable number of working hours a systematic double-reading scheme requires. A more profitable scheme might be to use systematic double reading for selected, high-risk examination types. A second conclusion is that there seems to be a value of sub-specialisation for increased report quality. A consequent implementation of this would have far-reaching organisational effects.

Key Points

• In double reading, two or more radiologists read the same images.

• A systematic literature review was performed.

• The discrepancy rates varied from 0.4 to 22% in various studies.

• Double reading by sub-specialists found high discrepancy rates.

Electronic supplementary material

The online version of this article (10.1007/s13244-018-0599-0) contains supplementary material, which is available to authorised users.

Radiologist-initiated double reading of abdominal CT: retrospective analysis of the clinical importance of changes to radiology reports

Background

Misinterpretation of radiological examinations is an important contributing factor to diagnostic errors. Consultant radiologists in Norwegian hospitals frequently request second reads by colleagues in real time. Our objective was to estimate the frequency of clinically important changes to radiology reports produced by these prospectively obtained double readings.

Methods

We retrospectively compared the preliminary and final reports from 1071 consecutive double-read abdominal CT examinations of surgical patients at five public hospitals in Norway. Experienced gastrointestinal surgeons rated the clinical importance of changes from the preliminary to final report. The severity of the radiological findings in clinically important changes was classified as increased, unchanged or decreased.

Results

Changes were classified as clinically important in 146 of 1071 reports (14%). Changes to 3 reports (0.3%) were critical (demanding immediate action), 35 (3%) were major (implying a change in treatment) and 108 (10%) were intermediate (requiring further investigations). The severity of the radiological findings was increased in 118 (81%) of the clinically important changes. Important changes were made less frequently when abdominal radiologists were first readers, more frequently when they were second readers, and more frequently to urgent examinations.

Conclusion

A 14% rate of clinically important changes made during double reading may justify quality assurance of radiological interpretation. Using expert second readers and a targeted selection of urgent cases and radiologists reading outside their specialty may increase the yield of discrepant cases.