Interrater agreement in the evaluation of discrepant imaging findings with the Radpeer system

Event-Based Learning and Improvement: Radiology's Move From Peer Review to Peer Learning

Over the past 15 years, the radiology community has made great progress moving from a system of score-based peer review to one of peer learning. Much has been learned along the way. In peer learning, cases in which learning opportunities are identified are reviewed solely for the purpose of fostering learning and improvement. This article defines peer learning and peer review and emphasizes the difference; looks back at the 20-year history of score-based peer review and transition to peer learning; outlines the problems with score-based peer review and the key elements of peer learning; discusses the current state of peer learning; and outlines future challenges and opportunities.

Improving the Reliability of Peer Review Without a Gold Standard

Peer review plays a crucial role in accreditation and credentialing processes as it can identify outliers and foster a peer learning approach, facilitating error analysis and knowledge sharing. However, traditional peer review methods may fall short in effectively addressing the interpretive variability among reviewing and primary reading radiologists, hindering scalability and effectiveness. Reducing this variability is key to enhancing the reliability of results and instilling confidence in the review process. In this paper, we propose a novel statistical approach called "Bayesian Inter-Reviewer Agreement Rate" (BIRAR) that integrates radiologist variability. By doing so, BIRAR aims to enhance the accuracy and consistency of peer review assessments, providing physicians involved in quality improvement and peer learning programs with valuable and reliable insights. A computer simulation was designed to assign predefined interpretive error rates to hypothetical interpreting and peer-reviewing radiologists. The Monte Carlo simulation then sampled (100 samples per experiment) the data that would be generated by peer reviews. The performances of BIRAR and four other peer review methods for measuring interpretive error rates were then evaluated, including a method that uses a gold standard diagnosis. Application of the BIRAR method resulted in 93% and 79% higher relative accuracy and 43% and 66% lower relative variability, compared to "Single/Standard" and "Majority Panel" peer review methods, respectively. Accuracy was defined by the median difference of Monte Carlo simulations between measured and pre-defined "actual" interpretive error rates. Variability was defined by the 95% CI around the median difference of Monte Carlo simulations between measured and pre-defined "actual" interpretive error rates. BIRAR is a practical and scalable peer review method that produces more accurate and less variable assessments of interpretive quality by accounting for variability within the group's radiologists, implicitly applying a standard derived from the level of consensus within the group across various types of interpretive findings.

Transition from Peer Review to Peer Learning: Lessons Learned

Landmark publications, such as To Err is Human, confronted the healthcare community with the egregious toll medical errors played in both patient safety and overall healthcare costs. This heralded a paradigm shift and a call for action by professional organizations to enact methods to ensure physician competency and quality assurance. The American College of Radiology similarly convened a task force to discuss these concerns and how best to address quality assurance in radiology practice, leading to the development of RADPEER, a score-based peer review system. However, critics were quick to point out the deficiencies of this model, highlighting it as punitive and a poor evaluator of physician performance. The recognized deficiencies in score-based peer review prompted the pursuit of an alternate model that would instead emphasize learning and improvement. Peer learning was proposed and highlighted the necessity of an inclusive and collaborative environment where colleagues could discuss case errors as learning opportunities without fear of punitive consequence. This paper explores peer learning, its benefits and challenges, as well as how to identify specific learning opportunities by utilizing case examples.

Simple changes to the reporting environment produce a large reduction in the frequency of interruptions to the reporting radiologist: an observational study

Background

Interruptions are a cause of discrepancy, errors, and potential safety incidents in radiology. The sources of radiological error are multifactorial and strategies to reduce error should include measures to reduce interruptions.

Purpose

To evaluate the effect of simple changes in the reporting environment on the frequency of interruptions to the reporting radiologist of a hospital radiology department.

Material and Methods

A prospective observational study was carried out. The number and type of potentially disruptive events (PDEs) to the radiologist reporting inpatient computed tomography (CT) scans were recorded during 20 separate 1-h observation periods during both pre- and post-intervention phases. The interventions were (i) relocation of the radiologist to a private, quiet room, and (ii) initial vetting of clinician enquiries via a separate duty radiologist

Results

After the intervention there was an 82% reduction in the number of frank interruptions (PDEs that require the radiologist to abandon the reporting task) from a median 6 events per hour to 1 (95% confidence interval [CI] = 4–6; P < 0.00001). The overall number of PDEs was reduced by 56% from a median 11 events per hour to 5 (95% CI = 4.5–11: P < 0.00001).

Conclusion

Relocation of inpatient CT reporting to a private, quiet room, coupled with vetting of clinician enquiries via the duty radiologist, resulted in a large reduction in the frequency of interruptions, a frequently cited avoidable source of radiological error.

Night Radiology Coverage for Trauma: Residents, Teleradiology, or Both?

BACKGROUND

Overnight radiology coverage for trauma patients is often addressed with a combination of on-call radiology residents (RR) and a teleradiology service; however, the accuracy of these 2 readers has not been studied for trauma. We aimed to compare the accuracy of RR versus teleradiologist interpretations of CT scans for trauma patients.

STUDY DESIGN

A retrospective analysis (March 2019 through May 2020) of trauma patients presenting to a single American College of Surgeons Level I trauma center was performed. Patients whose CT scans were performed between 10 pm to 8 am were included, because their scans were interpreted by both a RR and teleradiologist. Interpretations were compared with the final attending faculty radiologist's interpretation and graded for accuracy based on the RADPEER scoring system. Discrepancies were characterized as traumatic injury or incidental findings and missed findings or overcalls. Turnaround time was also compared.

RESULTS

A total of 1,053 patients and 8,226 interpretations were included. Compared with teleradiologists, RR had a lower discrepancy (7.7% vs 9.0%, p = 0.026) and major discrepancy rate (3.8% vs 5.2%, p = 0.003). Among major discrepancies, RR had a lower rate of traumatic injury discrepancies (3.2% vs 4.4%, p = 0.004) and missed findings (3.4% vs 5.1%, p < 0.001), but a higher rate of overcalls (0.5% vs 0.1%, p < 0.001) compared with teleradiologists. The mean turnaround time was shorter for RR (51.3 vs 78.8 minutes, p < 0.001). The combination of both RR and teleradiologist interpretations had a lower overall discrepancy rate than RR (5.0% vs 7.7%, p < 0.001).

CONCLUSIONS

This study identified lower discrepancy rates and a faster turnaround time by RR compared with teleradiologists for trauma CT studies. The combination of both interpreters had an even lower discrepancy rate, suggesting this combination is optimal when an in-house attending radiologist is not available.

Assessment of a Large-Scale Peer Learning Program's Value by Manual Review of Case Submissions

OBJECTIVE

Prior studies used submission numbers or report addendum rates to measure peer learning programs' (PLP) impact. We assessed the educational value of a PLP by manually reviewing cases submitted to identify factors correlating with meaningful learning opportunities (MLOs).

METHODS

This institutional review board-exempted, retrospective study was performed in a large academic radiology department generating >800,000 reports annually. A PLP facilitating radiologist-to-radiologist feedback was implemented May 1, 2017, with subsequent pay-for-performance initiatives encouraging increasing submissions, >18,000 by 2019. Two radiologists blinded to submitter and receiver identity categorized 336 randomly selected submissions as a MLO, not meaningful, or equivocal, resolving disagreements in consensus review. Primary outcome was proportion of MLOs. Secondary outcomes included percent engagement by subspecialty clinical division and comparing MLO and report addendum rates via Fisher's exact tests. We assessed association between peer learning category, pay-for-performance interventions, and subspecialty division with MLOs using logistic regression.

RESULTS

Of 336 PLP submissions, 65.2% (219 of 336) were categorized as meaningful, 27.4% (92 of 336) not meaningful, and 7.4% (25 of 336) equivocal, with substantial reviewer agreement (86.0% [289 of 336], κ = 0.71, 95% confidence interval 0.64-0.78). MLO rate (65.2% [219 of 336]) was five times higher than addendum rate (12.9% [43 of 333]) for the cohort. MLO proportion (adjusted odds ratios 0.05-1.09) and percent engagement (0.5%-3.6%) varied between subspecialty divisions, some submitting significantly fewer MLOs (P < .01). MLO proportion did not vary between peer learning categories.

CONCLUSION

Educational value of a large-scale PLP, estimated through manual review of case submissions, is likely a more accurate measure of program impact. Incentives to enhance PLP use did not diminish the program's educational value.

Peer learning in emergency radiology: effects on learning, error identification, and radiologist experience

PURPOSE

We established and evaluated a peer learning program in an emergency radiology (ER) division. Peer learning is an alternative to peer review focusing on non-punitive error reporting to mitigate consequences of inevitable human error. The central component is the peer learning conference, where cases are presented, key teaching points are discussed, and process improvement ideas are solicited.

METHODS

We established a prior imaging-based case identification system and a bimonthly remote videoconference where ER faculty discuss 5-15 cases selected for learning or process improvement opportunities. Case identification and conference characteristics were captured. A survey focused on learning and performance outcomes was administered to faculty initially and showed improved scores after 6 months.

RESULTS

Cases selected for conference favored perception errors (46%), with great calls (17%) and process improvement (15%) the next most common categories. A variety of anatomical regions were represented, with abdominal (35%) and musculoskeletal (29%) most common. Error detection was improved over peer review. All participants find the system easy to use and prefer peer learning to peer review for learning and process improvement.

CONCLUSION

A peer learning program can be successfully implemented within a busy academic emergency radiology division, as evidenced by increasing buy-in and engagement scores over time. When tied to a departmental peer learning infrastructure, interdisciplinary expertise and robust case identification can be leveraged to increase learning opportunities.

It's not about the errors, it's about the learning: How the Royal College of Radiologists has developed a Radiology Events and Learning process in the United Kingdom

The Royal College of Radiologists (RCR) is based in the United Kingdom but is a global organisation with members and fellows worldwide. In this invited article, the chair of the RCR Radiology Events and Learning (REAL) panel recounts his experience in looking at radiological errors. He starts with his personal work auditing his own mistakes as a junior consultant, describes what he learned in his departmental role in a large teaching hospital running a Radiology Events and Learning Meeting (REALM) and gives an overview of some of the work done over the last two decades by the RCR. This includes publishing national guidelines which set standards for running a REALM, setting up the REAL panel which produces a quarterly newsletter of cases from RCR members, and running an annual conference to share information with local radiology departments around the country. A review of the literature describing the drivers for this work and looking at the parallels with industry lies alongside the practical tips he found useful which he hopes would be helpful to anyone setting up their own departmental errors or discrepancy meeting.

CAR Peer Learning Guide

Peer learning is a quality initiative used to identify potential areas of practice improvement, both on a patient level and on a systemic level. Opportunities for peer learning include review of prior imaging studies, evaluation of cases from multidisciplinary case conferences, and review of radiology trainees' call cases. Peer learning is non-punitive and focuses on promoting life-long learning. It seeks to identify and disseminate learning opportunities and areas for systems improvement compared to traditional peer review. Learning opportunities arise from peer learning through both individual communication of cases reviewed for routine work, as well as through anonymous presentation of aggregate cases in an educational format. In conjunction with other tools such as root cause analysis, peer learning can be used to guide future practice improvement opportunities. This guide provides definitions of terms and a synthetic evidence review regarding peer review and peer learning, as well as medicolegal and jurisdictional considerations. Important aspects of what makes an effective peer learning program and best practices for implementing such a program are presented. The guide is intended to be a living document that will be updated regularly as new data emerges and peer learning continues to evolve in radiology practices.

Structured reporting of computed tomography in the staging of colon cancer: a Delphi consensus proposal

BACKGROUND

Structured reporting (SR) in radiology is becoming increasingly necessary and has been recognized recently by major scientific societies. This study aims to build structured CT-based reports in colon cancer during the staging phase in order to improve communication between the radiologist, members of multidisciplinary teams and patients.

MATERIALS AND METHODS

A panel of expert radiologists, members of the Italian Society of Medical and Interventional Radiology, was established. A modified Delphi process was used to develop the SR and to assess a level of agreement for all report sections. Cronbach's alpha (Cα) correlation coefficient was used to assess internal consistency for each section and to measure quality analysis according to the average inter-item correlation.

RESULTS

The final SR version was built by including n = 18 items in the "Patient Clinical Data" section, n = 7 items in the "Clinical Evaluation" section, n = 9 items in the "Imaging Protocol" section and n = 29 items in the "Report" section. Overall, 63 items were included in the final version of the SR. Both in the first and second round, all sections received a higher than good rating: a mean value of 4.6 and range 3.6-4.9 in the first round; a mean value of 5.0 and range 4.9-5 in the second round. In the first round, Cronbach's alpha (Cα) correlation coefficient was a questionable 0.61. In the first round, the overall mean score of the experts and the sum of scores for the structured report were 4.6 (range 1-5) and 1111 (mean value 74.07, STD 4.85), respectively. In the second round, Cronbach's alpha (Cα) correlation coefficient was an acceptable 0.70. In the second round, the overall mean score of the experts and the sum of score for structured report were 4.9 (range 4-5) and 1108 (mean value 79.14, STD 1.83), respectively. The overall mean score obtained by the experts in the second round was higher than the overall mean score of the first round, with a lower standard deviation value to underline greater agreement among the experts for the structured report reached in this round.

CONCLUSIONS

A wide implementation of SR is of critical importance in order to offer referring physicians and patients optimum quality of service and to provide researchers with the best quality data in the context of big data exploitation of available clinical data. Implementation is a complex procedure, requiring mature technology to successfully address the multiple challenges of user-friendliness, organization and interoperability.

Implementation of peer learning conferences throughout a multi-site abdominal radiology practice

PURPOSE

To initiate a peer learning conference for our abdominal radiology division across multiple geographically separated sites and different time zones, and to determine radiologist preference for peer learning versus traditional score-based peer review.

METHODS

We implemented a monthly peer learning videoconference for our abdominal radiology division. Surveys regarding radiologist opinion regarding traditional peer review and the new peer learning conferences were conducted before and after 6 months of conferences.

RESULTS

Peer learning conferences were well attended across our multiple sites, with an average of 43 participants per conference. Radiologist opinion regarding peer review was poor, with survey radiologists responding positively to only 1 out of 12 process questions. Opinion regarding peer learning was extremely favorable, with radiologists responding positively to 12 out of the same 12 process questions. After 6 months of peer learning conferences, 87.9% of surveyed radiologists wished to continue them in some fashion, and no one preferred to return to score-based peer review alone.

CONCLUSION

We successfully implemented a peer learning conference for our abdominal radiology division spread out over multiple geographic sites. Our radiologists strongly preferred peer learning conferences over our traditional peer review system for quality control.

Radiologist errors by modality, anatomic region, and pathology for 1.6 million exams: what we have learned

PURPOSE

To evaluate the feasibility of adding pathology to recent radiologist error characterization schemes of modality and anatomic region and the potential of this data to more specifically inform peer review and peer learning.

METHODS

Quality assurance data originating from 349 radiologists in a national teleradiology practice were collected for 2019. Interpretive errors were simply categorized as major or minor. Reporting or communication errors were classified as administrative errors. Interpretive errors were then divided by modality, anatomic region and placed into one of 64 pathologic categories.

RESULTS

Out of 1,628,464 studies, the discrepancy rate was 0.5% (8181/1,634,201). The 8181 total errors consisted of 2992 major errors (0.18%) and 5189 minor errors (0.32%). Precisely, 3.1% (257/8181) of total errors were administrative. Of major interpretive errors, 75.5% occurred on CT, with CT abdomen and pelvis accounting for 40.4%. The most common pathologic discrepancy for all exams was in the category of mass, nodule, or adenopathy (1583/8181), the majority of which were minor (1315/1583). The most common pathologic discrepancy for the 2937 major interpretive errors was fracture or dislocation (27%; 793/2937), followed by bleed (10.7%; 315/2937).

CONCLUSION

The addition of error-related pathology to peer review is both feasible and practical and provides a more detailed guide to targeted individual and practice-wide peer learning quality improvement efforts. Future research is needed to determine if there are measurable improvements in detection or interpretation of specific pathologies following error feedback and educational interventions.

Inter-rater reliability in quality assurance (QA) of pediatric chest X-rays

PURPOSE

The goal of the study is to determine the inter-rater agreement on multiple factors that were utilized to evaluate the quality of pediatric chest X-ray exams from different levels of healthcare provision in an African setting.

METHODS

The image quality of pediatric chest X-rays from 3 South African medical centers of varying level of healthcare service were retrospectively assessed by 3 raters for 12 quality factors including: (1) absent body parts; (2) under inspiration; (3) patient rotation; (4) scapula in the way; (5) patient kyphosis/lordosis; (6) artefact/foreign body; (7) central vessel visualization; (8) peripheral vessels visualization; (9) poor collimation; and (10) trachea and bronchi visualization; (11) post-cardiac vessel visualization; and (12) absent or wrong image orientation. Analysis was performed using the Brennan--Prediger coefficient of agreement for inter-rater reliability and Cochran's Q statistic and McNemar's test for inter-rater bias.

RESULTS

1077 X-rays were reviewed. The least difference between observers in the frequency of the errors was noticed for factors (1) absent body parts and (12) absent or wrong image orientation with almost perfect agreement between raters. κ score for these two factors among all raters and between each pair of raters was more than 0.95 with no significant inter-rater bias. Conversely, there was poor agreement for the remaining factors with the least agreed on being factor (3) patient rotation with a κ score of 0.23. This was followed by factors (2) under inspiration (κ score of 0.32) and factors (4) scapula in the way (κ score of 0.35) respectively. There was significant inter-rater bias for all these three factors.

CONCLUSION

Many of the factors used to assess the quality of a chest X-ray in children demonstrate poor reliability despite mitigation against variations in training, standard quality definitions and level of healthcare service provision. New definitions, objective measures and recording tools for assessing pediatric chest radiographic quality are required.

Structured Reporting of Computed Tomography in the Staging of Neuroendocrine Neoplasms: A Delphi Consensus Proposal

BACKGROUND

Structured reporting (SR) in radiology is becoming increasingly necessary and has been recognized recently by major scientific societies. This study aims to build structured CT-based reports in Neuroendocrine Neoplasms during the staging phase in order to improve communication between the radiologist and members of multidisciplinary teams.

MATERIALS AND METHODS

A panel of expert radiologists, members of the Italian Society of Medical and Interventional Radiology, was established. A Modified Delphi process was used to develop the SR and to assess a level of agreement for all report sections. Cronbach's alpha (Cα) correlation coefficient was used to assess internal consistency for each section and to measure quality analysis according to the average inter-item correlation.

RESULTS

The final SR version was built by including n=16 items in the "Patient Clinical Data" section, n=13 items in the "Clinical Evaluation" section, n=8 items in the "Imaging Protocol" section, and n=17 items in the "Report" section. Overall, 54 items were included in the final version of the SR. Both in the first and second round, all sections received more than a good rating: a mean value of 4.7 and range of 4.2-5.0 in the first round and a mean value 4.9 and range of 4.9-5 in the second round. In the first round, the Cα correlation coefficient was a poor 0.57: the overall mean score of the experts and the sum of scores for the structured report were 4.7 (range 1-5) and 728 (mean value 52.00 and standard deviation 2.83), respectively. In the second round, the Cα correlation coefficient was a good 0.82: the overall mean score of the experts and the sum of scores for the structured report were 4.9 (range 4-5) and 760 (mean value 54.29 and standard deviation 1.64), respectively.

CONCLUSIONS

The present SR, based on a multi-round consensus-building Delphi exercise following in-depth discussion between expert radiologists in gastro-enteric and oncological imaging, derived from a multidisciplinary agreement between a radiologist, medical oncologist and surgeon in order to obtain the most appropriate communication tool for referring physicians.

Optimizing Professional Practice Evaluation to Enable a Nonpunitive Learning Health System Approach to Peer Review

Healthcare organizations are focused on 2 different and sometimes conflicting tasks; (1) accelerate the improvement of clinical care delivery and (2) collect provider-specific data to determine the competency of providers. We describe creating a process to meet both of these aims while maintaining a culture that fosters improvement and teamwork.

Current Status and Future Wish List of Peer Review: A National Questionnaire of U.S. Radiologists

OBJECTIVE. Most peer review programs focus on error detection, numeric scoring, and radiologist-specific error rates. The effectiveness of this method on learning and systematic improvement is uncertain at best. Radiologists have been pushing for a transition from an individually punitive peer review system to a peer-learning model. This national questionnaire of U.S. radiologists aims to assess the current status of peer review and opportunities for improvement. MATERIALS AND METHODS. A 21-question multiple-choice questionnaire was developed and face validity assessed by the ARRS Performance Quality Improvement subcommittee. The questionnaire was e-mailed to 17,695 ARRS members and open for 4 weeks; two e-mail reminders were sent. Response collection was anonymous. Only responses from board-certified, practicing radiologists participating in peer review were analyzed. RESULTS. The response rate was 4.2% (742/17,695), and 73.7% (547/742) met inclusion criteria. Most responders were in private practice (51.7%, 283/547) with a group size of 11-50 radiologists (50.5%) and in an urban setting (61.6%). Significant diversity was noted in peer review systems, with RADPEER used by less than half (45.0%) and cases selected most commonly by commercial software (36.2%) or manually (31.2%). There was no consensus on the number of required peer reviews per month (10-20 cases, 32.1%; > 20 cases, 29.1%; < 10 cases, 21.7%). Less than half (43.7%) did not use peer review for group education. Whereas most (67.7%) were notified of their peer review results individually, 21.5% were not notified at all. Around half were dissatisfied (44.5%) because of insufficient learning (94.0%) and inaccurate representation of their performance improvement (75.5%). Overall, the group discrepancy rates were unknown to most radiologists who participate in peer review (54.3%). Submission bias was the main reason for underreporting of serious discrepancies (49.0%). Most found four peer-learning methods feasible in daily practice: incidental observation, 65.1%; focused practice review, 52.9%; professional auditing, 45.8%; and blinded double reading, 35.4%. CONCLUSION. More than half of participants reported that peer review data are used for educational purposes. However, significant diversity remains in current peer review practice with no agreement on number of required reviews, method of case selection, and oversight of results. Nearly half of the radiologists reported insufficient learning, although most feel a better system would be feasible in daily practice.

A Review of Perceptual Expertise in Radiology-How it develops, How we can test it, and Why humans still matter in the era of Artificial Intelligence

As the first step in image interpretation is detection, an error in perception can prematurely end the diagnostic process leading to missed diagnoses. Because perceptual errors of this sort-"failure to detect"-are the most common interpretive error (and cause of litigation) in radiology, understanding the nature of perceptual expertise is essential in decreasing radiology's long-standing error rates. In this article, we review what constitutes a perceptual error, the existing models of radiologic image perception, the development of perceptual expertise and how it can be tested, perceptual learning methods in training radiologists, and why understanding perceptual expertise is still relevant in the era of artificial intelligence. Adding targeted interventions, such as perceptual learning, to existing teaching practices, has the potential to enhance expertise and reduce medical error.

Survey of peer review programs among pediatric radiologists: report from the SPR Quality and Safety Committee

During the last 15 years, peer review has been widely incorporated into radiology quality improvement programs. However, current implementations are variable and carry concerns, including subjectivity of numerical scores and a sense of merely satisfying regulatory requirements. The Society for Pediatric Radiology (SPR) Quality and Safety Committee sought to evaluate the state of peer review programs in pediatric radiology practices, including implementation methods, perceived functions, strengths and weaknesses, and opportunities for improvement. We distributed an online 16-question survey to SPR members. Questions pertained to the type of peer review system, the use of numerical scores and comments, how feedback on discordances is given and received, and the use of peer learning conferences. We collected 219 responses (15% of survey invitations), 80% of which were from children's hospitals. Fifty percent of respondents said they use a picture archiving and communication system (PACS)-integrated peer review system. Comment-enhanced feedback for interpretive discordances was either very important or somewhat important to performance improvement in 86% of responses, compared to 48% with a similar perception of numerical scores. Sixty-eight percent of respondents said they either rarely or never check their numerical scores, and 82% either strongly or somewhat agreed that comments are more effective feedback than numerical scores. Ninety-three percent either strongly or somewhat agreed that peer learning conferences would be beneficial to their practice. Forty-eight percent thought that their current peer review system should be modified. Survey results demonstrate that peer review systems in pediatric radiology practices are implemented variably, and nearly half of respondents believe their systems should be modified. Most respondents prefer feedback in the form of comments and peer learning conferences, which are thought to be more beneficial for performance improvement than numerical scores.

Added Value of Dedicated Spine CT to Detect Fracture in Patients with CT Chest, Abdomen, and Pelvis in the Trauma Setting

PURPOSE

Fractures of the thoracolumbar spine account for up to 90% of spinal fractures, and are associated with significant disability. The advantage of acquiring dedicated spine CT imaging in addition to visceral CT studies of the chest, abdomen and pelvis for detection of spinal fractures has not been definitively established. This retrospective study seeks to determine the contribution of dedicated spine CT in the acute clinical setting.

METHODS

Patients who were diagnosed with fractures of the thoracic or lumbar spine at our institution between January 1, 2010 and June 30, 2014 were identified. Additional inclusion criteria included having a CT of the chest and/or abdomen and pelvis followed by a dedicated thoracic or lumbar spine CT within 30 days. Reports were reviewed for accuracy of fracture detection, and missed fractures were retrospectively analyzed on images for detectability.

RESULTS

A total of 102 patients met our inclusion criteria for a total of 312 fractures. Of the 312 fractures, 31 (10%) were missed on the initial visceral CT in 18 of the 102 patients. In all but two cases, at least one fracture was identified on the visceral spine CT. There were no cases in which the newly identified fractures changed patient management.

CONCLUSION

All fractures requiring surgical intervention were identified on the visceral CT. A dedicated spine CT does detect additional spine fractures but does not clearly alter patient management.

PathBot: A Radiology-Pathology Correlation Dashboard

Pathology is considered the "gold standard" of diagnostic medicine. The importance of radiology-pathology correlation is seen in interdepartmental patient conferences such as "tumor boards" and by the tradition of radiology resident immersion in a radiologic-pathology course at the American Institute of Radiologic Pathology. In practice, consistent pathology follow-up can be difficult due to time constraints and cumbersome electronic medical records. We present a radiology-pathology correlation dashboard that presents radiologists with pathology reports matched to their dictations, for both diagnostic imaging and image-guided procedures. In creating our dashboard, we utilized the RadLex ontology and National Center for Biomedical Ontology (NCBO) Annotator to identify anatomic concepts in pathology reports that could subsequently be mapped to relevant radiology reports, providing an automated method to match related radiology and pathology reports. Radiology-pathology matches are presented to the radiologist on a web-based dashboard. We found that our algorithm was highly specific in detecting matches. Our sensitivity was slightly lower than expected and could be attributed to missing anatomy concepts in the RadLex ontology, as well as limitations in our parent term hierarchical mapping and synonym recognition algorithms. By automating radiology-pathology correlation and presenting matches in a user-friendly dashboard format, we hope to encourage pathology follow-up in clinical radiology practice for purposes of self-education and to augment peer review. We also hope to provide a tool to facilitate the production of quality teaching files, lectures, and publications. Diagnostic images have a richer educational value when they are backed up by the gold standard of pathology.

Added value of double reading in diagnostic radiology,a systematic review

Objectives

Double reading in diagnostic radiology can find discrepancies in the original report, but a systematic program of double reading is resource consuming. There are conflicting opinions on the value of double reading. The purpose of the current study was to perform a systematic review on the value of double reading.

Methods

A systematic review was performed to find studies calculating the rate of misses and overcalls with the aim of establishing the added value of double reading by human observers.

Results

The literature search resulted in 1610 hits. After abstract and full-text reading, 46 articles were selected for analysis. The rate of discrepancy varied from 0.4 to 22% depending on study setting. Double reading by a sub-specialist, in general, led to high rates of changed reports.

Conclusions

The systematic review found rather low discrepancy rates. The benefit of double reading must be balanced by the considerable number of working hours a systematic double-reading scheme requires. A more profitable scheme might be to use systematic double reading for selected, high-risk examination types. A second conclusion is that there seems to be a value of sub-specialisation for increased report quality. A consequent implementation of this would have far-reaching organisational effects.

Key Points

• In double reading, two or more radiologists read the same images.

• A systematic literature review was performed.

• The discrepancy rates varied from 0.4 to 22% in various studies.

• Double reading by sub-specialists found high discrepancy rates.

Electronic supplementary material

The online version of this article (10.1007/s13244-018-0599-0) contains supplementary material, which is available to authorised users.

Practical Suggestions on How to Move From Peer Review to Peer Learning

OBJECTIVE

The purpose of this article is to outline practical steps that a department can take to transition to a peer learning model.

CONCLUSION

The 2015 Institute of Medicine report on improving diagnosis emphasized that organizations and industries that embrace error as an opportunity to learn tend to outperform those that do not. To meet this charge, radiology must transition from a peer review to a peer learning approach.

Structured Reporting in Radiology

Radiology reports are vital for patient care as referring physicians depend upon them for deciding appropriate patient management. Traditional narrative reports are associated with excessive variability in the language, length, and style, which can minimize report clarity and make it difficult for referring clinicians to identify key information needed for patient care. Structured reporting has been advocated as a potential solution for improving the quality of radiology reports. The Association of University Radiologists-Radiology Research Alliance Structured Reporting Task Force convened to explore the current and future role of structured reporting in radiology and summarized its finding in this article. We review the advantages and disadvantages of structured radiology reports and discuss the current prevailing sentiments among radiologists regarding structured reports. We also discuss the obstacles to the use of structured reports and highlight ways to overcome some of those challenges. We also discuss the future directions in radiology reporting in the era of personalized medicine.

Transition From Peer Review to Peer Learning: Experience in a Radiology Department

PURPOSE

To describe the process by which a radiology department moved from peer review to peer collaborative improvement (PCI) and review data from the first 16 months of the PCI process.

MATERIALS AND METHODS

Data from the first 16 months after PCI were reviewed: number of case reviews performed, number of learning opportunities identified, percentage yield of learning opportunities identified, type of learning opportunities identified, and comparison of the previous parameters between case randomly reviewed versus actively pushed (issues actively identified and entered). Changes in actively pushed cases were also assessed as volume per month over the 16 months (run chart). Faculty members were surveyed about their perception of the conversion to PCI.

RESULTS

In all, 12,197 cases were peer reviewed, yielding 1,140 learning opportunities (9.34%). The most common types of learning opportunities for all reviewed cases included perception (5.1%) and reporting (1.9%). The yield of learning opportunities from actively pushed cases was 96.3% compared with 3.88% for randomly reviewed cases. The number of actively pushed cases per month increased over the course of the period and established two new confidence intervals. The faculty survey revealed that the faculty perceived the new PCI process as positive, nonpunitive, and focused on improvement.

CONCLUSIONS

The study demonstrates that a switch to PCI is perceived as nonpunitive and associated with increased radiologist submission of learning opportunities. Active entering of identified learning opportunities had a greater yield and perceived value, compared with random review of cases.

Evidence-based anatomical review areas derived from systematic analysis of cases from a radiological departmental discrepancy meeting

AIM

To produce short checklists of specific anatomical review sites for different regions of the body based on the frequency of radiological errors reviewed at radiology discrepancy meetings, thereby creating "evidence-based" review areas for radiology reporting.

MATERIALS AND METHODS

A single centre discrepancy database was retrospectively reviewed from a 5-year period. All errors were classified by type, modality, body system, and specific anatomical location. Errors were assigned to one of four body regions: chest, abdominopelvic, central nervous system (CNS), and musculoskeletal (MSK). Frequencies of errors in anatomical locations were then analysed.

RESULTS

There were 561 errors in 477 examinations; 290 (46%) errors occurred in the abdomen/pelvis, 99 (15.7%) in the chest, 117 (18.5%) in the CNS, and 125 (19.9%) in the MSK system. In each body system, the five most common location were chest: lung bases on computed tomography (CT), apices on radiography, pulmonary vasculature, bones, and mediastinum; abdominopelvic: vasculature, colon, kidneys, liver, and pancreas; CNS: intracranial vasculature, peripheral cerebral grey matter, bone, parafalcine, and the frontotemporal lobes surrounding the Sylvian fissure; and MSK: calvarium, sacrum, pelvis, chest, and spine.

CONCLUSION

The five listed locations accounted for >50% of all perceptual errors suggesting an avenue for focused review at the end of reporting.

Error and discrepancy in radiology: inevitable or avoidable?

Errors and discrepancies in radiology practice are uncomfortably common, with an estimated day-to-day rate of 3-5% of studies reported, and much higher rates reported in many targeted studies. Nonetheless, the meaning of the terms "error" and "discrepancy" and the relationship to medical negligence are frequently misunderstood. This review outlines the incidence of such events, the ways they can be categorized to aid understanding, and potential contributing factors, both human- and system-based. Possible strategies to minimise error are considered, along with the means of dealing with perceived underperformance when it is identified. The inevitability of imperfection is explained, while the importance of striving to minimise such imperfection is emphasised.

TEACHING POINTS

• Discrepancies between radiology reports and subsequent patient outcomes are not inevitably errors. • Radiologist reporting performance cannot be perfect, and some errors are inevitable. • Error or discrepancy in radiology reporting does not equate negligence. • Radiologist errors occur for many reasons, both human- and system-derived. • Strategies exist to minimise error causes and to learn from errors made.

Implementation and Validation of PACS Integrated Peer Review for Discrepancy Recording of Radiology Reporting

The purpose of this work is to demonstrate the possibility of implementation of a PACS-integrated peer review system based on RADPEER™ classification providing a step-wise implementation plan utilizing features already present in the standard PACS implementation and without the requirement of additional software development. Furthermore, we show the usage and effects of the system during the first 30 months of usage. To allow fast and easy implementation into the daily workflow the key-word feature of the PACS was used. This feature allows to add a key-word to an imaging examination for easy searching in the PACS database (e.g. by entering keywords for different kinds of pathology). For peer review we implemented a keyword structure including a code for each of the existing RADPEER™ scoring language terms and a keyword with the phrase "second reading" followed by the name of the individual radiologist. The use of the short-keys to enter the codes in relation to the peer review was a simple to use solution. During the study 599 reports were peer reviewed. The active participation in this study of the radiologists varies and ranges from 3 to 327 reviews per radiologist. The number of peer review is highest in CT and CR. There are no significant technical obstacles to implement a PACS-integrated RADPEER™ -system based on key-words allowing easy integration of peer review into the daily routine without the requirement of additional software. Peer review implemented in a non-random setting based on relevant priors could already help in increasing the quality of radiological reporting and serve as continuing education among peers. Decisiveness, tact and trust are needed to promote use of the system and collaborative discussion of the results by radiologist.

The "Reading the Mind in the Eyes" Test: Investigation of Psychometric Properties and Test-Retest Reliability of the Persian Version

The psychometric properties of the Persian "Reading the Mind in the Eyes" test were investigated, so were the predictions from the Empathizing-Systemizing theory of psychological sex differences. Adults aged 16-69 years old (N = 545, female = 51.7 %) completed the test online. The analysis of items showed them to be generally acceptable. Test-retest reliability, as measured by Intra-class correlation coefficient, was 0.735 with a 95 % CI of (0.514, 0.855). The percentage of agreement for each item in the test-retest was satisfactory and the mean difference between test-retest scores was -0.159 (SD = 3.42). However, the internal consistency of Persian version, calculated by Cronbach's alpha (0.371), was poor. Females scored significantly higher than males but academic degree and field of study had no significant effect.

Updated Guidelines for the Medical Assessment and Care of Children Who May Have Been Sexually Abused

The medical evaluation is an important part of the clinical and legal process when child sexual abuse is suspected. Practitioners who examine children need to be up to date on current recommendations regarding when, how, and by whom these evaluations should be conducted, as well as how the medical findings should be interpreted. A previously published article on guidelines for medical care for sexually abused children has been widely used by physicians, nurses, and nurse practitioners to inform practice guidelines in this field. Since 2007, when the article was published, new research has suggested changes in some of the guidelines and in the table that lists medical and laboratory findings in children evaluated for suspected sexual abuse and suggests how these findings should be interpreted with respect to sexual abuse. A group of specialists in child abuse pediatrics met in person and via online communication from 2011 through 2014 to review published research as well as recommendations from the Centers for Disease Control and Prevention and the American Academy of Pediatrics and to reach consensus on if and how the guidelines and approach to interpretation table should be updated. The revisions are based, when possible, on data from well-designed, unbiased studies published in high-ranking, peer-reviewed, scientific journals that were reviewed and vetted by the authors. When such studies were not available, recommendations were based on expert consensus.

Radiologist-initiated double reading of abdominal CT: retrospective analysis of the clinical importance of changes to radiology reports

Background

Misinterpretation of radiological examinations is an important contributing factor to diagnostic errors. Consultant radiologists in Norwegian hospitals frequently request second reads by colleagues in real time. Our objective was to estimate the frequency of clinically important changes to radiology reports produced by these prospectively obtained double readings.

Methods

We retrospectively compared the preliminary and final reports from 1071 consecutive double-read abdominal CT examinations of surgical patients at five public hospitals in Norway. Experienced gastrointestinal surgeons rated the clinical importance of changes from the preliminary to final report. The severity of the radiological findings in clinically important changes was classified as increased, unchanged or decreased.

Results

Changes were classified as clinically important in 146 of 1071 reports (14%). Changes to 3 reports (0.3%) were critical (demanding immediate action), 35 (3%) were major (implying a change in treatment) and 108 (10%) were intermediate (requiring further investigations). The severity of the radiological findings was increased in 118 (81%) of the clinically important changes. Important changes were made less frequently when abdominal radiologists were first readers, more frequently when they were second readers, and more frequently to urgent examinations.

Conclusion

A 14% rate of clinically important changes made during double reading may justify quality assurance of radiological interpretation. Using expert second readers and a targeted selection of urgent cases and radiologists reading outside their specialty may increase the yield of discrepant cases.

Perceptual type error in everyday practice

Discrepancy or quality improvement meetings are good practice and are now commonplace in most Radiology Departments, with the aim of improving diagnostic accuracy, preventing recurrent and common mistakes, improving the radiological report and thereby improving patient care. A total of 122 cases were assessed from a two-year period. This review highlights some of the more common, recurrent and important issues encountered within a general hospital with an emphasis on learning points and review areas.

Performance results for a workstation-integrated radiology peer review quality assurance program

OBJECTIVE

To assess review completion rates, RADPEER score distribution, and sources of disagreement when using a workstation-integrated radiology peer review program, and to evaluate radiologist perceptions of the program.

DESIGN

Retrospective review of prospectively collected data.

SETTING

Large private outpatient radiology practice.

PARTICIPANTS

Radiologists (n = 66) with a mean of 16.0 (standard deviation, 9.2) years of experience.

INTERVENTIONS

Prior studies and reports of cases being actively reported were randomly selected for peer review using the RADPEER scoring system (a 4-point scale, with a score of 1 indicating agreement and scores of 2-4 indicating increasing levels of disagreement).

MAIN OUTCOME MEASURES

Assigned peer review completion rates, review scores, sources of disagreement and radiologist survey responses.

RESULTS

Of 31 293 assigned cases, 29 044 (92.8%; 95% CI 92.5-93.1%) were reviewed. Discrepant scores (score = 2, 3 or 4) were given in 0.69% (95% CI 0.60-0.79%) of cases and clinically significant discrepancy (score = 3 or 4) was assigned in 0.42% (95% CI 0.35-0.50%). The most common cause of disagreement was missed diagnosis (75.2%; 95% CI 66.8-82.1%). By anonymous survey, 94% of radiologists felt that peer review was worthwhile, 90% reported that the scores they received were appropriate and 78% felt that the received feedback was valuable.

CONCLUSION

Workstation-based peer review can increase completion rates and levels of radiologist acceptance while producing RADPEER scores similar to those previously reported. This approach may be one way to increase radiologist engagement in peer review quality assurance.

Peer Review: Lessons Learned in A Pediatric Radiology Department

The purpose of this article is to illustrate types of diagnostic errors and feedback given to radiologists, using cases to support and clarify these categories. A comment-enhanced peer review system may be leveraged to generate a comprehensive feedback categorization scheme. These include errors of observation, errors of interpretation, inadequate patient data gathering, errors of communication, interobserver variability, informational feedback, and compliments. Much of this feedback is captured through comments associated with interpretative agreements.