The Joint Commission Practice Performance Evaluation: A Primer for Radiologists

Automated annotation and classification of BI-RADS assessment from radiology reports

The Breast Imaging Reporting and Data System (BI-RADS) was developed to reduce variation in the descriptions of findings. Manual analysis of breast radiology report data is challenging but is necessary for clinical and healthcare quality assurance activities. The objective of this study is to develop a natural language processing (NLP) system for automated BI-RADS categories extraction from breast radiology reports. We evaluated an existing rule-based NLP algorithm, and then we developed and evaluated our own method using a supervised machine learning approach. We divided the BI-RADS category extraction task into two specific tasks: (1) annotation of all BI-RADS category values within a report, (2) classification of the laterality of each BI-RADS category value. We used one algorithm for task 1 and evaluated three algorithms for task 2. Across all evaluations and model training, we used a total of 2159 radiology reports from 18 hospitals, from 2003 to 2015. Performance with the existing rule-based algorithm was not satisfactory. Conditional random fields showed a high performance for task 1 with an F-1 measure of 0.95. Rules from partial decision trees (PART) algorithm showed the best performance across classes for task 2 with a weighted F-1 measure of 0.91 for BIRADS 0-6, and 0.93 for BIRADS 3-5. Classification performance by class showed that performance improved for all classes from Naïve Bayes to Support Vector Machine (SVM), and also from SVM to PART. Our system is able to annotate and classify all BI-RADS mentions present in a single radiology report and can serve as the foundation for future studies that will leverage automated BI-RADS annotation, to provide feedback to radiologists as part of a learning health system loop.

Quality metrics currently used in academic radiology departments: results of the QUALMET survey

OBJECTIVE

We present the results of the 2015 quality metrics (QUALMET) survey, which was designed to assess the commonalities and variability of selected quality and productivity metrics currently employed by a large sample of academic radiology departments representing all regions in the USA.

METHODS

The survey of key radiology metrics was distributed in March-April of 2015 via personal e-mail to 112 academic radiology departments.

RESULTS

There was a 34.8% institutional response rate. We found that most academic departments of radiology commonly utilize metrics of hand hygiene, report turn around time (RTAT), relative value unit (RVU) productivity, patient satisfaction and participation in peer review. RTAT targets were found to vary widely. The implementation of radiology peer review and the variety of ways in which peer review results are used within academic radiology departments, the use of clinical decision support tools and requirements for radiologist participation in Maintenance of Certification also varied. Policies for hand hygiene and critical results communication were very similar across all institutions reporting, and most departments utilized some form of missed case/difficult case conference as part of their quality and safety programme, as well as some form of periodic radiologist performance reviews.

CONCLUSION

Results of the QUALMET survey suggest many similarities in tracking and utilization of the selected quality and productivity metrics included in our survey. Use of quality indicators is not a fully standardized process among academic radiology departments. Advances in knowledge: This article examines the current quality and productivity metrics in academic radiology.

Key Principles in Quality and Safety in Radiology

OBJECTIVE

The purpose of this article is to introduce the reader to basic concepts of quality and safety in radiology.

CONCLUSION

Concepts are introduced that are keys to identifying, understanding, and utilizing certain quality tools with the aim of making process improvements. Challenges, opportunities, and change drivers can be mapped from the radiology quality perspective. Best practices, informatics, and benchmarks can profoundly affect the outcome of the quality improvement initiative we all aim to achieve.

Performance results for a workstation-integrated radiology peer review quality assurance program

OBJECTIVE

To assess review completion rates, RADPEER score distribution, and sources of disagreement when using a workstation-integrated radiology peer review program, and to evaluate radiologist perceptions of the program.

DESIGN

Retrospective review of prospectively collected data.

SETTING

Large private outpatient radiology practice.

PARTICIPANTS

Radiologists (n = 66) with a mean of 16.0 (standard deviation, 9.2) years of experience.

INTERVENTIONS

Prior studies and reports of cases being actively reported were randomly selected for peer review using the RADPEER scoring system (a 4-point scale, with a score of 1 indicating agreement and scores of 2-4 indicating increasing levels of disagreement).

MAIN OUTCOME MEASURES

Assigned peer review completion rates, review scores, sources of disagreement and radiologist survey responses.

RESULTS

Of 31 293 assigned cases, 29 044 (92.8%; 95% CI 92.5-93.1%) were reviewed. Discrepant scores (score = 2, 3 or 4) were given in 0.69% (95% CI 0.60-0.79%) of cases and clinically significant discrepancy (score = 3 or 4) was assigned in 0.42% (95% CI 0.35-0.50%). The most common cause of disagreement was missed diagnosis (75.2%; 95% CI 66.8-82.1%). By anonymous survey, 94% of radiologists felt that peer review was worthwhile, 90% reported that the scores they received were appropriate and 78% felt that the received feedback was valuable.

CONCLUSION

Workstation-based peer review can increase completion rates and levels of radiologist acceptance while producing RADPEER scores similar to those previously reported. This approach may be one way to increase radiologist engagement in peer review quality assurance.

Peer Review: Lessons Learned in A Pediatric Radiology Department

The purpose of this article is to illustrate types of diagnostic errors and feedback given to radiologists, using cases to support and clarify these categories. A comment-enhanced peer review system may be leveraged to generate a comprehensive feedback categorization scheme. These include errors of observation, errors of interpretation, inadequate patient data gathering, errors of communication, interobserver variability, informational feedback, and compliments. Much of this feedback is captured through comments associated with interpretative agreements.

Privileging and credentialing for interventional cardiology procedures

Local institutional-specific credentialing and privileging for procedures is an important process for ensuring the quality of care provided by interventional cardiologists. Recently revised standards for coronary intervention and the blossoming of structural heart disease programs have generated controversy over these processes. How standards are set for credentialing and privileging is poorly understood by most interventional cardiologists, including those responsible for credentialing and privileging. Requirements from The Joint Commission dictate how credentialing and privileging is performed at hospitals they accredit. Physicians must be recredentialed every 2 years at each hospital, with privileges renewed at that time. Hospitals must review quality of physicians even more frequently using Ongoing Professional Practice Evaluations. Hospitals must also evaluate the performance of physicians when they join a hospital staff or when they begin performing new procedures using Focused Professional Practice Evaluations. Cardiology department directors and catheterization laboratory directors are responsible for recredentialing and reprivileging members of their departments. Individual physicians are responsible for cooperating with these processes, and for periodic recertification with specialty boards and governmental agencies. We provide specific guidance to help physicians navigate these processes.

Consensus-oriented group peer review: a new process to review radiologist work output

The Joint Commission and other regulatory bodies have mandated that health care organizations implement processes for ongoing physician performance review. Software solutions, such as RADPEER™, have been created to meet this need efficiently. However, the authors believe that available systems are not optimally designed to produce changes in practice and overlook many important aspects of quality by excessive focus on diagnosis. The authors present a new model of peer review known as consensus-oriented group review, which is based on group discussion of cases in a conference setting and places greater emphasis on feedback than traditional systems of radiology peer review. By focusing on the process of peer review, consensus-oriented group review is intended to optimize performance improvement and foster group standards of practice. The authors also describe the software tool developed to implement this process of enriched peer review.

Optimizing radiation use during fluoroscopic procedures: a quality and safety improvement project

PURPOSE

The ionizing radiation used during fluoroscopically guided medical interventions carries risk. The teams performing these procedures seek to minimize those risks while preserving each procedure's benefits. This report describes a data-driven optimization strategy.

METHODS

Manual and automated data capture systems were used to collect a series of different metrics, including fluoroscopy time, kerma area product, and reference point air kerma, from both adult and pediatric interventional radiologic procedures. Tools from statistical process control were used to identify opportunities for improvement and assess which changes led to improvement.

RESULTS

Initial efforts focused on creating a system capable of reliably capturing fluoroscopy time from all interventional radiologic procedures. Ongoing data analysis and feedback to frontline teams led to the development of a manual workflow that reliably captured fluoroscopy time. Data capture was later supplemented by automatic capture of electronic records. This process exploited the standardized format (DICOM Structured Reporting) that newer fluoroscopy units use to record the radiation metrics. Data analysis found marked differences between the imaging protocols used for adults and children. Revision of the adult protocols led to a stable twofold reduction in average exposure per adult procedure. Analysis of balancing measures found no impact on workflow.

CONCLUSIONS

A systematic approach to improving radiation use during procedures led to a substantial and sustained reduction in risk with no reduction in benefits. Data were readily captured by both manual and automated processes. Concepts from cognitive psychology and information theory provided a theoretical basis for both data analysis and improvement opportunities.

Interrater variation in scoring radiological discrepancies

OBJECTIVE

Discrepancy meetings are an important aspect of clinical governance. The Royal College of Radiologists has published advice on how to conduct meetings, suggesting that discrepancies are scored using the scale: 0=no error, 1=minor error, 2=moderate error and 3=major error. We have noticed variation in scores attributed to individual cases by radiologists and have sought to quantify the variation in scoring at our meetings.

METHODS

The scores from six discrepancy meetings totalling 161 scored events were collected. The reliability of scoring was measured using Fleiss' kappa, which calculates the degree of agreement in classification.

RESULTS

The number of cases rated at the six meetings ranged from 18 to 31 (mean 27). The number of raters ranged from 11 to 16 (mean 14). Only cases where all the raters scored were included in the analysis. The Fleiss' kappa statistic ranged from 0.12 to 0.20, and mean kappa was 0.17 for the six meetings.

CONCLUSION

A kappa of 1.0 indicates perfect agreement above chance and 0.0 indicates agreement equal to chance. A rule of thumb is that a kappa ≥0.70 indicates adequate interrater agreement. Our mean result of 0.172 shows poor agreement between scorers. This could indicate a problem with the scoring system or may indicate a need for more formal training and agreement in how scores are applied.

ADVANCES IN KNOWLEDGE

Scoring of radiology discrepancies is highly subjective and shows poor interrater agreement.

A workstation-integrated peer review quality assurance program: pilot study

Background

The surrogate indicator of radiological excellence that has become accepted is consistency of assessments between radiologists, and the technique that has become the standard for evaluating concordance is peer review. This study describes the results of a workstation-integrated peer review program in a busy outpatient radiology practice.

Methods

Workstation-based peer review was performed using the software program Intelerad Peer Review. Cases for review were randomly chosen from those being actively reported. If an appropriate prior study was available, and if the reviewing radiologist and the original interpreting radiologist had not exceeded review targets, the case was scored using the modified RADPEER system.

Results

There were 2,241 cases randomly assigned for peer review. Of selected cases, 1,705 (76%) were interpreted. Reviewing radiologists agreed with prior reports in 99.1% of assessments. Positive feedback (score 0) was given in three cases (0.2%) and concordance (scores of 0 to 2) was assigned in 99.4%, similar to reported rates of 97.0% to 99.8%. Clinically significant discrepancies (scores of 3 or 4) were identified in 10 cases (0.6%). Eighty-eight percent of reviewed radiologists found the reviews worthwhile, 79% found scores appropriate, and 65% felt feedback was appropriate. Two-thirds of radiologists found case rounds discussing significant discrepancies to be valuable.

Conclusions

The workstation-based computerized peer review process used in this pilot project was seamlessly incorporated into the normal workday and met most criteria for an ideal peer review system. Clinically significant discrepancies were identified in 0.6% of cases, similar to published outcomes using the RADPEER system. Reviewed radiologists felt the process was worthwhile.

Familiarity with current practices of granting and maintaining privileges in pediatric interventional radiology--a worldwide survey of the members of the Society for Pediatric Interventional Radiology (SPIR)

BACKGROUND

Physician credentialing is a complex process driven by the demand for quality improvement in health care. In the U.S., the Joint Commission Standard of 2007 has tied hospital accreditation to credentialing through mandated use of the Focused Professional Practice Evaluation (FPPE) and Ongoing Professional Practice Evaluation (OPPE).

OBJECTIVE

To assess pediatric interventional radiologists' knowledge of how institutions grant them privileges.

MATERIALS AND METHODS

Members of the Society for Pediatric Interventional Radiology (SPIR) were sent a web-based survey regarding credentialing.

RESULTS

Of 122 members from 19 countries, 81 (66%) responded, and of these 81, 59 (73%) were familiar with their hospital's privileging process. Of 49 U.S. respondents and 32 non-U.S. respondents, 37 (76%) and 17 (53%), respectively, stated that interventional radiology credentialing was different from diagnostic radiology credentialing. Of the 49 U.S. respondents, 24 (49%) reported an OPPE, and of the 32 non-U.S. respondents, 8 (25%) reported an ongoing evaluation. The U.S. OPPE is performed at shorter intervals than its international equivalent.

CONCLUSION

Four years after the Joint Commission defined the FPPE and OPPE, separate credentialing of pediatric interventional radiology from pediatric diagnostic radiology is more likely in the U.S. than internationally, and U.S. pediatric interventional radiologists are more likely to have a defined ongoing professional evaluation and to be evaluated every 6 months or more frequently. There are many SPIR members who do not know how they obtain privileges and/or are not knowingly subject to an OPPE. This lack of knowledge may affect future education of interventional radiologists as well as the definition of pediatric interventional radiology practices within individual institutions.

Informatics in radiology: Measuring and improving quality in radiology: meeting the challenge with informatics

Quality is becoming a critical issue for radiology. Measuring and improving quality is essential not only to ensure optimum effectiveness of care and comply with increasing regulatory requirements, but also to combat current trends leading to commoditization of radiology services. A key challenge to implementing quality improvement programs is to develop methods to collect knowledge related to quality care and to deliver that knowledge to practitioners at the point of care. There are many dimensions to quality in radiology that need to be measured, monitored, and improved, including examination appropriateness, procedure protocol, accuracy of interpretation, communication of imaging results, and measuring and monitoring performance improvement in quality, safety, and efficiency. Informatics provides the key technologies that can enable radiologists to measure and improve quality. However, few institutions recognize the opportunities that informatics methods provide to improve safety and quality. The information technology infrastructure in most hospitals is limited, and they have suboptimal adoption of informatics techniques. Institutions can tackle the challenges of assessing and improving quality in radiology by means of informatics.