The Application of Machine Learning to Quality Improvement Through the Lens of the Radiology Value Network

Cross-Check QA: A Quality Assurance Workflow to Prevent Missed Diagnoses by Alerting Inadvertent Discordance Between the Radiologist and Artificial Intelligence in the Interpretation of High-Acuity CT Scans

PURPOSE

The aim of this study was to implement and evaluate a quality assurance (QA) workflow that leverages natural language processing to rapidly resolve inadvertent discordance between radiologists and an artificial intelligence (AI) decision support system (DSS) in the interpretation of high-acuity CT studies when the radiologist does not engage with AI DSS output.

METHODS

All consecutive high-acuity adult CT examinations performed in a health system between March 1, 2020, and September 20, 2022, were interpreted alongside an AI DSS (Aidoc) for intracranial hemorrhage, cervical spine fracture, and pulmonary embolus. CT studies were flagged for this QA workflow if they met three criteria: (1) negative results by radiologist report, (2) a high probability of positive results by the AI DSS, and (3) unviewed AI DSS output. In these cases, an automated e-mail notification was sent to our quality team. If discordance was confirmed on secondary review-an initially missed diagnosis-addendum and communication documentation was performed.

RESULTS

Of 111,674 high-acuity CT examinations interpreted alongside the AI DSS over this 2.5-year time period, the frequency of missed diagnoses (intracranial hemorrhage, pulmonary embolus, and cervical spine fracture) uncovered by this workflow was 0.02% (n = 26). Of 12,412 CT studies prioritized as depicting positive findings by the AI DSS, 0.4% (n = 46) were discordant, unengaged, and flagged for QA. Among these discordant cases, 57% (26 of 46) were determined to be true positives. Addendum and communication documentation was performed within 24 hours of the initial report signing in 85% of these cases.

CONCLUSIONS

Inadvertent discordance between radiologists and the AI DSS occurred in a small number of cases. This QA workflow leveraged natural language processing to rapidly detect, notify, and resolve these discrepancies and prevent potential missed diagnoses.

Development and Implementation of an Integrated Imaging Informatics Track for Radiology Residents: Our 3-Year Experience

RATIONALE AND OBJECTIVES

Imaging Informatics is an emerging and fast-evolving field that encompasses the management of information during all steps of the imaging value chain. With many information technology tools being essential to the radiologists' day-to-day work, there is an increasing need for qualified professionals with clinical background, technology expertise, and leadership skills. To answer this, we describe our experience in the development and implementation of an Integrated Imaging Informatics Track (I3T) for radiology residents at our institution.

MATERIALS AND METHODS

The I3T was created by a resident-driven initiative funded by an intradepartmental resident grant. Its curriculum is delivered through a combination of monthly small group discussions, operational meetings, recommended readings, lectures, and early exposure to the National Imaging Informatics Course. The track is steered and managed by the I3T Committee, including trainees and faculty advisors. Up to two first-year residents are selected annually based on their curriculum vitae and an interest application. Successful completion of the program requires submission of a capstone project and at least one academic deliverable (national meeting presentation, poster, exhibit, manuscript and/or grant).

RESULTS

In our three-year experience, the seven I3T radiology residents have reported a total of 58 scholarly activities related to Imaging Informatics. I3T residents have assumed leadership roles within our organization and nationally. All residents have successfully carried out their clinical responsibilities.

CONCLUSION

We have developed and implemented an I3T for radiology residents at our institution. These residents have been successful in their clinical, scholarship and leadership pursuits.

Optimization of Radiology Workflow with Artificial Intelligence

The potential of artificial intelligence (AI) in radiology goes far beyond image analysis. AI can be used to optimize all steps of the radiology workflow by supporting a variety of nondiagnostic tasks, including order entry support, patient scheduling, resource allocation, and improving the radiologist's workflow. This article discusses several principal directions of using AI algorithms to improve radiological operations and workflow management, with the intention of providing a broader understanding of the value of applying AI in the radiology department.