Development and External Validation of an Artificial Intelligence Model for Identifying Radiology Reports Containing Recommendations for Additional Imaging

BACKGROUND. Reported rates of recommendations for additional imaging (RAIs) in radiology reports are low. Bidirectional encoder representations from transformers (BERT), a deep learning model pretrained to understand language context and ambiguity, has potential for identifying RAIs and thereby assisting large-scale quality improvement efforts. OBJECTIVE. The purpose of this study was to develop and externally validate an artificial intelligence (AI)-based model for identifying radiology reports containing RAIs. METHODS. This retrospective study was performed at a multisite health center. A total of 6300 radiology reports generated at one site from January 1, 2015, to June 30, 2021, were randomly selected and split by 4:1 ratio to create training (n = 5040) and test (n = 1260) sets. A total of 1260 reports generated at the center's other sites (including academic and community hospitals) from April 1 to April 30, 2022, were randomly selected as an external validation group. Referring practitioners and radiologists of varying sub-specialties manually reviewed report impressions for presence of RAIs. A BERT-based technique for identifying RAIs was developed by use of the training set. Performance of the BERT-based model and a previously developed traditional machine learning (TML) model was assessed in the test set. Finally, performance was assessed in the external validation set. The code for the BERT-based RAI model is publicly available. RESULTS. Among a total of 7419 unique patients (4133 women, 3286 men; mean age, 58.8 years), 10.0% of 7560 reports contained RAI. In the test set, the BERT-based model had 94.4% precision, 98.5% recall, and an F1 score of 96.4%. In the test set, the TML model had 69.0% precision, 65.4% recall, and an F1 score of 67.2%. In the test set, accuracy was greater for the BERT-based than for the TML model (99.2% vs 93.1%, p < .001). In the external validation set, the BERT-based model had 99.2% precision, 91.6% recall, an F1 score of 95.2%, and 99.0% accuracy. CONCLUSION. The BERT-based AI model accurately identified reports with RAIs, outperforming the TML model. High performance in the external validation set suggests the potential for other health systems to adapt the model without requiring institution-specific training. CLINICAL IMPACT. The model could potentially be used for real-time EHR monitoring for RAIs and other improvement initiatives to help ensure timely performance of clinically necessary recommended follow-up.

Frequency and Clinical Utility of Alerts for Intra-Institutional Radiologist Discrepant Opinions

OBJECTIVE

Determine the rate of documented notification, via an alert, for intra-institutional discrepant radiologist opinions and addended reports and resulting clinical management changes.

METHODS

This institutional review board-exempt, retrospective study was performed at a large academic medical center. We defined an intra-institutional discrepant opinion as when a consultant radiologist provides a different interpretation from that formally rendered by a colleague at our institution. We implemented a discrepant opinion policy requiring closed-loop notification of the consulting radiologist's second opinion to the original radiologist, who must acknowledge this alert within 30 days. This study included all discrepant opinion alerts created December 1, 2019, to December 31, 2021, of which two radiologists and an internal medicine physician performed consensus review. Primary outcomes were degree of discrepancy and percent of discrepant opinions leading to change in clinical management. Secondary outcome was report addendum rate compared with an existing peer learning program using Fisher's exact test.

RESULTS

Of 114 discrepant opinion alerts among 1,888,147 reports generated during the study period (0.006%), 58 alerts were categorized as major (50.9%), 41 as moderate (36.0%), and 15 as minor discrepancies (13.1%). Clinical management change occurred in 64 of 114 cases (56.1%). Report addendum rate for discrepant opinion alerts was 4-fold higher than for peer learning alerts at our institution (66 of 315 = 21% versus 432 of 8,273 =5.2%; P < .0001).

DISCUSSION

Although discrepant intra-institutional radiologist second opinions were rare, they frequently led to changes in clinical management. Capturing these discrepancies by encouraging alert use may help optimize patient care and document what was communicated to the referring or consulting care team by consulting radiologists.

Representing narrative evidence as clinical evidence logic statements

Objective

Clinical evidence logic statements (CELS) are shareable knowledge artifacts in a semistructured “If-Then” format that can be used for clinical decision support systems. This project aimed to assess factors facilitating CELS representation.

Materials and Methods

We described CELS representation of clinical evidence. We assessed factors that facilitate representation, including authoring instruction, evidence structure, and educational level of CELS authors. Five researchers were tasked with representing CELS from published evidence. Represented CELS were compared with the formal representation. After an authoring instruction intervention, the same researchers were asked to represent the same CELS and accuracy was compared with that preintervention using McNemar’s test. Moreover, CELS representation accuracy was compared between evidence that is structured versus semistructured, and between CELS authored by specialty-trained versus nonspecialty-trained researchers, using χ2 analysis.

Results

261 CELS were represented from 10 different pieces of published evidence by the researchers pre- and postintervention. CELS representation accuracy significantly increased post-intervention, from 20/261 (8%) to 63/261 (24%, P value &lt; .00001). More CELS were assigned for representation with 379 total CELS subsequently included in the analysis (278 structured and 101 semistructured) postintervention. Representing CELS from structured evidence was associated with significantly higher CELS representation accuracy (P = .002), as well as CELS representation by specialty-trained authors (P = .0004).

Discussion

CELS represented from structured evidence had a higher representation accuracy compared with semistructured evidence. Similarly, specialty-trained authors had higher accuracy when representing structured evidence.

Conclusion

Authoring instructions significantly improved CELS representation with a 3-fold increase in accuracy. However, CELS representation remains a challenging task.