Analysis of Radiology Report Recommendation Characteristics and Rate of Recommended Action Performance

Recommendations for Additional Imaging on Head and Neck Imaging Examinations: Interradiologist Variation and Associated Factors

BACKGROUND. A paucity of relevant guidelines may lead to pronounced variation among radiologists in issuing recommendations for additional imaging (RAI) for head and neck imaging. OBJECTIVE. The purpose of this article was to explore associations of RAI for head and neck imaging examinations with examination, patient, and radiologist factors and to assess the role of individual radiologist-specific behavior in issuing such RAI. METHODS. This retrospective study included 39,200 patients (median age, 58 years; 21,855 women, 17,315 men, 30 with missing sex information) who underwent 39,200 head and neck CT or MRI examinations, interpreted by 61 radiologists, from June 1, 2021, through May 31, 2022. A natural language processing (NLP) tool with manual review of NLP results was used to identify RAI in report impressions. Interradiologist variation in RAI rates was assessed. A generalized mixed-effects model was used to assess associations between RAI and examination, patient, and radiologist factors. RESULTS. A total of 2943 (7.5%) reports contained RAI. Individual radiologist RAI rates ranged from 0.8% to 22.0% (median, 7.1%; IQR, 5.2-10.2%), representing a 27.5-fold difference between minimum and a maximum values and 1.8-fold difference between 25th and 75th percentiles. In multivariable analysis, RAI likelihood was higher for CTA than for CT examinations (OR, 1.32), for examinations that included a trainee in report generation (OR, 1.23), and for patients with self-identified race of Black or African American versus White (OR, 1.25); was lower for male than female patients (OR, 0.90); and was associated with increasing patient age (OR, 1.09 per decade) and inversely associated with radiologist years since training (OR, 0.90 per 5 years). The model accounted for 10.9% of the likelihood of RAI. Of explainable likelihood of RAI, 25.7% was attributable to examination, patient, and radiologist factors; 74.3% was attributable to radiologist-specific behavior. CONCLUSION. Interradiologist variation in RAI rates for head and neck imaging was substantial. RAI appear to be more substantially associated with individual radiologist-specific behavior than with measurable systemic factors. CLINICAL IMPACT. Quality improvement initiatives, incorporating best practices for incidental findings management, may help reduce radiologist preference-sensitive decision-making in issuing RAI for head and neck imaging and associated care variation.

From technical to understandable: Artificial Intelligence Large Language Models improve the readability of knee radiology reports

PURPOSE

The purpose of this study was to evaluate the effectiveness of an Artificial Intelligence-Large Language Model (AI-LLM) at improving the readability of knee radiology reports.

METHODS

Reports of 100 knee X-rays, 100 knee computed tomography (CT) scans and 100 knee magnetic resonance imaging (MRI) scans were retrieved. The following prompt command was inserted into the AI-LLM: 'Explain this radiology report to a patient in layman's terms in the second person:[Report Text]'. The Flesch-Kincaid reading level (FKRL) score, Flesch reading ease (FRE) score and report length were calculated for the original radiology report and the AI-LLM generated report. Any 'hallucination' or inaccurate text produced by the AI-LLM-generated report was documented.

RESULTS

Statistically significant improvements in mean FKRL scores in the AI-LLM generated X-ray report (12.7 ± 1.0-7.2 ± 0.6), CT report (13.4 ± 1.0-7.5 ± 0.5) and MRI report (13.5 ± 0.9-7.5 ± 0.6) were observed. Statistically significant improvements in mean FRE scores in the AI-LLM generated X-ray report (39.5 ± 7.5-76.8 ± 5.1), CT report (27.3 ± 5.9-73.1 ± 5.6) and MRI report (26.8 ± 6.4-73.4 ± 5.0) were observed. Superior FKRL scores and FRE scores were observed in the AI-LLM-generated X-ray report compared to the AI-LLM-generated CT report and MRI report, p < 0.001. The hallucination rates in the AI-LLM generated X-ray report, CT report and MRI report were 2%, 5% and 5%, respectively.

CONCLUSIONS

This study highlights the promising use of AI-LLMs as an innovative, patient-centred strategy to improve the readability of knee radiology reports. The clinical relevance of this study is that an AI-LLM-generated knee radiology report may enhance patients' understanding of their imaging reports, potentially reducing the responder burden placed on the ordering physicians. However, due to the 'hallucinations' produced by the AI-LLM-generated report, the ordering physician must always engage in a collaborative discussion with the patient regarding both reports and the corresponding images.

LEVEL OF EVIDENCE

Level IV.

Association of patient and health care organization factors with incidental nodule guidelines adherence: A multi-system observational study

BACKGROUND

Health care organizations are increasingly developing systems to ensure patients with pulmonary nodules receive guideline-adherent care. Our goal was to determine patient and organization factors that are associated with radiologist adherence as well as clinician and patient concordance to 2005 Fleischner Society guidelines for incidental pulmonary nodule follow-up.

MATERIALS

Trained researchers abstracted data from the electronic health record from two Veterans Affairs health care systems for patients with incidental pulmonary nodules as identified by interpreting radiologists from 2008 to 2016.

METHODS

We classified radiology reports and patient follow-up into two categories. Radiologist-Fleischner Adherence was the agreement between the radiologist's recommendation in the computed tomography report and the 2005 Fleischner Society guidelines. Clinician/Patient-Fleischner Concordance was agreement between patient follow-up and the guidelines. We calculated multivariable-adjusted predicted probabilities for factors associated with Radiologist-Fleischner Adherence and Clinician/Patient-Fleischner Concordance.

RESULTS

Among 3150 patients, 69% of radiologist recommendations were adherent to 2005 Fleischner guidelines, 4% were more aggressive, and 27% recommended less aggressive follow-up. Overall, only 48% of patients underwent follow-up concordant with 2005 Fleischner Society guidelines, 37% had less aggressive follow-up, and 15% had more aggressive follow-up. Radiologist-Fleischner Adherence was associated with Clinician/Patient-Fleischner Concordance with evidence for effect modification by health care system.

CONCLUSION

Clinicians and patients seem to follow radiologists' recommendations but often do not obtain concordant follow-up, likely due to downstream differential processes in each health care system. Health care organizations need to develop comprehensive and rigorous tools to ensure high levels of appropriate follow-up for patients with pulmonary nodules.

To report hounsfeld units or not: There is no question

INTRODUCTION

Most adrenal incidentalomas are not appropriately evaluated. Reporting the mass in the radiology report summary and providing recommendations in the report can positively impact evaluation. This study evaluated the effect of reporting Hounsfield units(HU) on adrenal incidentaloma follow-up.

METHODS

Patients with adrenal incidentalomas identified on noncontrast CT scan from 2015 to 2020 ​at a tertiary care institution were studied. Chart review was conducted. Patient and imaging characteristics were compared between patients who did and did not have HU reported. Outcomes of interest were 1)outpatient referral, 2)biochemical evaluation, and 3)dedicated imaging if appropriate. Multivariate analysis determined the impact of HU, reporting in the summary and provision of recommendations on the outcomes.

RESULTS

363 patients were studied, 36(9.9 ​%) had HU reported. When HU were used in addition to recommendations and reporting in the summary, the likelihood of outpatient referral increased from 10.1 to 32.6-fold (95%CI 7.7-138.1, p ​< ​0.001). Similarly, the likelihood of biochemical workup increased from 2.5 to 7.8-fold (95%CI 2.5-24.1, p ​< ​0.001).

CONCLUSION

Recording adrenal incidentaloma HU on non-contrast CT scans was associated with increased rates of outpatient referral and biochemical workup.

Do patients who read visit notes on the patient portal have a higher rate of "loop closure" on diagnostic tests and referrals in primary care? A retrospective cohort study

OBJECTIVES

The 2021 US Cures Act may engage patients to help reduce diagnostic errors/delays. We examined the relationship between patient portal registration with/without note reading and test/referral completion in primary care.

MATERIALS AND METHODS

Retrospective cohort study of patients with visits from January 1, 2018 to December 31, 2021, and order for (1) colonoscopy, (2) dermatology referral for concerning lesions, or (3) cardiac stress test at 2 academic primary care clinics. We examined differences in timely completion ("loop closure") of tests/referrals for (1) patients who used the portal and read ≥1 note (Portal + Notes); (2) those with a portal account but who did not read notes (Portal Account Only); and (3) those who did not register for the portal (No Portal). We estimated the predictive probability of loop closure in each group after adjusting for socio-demographic and clinical factors using multivariable logistic regression.

RESULTS

Among 12 849 tests/referrals, loop closure was more common among Portal+Note-readers compared to their counterparts for all tests/referrals (54.2% No Portal, 57.4% Portal Account Only, 61.6% Portal+Notes, P < .001). In adjusted analysis, compared to the No Portal group, the odds of loop closure were significantly higher for Portal Account Only (OR 1.2; 95% CI, 1.1-1.4), and Portal+Notes (OR 1.4; 95% CI, 1.3-1.6) groups. Beyond portal registration, note reading was independently associated with loop closure (P = .002).

DISCUSSION AND CONCLUSION

Compared to no portal registration, the odds of loop closure were 20% higher in tests/referrals for patients with a portal account, and 40% higher in tests/referrals for note readers, after controlling for sociodemographic and clinical factors. However, important safety gaps from unclosed loops remain, requiring additional engagement strategies.

DXA Reporting Updates: 2023 Official Positions of the International Society for Clinical Densitometry

INTRODUCTION

Professional guidance and standards assist radiologic interpreters in generating high quality reports. Initially DXA reporting Official Positions were provided by the ISCD in 2003; however, as the field has progressed, some of the current recommendations require revision and updating. This manuscript details the research approach and provides updated DXA reporting guidance.

METHODS

Key Questions were proposed by ISCD established protocols and approved by the Position Development Conference Steering Committee. Literature related to each question was accumulated by searching PubMed, and existing guidelines from other organizations were extracted from websites. Modifications and additions to the ISCD Official Positions were determined by an expert panel after reviewing the Task Force proposals and position papers.

RESULTS

Since most DXA is now performed in radiology departments, an approach was endorsed that better aligns with standard radiologic reports. To achieve this, reporting elements were divided into required minimum or optional. Collectively, required components comprise a standard diagnostic report and are considered the minimum necessary to generate an acceptable report. Additional elements were retained and categorized as optional. These optional components were considered relevant but tailored to a consultative, clinically oriented report. Although this information is beneficial, not all interpreters have access to sufficient clinical information, or may not have the clinical expertise to expand beyond a diagnostic report. Consequently, these are not required for an acceptable report.

CONCLUSION

These updated ISCD positions conform with the DXA field's evolution over the past 20 years. Specifically, a basic diagnostic report better aligns with radiology standards, and additional elements (which are valued by treating clinicians) remain acceptable but are optional and not required. Additionally, reporting guidance for newer elements such as fracture risk assessment are incorporated. It is our expectation that these updated Official Positions will improve compliance with required standards and generate high quality DXA reports that are valuable to the recipient clinician and contribute to best patient care.

Development and External Validation of an Artificial Intelligence Model for Identifying Radiology Reports Containing Recommendations for Additional Imaging

BACKGROUND. Reported rates of recommendations for additional imaging (RAIs) in radiology reports are low. Bidirectional encoder representations from transformers (BERT), a deep learning model pretrained to understand language context and ambiguity, has potential for identifying RAIs and thereby assisting large-scale quality improvement efforts. OBJECTIVE. The purpose of this study was to develop and externally validate an artificial intelligence (AI)-based model for identifying radiology reports containing RAIs. METHODS. This retrospective study was performed at a multisite health center. A total of 6300 radiology reports generated at one site from January 1, 2015, to June 30, 2021, were randomly selected and split by 4:1 ratio to create training (n = 5040) and test (n = 1260) sets. A total of 1260 reports generated at the center's other sites (including academic and community hospitals) from April 1 to April 30, 2022, were randomly selected as an external validation group. Referring practitioners and radiologists of varying sub-specialties manually reviewed report impressions for presence of RAIs. A BERT-based technique for identifying RAIs was developed by use of the training set. Performance of the BERT-based model and a previously developed traditional machine learning (TML) model was assessed in the test set. Finally, performance was assessed in the external validation set. The code for the BERT-based RAI model is publicly available. RESULTS. Among a total of 7419 unique patients (4133 women, 3286 men; mean age, 58.8 years), 10.0% of 7560 reports contained RAI. In the test set, the BERT-based model had 94.4% precision, 98.5% recall, and an F1 score of 96.4%. In the test set, the TML model had 69.0% precision, 65.4% recall, and an F1 score of 67.2%. In the test set, accuracy was greater for the BERT-based than for the TML model (99.2% vs 93.1%, p < .001). In the external validation set, the BERT-based model had 99.2% precision, 91.6% recall, an F1 score of 95.2%, and 99.0% accuracy. CONCLUSION. The BERT-based AI model accurately identified reports with RAIs, outperforming the TML model. High performance in the external validation set suggests the potential for other health systems to adapt the model without requiring institution-specific training. CLINICAL IMPACT. The model could potentially be used for real-time EHR monitoring for RAIs and other improvement initiatives to help ensure timely performance of clinically necessary recommended follow-up.

Opportunistic CT Assessment of Parathyroid Glands: Utility of Radiologist-Recommended Biochemical Evaluation for Diagnosing Primary Hyperparathyroidism

BACKGROUND. Existing gaps in primary hyperparathyroidism (PHPT) diagnosis and treatment have prompted calls for systemic change in the approach to this disease. One proposed change is opportunistic assessment for enlarged parathyroid glands on routine CT examinations, to target biochemical testing to individuals most likely to have un-diagnosed PHPT. OBJECTIVE. The purpose of our study was to assess the utility of a radiologist recommendation for biochemical testing in patients with a suspected enlarged parathyroid gland on routine CT for identifying previously undiagnosed PHPT. METHODS. This retrospective study included patients without known or suspected PHPT who underwent routine CT (i.e., performed for reasons other than known or suspected parathyroid disease) between August 2019 and September 2021 in which the clinical CT report included a radiologist recommendation for biochemical testing to evaluate for possible PHPT because of a suspected enlarged parathyroid gland. Neuroradiologists at the study institution included this recommendation on the basis of individual judgment without formal criteria. The EHR was reviewed to identify patients who underwent subsequent laboratory evaluation for PHPT. An endocrine surgeon used available laboratory results and clinical data to classify patients as having PHPT, secondary hyper-parathyroidism, or no parathyroid disorder independent of the CT findings. RESULTS. The sample comprised 39 patients (median age, 68 years; 20 women, 19 men) who received the radiologist recommendation for biochemical evaluation. Of these patients, 13 (33.3%) received the recommended biochemical evaluation. Of the 13 tested patients, three (23.1%) were classified as having PHPT, four (30.8%) as having secondary hyperparathyroidism, and six (46.2%) as having no parathyroid disorder. Thus, the number of patients needing to receive a radiologist recommendation for biochemical testing per correct PHPT diagnosis was 13.0, and the number of patients needing to undergo laboratory testing per correct PHPT diagnosis was 4.3. One of the three patients classified as having PHPT underwent surgical resection of the lesion identified by CT, which was shown on histopathologic evaluation to represent hypercellular parathyroid tissue. CONCLUSION. Radiologist recommendations for biochemical testing in patients with suspected enlarged parathyroid glands on routine CT helped to identify individuals with undiagnosed PHPT. CLINICAL IMPACT. Opportunistic assessment for enlarged parathyroid glands on routine CT may facilitate PHPT diagnosis.

Impact of an Automated Closed-Loop Communication and Tracking Tool on the Rate of Recommendations for Additional Imaging in Thoracic Radiology Reports

OBJECTIVE

Assess the effects of feedback reports and implementing a closed-loop communication system on rates of recommendations for additional imaging (RAIs) in thoracic radiology reports.

METHODS

In this retrospective, institutional review board-approved study at an academic quaternary care hospital, we analyzed 176,498 thoracic radiology reports during a pre-intervention (baseline) period from April 1, 2018, to November 30, 2018; a feedback report only period from December 1, 2018, to September 30, 2019; and a closed-loop communication system plus feedback report (IT intervention) period from October 1, 2019, to December 31, 2020, promoting explicit documentation of rationale, time frame, and imaging modality for RAI, defined as complete RAI. A previously validated natural language processing tool was used to classify reports with an RAI. Primary outcome of rate of RAI was compared using a control chart. Multivariable logistic regression determined factors associated with likelihood of RAI. We also estimated the completeness of RAI in reports comparing IT intervention to baseline using χ2 statistic.

RESULTS

The natural language processing tool classified 3.2% (5,682 of 176,498) reports as having an RAI; 3.5% (1,783 of 51,323) during the pre-intervention period, 3.8% (2,147 of 56,722) during the feedback report only period (odds ratio: 1.1, P = .03), and 2.6% (1,752 of 68,453) during the IT intervention period (odds ratio: 0.60, P < .001). In subanalysis, the proportion of incomplete RAI decreased from 84.0% (79 of 94) during the pre-intervention period to 48.5% (47 of 97) during the IT intervention period (P < .001).

DISCUSSION

Feedback reports alone increased RAI rates, and an IT intervention promoting documentation of complete RAI in addition to feedback reports led to significant reductions in RAI rate, incomplete RAI, and improved overall completeness of the radiology recommendations.

White Paper: Best Practices in the Communication and Management of Actionable Incidental Findings in Emergency Department Imaging

PURPOSE

Actionable incidental findings (AIFs) are common in radiologic imaging. Imaging is commonly performed in emergency department (ED) visits, and AIFs are frequently encountered, but the ED presents unique challenges for communication and follow-up of these findings. The authors formed a multidisciplinary panel to seek consensus regarding best practices in the reporting, communication, and follow-up of AIFs on ED imaging tests.

METHODS

A 15-member panel was formed, nominated by the ACR and American College of Emergency Physicians, to represent radiologists, emergency physicians, patients, and those involved in health care systems and quality. A modified Delphi process was used to identify areas of best practice and seek consensus. The panel identified four areas: (1) report elements and structure, (2) communication of findings with patients, (3) communication of findings with clinicians, and (4) follow-up and tracking systems. A survey was constructed to seek consensus and was anonymously administered in two rounds, with a priori agreement requiring at least 80% consensus. Discussion occurred after the first round, with readministration of questions where consensus was not initially achieved.

RESULTS

Consensus was reached in the four areas identified. There was particularly strong consensus that AIFs represent a system-level issue, with need for approaches that do not depend on individual clinicians or patients to ensure communication and completion of recommended follow-up.

CONCLUSIONS

This multidisciplinary collaboration represents consensus results on best practices regarding the reporting and communication of AIFs in the ED setting.

Closing the Loop: Re-engineering the Assessment and Tracking of Symptoms in Primary Care

Reliable systems that track the continuation, progression, or resolution of a patient's symptoms over time are essential for reliable diagnosis and ensuring that patients harboring more worrisome diagnoses are safely followed up. Given their first-contact role and increasing stresses on busy primary care clinicians and practices, new processes that make these tasks easier rather than creating more work for busy clinicians are especially needed.Some symptoms are sufficiently worrisome that they demand an urgent diagnosis and treatment while others result in a differential that can be more safely explored over time, or less differentiated and worrisome that they are best managed with the "test of time" to see if they resolve, worsen, or evolve into symptoms that are more worrisome. Regardless, it is essential that clinicians are able to reliably track symptoms over time, yet this capacity is rarely available or explicit. Working with systems engineers, we are developing prototypes for such systems and are working on their implementation and evaluation. In this commentary, we describe approaches to this essential, but underappreciated, problem in primary care.