Multivariate Analysis of Radiologists' Usage of Phrases that Convey Diagnostic Certainty

Quality assurance of late gadolinium enhancement cardiac magnetic resonance images: a deep learning classifier for confidence in the presence or absence of abnormality with potential to prompt real-time image optimization

BACKGROUND

Late gadolinium enhancement (LGE) of the myocardium has significant diagnostic and prognostic implications, with even small areas of enhancement being important. Distinguishing between definitely normal and definitely abnormal LGE images is usually straightforward, but diagnostic uncertainty arises when reporters are not sure whether the observed LGE is genuine or not. This uncertainty might be resolved by repetition (to remove artifact) or further acquisition of intersecting images, but this must take place before the scan finishes. Real-time quality assurance by humans is a complex task requiring training and experience, so being able to identify which images have an intermediate likelihood of LGE while the scan is ongoing, without the presence of an expert is of high value. This decision-support could prompt immediate image optimization or acquisition of supplementary images to confirm or refute the presence of genuine LGE. This could reduce ambiguity in reports.

METHODS

Short-axis, phase-sensitive inversion recovery late gadolinium images were extracted from our clinical cardiac magnetic resonance (CMR) database and shuffled. Two, independent, blinded experts scored each individual slice for "LGE likelihood" on a visual analog scale, from 0 (absolute certainty of no LGE) to 100 (absolute certainty of LGE), with 50 representing clinical equipoise. The scored images were split into two classes-either "high certainty" of whether LGE was present or not, or "low certainty." The dataset was split into training, validation, and test sets (70:15:15). A deep learning binary classifier based on the EfficientNetV2 convolutional neural network architecture was trained to distinguish between these categories. Classifier performance on the test set was evaluated by calculating the accuracy, precision, recall, F1-score, and area under the receiver operating characteristics curve (ROC AUC). Performance was also evaluated on an external test set of images from a different center.

RESULTS

One thousand six hundred and forty-five images (from 272 patients) were labeled and split at the patient level into training (1151 images), validation (247 images), and test (247 images) sets for the deep learning binary classifier. Of these, 1208 images were "high certainty" (255 for LGE, 953 for no LGE), and 437 were "low certainty". An external test comprising 247 images from 41 patients from another center was also employed. After 100 epochs, the performance on the internal test set was accuracy = 0.94, recall = 0.80, precision = 0.97, F1-score = 0.87, and ROC AUC = 0.94. The classifier also performed robustly on the external test set (accuracy = 0.91, recall = 0.73, precision = 0.93, F1-score = 0.82, and ROC AUC = 0.91). These results were benchmarked against a reference inter-expert accuracy of 0.86.

CONCLUSION

Deep learning shows potential to automate quality control of late gadolinium imaging in CMR. The ability to identify short-axis images with intermediate LGE likelihood in real-time may serve as a useful decision-support tool. This approach has the potential to guide immediate further imaging while the patient is still in the scanner, thereby reducing the frequency of recalls and inconclusive reports due to diagnostic indecision.

Improved Diagnostic Confidence Imparted by Radiologists in Radiology Reports After Educational Interventions on Reporting Styles

Background Radiology reports are important medico-legal documents facilitating communication between radiologists and referring doctors. Language clarity and precision are crucial for effective communication in these reports. Radiology reporting has changed with the evolution of imaging technology, prompting the adoption of precise terminology. Diagnostic certainty phrases (DCPs) play an important role in communicating diagnostic confidence in radiology reports. Objective The aim of this study was to evaluate the use of DCPs in radiology reports, before and after targeted educational interventions. Materials and methods The study was approved by the Aga Khan University Hospital's Ethical Review Committee and includes cross-sectional radiology reports. It involved three cycles of retrospective evaluation, with educational interventions in between to improve the use of DCPs. Results The study found a dynamic shift in the use of DCPs during the three cycles. Initially, intermediate-certainty phrases prevailed, followed by an increase in high-certainty phrases and a drop in low-certainty phrases. Later cycles showed a significant decline in DCPs and an increase in the use of definitive language. Across all subspecialties, there was a consistent decrease in intermediate- and low-certainty DCPs. Conclusion The study validates the transformative impact of educational interventions on the use of DCPs in radiology reports. The radiology reports frequently used DCPs with intermediate to low diagnostic certainty with improvement in the subsequent cycles of the study after educational interventions. It emphasizes the significance of continuing education to ensure the use of precise nomenclature.

Development and External Validation of an Artificial Intelligence Model for Identifying Radiology Reports Containing Recommendations for Additional Imaging

BACKGROUND. Reported rates of recommendations for additional imaging (RAIs) in radiology reports are low. Bidirectional encoder representations from transformers (BERT), a deep learning model pretrained to understand language context and ambiguity, has potential for identifying RAIs and thereby assisting large-scale quality improvement efforts. OBJECTIVE. The purpose of this study was to develop and externally validate an artificial intelligence (AI)-based model for identifying radiology reports containing RAIs. METHODS. This retrospective study was performed at a multisite health center. A total of 6300 radiology reports generated at one site from January 1, 2015, to June 30, 2021, were randomly selected and split by 4:1 ratio to create training (n = 5040) and test (n = 1260) sets. A total of 1260 reports generated at the center's other sites (including academic and community hospitals) from April 1 to April 30, 2022, were randomly selected as an external validation group. Referring practitioners and radiologists of varying sub-specialties manually reviewed report impressions for presence of RAIs. A BERT-based technique for identifying RAIs was developed by use of the training set. Performance of the BERT-based model and a previously developed traditional machine learning (TML) model was assessed in the test set. Finally, performance was assessed in the external validation set. The code for the BERT-based RAI model is publicly available. RESULTS. Among a total of 7419 unique patients (4133 women, 3286 men; mean age, 58.8 years), 10.0% of 7560 reports contained RAI. In the test set, the BERT-based model had 94.4% precision, 98.5% recall, and an F1 score of 96.4%. In the test set, the TML model had 69.0% precision, 65.4% recall, and an F1 score of 67.2%. In the test set, accuracy was greater for the BERT-based than for the TML model (99.2% vs 93.1%, p < .001). In the external validation set, the BERT-based model had 99.2% precision, 91.6% recall, an F1 score of 95.2%, and 99.0% accuracy. CONCLUSION. The BERT-based AI model accurately identified reports with RAIs, outperforming the TML model. High performance in the external validation set suggests the potential for other health systems to adapt the model without requiring institution-specific training. CLINICAL IMPACT. The model could potentially be used for real-time EHR monitoring for RAIs and other improvement initiatives to help ensure timely performance of clinically necessary recommended follow-up.

Performance and Agreement When Annotating Chest X-ray Text Reports—A Preliminary Step in the Development of a Deep Learning-Based Prioritization and Detection System

A chest X-ray report is a communicative tool and can be used as data for developing artificial intelligence-based decision support systems. For both, consistent understanding and labeling is important. Our aim was to investigate how readers would comprehend and annotate 200 chest X-ray reports. Reports written between 1 January 2015 and 11 March 2022 were selected based on search words. Annotators included three board-certified radiologists, two trained radiologists (physicians), two radiographers (radiological technicians), a non-radiological physician, and a medical student. Consensus labels by two or more of the experienced radiologists were considered “gold standard”. Matthew’s correlation coefficient (MCC) was calculated to assess annotation performance, and descriptive statistics were used to assess agreement between individual annotators and labels. The intermediate radiologist had the best correlation to “gold standard” (MCC 0.77). This was followed by the novice radiologist and medical student (MCC 0.71 for both), the novice radiographer (MCC 0.65), non-radiological physician (MCC 0.64), and experienced radiographer (MCC 0.57). Our findings showed that for developing an artificial intelligence-based support system, if trained radiologists are not available, annotations from non-radiological annotators with basic and general knowledge may be more aligned with radiologists compared to annotations from sub-specialized medical staff, if their sub-specialization is outside of diagnostic radiology.

Lexicon for adrenal terms at CT and MRI: a consensus of the Society of Abdominal Radiology adrenal neoplasm disease-focused panel

PURPOSE

Substantial variation in imaging terms used to describe the adrenal gland and adrenal findings leads to ambiguity and uncertainty in radiology reports and subsequently their understanding by referring clinicians. The purpose of this study was to develop a standardized lexicon to describe adrenal imaging findings at CT and MRI.

METHODS

Fourteen members of the Society of Abdominal Radiology adrenal neoplasm disease-focused panel (SAR-DFP) including one endocrine surgeon participated to develop an adrenal lexicon using a modified Delphi process to reach consensus. Five radiologists prepared a preliminary list of 35 imaging terms that was sent to the full group as an online survey (19 general imaging terms, 9 specific to CT, and 7 specific to MRI). In the first round, members voted on terms to be included and proposed definitions; subsequent two rounds were used to achieve consensus on definitions (defined as ≥ 80% agreement).

RESULTS

Consensus for inclusion was reached on 33/35 terms with two terms excluded (anterior limb and normal adrenal size measurements). Greater than 80% consensus was reached on the definitions for 15 terms following the first round, with subsequent consensus achieved for the definitions of the remaining 18 terms following two additional rounds. No included term had remaining disagreement.

CONCLUSION

Expert consensus produced a standardized lexicon for reporting adrenal findings at CT and MRI. The use of this consensus lexicon should improve radiology report clarity, standardize clinical and research terminology, and reduce uncertainty for referring providers when adrenal findings are present.

Positive oral contrast material for CT evaluation of non-traumatic abdominal pain in the ED: prospective assessment of diagnostic confidence and throughput metrics

OBJECTIVE

Evaluate the impact of positive oral contrast material (POCM) for non-traumatic abdominal pain on diagnostic confidence, diagnostic rate, and ED throughput.

MATERIALS AND METHODS

ED oral contrast guidelines were changed to limit use of POCM. A total of 2,690 abdominopelvic CT exams performed for non-traumatic abdominal pain were prospectively evaluated for diagnostic confidence (5-point scale at 20% increments; 5 = 80-100% confidence) during a 24-month period. Impact on ED metrics including time from CT order to exam, preliminary read, ED length of stay (LOS), and repeat CT scan within 7 days was assessed. A subset of cases (n = 729) was evaluated for diagnostic rate. Data were collected at 2 time points, 6 and 24 months following the change.

RESULTS

A total of 38 reviewers were participated (28 trainees, 10 staff). 1238 exams (46%) were done with POCM, 1452 (54%) were performed without POCM. For examinations with POCM, 80% of exams received a diagnostic confidence score of 5 (mean, 4.78 ± 0.43; 99% ≥ 4), whereas 60% of exams without POCM received a score of 5 (mean, 4.51 ± 0.70; 92% ≥ 4; p < .001). Trainees scored 1,523 exams (57%, 722 + POCM, 801 -POCM) and showed even lower diagnostic confidence in cases without PCOM compared with faculty (mean, 4.43 ± 0.68 vs. 4.59 ± 0.71; p < 0.001). Diagnostic rate in a randomly selected subset of exams (n = 729) was 54.2% in the POCM group versus 56.1% without POCM (p < 0.655). CT order to exam time decreased by 31 min, order to preliminary read decreased by 33 min, and ED LOS decreased by 30 min (approximately 8% of total LOS) in the group without POCM compared to those with POCM (p < 0.001 for all). 205 patients had a repeat scan within 7 days, 74 (36%) had IV contrast only, 131 (64%) had both IV and oral contrast on initial exam. Findings were consistent both over a 6-month evaluation period as well as the full 24-month study period.

CONCLUSION

Limiting use of POCM in the ED for non-traumatic abdominal pain improved ED throughput but impaired diagnostic confidence, particularly in trainees; however, it did not significantly impact diagnostic rates nor proportion of repeat CT exams.

Talking Points: Enhancing Communication Between Radiologists and Patients

Radiologists communicate along multiple pathways, using written, verbal, and non-verbal means. Radiology trainees must gain skills in all forms of communication, with attention to developing effective professional communication in all forms. This manuscript reviews evidence-based strategies for enhancing effective communication between radiologists and patients through direct communication, written means and enhanced reporting. We highlight patient-centered communication efforts, available evidence, and opportunities to engage learners and enhance training and simulation efforts that improve communication with patients at all levels of clinical care.

Adoption of a diagnostic certainty scale in abdominal imaging: 2-year experience at an academic institution

PURPOSE

Assess use of a diagnostic certainty scale (CS) for abdominal imaging reports and identify factors associated with greater adoption.

METHODS

This retrospective, Institutional Review Board-exempt study was conducted at an academic health system. Abdominal radiology reports containing diagnostic certainty phrases (DCPs) generated 4/1/2019-3/31/2021 were identified by a natural language processing tool. Reports containing DCPs were subdivided into those with/without a CS inserted at the end. Primary outcome was monthly CS use rate in reports containing DCPs. Secondary outcomes were assessment of factors associated with CS use, and usage of recommended DCPs over time. Chi-square test was used to compare proportions; univariable and multivariable regression assessed impact of other variables.

RESULTS

DCPs were used in 81,281/124,501 reports (65.3%). One-month post-implementation, 82/2310 (3.6%) of reports with DCPs contained the CS, increasing to 1862/4644 (40.1%) by study completion (p < 0.001). Multivariable analysis demonstrated reports containing recommended DCPs were more likely to have the CS (Odds Ratio [OR] 4.5; p < 0.001). Using CT as a reference, CS use was lower for ultrasound (OR 0.73; p < 0.001) and X-ray (OR 0.38; p < 0.001). There was substantial inter-radiologist variation in CS use (OR 0.01-26.3, multiple p values).

CONCLUSION

DCPs are very common in abdominal imaging reports and can be further clarified with CS use. Although voluntary CS adoption increased 13-fold over 2 years, > 50% of reports with DCPs lacked the CS at study's end. More stringent interventions, including embedding the scale in report templates, are likely needed to reduce inter-radiologist variation and decrease ambiguity in conveying diagnostic certainty to referring providers and patients.

Frequency of medical students' language expressing implicit uncertainty in simulated handovers

OBJECTIVES

The aim of this study was to investigate the number and type of implicit expressions of uncertainty by medical students during simulated patient handovers.

METHODS

Eighty-seven volunteer medical students, a convenience sample collected on a first-come, first-served basis, participated in simulated handovers. They each worked with three simulated patients who presented with different chief complaints and personal conditions. The handovers were video recorded and transcribed. A framework of implicit expressions of uncertainty was used to identify and count modifiers that attenuate or strengthen medical information using MAXQDA lexical search. We analysed the findings with respect to the patients' contexts.

RESULTS

Implicit uncertainty expressions which attenuate or strengthen information occurred in almost equal frequency, 1879 (55%) versus 1505 (45%). Attenuators were found most frequently in the category 'Questionable', 1041 (55.4%), strengtheners in the category 'Focused', 1031 (68.5%). Most attenuators and strengtheners were found in the handover of two patients with challenging personal conditions ('angry man', 434 (23.1%) versus 323 (21.5%); 'unfocused woman', 354 (19.4%) versus 322 (21.4%)) and one patient with abnormal laboratory findings ('elevated creatinine', 379 (20.2%) versus 285 (18.9%)).

CONCLUSIONS

Medical students use a variety of implicit expressions of uncertainty in simulated handovers. These findings provide an opportunity for medical educators to design communication courses that raise students' awareness for content-dependent implicit expressions of uncertainty and provide strategies to communicate uncertainty explicitly.

Behind the hedges: how to convey uncertainty in imaging reports

The imaging report is a summary document of findings and the primary form of communication of such to referring clinicians. Expressing uncertainty in the summary report is clearly difficult and the literature is unanimous that there is no agreement between imaging consultants and clinicians, and even between imaging consultants themselves, as to the meaning of uncertainty phrases. It is important for the imaging consultants to express uncertainty in the imaging report, but it is equally important that the referring clinician understands the degree of that uncertainty. Individual terminology does not bridge that gap. The present study reviews the literature in order to differentiate between uncertainty phrasing and hedging, and to find best practice examples to inform practice. We suggest three approaches that may be applied.

Early Adoption of a Certainty Scale to Improve Diagnostic Certainty Communication

OBJECTIVE

Assess the early voluntary adoption of a certainty scale to improve communicating diagnostic certainty in radiology reports.

METHODS

This institutional review board-approved study was part of a multifaceted initiative to improve radiology report quality at a tertiary academic hospital. A committee comprised of radiology subspecialty division representatives worked to develop recommendations for communicating varying degrees of diagnostic certainty in radiology reports in the form of a certainty scale, made publicly available online, which specified the terms recommended and the terms to be avoided in radiology reports. Twelve radiologists voluntarily piloted the scale; use was not mandatory. We assessed proportion of recommended terms among all diagnostic certainty terms in the Impression section (primary outcome) of all reports generated by the radiologists. Certainty terms were extracted via natural language processing over a 22-week postintervention period (31,399 reports) and compared with the same 22 calendar weeks 1 year pre-intervention (24,244 reports) using Fisher's exact test and statistical process control charts.

RESULTS

Overall, the proportion of recommended terms significantly increased from 8,498 of 10,650 (80.0%) pre-intervention to 9,646 of 11,239 (85.8%) postintervention (P < .0001 and by statistical process control). The proportion of recommended terms significantly increased for 8 of 12 radiologists (P < .0005 each), increased insignificantly for 3 radiologists (P > .05), and decreased without significance for 1 radiologist.

CONCLUSION

Designing and implementing a certainty scale was associated with increased voluntary use of recommended certainty terms in a small radiologist cohort. Larger-scale interventions will be needed for adoption of the scale across a broad range of radiologists.

Implicit expression of uncertainty - suggestion of an empirically derived framework

BACKGROUND

Uncertainty occurs in physicians' daily work in almost every clinical context and is also present in the clinical reasoning process. The way physicians communicate uncertainty in their thinking process during handoffs is crucial for patient safety because uncertainty has diverse effects on individuals involved in patient care. Dealing with uncertainty and expressing uncertainty are important processes in the development of professional identity of undergraduate medical students. Many studies focused on how to deal with uncertainty and whether uncertainty is explicitly expressed. Hardly any research has been done regarding implicit expression of uncertainty. Therefore, we studied the ways in which medical students in the role of beginning residents implicitly express uncertainty during simulated handoffs.

METHODS

Sixty-seven advanced undergraduate medical students participated in a simulated first day of residency including a consultation hour, a patient management phase with interprofessional interaction, and a patient handoff. We transcribed the videographed handoffs verbatim and extracted language with respect to expression of uncertainty using a grounded theory approach. Text sequences expressing patient related information were analyzed and coded with respect to language aspects which implicitly modified plain information with respect to increasing or decreasing uncertainty. Concepts and categories were developed and discussed until saturation of all aspects was reached.

RESULTS

We discovered a framework of implicit expressions of uncertainty regarding diagnostic and treatment-related decisions within four categories: "Statement", "Assessment", "Consideration", and "Implication". Each category was related to either the subcategory "Actions" or "Results" within the diagnostic or therapeutic decisions. Within each category and subcategory, we found a subset of expressions, which implicitly attenuated or strengthened plain information thereby increasing uncertainty or certainty, respectively. Language that implicitly attenuated plain information belonged to the categories questionable, incomplete, alterable, and unreliable while we could ascribe implicit strengtheners to the categories assertive, adequate, focused, and reliable.

CONCLUSIONS

Our suggested framework of implicit expression of uncertainty may help to raise the awareness for expression of uncertainty in the clinical reasoning process and provide support for making uncertainty explicit in the teaching process. This may lead to more transparent communication processes among health care professionals and eventually to improved patient safety.

Effect of a Report Template-Enabled Quality Improvement Initiative on Use of Preferred Phrases for Communicating Normal Findings in Structured Abdominal CT and MRI Reports

OBJECTIVE. The objective of this study was to assess impact of a report template quality improvement (QI) initiative on use of preferred phrases for communicating normal findings in structured abdominal CT and MRI reports. SUBJECTS AND METHODS. This prospective QI initiative, designed to decrease use of equivocal phrases and increase use of preferred and acceptable phrases (defined by multidisciplinary experts including patient advocates) in radiology reports, was performed in an academic medical center with over 800,000 annual radiologic examinations and was exempt from institutional review board approval. The intervention populated the preferred term "normal" (default) and acceptable specified pertinent negative phrases (pick-list option) when describing abdominal organ subheadings (liver, pancreas, spleen, adrenal glands, kidneys) within the "Findings" heading of abdominal CT and MRI report templates. We tabulated frequencies of the term "normal", specified pertinent negatives, and equivocal phrases in 21,629 reports before (June 1, 2017, to February 28, 2018) and 23,051 reports after (April 1, 2018, to December 31, 2018) the intervention using natural language processing and recorded trainee participation in report generation. We assessed intervention impact using statistical process control (SPC) charts and the Fisher exact test. RESULTS. Equivocal phrases were used less frequently in abdominal CT and MRI reports for both attending radiologists and trainees after the intervention (p < 0.05, SPC). Use of the term "normal" increased for reports generated by attending radiologists alone but decreased for reports created with trainee participation (p < 0.05, SPC). Frequency of pertinent negatives increased for reports with trainee participation (p < 0.05, SPC). CONCLUSION. A QI intervention decreased use of equivocal terms and increased use of preferred and acceptable phrases when communicating normal findings in abdominal CT and MRI reports.

Use of Machine Learning to Identify Follow-Up Recommendations in Radiology Reports

PURPOSE

The aims of this study were to assess follow-up recommendations in radiology reports, develop and assess traditional machine learning (TML) and deep learning (DL) models in identifying follow-up, and benchmark them against a natural language processing (NLP) system.

METHODS

This HIPAA-compliant, institutional review board-approved study was performed at an academic medical center generating >500,000 radiology reports annually. One thousand randomly selected ultrasound, radiography, CT, and MRI reports generated in 2016 were manually reviewed and annotated for follow-up recommendations. TML (support vector machines, random forest, logistic regression) and DL (recurrent neural nets) algorithms were constructed and trained on 850 reports (training data), with subsequent optimization of model architectures and parameters. Precision, recall, and F1 score were calculated on the remaining 150 reports (test data). A previously developed and validated NLP system (iSCOUT) was also applied to the test data, with equivalent metrics calculated.

RESULTS

Follow-up recommendations were present in 12.7% of reports. The TML algorithms achieved F1 scores of 0.75 (random forest), 0.83 (logistic regression), and 0.85 (support vector machine) on the test data. DL recurrent neural nets had an F1 score of 0.71; iSCOUT also had an F1 score of 0.71. Performance of both TML and DL methods by F1 scores appeared to plateau after 500 to 700 samples while training.

CONCLUSIONS

TML and DL are feasible methods to identify follow-up recommendations. These methods have great potential for near real-time monitoring of follow-up recommendations in radiology reports.