Use of Machine Learning to Identify Follow-Up Recommendations in Radiology Reports

Demographics and socioeconomic determinants of health predict continued participation in a CT lung cancer screening program

PURPOSE

We developed machine learning (ML) models to assess demographic and socioeconomic status (SES) variables' value in predicting continued participation in a low-dose CT lung cancer screening (LCS) program.

MATERIALS AND METHODS

480 LCS subjects were retrospectively examined for the following outcomes: (#1) no follow-up (single LCS scan) vs. multiple follow-ups (220 and 260 subjects respectively) and (#2) absent or delayed (>1 month past the due date) follow-up vs timely follow-up (356 and 124 subjects respectively). We quantified the contributions of 14 socioeconomic, demographic, and clinical predictors to LCS adherence, and validated and compared prediction performances of multivariate logistic regression (MLR), support vector machine (SVM) and shallow neural network (NN) models.

RESULTS

For outcome #1, age, sex, race, insurance status, personal cancer history, and median household income were found to be associated with returning for follow-ups. For outcome #2, age, sex, race, and insurance status were significant predictor of absent/delayed LCS follow-up. Across 5-fold cross-validation, the MLR model achieved an average AUC of 0.732 (95% CI, 0.661-0.803) for outcome #1 and 0.633 (95% CI, 0.602-0.664) for outcome #2 and is the model with best predictive performance overall, whereas NN and SVM tended to overfit training data and fell short on testing data performance for either outcome.

CONCLUSIONS

We identified significant predictors of LCS adherence, and our ML models can predict which subjects are at higher risk of receiving no or delayed LCS follow-ups. Our results could inform data-driven interventions to engage vulnerable populations and extend the benefits of LCS.

Artificial Intelligence in Radiology: Opportunities and Challenges

Artificial intelligence's (AI) emergence in radiology elicits both excitement and uncertainty. AI holds promise for improving radiology with regards to clinical practice, education, and research opportunities. Yet, AI systems are trained on select datasets that can contain bias and inaccuracies. Radiologists must understand these limitations and engage with AI developers at every step of the process - from algorithm initiation and design to development and implementation - to maximize benefit and minimize harm that can be enabled by this technology.

How AI May Transform Musculoskeletal Imaging

While musculoskeletal imaging volumes are increasing, there is a relative shortage of subspecialized musculoskeletal radiologists to interpret the studies. Will artificial intelligence (AI) be the solution? For AI to be the solution, the wide implementation of AI-supported data acquisition methods in clinical practice requires establishing trusted and reliable results. This implementation will demand close collaboration between core AI researchers and clinical radiologists. Upon successful clinical implementation, a wide variety of AI-based tools can improve the musculoskeletal radiologist's workflow by triaging imaging examinations, helping with image interpretation, and decreasing the reporting time. Additional AI applications may also be helpful for business, education, and research purposes if successfully integrated into the daily practice of musculoskeletal radiology. The question is not whether AI will replace radiologists, but rather how musculoskeletal radiologists can take advantage of AI to enhance their expert capabilities.

Artificial intelligence in cardiac computed tomography

Artificial Intelligence (AI) is a broad discipline of computer science and engineering. Modern application of AI encompasses intelligent models and algorithms for automated data analysis and processing, data generation, and prediction with applications in visual perception, speech understanding, and language translation. AI in healthcare uses machine learning (ML) and other predictive analytical techniques to help sort through vast amounts of data and generate outputs that aid in diagnosis, clinical decision support, workflow automation, and prognostication. Coronary computed tomography angiography (CCTA) is an ideal union for these applications due to vast amounts of data generation and analysis during cardiac segmentation, coronary calcium scoring, plaque quantification, adipose tissue quantification, peri-operative planning, fractional flow reserve quantification, and cardiac event prediction. In the past 5 years, there has been an exponential increase in the number of studies exploring the use of AI for cardiac computed tomography (CT) image acquisition, de-noising, analysis, and prognosis. Beyond image processing, AI has also been applied to improve the imaging workflow in areas such as patient scheduling, urgent result notification, report generation, and report communication. In this review, we discuss algorithms applicable to AI and radiomic analysis; we then present a summary of current and emerging clinical applications of AI in cardiac CT. We conclude with AI's advantages and limitations in this new field.

Advancements in Standardizing Radiological Reports: A Comprehensive Review

Standardized radiological reports stimulate debate in the medical imaging field. This review paper explores the advantages and challenges of standardized reporting. Standardized reporting can offer improved clarity and efficiency of communication among radiologists and the multidisciplinary team. However, challenges include limited flexibility, initially increased time and effort, and potential user experience issues. The efforts toward standardization are examined, encompassing the establishment of reporting templates, use of common imaging lexicons, and integration of clinical decision support tools. Recent technological advancements, including multimedia-enhanced reporting and AI-driven solutions, are discussed for their potential to improve the standardization process. Organizations such as the ACR, ESUR, RSNA, and ESR have developed standardized reporting systems, templates, and platforms to promote uniformity and collaboration. However, challenges remain in terms of workflow adjustments, language and format variability, and the need for validation. The review concludes by presenting a set of ten essential rules for creating standardized radiology reports, emphasizing clarity, consistency, and adherence to structured formats.

Development and External Validation of an Artificial Intelligence Model for Identifying Radiology Reports Containing Recommendations for Additional Imaging

BACKGROUND. Reported rates of recommendations for additional imaging (RAIs) in radiology reports are low. Bidirectional encoder representations from transformers (BERT), a deep learning model pretrained to understand language context and ambiguity, has potential for identifying RAIs and thereby assisting large-scale quality improvement efforts. OBJECTIVE. The purpose of this study was to develop and externally validate an artificial intelligence (AI)-based model for identifying radiology reports containing RAIs. METHODS. This retrospective study was performed at a multisite health center. A total of 6300 radiology reports generated at one site from January 1, 2015, to June 30, 2021, were randomly selected and split by 4:1 ratio to create training (n = 5040) and test (n = 1260) sets. A total of 1260 reports generated at the center's other sites (including academic and community hospitals) from April 1 to April 30, 2022, were randomly selected as an external validation group. Referring practitioners and radiologists of varying sub-specialties manually reviewed report impressions for presence of RAIs. A BERT-based technique for identifying RAIs was developed by use of the training set. Performance of the BERT-based model and a previously developed traditional machine learning (TML) model was assessed in the test set. Finally, performance was assessed in the external validation set. The code for the BERT-based RAI model is publicly available. RESULTS. Among a total of 7419 unique patients (4133 women, 3286 men; mean age, 58.8 years), 10.0% of 7560 reports contained RAI. In the test set, the BERT-based model had 94.4% precision, 98.5% recall, and an F1 score of 96.4%. In the test set, the TML model had 69.0% precision, 65.4% recall, and an F1 score of 67.2%. In the test set, accuracy was greater for the BERT-based than for the TML model (99.2% vs 93.1%, p < .001). In the external validation set, the BERT-based model had 99.2% precision, 91.6% recall, an F1 score of 95.2%, and 99.0% accuracy. CONCLUSION. The BERT-based AI model accurately identified reports with RAIs, outperforming the TML model. High performance in the external validation set suggests the potential for other health systems to adapt the model without requiring institution-specific training. CLINICAL IMPACT. The model could potentially be used for real-time EHR monitoring for RAIs and other improvement initiatives to help ensure timely performance of clinically necessary recommended follow-up.

Artificial Intelligence to Improve Patient Understanding of Radiology Reports

Diagnostic imaging reports are generally written with a target audience of other providers. As a result, the reports are written with medical jargon and technical detail to ensure accurate communication. With implementation of the 21st Century Cures Act, patients have greater and quicker access to their imaging reports, but these reports are still written above the comprehension level of the average patient. Consequently, many patients have requested reports to be conveyed in language accessible to them. Numerous studies have shown that improving patient understanding of their condition results in better outcomes, so driving comprehension of imaging reports is essential. Summary statements, second reports, and the inclusion of the radiologist's phone number have been proposed, but these solutions have implications for radiologist workflow. Artificial intelligence (AI) has the potential to simplify imaging reports without significant disruptions. Many AI technologies have been applied to radiology reports in the past for various clinical and research purposes, but patient focused solutions have largely been ignored. New natural language processing technologies and large language models (LLMs) have the potential to improve patient understanding of their imaging reports. However, LLMs are a nascent technology and significant research is required before LLM-driven report simplification is used in patient care.

The promise and limitations of artificial intelligence in musculoskeletal imaging

With the recent developments in deep learning and the rapid growth of convolutional neural networks, artificial intelligence has shown promise as a tool that can transform several aspects of the musculoskeletal imaging cycle. Its applications can involve both interpretive and non-interpretive tasks such as the ordering of imaging, scheduling, protocoling, image acquisition, report generation and communication of findings. However, artificial intelligence tools still face a number of challenges that can hinder effective implementation into clinical practice. The purpose of this review is to explore both the successes and limitations of artificial intelligence applications throughout the muscuskeletal imaging cycle and to highlight how these applications can help enhance the service radiologists deliver to their patients, resulting in increased efficiency as well as improved patient and provider satisfaction.

Impact of an Automated Closed-Loop Communication and Tracking Tool on the Rate of Recommendations for Additional Imaging in Thoracic Radiology Reports

OBJECTIVE

Assess the effects of feedback reports and implementing a closed-loop communication system on rates of recommendations for additional imaging (RAIs) in thoracic radiology reports.

METHODS

In this retrospective, institutional review board-approved study at an academic quaternary care hospital, we analyzed 176,498 thoracic radiology reports during a pre-intervention (baseline) period from April 1, 2018, to November 30, 2018; a feedback report only period from December 1, 2018, to September 30, 2019; and a closed-loop communication system plus feedback report (IT intervention) period from October 1, 2019, to December 31, 2020, promoting explicit documentation of rationale, time frame, and imaging modality for RAI, defined as complete RAI. A previously validated natural language processing tool was used to classify reports with an RAI. Primary outcome of rate of RAI was compared using a control chart. Multivariable logistic regression determined factors associated with likelihood of RAI. We also estimated the completeness of RAI in reports comparing IT intervention to baseline using χ2 statistic.

RESULTS

The natural language processing tool classified 3.2% (5,682 of 176,498) reports as having an RAI; 3.5% (1,783 of 51,323) during the pre-intervention period, 3.8% (2,147 of 56,722) during the feedback report only period (odds ratio: 1.1, P = .03), and 2.6% (1,752 of 68,453) during the IT intervention period (odds ratio: 0.60, P < .001). In subanalysis, the proportion of incomplete RAI decreased from 84.0% (79 of 94) during the pre-intervention period to 48.5% (47 of 97) during the IT intervention period (P < .001).

DISCUSSION

Feedback reports alone increased RAI rates, and an IT intervention promoting documentation of complete RAI in addition to feedback reports led to significant reductions in RAI rate, incomplete RAI, and improved overall completeness of the radiology recommendations.

Natural language processing in radiology: Clinical applications and future directions

Natural language processing (NLP) is a wide range of techniques that allows computers to interact with human text. Applications of NLP in everyday life include language translation aids, chat bots, and text prediction. It has been increasingly utilized in the medical field with increased reliance on electronic health records. As findings in radiology are primarily communicated via text, the field is particularly suited to benefit from NLP based applications. Furthermore, rapidly increasing imaging volume will continue to increase burden on clinicians, emphasizing the need for improvements in workflow. In this article, we highlight the numerous non-clinical, provider focused, and patient focused applications of NLP in radiology. We also comment on challenges associated with development and incorporation of NLP based applications in radiology as well as potential future directions.

Application of Deep Learning in Generating Structured Radiology Reports: A Transformer-Based Technique

Since radiology reports needed for clinical practice and research are written and stored in free-text narrations, extraction of relative information for further analysis is difficult. In these circumstances, natural language processing (NLP) techniques can facilitate automatic information extraction and transformation of free-text formats to structured data. In recent years, deep learning (DL)-based models have been adapted for NLP experiments with promising results. Despite the significant potential of DL models based on artificial neural networks (ANN) and convolutional neural networks (CNN), the models face some limitations to implement in clinical practice. Transformers, another new DL architecture, have been increasingly applied to improve the process. Therefore, in this study, we propose a transformer-based fine-grained named entity recognition (NER) architecture for clinical information extraction. We collected 88 abdominopelvic sonography reports in free-text formats and annotated them based on our developed information schema. The text-to-text transfer transformer model (T5) and Scifive, a pre-trained domain-specific adaptation of the T5 model, were applied for fine-tuning to extract entities and relations and transform the input into a structured format. Our transformer-based model in this study outperformed previously applied approaches such as ANN and CNN models based on ROUGE-1, ROUGE-2, ROUGE-L, and BLEU scores of 0.816, 0.668, 0.528, and 0.743, respectively, while providing an interpretable structured report.

Machine Learning to Optimize the Echocardiographic Follow-Up of Aortic Stenosis

BACKGROUND

Disease progression in patients with mild-to-moderate aortic stenosis is heterogenous and requires periodic echocardiographic examinations to evaluate severity.

OBJECTIVES

This study sought to explore the use of machine learning to optimize aortic stenosis echocardiographic surveillance automatically.

METHODS

The study investigators trained, validated, and externally applied a machine learning model to predict whether a patient with mild-to-moderate aortic stenosis will develop severe valvular disease at 1, 2, or 3 years. Demographic and echocardiographic patient data to develop the model were obtained from a tertiary hospital consisting of 4,633 echocardiograms from 1,638 consecutive patients. The external cohort was obtained from an independent tertiary hospital, consisting of 4,531 echocardiograms from 1,533 patients. Echocardiographic surveillance timing results were compared with the European and American guidelines echocardiographic follow-up recommendations.

RESULTS

In internal validation, the model discriminated severe from nonsevere aortic stenosis development with an area under the receiver-operating characteristic curve (AUC-ROC) of 0.90, 0.92, and 0.92 for the 1-, 2-, or 3-year interval, respectively. In external application, the model showed an AUC-ROC of 0.85, 0.85, and 0.85, for the 1-, 2-, or 3-year interval. A simulated application of the model in the external validation cohort resulted in savings of 49% and 13% of unnecessary echocardiographic examinations per year compared with European and American guideline recommendations, respectively.

CONCLUSIONS

Machine learning provides real-time, automated, personalized timing of next echocardiographic follow-up examination for patients with mild-to-moderate aortic stenosis. Compared with European and American guidelines, the model reduces the number of patient examinations.

Identifying stroke diagnosis-related features from medical imaging reports to improve clinical decision-making support

BACKGROUND

Medical imaging reports play an important role in communication of diagnostic information between radiologists and clinicians. Head magnetic resonance imaging (MRI) reports can provide evidence that is widely used in the diagnosis and treatment of ischaemic stroke. The high-signal regions of diffusion-weighted imaging (DWI) images in MRI reports are key evidence. Correctly identifying high-signal regions of DWI images is helpful for the treatment of ischaemic stroke patients. Since most of the multiple signals recorded in head MRI reports appear in the same part, it is challenging to identify high-signal regions of DWI images from MRI reports.

METHODS

We developed a deep learning model to automatically identify high-signal regions of DWI images from head MRI reports. We proposed a fine-grained entity typing model based on machine reading comprehension that transformed the traditional two-step fine-grained entity typing task into a question-answering task.

RESULTS

To prove the validity of the model proposed, we compared it with the fine-grained entity typing model, of which the F1 measure was 5.9% and 3.2% higher than the F1 measures of the models based on LSTM and BERT, respectively.

CONCLUSION

In this study, we explore the automatic identification of high-signal regions of DWI images from the description part of a head MRI report. We transformed the identification of high-signal regions of DWI images to an FET task and proposed an MRC-FET model. Compared with the traditional two-step FET method, the model we proposed not only simplifies the task but also has better performance. The comparable result shows that the work in this study can contribute to improving the clinical decision support system.

Machine Learning Model Drift: Predicting Diagnostic Imaging Follow-Up as a Case Example

OBJECTIVE

Address model drift in a machine learning (ML) model for predicting diagnostic imaging follow-up using data augmentation with more recent data versus retraining new predictive models.

METHODS

This institutional review board-approved retrospective study was conducted January 1, 2016, to December 31, 2020, at a large academic institution. A previously trained ML model was trained on 1,000 radiology reports from 2016 (old data). An additional 1,385 randomly selected reports from 2019 to 2020 (new data) were annotated for follow-up recommendations and randomly divided into two sets: training (n = 900) and testing (n = 485). Support vector machine and random forest (RF) algorithms were constructed and trained using 900 new data reports plus old data (augmented data, new models) and using only new data (new data, new models). The 2016 baseline model was used as comparator as is and trained with augmented data. Recall was compared with baseline using McNemar's test.

RESULTS

Follow-up recommendations were contained in 11.3% of reports (157 or 1,385). The baseline model retrained with new data had precision = 0.83 and recall = 0.54; none significantly different from baseline. A new RF model trained with augmented data had significantly better recall versus the baseline model (0.80 versus 0.66, P = .04) and comparable precision (0.90 versus 0.86).

DISCUSSION

ML methods for monitoring follow-up recommendations in radiology reports suffer model drift over time. A newly developed RF model achieved better recall with comparable precision versus simply retraining a previously trained original model with augmented data. Thus, regularly assessing and updating these models is necessary using more recent historical data.

Tempering Expectations on the Medical Artificial Intelligence Revolution: The Medical Trainee Viewpoint

The rapid development of artificial intelligence (AI) in medicine has resulted in an increased number of applications deployed in clinical trials. AI tools have been developed with goals of improving diagnostic accuracy, workflow efficiency through automation, and discovery of novel features in clinical data. There is subsequent concern on the role of AI in replacing existing tasks traditionally entrusted to physicians. This has implications for medical trainees who may make decisions based on the perception of how disruptive AI may be to their future career. This commentary discusses current barriers to AI adoption to moderate concerns of the role of AI in the clinical setting, particularly as a standalone tool that replaces physicians. Technical limitations of AI include generalizability of performance and deficits in existing infrastructure to accommodate data, both of which are less obvious in pilot studies, where high performance is achieved in a controlled data processing environment. Economic limitations include rigorous regulatory requirements to deploy medical devices safely, particularly if AI is to replace human decision-making. Ethical guidelines are also required in the event of dysfunction to identify responsibility of the developer of the tool, health care authority, and patient. The consequences are apparent when identifying the scope of existing AI tools, most of which aim to be physician assisting rather than a physician replacement. The combination of the limitations will delay the onset of ubiquitous AI tools that perform standalone clinical tasks. The role of the physician likely remains paramount to clinical decision-making in the near future.

Automated vetting of radiology referrals: exploring natural language processing and traditional machine learning approaches

Background

With a significant increase in utilisation of computed tomography (CT), inappropriate imaging is a significant concern. Manual justification audits of radiology referrals are time-consuming and require financial resources. We aimed to retrospectively audit justification of brain CT referrals by applying natural language processing and traditional machine learning (ML) techniques to predict their justification based on the audit outcomes.

Methods

Two human experts retrospectively analysed justification of 375 adult brain CT referrals performed in a tertiary referral hospital during the 2019 calendar year, using a cloud-based platform for structured referring. Cohen’s kappa was computed to measure inter-rater reliability. Referrals were represented as bag-of-words (BOW) and term frequency-inverse document frequency models. Text preprocessing techniques, including custom stop words (CSW) and spell correction (SC), were applied to the referral text. Logistic regression, random forest, and support vector machines (SVM) were used to predict the justification of referrals. A test set (300/75) was used to compute weighted accuracy, sensitivity, specificity, and the area under the curve (AUC).

Results

In total, 253 (67.5%) examinations were deemed justified, 75 (20.0%) as unjustified, and 47 (12.5%) as maybe justified. The agreement between the annotators was strong (κ = 0.835). The BOW + CSW + SC + SVM outperformed other binary models with a weighted accuracy of 92%, a sensitivity of 91%, a specificity of 93%, and an AUC of 0.948.

Conclusions

Traditional ML models can accurately predict justification of unstructured brain CT referrals. This offers potential for automated justification analysis of CT referrals in clinical departments.

Automatic text classification of actionable radiology reports of tinnitus patients using bidirectional encoder representations from transformer (BERT) and in-domain pre-training (IDPT)

Background

Given the increasing number of people suffering from tinnitus, the accurate categorization of patients with actionable reports is attractive in assisting clinical decision making. However, this process requires experienced physicians and significant human labor. Natural language processing (NLP) has shown great potential in big data analytics of medical texts; yet, its application to domain-specific analysis of radiology reports is limited.

Objective

The aim of this study is to propose a novel approach in classifying actionable radiology reports of tinnitus patients using bidirectional encoder representations from transformer BERT-based models and evaluate the benefits of in domain pre-training (IDPT) along with a sequence adaptation strategy.

Methods

A total of 5864 temporal bone computed tomography(CT) reports are labeled by two experienced radiologists as follows: (1) normal findings without notable lesions; (2) notable lesions but uncorrelated to tinnitus; and (3) at least one lesion considered as potential cause of tinnitus. We then constructed a framework consisting of deep learning (DL) neural networks and self-supervised BERT models. A tinnitus domain-specific corpus is used to pre-train the BERT model to further improve its embedding weights. In addition, we conducted an experiment to evaluate multiple groups of max sequence length settings in BERT to reduce the excessive quantity of calculations. After a comprehensive comparison of all metrics, we determined the most promising approach through the performance comparison of F1-scores and AUC values.

Results

In the first experiment, the BERT finetune model achieved a more promising result (AUC-0.868, F1-0.760) compared with that of the Word2Vec-based models(AUC-0.767, F1-0.733) on validation data. In the second experiment, the BERT in-domain pre-training model (AUC-0.948, F1-0.841) performed significantly better than the BERT based model(AUC-0.868, F1-0.760). Additionally, in the variants of BERT fine-tuning models, Mengzi achieved the highest AUC of 0.878 (F1-0.764). Finally, we found that the BERT max-sequence-length of 128 tokens achieved an AUC of 0.866 (F1-0.736), which is almost equal to the BERT max-sequence-length of 512 tokens (AUC-0.868,F1-0.760).

Conclusion

In conclusion, we developed a reliable BERT-based framework for tinnitus diagnosis from Chinese radiology reports, along with a sequence adaptation strategy to reduce computational resources while maintaining accuracy. The findings could provide a reference for NLP development in Chinese radiology reports.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-022-01946-y.

Analysis of Radiology Report Recommendation Characteristics and Rate of Recommended Action Performance

IMPORTANCE

Following up on recommendations from radiologic findings is important for patient care, but frequently there are failures to carry out these recommendations. The lack of reliable systems to characterize and track completion of actionable radiology report recommendations poses an important patient safety challenge.

OBJECTIVES

To characterize actionable radiology recommendations and, using this taxonomy, track and understand rates of loop closure for radiology recommendations in a primary care setting.

DESIGN, SETTING, AND PARTICIPANTS

Radiology reports in a primary care clinic at a large academic center were redesigned to include actionable recommendations in a separate dedicated field. Manual review of all reports generated from imaging tests ordered between January 1 and December 31, 2018, by primary care physicians that contained actionable recommendations was performed. For this quality improvement study, a taxonomy system that conceptualized recommendations was developed based on 3 domains: (1) what is recommended (eg, repeat a test or perform a different test, specialty referral), (2) specified time frame in which to perform the recommended action, and (3) contingency language qualifying the recommendation. Using this framework, a 2-stage process was used to review patients' records to classify recommendations and determine loop closure rates and factors associated with failure to complete recommended actions. Data analysis was conducted from April to July 2021.

MAIN OUTCOMES AND MEASURES

Radiology recommendations, time frames, and contingencies. Rates of carrying out vs not closing the loop on these recommendations in the recommended time frame were assessed.

RESULTS

A total of 598 radiology reports were identified with structured recommendations: 462 for additional or future radiologic studies and 196 for nonradiologic actions (119 specialty referrals, 47 invasive procedures, and 43 other actions). The overall rate of completed actions (loop closure) within the recommended time frame was 87.4%, with 31 open loop cases rated by quality expert reviewers to pose substantial clinical risks. Factors associated with successful loop closure included (1) absence of accompanying contingency language, (2) shorter recommended time frames, and (3) evidence of direct radiologist communication with the ordering primary care physicians. A clinically significant lack of loop closure was found in approximately 5% of cases.

CONCLUSIONS AND RELEVANCE

The findings of this study suggest that creating structured radiology reports featuring a dedicated recommendations field permits the development of taxonomy to classify such recommendations and determine whether they were carried out. The lack of loop closure suggests the need for more reliable systems.

A Machine Learning-Based Approach to Discriminating Basaltic Tectonic Settings

The geochemical characteristics of magmatic rocks can distinguish the tectonic setting of magma formation and their geochemical signatures are discriminated by using the whole-rock geochemical data. As a new attempt of artificial intelligence technology in geochemistry, the machine learning discrimination method is gradually complementary to the classical discriminative graphical method. However, the feature selection of high-dimensional data and the determination of many unknown parameters are the two main factors affecting the classification accuracy of the algorithm. In this paper, a particle swarm optimized support vector machine (PSO-SVM) model is established to classify the tectonic environments of basaltic rocks in the GEOROC database. The model mainly relies on the powerful search capability of the particle swarm algorithm to find the best parameter combination selected by the SVM based on experience to improve the accuracy. In this study, based on the basalt samples in the database and the confusion matrix, the performance of PSO-SVM model is evaluated by simulation experiments. The results show that the model proposed in this paper is more effective in distinguishing the basaltic tectonic environments, with an accuracy of more than 90%. Therefore, compared with the traditional discriminant map method, the machine learning method based on the fusion of two algorithms performs better in the tectonic environment classification problems.

Informatics Solutions to Mitigate Legal Risk Associated With Communication Failures

Communication failures are a documented cause of malpractice litigation against radiologists. As imaging volumes have increased, and with them the number of findings requiring further workup, radiologists are increasingly expected to communicate with ordering clinicians. However, communication may be unsuccessful for a variety of reasons that expose radiologists to potential malpractice risk. Informatics solutions have the potential to improve communication and decrease this risk. We discuss human-powered, purely automated, and hybrid approaches to closing the communications loop. In addition, we describe the Patient Test Results Information Act (Pennsylvania Act 112) and its implications for closing the loop on noncritical actionable findings.

Management of Incidental Thyroid Nodules on Chest CT: Using Natural Language Processing to Assess White Paper Adherence and Track Patient Outcomes

OBJECTIVE

The purpose of this study was to develop a natural language processing (NLP) pipeline to identify incidental thyroid nodules (ITNs) meeting criteria for sonographic follow-up and to assess both adherence rates to white paper recommendations and downstream outcomes related to these incidental findings.

METHODS

21583 non-contrast chest CT reports from 2017 and 2018 were retrospectively evaluated to identify reports which included either an explicit recommendation for thyroid ultrasound, a description of a nodule ≥ 1.5 cm, or description of a nodule with suspicious features. Reports from 2018 were used to train an NLP algorithm called fastText for automated identification of such reports. Algorithm performance was then evaluated on the 2017 reports. Next, any patient from 2017 with a report meeting criteria for ultrasound follow-up was further evaluated with manual chart review to determine follow-up adherence rates and nodule-related outcomes.

RESULTS

NLP identified reports with ITNs meeting criteria for sonographic follow-up with an accuracy of 96.5% (95% CI 96.2-96.7) and sensitivity of 92.1% (95% CI 89.8-94.3). In 10006 chest CTs from 2017, ITN follow-up ultrasound was indicated according to white paper criteria in 81 patients (0.8%), explicitly recommended in 46.9% (38/81) of patients, and obtained in less than half of patients in which it was appropriately recommended (17/35, 48.6%).

DISCUSSION

NLP accurately identified chest CT reports meeting criteria for ITN ultrasound follow-up. Radiologist adherence to white paper guidelines and subsequent referrer adherence to radiologist recommendations showed room for improvement.

Overview of Noninterpretive Artificial Intelligence Models for Safety, Quality, Workflow, and Education Applications in Radiology Practice

Artificial intelligence has become a ubiquitous term in radiology over the past several years, and much attention has been given to applications that aid radiologists in the detection of abnormalities and diagnosis of diseases. However, there are many potential applications related to radiologic image quality, safety, and workflow improvements that present equal, if not greater, value propositions to radiology practices, insurance companies, and hospital systems. This review focuses on six major categories for artificial intelligence applications: study selection and protocoling, image acquisition, worklist prioritization, study reporting, business applications, and resident education. All of these categories can substantially affect different aspects of radiology practices and workflows. Each of these categories has different value propositions in terms of whether they could be used to increase efficiency, improve patient safety, increase revenue, or save costs. Each application is covered in depth in the context of both current and future areas of work. Keywords: Use of AI in Education, Application Domain, Supervised Learning, Safety © RSNA, 2022.

Automatic Diagnosis Labeling of Cardiovascular MRI by Using Semisupervised Natural Language Processing of Text Reports

PURPOSE

To assess whether the semisupervised natural language processing (NLP) of text from clinical radiology reports could provide useful automated diagnosis categorization for ground truth labeling to overcome manual labeling bottlenecks in the machine learning pipeline.

MATERIALS AND METHODS

In this retrospective study, 1503 text cardiac MRI reports from 2016 to 2019 were manually annotated for five diagnoses by clinicians: normal, dilated cardiomyopathy (DCM), hypertrophic cardiomyopathy, myocardial infarction (MI), and myocarditis. A semisupervised method that uses bidirectional encoder representations from transformers (BERT) pretrained on 1.14 million scientific publications was fine-tuned by using the manually extracted labels, with a report dataset split into groups of 801 for training, 302 for validation, and 400 for testing. The model's performance was compared with two traditional NLP models: a rule-based model and a support vector machine (SVM) model. The models' F1 scores and receiver operating characteristic curves were used to analyze performance.

RESULTS

After 15 epochs, the F1 scores on the test set of 400 reports were as follows: normal, 84%; DCM, 79%; hypertrophic cardiomyopathy, 86%; MI, 91%; and myocarditis, 86%. The pooled F1 score and area under the receiver operating curve were 86% and 0.96, respectively. On the same test set, the BERT model had a higher performance than the rule-based model (F1 score, 42%) and SVM model (F1 score, 82%). Diagnosis categories classified by using the BERT model performed the labeling of 1000 MR images in 0.2 second.

CONCLUSION

The developed model used labels extracted from radiology reports to provide automated diagnosis categorization of MR images with a high level of performance.Keywords: Semisupervised Learning, Diagnosis/Classification/Application Domain, Named Entity Recognition, MRI Supplemental material is available for this article. © RSNA, 2021.

The role of artificial intelligence in paediatric cardiovascular magnetic resonance imaging

Artificial intelligence (AI) offers the potential to change many aspects of paediatric cardiac imaging. At present, there are only a few clinically validated examples of AI applications in this field. This review focuses on the use of AI in paediatric cardiovascular MRI, using examples from paediatric cardiovascular MRI, adult cardiovascular MRI and other radiologic experience.

Basic Artificial Intelligence Techniques: Natural Language Processing of Radiology Reports

Natural language processing (NLP) is a subfield of computer science and linguistics that can be applied to extract meaningful information from radiology reports. Symbolic NLP is rule based and well suited to problems that can be explicitly defined by a set of rules. Statistical NLP is better situated to problems that cannot be well defined and requires annotated or labeled examples from which machine learning algorithms can infer the rules. Both symbolic and statistical NLP have found success in a variety of radiology use cases. More recently, deep learning approaches, including transformers, have gained traction and demonstrated good performance.

Automatic detection of actionable radiology reports using bidirectional encoder representations from transformers

Background

It is essential for radiologists to communicate actionable findings to the referring clinicians reliably. Natural language processing (NLP) has been shown to help identify free-text radiology reports including actionable findings. However, the application of recent deep learning techniques to radiology reports, which can improve the detection performance, has not been thoroughly examined. Moreover, free-text that clinicians input in the ordering form (order information) has seldom been used to identify actionable reports. This study aims to evaluate the benefits of two new approaches: (1) bidirectional encoder representations from transformers (BERT), a recent deep learning architecture in NLP, and (2) using order information in addition to radiology reports.

Methods

We performed a binary classification to distinguish actionable reports (i.e., radiology reports tagged as actionable in actual radiological practice) from non-actionable ones (those without an actionable tag). 90,923 Japanese radiology reports in our hospital were used, of which 788 (0.87%) were actionable. We evaluated four methods, statistical machine learning with logistic regression (LR) and with gradient boosting decision tree (GBDT), and deep learning with a bidirectional long short-term memory (LSTM) model and a publicly available Japanese BERT model. Each method was used with two different inputs, radiology reports alone and pairs of order information and radiology reports. Thus, eight experiments were conducted to examine the performance.

Results

Without order information, BERT achieved the highest area under the precision-recall curve (AUPRC) of 0.5138, which showed a statistically significant improvement over LR, GBDT, and LSTM, and the highest area under the receiver operating characteristic curve (AUROC) of 0.9516. Simply coupling the order information with the radiology reports slightly increased the AUPRC of BERT but did not lead to a statistically significant improvement. This may be due to the complexity of clinical decisions made by radiologists.

Conclusions

BERT was assumed to be useful to detect actionable reports. More sophisticated methods are required to use order information effectively.

Machine learning based natural language processing of radiology reports in orthopaedic trauma

OBJECTIVES

To compare different Machine Learning (ML) Natural Language Processing (NLP) methods to classify radiology reports in orthopaedic trauma for the presence of injuries. Assessing NLP performance is a prerequisite for downstream tasks and therefore of importance from a clinical perspective (avoiding missed injuries, quality check, insight in diagnostic yield) as well as from a research perspective (identification of patient cohorts, annotation of radiographs).

METHODS

Datasets of Dutch radiology reports of injured extremities (n = 2469, 33% fractures) and chest radiographs (n = 799, 20% pneumothorax) were collected in two different hospitals and labeled by radiologists and trauma surgeons for the presence or absence of injuries. NLP classification was applied and optimized by testing different preprocessing steps and different classifiers (Rule-based, ML, and Bidirectional Encoder Representations from Transformers (BERT)). Performance was assessed by F1-score, AUC, sensitivity, specificity and accuracy.

RESULTS

The deep learning based BERT model outperforms all other classification methods which were assessed. The model achieved an F1-score of (95 ± 2)% and accuracy of (96 ± 1)% on a dataset of simple reports (n= 2469), and an F1 of (83 ± 7)% with accuracy (93 ± 2)% on a dataset of complex reports (n= 799).

CONCLUSION

BERT NLP outperforms traditional ML and rule-base classifiers when applied to Dutch radiology reports in orthopaedic trauma.

Closing the Compliance Loop on Follow-Up Imaging Recommendations: Comparing Radiologists' and Administrators' Attitudes

PURPOSE

To compare non-physician healthcare professional and radiologists' survey responses regarding attitudes and current practices, policies, and procedures related to the follow-up of nonemergent actionable incidental findings (AIF).

MATERIALS AND METHODS

The American College of Radiology (ACR) developed a survey with input from a technical expert panel (TEP). Survey items were developed by TEP members, refined by an ACR market research expert, and were examined for face and construct validity. The survey was distributed among ACR membership and various medical professional organizations. Responses from non-physician responders and radiologists were analyzed and compared using descriptive statistics.

RESULTS

The analysis included 375 responses, 247 from radiologists and 128 from non-physicians. All respondent groups stated that radiology follow-up recommendations are evidence-based. Both respondent groups indicated that there is up to moderate risk associated with AIF follow-up. Both respondent groups similarly favored that the accountability for communicating AIF lies first with the ordering provider, followed by primary care providers, then the patient, and lastly an automated process that is managed by a staff member and/or the radiologist. All respondent groups indicated that tracking processes were more commonly funded by the healthcare system than through the radiology budget.

CONCLUSION

There is alignment between non-physicians and radiologists regarding the implementation of tracking systems that assure completion of radiology follow-up recommendations. Building tracking systems represents an opportunity for multi-disciplinary collaboration to address care transition communication and process gaps.

Variation in Radiologists' Follow-Up Imaging Recommendations for Small Cystic Pancreatic Lesions

OBJECTIVE

This study aimed to determine the incidence, identify imaging and patient factors, and measure individual radiologist variation associated with follow-up recommendations for small focal cystic pancreatic lesions (FCPLs), a common incidental imaging finding.

METHODS

This institutional review board-approved retrospective study analyzed 146,709 reports from abdominal CTs and MRIs performed in a large academic hospital from July 1, 2016, to June 30, 2018. A trained natural language processing tool identified 4,345 reports with FCPLs, which were manually reviewed to identify those containing one or more <1.5-cm pancreatic cysts. For these patients, patient, lesion, and radiologist features and follow-up recommendations for FCPL were extracted. A nonlinear random-effects model estimated degree of variation in follow-up recommendations across radiologists at department and division levels.

RESULTS

Of 2,872 reports with FCPLs < 1.5 cm, 708 (24.7%) had FCPL-related follow-up recommendations. Average patient age was 67 years (SD ± 11). In all, 1,721 (60.0%) reports were for female patients; 59.3% of patients had only one cyst. In multivariable analysis, older patients had slightly lower follow-up recommendation rates (odds ratio [OR]: 0.98 [0.98-1.00] per additional year), and lesions associated with main duct dilatation and septation were more likely to have a follow-up recommendation (ORs: 1.93 [1.11-3.36] and 2.88 [1.89-4.38], respectively). Radiologist years in practice (P = .51), trainee presence (P = .21), and radiologist gender (P = .52) were not associated with increased follow-up recommendations. There was significant interradiologist variation in the Abdominal Imaging Division (P = .04), but not in Emergency Radiology (P = .31) or Cancer Imaging Divisions (P = .29).

DISCUSSION

Interradiologist variation significantly contributes to variability in follow-up imaging recommendations for FCPLs.

Workflow Applications of Artificial Intelligence in Radiology and an Overview of Available Tools

In the past decade, there has been tremendous interest in applying artificial intelligence (AI) to improve the field of radiology. Currently, numerous AI applications are in development, with potential benefits spanning all steps of the imaging chain from test ordering to report communication. AI has been proposed as a means to optimize patient scheduling, improve worklist management, enhance image acquisition, and help radiologists interpret diagnostic studies. Although the potential for AI in radiology appears almost endless, the field is still in the early stages, with many uses still theoretical, in development, or limited to single institutions. Moreover, although the current use of AI in radiology has emphasized its clinical applications, some of which are in the distant future, it is increasingly clear that AI algorithms could also be used in the more immediate future for a variety of noninterpretive and quality improvement uses. Such uses include the integration of AI into electronic health record systems to reduce unwarranted variation in radiologists' follow-up recommendations and to improve other dimensions of radiology report quality. In the end, the potential of AI in radiology must be balanced with acknowledgment of its current limitations regarding generalizability and data privacy.

Improvement in Follow-up Imaging With a Patient Tracking System and Computerized Registry for Lung Nodule Management

PURPOSE

Despite established guidelines, radiologists' recommendations and timely follow-up of incidental lung nodules remain variable. To improve follow-up of nodules, a system using standardized language (tracker phrases) recommending time-based follow-up in chest CT reports, coupled with a computerized registry, was created.

MATERIALS AND METHODS

Data were obtained from the electronic health record and a facility-built electronic lung nodule registry. We evaluated two randomly selected patient cohorts with incidental nodules on chest CT reports: before intervention (September 2008 to March 2011) and after intervention (August 2011 to December 2016). Multivariable logistic regression was used to compare the cohorts for the main outcome of timely follow-up, defined as a subsequent report within 13 months of the initial report.

RESULTS

In all, 410 patients were included in the pretracker cohort versus 626 in the tracker cohort. Before system inception, 30% of CT reports lacked an explicit time-based recommendation for nodule follow-up. The proportion of patients with timely follow-up increased from 46% to 55%, and the proportion of those with no documented follow-up or follow-up beyond 24 months decreased from 48% to 31%. The likelihood of timely follow-up increased 41%, adjusted for high risk for lung cancer and age 65 years or older. After system inception, reports missing a tracker phrase for nodule recommendation averaged 6%, without significant interyear variation.

CONCLUSIONS

Standardized language added to CT reports combined with a computerized registry designed to identify and track patients with incidental lung nodules was associated with improved likelihood of follow-up imaging.

A survey on Machine learning based Medical Assistive systems in Current Oncological Sciences

BACKGROUND

Cancer is one of the life threatening disease which is affecting a large number of population worldwide. The cancer cells multiply inside the body without showing much symptoms on the surface of the skin thereby making it difficult to predict and detect at the onset of disease. Many organizations are working towards automating the process of cancer detection with minimal false detection rates.

INTRODUCTION

The machine learning algorithms serve to be a promising alternative to support health care practitioners to rule out the disease and predict the growth with various imaging and statistical analysis tools. The medical practitioners are utilizing the output of these algorithms to diagnose and design the course of treatment. These algorithms are capable of finding out the risk level of the patient and can reduce the mortality rate concerning to cancer disease.

METHOD

This article presents the existing state of art techniques for identifying cancer affecting human organs based on machine learning models. The supported set of imaging operations are also elaborated for each type of Cancer.

CONCLUSION

The CAD tools are the aid for the diagnostic radiologists for preliminary investigations and detecting the nature of tumor cells.

Radiologist Variation in the Rates of Follow-up Imaging Recommendations Made for Pulmonary Nodules

OBJECTIVE

Determine whether differences exist in rates of follow-up recommendations made for pulmonary nodules after accounting for multiple patient and radiologist factors.

METHODS

This Institutional Review Board-approved, retrospective study was performed at an urban academic quaternary care hospital. We analyzed 142,001 chest and abdominal CT reports from January 1, 2016, to December 31, 2018, from abdominal, thoracic, and emergency radiology subspecialty divisions. A previously validated natural language processing (NLP) tool identified 24,512 reports documenting pulmonary nodule(s), excluding reports NLP-positive for lung cancer. A second validated NLP tool identified reports with follow-up recommendations specifically for pulmonary nodules. Multivariable logistic regression was used to determine the likelihood of pulmonary nodule follow-up recommendation. Interradiologist variability was quantified within subspecialty divisions.

RESULTS

NLP classified 4,939 of 24,512 (20.1%) reports as having a follow-up recommendation for pulmonary nodule. Male patients comprised 45.3% (11,097) of the patient cohort; average patient age was 61.4 years (±14.1 years). The majority of reports were from outpatient studies (62.7%, 15,376 of 24,512), were chest CTs (75.9%, 18,615 of 24,512), and were interpreted by thoracic radiologists (63.7%, 15,614 of 24,512). In multivariable analysis, studies for male patients (odds ratio [OR]: 0.9 [0.8-0.9]) and abdominal CTs (OR: 0.6 [0.6-0.7] compared with chest CT) were less likely to have a pulmonary nodule follow-up recommendation. Older patients had higher rates of follow-up recommendation (OR: 1.01 for each additional year). Division-level analysis showed up to 4.3-fold difference between radiologists in the probability of making a follow-up recommendation for a pulmonary nodule.

DISCUSSION

Significant differences exist in the probability of making a follow-up recommendation for pulmonary nodules among radiologists within the same subspecialty division.

Adding Value in Radiology Reporting

The major goal of the radiology report is to deliver timely, accurate, and actionable information to the patient care team and the patient. Structured reporting offers multiple advantages over traditional free-text reporting, including reduction in diagnostic error, comprehensiveness, adherence to national consensus guidelines, revenue capture, data collection, and research. Various technological innovations enhance integration of structured reporting into everyday clinical practice. This review discusses the benefits of innovations in radiology reporting to the clinical decision process, the patient experience, the cost of imaging, and the overall contributions to the health of the population. Future directions, including the use of artificial intelligence, are reviewed.

Applications of Artificial Intelligence in Musculoskeletal Imaging: From the Request to the Report

Artificial intelligence (AI) will transform every step in the imaging value chain, including interpretive and noninterpretive components. Radiologists should familiarize themselves with AI developments to become leaders in their clinical implementation. This article explores the impact of AI through the entire imaging cycle of musculoskeletal radiology, from the placement of the requisition to the generation of the report, with an added Canadian perspective. Noninterpretive tasks which may be assisted by AI include the ordering of appropriate imaging tests, automatic exam protocoling, optimized scheduling, shorter magnetic resonance imaging acquisition time, computed tomography imaging with reduced artifact and radiation dose, and new methods of generation and utilization of radiology reports. Applications of AI for image interpretation consist of the determination of bone age, body composition measurements, screening for osteoporosis, identification of fractures, evaluation of segmental spine pathology, detection and temporal monitoring of osseous metastases, diagnosis of primary bone and soft tissue tumors, and grading of osteoarthritis.

Identifying diagnosis evidence of cardiogenic stroke from Chinese echocardiograph reports

BACKGROUND

Cardiogenic stroke has increasing morbidity in China and brought economic burden to patient families. In cardiogenic stroke diagnosis, echocardiograph examination is one of the most important examinations. Sonographers will investigate patients' heart via echocardiograph, and describe them in the echocardiograph reports. In this study, we developed a machine learning model to automatically identify diagnosis evidences of cardiogenic stroke providing to neurologist for clinical decision making.

METHODS

We collected 4188 Chinese echocardiograph reports of 4018 patients, with average length 177 Chinese characters in free-text style. Collaborating with neurologists and sonographers, we summarized 149 phrases on diagnosis evidence of cardiogenic stroke such as "" (severe mitral stenosis), "" (aortic valve degeneration) and so on. Furthermore, we developed an annotated corpus via mapping 149 phrases to the 4188 reports. We selected 11 most frequent diagnosis evidence types such as "" (mitral stenosis) for further identifying. The generated corpus is divided into training set and testing set in the ratio of 8:2, which is used to train and validate a machine learning model to identify the evidence of cardiogenic stroke using BiLSTM-CRF algorithm.

RESULTS

Our machine learning method achieved the average performance on the diagnosis evidence identification is 98.03, 90.17 and 93.94% respectively. In addition, our method is capable to identify the novel diagnosis evidence of cardiogenic stroke description such as "-" (mitral stenosis), "" (aortic valve calcification) et al. CONCLUSIONS: In this study, we analyze the structure of the echocardiograph reports and summarized 149 phrases on diagnosis evidence of cardiogenic stroke. We use the phrases to generate an annotated corpus automatically, which greatly reduces the cost of manual annotation. The model trained based on the corpus also has a good performance on the testing set. The method of automatically identifying diagnosis evidence of cardiogenic stroke proposed in this study will be further refined in the practice.

Extraction and Analysis of Clinically Important Follow-up Recommendations in a Large Radiology Dataset

Communication of follow-up recommendations when abnormalities are identified on imaging studies is prone to error. In this paper, we present a natural language processing approach based on deep learning to automatically identify clinically important recommendations in radiology reports. Our approach first identifies the recommendation sentences and then extracts reason, test, and time frame of the identified recommendations. To train our extraction models, we created a corpus of 1367 radiology reports annotated for recommendation information. Our extraction models achieved 0.93 f-score for recommendation sentence, 0.65 f-score for reason, 0.73 f-score for test, and 0.84 f-score for time frame. We applied the extraction models to a set of over 3.3 million radiology reports and analyzed the adherence of follow-up recommendations.

Statistical modeling can determine what factors are predictive of appropriate follow-up in patients presenting with incidental pulmonary nodules on CT

PURPOSE

To assess the performance of statistical modeling in predicting follow-up adherence of incidentally detected pulmonary nodules (IPN) on CT, based on patient variables (PV), radiology report related variables (RRRV) and physician-patient communication variables (PPCV).

METHODS

200 patients with IPN on CT were retrospectively identified and randomly selected. PV (age, gender, smoking status, ethnicity), RRRV (nodule size, patient context, whether follow-up recommendations were provided) and PPCV (whether referring physician documented IPN and ordered follow-up on the electronic medical record) were recorded. Primary outcome was whether patients received appropriate follow-up within +/- 1 month of the recommended time frame. Statistical methods included logistic regression and machine learning (K-nearest neighbors and support vector machine).

RESULTS

Adherence was low, with or without recommendations provided in the radiology report (23.4 %-27.4 %). Whether the referring physician ordered follow-up was the dominant predictor of adherence in all models. The following variables were statistically significant predictors of whether referring physician ordered follow-up: recommendations provided in the radiology report, smoking status, patient context and nodule size (FDR logworth of respectively 21.18, 11.66, 2.35, 1.63, p < 0.05). Prediction accuracy varied from 72 % (PV) to 93 % (PPCV, all variables).

CONCLUSION

PPCV are the most important predictors of adherence. Amongst all variables, patient context, smoking status, nodule size, and whether the radiologist provided follow-up recommendations in the report were all statistically significant predictors of patient follow-up adherence, supporting the utility of statistical modeling for analytics, quality assurance and optimization of outcomes related to IPN.

Comparison of deep learning models for natural language processing-based classification of non-English head CT reports

PURPOSE

Natural language processing (NLP) can be used for automatic flagging of radiology reports. We assessed deep learning models for classifying non-English head CT reports.

METHODS

We retrospectively collected head CT reports (2011-2018). Reports were signed in Hebrew. Emergency department (ED) reports of adult patients from January to February for each year (2013-2018) were manually labeled. All other reports were used to pre-train an embedding layer. We explored two use cases: (1) general labeling use case, in which reports were labeled as normal vs. pathological; (2) specific labeling use case, in which reports were labeled as with and without intra-cranial hemorrhage. We tested long short-term memory (LSTM) and LSTM-attention (LSTM-ATN) networks for classifying reports. We also evaluated the improvement of adding Word2Vec word embedding. Deep learning models were compared with a bag-of-words (BOW) model.

RESULTS

We retrieved 176,988 head CT reports for pre-training. We manually labeled 7784 reports as normal (46.3%) or pathological (53.7%), and 7.1% with intra-cranial hemorrhage. For the general labeling, LSTM-ATN-Word2Vec showed the best results (AUC = 0.967 ± 0.006, accuracy 90.8% ± 0.01). For the specific labeling, all methods showed similar accuracies between 95.0 and 95.9%. Both LSTM-ATN-Word2Vec and BOW had the highest AUC (0.970).

CONCLUSION

For a general use case, word embedding using a large cohort of non-English head CT reports and ATN improves NLP performance. For a more specific task, BOW and deep learning showed similar results. Models should be explored and tailored to the NLP task.

Artificial Intelligence Pertaining to Cardiothoracic Imaging and Patient Care: Beyond Image Interpretation

Artificial intelligence (AI) is a broad field of computational science that includes many subsets. Today the most widely used subset in medical imaging is machine learning (ML). Many articles have focused on the use of ML for pattern recognition to detect and potentially diagnose various pathologies. However, AI algorithm development is now directed toward workflow management. AI can impact patient care at multiple stages of their imaging experience and assist in efficient and effective scheduling, imaging performance, worklist prioritization, image interpretation, and quality assurance. The purpose of this manuscript was to review the potential AI applications in radiology focusing on workflow management and discuss how ML will affect cardiothoracic imaging.

Deep Learning for Natural Language Processing in Radiology-Fundamentals and a Systematic Review

PURPOSE

Natural language processing (NLP) enables conversion of free text into structured data. Recent innovations in deep learning technology provide improved NLP performance. We aimed to survey deep learning NLP fundamentals and review radiology-related research.

METHODS

This systematic review was reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. We searched for deep learning NLP radiology studies published up to September 2019. MEDLINE, Scopus, and Google Scholar were used as search databases.

RESULTS

Ten relevant studies published between 2018 and 2019 were identified. Deep learning models applied for NLP in radiology are convolutional neural networks, recurrent neural networks, long short-term memory networks, and attention networks. Deep learning NLP applications in radiology include flagging of diagnoses such as pulmonary embolisms and fractures, labeling follow-up recommendations, and automatic selection of imaging protocols. Deep learning NLP models perform as well as or better than traditional NLP models.

CONCLUSION

Research and use of deep learning NLP in radiology is increasing. Acquaintance with this technology can help prepare radiologists for the coming changes in their field.

Variation in Follow-up Imaging Recommendations in Radiology Reports: Patient, Modality, and Radiologist Predictors

Background Variation between radiologists when making recommendations for additional imaging and associated factors are, to the knowledge of the authors, unknown. Clear identification of factors that account for variation in follow-up recommendations might prevent unnecessary tests for incidental or ambiguous image findings. Purpose To determine incidence and identify factors associated with follow-up recommendations in radiology reports from multiple modalities, patient care settings, and imaging divisions. Materials and Methods This retrospective study analyzed 318 366 reports obtained from diagnostic imaging examinations performed at a large urban quaternary care hospital from January 1 to December 31, 2016, excluding breast and US reports. A subset of 1000 reports were randomly selected and manually annotated to train and validate a machine learning algorithm to predict whether a report included a follow-up imaging recommendation (training-and-validation set consisted of 850 reports and test set of 150 reports). The trained algorithm was used to classify 318 366 reports. Multivariable logistic regression was used to determine the likelihood of follow-up recommendation. Additional analysis by imaging subspecialty division was performed, and intradivision and interradiologist variability was quantified. Results The machine learning algorithm classified 38 745 of 318 366 (12.2%) reports as containing follow-up recommendations. Average patient age was 59 years ± 17 (standard deviation); 45.2% (143 767 of 318 366) of reports were from male patients. Among 65 radiologists, 57% (37 of 65) were men. At multivariable analysis, older patients had higher rates of follow-up recommendations (odds ratio [OR], 1.01 [95% confidence interval {CI}: 1.01, 1.01] for each additional year), male patients had lower rates of follow-up recommendations (OR, 0.9; 95% CI: 0.9, 1.0), and follow-up recommendations were most common among CT studies (OR, 4.2 [95% CI: 4.0, 4.4] compared with radiography). Radiologist sex (P = .54), presence of a trainee (P = .45), and years in practice (P = .49) were not significant predictors overall. A division-level analysis showed 2.8-fold to 6.7-fold interradiologist variation. Conclusion Substantial interradiologist variation exists in the probability of recommending a follow-up examination in a radiology report, after adjusting for patient, examination, and radiologist factors. © RSNA, 2019 See also the editorial by Russell in this issue.