Deep Learning to Classify Radiology Free-Text Reports

Purpose To evaluate the performance of a deep learning convolutional neural network (CNN) model compared with a traditional natural language processing (NLP) model in extracting pulmonary embolism (PE) findings from thoracic computed tomography (CT) reports from two institutions. Materials and Methods Contrast material-enhanced CT examinations of the chest performed between January 1, 1998, and January 1, 2016, were selected. Annotations by two human radiologists were made for three categories: the presence, chronicity, and location of PE. Classification of performance of a CNN model with an unsupervised learning algorithm for obtaining vector representations of words was compared with the open-source application PeFinder. Sensitivity, specificity, accuracy, and F1 scores for both the CNN model and PeFinder in the internal and external validation sets were determined. Results The CNN model demonstrated an accuracy of 99% and an area under the curve value of 0.97. For internal validation report data, the CNN model had a statistically significant larger F1 score (0.938) than did PeFinder (0.867) when classifying findings as either PE positive or PE negative, but no significant difference in sensitivity, specificity, or accuracy was found. For external validation report data, no statistical difference between the performance of the CNN model and PeFinder was found. Conclusion A deep learning CNN model can classify radiology free-text reports with accuracy equivalent to or beyond that of an existing traditional NLP model. ^© RSNA, 2017 Online supplemental material is available for this article.

What Does a Shoulder MRI Cost the Consumer?

BACKGROUND

More than 100 MRIs per 1000 inhabitants are performed in the United States annually, more than almost every other country. Little is known regarding the cost of obtaining an MRI and factors associated with differences in cost.

QUESTIONS/PURPOSES

By surveying all hospital-owned and independent imaging centers in Iowa, we wished to determine (1) the cost to the consumer of obtaining a noncontrast shoulder MRI, (2) the frequency and magnitude of discounts provided, and (3) factors associated with differences in cost including location (hospital-owned or independent) and Centers for Medicare & Medicaid Services designation (rural, urban, and critical access).

METHODS

There were 71 hospitals and 26 independent imaging centers that offered MRI services in Iowa. Each site was contacted via telephone and posed a scripted request for the cost of the technical component of a noncontrast shoulder MRI. Radiologists' reading fees were not considered. Statistical analysis was performed using standard methods and significance was defined as a probability less than 0.05.

RESULTS

The mean technical component cost to consumers for an MRI was USD 1874 ± USD 694 (range, USD 500-USD 4000). Discounts were offered by 49% of imaging centers, with a mean savings of 21%. Factors associated with increased cost include hospital-owned imaging centers (USD 2062 ± USD 664 versus USD 1400 ± USD 441 at independent imaging centers; p < 0.001; mean difference, USD 662; 95% CI, USD 351-USD 893) and rural imaging centers, unless designated as a critical access hospital (USD 2213 ± USD 668 versus USD 1794 ± USD 680; p = 0.0202; mean difference, USD 419; 95% CI, USD 66-USD 772).

CONCLUSIONS

In Iowa, the cost to the consumer of a shoulder MRI is significantly less at independent imaging centers compared with hospital-owned centers. Referring physicians and healthcare consumers should be aware that there may be substantial price discrepancies between centers that provide advanced imaging services.

LEVEL OF EVIDENCE

Level IV, Economic and decision analysis.

Performance of a Machine Learning Classifier of Knee MRI Reports in Two Large Academic Radiology Practices: A Tool to Estimate Diagnostic Yield

OBJECTIVE

The purpose of this study is to evaluate the performance of a natural language processing (NLP) system in classifying a database of free-text knee MRI reports at two separate academic radiology practices.

MATERIALS AND METHODS

An NLP system that uses terms and patterns in manually classified narrative knee MRI reports was constructed. The NLP system was trained and tested on expert-classified knee MRI reports from two major health care organizations. Radiology reports were modeled in the training set as vectors, and a support vector machine framework was used to train the classifier. A separate test set from each organization was used to evaluate the performance of the system. We evaluated the performance of the system both within and across organizations. Standard evaluation metrics, such as accuracy, precision, recall, and F1 score (i.e., the weighted average of the precision and recall), and their respective 95% CIs were used to measure the efficacy of our classification system.

RESULTS

The accuracy for radiology reports that belonged to the model's clinically significant concept classes after training data from the same institution was good, yielding an F1 score greater than 90% (95% CI, 84.6-97.3%). Performance of the classifier on cross-institutional application without institution-specific training data yielded F1 scores of 77.6% (95% CI, 69.5-85.7%) and 90.2% (95% CI, 84.5-95.9%) at the two organizations studied.

CONCLUSION

The results show excellent accuracy by the NLP machine learning classifier in classifying free-text knee MRI reports, supporting the institution-independent reproducibility of knee MRI report classification. Furthermore, the machine learning classifier performed well on free-text knee MRI reports from another institution. These data support the feasibility of multiinstitutional classification of radiologic imaging text reports with a single machine learning classifier without requiring institution-specific training data.

Patient Attitudes Toward Orthopedic Surgeon Ownership of Related Ancillary Businesses

BACKGROUND

Physician ownership of businesses related to orthopedic surgery, such as surgery centers, has been criticized as potentially leading to misuse of health care resources. The purpose of this study was to determine patients' attitudes toward surgeon ownership of orthopedic-related businesses.

METHODS

We surveyed 280 consecutive patients at 2 centers regarding their attitudes toward surgeon ownership of orthopedic-related businesses using an anonymous questionnaire. Three surgeon ownership scenarios were presented: (1) owning a surgery center, (2) physical therapy (PT), and (3) imaging facilities (eg, Magnetic Resonance Imaging scanner).

RESULTS

Two hundred fourteen patients (76%) completed the questionnaire. The majority agreed that it is ethical for a surgeon to own a surgery center (73%), PT practice (77%), or imaging facility (77%). Most (>67%) indicated that their surgeon owning such a business would have no effect on the trust they have in their surgeon. Although >70% agreed that a surgeon in all 3 scenarios would make the same treatment decisions, many agreed that such surgeons might perform more surgery (47%), refer more patients to PT (61%), or order more imaging (58%). Patients favored surgeon autonomy, however, believing that surgeons should be allowed to own such businesses (78%). Eighty-five percent agreed that patients should be informed if their surgeon owns an orthopedic-related business.

CONCLUSION

Although patients express concern over and desire disclosure of surgeon ownership of orthopedic-related businesses, the majority believes that it is an ethical practice and feel comfortable receiving care at such a facility.

Physician self-referral and imaging use appropriateness: negative cervical spine MRI frequency as an assessment metric

BACKGROUND AND PURPOSE

Imaging self-referral is increasingly cited as a contributor to diagnostic imaging overuse. The purpose of this study was to determine whether ownership of MR imaging equipment by ordering physicians influences the frequency of negative cervical spine MR imaging findings.

MATERIALS AND METHODS

A retrospective review was performed of 500 consecutive cervical spine MRIs ordered by 2 separate referring-physician groups serving the same geographic community. The first group owned the scanners used and received technical fees for their use, while the second group did not. Final reports were reviewed, and for each group, the percentage of negative study findings and the frequency of abnormalities were calculated. The number of concomitant shoulder MRIs was recorded.

RESULTS

Five hundred MRIs meeting inclusion criteria were reviewed (250 with financial interest, 250 with no financial interest). Three hundred fifty-two had negative findings (190 with financial interest, 162 with no financial interest); there were 17.3% more scans with negative findings in the financial interest group (P = .006). Among scans with positive findings, there was no significant difference in the mean number of lesions per scan, controlled for age (1.90 with financial interest, 2.19 with no financial interest; P = .23). Patients in the financial interest group were more likely to undergo concomitant shoulder MR imaging (24 with financial interest, 11 with no financial interest; P = .02).

CONCLUSIONS

Cervical spine MRIs referred by physicians with a financial interest in the imaging equipment used were significantly more likely to have negative findings. There was otherwise a highly similar distribution and severity of disease between the 2 patient samples. Patients in the financial interest group were more likely to undergo concomitant shoulder MR imaging.