The association of clinically relevant variables with chest radiograph lung disease burden quantified in real-time by radiologists upon initial presentation in individuals hospitalized with COVID-19

Evaluating the accuracy of lung-RADS score extraction from radiology reports: Manual entry versus natural language processing

INTRODUCTION

Radiology scoring systems are critical to the success of lung cancer screening (LCS) programs, impacting patient care, adherence to follow-up, data management and reporting, and program evaluation. LungCT ScreeningReporting and Data System (Lung-RADS) is a structured radiology scoring system that provides recommendations for LCS follow-up that are utilized (a) in clinical care and (b) by LCS programs monitoring rates of adherence to follow-up. Thus, accurate reporting and reliable collection of Lung-RADS scores are fundamental components of LCS program evaluation and improvement. Unfortunately, due to variability in radiology reports, extraction of Lung-RADS scores is non-trivial, and best practices do not exist. The purpose of this project is to compare mechanisms to extract Lung-RADS scores from free-text radiology reports.

METHODS

We retrospectively analyzed reports of LCS low-dose computed tomography (LDCT) examinations performed at a multihospital integrated healthcare network in New York State between January 2016 and July 2023. We compared three methods of Lung-RADS score extraction: manual physician entry at time of report creation, manual LCS specialist entry after report creation, and an internally developed, rule-based natural language processing (NLP) algorithm. Accuracy, recall, precision, and completeness (i.e., the proportion of LCS exams to which a Lung-RADS score has been assigned) were compared between the three methods.

RESULTS

The dataset includes 24,060 LCS examinations on 14,243 unique patients. The mean patient age was 65 years, and most patients were male (54 %) and white (75 %). Completeness rate was 65 %, 68 %, and 99 % for radiologists' manual entry, LCS specialists' entry, and NLP algorithm, respectively. Accuracy, recall, and precision were high across all extraction methods (>94 %), though the NLP-based approach was consistently higher than both manual entries in all metrics.

DISCUSSION

An NLP-based method of LCS score determination is an efficient and more accurate means of extracting Lung-RADS scores than manual review and data entry. NLP-based methods should be considered best practice for extracting structured Lung-RADS scores from free-text radiology reports.

The Neiman Imaging Comorbidity Index: Development and Validation in a National Commercial Claims Database

OBJECTIVE

To build the Neiman Imaging Comorbidity Index (NICI), based on variables available in claims datasets, which provides good discrimination of an individual's chance of receiving advanced imaging (CT, MR, PET), and thus, utility as a control variable in research.

METHODS

This retrospective study used national commercial claims data from Optum's deidentified Clinformatics Data Mart database from the period January 1, 2018 to December 31, 2019. Individuals with continuous enrollment during this 2-year study period were included. Lasso (least absolute shrinkage and selection operator) regression was used to predict the chance of receiving advanced imaging in 2019 based on the presence of comorbidities in 2018. A numerical index was created in a development cohort (70% of the total dataset) using weights assigned to each comorbidity, based on regression β coefficients. Internal validation of assigned scores was performed in the remaining 30% of claims, with comparison to the commonly used Charlson Comorbidity Index.

RESULTS

The final sample (development and validation cohorts) included 10,532,734 beneficiaries, of whom 2,116,348 (20.1%) received advanced imaging. After model development, the NICI included nine comorbidities. In the internal validation set, the NICI achieved good discrimination of receipt of advanced imaging with a C statistic of 0.709 (95% confidence interval [CI] 0.708-0.709), which predicted advanced imaging better than the CCI (C 0.692, 95% CI 0.691-0.692). Controlling for age and sex yielded better discrimination (C 0.748, 95% CI 0.748-0.749).

DISCUSSION

The NICI is an easily calculated measure of comorbidity burden that can be used to adjust for patients' chances of receiving advanced imaging. Future work should explore external validation of the NICI.