Natural Language Processing in Nephrology

Use of Generative AI to Identify Helmet Status Among Patients With Micromobility-Related Injuries From Unstructured Clinical Notes

Importance

Large language models (LLMs) have potential to increase the efficiency of information extraction from unstructured clinical notes in electronic medical records.

Objective

To assess the utility and reliability of an LLM, ChatGPT-4 (OpenAI), to analyze clinical narratives and identify helmet use status of patients injured in micromobility-related accidents.

Design, Setting, and Participants

This cross-sectional study used publicly available, deidentified 2019 to 2022 data from the US Consumer Product Safety Commission's National Electronic Injury Surveillance System, a nationally representative stratified probability sample of 96 hospitals in the US. Unweighted estimates of e-bike, bicycle, hoverboard, and powered scooter-related injuries that resulted in an emergency department visit were used. Statistical analysis was performed from November 2023 to April 2024.

Main Outcomes and Measures

Patient helmet status (wearing vs not wearing vs unknown) was extracted from clinical narratives using (1) a text string search using researcher-generated text strings and (2) the LLM by prompting the system with low-, intermediate-, and high-detail prompts. The level of agreement between the 2 approaches across all 3 prompts was analyzed using Cohen κ test statistics. Fleiss κ was calculated to measure the test-retest reliability of the high-detail prompt across 5 new chat sessions and days. Performance statistics were calculated by comparing results from the high-detail prompt to classifications of helmet status generated by researchers reading the clinical notes (ie, a criterion standard review).

Results

Among 54 569 clinical notes, moderate (Cohen κ = 0.74 [95% CI, 0.73-0.75) and weak (Cohen κ = 0.53 [95% CI, 0.52-0.54]) agreement were found between the text string-search approach and the LLM for the low- and intermediate-detail prompts, respectively. The high-detail prompt had almost perfect agreement (κ = 1.00 [95% CI, 1.00-1.00]) but required the greatest amount of time to complete. The LLM did not perfectly replicate its analyses across new sessions and days (Fleiss κ = 0.91 across 5 trials; P < .001). The LLM often hallucinated and was consistent in replicating its hallucinations. It also showed high validity compared with the criterion standard (n = 400; κ = 0.98 [95% CI, 0.96-1.00]).

Conclusions and Relevance

This study's findings suggest that although there are efficiency gains for using the LLM to extract information from clinical notes, the inadequate reliability compared with a text string-search approach, hallucinations, and inconsistent performance significantly hinder the potential of the currently available LLM.

Role of artificial intelligence in colorectal cancer

The sphere of artificial intelligence (AI) is ever expanding. Applications for clinical practice have been emerging over recent years. Although its uptake has been most prominent in endoscopy, this represents only one aspect of holistic patient care. There are a multitude of other potential avenues in which gastrointestinal care may be involved. We aim to review the role of AI in colorectal cancer as a whole. We performed broad scoping and focused searches of the applications of AI in the field of colorectal cancer. All trials including qualitative research were included from the year 2000 onwards. Studies were grouped into pre-operative, intra-operative and post-operative aspects. Pre-operatively, the major use is with endoscopic recognition. Colonoscopy has embraced the use for human derived classifications such as Narrow-band Imaging International Colorectal Endoscopic, Japan Narrow-band Imaging Expert Team, Paris and Kudo. However, novel detection and diagnostic methods have arisen from advances in AI classification. Intra-operatively, adjuncts such as image enhanced identification of structures and assessment of perfusion have led to improvements in clinical outcomes. Post-operatively, monitoring and surveillance have taken strides with potential socioeconomic and environmental savings. The uses of AI within the umbrella of colorectal surgery are multiple. We have identified existing technologies which are already augmenting cancer care. The future applications are exciting and could at least match, if not surpass human standards.

New marker for chronic kidney disease progression and mortality in medical-word virtual space

A new marker reflecting the pathophysiology of chronic kidney disease (CKD) has been desired for its therapy. In this study, we developed a virtual space where data in medical words and those of actual CKD patients were unified by natural language processing and category theory. A virtual space of medical words was constructed from the CKD-related literature (n = 165,271) using Word2Vec, in which 106,612 words composed a network. The network satisfied vector calculations, and retained the meanings of medical words. The data of CKD patients of a cohort study for 3 years (n = 26,433) were transformed into the network as medical-word vectors. We let the relationship between vectors of patient data and the outcome (dialysis or death) be a marker (inner product). Then, the inner product accurately predicted the outcomes: C-statistics of 0.911 (95% CI 0.897, 0.924). Cox proportional hazards models showed that the risk of the outcomes in the high-inner-product group was 21.92 (95% CI 14.77, 32.51) times higher than that in the low-inner-product group. This study showed that CKD patients can be treated as a network of medical words that reflect the pathophysiological condition of CKD and the risks of CKD progression and mortality.

Symptom Trajectories in the Electronic Health Record during the Transition to Maintenance Dialysis

How symptoms recorded in the electronic health record change during the transition to dialysis has not been fully explored. We used the Optum deidentified Integrated Claims-Clinical dataset to identify individuals with CKD stages 4 or 5 who transitioned to dialysis. We searched structured data elements from clinical notes, identified by natural language processing, for symptoms recorded across weekly intervals in the 6 months before and after dialysis initiation and estimated changes in the odds of a symptom being recorded with an interrupted time series analysis using segmented logistic regression. The cohort comprised 728 individuals (aged 68±13 years, 44% women, 56% White, 30% Black). Before dialysis initiation, 83% were recorded as having pain, 68% fatigue/weakness, 66% shortness of breath, 61% nausea/vomiting, and 37% difficulty concentrating. Before dialysis initiation, odds of pain being recorded increased (slope: odds ratio [OR] 1.02 per week, 95% confidence interval [CI], 1.01 to 1.03); initiation was associated with a decrease (intercept change: OR 0.70, 95% CI, 0.59 to 0.82). After initiation, odds of pain were unchanged (postdialysis slope: OR 1.00 per week, 95% CI, 0.99 to 1.01), although this represented an improved trajectory relative to the predialysis period (change in slope: OR 0.98 per week, 95% CI, 0.96 to 0.99). For fatigue/weakness, odds increased before initiation (OR 1.03 per week, 95% CI, 1.02 to 1.04) but decreased on initiation (OR 0.62, 95% CI, 0.51 to 0.75) and thereafter (OR 0.98 per week, 95% CI, 0.97 to 0.99), representing a reduction in slope (OR 0.95 per week, 95% CI, 0.94 to 0.97). Patterns for shortness of breath, nausea/vomiting, and difficulty concentrating were similar to those of pain. Thus, the odds of five key symptoms being recorded in the electronic health record increased over time in the 6 months before dialysis initiation, decreased immediately on initiation, and, generally, remained unchanged in the 6 months thereafter.

A novel staging system derived from natural language processing of pathology reports to predict prognostic outcomes of pancreatic cancer: a retrospective cohort study

OBJECTIVE

To construct a novel tumor-node-morphology (TNMor) staging system derived from natural language processing (NLP) of pathology reports to predict outcomes of pancreatic ductal adenocarcinoma.

METHOD

This retrospective study with 1657 participants was based on a large referral center and The Cancer Genome Atlas Program (TCGA) dataset. In the training cohort, NLP was used to extract and screen prognostic predictors from pathology reports to develop the TNMor system, which was further evaluated with the tumor-node-metastasis (TNM) system in the internal and external validation cohort, respectively. Main outcomes were evaluated by the log-rank test of Kaplan-Meier curves, the concordance index (C-index), and the area under the receiver operating curve (AUC).

RESULTS

The precision, recall, and F1 scores of the NLP model were 88.83, 89.89, and 89.21%, respectively. In Kaplan-Meier analysis, survival differences between stages in the TNMor system were more significant than that in the TNM system. In addition, our system provided an improved C-index (internal validation, 0.58 vs. 0.54, P <0.001; external validation, 0.64 vs. 0.63, P <0.001), and higher AUCs for 1, 2, and 3-year survival (internal validation: 0.62 vs. 0.54, P <0.001; 0.64 vs. 0.60, P= 0.017; 0.69 vs. 0.62, P= 0.001; external validation: 0.69 vs. 0.65, P= 0.098; 0.68 vs. 0.64, P= 0.154; 0.64 vs. 0.55, P= 0.032, respectively). Finally, our system was particularly beneficial for precise stratification of patients receiving adjuvant therapy, with an improved C-index (0.61 vs. 0.57, P <0.001), and higher AUCs for 1-year, 2-year, and 3-year survival (0.64 vs. 0.57, P <0.001; 0.64 vs. 0.58, P <0.001; 0.67 vs. 0.61, P <0.001; respectively) compared with the TNM system.

CONCLUSION

These findings suggest that the TNMor system performed better than the TNM system in predicting pancreatic ductal adenocarcinoma prognosis. It is a promising system to screen risk-adjusted strategies for precision medicine.