Using natural language processing for identification of herpes zoster ophthalmicus cases to support population-based study

Risk Factors for the Development of Ocular Complications in Herpes Zoster Ophthalmicus and Zoster Vaccine Utilization in a Large, Urban Health System

PURPOSE

To characterize the epidemiology of herpes zoster (HZ) and herpes zoster ophthalmicus (HZO) in an urban hospital system and determine risk factors associated with developing ocular complications in HZO. To report the frequency of shingles vaccination and HZ reactivation following shingles vaccination in this population.

METHODS

A retrospective cohort study was conducted on patients seen at the University of Illinois Hospital system from January 1, 2010 to December 1, 2021 with HZ and HZO identified by diagnosis code. Charts of HZO patients seen within 1 year of diagnosis were abstracted. Multivariable logistic regression analysis identified factors associated with the development of ocular complications in HZO.

RESULTS

During the study period, 3283 patients had HZ; mean age of onset was 52.3 years, 61.6% were female, and 37% were Black. HZO with ocular involvement was seen in 110 (3.4%) patients. Ocular complications developed in 40 (36.4%) patients; the most common complication was corneal scarring (70%). Age (odds ratio [OR] 1.04, 95%CI 1.0-1.1), female gender (OR 2.86, 95%CI 1.0-8.1), steroids at initial visit (4.46, 95%CI 1.4-14.6), and stromal keratitis (OR 3.45, 95% CI 1.2, 9.8) were associated with developing ocular complications. Of eligible populations, 5333 (1.5%) received shingles vaccination; 43 patients developed reactivation of HZ following vaccination.

CONCLUSIONS

In HZO, age, female gender, steroids at initial visit, and stromal keratitis are strongly associated with developing ocular complications. Shingles vaccination rates were low in this study population. Understanding potential for complications in HZ/HZO and vaccination uptake can help identify at risk populations to prevent disease.

Natural Language Processing Versus Diagnosis Code-Based Methods for Postherpetic Neuralgia Identification: Algorithm Development and Validation

Background

Diagnosis codes and prescription data are used in algorithms to identify postherpetic neuralgia (PHN), a debilitating complication of herpes zoster (HZ). Because of the questionable accuracy of codes and prescription data, manual chart review is sometimes used to identify PHN in electronic health records (EHRs), which can be costly and time-consuming.

Objective

This study aims to develop and validate a natural language processing (NLP) algorithm for automatically identifying PHN from unstructured EHR data and to compare its performance with that of code-based methods.

Methods

This retrospective study used EHR data from Kaiser Permanente Southern California, a large integrated health care system that serves over 4.8 million members. The source population included members aged ≥50 years who received an incident HZ diagnosis and accompanying antiviral prescription between 2018 and 2020 and had ≥1 encounter within 90-180 days of the incident HZ diagnosis. The study team manually reviewed the EHR and identified PHN cases. For NLP development and validation, 500 and 800 random samples from the source population were selected, respectively. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F-score, and Matthews correlation coefficient (MCC) of NLP and the code-based methods were evaluated using chart-reviewed results as the reference standard.

Results

The NLP algorithm identified PHN cases with a 90.9% sensitivity, 98.5% specificity, 82% PPV, and 99.3% NPV. The composite scores of the NLP algorithm were 0.89 (F-score) and 0.85 (MCC). The prevalences of PHN in the validation data were 6.9% (reference standard), 7.6% (NLP), and 5.4%-13.1% (code-based). The code-based methods achieved a 52.7%-61.8% sensitivity, 89.8%-98.4% specificity, 27.6%-72.1% PPV, and 96.3%-97.1% NPV. The F-scores and MCCs ranged between 0.45 and 0.59 and between 0.32 and 0.61, respectively.

Conclusions

The automated NLP-based approach identified PHN cases from the EHR with good accuracy. This method could be useful in population-based PHN research.

Utility of artificial intelligence-based large language models in ophthalmic care

PURPOSE

With the introduction of ChatGPT, artificial intelligence (AI)-based large language models (LLMs) are rapidly becoming popular within the scientific community. They use natural language processing to generate human-like responses to queries. However, the application of LLMs and comparison of the abilities among different LLMs with their human counterparts in ophthalmic care remain under-reported.

RECENT FINDINGS

Hitherto, studies in eye care have demonstrated the utility of ChatGPT in generating patient information, clinical diagnosis and passing ophthalmology question-based examinations, among others. LLMs' performance (median accuracy, %) is influenced by factors such as the iteration, prompts utilised and the domain. Human expert (86%) demonstrated the highest proficiency in disease diagnosis, while ChatGPT-4 outperformed others in ophthalmology examinations (75.9%), symptom triaging (98%) and providing information and answering questions (84.6%). LLMs exhibited superior performance in general ophthalmology but reduced accuracy in ophthalmic subspecialties. Although AI-based LLMs like ChatGPT are deemed more efficient than their human counterparts, these AIs are constrained by their nonspecific and outdated training, no access to current knowledge, generation of plausible-sounding 'fake' responses or hallucinations, inability to process images, lack of critical literature analysis and ethical and copyright issues. A comprehensive evaluation of recently published studies is crucial to deepen understanding of LLMs and the potential of these AI-based LLMs.

SUMMARY

Ophthalmic care professionals should undertake a conservative approach when using AI, as human judgement remains essential for clinical decision-making and monitoring the accuracy of information. This review identified the ophthalmic applications and potential usages which need further exploration. With the advancement of LLMs, setting standards for benchmarking and promoting best practices is crucial. Potential clinical deployment requires the evaluation of these LLMs to move away from artificial settings, delve into clinical trials and determine their usefulness in the real world.

Identification of recurrent atrial fibrillation using natural language processing applied to electronic health records

AIMS

This study aimed to develop and apply natural language processing (NLP) algorithms to identify recurrent atrial fibrillation (AF) episodes following rhythm control therapy initiation using electronic health records (EHRs).

METHODS AND RESULTS

We included adults with new-onset AF who initiated rhythm control therapies (ablation, cardioversion, or antiarrhythmic medication) within two US integrated healthcare delivery systems. A code-based algorithm identified potential AF recurrence using diagnosis and procedure codes. An automated NLP algorithm was developed and validated to capture AF recurrence from electrocardiograms, cardiac monitor reports, and clinical notes. Compared with the reference standard cases confirmed by physicians' adjudication, the F-scores, sensitivity, and specificity were all above 0.90 for the NLP algorithms at both sites. We applied the NLP and code-based algorithms to patients with incident AF (n = 22 970) during the 12 months after initiating rhythm control therapy. Applying the NLP algorithms, the percentages of patients with AF recurrence for sites 1 and 2 were 60.7% and 69.9% (ablation), 64.5% and 73.7% (cardioversion), and 49.6% and 55.5% (antiarrhythmic medication), respectively. In comparison, the percentages of patients with code-identified AF recurrence for sites 1 and 2 were 20.2% and 23.7% for ablation, 25.6% and 28.4% for cardioversion, and 20.0% and 27.5% for antiarrhythmic medication, respectively.

CONCLUSION

When compared with a code-based approach alone, this study's high-performing automated NLP method identified significantly more patients with recurrent AF. The NLP algorithms could enable efficient evaluation of treatment effectiveness of AF therapies in large populations and help develop tailored interventions.

Applications of natural language processing in ophthalmology: present and future

Advances in technology, including novel ophthalmic imaging devices and adoption of the electronic health record (EHR), have resulted in significantly increased data available for both clinical use and research in ophthalmology. While artificial intelligence (AI) algorithms have the potential to utilize these data to transform clinical care, current applications of AI in ophthalmology have focused mostly on image-based deep learning. Unstructured free-text in the EHR represents a tremendous amount of underutilized data in big data analyses and predictive AI. Natural language processing (NLP) is a type of AI involved in processing human language that can be used to develop automated algorithms using these vast quantities of available text data. The purpose of this review was to introduce ophthalmologists to NLP by (1) reviewing current applications of NLP in ophthalmology and (2) exploring potential applications of NLP. We reviewed current literature published in Pubmed and Google Scholar for articles related to NLP and ophthalmology, and used ancestor search to expand our references. Overall, we found 19 published studies of NLP in ophthalmology. The majority of these publications (16) focused on extracting specific text such as visual acuity from free-text notes for the purposes of quantitative analysis. Other applications included: domain embedding, predictive modeling, and topic modeling. Future ophthalmic applications of NLP may also focus on developing search engines for data within free-text notes, cleaning notes, automated question-answering, and translating ophthalmology notes for other specialties or for patients, especially with a growing interest in open notes. As medicine becomes more data-oriented, NLP offers increasing opportunities to augment our ability to harness free-text data and drive innovations in healthcare delivery and treatment of ophthalmic conditions.

New meaning for NLP: the trials and tribulations of natural language processing with GPT-3 in ophthalmology

Natural language processing (NLP) is a subfield of machine intelligence focused on the interaction of human language with computer systems. NLP has recently been discussed in the mainstream media and the literature with the advent of Generative Pre-trained Transformer 3 (GPT-3), a language model capable of producing human-like text. The release of GPT-3 has also sparked renewed interest on the applicability of NLP to contemporary healthcare problems. This article provides an overview of NLP models, with a focus on GPT-3, as well as discussion of applications specific to ophthalmology. We also outline the limitations of GPT-3 and the challenges with its integration into routine ophthalmic care.

Application of Artificial Intelligence in Community-Based Primary Health Care: Systematic Scoping Review and Critical Appraisal

BACKGROUND

Research on the integration of artificial intelligence (AI) into community-based primary health care (CBPHC) has highlighted several advantages and disadvantages in practice regarding, for example, facilitating diagnosis and disease management, as well as doubts concerning the unintended harmful effects of this integration. However, there is a lack of evidence about a comprehensive knowledge synthesis that could shed light on AI systems tested or implemented in CBPHC.

OBJECTIVE

We intended to identify and evaluate published studies that have tested or implemented AI in CBPHC settings.

METHODS

We conducted a systematic scoping review informed by an earlier study and the Joanna Briggs Institute (JBI) scoping review framework and reported the findings according to PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analysis-Scoping Reviews) reporting guidelines. An information specialist performed a comprehensive search from the date of inception until February 2020, in seven bibliographic databases: Cochrane Library, MEDLINE, EMBASE, Web of Science, Cumulative Index to Nursing and Allied Health Literature (CINAHL), ScienceDirect, and IEEE Xplore. The selected studies considered all populations who provide and receive care in CBPHC settings, AI interventions that had been implemented, tested, or both, and assessed outcomes related to patients, health care providers, or CBPHC systems. Risk of bias was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST). Two authors independently screened the titles and abstracts of the identified records, read the selected full texts, and extracted data from the included studies using a validated extraction form. Disagreements were resolved by consensus, and if this was not possible, the opinion of a third reviewer was sought. A third reviewer also validated all the extracted data.

RESULTS

We retrieved 22,113 documents. After the removal of duplicates, 16,870 documents were screened, and 90 peer-reviewed publications met our inclusion criteria. Machine learning (ML) (41/90, 45%), natural language processing (NLP) (24/90, 27%), and expert systems (17/90, 19%) were the most commonly studied AI interventions. These were primarily implemented for diagnosis, detection, or surveillance purposes. Neural networks (ie, convolutional neural networks and abductive networks) demonstrated the highest accuracy, considering the given database for the given clinical task. The risk of bias in diagnosis or prognosis studies was the lowest in the participant category (4/49, 4%) and the highest in the outcome category (22/49, 45%).

CONCLUSIONS

We observed variabilities in reporting the participants, types of AI methods, analyses, and outcomes, and highlighted the large gap in the effective development and implementation of AI in CBPHC. Further studies are needed to efficiently guide the development and implementation of AI interventions in CBPHC settings.

Deep learning-based natural language processing in ophthalmology: applications, challenges and future directions

PURPOSE OF REVIEW

Artificial intelligence (AI) is the fourth industrial revolution in mankind's history. Natural language processing (NLP) is a type of AI that transforms human language, to one that computers can interpret and process. NLP is still in the formative stages of development in healthcare, with promising applications and potential challenges in its applications. This review provides an overview of AI-based NLP, its applications in healthcare and ophthalmology, next-generation use case, as well as potential challenges in deployment.

RECENT FINDINGS

The integration of AI-based NLP systems into existing clinical care shows considerable promise in disease screening, risk stratification, and treatment monitoring, amongst others. Stakeholder collaboration, greater public acceptance, and advancing technologies will continue to shape the NLP landscape in healthcare and ophthalmology.

SUMMARY

Healthcare has always endeavored to be patient centric and personalized. For AI-based NLP systems to become an eventual reality in larger-scale applications, it is pertinent for key stakeholders to collaborate and address potential challenges in application. Ultimately, these would enable more equitable and generalizable use of NLP systems for the betterment of healthcare and society.

Identifying Cases of Shoulder Injury Related to Vaccine Administration (SIRVA) in the United States: Development and Validation of a Natural Language Processing Method (Preprint)

Background

Shoulder injury related to vaccine administration (SIRVA) accounts for more than half of all claims received by the National Vaccine Injury Compensation Program. However, due to the difficulty of finding SIRVA cases in large health care databases, population-based studies are scarce.

Objective

The goal of the research was to develop a natural language processing (NLP) method to identify SIRVA cases from clinical notes.

Methods

We conducted the study among members of a large integrated health care organization who were vaccinated between April 1, 2016, and December 31, 2017, and had subsequent diagnosis codes indicative of shoulder injury. Based on a training data set with a chart review reference standard of 164 cases, we developed an NLP algorithm to extract shoulder disorder information, including prior vaccination, anatomic location, temporality and causality. The algorithm identified 3 groups of positive SIRVA cases (definite, probable, and possible) based on the strength of evidence. We compared NLP results to a chart review reference standard of 100 vaccinated cases. We then applied the final automated NLP algorithm to a broader cohort of vaccinated persons with a shoulder injury diagnosis code and performed manual chart confirmation on a random sample of NLP-identified definite cases and all NLP-identified probable and possible cases.

Results

In the validation sample, the NLP algorithm had 100% accuracy for identifying 4 SIRVA cases and 96 cases without SIRVA. In the broader cohort of 53,585 vaccinations, the NLP algorithm identified 291 definite, 124 probable, and 52 possible SIRVA cases. The chart-confirmation rates for these groups were 95.5% (278/291), 67.7% (84/124), and 17.3% (9/52), respectively.

Conclusions

The algorithm performed with high sensitivity and reasonable specificity in identifying positive SIRVA cases. The NLP algorithm can potentially be used in future population-based studies to identify this rare adverse event, avoiding labor-intensive chart review validation.

The Epidemiology of Herpes Zoster in Immunocompetent, Unvaccinated Adults ≥50 Years Old: Incidence, Complications, Hospitalization, Mortality, and Recurrence

Background

Data on the epidemiology of herpes zoster (HZ), particularly in the unvaccinated immunocompetent population, are needed to assess disease burden and the potential impact of vaccination.

Methods

The study at a large health care organization comprised: (1) incidence estimated from immunocompetent adults aged ≥50 years unvaccinated with zoster vaccine live who had incident HZ in 2011–2015; (2) proportion of HZ-related nonpain complications assessed by double abstraction of electronic health records (EHRs) of 600 incident patients 2011–2015; (3) HZ-related hospitalizations among HZ patients diagnosed in 2015; (4) HZ-related death determined from automated data and EHRs; and (5) recurrent HZ identified from a cohort initially diagnosed with HZ in 2007–2008 and followed through 2016.

Results

HZ incidence rate was 9.92/1000 person-years (95% confidence interval [CI], 9.82–10.01). Proportions of cutaneous, neurologic, and other complications were 6.40% (95% CI,1.73%–11.07%), 0.77% (95% CI, .00%–2.36%), and 1.01% (95% CI, .00%–2.93%), respectively. Only 0.86% of patients had an HZ-related hospitalization. The case-fatality rate was 0.04%. Recurrence rate was 10.96/1000 person-years (95% CI, 10.18–11.79) with 10-year recurrence risk of 10.26% (95% CI, 9.36%–11.23%).

Conclusions

These recent HZ epidemiology data among an immunocompetent, unvaccinated population measure real-world disease burden.

Can antiepileptic efficacy and epilepsy variables be studied from electronic health records? A review of current approaches

As automated data extraction and natural language processing (NLP) are rapidly evolving, improving healthcare delivery by harnessing large data is garnering great interest. Assessing antiepileptic drug (AED) efficacy and other epilepsy variables pertinent to healthcare delivery remain a critical barrier to improving patient care. In this systematic review, we examined automatic electronic health record (EHR) extraction methodologies pertinent to epilepsy. We also reviewed more generalizable NLP pipelines to extract other critical patient variables. Our review found varying reports of performance measures. Whereas automated data extraction pipelines are a crucial advancement, this review calls attention to standardizing NLP methodology and accuracy reporting for greater generalizability. Moreover, the use of crowdsourcing competitions to spur innovative NLP pipelines would further advance this field.

Text-Based Identification of Herpes Zoster Ophthalmicus With Ocular Involvement in the Electronic Health Record: A Population-Based Study

Background

Diagnosis codes are inadequate for accurately identifying herpes zoster ophthalmicus (HZO). Manual review of medical records is expensive and time-consuming, resulting in a lack of population-based data on HZO.

Methods

We conducted a retrospective cohort study, including 87 673 patients aged ≥50 years who had a new HZ diagnosis and associated antiviral prescription between 2010 and 2018. We developed and validated an automated natural language processing (NLP) algorithm to identify HZO with ocular involvement (ocular HZO). We compared the characteristics of NLP-identified ocular HZO, nonocular HZO, and non-HZO cases among HZ patients and identified the factors associated with ocular HZO among HZ patients.

Results

The NLP algorithm achieved 94.9% sensitivity and 94.2% specificity in identifying ocular HZO cases. Among 87 673 incident HZ cases, the proportion identified as ocular HZO was 9.0% (n = 7853) by NLP and 2.3% (n = 1988) by International Classification of Diseases codes. In adjusted analyses, older age and male sex were associated with an increased risk of ocular HZO; Hispanic and black race/ethnicity each were associated with a lower risk of ocular HZO compared with non-Hispanic white.

Conclusions

The NLP algorithm achieved high accuracy and can be used in large population-based studies to identify ocular HZO, avoiding labor-intensive chart review. Age, sex, and race were strongly associated with ocular HZO among HZ patients. We should consider these risk factors when planning for zoster vaccination.

Text Processing for Detection of Fungal Ocular Involvement in Critical Care Patients: Cross-Sectional Study

Background

Fungal ocular involvement can develop in patients with fungal bloodstream infections and can be vision-threatening. Ocular involvement has become less common in the current era of improved antifungal therapies. Retrospectively determining the prevalence of fungal ocular involvement is important for informing clinical guidelines, such as the need for routine ophthalmologic consultations. However, manual retrospective record review to detect cases is time-consuming.

Objective

This study aimed to determine the prevalence of fungal ocular involvement in a critical care database using both structured and unstructured electronic health record (EHR) data.

Methods

We queried microbiology data from 46,467 critical care patients over 12 years (2000-2012) from the Medical Information Mart for Intensive Care III (MIMIC-III) to identify 265 patients with culture-proven fungemia. For each fungemic patient, demographic data, fungal species present in blood culture, and risk factors for fungemia (eg, presence of indwelling catheters, recent major surgery, diabetes, immunosuppressed status) were ascertained. All structured diagnosis codes and free-text narrative notes associated with each patient’s hospitalization were also extracted. Screening for fungal endophthalmitis was performed using two approaches: (1) by querying a wide array of eye- and vision-related diagnosis codes, and (2) by utilizing a custom regular expression pipeline to identify and collate relevant text matches pertaining to fungal ocular involvement. Both approaches were validated using manual record review. The main outcome measure was the documentation of any fungal ocular involvement.

Results

In total, 265 patients had culture-proven fungemia, with Candida albicans (n=114, 43%) and Candida glabrata (n=74, 28%) being the most common fungal species in blood culture. The in-hospital mortality rate was 121 (46%). In total, 7 patients were identified as having eye- or vision-related diagnosis codes, none of whom had fungal endophthalmitis based on record review. There were 26,830 free-text narrative notes associated with these 265 patients. A regular expression pipeline based on relevant terms yielded possible matches in 683 notes from 108 patients. Subsequent manual record review again demonstrated that no patients had fungal ocular involvement. Therefore, the prevalence of fungal ocular involvement in this cohort was 0%.

Conclusions

MIMIC-III contained no cases of ocular involvement among fungemic patients, consistent with prior studies reporting low rates of ocular involvement in fungemia. This study demonstrates an application of natural language processing to expedite the review of narrative notes. This approach is highly relevant for ophthalmology, where diagnoses are often based on physical examination findings that are documented within clinical notes.