1
|
Lucas A, Revell A, Davis KA. Artificial intelligence in epilepsy - applications and pathways to the clinic. Nat Rev Neurol 2024; 20:319-336. [PMID: 38720105 DOI: 10.1038/s41582-024-00965-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/16/2024] [Indexed: 06/06/2024]
Abstract
Artificial intelligence (AI) is rapidly transforming health care, and its applications in epilepsy have increased exponentially over the past decade. Integration of AI into epilepsy management promises to revolutionize the diagnosis and treatment of this complex disorder. However, translation of AI into neurology clinical practice has not yet been successful, emphasizing the need to consider progress to date and assess challenges and limitations of AI. In this Review, we provide an overview of AI applications that have been developed in epilepsy using a variety of data modalities: neuroimaging, electroencephalography, electronic health records, medical devices and multimodal data integration. For each, we consider potential applications, including seizure detection and prediction, seizure lateralization, localization of the seizure-onset zone and assessment for surgical or neurostimulation interventions, and review the performance of AI tools developed to date. We also discuss methodological considerations and challenges that must be addressed to successfully integrate AI into clinical practice. Our goal is to provide an overview of the current state of the field and provide guidance for leveraging AI in future to improve management of epilepsy.
Collapse
Affiliation(s)
- Alfredo Lucas
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA
| | - Andrew Revell
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Kathryn A Davis
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, USA.
| |
Collapse
|
2
|
Xie K, Ojemann WKS, Gallagher RS, Shinohara RT, Lucas A, Hill CE, Hamilton RH, Johnson KB, Roth D, Litt B, Ellis CA. Disparities in seizure outcomes revealed by large language models. J Am Med Inform Assoc 2024; 31:1348-1355. [PMID: 38481027 PMCID: PMC11105138 DOI: 10.1093/jamia/ocae047] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2023] [Revised: 02/21/2024] [Accepted: 02/23/2024] [Indexed: 03/26/2024] Open
Abstract
OBJECTIVE Large-language models (LLMs) can potentially revolutionize health care delivery and research, but risk propagating existing biases or introducing new ones. In epilepsy, social determinants of health are associated with disparities in care access, but their impact on seizure outcomes among those with access remains unclear. Here we (1) evaluated our validated, epilepsy-specific LLM for intrinsic bias, and (2) used LLM-extracted seizure outcomes to determine if different demographic groups have different seizure outcomes. MATERIALS AND METHODS We tested our LLM for differences and equivalences in prediction accuracy and confidence across demographic groups defined by race, ethnicity, sex, income, and health insurance, using manually annotated notes. Next, we used LLM-classified seizure freedom at each office visit to test for demographic outcome disparities, using univariable and multivariable analyses. RESULTS We analyzed 84 675 clinic visits from 25 612 unique patients seen at our epilepsy center. We found little evidence of bias in the prediction accuracy or confidence of outcome classifications across demographic groups. Multivariable analysis indicated worse seizure outcomes for female patients (OR 1.33, P ≤ .001), those with public insurance (OR 1.53, P ≤ .001), and those from lower-income zip codes (OR ≥1.22, P ≤ .007). Black patients had worse outcomes than White patients in univariable but not multivariable analysis (OR 1.03, P = .66). CONCLUSION We found little evidence that our LLM was intrinsically biased against any demographic group. Seizure freedom extracted by LLM revealed disparities in seizure outcomes across several demographic groups. These findings quantify the critical need to reduce disparities in the care of people with epilepsy.
Collapse
Affiliation(s)
- Kevin Xie
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA 19104, United States
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - William K S Ojemann
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA 19104, United States
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Ryan S Gallagher
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA 19104, United States
- Department of Neurology, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Russell T Shinohara
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Alfredo Lucas
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA 19104, United States
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA 19104, United States
- Department of Neurology, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Chloé E Hill
- Department of Neurology, University of Michigan, Ann Arbor, MI 48109, United States
| | - Roy H Hamilton
- Department of Neurology, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Kevin B Johnson
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA 19104, United States
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA 19104, United States
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, United States
- Department of Pediatrics, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Dan Roth
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Brian Litt
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA 19104, United States
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA 19104, United States
- Department of Neurology, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Colin A Ellis
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA 19104, United States
- Department of Neurology, University of Pennsylvania, Philadelphia, PA 19104, United States
| |
Collapse
|
3
|
van Diessen E, van Amerongen RA, Zijlmans M, Otte WM. Potential merits and flaws of large language models in epilepsy care: A critical review. Epilepsia 2024; 65:873-886. [PMID: 38305763 DOI: 10.1111/epi.17907] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2023] [Revised: 12/30/2023] [Accepted: 01/19/2024] [Indexed: 02/03/2024]
Abstract
The current pace of development and applications of large language models (LLMs) is unprecedented and will impact future medical care significantly. In this critical review, we provide the background to better understand these novel artificial intelligence (AI) models and how LLMs can be of future use in the daily care of people with epilepsy. Considering the importance of clinical history taking in diagnosing and monitoring epilepsy-combined with the established use of electronic health records-a great potential exists to integrate LLMs in epilepsy care. We present the current available LLM studies in epilepsy. Furthermore, we highlight and compare the most commonly used LLMs and elaborate on how these models can be applied in epilepsy. We further discuss important drawbacks and risks of LLMs, and we provide recommendations for overcoming these limitations.
Collapse
Affiliation(s)
- Eric van Diessen
- Department of Child Neurology, UMC Utrecht Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands
- Department of Pediatrics, Franciscus Gasthuis & Vlietland, Rotterdam, The Netherlands
| | - Ramon A van Amerongen
- Faculty of Science, Bioinformatics and Biocomplexity, Utrecht University, Utrecht, The Netherlands
| | - Maeike Zijlmans
- Department of Neurology and Neurosurgery, UMC Utrecht Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands
- Stichting Epilepsie Instellingen Nederland, Heemstede, The Netherlands
| | - Willem M Otte
- Department of Child Neurology, UMC Utrecht Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
4
|
Lo Barco T, Garcelon N, Neuraz A, Nabbout R. Natural history of rare diseases using natural language processing of narrative unstructured electronic health records: The example of Dravet syndrome. Epilepsia 2024; 65:350-361. [PMID: 38065926 DOI: 10.1111/epi.17855] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/04/2023] [Revised: 12/07/2023] [Accepted: 12/07/2023] [Indexed: 12/31/2023]
Abstract
OBJECTIVE The increasing implementation of electronic health records allows the use of advanced text-mining methods for establishing new patient phenotypes and stratification, and for revealing outcome correlations. In this study, we aimed to explore the electronic narrative clinical reports of a cohort of patients with Dravet syndrome (DS) longitudinally followed at our center, to identify the capacity of this methodology to retrace natural history of DS during the early years. METHODS We used a document-based clinical data warehouse employing natural language processing to recognize the phenotype concepts in the narrative medical reports. We included patients with DS who have a medical report produced before the age of 2 years and a follow-up after the age of 3 years ("DS cohort," 56 individuals). We selected two control populations, a "general control cohort" (275 individuals) and a "neurological control cohort" (281 individuals), with similar characteristics in terms of gender, number of reports, and age at last report. To find concepts specifically associated with DS, we performed a phenome-wide association study using Cox regression, comparing the reports of the three cohorts. We then performed a qualitative analysis of the surviving concepts based on their median age at first appearance. RESULTS A total of 76 concepts were prevalent in the reports of children with DS. Concepts appearing during the first 2 years were mostly related with the epilepsy features at the onset of DS (convulsive and prolonged seizures triggered by fever, often requiring in-hospital care). Subsequently, concepts related to new types of seizures and to drug resistance appeared. A series of non-seizure-related concepts emerged after the age of 2-3 years, referring to the nonseizure comorbidities classically associated with DS. SIGNIFICANCE The extraction of clinical terms by narrative reports of children with DS allows outlining the known natural history of this rare disease in early childhood. This original model of "longitudinal phenotyping" could be applied to other rare and very rare conditions with poor natural history description.
Collapse
Affiliation(s)
- Tommaso Lo Barco
- Department of Pediatric Neurology, Necker-Enfants Malades Hospital, Assistance Publique-Hôpitaux de Paris, Reference Center for Rare Epilepsies, Member of European Reference Network EpiCARE, Université Paris Cité, Paris, France
| | - Nicolas Garcelon
- Data Science Platform, Institut National de la Santé et de la Recherche Médicale Unité Mixte de Recherche 1163, Imagine Institute, Université Paris Cité, Paris, France
| | - Antoine Neuraz
- Data Science Platform, Institut National de la Santé et de la Recherche Médicale Unité Mixte de Recherche 1163, Imagine Institute, Université Paris Cité, Paris, France
| | - Rima Nabbout
- Department of Pediatric Neurology, Necker-Enfants Malades Hospital, Assistance Publique-Hôpitaux de Paris, Reference Center for Rare Epilepsies, Member of European Reference Network EpiCARE, Université Paris Cité, Paris, France
- Translational Research for Neurological Disorders, Institut National de la Santé et de la Recherche Médicale Unité Mixte de Recherche 1163, Imagine Institute, Université Paris Cité, Paris, France
| |
Collapse
|
5
|
Emmert BE, Xie K, Conrad EC, Ghosn NJ, Bauman K, Korzun J, Kulick-Soper CV, Naveed O, Hartmann N, LaRocque JJ, Mindy Ganguly T, Gugger JJ, Raghupathi R, Gelfand MA, Davis KA, Sinha SR, Litt B, Shinohara RT, Ellis CA. Alcohol for seizure induction in the epilepsy monitoring unit. Epilepsy Behav 2024; 150:109572. [PMID: 38070406 PMCID: PMC10842723 DOI: 10.1016/j.yebeh.2023.109572] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/09/2023] [Revised: 11/17/2023] [Accepted: 11/28/2023] [Indexed: 01/14/2024]
Abstract
RATIONALE Seizure induction techniques are used in the epilepsy monitoring unit (EMU) to increase diagnostic yield and reduce length of stay. There are insufficient data on the efficacy of alcohol as an induction technique. METHODS We performed a retrospective cohort study using six years of EMU data at our institution. We compared cases who received alcohol for seizure induction to matched controls who did not. The groups were matched on the following variables: age, reason for admission, length of stay, number of antiseizure medications (ASM) at admission, whether ASMs were tapered during admission, and presence of interictal epileptiform discharges. We used both propensity score and exact matching strategies. We compared the likelihood of epileptic seizures and nonepileptic events in cases versus controls using Kaplan-Meier time-to-event analysis, as well as odds ratios for these outcomes occurring at any time during the admission. RESULTS We analyzed 256 cases who received alcohol (median dose 2.5 standard drinks) and 256 propensity score-matched controls. Cases who received alcohol were no more likely than controls to have an epileptic seizure (X2(1) = 0.01, p = 0.93) or nonepileptic event (X2(1) = 2.1, p = 0.14) in the first 48 h after alcohol administration. For the admission overall, cases were no more likely to have an epileptic seizure (OR 0.89, 95 % CI 0.61-1.28, p = 0.58), nonepileptic event (OR 0.97, CI 0.62-1.53, p = 1.00), nor require rescue benzodiazepine (OR 0.63, CI 0.35-1.12, p = 0.15). Stratified analyses revealed no increased risk of epileptic seizure in any subgroups. Sensitivity analysis using exact matching showed that results were robust to matching strategy. CONCLUSIONS Alcohol was not an effective induction technique in the EMU. This finding has implications for counseling patients with epilepsy about the risks of drinking alcohol in moderation in their daily lives.
Collapse
Affiliation(s)
- Brian E Emmert
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA.
| | - Kevin Xie
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA; Center for Neuroengineering & Therapeutics, University of Pennsylvania, Philadelphia, PA, USA
| | - Erin C Conrad
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA; Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA; Center for Neuroengineering & Therapeutics, University of Pennsylvania, Philadelphia, PA, USA
| | - Nina J Ghosn
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA; Center for Neuroengineering & Therapeutics, University of Pennsylvania, Philadelphia, PA, USA
| | - Kristie Bauman
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Jacob Korzun
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Catherine V Kulick-Soper
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Omer Naveed
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Nicole Hartmann
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Joshua J LaRocque
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Taneeta Mindy Ganguly
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - James J Gugger
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA; Center for Neuroengineering & Therapeutics, University of Pennsylvania, Philadelphia, PA, USA
| | - Ramya Raghupathi
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Michael A Gelfand
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Kathryn A Davis
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA; Center for Neuroengineering & Therapeutics, University of Pennsylvania, Philadelphia, PA, USA
| | - Saurabh R Sinha
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Brian Litt
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA; Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA; Center for Neuroengineering & Therapeutics, University of Pennsylvania, Philadelphia, PA, USA
| | - Russell T Shinohara
- Penn Statistics in Imaging and Visualization Center (PennSIVE), Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia PA, USA; Center for Biomedical Image Computing and Analytics, Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia PA, USA
| | - Colin A Ellis
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| |
Collapse
|
6
|
Kerr WT, McFarlane KN. Machine Learning and Artificial Intelligence Applications to Epilepsy: a Review for the Practicing Epileptologist. Curr Neurol Neurosci Rep 2023; 23:869-879. [PMID: 38060133 DOI: 10.1007/s11910-023-01318-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/24/2023] [Indexed: 12/08/2023]
Abstract
PURPOSE OF REVIEW Machine Learning (ML) and Artificial Intelligence (AI) are data-driven techniques to translate raw data into applicable and interpretable insights that can assist in clinical decision making. Some of these tools have extremely promising initial results, earning both great excitement and creating hype. This non-technical article reviews recent developments in ML/AI in epilepsy to assist the current practicing epileptologist in understanding both the benefits and limitations of integrating ML/AI tools into their clinical practice. RECENT FINDINGS ML/AI tools have been developed to assist clinicians in almost every clinical decision including (1) predicting future epilepsy in people at risk, (2) detecting and monitoring for seizures, (3) differentiating epilepsy from mimics, (4) using data to improve neuroanatomic localization and lateralization, and (5) tracking and predicting response to medical and surgical treatments. We also discuss practical, ethical, and equity considerations in the development and application of ML/AI tools including chatbots based on Large Language Models (e.g., ChatGPT). ML/AI tools will change how clinical medicine is practiced, but, with rare exceptions, the transferability to other centers, effectiveness, and safety of these approaches have not yet been established rigorously. In the future, ML/AI will not replace epileptologists, but epileptologists with ML/AI will replace epileptologists without ML/AI.
Collapse
Affiliation(s)
- Wesley T Kerr
- Department of Neurology, University of Pittsburgh, 3471 Fifth Ave, Kaufmann 811.22, Pittsburgh, PA, 15213, USA.
- Department of Biomedical Informatics, University of Pittsburgh, 3471 Fifth Ave, Kaufmann 811.22, Pittsburgh, PA, 15213, USA.
- Department of Neurology, Michigan Medicine, University of Michigan, Ann Arbor, MI, USA.
| | - Katherine N McFarlane
- Department of Neurology, University of Pittsburgh, 3471 Fifth Ave, Kaufmann 811.22, Pittsburgh, PA, 15213, USA
| |
Collapse
|
7
|
Knight A, Gschwind T, Galer P, Worrell GA, Litt B, Soltesz I, Beniczky S. Artificial intelligence in epilepsy phenotyping. Epilepsia 2023:10.1111/epi.17833. [PMID: 37983589 PMCID: PMC11102939 DOI: 10.1111/epi.17833] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2023] [Accepted: 11/17/2023] [Indexed: 11/22/2023]
Abstract
Artificial intelligence (AI) allows data analysis and integration at an unprecedented granularity and scale. Here we review the technological advances, challenges, and future perspectives of using AI for electro-clinical phenotyping of animal models and patients with epilepsy. In translational research, AI models accurately identify behavioral states in animal models of epilepsy, allowing identification of correlations between neural activity and interictal and ictal behavior. Clinical applications of AI-based automated and semi-automated analysis of audio and video recordings of people with epilepsy, allow significant data reduction and reliable detection and classification of major motor seizures. AI models can accurately identify electrographic biomarkers of epilepsy, such as spikes, high-frequency oscillations, and seizure patterns. Integrating AI analysis of electroencephalographic, clinical, and behavioral data will contribute to optimizing therapy for patients with epilepsy.
Collapse
Affiliation(s)
| | - Tilo Gschwind
- Department of Neurosurgery, Stanford University, Stanford, USA
| | - Peter Galer
- Center for Neuroengineering and Therapeutics; Department of Bioengineering; Department of Neurology, Perelman School of Medicine, University of Pennsylvania, USA
| | | | - Brian Litt
- Center for Neuroengineering and Therapeutics; Department of Bioengineering; Department of Neurology, Perelman School of Medicine, University of Pennsylvania, USA
| | - Ivan Soltesz
- Department of Neurosurgery, Stanford University, Stanford, USA
| | - Sándor Beniczky
- Danish Epilepsy Centre Filadelfia, Dianalund, Aarhus University Hospital and Aarhus University, Aarhus, Denmark
| |
Collapse
|
8
|
Xie K, Terman SW, Gallagher RS, Hill CE, Davis KA, Litt B, Roth D, Ellis CA. Generalization of finetuned transformer language models to new clinical contexts. JAMIA Open 2023; 6:ooad070. [PMID: 37600072 PMCID: PMC10432353 DOI: 10.1093/jamiaopen/ooad070] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Revised: 08/01/2023] [Accepted: 08/03/2023] [Indexed: 08/22/2023] Open
Abstract
Objective We have previously developed a natural language processing pipeline using clinical notes written by epilepsy specialists to extract seizure freedom, seizure frequency text, and date of last seizure text for patients with epilepsy. It is important to understand how our methods generalize to new care contexts. Materials and methods We evaluated our pipeline on unseen notes from nonepilepsy-specialist neurologists and non-neurologists without any additional algorithm training. We tested the pipeline out-of-institution using epilepsy specialist notes from an outside medical center with only minor preprocessing adaptations. We examined reasons for discrepancies in performance in new contexts by measuring physical and semantic similarities between documents. Results Our ability to classify patient seizure freedom decreased by at least 0.12 agreement when moving from epilepsy specialists to nonspecialists or other institutions. On notes from our institution, textual overlap between the extracted outcomes and the gold standard annotations attained from manual chart review decreased by at least 0.11 F1 when an answer existed but did not change when no answer existed; here our models generalized on notes from the outside institution, losing at most 0.02 agreement. We analyzed textual differences and found that syntactic and semantic differences in both clinically relevant sentences and surrounding contexts significantly influenced model performance. Discussion and conclusion Model generalization performance decreased on notes from nonspecialists; out-of-institution generalization on epilepsy specialist notes required small changes to preprocessing but was especially good for seizure frequency text and date of last seizure text, opening opportunities for multicenter collaborations using these outcomes.
Collapse
Affiliation(s)
- Kevin Xie
- Department of Bioengineering, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Samuel W Terman
- Department of Neurology, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Ryan S Gallagher
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Chloe E Hill
- Department of Neurology, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Kathryn A Davis
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Brian Litt
- Department of Bioengineering, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Dan Roth
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Colin A Ellis
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| |
Collapse
|
9
|
Xie K, Ojemann WKS, Gallagher RS, Lucas A, Hill CE, Hamilton RH, Johnson KB, Roth D, Litt B, Ellis CA. Disparities in seizure outcomes revealed by large language models. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.09.20.23295842. [PMID: 37790442 PMCID: PMC10543059 DOI: 10.1101/2023.09.20.23295842] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/05/2023]
Abstract
Objective Large-language models (LLMs) in healthcare have the potential to propagate existing biases or introduce new ones. For people with epilepsy, social determinants of health are associated with disparities in access to care, but their impact on seizure outcomes among those with access to specialty care remains unclear. Here we (1) evaluated our validated, epilepsy-specific LLM for intrinsic bias, and (2) used LLM-extracted seizure outcomes to test the hypothesis that different demographic groups have different seizure outcomes. Methods First, we tested our LLM for intrinsic bias in the form of differential performance in demographic groups by race, ethnicity, sex, income, and health insurance in manually annotated notes. Next, we used LLM-classified seizure freedom at each office visit to test for outcome disparities in the same demographic groups, using univariable and multivariable analyses. Results We analyzed 84,675 clinic visits from 25,612 patients seen at our epilepsy center 2005-2022. We found no differences in the accuracy, or positive or negative class balance of outcome classifications across demographic groups. Multivariable analysis indicated worse seizure outcomes for female patients (OR 1.33, p = 3×10-8), those with public insurance (OR 1.53, p = 2×10-13), and those from lower-income zip codes (OR ≥ 1.22, p ≤ 6.6×10-3). Black patients had worse outcomes than White patients in univariable but not multivariable analysis (OR 1.03, p = 0.66). Significance We found no evidence that our LLM was intrinsically biased against any demographic group. Seizure freedom extracted by LLM revealed disparities in seizure outcomes across several demographic groups. These findings highlight the critical need to reduce disparities in the care of people with epilepsy.
Collapse
Affiliation(s)
- Kevin Xie
- University of Pennsylvania, Dept. of Bioengineering, Philadelphia, PA, USA
- University of Pennsylvania, Center for Neuroengineering and Therapeutics, Philadelphia, PA, USA
| | - William K S Ojemann
- University of Pennsylvania, Dept. of Bioengineering, Philadelphia, PA, USA
- University of Pennsylvania, Center for Neuroengineering and Therapeutics, Philadelphia, PA, USA
| | - Ryan S Gallagher
- University of Pennsylvania, Center for Neuroengineering and Therapeutics, Philadelphia, PA, USA
- University of Pennsylvania, Dept. of Neurology, Philadelphia, PA, USA
| | - Alfredo Lucas
- University of Pennsylvania, Dept. of Bioengineering, Philadelphia, PA, USA
- University of Pennsylvania, Center for Neuroengineering and Therapeutics, Philadelphia, PA, USA
- University of Pennsylvania, Dept. of Neurology, Philadelphia, PA, USA
| | - Chloé E Hill
- University of Michigan, Dept. of Neurology, Ann Arbor, MI, USA
| | - Roy H Hamilton
- University of Pennsylvania, Dept. of Neurology, Philadelphia, PA, USA
| | - Kevin B Johnson
- University of Pennsylvania, Dept. of Bioengineering, Philadelphia, PA, USA
- University of Pennsylvania, Dept. Of Biostatistics, Epidemiology and Informatics, Philadelphia, PA USA
- University of Pennsylvania, Dept. of Computer and Information Science, Philadelphia, PA, USA
- University of Pennsylvania, Dept. of Pediatrics, Philadelphia, PA, USA
| | - Dan Roth
- University of Pennsylvania, Dept. of Computer and Information Science, Philadelphia, PA, USA
| | - Brian Litt
- University of Pennsylvania, Dept. of Bioengineering, Philadelphia, PA, USA
- University of Pennsylvania, Center for Neuroengineering and Therapeutics, Philadelphia, PA, USA
- University of Pennsylvania, Dept. of Neurology, Philadelphia, PA, USA
| | - Colin A Ellis
- University of Pennsylvania, Center for Neuroengineering and Therapeutics, Philadelphia, PA, USA
- University of Pennsylvania, Dept. of Neurology, Philadelphia, PA, USA
| |
Collapse
|