1
|
Muizelaar H, Haas M, van Dortmont K, van der Putten P, Spruit M. Extracting patient lifestyle characteristics from Dutch clinical text with BERT models. BMC Med Inform Decis Mak 2024; 24:151. [PMID: 38831420 PMCID: PMC11149227 DOI: 10.1186/s12911-024-02557-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2024] [Accepted: 05/28/2024] [Indexed: 06/05/2024] Open
Abstract
BACKGROUND BERT models have seen widespread use on unstructured text within the clinical domain. However, little to no research has been conducted into classifying unstructured clinical notes on the basis of patient lifestyle indicators, especially in Dutch. This article aims to test the feasibility of deep BERT models on the task of patient lifestyle classification, as well as introducing an experimental framework that is easily reproducible in future research. METHODS This study makes use of unstructured general patient text data from HagaZiekenhuis, a large hospital in The Netherlands. Over 148 000 notes were provided to us, which were each automatically labelled on the basis of the respective patients' smoking, alcohol usage and drug usage statuses. In this paper we test feasibility of automatically assigning labels, and justify it using hand-labelled input. Ultimately, we compare macro F1-scores of string matching, SGD and several BERT models on the task of classifying smoking, alcohol and drug usage. We test Dutch BERT models and English models with translated input. RESULTS We find that our further pre-trained MedRoBERTa.nl-HAGA model outperformed every other model on smoking (0.93) and drug usage (0.77). Interestingly, our ClinicalBERT model that was merely fine-tuned on translated text performed best on the alcohol task (0.80). In t-SNE visualisations, we show our MedRoBERTa.nl-HAGA model is the best model to differentiate between classes in the embedding space, explaining its superior classification performance. CONCLUSIONS We suggest MedRoBERTa.nl-HAGA to be used as a baseline in future research on Dutch free text patient lifestyle classification. We furthermore strongly suggest further exploring the application of translation to input text in non-English clinical BERT research, as we only translated a subset of the full set and yet achieved very promising results.
Collapse
Affiliation(s)
- Hielke Muizelaar
- LIACS, Leiden University, P.O. Box 9512, Leiden, 2300RA, The Netherlands.
- Department of Public Health and Primary Care, Leiden University Medical Center, Albinusdreef 2, Leiden, 2333ZA, The Netherlands.
| | - Marcel Haas
- Department of Public Health and Primary Care, Leiden University Medical Center, Albinusdreef 2, Leiden, 2333ZA, The Netherlands
| | - Koert van Dortmont
- Department of Business Intelligence, HagaZiekenhuis, Els Borst-Eilersplein 275, Den Haag, 2545AA, The Netherlands
| | | | - Marco Spruit
- LIACS, Leiden University, P.O. Box 9512, Leiden, 2300RA, The Netherlands
- Department of Public Health and Primary Care, Leiden University Medical Center, Albinusdreef 2, Leiden, 2333ZA, The Netherlands
| |
Collapse
|
2
|
Lucas A, Revell A, Davis KA. Artificial intelligence in epilepsy - applications and pathways to the clinic. Nat Rev Neurol 2024; 20:319-336. [PMID: 38720105 DOI: 10.1038/s41582-024-00965-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/16/2024] [Indexed: 06/06/2024]
Abstract
Artificial intelligence (AI) is rapidly transforming health care, and its applications in epilepsy have increased exponentially over the past decade. Integration of AI into epilepsy management promises to revolutionize the diagnosis and treatment of this complex disorder. However, translation of AI into neurology clinical practice has not yet been successful, emphasizing the need to consider progress to date and assess challenges and limitations of AI. In this Review, we provide an overview of AI applications that have been developed in epilepsy using a variety of data modalities: neuroimaging, electroencephalography, electronic health records, medical devices and multimodal data integration. For each, we consider potential applications, including seizure detection and prediction, seizure lateralization, localization of the seizure-onset zone and assessment for surgical or neurostimulation interventions, and review the performance of AI tools developed to date. We also discuss methodological considerations and challenges that must be addressed to successfully integrate AI into clinical practice. Our goal is to provide an overview of the current state of the field and provide guidance for leveraging AI in future to improve management of epilepsy.
Collapse
Affiliation(s)
- Alfredo Lucas
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA
| | - Andrew Revell
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Kathryn A Davis
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, USA.
| |
Collapse
|
3
|
Xie K, Ojemann WKS, Gallagher RS, Shinohara RT, Lucas A, Hill CE, Hamilton RH, Johnson KB, Roth D, Litt B, Ellis CA. Disparities in seizure outcomes revealed by large language models. J Am Med Inform Assoc 2024; 31:1348-1355. [PMID: 38481027 PMCID: PMC11105138 DOI: 10.1093/jamia/ocae047] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2023] [Revised: 02/21/2024] [Accepted: 02/23/2024] [Indexed: 03/26/2024] Open
Abstract
OBJECTIVE Large-language models (LLMs) can potentially revolutionize health care delivery and research, but risk propagating existing biases or introducing new ones. In epilepsy, social determinants of health are associated with disparities in care access, but their impact on seizure outcomes among those with access remains unclear. Here we (1) evaluated our validated, epilepsy-specific LLM for intrinsic bias, and (2) used LLM-extracted seizure outcomes to determine if different demographic groups have different seizure outcomes. MATERIALS AND METHODS We tested our LLM for differences and equivalences in prediction accuracy and confidence across demographic groups defined by race, ethnicity, sex, income, and health insurance, using manually annotated notes. Next, we used LLM-classified seizure freedom at each office visit to test for demographic outcome disparities, using univariable and multivariable analyses. RESULTS We analyzed 84 675 clinic visits from 25 612 unique patients seen at our epilepsy center. We found little evidence of bias in the prediction accuracy or confidence of outcome classifications across demographic groups. Multivariable analysis indicated worse seizure outcomes for female patients (OR 1.33, P ≤ .001), those with public insurance (OR 1.53, P ≤ .001), and those from lower-income zip codes (OR ≥1.22, P ≤ .007). Black patients had worse outcomes than White patients in univariable but not multivariable analysis (OR 1.03, P = .66). CONCLUSION We found little evidence that our LLM was intrinsically biased against any demographic group. Seizure freedom extracted by LLM revealed disparities in seizure outcomes across several demographic groups. These findings quantify the critical need to reduce disparities in the care of people with epilepsy.
Collapse
Affiliation(s)
- Kevin Xie
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA 19104, United States
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - William K S Ojemann
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA 19104, United States
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Ryan S Gallagher
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA 19104, United States
- Department of Neurology, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Russell T Shinohara
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Alfredo Lucas
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA 19104, United States
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA 19104, United States
- Department of Neurology, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Chloé E Hill
- Department of Neurology, University of Michigan, Ann Arbor, MI 48109, United States
| | - Roy H Hamilton
- Department of Neurology, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Kevin B Johnson
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA 19104, United States
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA 19104, United States
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, United States
- Department of Pediatrics, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Dan Roth
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Brian Litt
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA 19104, United States
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA 19104, United States
- Department of Neurology, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Colin A Ellis
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA 19104, United States
- Department of Neurology, University of Pennsylvania, Philadelphia, PA 19104, United States
| |
Collapse
|
4
|
van Diessen E, van Amerongen RA, Zijlmans M, Otte WM. Potential merits and flaws of large language models in epilepsy care: A critical review. Epilepsia 2024; 65:873-886. [PMID: 38305763 DOI: 10.1111/epi.17907] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2023] [Revised: 12/30/2023] [Accepted: 01/19/2024] [Indexed: 02/03/2024]
Abstract
The current pace of development and applications of large language models (LLMs) is unprecedented and will impact future medical care significantly. In this critical review, we provide the background to better understand these novel artificial intelligence (AI) models and how LLMs can be of future use in the daily care of people with epilepsy. Considering the importance of clinical history taking in diagnosing and monitoring epilepsy-combined with the established use of electronic health records-a great potential exists to integrate LLMs in epilepsy care. We present the current available LLM studies in epilepsy. Furthermore, we highlight and compare the most commonly used LLMs and elaborate on how these models can be applied in epilepsy. We further discuss important drawbacks and risks of LLMs, and we provide recommendations for overcoming these limitations.
Collapse
Affiliation(s)
- Eric van Diessen
- Department of Child Neurology, UMC Utrecht Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands
- Department of Pediatrics, Franciscus Gasthuis & Vlietland, Rotterdam, The Netherlands
| | - Ramon A van Amerongen
- Faculty of Science, Bioinformatics and Biocomplexity, Utrecht University, Utrecht, The Netherlands
| | - Maeike Zijlmans
- Department of Neurology and Neurosurgery, UMC Utrecht Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands
- Stichting Epilepsie Instellingen Nederland, Heemstede, The Netherlands
| | - Willem M Otte
- Department of Child Neurology, UMC Utrecht Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
5
|
Emmert BE, Xie K, Conrad EC, Ghosn NJ, Bauman K, Korzun J, Kulick-Soper CV, Naveed O, Hartmann N, LaRocque JJ, Mindy Ganguly T, Gugger JJ, Raghupathi R, Gelfand MA, Davis KA, Sinha SR, Litt B, Shinohara RT, Ellis CA. Alcohol for seizure induction in the epilepsy monitoring unit. Epilepsy Behav 2024; 150:109572. [PMID: 38070406 PMCID: PMC10842723 DOI: 10.1016/j.yebeh.2023.109572] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/09/2023] [Revised: 11/17/2023] [Accepted: 11/28/2023] [Indexed: 01/14/2024]
Abstract
RATIONALE Seizure induction techniques are used in the epilepsy monitoring unit (EMU) to increase diagnostic yield and reduce length of stay. There are insufficient data on the efficacy of alcohol as an induction technique. METHODS We performed a retrospective cohort study using six years of EMU data at our institution. We compared cases who received alcohol for seizure induction to matched controls who did not. The groups were matched on the following variables: age, reason for admission, length of stay, number of antiseizure medications (ASM) at admission, whether ASMs were tapered during admission, and presence of interictal epileptiform discharges. We used both propensity score and exact matching strategies. We compared the likelihood of epileptic seizures and nonepileptic events in cases versus controls using Kaplan-Meier time-to-event analysis, as well as odds ratios for these outcomes occurring at any time during the admission. RESULTS We analyzed 256 cases who received alcohol (median dose 2.5 standard drinks) and 256 propensity score-matched controls. Cases who received alcohol were no more likely than controls to have an epileptic seizure (X2(1) = 0.01, p = 0.93) or nonepileptic event (X2(1) = 2.1, p = 0.14) in the first 48 h after alcohol administration. For the admission overall, cases were no more likely to have an epileptic seizure (OR 0.89, 95 % CI 0.61-1.28, p = 0.58), nonepileptic event (OR 0.97, CI 0.62-1.53, p = 1.00), nor require rescue benzodiazepine (OR 0.63, CI 0.35-1.12, p = 0.15). Stratified analyses revealed no increased risk of epileptic seizure in any subgroups. Sensitivity analysis using exact matching showed that results were robust to matching strategy. CONCLUSIONS Alcohol was not an effective induction technique in the EMU. This finding has implications for counseling patients with epilepsy about the risks of drinking alcohol in moderation in their daily lives.
Collapse
Affiliation(s)
- Brian E Emmert
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA.
| | - Kevin Xie
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA; Center for Neuroengineering & Therapeutics, University of Pennsylvania, Philadelphia, PA, USA
| | - Erin C Conrad
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA; Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA; Center for Neuroengineering & Therapeutics, University of Pennsylvania, Philadelphia, PA, USA
| | - Nina J Ghosn
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA; Center for Neuroengineering & Therapeutics, University of Pennsylvania, Philadelphia, PA, USA
| | - Kristie Bauman
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Jacob Korzun
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Catherine V Kulick-Soper
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Omer Naveed
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Nicole Hartmann
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Joshua J LaRocque
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Taneeta Mindy Ganguly
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - James J Gugger
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA; Center for Neuroengineering & Therapeutics, University of Pennsylvania, Philadelphia, PA, USA
| | - Ramya Raghupathi
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Michael A Gelfand
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Kathryn A Davis
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA; Center for Neuroengineering & Therapeutics, University of Pennsylvania, Philadelphia, PA, USA
| | - Saurabh R Sinha
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| | - Brian Litt
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA; Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA; Center for Neuroengineering & Therapeutics, University of Pennsylvania, Philadelphia, PA, USA
| | - Russell T Shinohara
- Penn Statistics in Imaging and Visualization Center (PennSIVE), Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia PA, USA; Center for Biomedical Image Computing and Analytics, Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia PA, USA
| | - Colin A Ellis
- Department of Neurology, University of Pennsylvania Perelman School of Medicine, Philadelphia PA, USA
| |
Collapse
|
6
|
Romano MF, Shih LC, Paschalidis IC, Au R, Kolachalama VB. Large Language Models in Neurology Research and Future Practice. Neurology 2023; 101:1058-1067. [PMID: 37816646 PMCID: PMC10752640 DOI: 10.1212/wnl.0000000000207967] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2023] [Accepted: 09/06/2023] [Indexed: 10/12/2023] Open
Abstract
Recent advancements in generative artificial intelligence, particularly using large language models (LLMs), are gaining increased public attention. We provide a perspective on the potential of LLMs to analyze enormous amounts of data from medical records and gain insights on specific topics in neurology. In addition, we explore use cases for LLMs, such as early diagnosis, supporting patient and caregivers, and acting as an assistant for clinicians. We point to the potential ethical and technical challenges raised by LLMs, such as concerns about privacy and data security, potential biases in the data for model training, and the need for careful validation of results. Researchers must consider these challenges and take steps to address them to ensure that their work is conducted in a safe and responsible manner. Despite these challenges, LLMs offer promising opportunities for improving care and treatment of various neurologic disorders.
Collapse
Affiliation(s)
- Michael F Romano
- From the Department of Medicine (M.F.R., R.A., V.B.K.), Boston University Chobanian & Avedisian School of Medicine, MA; Department of Radiology and Biomedical Imaging (M.F.R.), University of California, San Francisco; Department of Neurology (L.C.S., R.A.), Boston University Chobanian & Avedisian School of Medicine; Department of Electrical and Computer Engineering (I.C.P.), Division of Systems Engineering, and Department of Biomedical Engineering; Faculty of Computing and Data Sciences (I.C.P., V.B.K.), Boston University; Department of Anatomy and Neurobiology (R.A.); The Framingham Heart Study, Boston University Chobanian & Avedisian School of Medicine; Department of Epidemiology, Boston University School of Public Health; Boston University Alzheimer's Disease Research Center (R.A.); and Department of Computer Science (V.B.K.), Boston University, MA
| | - Ludy C Shih
- From the Department of Medicine (M.F.R., R.A., V.B.K.), Boston University Chobanian & Avedisian School of Medicine, MA; Department of Radiology and Biomedical Imaging (M.F.R.), University of California, San Francisco; Department of Neurology (L.C.S., R.A.), Boston University Chobanian & Avedisian School of Medicine; Department of Electrical and Computer Engineering (I.C.P.), Division of Systems Engineering, and Department of Biomedical Engineering; Faculty of Computing and Data Sciences (I.C.P., V.B.K.), Boston University; Department of Anatomy and Neurobiology (R.A.); The Framingham Heart Study, Boston University Chobanian & Avedisian School of Medicine; Department of Epidemiology, Boston University School of Public Health; Boston University Alzheimer's Disease Research Center (R.A.); and Department of Computer Science (V.B.K.), Boston University, MA
| | - Ioannis C Paschalidis
- From the Department of Medicine (M.F.R., R.A., V.B.K.), Boston University Chobanian & Avedisian School of Medicine, MA; Department of Radiology and Biomedical Imaging (M.F.R.), University of California, San Francisco; Department of Neurology (L.C.S., R.A.), Boston University Chobanian & Avedisian School of Medicine; Department of Electrical and Computer Engineering (I.C.P.), Division of Systems Engineering, and Department of Biomedical Engineering; Faculty of Computing and Data Sciences (I.C.P., V.B.K.), Boston University; Department of Anatomy and Neurobiology (R.A.); The Framingham Heart Study, Boston University Chobanian & Avedisian School of Medicine; Department of Epidemiology, Boston University School of Public Health; Boston University Alzheimer's Disease Research Center (R.A.); and Department of Computer Science (V.B.K.), Boston University, MA
| | - Rhoda Au
- From the Department of Medicine (M.F.R., R.A., V.B.K.), Boston University Chobanian & Avedisian School of Medicine, MA; Department of Radiology and Biomedical Imaging (M.F.R.), University of California, San Francisco; Department of Neurology (L.C.S., R.A.), Boston University Chobanian & Avedisian School of Medicine; Department of Electrical and Computer Engineering (I.C.P.), Division of Systems Engineering, and Department of Biomedical Engineering; Faculty of Computing and Data Sciences (I.C.P., V.B.K.), Boston University; Department of Anatomy and Neurobiology (R.A.); The Framingham Heart Study, Boston University Chobanian & Avedisian School of Medicine; Department of Epidemiology, Boston University School of Public Health; Boston University Alzheimer's Disease Research Center (R.A.); and Department of Computer Science (V.B.K.), Boston University, MA
| | - Vijaya B Kolachalama
- From the Department of Medicine (M.F.R., R.A., V.B.K.), Boston University Chobanian & Avedisian School of Medicine, MA; Department of Radiology and Biomedical Imaging (M.F.R.), University of California, San Francisco; Department of Neurology (L.C.S., R.A.), Boston University Chobanian & Avedisian School of Medicine; Department of Electrical and Computer Engineering (I.C.P.), Division of Systems Engineering, and Department of Biomedical Engineering; Faculty of Computing and Data Sciences (I.C.P., V.B.K.), Boston University; Department of Anatomy and Neurobiology (R.A.); The Framingham Heart Study, Boston University Chobanian & Avedisian School of Medicine; Department of Epidemiology, Boston University School of Public Health; Boston University Alzheimer's Disease Research Center (R.A.); and Department of Computer Science (V.B.K.), Boston University, MA.
| |
Collapse
|
7
|
Kerr WT, McFarlane KN. Machine Learning and Artificial Intelligence Applications to Epilepsy: a Review for the Practicing Epileptologist. Curr Neurol Neurosci Rep 2023; 23:869-879. [PMID: 38060133 DOI: 10.1007/s11910-023-01318-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/24/2023] [Indexed: 12/08/2023]
Abstract
PURPOSE OF REVIEW Machine Learning (ML) and Artificial Intelligence (AI) are data-driven techniques to translate raw data into applicable and interpretable insights that can assist in clinical decision making. Some of these tools have extremely promising initial results, earning both great excitement and creating hype. This non-technical article reviews recent developments in ML/AI in epilepsy to assist the current practicing epileptologist in understanding both the benefits and limitations of integrating ML/AI tools into their clinical practice. RECENT FINDINGS ML/AI tools have been developed to assist clinicians in almost every clinical decision including (1) predicting future epilepsy in people at risk, (2) detecting and monitoring for seizures, (3) differentiating epilepsy from mimics, (4) using data to improve neuroanatomic localization and lateralization, and (5) tracking and predicting response to medical and surgical treatments. We also discuss practical, ethical, and equity considerations in the development and application of ML/AI tools including chatbots based on Large Language Models (e.g., ChatGPT). ML/AI tools will change how clinical medicine is practiced, but, with rare exceptions, the transferability to other centers, effectiveness, and safety of these approaches have not yet been established rigorously. In the future, ML/AI will not replace epileptologists, but epileptologists with ML/AI will replace epileptologists without ML/AI.
Collapse
Affiliation(s)
- Wesley T Kerr
- Department of Neurology, University of Pittsburgh, 3471 Fifth Ave, Kaufmann 811.22, Pittsburgh, PA, 15213, USA.
- Department of Biomedical Informatics, University of Pittsburgh, 3471 Fifth Ave, Kaufmann 811.22, Pittsburgh, PA, 15213, USA.
- Department of Neurology, Michigan Medicine, University of Michigan, Ann Arbor, MI, USA.
| | - Katherine N McFarlane
- Department of Neurology, University of Pittsburgh, 3471 Fifth Ave, Kaufmann 811.22, Pittsburgh, PA, 15213, USA
| |
Collapse
|
8
|
Ge W, Rice HJ, Sheikh IS, Westover MB, Weathers AL, Jones LK, Moura L. Improving Neurology Clinical Care With Natural Language Processing Tools. Neurology 2023; 101:1010-1018. [PMID: 37816638 PMCID: PMC10727205 DOI: 10.1212/wnl.0000000000207853] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2023] [Accepted: 08/25/2023] [Indexed: 10/12/2023] Open
Abstract
The integration of natural language processing (NLP) tools into neurology workflows has the potential to significantly enhance clinical care. However, it is important to address the limitations and risks associated with integrating this new technology. Recent advances in transformer-based NLP algorithms (e.g., GPT, BERT) could augment neurology clinical care by summarizing patient health information, suggesting care options, and assisting research involving large datasets. However, these NLP platforms have potential risks including fabricated facts and data security and substantial barriers for implementation. Although these risks and barriers need to be considered, the benefits for providers, patients, and communities are substantial. With these systems achieving greater functionality and the pace of medical need increasing, integrating these tools into clinical care may prove not only beneficial but necessary. Further investigation is needed to design implementation strategies, mitigate risks, and overcome barriers.
Collapse
Affiliation(s)
- Wendong Ge
- From the Department of Neurology (W.G., H.J.R., I.S.S., L.M.), Massachusetts General Hospital, Boston; Department of Neurology (M.B.W.), Beth Israel Deaconess Medical Center, Boston, MA; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Neurology (L.K.J.), Mayo Clinic, Rochester, MN; and Department of Neurology (L.M.), Harvard Medical School, Boston, MA
| | - Hunter J Rice
- From the Department of Neurology (W.G., H.J.R., I.S.S., L.M.), Massachusetts General Hospital, Boston; Department of Neurology (M.B.W.), Beth Israel Deaconess Medical Center, Boston, MA; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Neurology (L.K.J.), Mayo Clinic, Rochester, MN; and Department of Neurology (L.M.), Harvard Medical School, Boston, MA
| | - Irfan S Sheikh
- From the Department of Neurology (W.G., H.J.R., I.S.S., L.M.), Massachusetts General Hospital, Boston; Department of Neurology (M.B.W.), Beth Israel Deaconess Medical Center, Boston, MA; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Neurology (L.K.J.), Mayo Clinic, Rochester, MN; and Department of Neurology (L.M.), Harvard Medical School, Boston, MA
| | - M Brandon Westover
- From the Department of Neurology (W.G., H.J.R., I.S.S., L.M.), Massachusetts General Hospital, Boston; Department of Neurology (M.B.W.), Beth Israel Deaconess Medical Center, Boston, MA; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Neurology (L.K.J.), Mayo Clinic, Rochester, MN; and Department of Neurology (L.M.), Harvard Medical School, Boston, MA
| | - Allison L Weathers
- From the Department of Neurology (W.G., H.J.R., I.S.S., L.M.), Massachusetts General Hospital, Boston; Department of Neurology (M.B.W.), Beth Israel Deaconess Medical Center, Boston, MA; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Neurology (L.K.J.), Mayo Clinic, Rochester, MN; and Department of Neurology (L.M.), Harvard Medical School, Boston, MA
| | - Lyell K Jones
- From the Department of Neurology (W.G., H.J.R., I.S.S., L.M.), Massachusetts General Hospital, Boston; Department of Neurology (M.B.W.), Beth Israel Deaconess Medical Center, Boston, MA; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Neurology (L.K.J.), Mayo Clinic, Rochester, MN; and Department of Neurology (L.M.), Harvard Medical School, Boston, MA
| | - Lidia Moura
- From the Department of Neurology (W.G., H.J.R., I.S.S., L.M.), Massachusetts General Hospital, Boston; Department of Neurology (M.B.W.), Beth Israel Deaconess Medical Center, Boston, MA; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Neurology (L.K.J.), Mayo Clinic, Rochester, MN; and Department of Neurology (L.M.), Harvard Medical School, Boston, MA.
| |
Collapse
|
9
|
Knight A, Gschwind T, Galer P, Worrell GA, Litt B, Soltesz I, Beniczky S. Artificial intelligence in epilepsy phenotyping. Epilepsia 2023:10.1111/epi.17833. [PMID: 37983589 PMCID: PMC11102939 DOI: 10.1111/epi.17833] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2023] [Accepted: 11/17/2023] [Indexed: 11/22/2023]
Abstract
Artificial intelligence (AI) allows data analysis and integration at an unprecedented granularity and scale. Here we review the technological advances, challenges, and future perspectives of using AI for electro-clinical phenotyping of animal models and patients with epilepsy. In translational research, AI models accurately identify behavioral states in animal models of epilepsy, allowing identification of correlations between neural activity and interictal and ictal behavior. Clinical applications of AI-based automated and semi-automated analysis of audio and video recordings of people with epilepsy, allow significant data reduction and reliable detection and classification of major motor seizures. AI models can accurately identify electrographic biomarkers of epilepsy, such as spikes, high-frequency oscillations, and seizure patterns. Integrating AI analysis of electroencephalographic, clinical, and behavioral data will contribute to optimizing therapy for patients with epilepsy.
Collapse
Affiliation(s)
| | - Tilo Gschwind
- Department of Neurosurgery, Stanford University, Stanford, USA
| | - Peter Galer
- Center for Neuroengineering and Therapeutics; Department of Bioengineering; Department of Neurology, Perelman School of Medicine, University of Pennsylvania, USA
| | | | - Brian Litt
- Center for Neuroengineering and Therapeutics; Department of Bioengineering; Department of Neurology, Perelman School of Medicine, University of Pennsylvania, USA
| | - Ivan Soltesz
- Department of Neurosurgery, Stanford University, Stanford, USA
| | - Sándor Beniczky
- Danish Epilepsy Centre Filadelfia, Dianalund, Aarhus University Hospital and Aarhus University, Aarhus, Denmark
| |
Collapse
|
10
|
Xie K, Terman SW, Gallagher RS, Hill CE, Davis KA, Litt B, Roth D, Ellis CA. Generalization of finetuned transformer language models to new clinical contexts. JAMIA Open 2023; 6:ooad070. [PMID: 37600072 PMCID: PMC10432353 DOI: 10.1093/jamiaopen/ooad070] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Revised: 08/01/2023] [Accepted: 08/03/2023] [Indexed: 08/22/2023] Open
Abstract
Objective We have previously developed a natural language processing pipeline using clinical notes written by epilepsy specialists to extract seizure freedom, seizure frequency text, and date of last seizure text for patients with epilepsy. It is important to understand how our methods generalize to new care contexts. Materials and methods We evaluated our pipeline on unseen notes from nonepilepsy-specialist neurologists and non-neurologists without any additional algorithm training. We tested the pipeline out-of-institution using epilepsy specialist notes from an outside medical center with only minor preprocessing adaptations. We examined reasons for discrepancies in performance in new contexts by measuring physical and semantic similarities between documents. Results Our ability to classify patient seizure freedom decreased by at least 0.12 agreement when moving from epilepsy specialists to nonspecialists or other institutions. On notes from our institution, textual overlap between the extracted outcomes and the gold standard annotations attained from manual chart review decreased by at least 0.11 F1 when an answer existed but did not change when no answer existed; here our models generalized on notes from the outside institution, losing at most 0.02 agreement. We analyzed textual differences and found that syntactic and semantic differences in both clinically relevant sentences and surrounding contexts significantly influenced model performance. Discussion and conclusion Model generalization performance decreased on notes from nonspecialists; out-of-institution generalization on epilepsy specialist notes required small changes to preprocessing but was especially good for seizure frequency text and date of last seizure text, opening opportunities for multicenter collaborations using these outcomes.
Collapse
Affiliation(s)
- Kevin Xie
- Department of Bioengineering, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Samuel W Terman
- Department of Neurology, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Ryan S Gallagher
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Chloe E Hill
- Department of Neurology, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Kathryn A Davis
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Brian Litt
- Department of Bioengineering, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Dan Roth
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Colin A Ellis
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| |
Collapse
|
11
|
Xie K, Ojemann WKS, Gallagher RS, Lucas A, Hill CE, Hamilton RH, Johnson KB, Roth D, Litt B, Ellis CA. Disparities in seizure outcomes revealed by large language models. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.09.20.23295842. [PMID: 37790442 PMCID: PMC10543059 DOI: 10.1101/2023.09.20.23295842] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/05/2023]
Abstract
Objective Large-language models (LLMs) in healthcare have the potential to propagate existing biases or introduce new ones. For people with epilepsy, social determinants of health are associated with disparities in access to care, but their impact on seizure outcomes among those with access to specialty care remains unclear. Here we (1) evaluated our validated, epilepsy-specific LLM for intrinsic bias, and (2) used LLM-extracted seizure outcomes to test the hypothesis that different demographic groups have different seizure outcomes. Methods First, we tested our LLM for intrinsic bias in the form of differential performance in demographic groups by race, ethnicity, sex, income, and health insurance in manually annotated notes. Next, we used LLM-classified seizure freedom at each office visit to test for outcome disparities in the same demographic groups, using univariable and multivariable analyses. Results We analyzed 84,675 clinic visits from 25,612 patients seen at our epilepsy center 2005-2022. We found no differences in the accuracy, or positive or negative class balance of outcome classifications across demographic groups. Multivariable analysis indicated worse seizure outcomes for female patients (OR 1.33, p = 3×10-8), those with public insurance (OR 1.53, p = 2×10-13), and those from lower-income zip codes (OR ≥ 1.22, p ≤ 6.6×10-3). Black patients had worse outcomes than White patients in univariable but not multivariable analysis (OR 1.03, p = 0.66). Significance We found no evidence that our LLM was intrinsically biased against any demographic group. Seizure freedom extracted by LLM revealed disparities in seizure outcomes across several demographic groups. These findings highlight the critical need to reduce disparities in the care of people with epilepsy.
Collapse
Affiliation(s)
- Kevin Xie
- University of Pennsylvania, Dept. of Bioengineering, Philadelphia, PA, USA
- University of Pennsylvania, Center for Neuroengineering and Therapeutics, Philadelphia, PA, USA
| | - William K S Ojemann
- University of Pennsylvania, Dept. of Bioengineering, Philadelphia, PA, USA
- University of Pennsylvania, Center for Neuroengineering and Therapeutics, Philadelphia, PA, USA
| | - Ryan S Gallagher
- University of Pennsylvania, Center for Neuroengineering and Therapeutics, Philadelphia, PA, USA
- University of Pennsylvania, Dept. of Neurology, Philadelphia, PA, USA
| | - Alfredo Lucas
- University of Pennsylvania, Dept. of Bioengineering, Philadelphia, PA, USA
- University of Pennsylvania, Center for Neuroengineering and Therapeutics, Philadelphia, PA, USA
- University of Pennsylvania, Dept. of Neurology, Philadelphia, PA, USA
| | - Chloé E Hill
- University of Michigan, Dept. of Neurology, Ann Arbor, MI, USA
| | - Roy H Hamilton
- University of Pennsylvania, Dept. of Neurology, Philadelphia, PA, USA
| | - Kevin B Johnson
- University of Pennsylvania, Dept. of Bioengineering, Philadelphia, PA, USA
- University of Pennsylvania, Dept. Of Biostatistics, Epidemiology and Informatics, Philadelphia, PA USA
- University of Pennsylvania, Dept. of Computer and Information Science, Philadelphia, PA, USA
- University of Pennsylvania, Dept. of Pediatrics, Philadelphia, PA, USA
| | - Dan Roth
- University of Pennsylvania, Dept. of Computer and Information Science, Philadelphia, PA, USA
| | - Brian Litt
- University of Pennsylvania, Dept. of Bioengineering, Philadelphia, PA, USA
- University of Pennsylvania, Center for Neuroengineering and Therapeutics, Philadelphia, PA, USA
- University of Pennsylvania, Dept. of Neurology, Philadelphia, PA, USA
| | - Colin A Ellis
- University of Pennsylvania, Center for Neuroengineering and Therapeutics, Philadelphia, PA, USA
- University of Pennsylvania, Dept. of Neurology, Philadelphia, PA, USA
| |
Collapse
|
12
|
Xie K, Gallagher RS, Shinohara RT, Xie SX, Hill CE, Conrad EC, Davis KA, Roth D, Litt B, Ellis CA. Long-term epilepsy outcome dynamics revealed by natural language processing of clinic notes. Epilepsia 2023; 64:1900-1909. [PMID: 37114472 PMCID: PMC10523917 DOI: 10.1111/epi.17633] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2022] [Revised: 04/26/2023] [Accepted: 04/26/2023] [Indexed: 04/29/2023]
Abstract
OBJECTIVE Electronic medical records allow for retrospective clinical research with large patient cohorts. However, epilepsy outcomes are often contained in free text notes that are difficult to mine. We recently developed and validated novel natural language processing (NLP) algorithms to automatically extract key epilepsy outcome measures from clinic notes. In this study, we assessed the feasibility of extracting these measures to study the natural history of epilepsy at our center. METHODS We applied our previously validated NLP algorithms to extract seizure freedom, seizure frequency, and date of most recent seizure from outpatient visits at our epilepsy center from 2010 to 2022. We examined the dynamics of seizure outcomes over time using Markov model-based probability and Kaplan-Meier analyses. RESULTS Performance of our algorithms on classifying seizure freedom was comparable to that of human reviewers (algorithm F1 = .88 vs. human annotatorκ = .86). We extracted seizure outcome data from 55 630 clinic notes from 9510 unique patients written by 53 unique authors. Of these, 30% were classified as seizure-free since the last visit, 48% of non-seizure-free visits contained a quantifiable seizure frequency, and 47% of all visits contained the date of most recent seizure occurrence. Among patients with at least five visits, the probabilities of seizure freedom at the next visit ranged from 12% to 80% in patients having seizures or seizure-free at the prior three visits, respectively. Only 25% of patients who were seizure-free for 6 months remained seizure-free after 10 years. SIGNIFICANCE Our findings demonstrate that epilepsy outcome measures can be extracted accurately from unstructured clinical note text using NLP. At our tertiary center, the disease course often followed a remitting and relapsing pattern. This method represents a powerful new tool for clinical research with many potential uses and extensions to other clinical questions.
Collapse
Affiliation(s)
- Kevin Xie
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Ryan S. Gallagher
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Russell T. Shinohara
- Penn Statistics in Imaging and Visualization Center, Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Sharon X. Xie
- Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Chloe E. Hill
- Department of Neurology, University of Michigan, Ann Arbor, MI, 48109, USA
| | - Erin C. Conrad
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Kathryn A. Davis
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Dan Roth
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Brian Litt
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Colin A. Ellis
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, 19104, USA
| |
Collapse
|
13
|
Houssein EH, Mohamed RE, Ali AA. Heart disease risk factors detection from electronic health records using advanced NLP and deep learning techniques. Sci Rep 2023; 13:7173. [PMID: 37138014 PMCID: PMC10156668 DOI: 10.1038/s41598-023-34294-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2022] [Accepted: 04/27/2023] [Indexed: 05/05/2023] Open
Abstract
Heart disease remains the major cause of death, despite recent improvements in prediction and prevention. Risk factor identification is the main step in diagnosing and preventing heart disease. Automatically detecting risk factors for heart disease in clinical notes can help with disease progression modeling and clinical decision-making. Many studies have attempted to detect risk factors for heart disease, but none have identified all risk factors. These studies have proposed hybrid systems that combine knowledge-driven and data-driven techniques, based on dictionaries, rules, and machine learning methods that require significant human effort. The National Center for Informatics for Integrating Biology and Beyond (i2b2) proposed a clinical natural language processing (NLP) challenge in 2014, with a track (track2) focused on detecting risk factors for heart disease risk factors in clinical notes over time. Clinical narratives provide a wealth of information that can be extracted using NLP and Deep Learning techniques. The objective of this paper is to improve on previous work in this area as part of the 2014 i2b2 challenge by identifying tags and attributes relevant to disease diagnosis, risk factors, and medications by providing advanced techniques of using stacked word embeddings. The i2b2 heart disease risk factors challenge dataset has shown significant improvement by using the approach of stacking embeddings, which combines various embeddings. Our model achieved an F1 score of 93.66% by using BERT and character embeddings (CHARACTER-BERT Embedding) stacking. The proposed model has significant results compared to all other models and systems that we developed for the 2014 i2b2 challenge.
Collapse
Affiliation(s)
- Essam H Houssein
- Faculty of Computers and Information, Minia University, Minia, Egypt.
| | - Rehab E Mohamed
- Faculty of Computers and Information, Minia University, Minia, Egypt
| | - Abdelmgeid A Ali
- Faculty of Computers and Information, Minia University, Minia, Egypt
| |
Collapse
|
14
|
Ghosn NJ, Xie K, Pattnaik AR, Gugger JJ, Ellis CA, Sweeney E, Fox E, Bernabei JM, Johnson J, Boccanfuso J, Litt B, Conrad EC. A pharmacokinetic model of antiseizure medication load to guide care in the epilepsy monitoring unit. Epilepsia 2023; 64:1236-1247. [PMID: 36815252 PMCID: PMC10424095 DOI: 10.1111/epi.17558] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2022] [Revised: 02/21/2023] [Accepted: 02/21/2023] [Indexed: 02/24/2023]
Abstract
OBJECTIVE Evaluating patients with drug-resistant epilepsy often requires inducing seizures by tapering antiseizure medications (ASMs) in the epilepsy monitoring unit (EMU). The relationship between ASM taper strategy, seizure timing, and severity remains unclear. In this study, we developed and validated a pharmacokinetic model of total ASM load and tested its association with seizure occurrence and severity in the EMU. METHODS We studied 80 patients who underwent intracranial electroencephalographic recording for epilepsy surgery planning. We developed a first order pharmacokinetic model of the ASMs administered in the EMU to generate a continuous metric of overall ASM load. We then related modeled ASM load to seizure likelihood and severity. We determined the association between the rate of ASM load reduction, the length of hospital stay, and the probability of having a severe seizure. Finally, we used modeled ASM load to predict oncoming seizures. RESULTS Seizures occurred in the bottom 50th percentile of sampled ASM loads across the cohort (p < .0001, Wilcoxon signed-rank test), and seizures requiring rescue therapy occurred at lower ASM loads than seizures that did not require rescue therapy (logistic regression mixed effects model, odds ratio = .27, p = .01). Greater ASM decrease early in the EMU was not associated with an increased likelihood of having a severe seizure, nor with a shorter length of stay. SIGNIFICANCE A pharmacokinetic model can accurately estimate ASM levels for patients in the EMU. Lower modeled ASM levels are associated with increased seizure likelihood and seizure severity. We show that ASM load, rather than ASM taper speed, is associated with severe seizures. ASM modeling has the potential to help optimize taper strategy to minimize severe seizures while maximizing diagnostic yield.
Collapse
Affiliation(s)
- Nina J. Ghosn
- Department of Bioengineering, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Kevin Xie
- Department of Bioengineering, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Akash R. Pattnaik
- Department of Bioengineering, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - James J. Gugger
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Colin A. Ellis
- Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Elizabeth Sweeney
- Penn Statistics in Imaging and Visualization Endeavor Center, Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Emily Fox
- Department of Statistics, Stanford University, Stanford, California, USA
- Department of Computer Science, Stanford University, Stanford, California, USA
- Chan Zuckerberg Biohub, San Francisco, California, USA
| | - John M. Bernabei
- Department of Bioengineering, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Jenaye Johnson
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Jacqueline Boccanfuso
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Brian Litt
- Department of Bioengineering, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Erin C. Conrad
- Department of Bioengineering, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| |
Collapse
|
15
|
Gombolay GY, Gopalan N, Bernasconi A, Nabbout R, Megerian JT, Siegel B, Hallman-Cooper J, Bhalla S, Gombolay MC. Review of Machine Learning and Artificial Intelligence (ML/AI) for the Pediatric Neurologist. Pediatr Neurol 2023; 141:42-51. [PMID: 36773406 PMCID: PMC10040433 DOI: 10.1016/j.pediatrneurol.2023.01.004] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/22/2022] [Revised: 01/03/2023] [Accepted: 01/09/2023] [Indexed: 01/15/2023]
Abstract
Artificial intelligence (AI) and a popular branch of AI known as machine learning (ML) are increasingly being utilized in medicine and to inform medical research. This review provides an overview of AI and ML (AI/ML), including definitions of common terms. We discuss the history of AI and provide instances of how AI/ML can be applied to pediatric neurology. Examples include imaging in neuro-oncology, autism diagnosis, diagnosis from charts, epilepsy, cerebral palsy, and neonatal neurology. Topics such as supervised learning, unsupervised learning, and reinforcement learning are discussed.
Collapse
Affiliation(s)
- Grace Y Gombolay
- Division of Neurology, Department of Pediatrics, Emory University School of Medicine, Atlanta Georgia; Division of Pediatric Neurology, Children's Healthcare of Atlanta, Atlanta Georgia.
| | - Nakul Gopalan
- Georgia Institute of Technology, Interactive Computing, Atlanta, Georgia
| | - Andrea Bernasconi
- Neuroimaging of Epilepsy Laboratory, McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University, Montreal, UK
| | - Rima Nabbout
- Department of Pediatric Neurology, Necker Enfants Malades Hospital, Reference Centre for Rare Epilepsies and Member of the ERN EpiCARE, Imagine Institute UMR1163, Paris Descartes University, Paris, France
| | - Jonathan T Megerian
- Department of Pediatrics, CHOC Children's, Irvine School of Medicine, University of California, Orange, California
| | - Benjamin Siegel
- Division of Neurology, Department of Pediatrics, Emory University School of Medicine, Atlanta Georgia; Division of Pediatric Neurology, Children's Healthcare of Atlanta, Atlanta Georgia
| | - Jamika Hallman-Cooper
- Division of Neurology, Department of Pediatrics, Emory University School of Medicine, Atlanta Georgia; Division of Pediatric Neurology, Children's Healthcare of Atlanta, Atlanta Georgia
| | - Sonam Bhalla
- Division of Neurology, Department of Pediatrics, Emory University School of Medicine, Atlanta Georgia; Division of Pediatric Neurology, Children's Healthcare of Atlanta, Atlanta Georgia
| | - Matthew C Gombolay
- Georgia Institute of Technology, Interactive Computing, Atlanta, Georgia
| |
Collapse
|
16
|
Boßelmann CM, Leu C, Lal D. Are AI language models such as ChatGPT ready to improve the care of individuals with epilepsy? Epilepsia 2023; 64:1195-1199. [PMID: 36869421 DOI: 10.1111/epi.17570] [Citation(s) in RCA: 17] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2023] [Revised: 02/23/2023] [Accepted: 03/01/2023] [Indexed: 03/05/2023]
Affiliation(s)
- Christian M Boßelmann
- Genomic Medicine Institute, Lerner Research Institute, Cleveland Clinic, Cleveland, Ohio, USA.,Epilepsy Center, Neurological Institute, Cleveland Clinic, Cleveland, Ohio, USA
| | - Costin Leu
- Genomic Medicine Institute, Lerner Research Institute, Cleveland Clinic, Cleveland, Ohio, USA.,Department of Clinical and Experimental Epilepsy, Institute of Neurology, University College London, London, UK
| | - Dennis Lal
- Genomic Medicine Institute, Lerner Research Institute, Cleveland Clinic, Cleveland, Ohio, USA.,Epilepsy Center, Neurological Institute, Cleveland Clinic, Cleveland, Ohio, USA.,Stanley Center for Psychiatric Research, Broad Institute of Harvard and M.I.T., Cambridge, Massachusetts, USA.,Cologne Center for Genomics (CCG), University of Cologne, Cologne, Delaware, USA
| |
Collapse
|
17
|
Yew ANJ, Schraagen M, Otte WM, van Diessen E. Transforming epilepsy research: A systematic review on natural language processing applications. Epilepsia 2023; 64:292-305. [PMID: 36462150 PMCID: PMC10108221 DOI: 10.1111/epi.17474] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2022] [Revised: 11/23/2022] [Accepted: 12/01/2022] [Indexed: 12/05/2022]
Abstract
Despite improved ancillary investigations in epilepsy care, patients' narratives remain indispensable for diagnosing and treatment monitoring. This wealth of information is typically stored in electronic health records and accumulated in medical journals in an unstructured manner, thereby restricting complete utilization in clinical decision-making. To this end, clinical researchers increasing apply natural language processing (NLP)-a branch of artificial intelligence-as it removes ambiguity, derives context, and imbues standardized meaning from free-narrative clinical texts. This systematic review presents an overview of the current NLP applications in epilepsy and discusses the opportunities and drawbacks of NLP alongside its future implications. We searched the PubMed and Embase databases with a "natural language processing" and "epilepsy" query (March 4, 2022) and included original research articles describing the application of NLP techniques for textual analysis in epilepsy. Twenty-six studies were included. Fifty-eight percent of these studies used NLP to classify clinical records into predefined categories, improving patient identification and treatment decisions. Other applications of NLP had structured clinical information retrieval from electronic health records, scientific papers, and online posts of patients. Challenges and opportunities of NLP applications for enhancing epilepsy care and research are discussed. The field could further benefit from NLP by replicating successes in other health care domains, such as NLP-aided quality evaluation for clinical decision-making, outcome prediction, and clinical record summarization.
Collapse
Affiliation(s)
- Arister N J Yew
- University College Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Marijn Schraagen
- Department of Information and Computing Sciences, Faculty of Science, Utrecht University, Utrecht, The Netherlands
| | - Willem M Otte
- Department of Child Neurology, Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands
| | - Eric van Diessen
- Department of Child Neurology, Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
18
|
Yang S, Varghese P, Stephenson E, Tu K, Gronsbell J. Machine learning approaches for electronic health records phenotyping: a methodical review. J Am Med Inform Assoc 2023; 30:367-381. [PMID: 36413056 PMCID: PMC9846699 DOI: 10.1093/jamia/ocac216] [Citation(s) in RCA: 23] [Impact Index Per Article: 23.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2022] [Revised: 09/27/2022] [Accepted: 10/27/2022] [Indexed: 11/23/2022] Open
Abstract
OBJECTIVE Accurate and rapid phenotyping is a prerequisite to leveraging electronic health records for biomedical research. While early phenotyping relied on rule-based algorithms curated by experts, machine learning (ML) approaches have emerged as an alternative to improve scalability across phenotypes and healthcare settings. This study evaluates ML-based phenotyping with respect to (1) the data sources used, (2) the phenotypes considered, (3) the methods applied, and (4) the reporting and evaluation methods used. MATERIALS AND METHODS We searched PubMed and Web of Science for articles published between 2018 and 2022. After screening 850 articles, we recorded 37 variables on 100 studies. RESULTS Most studies utilized data from a single institution and included information in clinical notes. Although chronic conditions were most commonly considered, ML also enabled the characterization of nuanced phenotypes such as social determinants of health. Supervised deep learning was the most popular ML paradigm, while semi-supervised and weakly supervised learning were applied to expedite algorithm development and unsupervised learning to facilitate phenotype discovery. ML approaches did not uniformly outperform rule-based algorithms, but deep learning offered a marginal improvement over traditional ML for many conditions. DISCUSSION Despite the progress in ML-based phenotyping, most articles focused on binary phenotypes and few articles evaluated external validity or used multi-institution data. Study settings were infrequently reported and analytic code was rarely released. CONCLUSION Continued research in ML-based phenotyping is warranted, with emphasis on characterizing nuanced phenotypes, establishing reporting and evaluation standards, and developing methods to accommodate misclassified phenotypes due to algorithm errors in downstream applications.
Collapse
Affiliation(s)
- Siyue Yang
- Department of Statistical Sciences, University of Toronto, Toronto, Ontario, Canada
| | | | - Ellen Stephenson
- Department of Family & Community Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Karen Tu
- Department of Family & Community Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Jessica Gronsbell
- Department of Statistical Sciences, University of Toronto, Toronto, Ontario, Canada
- Department of Family & Community Medicine, University of Toronto, Toronto, Ontario, Canada
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
| |
Collapse
|
19
|
Bombieri M, Rospocher M, Ponzetto SP, Fiorini P. Machine understanding surgical actions from intervention procedure textbooks. Comput Biol Med 2023; 152:106415. [PMID: 36527782 DOI: 10.1016/j.compbiomed.2022.106415] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2022] [Revised: 11/23/2022] [Accepted: 12/04/2022] [Indexed: 12/12/2022]
Abstract
The automatic extraction of procedural surgical knowledge from surgery manuals, academic papers or other high-quality textual resources, is of the utmost importance to develop knowledge-based clinical decision support systems, to automatically execute some procedure's step or to summarize the procedural information, spread throughout the texts, in a structured form usable as a study resource by medical students. In this work, we propose a first benchmark on extracting detailed surgical actions from available intervention procedure textbooks and papers. We frame the problem as a Semantic Role Labeling task. Exploiting a manually annotated dataset, we apply different Transformer-based information extraction methods. Starting from RoBERTa and BioMedRoBERTa pre-trained language models, we first investigate a zero-shot scenario and compare the obtained results with a full fine-tuning setting. We then introduce a new ad-hoc surgical language model, named SurgicBERTa, pre-trained on a large collection of surgical materials, and we compare it with the previous ones. In the assessment, we explore different dataset splits (one in-domain and two out-of-domain) and we investigate also the effectiveness of the approach in a few-shot learning scenario. Performance is evaluated on three correlated sub-tasks: predicate disambiguation, semantic argument disambiguation and predicate-argument disambiguation. Results show that the fine-tuning of a pre-trained domain-specific language model achieves the highest performance on all splits and on all sub-tasks. All models are publicly released.
Collapse
Affiliation(s)
- Marco Bombieri
- Department of Computer Science, University of Verona, Verona, Italy.
| | - Marco Rospocher
- Department of Foreign Languages and Literatures, University of Verona, Verona, Italy
| | | | - Paolo Fiorini
- Department of Computer Science, University of Verona, Verona, Italy
| |
Collapse
|
20
|
Decker BM, Turco A, Xu J, Terman SW, Kosaraju N, Jamil A, Davis KA, Litt B, Ellis CA, Khankhanian P, Hill CE. Development of a natural language processing algorithm to extract seizure types and frequencies from the electronic health record. Seizure 2022; 101:48-51. [PMID: 35882104 PMCID: PMC9547963 DOI: 10.1016/j.seizure.2022.07.010] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2022] [Revised: 07/16/2022] [Accepted: 07/18/2022] [Indexed: 11/21/2022] Open
Abstract
OBJECTIVE To develop a natural language processing (NLP) algorithm to abstract seizure types and frequencies from electronic health records (EHR). BACKGROUND Seizure frequency measurement is an epilepsy quality metric. Yet, abstraction of seizure frequency from the EHR is laborious. We present an NLP algorithm to extract seizure data from unstructured text of clinic notes. Algorithm performance was assessed at two epilepsy centers. METHODS We developed a rules-based NLP algorithm to recognize terms related to seizures and frequency within the text of an outpatient encounter. Algorithm output (e.g. number of seizures of a particular type within a time interval) was compared to seizure data manually annotated by two expert reviewers ("gold standard"). The algorithm was developed from 150 clinic notes from institution #1 (development set), then tested on a separate set of 219 notes from institution #1 (internal test set) with 248 unique seizure frequency elements. The algorithm was separately applied to 100 notes from institution #2 (external test set) with 124 unique seizure frequency elements. Algorithm performance was measured by recall (sensitivity), precision (positive predictive value), and F1 score (geometric mean of precision and recall). RESULTS In the internal test set, the algorithm demonstrated 70% recall (173/248), 95% precision (173/182), and 0.82 F1 score compared to manual review. Algorithm performance in the external test set was lower with 22% recall (27/124), 73% precision (27/37), and 0.40 F1 score. CONCLUSIONS These results suggest NLP extraction of seizure types and frequencies is feasible, though not without challenges in generalizability for large-scale implementation.
Collapse
Affiliation(s)
- Barbara M Decker
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, United States; Department of Neurological Sciences, University of Vermont Medical Center, Burlington, VT, United States.
| | - Alexandra Turco
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, United States
| | - Jian Xu
- Department of Neurology, Henry Ford Health System, Detroit, MI, United States
| | - Samuel W Terman
- Department of Neurology, University of Michigan, Ann Arbor, MI, United States
| | - Nikitha Kosaraju
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, United States
| | - Alisha Jamil
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, United States
| | - Kathryn A Davis
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, United States
| | - Brian Litt
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, United States
| | - Colin A Ellis
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, United States
| | | | - Chloe E Hill
- Department of Neurology, University of Michigan, Ann Arbor, MI, United States
| |
Collapse
|