1
|
Moura L, Jones DT, Sheikh IS, Murphy S, Kalfin M, Kummer BR, Weathers AL, Grinspan ZM, Silsbee HM, Jones LK, Patel AD. Implications of Large Language Models for Quality and Efficiency of Neurologic Care: Emerging Issues in Neurology. Neurology 2024; 102:e209497. [PMID: 38759131 DOI: 10.1212/wnl.0000000000209497] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/19/2024] Open
Abstract
Large language models (LLMs) are advanced artificial intelligence (AI) systems that excel in recognizing and generating human-like language, possibly serving as valuable tools for neurology-related information tasks. Although LLMs have shown remarkable potential in various areas, their performance in the dynamic environment of daily clinical practice remains uncertain. This article outlines multiple limitations and challenges of using LLMs in clinical settings that need to be addressed, including limited clinical reasoning, variable reliability and accuracy, reproducibility bias, self-serving bias, sponsorship bias, and potential for exacerbating health care disparities. These challenges are further compounded by practical business considerations and infrastructure requirements, including associated costs. To overcome these hurdles and harness the potential of LLMs effectively, this article includes considerations for health care organizations, researchers, and neurologists contemplating the use of LLMs in clinical practice. It is essential for health care organizations to cultivate a culture that welcomes AI solutions and aligns them seamlessly with health care operations. Clear objectives and business plans should guide the selection of AI solutions, ensuring they meet organizational needs and budget considerations. Engaging both clinical and nonclinical stakeholders can help secure necessary resources, foster trust, and ensure the long-term sustainability of AI implementations. Testing, validation, training, and ongoing monitoring are pivotal for successful integration. For neurologists, safeguarding patient data privacy is paramount. Seeking guidance from institutional information technology resources for informed, compliant decisions, and remaining vigilant against biases in LLM outputs are essential practices in responsible and unbiased utilization of AI tools. In research, obtaining institutional review board approval is crucial when dealing with patient data, even if deidentified, to ensure ethical use. Compliance with established guidelines like SPIRIT-AI, MI-CLAIM, and CONSORT-AI is necessary to maintain consistency and mitigate biases in AI research. In summary, the integration of LLMs into clinical neurology offers immense promise while presenting formidable challenges. Awareness of these considerations is vital for harnessing the potential of AI in neurologic care effectively and enhancing patient care quality and safety. The article serves as a guide for health care organizations, researchers, and neurologists navigating this transformative landscape.
Collapse
Affiliation(s)
- Lidia Moura
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - David T Jones
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Irfan S Sheikh
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Shawn Murphy
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Michael Kalfin
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Benjamin R Kummer
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Allison L Weathers
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Zachary M Grinspan
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Heather M Silsbee
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Lyell K Jones
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Anup D Patel
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| |
Collapse
|
2
|
Osborne G, Valenti O, Jarvis J, Wentzel E, Vidaurre J, Clarke DF, Patel AD. Implementing American Academy of Neurology Quality Measures in Antigua Using Quality Improvement Methodology. Neurol Clin Pract 2024; 14:e200231. [PMID: 38152065 PMCID: PMC10751012 DOI: 10.1212/cpj.0000000000200231] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2023] [Accepted: 10/10/2023] [Indexed: 12/29/2023]
Abstract
Background and Objectives The American Academy of Neurology has developed quality measures related to various neurologic disorders. A gap exists in the implementation of these measures in the different health care systems. To date, there has been no electronic health care record nor implementation of quality measures in Antigua. Therefore, we aimed to increase the percent of patients who have epilepsy quality measures documented using standardized common data elements in the outpatient neurology clinic at Sir Lester Bird Medical Center from 0% to 80% per week by June 1, 2022 and sustain for 6 months. Methods We used the Institute for Health care Improvement Model for Improvement methodology. A data use agreement was implemented. Data were displayed using statistical process control charts and the American Society for Quality criteria to determine statistical significance and centerline shifts. Results Current and future state process maps were developed to determine areas of opportunity for interventions. Interventions were developed following a "Plan-Do-Study-Act cycle." One intervention was the creation of a RedCap survey and database to be used by health care providers during clinical patient encounters. Because of multiple interventions, we achieved a 100% utilization of the survey for clinical care. Discussion Quality improvement (QI) methodology can be used for implementation of quality measures in various settings to improve patient care outcomes without use of significant resources. Implementation of quality measures can increase efficiency in clinical delivery. Similar QI methodology could be implemented in other resource-limited countries of the Caribbean and globally.
Collapse
Affiliation(s)
- Gaden Osborne
- Neurology Department (GO, JJ), Sir Lester Bird Medical Centre, St. John's, Antigua, West Indies; The Center for Clinical Excellence (OV, ADP); Division of Neurology (EW, JV, ADP), Nationwide Children's Hospital, Columbus, OH; and Pediatric Neurology (DFC), Dell Medical School, the University of Texas at Austin
| | - Olivia Valenti
- Neurology Department (GO, JJ), Sir Lester Bird Medical Centre, St. John's, Antigua, West Indies; The Center for Clinical Excellence (OV, ADP); Division of Neurology (EW, JV, ADP), Nationwide Children's Hospital, Columbus, OH; and Pediatric Neurology (DFC), Dell Medical School, the University of Texas at Austin
| | - Juniella Jarvis
- Neurology Department (GO, JJ), Sir Lester Bird Medical Centre, St. John's, Antigua, West Indies; The Center for Clinical Excellence (OV, ADP); Division of Neurology (EW, JV, ADP), Nationwide Children's Hospital, Columbus, OH; and Pediatric Neurology (DFC), Dell Medical School, the University of Texas at Austin
| | - Evelynne Wentzel
- Neurology Department (GO, JJ), Sir Lester Bird Medical Centre, St. John's, Antigua, West Indies; The Center for Clinical Excellence (OV, ADP); Division of Neurology (EW, JV, ADP), Nationwide Children's Hospital, Columbus, OH; and Pediatric Neurology (DFC), Dell Medical School, the University of Texas at Austin
| | - Jorge Vidaurre
- Neurology Department (GO, JJ), Sir Lester Bird Medical Centre, St. John's, Antigua, West Indies; The Center for Clinical Excellence (OV, ADP); Division of Neurology (EW, JV, ADP), Nationwide Children's Hospital, Columbus, OH; and Pediatric Neurology (DFC), Dell Medical School, the University of Texas at Austin
| | - Dave F Clarke
- Neurology Department (GO, JJ), Sir Lester Bird Medical Centre, St. John's, Antigua, West Indies; The Center for Clinical Excellence (OV, ADP); Division of Neurology (EW, JV, ADP), Nationwide Children's Hospital, Columbus, OH; and Pediatric Neurology (DFC), Dell Medical School, the University of Texas at Austin
| | - Anup D Patel
- Neurology Department (GO, JJ), Sir Lester Bird Medical Centre, St. John's, Antigua, West Indies; The Center for Clinical Excellence (OV, ADP); Division of Neurology (EW, JV, ADP), Nationwide Children's Hospital, Columbus, OH; and Pediatric Neurology (DFC), Dell Medical School, the University of Texas at Austin
| |
Collapse
|
3
|
Fernandes M, Donahue MA, Hoch D, Cash S, Zafar S, Jacobs C, Hosford M, Voinescu PE, Fureman B, Buchhalter J, McGraw CM, Westover MB, Moura LMVR. A replicable, open-source, data integration method to support national practice-based research & quality improvement systems. Epilepsy Res 2022; 186:107013. [PMID: 35994859 PMCID: PMC9810436 DOI: 10.1016/j.eplepsyres.2022.107013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2022] [Revised: 04/28/2022] [Accepted: 08/13/2022] [Indexed: 01/07/2023]
Abstract
OBJECTIVES The Epilepsy Learning Healthcare System (ELHS) was created in 2018 to address measurable improvements in outcomes for people with epilepsy. However, fragmentation of data systems has been a major barrier for reporting and participation. In this study, we aimed to test the feasibility of an open-source Data Integration (DI) method that connects real-life clinical data to national research and quality improvement (QI) systems. METHODS The ELHS case report forms were programmed as EPIC SmartPhrases at Mass General Brigham (MGB) in December 2018 and subsequently as EPIC SmartForms in June 2021 to collect actionable, standardized, structured epilepsy data in the electronic health record (EHR) for subsequent pull into the external national registry of the ELHS. Following the QI methodology in the Chronic Care Model, 39 providers, epileptologists and neurologists, incorporated the ELHS SmartPhrase into their clinical workflow, focusing on collecting diagnosis of epilepsy, seizure type according to the International League Against Epilepsy, seizure frequency, date of last seizure, medication adherence and side effects. The collected data was stored in the Enterprise Data Warehouse (EDW) without integration with external systems. We developed and validated a DI method that extracted the data from EDW using structured query language and later preprocessed using text mining. We used the ELHS data dictionary to match fields in the preprocessed notes to obtain the final structured dataset with seizure control information. For illustration, we described the data curated from the care period of 12/2018-12/2021. RESULTS The cohort comprised a total of 1806 patients with a mean age of 43 years old (SD: 17.0), where 57% were female, 80% were white, and 84% were non-Hispanic/Latino. Using our DI method, we automated the data mining, preprocessing, and exporting of the structured dataset into a local database, to be weekly accessible to clinicians and quality improvers. During the period of SmartPhrase implementation, there were 5168 clinic visits logged by providers documenting each patient's seizure type and frequency. During this period, providers documented 59% patients having focal seizures, 35% having generalized seizures and 6% patients having another type. Of the cohort, 45% patients had private insurance. The resulting structured dataset was bulk uploaded via web interface into the external national registry of the ELHS. CONCLUSIONS Structured data can be feasibly extracted from text notes of epilepsy patients for weekly reporting to a national learning healthcare system.
Collapse
Affiliation(s)
- Marta Fernandes
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States; Harvard Medical School, Boston, MA, United States; Clinical Data Animation Center (CDAC), MGH, Boston, MA, United States.
| | - Maria A Donahue
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States; Harvard Medical School, Boston, MA, United States; The NeuroValue Lab, MGH, Boston, MA, United States.
| | - Dan Hoch
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States; Harvard Medical School, Boston, MA, United States.
| | - Sydney Cash
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States; Harvard Medical School, Boston, MA, United States.
| | - Sahar Zafar
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States; Harvard Medical School, Boston, MA, United States.
| | - Claire Jacobs
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States; Harvard Medical School, Boston, MA, United States.
| | - Mackenzie Hosford
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States; Harvard Medical School, Boston, MA, United States.
| | - P Emanuela Voinescu
- Harvard Medical School, Boston, MA, United States; Department of Neurology, Division of Epilepsy, Division of Women's Health, Brigham and Women's Hospital, Boston, MA, United States.
| | | | - Jeffrey Buchhalter
- Department of Pediatrics, University of Calgary School of Medicine, Calgary, Canada.
| | - Christopher Michael McGraw
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States; Harvard Medical School, Boston, MA, United States.
| | - M Brandon Westover
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States; Harvard Medical School, Boston, MA, United States; Clinical Data Animation Center (CDAC), MGH, Boston, MA, United States; McCance Center for Brain Health, MGH, Boston, MA, United States.
| | - Lidia M V R Moura
- Department of Neurology, Massachusetts General Hospital (MGH), Boston, MA, United States; Harvard Medical School, Boston, MA, United States; The NeuroValue Lab, MGH, Boston, MA, United States.
| |
Collapse
|