4
|
Moura L, Jones DT, Sheikh IS, Murphy S, Kalfin M, Kummer BR, Weathers AL, Grinspan ZM, Silsbee HM, Jones LK, Patel AD. Implications of Large Language Models for Quality and Efficiency of Neurologic Care: Emerging Issues in Neurology. Neurology 2024; 102:e209497. [PMID: 38759131 DOI: 10.1212/wnl.0000000000209497] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/19/2024] Open
Abstract
Large language models (LLMs) are advanced artificial intelligence (AI) systems that excel in recognizing and generating human-like language, possibly serving as valuable tools for neurology-related information tasks. Although LLMs have shown remarkable potential in various areas, their performance in the dynamic environment of daily clinical practice remains uncertain. This article outlines multiple limitations and challenges of using LLMs in clinical settings that need to be addressed, including limited clinical reasoning, variable reliability and accuracy, reproducibility bias, self-serving bias, sponsorship bias, and potential for exacerbating health care disparities. These challenges are further compounded by practical business considerations and infrastructure requirements, including associated costs. To overcome these hurdles and harness the potential of LLMs effectively, this article includes considerations for health care organizations, researchers, and neurologists contemplating the use of LLMs in clinical practice. It is essential for health care organizations to cultivate a culture that welcomes AI solutions and aligns them seamlessly with health care operations. Clear objectives and business plans should guide the selection of AI solutions, ensuring they meet organizational needs and budget considerations. Engaging both clinical and nonclinical stakeholders can help secure necessary resources, foster trust, and ensure the long-term sustainability of AI implementations. Testing, validation, training, and ongoing monitoring are pivotal for successful integration. For neurologists, safeguarding patient data privacy is paramount. Seeking guidance from institutional information technology resources for informed, compliant decisions, and remaining vigilant against biases in LLM outputs are essential practices in responsible and unbiased utilization of AI tools. In research, obtaining institutional review board approval is crucial when dealing with patient data, even if deidentified, to ensure ethical use. Compliance with established guidelines like SPIRIT-AI, MI-CLAIM, and CONSORT-AI is necessary to maintain consistency and mitigate biases in AI research. In summary, the integration of LLMs into clinical neurology offers immense promise while presenting formidable challenges. Awareness of these considerations is vital for harnessing the potential of AI in neurologic care effectively and enhancing patient care quality and safety. The article serves as a guide for health care organizations, researchers, and neurologists navigating this transformative landscape.
Collapse
Affiliation(s)
- Lidia Moura
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - David T Jones
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Irfan S Sheikh
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Shawn Murphy
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Michael Kalfin
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Benjamin R Kummer
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Allison L Weathers
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Zachary M Grinspan
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Heather M Silsbee
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Lyell K Jones
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| | - Anup D Patel
- From the Center for Value-based Health Care and Sciences (L.M.), and Department of Neurology (L.M., S.M.), Massachusetts General Hospital, Boston; Harvard Medical School (L.M., S.M.), Boston, MA; Department of Neurology (D.T.J., L.K.J.), Mayo Clinic, Rochester, MN; Department of Neurology (I.S.S.), University of Texas Southwestern Medical Center, Dallas; Department of Neurology (M.K.), University of Pennsylvania Health System, Philadelphia; Department of Neurology (B.R.K.), Icahn School of Medicine at Mount Sinai, New York, NY; Information Technology Division (A.L.W.), Cleveland Clinic, OH; Department of Pediatrics (Z.M.G.), Weill Cornell Medicine, New York, NY; American Academy of Neurology (H.M.S.), Minneapolis, MN; and The Center for Clinical Excellence (A.D.P.), Nationwide Children's Hospital, Division of Neurology, The Ohio State University College of Medicine, Columbus
| |
Collapse
|
5
|
Duarte M, Salamanca M, Gonzalez JM, Roman Laporte R, Gattamorta K, Lopez Martinez FE, Clochesy J, Rincon Acuna JC. Prediction of Positive Patient Health Questionnaire-2 Screening Using Area Deprivation Index in Primary Care. Clin Nurs Res 2024; 33:355-369. [PMID: 38801166 DOI: 10.1177/10547738241252887] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/29/2024]
Abstract
Depression is recognized as a significant public health issue in the United States. The National Survey on Drug Use and Health reports that 21.0 million adults aged 18 or older had major depressive disorder in 2020, including 14.8 million experiencing a major depressive episode with severe impairment. The aim is to predict the positivity of Patient Health Questionnaire-2 (PHQ-2) outcomes among patients in primary care settings by analyzing a range of variables, including socioeconomic status, demographic characteristics, and health behaviors, thereby identifying those at increased risk for depression. Employing a machine learning approach, the study utilizes retrospective data from electronic health records across 15 primary care clinics in South Florida to explore the relationship between social determinants of health (SDoH), including area of deprivation index (ADI) and PHQ-2 positivity. The study encompasses 15 primary care clinics located in South Florida, where a diverse patient population receives care. Analysis included 94,572 patient visits; 74,636 records were included in the study. If a zip+4 was not available or an ADI score did not exist, the visit was not included in the final analysis. Screening involved the PHQ-2, assessing depressed mood and anhedonia, with a cutoff >2 indicating positive screening. ADI was used to assess SDoH by matching patients' residential postal codes to ADI national percentiles. Demographics, sexual history, tobacco use, caffeine intake, and community involvement were also evaluated in the study. Over 40 machine learning algorithms were explored for their accuracy in predicting PHQ-2 outcomes, using software tools including Scikit-learn and stats models in Python. Variables were normalized, scored, and then subjected to predictive regression models, with Random Forest showing outstanding performance. Feature engineering and correlation analysis identified ADI, age, education, visit type, coffee intake, and marital status as significant predictors of PHQ-2 positivity. The area under the curve and model accuracies varied across clinics, with specific clinics showing higher predictive accuracy and others (p > .05). The study concludes that the ADI, as a proxy for SDoH, alongside other individual factors, can predict PHQ-2 positivity. Health organizations can use this information to anticipate health needs and resource allocation.
Collapse
Affiliation(s)
| | | | - Juan M Gonzalez
- University of Miami School of Nursing and Health Studies, Coral Gables, FL, USA
| | | | - Karina Gattamorta
- University of Miami School of Nursing and Health Studies, Coral Gables, FL, USA
| | | | - John Clochesy
- University of Miami School of Nursing and Health Studies, Coral Gables, FL, USA
| | | |
Collapse
|
8
|
Xu R, Chen Y, Yao Z, Wu W, Cui J, Wang R, Diao Y, Jin C, Hong Z, Li X. Application of machine learning algorithms to identify people with low bone density. Front Public Health 2024; 12:1347219. [PMID: 38726233 PMCID: PMC11080984 DOI: 10.3389/fpubh.2024.1347219] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2023] [Accepted: 03/29/2024] [Indexed: 05/12/2024] Open
Abstract
Background Osteoporosis is becoming more common worldwide, imposing a substantial burden on individuals and society. The onset of osteoporosis is subtle, early detection is challenging, and population-wide screening is infeasible. Thus, there is a need to develop a method to identify those at high risk for osteoporosis. Objective This study aimed to develop a machine learning algorithm to effectively identify people with low bone density, using readily available demographic and blood biochemical data. Methods Using NHANES 2017-2020 data, participants over 50 years old with complete femoral neck BMD data were selected. This cohort was randomly divided into training (70%) and test (30%) sets. Lasso regression selected variables for inclusion in six machine learning models built on the training data: logistic regression (LR), support vector machine (SVM), gradient boosting machine (GBM), naive Bayes (NB), artificial neural network (ANN) and random forest (RF). NHANES data from the 2013-2014 cycle was used as an external validation set input into the models to verify their generalizability. Model discrimination was assessed via AUC, accuracy, sensitivity, specificity, precision and F1 score. Calibration curves evaluated goodness-of-fit. Decision curves determined clinical utility. The SHAP framework analyzed variable importance. Results A total of 3,545 participants were included in the internal validation set of this study, of whom 1870 had normal bone density and 1,675 had low bone density Lasso regression selected 19 variables. In the test set, AUC was 0.785 (LR), 0.780 (SVM), 0.775 (GBM), 0.729 (NB), 0.771 (ANN), and 0.768 (RF). The LR model has the best discrimination and a better calibration curve fit, the best clinical net benefit for the decision curve, and it also reflects good predictive power in the external validation dataset The top variables in the LR model were: age, BMI, gender, creatine phosphokinase, total cholesterol and alkaline phosphatase. Conclusion The machine learning model demonstrated effective classification of low BMD using blood biomarkers. This could aid clinical decision making for osteoporosis prevention and management.
Collapse
Affiliation(s)
- Rongxuan Xu
- Department of Epidemiology and Health Statistics, Dalian Medical University, Dalian, China
| | - Yongxing Chen
- Department of Epidemiology and Health Statistics, Dalian Medical University, Dalian, China
| | - Zhihan Yao
- Department of Epidemiology and Health Statistics, Dalian Medical University, Dalian, China
| | - Wei Wu
- Department of Epidemiology and Health Statistics, Dalian Medical University, Dalian, China
| | - Jiaxue Cui
- Department of Epidemiology and Health Statistics, Dalian Medical University, Dalian, China
| | - Ruiqi Wang
- Department of Epidemiology and Health Statistics, Dalian Medical University, Dalian, China
| | - Yizhuo Diao
- Department of Epidemiology and Health Statistics, Dalian Medical University, Dalian, China
| | - Chenxin Jin
- Department of Epidemiology and Health Statistics, Dalian Medical University, Dalian, China
| | - Zhijun Hong
- The Health Management Center, The First Affiliated Hospital of Dalian Medical University, Dalian, Liaoning, China
| | - Xiaofeng Li
- Department of Epidemiology and Health Statistics, Dalian Medical University, Dalian, China
| |
Collapse
|