1
|
Nunes M, Boné J, Ferreira JC, Chaves P, Elvas LB. MediAlbertina: An European Portuguese medical language model. Comput Biol Med 2024; 182:109233. [PMID: 39362002 DOI: 10.1016/j.compbiomed.2024.109233] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2024] [Revised: 09/28/2024] [Accepted: 09/30/2024] [Indexed: 10/05/2024]
Abstract
BACKGROUND Patient medical information often exists in unstructured text containing abbreviations and acronyms deemed essential to conserve time and space but posing challenges for automated interpretation. Leveraging the efficacy of Transformers in natural language processing, our objective was to use the knowledge acquired by a language model and continue its pre-training to develop an European Portuguese (PT-PT) healthcare-domain language model. METHODS After carrying out a filtering process, Albertina PT-PT 900M was selected as our base language model, and we continued its pre-training using more than 2.6 million electronic medical records from Portugal's largest public hospital. MediAlbertina 900M has been created through domain adaptation on this data using masked language modelling. RESULTS The comparison with our baseline was made through the usage of both perplexity, which decreased from about 20 to 1.6 values, and the fine-tuning and evaluation of information extraction models such as Named Entity Recognition and Assertion Status. MediAlbertina PT-PT outperformed Albertina PT-PT in both tasks by 4-6% on recall and f1-score. CONCLUSIONS This study contributes with the first publicly available medical language model trained with PT-PT data. It underscores the efficacy of domain adaptation and offers a contribution to the scientific community in overcoming obstacles of non-English languages. With MediAlbertina, further steps can be taken to assist physicians, in creating decision support systems or building medical timelines in order to perform profiling, by fine-tuning MediAlbertina for PT- PT medical tasks.
Collapse
Affiliation(s)
- Miguel Nunes
- ISTAR, Instituto Universitário de Lisboa (ISCTE-IUL), 1649-026, Lisbon, Portugal
| | - João Boné
- Select Data, Anaheim, CA, 92807, USA
| | - João C Ferreira
- Department of Logistics, Molde University College, Molde, 6410, Norway; ISTAR, Instituto Universitário de Lisboa (ISCTE-IUL), 1649-026, Lisbon, Portugal; Inov Inesc Inovação - Instituto de Novas Tecnologias, 1000-029, Lisbon, Portugal
| | | | - Luis B Elvas
- Department of Logistics, Molde University College, Molde, 6410, Norway; ISTAR, Instituto Universitário de Lisboa (ISCTE-IUL), 1649-026, Lisbon, Portugal; Inov Inesc Inovação - Instituto de Novas Tecnologias, 1000-029, Lisbon, Portugal.
| |
Collapse
|
2
|
Osman M, Cooper R, Sayer AA, Witham MD. The use of natural language processing for the identification of ageing syndromes including sarcopenia, frailty and falls in electronic healthcare records: a systematic review. Age Ageing 2024; 53:afae135. [PMID: 38970549 PMCID: PMC11227113 DOI: 10.1093/ageing/afae135] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2023] [Indexed: 07/08/2024] Open
Abstract
BACKGROUND Recording and coding of ageing syndromes in hospital records is known to be suboptimal. Natural Language Processing algorithms may be useful to identify diagnoses in electronic healthcare records to improve the recording and coding of these ageing syndromes, but the feasibility and diagnostic accuracy of such algorithms are unclear. METHODS We conducted a systematic review according to a predefined protocol and in line with Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines. Searches were run from the inception of each database to the end of September 2023 in PubMed, Medline, Embase, CINAHL, ACM digital library, IEEE Xplore and Scopus. Eligible studies were identified via independent review of search results by two coauthors and data extracted from each study to identify the computational method, source of text, testing strategy and performance metrics. Data were synthesised narratively by ageing syndrome and computational method in line with the Studies Without Meta-analysis guidelines. RESULTS From 1030 titles screened, 22 studies were eligible for inclusion. One study focussed on identifying sarcopenia, one frailty, twelve falls, five delirium, five dementia and four incontinence. Sensitivity (57.1%-100%) of algorithms compared with a reference standard was reported in 20 studies, and specificity (84.0%-100%) was reported in only 12 studies. Study design quality was variable with results relevant to diagnostic accuracy not always reported, and few studies undertaking external validation of algorithms. CONCLUSIONS Current evidence suggests that Natural Language Processing algorithms can identify ageing syndromes in electronic health records. However, algorithms require testing in rigorously designed diagnostic accuracy studies with appropriate metrics reported.
Collapse
Affiliation(s)
- Mo Osman
- AGE Research Group, Translational and Clinical Research Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, UK
- NIHR Newcastle Biomedical Research Centre, Newcastle upon Tyne NHS Foundation Trust, Cumbria Northumberland Tyne and Wear NHS Foundation Trust and Newcastle University, Newcastle upon Tyne, UK
| | - Rachel Cooper
- AGE Research Group, Translational and Clinical Research Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, UK
- NIHR Newcastle Biomedical Research Centre, Newcastle upon Tyne NHS Foundation Trust, Cumbria Northumberland Tyne and Wear NHS Foundation Trust and Newcastle University, Newcastle upon Tyne, UK
| | - Avan A Sayer
- AGE Research Group, Translational and Clinical Research Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, UK
- NIHR Newcastle Biomedical Research Centre, Newcastle upon Tyne NHS Foundation Trust, Cumbria Northumberland Tyne and Wear NHS Foundation Trust and Newcastle University, Newcastle upon Tyne, UK
| | - Miles D Witham
- AGE Research Group, Translational and Clinical Research Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, UK
- NIHR Newcastle Biomedical Research Centre, Newcastle upon Tyne NHS Foundation Trust, Cumbria Northumberland Tyne and Wear NHS Foundation Trust and Newcastle University, Newcastle upon Tyne, UK
| |
Collapse
|
3
|
Pillai M, Blumke TL, Studnia J, Wang Y, Veigulis ZP, Ware AD, Hoover PJ, Carroll IR, Humphreys K, Osborne TF, Asch SM, Hernandez-Boussard T, Curtin CM. Improving postsurgical fall detection for older Americans using LLM-driven analysis of clinical narratives. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.06.25.24309480. [PMID: 38978655 PMCID: PMC11230313 DOI: 10.1101/2024.06.25.24309480] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/10/2024]
Abstract
Postsurgical falls have significant patient and societal implications but remain challenging to identify and track. Detecting postsurgical falls is crucial to improve patient care for older adults and reduce healthcare costs. Large language models (LLMs) offer a promising solution for reliable and automated fall detection using unstructured data in clinical notes. We tested several LLM prompting approaches to postsurgical fall detection in two different healthcare systems with three open-source LLMs. The Mixtral-8×7B zero-shot had the best performance at Stanford Health Care (PPV = 0.81, recall = 0.67) and the Veterans Health Administration (PPV = 0.93, recall = 0.94). These results demonstrate that LLMs can detect falls with little to no guidance and lay groundwork for applications of LLMs in fall prediction and prevention across many different settings.
Collapse
Affiliation(s)
- Malvika Pillai
- Veterans Affairs Palo Alto Health Care System, Palo Alto, California, USA
- Department of Medicine (Biomedical Informatics), Stanford University, Stanford, CA, USA
| | - Terri L Blumke
- National Center for Collaborative Healthcare Innovation, Palo Alto VA Healthcare System, Palo Alto, CA, USA
| | - Joachim Studnia
- Institute for Computational and Mathematical Engineering, Stanford University, Stanford, CA, USA
| | - Yuqing Wang
- Department of Medicine (Biomedical Informatics), Stanford University, Stanford, CA, USA
| | | | - Anna D Ware
- National Center for Collaborative Healthcare Innovation, Palo Alto VA Healthcare System, Palo Alto, CA, USA
| | - Peter J Hoover
- National Center for Collaborative Healthcare Innovation, Palo Alto VA Healthcare System, Palo Alto, CA, USA
| | - Ian R Carroll
- Department of Anesthesiology, Stanford University School of Medicine, Stanford, CA, USA
| | - Keith Humphreys
- Department of Psychiatry and the Behavioral Sciences, Stanford University, Stanford, CA USA
| | - Thomas F Osborne
- National Center for Collaborative Healthcare Innovation, Palo Alto VA Healthcare System, Palo Alto, CA, USA
- Department of Radiology, Stanford University School of Medicine, Stanford, CA, USA
| | - Steven M Asch
- Veterans Affairs Palo Alto Health Care System, Palo Alto, California, USA
- Division of Primary Care and Population Health, Stanford University School of Medicine, Stanford, CA, USA
| | | | - Catherine M Curtin
- Veterans Affairs Palo Alto Health Care System, Palo Alto, California, USA
- Division of Plastic and Reconstructive Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA
| |
Collapse
|
4
|
Fu S, Wang L, He H, Wen A, Zong N, Kumari A, Liu F, Zhou S, Zhang R, Li C, Wang Y, St Sauver J, Liu H, Sohn S. A taxonomy for advancing systematic error analysis in multi-site electronic health record-based clinical concept extraction. J Am Med Inform Assoc 2024; 31:1493-1502. [PMID: 38742455 PMCID: PMC11187420 DOI: 10.1093/jamia/ocae101] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2024] [Revised: 03/26/2024] [Accepted: 04/19/2024] [Indexed: 05/16/2024] Open
Abstract
BACKGROUND Error analysis plays a crucial role in clinical concept extraction, a fundamental subtask within clinical natural language processing (NLP). The process typically involves a manual review of error types, such as contextual and linguistic factors contributing to their occurrence, and the identification of underlying causes to refine the NLP model and improve its performance. Conducting error analysis can be complex, requiring a combination of NLP expertise and domain-specific knowledge. Due to the high heterogeneity of electronic health record (EHR) settings across different institutions, challenges may arise when attempting to standardize and reproduce the error analysis process. OBJECTIVES This study aims to facilitate a collaborative effort to establish common definitions and taxonomies for capturing diverse error types, fostering community consensus on error analysis for clinical concept extraction tasks. MATERIALS AND METHODS We iteratively developed and evaluated an error taxonomy based on existing literature, standards, real-world data, multisite case evaluations, and community feedback. The finalized taxonomy was released in both .dtd and .owl formats at the Open Health Natural Language Processing Consortium. The taxonomy is compatible with several different open-source annotation tools, including MAE, Brat, and MedTator. RESULTS The resulting error taxonomy comprises 43 distinct error classes, organized into 6 error dimensions and 4 properties, including model type (symbolic and statistical machine learning), evaluation subject (model and human), evaluation level (patient, document, sentence, and concept), and annotation examples. Internal and external evaluations revealed strong variations in error types across methodological approaches, tasks, and EHR settings. Key points emerged from community feedback, including the need to enhancing clarity, generalizability, and usability of the taxonomy, along with dissemination strategies. CONCLUSION The proposed taxonomy can facilitate the acceleration and standardization of the error analysis process in multi-site settings, thus improving the provenance, interpretability, and portability of NLP models. Future researchers could explore the potential direction of developing automated or semi-automated methods to assist in the classification and standardization of error analysis.
Collapse
Affiliation(s)
- Sunyang Fu
- Department of AI and Informatics, Mayo Clinic, Rochester, MN 55902, United States
- Center for Translational AI Excellence and Applications in Medicine, University of Texas Health Science Center at Houston, Houston, TX 77030, United States
| | - Liwei Wang
- Department of AI and Informatics, Mayo Clinic, Rochester, MN 55902, United States
- Center for Translational AI Excellence and Applications in Medicine, University of Texas Health Science Center at Houston, Houston, TX 77030, United States
| | - Huan He
- Department of Biomedical Informatics & Data Science, Yale University, New Haven, CT 06520, United States
| | - Andrew Wen
- Center for Translational AI Excellence and Applications in Medicine, University of Texas Health Science Center at Houston, Houston, TX 77030, United States
| | - Nansu Zong
- Department of AI and Informatics, Mayo Clinic, Rochester, MN 55902, United States
| | - Anamika Kumari
- Department of Population and Quantitative Health Sciences, University of Massachusetts Chan Medical School, Boston, MA 01655, United States
| | - Feifan Liu
- Department of Population and Quantitative Health Sciences, University of Massachusetts Chan Medical School, Boston, MA 01655, United States
| | - Sicheng Zhou
- Division of Computational Health Sciences, University of Minnesota, Minneapolis, MN 55455, United States
| | - Rui Zhang
- Division of Computational Health Sciences, University of Minnesota, Minneapolis, MN 55455, United States
| | - Chenyu Li
- Department of Health Information Management, University of Pittsburgh, Pittsburgh, PA 15260, United States
| | - Yanshan Wang
- Department of Health Information Management, University of Pittsburgh, Pittsburgh, PA 15260, United States
| | - Jennifer St Sauver
- Department of Quantitative Health Sciences, Mayo Clinic, Rochester, MN 55902, United States
| | - Hongfang Liu
- Department of AI and Informatics, Mayo Clinic, Rochester, MN 55902, United States
- Center for Translational AI Excellence and Applications in Medicine, University of Texas Health Science Center at Houston, Houston, TX 77030, United States
| | - Sunghwan Sohn
- Department of AI and Informatics, Mayo Clinic, Rochester, MN 55902, United States
| |
Collapse
|
5
|
Bonnechère B. Unlocking the Black Box? A Comprehensive Exploration of Large Language Models in Rehabilitation. Am J Phys Med Rehabil 2024; 103:532-537. [PMID: 38261757 DOI: 10.1097/phm.0000000000002440] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/25/2024]
Abstract
ABSTRACT Rehabilitation is a vital component of health care, aiming to restore function and improve the well-being of individuals with disabilities or injuries. Nevertheless, the rehabilitation process is often likened to a " black box ," with complexities that pose challenges for comprehensive analysis and optimization. The emergence of large language models offers promising solutions to better understand this " black box ." Large language models excel at comprehending and generating human-like text, making them valuable in the healthcare sector. In rehabilitation, healthcare professionals must integrate a wide range of data to create effective treatment plans, akin to selecting the best ingredients for the " black box. " Large language models enhance data integration, communication, assessment, and prediction.This article delves into the ground-breaking use of large language models as a tool to further understand the rehabilitation process. Large language models address current rehabilitation issues, including data bias, contextual comprehension, and ethical concerns. Collaboration with healthcare experts and rigorous validation is crucial when deploying large language models. Integrating large language models into rehabilitation yields insights into this intricate process, enhancing data-driven decision making, refining clinical practices, and predicting rehabilitation outcomes. Although challenges persist, large language models represent a significant stride in rehabilitation, underscoring the importance of ethical use and collaboration.
Collapse
Affiliation(s)
- Bruno Bonnechère
- From the REVAL Rehabilitation Research Center, Faculty of Rehabilitation Sciences, Hasselt University, Diepenbeek, Belgium; Technology-Supported and Data-Driven Rehabilitation, Data Sciences Institute, Hasselt University, Diepenbeek, Belgium; and Department of PXL-Healthcare, PXL University of Applied Sciences and Arts, Hasselt, Belgium
| |
Collapse
|
6
|
Calley DQ, Fu S, Hamilton MD, Kalla AW, Lee CK, Rasmussen VA, Hollman JH, Liu H. Assessment of Gender Differences in Letters of Recommendation for Physical Therapy Residency Applications. JOURNAL, PHYSICAL THERAPY EDUCATION 2024:00001416-990000000-00105. [PMID: 38640081 DOI: 10.1097/jte.0000000000000337] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/16/2023] [Accepted: 12/27/2023] [Indexed: 04/21/2024]
Abstract
INTRODUCTION Letters of recommendation (LOR) are an integral component of physical therapy residency applications. Identifying the influence of applicant and writer gender in LOR will help identify whether potential implicit gender bias exists in physical therapy residency application processes. REVIEW OF LITERATURE Several medical and surgical residency education programs have reported positive, neutral, or negative LOR female gender bias among applicants and writers. Little research exists on gender differences in LOR to physical therapy education programs or physical therapy residency programs. SUBJECTS Seven hundred sixty-eight LOR were analyzed from 256 applications to 3 physical therapy residency programs (neurologic, orthopaedic, sports) at one institution from 2014 to 2020. METHODS Thematic categories were developed to identify themes in a sample of LOR. Associations between writer and applicant gender were analyzed using summary statistics, word counts, thematic and psycholinguistic extraction, and rule-based and deep learning Natural Language Processing . RESULTS No significant difference in LOR word counts were found based on writer or applicant gender. Increased word counts were seen in sports residency LOR compared with the orthopaedic residency. Thematic analysis showed LOR gender differences with male applicants receiving more positive generalized recommendations and female applicants receiving more comments regarding interpersonal relationship skills. No thematic or psycholinguistic gender differences were seen by LOR writer. Male applicants were 1.9 times more likely to select all male LOR writers, whereas female applicants were 2.1 times more likely to choose all female LOR writers. DISCUSSION AND CONCLUSION Gender differences in LORs for physical therapy residencies were found using a comprehensive Natural Language Processing approach that identified both a positive recommendation male applicant gender bias and a positive interpersonal relationship skill female applicant gender bias. Applicants were not harmed nor helped by selecting LOR writers of the opposite gender. Admissions committees and LOR writers should be mindful of potential implicit gender biases in LOR submitted to physical therapy residency programs.
Collapse
Affiliation(s)
- Darren Q Calley
- Darren Q. Calley is the residency director for the Mayo Clinic Physical Therapy Neurologic, Orthopaedic, & Sports Residency Programs in the Mayo Clinic School of Health Sciences, and is an assistant professor of Physical Therapy at the Mayo Clinic College of Medicine and Science, and is a faculty member in the Physical Therapy Doctoral Education Program at the Mayo Clinic, Siebens 7-57, 200 First Street SW, Rochester, MN, 55905 . Please address all correspondence to Darren Calley
- Sunyang Fu is an assistant professor of Biomedical Informatics, and is an associate director of Team Science at the Center for Translational AI Excellence and Applications in Medicine (TEAM-AI) at the University of Texas Health Science Center
- Marissa D. Hamilton is a physical therapist at Mayo Clinic
- Austin W. Kalla is a physical therapist graduate at Mayo Clinic
- Christopher K. Lee is a physical therapist at Mayo Clinic
- Veronica A. Rasmussen is a physical therapist at the Hennepin County Medical Center
- John H. Hollman is the program director of the Physical Therapy Doctoral Education Program, and is an associate dean for Academic Affairs in the Mayo Clinic School of Health Sciences, and is a professor of Physical Therapy in the Mayo Clinic College of Medicine and Science
- Hongfang Liu is a professor in the Biomedical Informatics, and is a director of Translational AI Excellence and Applications in Medicine (TEAM-AI), University of Texas Health Science Center
| | - Sunyang Fu
- Darren Q. Calley is the residency director for the Mayo Clinic Physical Therapy Neurologic, Orthopaedic, & Sports Residency Programs in the Mayo Clinic School of Health Sciences, and is an assistant professor of Physical Therapy at the Mayo Clinic College of Medicine and Science, and is a faculty member in the Physical Therapy Doctoral Education Program at the Mayo Clinic, Siebens 7-57, 200 First Street SW, Rochester, MN, 55905 . Please address all correspondence to Darren Calley
- Sunyang Fu is an assistant professor of Biomedical Informatics, and is an associate director of Team Science at the Center for Translational AI Excellence and Applications in Medicine (TEAM-AI) at the University of Texas Health Science Center
- Marissa D. Hamilton is a physical therapist at Mayo Clinic
- Austin W. Kalla is a physical therapist graduate at Mayo Clinic
- Christopher K. Lee is a physical therapist at Mayo Clinic
- Veronica A. Rasmussen is a physical therapist at the Hennepin County Medical Center
- John H. Hollman is the program director of the Physical Therapy Doctoral Education Program, and is an associate dean for Academic Affairs in the Mayo Clinic School of Health Sciences, and is a professor of Physical Therapy in the Mayo Clinic College of Medicine and Science
- Hongfang Liu is a professor in the Biomedical Informatics, and is a director of Translational AI Excellence and Applications in Medicine (TEAM-AI), University of Texas Health Science Center
| | - Marissa D Hamilton
- Darren Q. Calley is the residency director for the Mayo Clinic Physical Therapy Neurologic, Orthopaedic, & Sports Residency Programs in the Mayo Clinic School of Health Sciences, and is an assistant professor of Physical Therapy at the Mayo Clinic College of Medicine and Science, and is a faculty member in the Physical Therapy Doctoral Education Program at the Mayo Clinic, Siebens 7-57, 200 First Street SW, Rochester, MN, 55905 . Please address all correspondence to Darren Calley
- Sunyang Fu is an assistant professor of Biomedical Informatics, and is an associate director of Team Science at the Center for Translational AI Excellence and Applications in Medicine (TEAM-AI) at the University of Texas Health Science Center
- Marissa D. Hamilton is a physical therapist at Mayo Clinic
- Austin W. Kalla is a physical therapist graduate at Mayo Clinic
- Christopher K. Lee is a physical therapist at Mayo Clinic
- Veronica A. Rasmussen is a physical therapist at the Hennepin County Medical Center
- John H. Hollman is the program director of the Physical Therapy Doctoral Education Program, and is an associate dean for Academic Affairs in the Mayo Clinic School of Health Sciences, and is a professor of Physical Therapy in the Mayo Clinic College of Medicine and Science
- Hongfang Liu is a professor in the Biomedical Informatics, and is a director of Translational AI Excellence and Applications in Medicine (TEAM-AI), University of Texas Health Science Center
| | - Austin W Kalla
- Darren Q. Calley is the residency director for the Mayo Clinic Physical Therapy Neurologic, Orthopaedic, & Sports Residency Programs in the Mayo Clinic School of Health Sciences, and is an assistant professor of Physical Therapy at the Mayo Clinic College of Medicine and Science, and is a faculty member in the Physical Therapy Doctoral Education Program at the Mayo Clinic, Siebens 7-57, 200 First Street SW, Rochester, MN, 55905 . Please address all correspondence to Darren Calley
- Sunyang Fu is an assistant professor of Biomedical Informatics, and is an associate director of Team Science at the Center for Translational AI Excellence and Applications in Medicine (TEAM-AI) at the University of Texas Health Science Center
- Marissa D. Hamilton is a physical therapist at Mayo Clinic
- Austin W. Kalla is a physical therapist graduate at Mayo Clinic
- Christopher K. Lee is a physical therapist at Mayo Clinic
- Veronica A. Rasmussen is a physical therapist at the Hennepin County Medical Center
- John H. Hollman is the program director of the Physical Therapy Doctoral Education Program, and is an associate dean for Academic Affairs in the Mayo Clinic School of Health Sciences, and is a professor of Physical Therapy in the Mayo Clinic College of Medicine and Science
- Hongfang Liu is a professor in the Biomedical Informatics, and is a director of Translational AI Excellence and Applications in Medicine (TEAM-AI), University of Texas Health Science Center
| | - Christopher K Lee
- Darren Q. Calley is the residency director for the Mayo Clinic Physical Therapy Neurologic, Orthopaedic, & Sports Residency Programs in the Mayo Clinic School of Health Sciences, and is an assistant professor of Physical Therapy at the Mayo Clinic College of Medicine and Science, and is a faculty member in the Physical Therapy Doctoral Education Program at the Mayo Clinic, Siebens 7-57, 200 First Street SW, Rochester, MN, 55905 . Please address all correspondence to Darren Calley
- Sunyang Fu is an assistant professor of Biomedical Informatics, and is an associate director of Team Science at the Center for Translational AI Excellence and Applications in Medicine (TEAM-AI) at the University of Texas Health Science Center
- Marissa D. Hamilton is a physical therapist at Mayo Clinic
- Austin W. Kalla is a physical therapist graduate at Mayo Clinic
- Christopher K. Lee is a physical therapist at Mayo Clinic
- Veronica A. Rasmussen is a physical therapist at the Hennepin County Medical Center
- John H. Hollman is the program director of the Physical Therapy Doctoral Education Program, and is an associate dean for Academic Affairs in the Mayo Clinic School of Health Sciences, and is a professor of Physical Therapy in the Mayo Clinic College of Medicine and Science
- Hongfang Liu is a professor in the Biomedical Informatics, and is a director of Translational AI Excellence and Applications in Medicine (TEAM-AI), University of Texas Health Science Center
| | - Veronica A Rasmussen
- Darren Q. Calley is the residency director for the Mayo Clinic Physical Therapy Neurologic, Orthopaedic, & Sports Residency Programs in the Mayo Clinic School of Health Sciences, and is an assistant professor of Physical Therapy at the Mayo Clinic College of Medicine and Science, and is a faculty member in the Physical Therapy Doctoral Education Program at the Mayo Clinic, Siebens 7-57, 200 First Street SW, Rochester, MN, 55905 . Please address all correspondence to Darren Calley
- Sunyang Fu is an assistant professor of Biomedical Informatics, and is an associate director of Team Science at the Center for Translational AI Excellence and Applications in Medicine (TEAM-AI) at the University of Texas Health Science Center
- Marissa D. Hamilton is a physical therapist at Mayo Clinic
- Austin W. Kalla is a physical therapist graduate at Mayo Clinic
- Christopher K. Lee is a physical therapist at Mayo Clinic
- Veronica A. Rasmussen is a physical therapist at the Hennepin County Medical Center
- John H. Hollman is the program director of the Physical Therapy Doctoral Education Program, and is an associate dean for Academic Affairs in the Mayo Clinic School of Health Sciences, and is a professor of Physical Therapy in the Mayo Clinic College of Medicine and Science
- Hongfang Liu is a professor in the Biomedical Informatics, and is a director of Translational AI Excellence and Applications in Medicine (TEAM-AI), University of Texas Health Science Center
| | - John H Hollman
- Darren Q. Calley is the residency director for the Mayo Clinic Physical Therapy Neurologic, Orthopaedic, & Sports Residency Programs in the Mayo Clinic School of Health Sciences, and is an assistant professor of Physical Therapy at the Mayo Clinic College of Medicine and Science, and is a faculty member in the Physical Therapy Doctoral Education Program at the Mayo Clinic, Siebens 7-57, 200 First Street SW, Rochester, MN, 55905 . Please address all correspondence to Darren Calley
- Sunyang Fu is an assistant professor of Biomedical Informatics, and is an associate director of Team Science at the Center for Translational AI Excellence and Applications in Medicine (TEAM-AI) at the University of Texas Health Science Center
- Marissa D. Hamilton is a physical therapist at Mayo Clinic
- Austin W. Kalla is a physical therapist graduate at Mayo Clinic
- Christopher K. Lee is a physical therapist at Mayo Clinic
- Veronica A. Rasmussen is a physical therapist at the Hennepin County Medical Center
- John H. Hollman is the program director of the Physical Therapy Doctoral Education Program, and is an associate dean for Academic Affairs in the Mayo Clinic School of Health Sciences, and is a professor of Physical Therapy in the Mayo Clinic College of Medicine and Science
- Hongfang Liu is a professor in the Biomedical Informatics, and is a director of Translational AI Excellence and Applications in Medicine (TEAM-AI), University of Texas Health Science Center
| | - Hongfang Liu
- Darren Q. Calley is the residency director for the Mayo Clinic Physical Therapy Neurologic, Orthopaedic, & Sports Residency Programs in the Mayo Clinic School of Health Sciences, and is an assistant professor of Physical Therapy at the Mayo Clinic College of Medicine and Science, and is a faculty member in the Physical Therapy Doctoral Education Program at the Mayo Clinic, Siebens 7-57, 200 First Street SW, Rochester, MN, 55905 . Please address all correspondence to Darren Calley
- Sunyang Fu is an assistant professor of Biomedical Informatics, and is an associate director of Team Science at the Center for Translational AI Excellence and Applications in Medicine (TEAM-AI) at the University of Texas Health Science Center
- Marissa D. Hamilton is a physical therapist at Mayo Clinic
- Austin W. Kalla is a physical therapist graduate at Mayo Clinic
- Christopher K. Lee is a physical therapist at Mayo Clinic
- Veronica A. Rasmussen is a physical therapist at the Hennepin County Medical Center
- John H. Hollman is the program director of the Physical Therapy Doctoral Education Program, and is an associate dean for Academic Affairs in the Mayo Clinic School of Health Sciences, and is a professor of Physical Therapy in the Mayo Clinic College of Medicine and Science
- Hongfang Liu is a professor in the Biomedical Informatics, and is a director of Translational AI Excellence and Applications in Medicine (TEAM-AI), University of Texas Health Science Center
| |
Collapse
|
7
|
Fu S, Jia H, Vassilaki M, Keloth VK, Dang Y, Zhou Y, Garg M, Petersen RC, St Sauver J, Moon S, Wang L, Wen A, Li F, Xu H, Tao C, Fan J, Liu H, Sohn S. FedFSA: Hybrid and federated framework for functional status ascertainment across institutions. J Biomed Inform 2024; 152:104623. [PMID: 38458578 PMCID: PMC11005095 DOI: 10.1016/j.jbi.2024.104623] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Revised: 01/12/2024] [Accepted: 03/04/2024] [Indexed: 03/10/2024]
Abstract
INTRODUCTION Patients' functional status assesses their independence in performing activities of daily living, including basic ADLs (bADL), and more complex instrumental activities (iADL). Existing studies have discovered that patients' functional status is a strong predictor of health outcomes, particularly in older adults. Depite their usefulness, much of the functional status information is stored in electronic health records (EHRs) in either semi-structured or free text formats. This indicates the pressing need to leverage computational approaches such as natural language processing (NLP) to accelerate the curation of functional status information. In this study, we introduced FedFSA, a hybrid and federated NLP framework designed to extract functional status information from EHRs across multiple healthcare institutions. METHODS FedFSA consists of four major components: 1) individual sites (clients) with their private local data, 2) a rule-based information extraction (IE) framework for ADL extraction, 3) a BERT model for functional status impairment classification, and 4) a concept normalizer. The framework was implemented using the OHNLP Backbone for rule-based IE and open-source Flower and PyTorch library for federated BERT components. For gold standard data generation, we carried out corpus annotation to identify functional status-related expressions based on ICF definitions. Four healthcare institutions were included in the study. To assess FedFSA, we evaluated the performance of category- and institution-specific ADL extraction across different experimental designs. RESULTS ADL extraction performance ranges from an F1-score of 0.907 to 0.986 for bADL and 0.825 to 0.951 for iADL across the four healthcare sites. The performance for ADL extraction with impairment ranges from an F1-score of 0.722 to 0.954 for bADL and 0.674 to 0.813 for iADL across four healthcare sites. For category-specific ADL extraction, laundry and transferring yielded relatively high performance, while dressing, medication, bathing, and continence achieved moderate-high performance. Conversely, food preparation and toileting showed low performance. CONCLUSION NLP performance varied across ADL categories and healthcare sites. Federated learning using a FedFSA framework performed higher than non-federated learning for impaired ADL extraction at all healthcare sites. Our study demonstrated the potential of the federated learning framework in functional status extraction and impairment classification in EHRs, exemplifying the importance of a large-scale, multi-institutional collaborative development effort.
Collapse
Affiliation(s)
- Sunyang Fu
- Mayo Clinic, Rochester, MN, United States; University of Texas Health Science Center, Houston, TX, United States.
| | - Heling Jia
- Mayo Clinic, Rochester, MN, United States.
| | | | | | - Yifang Dang
- University of Texas Health Science Center, Houston, TX, United States.
| | - Yujia Zhou
- University of Texas Health Science Center, Houston, TX, United States.
| | | | | | | | | | - Liwei Wang
- Mayo Clinic, Rochester, MN, United States.
| | - Andrew Wen
- University of Texas Health Science Center, Houston, TX, United States.
| | - Fang Li
- University of Texas Health Science Center, Houston, TX, United States.
| | - Hua Xu
- Yale University, New Haven, CT, United States.
| | - Cui Tao
- University of Texas Health Science Center, Houston, TX, United States.
| | | | - Hongfang Liu
- Mayo Clinic, Rochester, MN, United States; University of Texas Health Science Center, Houston, TX, United States.
| | | |
Collapse
|
8
|
Cheligeer C, Wu G, Lee S, Pan J, Southern DA, Martin EA, Sapiro N, Eastwood CA, Quan H, Xu Y. BERT-Based Neural Network for Inpatient Fall Detection From Electronic Medical Records: Retrospective Cohort Study. JMIR Med Inform 2024; 12:e48995. [PMID: 38289643 PMCID: PMC10865188 DOI: 10.2196/48995] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2023] [Revised: 07/24/2023] [Accepted: 12/23/2023] [Indexed: 02/01/2024] Open
Abstract
BACKGROUND Inpatient falls are a substantial concern for health care providers and are associated with negative outcomes for patients. Automated detection of falls using machine learning (ML) algorithms may aid in improving patient safety and reducing the occurrence of falls. OBJECTIVE This study aims to develop and evaluate an ML algorithm for inpatient fall detection using multidisciplinary progress record notes and a pretrained Bidirectional Encoder Representation from Transformers (BERT) language model. METHODS A cohort of 4323 adult patients admitted to 3 acute care hospitals in Calgary, Alberta, Canada from 2016 to 2021 were randomly sampled. Trained reviewers determined falls from patient charts, which were linked to electronic medical records and administrative data. The BERT-based language model was pretrained on clinical notes, and a fall detection algorithm was developed based on a neural network binary classification architecture. RESULTS To address various use scenarios, we developed 3 different Alberta hospital notes-specific BERT models: a high sensitivity model (sensitivity 97.7, IQR 87.7-99.9), a high positive predictive value model (positive predictive value 85.7, IQR 57.2-98.2), and the high F1-score model (F1=64.4). Our proposed method outperformed 3 classical ML algorithms and an International Classification of Diseases code-based algorithm for fall detection, showing its potential for improved performance in diverse clinical settings. CONCLUSIONS The developed algorithm provides an automated and accurate method for inpatient fall detection using multidisciplinary progress record notes and a pretrained BERT language model. This method could be implemented in clinical practice to improve patient safety and reduce the occurrence of falls in hospitals.
Collapse
Affiliation(s)
- Cheligeer Cheligeer
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Provincial Research Data Services, Alberta Health Services, Calgary, AB, Canada
| | - Guosong Wu
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Seungwon Lee
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Provincial Research Data Services, Alberta Health Services, Calgary, AB, Canada
| | - Jie Pan
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Danielle A Southern
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Elliot A Martin
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Provincial Research Data Services, Alberta Health Services, Calgary, AB, Canada
| | - Natalie Sapiro
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Cathy A Eastwood
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Hude Quan
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Yuan Xu
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Department of Oncology, University of Calgary, Calgary, AB, Canada
- Department of Surgery, University of Calgary, Calgary, AB, Canada
| |
Collapse
|
9
|
Trinh VQN, Zhang S, Kovoor J, Gupta A, Chan WO, Gilbert T, Bacchi S. The use of natural language processing in detecting and predicting falls within the healthcare setting: a systematic review. Int J Qual Health Care 2023; 35:mzad077. [PMID: 37758209 PMCID: PMC10585351 DOI: 10.1093/intqhc/mzad077] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2023] [Revised: 08/30/2023] [Accepted: 09/23/2023] [Indexed: 10/03/2023] Open
Abstract
Falls are a common problem associated with significant morbidity, mortality, and economic costs. Current fall prevention policies in local healthcare settings are often guided by information provided by fall risk assessment tools, incident reporting, and coding data. This review was conducted with the aim of identifying studies which utilized natural language processing (NLP) for the automated detection and prediction of falls in the healthcare setting. The databases Ovid Medline, Ovid Embase, Ovid Emcare, PubMed, CINAHL, IEEE Xplore, and Ei Compendex were searched from 2012 until April 2023. Retrospective derivation, validation, and implementation studies wherein patients experienced falls within a healthcare setting were identified for inclusion. The initial search yielded 2611 publications for title and abstract screening. Full-text screening was conducted on 105 publications, resulting in 26 unique studies that underwent qualitative analyses. Studies applied NLP towards falls risk factor identification, known falls detection, future falls prediction, and falls severity stratification with reasonable success. The NLP pipeline was reviewed in detail between studies and models utilizing rule-based, machine learning (ML), deep learning (DL), and hybrid approaches were examined. With a growing literature surrounding falls prediction in both inpatient and outpatient environments, the absence of studies examining the impact of these models on patient and system outcomes highlights the need for further implementation studies. Through an exploration of the application of NLP techniques, it may be possible to develop models with higher performance in automated falls prediction and detection.
Collapse
Affiliation(s)
| | - Steven Zhang
- University of Adelaide, Adelaide, South Australia 5005, Australia
| | - Joshua Kovoor
- University of Adelaide, Adelaide, South Australia 5005, Australia
- Queen Elizabeth Hospital, Adelaide, South Australia 5011, Australia
| | - Aashray Gupta
- University of Adelaide, Adelaide, South Australia 5005, Australia
- Gold Coast University Hospital, Gold Coast, Queensland 4215, Australia
| | - Weng Onn Chan
- Queen Elizabeth Hospital, Adelaide, South Australia 5011, Australia
- Discipline of Ophthalmology and Visual Sciences, The University of Adelaide, Adelaide, South Australia 5005, Australia
- Royal Adelaide Hospital, Adelaide, South Australia 5000, Australia
| | - Toby Gilbert
- University of Adelaide, Adelaide, South Australia 5005, Australia
- Northern Adelaide Local Health Network, Adelaide, South Australia 5112, Australia
| | - Stephen Bacchi
- Royal Adelaide Hospital, Adelaide, South Australia 5000, Australia
- Flinders University, Adelaide, South Australia 5042, Australia
| |
Collapse
|
10
|
Chaieb S, Ben Mrad A, Hnich B. From Personal Observations to Recommendation of Tailored Interventions based on Causal Reasoning: a case study of Falls Prevention in Elderly Patients. Int J Med Inform 2022; 163:104765. [PMID: 35461148 DOI: 10.1016/j.ijmedinf.2022.104765] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2022] [Revised: 03/31/2022] [Accepted: 04/12/2022] [Indexed: 10/18/2022]
Abstract
OBJECTIVE While the challenge of estimating the efficacy of therapies using observational data has received a lot of attention, little work has been done on estimating the treatment effect from interventions. In this paper, we tackle this problem by proposing an early guidance system based on a causal Bayesian network (CBN) for recommending personalized interventions. We are interested in the elderly fall prevention context. The objective is to develop a practical tool to help doctors estimate the effects of each intervention (or compound interventions) on a given patient and then choose the one that best fits each patient's health situation to reduce the risk of falling. METHODS On a real-world elderly information base, we undertake an empirical investigation for the proposed approach, which is based on a 44-node CBN. Then, we describe what is possible to achieve using state-of-the-art machine learning methods, namely Support Virtual Machine (SVM), Decision Tree (DT), and Bayesian Network (BN), and how well these methods can be used in recommending personalized interventions compared to the proposed approach. RESULTS 1174 elderly patients from Lille University Hospital, between January 2005 and December 2018 are included. The results reveal that none of the classifiers is significantly superior to the others, even if BN delivers somewhat better results (41.6%) and DT most often slightly lower performance (31.2%). Results also show that none of these three classifiers performs comparable to the proposed system (89.7%). The interventions recommended by the system are in agreement with the expert's judgment to a satisfactory level. The reaction of the physicians to the proposed system in its first trial version was very favorable. CONCLUSION The study showed the efficacy and utility of the causality-based strategy in recommending tailored interventions to prevent elderly falls compared to automated learning methods that had failed to infer a solid interventional paradigm for precision medicine.
Collapse
Affiliation(s)
- Salma Chaieb
- University of Sousse, ISITCom, 4011 Sousse, Tunisia; University of Sfax, CES Lab, 3038 Sfax, Tunisia.
| | - Ali Ben Mrad
- University of Sfax, ISAAS, 1013 Sfax, Tunisia; University of Sfax, CES Lab, 3038 Sfax, Tunisia
| | - Brahim Hnich
- University of Monastir, FSM, 5000 Monastir, Tunisia; University of Sfax, CES Lab, 3038 Sfax, Tunisia.
| |
Collapse
|