1
|
Sivarajkumar S, Tam TYC, Mohammad HA, Viggiano S, Oniani D, Visweswaran S, Wang Y. Extraction of sleep information from clinical notes of Alzheimer's disease patients using natural language processing. J Am Med Inform Assoc 2024:ocae177. [PMID: 39001795 DOI: 10.1093/jamia/ocae177] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/29/2024] [Revised: 06/19/2024] [Accepted: 07/01/2024] [Indexed: 07/15/2024] Open
Abstract
OBJECTIVES Alzheimer's disease (AD) is the most common form of dementia in the United States. Sleep is one of the lifestyle-related factors that has been shown critical for optimal cognitive function in old age. However, there is a lack of research studying the association between sleep and AD incidence. A major bottleneck for conducting such research is that the traditional way to acquire sleep information is time-consuming, inefficient, non-scalable, and limited to patients' subjective experience. We aim to automate the extraction of specific sleep-related patterns, such as snoring, napping, poor sleep quality, daytime sleepiness, night wakings, other sleep problems, and sleep duration, from clinical notes of AD patients. These sleep patterns are hypothesized to play a role in the incidence of AD, providing insight into the relationship between sleep and AD onset and progression. MATERIALS AND METHODS A gold standard dataset is created from manual annotation of 570 randomly sampled clinical note documents from the adSLEEP, a corpus of 192 000 de-identified clinical notes of 7266 AD patients retrieved from the University of Pittsburgh Medical Center (UPMC). We developed a rule-based natural language processing (NLP) algorithm, machine learning models, and large language model (LLM)-based NLP algorithms to automate the extraction of sleep-related concepts, including snoring, napping, sleep problem, bad sleep quality, daytime sleepiness, night wakings, and sleep duration, from the gold standard dataset. RESULTS The annotated dataset of 482 patients comprised a predominantly White (89.2%), older adult population with an average age of 84.7 years, where females represented 64.1%, and a vast majority were non-Hispanic or Latino (94.6%). Rule-based NLP algorithm achieved the best performance of F1 across all sleep-related concepts. In terms of positive predictive value (PPV), the rule-based NLP algorithm achieved the highest PPV scores for daytime sleepiness (1.00) and sleep duration (1.00), while the machine learning models had the highest PPV for napping (0.95) and bad sleep quality (0.86), and LLAMA2 with finetuning had the highest PPV for night wakings (0.93) and sleep problem (0.89). DISCUSSION Although sleep information is infrequently documented in the clinical notes, the proposed rule-based NLP algorithm and LLM-based NLP algorithms still achieved promising results. In comparison, the machine learning-based approaches did not achieve good results, which is due to the small size of sleep information in the training data. CONCLUSION The results show that the rule-based NLP algorithm consistently achieved the best performance for all sleep concepts. This study focused on the clinical notes of patients with AD but could be extended to general sleep information extraction for other diseases.
Collapse
Affiliation(s)
- Sonish Sivarajkumar
- Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA 15260, United States
| | - Thomas Yu Chow Tam
- Department of Health Information Management, University of Pittsburgh, Pittsburgh, PA 15260, United States
| | - Haneef Ahamed Mohammad
- Department of Health Information Management, University of Pittsburgh, Pittsburgh, PA 15260, United States
| | - Samuel Viggiano
- Department of Health Information Management, University of Pittsburgh, Pittsburgh, PA 15260, United States
| | - David Oniani
- Department of Health Information Management, University of Pittsburgh, Pittsburgh, PA 15260, United States
| | - Shyam Visweswaran
- Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA 15260, United States
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA 15260, United States
| | - Yanshan Wang
- Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA 15260, United States
- Department of Health Information Management, University of Pittsburgh, Pittsburgh, PA 15260, United States
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA 15260, United States
- Clinical and Translational Science Institute, University of Pittsburgh, Pittsburgh, PA 15260, United States
| |
Collapse
|
2
|
Hsu E, Roberts K. Leveraging Large Language Models for Knowledge-free Weak Supervision in Clinical Natural Language Processing. RESEARCH SQUARE 2024:rs.3.rs-4559971. [PMID: 38978609 PMCID: PMC11230489 DOI: 10.21203/rs.3.rs-4559971/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/10/2024]
Abstract
The performance of deep learning-based natural language processing systems is based on large amounts of labeled training data which, in the clinical domain, are not easily available or affordable. Weak supervision and in-context learning offer partial solutions to this issue, particularly using large language models (LLMs), but their performance still trails traditional supervised methods with moderate amounts of gold-standard data. In particular, inferencing with LLMs is computationally heavy. We propose an approach leveraging fine-tuning LLMs and weak supervision with virtually no domain knowledge that still achieves consistently dominant performance. Using a prompt-based approach, the LLM is used to generate weakly-labeled data for training a downstream BERT model. The weakly supervised model is then further fine-tuned on small amounts of gold standard data. We evaluate this approach using Llama2 on three different n2c2 datasets. With no more than 10 gold standard notes, our final BERT models weakly supervised by fine-tuned Llama2-13B consistently outperformed out-of-the-box PubMedBERT by 4.7-47.9% in F1 scores. With only 50 gold standard notes, our models achieved close performance to fully fine-tuned systems.
Collapse
Affiliation(s)
- Enshuo Hsu
- University of Texas Health Science Center at Houston
| | - Kirk Roberts
- University of Texas Health Science Center at Houston
| |
Collapse
|
3
|
Yang C, Huebner ES, Tian L. Prediction of suicidal ideation among preadolescent children with machine learning models: A longitudinal study. J Affect Disord 2024; 352:403-409. [PMID: 38387673 DOI: 10.1016/j.jad.2024.02.070] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/19/2023] [Revised: 02/15/2024] [Accepted: 02/19/2024] [Indexed: 02/24/2024]
Abstract
BACKGROUND Machine learning (ML) has been widely used to predict suicidal ideation (SI) in adolescents and adults. Nevertheless, studies of accurate and efficient models of SI prediction with preadolescent children are still needed because SI is surprisingly prevalent during the transition into adolescence. This study aimed to explore the potential of ML models to predict SI among preadolescent children. METHODS A total of 4691 Chinese children (54.89 % boys, Mage = 10.92 at baseline) and their parents completed relevant measures at baseline and the children provided 6-month follow-up data for SI. The current study compared four ML models: Random Forest (RF), Decision Tree (DT), Support Vector Machine (SVM), and Multilayer Perceptron (MLP), to predict SI and to identify variables with predictive value based on the best-performing model among Chinese preadolescent children. RESULTS The RF model achieved the highest discriminant performance with an AUC of 0.92, accuracy of 0.93 (balanced accuracy = 0.88). The factors of internalizing problems, externalizing problems, neuroticism, childhood maltreatment, and subjective well-being in school demonstrated the highest values in predicting SI. CONCLUSION The findings of this study suggested that ML models based on the observation and assessment of children's general characteristics and experiences in everyday life can serve as convenient screening and evaluation tools for suicide risk assessment among Chinese preadolescent children. The findings also provide insights for early intervention.
Collapse
Affiliation(s)
- Chi Yang
- Philosophy and Social Science Laboratory of Reading and Development in Children and Adolescents, South China Normal University, Ministry of Education, Guangzhou 510631, People's Republic of China; School of Psychology, South China Normal University, Guangzhou 510631, People's Republic of China
| | - E Scott Huebner
- Department of Psychology, University of South Carolina, Columbia, SC 29208, USA
| | - Lili Tian
- Philosophy and Social Science Laboratory of Reading and Development in Children and Adolescents, South China Normal University, Ministry of Education, Guangzhou 510631, People's Republic of China.
| |
Collapse
|
4
|
Mitra A, Chen K, Liu W, Kessler RC, Yu H. Predicting Suicide Among US Veterans Using Natural Language Processing-enriched Social and Behavioral Determinants of Health. RESEARCH SQUARE 2024:rs.3.rs-4290732. [PMID: 38746180 PMCID: PMC11092830 DOI: 10.21203/rs.3.rs-4290732/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/16/2024]
Abstract
Despite recognizing the critical association between social and behavioral determinants of health (SBDH) and suicide risk, SBDHs from unstructured electronic health record (EHR) notes for suicide predictive modeling remain underutilized. This study investigates the impact of SBDH, identified from both structured and unstructured data utilizing a natural language processing (NLP) system, on suicide prediction within 7, 30, 90, and 180 days of discharge. Using EHR data of 2,987,006 Veterans between October 1, 2009, and September 30, 2015, from the US Veterans Health Administration (VHA), we designed a case-control study that demonstrates that incorporating structured and NLP-extracted SBDH significantly enhances the performance of three architecturally distinct suicide predictive models - elastic-net logistic regression, random forest (RF), and multilayer perceptron. For example, RF achieved notable improvements in suicide prediction within 180 days of discharge, with an increase in the area under the receiver operating characteristic curve from 83.57-84.25% (95% CI = 0.63%-0.98%, p-val < 0.001) and the area under the precision recall curve from 57.38-59.87% (95% CI = 3.86%-4.82%, p-val < 0.001) after integrating NLP-extracted SBDH. These findings underscore the potential of NLP-extracted SBDH in enhancing suicide prediction across various prediction timeframes, offering valuable insights for healthcare practitioners and policymakers.
Collapse
Affiliation(s)
| | | | | | | | - Hong Yu
- University of Massachusetts Amherst
| |
Collapse
|
5
|
Adekkanattu P, Furmanchuk A, Wu Y, Pathak A, Patra BG, Bost S, Morrow D, Wang GHM, Yang Y, Forrest NJ, Luo Y, Walunas TL, Jenny WHLC, Gelad W, Bian J, Bao Y, Weiner M, Oslin D, Pathak J. Detection of Personal and Family History of Suicidal Thoughts and Behaviors using Deep Learning and Natural Language Processing: A Multi-Site Study. RESEARCH SQUARE 2024:rs.3.rs-4014472. [PMID: 38559051 PMCID: PMC10980141 DOI: 10.21203/rs.3.rs-4014472/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/04/2024]
Abstract
Objective Personal and family history of suicidal thoughts and behaviors (PSH and FSH, respectively) are significant risk factors associated with future suicide events. These are often captured in narrative clinical notes in electronic health records (EHRs). Collaboratively, Weill Cornell Medicine (WCM), Northwestern Medicine (NM), and the University of Florida (UF) developed and validated deep learning (DL)-based natural language processing (NLP) tools to detect PSH and FSH from such notes. The tool's performance was further benchmarked against a method relying exclusively on ICD-9/10 diagnosis codes. Materials and Methods We developed DL-based NLP tools utilizing pre-trained transformer models Bio_ClinicalBERT and GatorTron, and compared them with expert-informed, rule-based methods. The tools were initially developed and validated using manually annotated clinical notes at WCM. Their portability and performance were further evaluated using clinical notes at NM and UF. Results The DL tools outperformed the rule-based NLP tool in identifying PSH and FHS. For detecting PSH, the rule-based system obtained an F1-score of 0.75 ± 0.07, while the Bio_ClinicalBERT and GatorTron DL tools scored 0.83 ± 0.09 and 0.84 ± 0.07, respectively. For detecting FSH, the rule-based NLP tool's F1-score was 0.69 ± 0.11, compared to 0.89 ± 0.10 for Bio_ClinicalBERT and 0.92 ± 0.07 for GatorTron. For the gold standard corpora across the three sites, only 2.2% (WCM), 9.3% (NM), and 7.8% (UF) of patients reported to have an ICD-9/10 diagnosis code for suicidal thoughts and behaviors prior to the clinical notes report date. The best performing GatorTron DL tool identified 93.0% (WCM), 80.4% (NM), and 89.0% (UF) of patients with documented PSH, and 85.0%(WCM), 89.5%(NM), and 100%(UF) of patients with documented FSH in their notes. Discussion While PSH and FSH are significant risk factors for future suicide events, little effort has been made previously to identify individuals with these history. To address this, we developed a transformer based DL method and compared with conventional rule-based NLP approach. The varying effectiveness of the rule-based tools across sites suggests a need for improvement in its dictionary-based approach. In contrast, the performances of the DL tools were higher and comparable across sites. Furthermore, DL tools were fine-tuned using only small number of annotated notes at each site, underscores its greater adaptability to local documentation practices and lexical variations. Conclusion Variations in local documentation practices across health care systems pose challenges to rule-based NLP tools. In contrast, the developed DL tools can effectively extract PSH and FSH information from unstructured clinical notes. These tools will provide clinicians with crucial information for assessing and treating patients at elevated risk for suicide who are rarely been diagnosed.
Collapse
Affiliation(s)
| | - Al'ona Furmanchuk
- Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Yonghui Wu
- University of Florida College of Medicine, Gainesville, FL, USA
| | - Aman Pathak
- University of Florida College of Medicine, Gainesville, FL, USA
| | | | - Sarah Bost
- University of Florida College of Medicine, Gainesville, FL, USA
| | | | | | - Yuyang Yang
- Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | | | - Yuan Luo
- Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Theresa L Walunas
- Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Wei-Hsuan Lo-Ciganic Jenny
- University of Florida College of Medicine, Gainesville, FL, USA
- University of Pittsburgh School of Medicine, Pittsburgh, PA, USA
| | - Walid Gelad
- University of Pittsburgh School of Medicine, Pittsburgh, PA, USA
| | - Jiang Bian
- University of Florida College of Medicine, Gainesville, FL, USA
| | - Yuhua Bao
- Weill Cornell Medicine, New York, NY, USA
| | | | - David Oslin
- Corporal Michael J Crescenz Veterans Affairs Medical Center, Philadelphia, PA, USA
| | | |
Collapse
|
6
|
Pigoni A, Delvecchio G, Turtulici N, Madonna D, Pietrini P, Cecchetti L, Brambilla P. Machine learning and the prediction of suicide in psychiatric populations: a systematic review. Transl Psychiatry 2024; 14:140. [PMID: 38461283 PMCID: PMC10925059 DOI: 10.1038/s41398-024-02852-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Revised: 02/22/2024] [Accepted: 02/22/2024] [Indexed: 03/11/2024] Open
Abstract
Machine learning (ML) has emerged as a promising tool to enhance suicidal prediction. However, as many large-sample studies mixed psychiatric and non-psychiatric populations, a formal psychiatric diagnosis emerged as a strong predictor of suicidal risk, overshadowing more subtle risk factors specific to distinct populations. To overcome this limitation, we conducted a systematic review of ML studies evaluating suicidal behaviors exclusively in psychiatric clinical populations. A systematic literature search was performed from inception through November 17, 2022 on PubMed, EMBASE, and Scopus following the PRISMA guidelines. Original research using ML techniques to assess the risk of suicide or predict suicide attempts in the psychiatric population were included. An assessment for bias risk was performed using the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guidelines. About 1032 studies were retrieved, and 81 satisfied the inclusion criteria and were included for qualitative synthesis. Clinical and demographic features were the most frequently employed and random forest, support vector machine, and convolutional neural network performed better in terms of accuracy than other algorithms when directly compared. Despite heterogeneity in procedures, most studies reported an accuracy of 70% or greater based on features such as previous attempts, severity of the disorder, and pharmacological treatments. Although the evidence reported is promising, ML algorithms for suicidal prediction still present limitations, including the lack of neurobiological and imaging data and the lack of external validation samples. Overcoming these issues may lead to the development of models to adopt in clinical practice. Further research is warranted to boost a field that holds the potential to critically impact suicide mortality.
Collapse
Affiliation(s)
- Alessandro Pigoni
- Social and Affective Neuroscience Group, MoMiLab, IMT School for Advanced Studies Lucca, Lucca, Italy
- Department of Neurosciences and Mental Health, Fondazione IRCCS Ca' Granda, Ospedale Maggiore Policlinico, Milan, Italy
| | - Giuseppe Delvecchio
- Department of Neurosciences and Mental Health, Fondazione IRCCS Ca' Granda, Ospedale Maggiore Policlinico, Milan, Italy
| | - Nunzio Turtulici
- Department of Pathophysiology and Transplantation, University of Milan, Milan, Italy
| | - Domenico Madonna
- Department of Neurosciences and Mental Health, Fondazione IRCCS Ca' Granda, Ospedale Maggiore Policlinico, Milan, Italy
| | - Pietro Pietrini
- MoMiLab, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Luca Cecchetti
- Social and Affective Neuroscience Group, MoMiLab, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Paolo Brambilla
- Department of Neurosciences and Mental Health, Fondazione IRCCS Ca' Granda, Ospedale Maggiore Policlinico, Milan, Italy.
- Department of Pathophysiology and Transplantation, University of Milan, Milan, Italy.
| |
Collapse
|
7
|
Workman TE, Goulet JL, Brandt CA, Warren AR, Eleazer J, Skanderson M, Lindemann L, Blosnich JR, O'Leary J, Zeng‐Treitler Q. Identifying suicide documentation in clinical notes through zero-shot learning. Health Sci Rep 2023; 6:e1526. [PMID: 37706016 PMCID: PMC10495736 DOI: 10.1002/hsr2.1526] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2023] [Revised: 08/08/2023] [Accepted: 08/11/2023] [Indexed: 09/15/2023] Open
Abstract
Background and Aims In deep learning, a major difficulty in identifying suicidality and its risk factors in clinical notes is the lack of training samples given the small number of true positive instances among the number of patients screened. This paper describes a novel methodology that identifies suicidality in clinical notes by addressing this data sparsity issue through zero-shot learning. Our general aim was to develop a tool that leveraged zero-shot learning to effectively identify suicidality documentation in all types of clinical notes. Methods US Veterans Affairs clinical notes served as data. The training data set label was determined using diagnostic codes of suicide attempt and self-harm. We used a base string associated with the target label of suicidality to provide auxiliary information by narrowing the positive training cases to those containing the base string. We trained a deep neural network by mapping the training documents' contents to a semantic space. For comparison, we trained another deep neural network using the identical training data set labels, and bag-of-words features. Results The zero-shot learning model outperformed the baseline model in terms of area under the curve, sensitivity, specificity, and positive predictive value at multiple probability thresholds. In applying a 0.90 probability threshold, the methodology identified notes documenting suicidality but not associated with a relevant ICD-10-CM code, with 94% accuracy. Conclusion This method can effectively identify suicidality without manual annotation.
Collapse
Affiliation(s)
- Terri Elizabeth Workman
- Biomedical Informatics CenterThe George Washington UniversityWashingtonDistrict of ColumbiaUSA
- VA Medical CenterWashingtonDistrict of ColumbiaUSA
| | - Joseph L. Goulet
- Department of Emergency MedicineYale School of MedicineNew HavenConnecticutUSA
- VA Connecticut Healthcare SystemWest HavenConnecticutUSA
| | - Cynthia A. Brandt
- Department of Emergency MedicineYale School of MedicineNew HavenConnecticutUSA
- VA Connecticut Healthcare SystemWest HavenConnecticutUSA
| | - Allison R. Warren
- PRIME Center, VA Connecticut Healthcare SystemWest HavenConnecticutUSA
| | - Jacob Eleazer
- PRIME Center, VA Connecticut Healthcare SystemWest HavenConnecticutUSA
| | | | - Luke Lindemann
- VA Connecticut Healthcare SystemWest HavenConnecticutUSA
| | - John R. Blosnich
- Suzanne Dworak‐Peck School of Social WorkUniversity of Southern CaliforniaLos AngelesCaliforniaUSA
| | - John O'Leary
- VA Connecticut Healthcare SystemWest HavenConnecticutUSA
- Department of Internal MedicineYale School of MedicineWest HavenConnecticutUSA
| | - Qing Zeng‐Treitler
- Biomedical Informatics CenterThe George Washington UniversityWashingtonDistrict of ColumbiaUSA
- VA Medical CenterWashingtonDistrict of ColumbiaUSA
| |
Collapse
|
8
|
Cliffe C, Cusick M, Vellupillai S, Shear M, Downs J, Epstein S, Pathak J, Dutta R. A multisite comparison using electronic health records and natural language processing to identify the association between suicidality and hospital readmission amongst patients with eating disorders. Int J Eat Disord 2023; 56:1581-1592. [PMID: 37194359 PMCID: PMC10524005 DOI: 10.1002/eat.23980] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/14/2022] [Revised: 04/21/2023] [Accepted: 04/21/2023] [Indexed: 05/18/2023]
Abstract
OBJECTIVES To describe and compare the association between suicidality and subsequent readmission for patients hospitalized for eating disorder treatment, within 2 years of discharge, at two large academic medical centers in two different countries. METHODS Over an 8-year study window from January 2009 to March 2017, we identified all inpatient eating disorder admissions at Weill Cornell Medicine, New York, USA (WCM) and South London and Maudsley Foundation NHS Trust, London, UK (SLaM). To establish each patient's-suicidality profile, we applied two natural language processing (NLP) algorithms, independently developed at the two institutions, and detected suicidality in clinical notes documented in the first week of admission. We calculated the odds ratios (OR) for any subsequent readmission within 2 years postdischarge and determined whether this was to another eating disorder unit, other psychiatric unit, a general medical hospital admission or emergency room attendance. RESULTS We identified 1126 and 420 eating disorder inpatient admissions at WCM and SLaM, respectively. In the WCM cohort, evidence of above average suicidality during the first week of admission was significantly associated with an increased risk of noneating disorder-related psychiatric readmission (OR 3.48 95% CI = 2.03-5.99, p-value < .001), but a similar pattern was not observed in the SLaM cohort (OR 1.34, 95% CI = 0.75-2.37, p = .32), there was no significant increase in risk of admission. In both cohorts, personality disorder increased the risk of any psychiatric readmission within 2 years. DISCUSSION Patterns of increased risk of psychiatric readmission from above average suicidality detected via NLP during inpatient eating disorder admissions differed in our two patient cohorts. However, comorbid diagnoses such as personality disorder increased the risk of any psychiatric readmission across both cohorts. PUBLIC SIGNIFICANCE Suicidality amongst is eating disorders is an extremely common presentation and it is important we further our understanding of identifying those most at risk. This research also provides a novel study design, comparing two NLP algorithms on electronic health record data based in the United States and United Kingdom on eating disorder inpatients. Studies researching both UK and US mental health patients are sparse therefore this study provides novel data.
Collapse
Affiliation(s)
- Charlotte Cliffe
- Department of Psychological Medicine, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
- South London & Maudsley Foundation NHS Trust, London, UK
| | - Marika Cusick
- Division of Population Health Sciences, Cornell University, New York, New York, USA
- Department of Health Policy, Stanford School of Medicine, Stanford, CA, USA
| | - Sumithra Vellupillai
- Department of Psychological Medicine, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
| | - Matthew Shear
- Department of Psychiatry, Weill Cornell Medicine, New York, New York, USA
- Psychiatry, New York Presbyterian Hospital, White Plains, New York, USA
| | - Johnny Downs
- South London & Maudsley Foundation NHS Trust, London, UK
- Department of Child and Adolescent Psychiatry, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom
| | - Sophie Epstein
- South London & Maudsley Foundation NHS Trust, London, UK
- Department of Child and Adolescent Psychiatry, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom
| | - Jyotishman Pathak
- Division of Population Health Sciences, Cornell University, New York, New York, USA
| | - Rina Dutta
- Department of Psychological Medicine, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
- South London & Maudsley Foundation NHS Trust, London, UK
| |
Collapse
|
9
|
Parsapoor (Mah Parsa) M, Koudys JW, Ruocco AC. Suicide risk detection using artificial intelligence: the promise of creating a benchmark dataset for research on the detection of suicide risk. Front Psychiatry 2023; 14:1186569. [PMID: 37564247 PMCID: PMC10411603 DOI: 10.3389/fpsyt.2023.1186569] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/15/2023] [Accepted: 06/14/2023] [Indexed: 08/12/2023] Open
Abstract
Suicide is a leading cause of death that demands cross-disciplinary research efforts to develop and deploy suicide risk screening tools. Such tools, partly informed by influential suicide theories, can help identify individuals at the greatest risk of suicide and should be able to predict the transition from suicidal thoughts to suicide attempts. Advances in artificial intelligence have revolutionized the development of suicide screening tools and suicide risk detection systems. Thus, various types of AI systems, including text-based systems, have been proposed to identify individuals at risk of suicide. Although these systems have shown acceptable performance, most of them have not incorporated suicide theories in their design. Furthermore, directly applying suicide theories may be difficult because of the diversity and complexity of these theories. To address these challenges, we propose an approach to develop speech- and language-based suicide risk detection systems. We highlight the promise of establishing a benchmark textual and vocal dataset using a standardized speech and language assessment procedure, and research designs that distinguish between the risk factors for suicide attempt above and beyond those for suicidal ideation alone. The benchmark dataset could be used to develop trustworthy machine learning or deep learning-based suicide risk detection systems, ultimately constructing a foundation for vocal and textual-based suicide risk detection systems.
Collapse
Affiliation(s)
| | - Jacob W. Koudys
- Department of Psychological Clinical Science, University of Toronto, Toronto, ON, Canada
| | - Anthony C. Ruocco
- Department of Psychological Clinical Science, University of Toronto, Toronto, ON, Canada
- Department of Psychology, University of Toronto Scarborough Toronto, Toronto, ON, Canada
| |
Collapse
|
10
|
Datta S, Roberts K. Weakly supervised spatial relation extraction from radiology reports. JAMIA Open 2023; 6:ooad027. [PMID: 37096148 PMCID: PMC10122604 DOI: 10.1093/jamiaopen/ooad027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2022] [Revised: 03/16/2023] [Accepted: 04/04/2023] [Indexed: 04/26/2023] Open
Abstract
Objective Weak supervision holds significant promise to improve clinical natural language processing by leveraging domain resources and expertise instead of large manually annotated datasets alone. Here, our objective is to evaluate a weak supervision approach to extract spatial information from radiology reports. Materials and Methods Our weak supervision approach is based on data programming that uses rules (or labeling functions) relying on domain-specific dictionaries and radiology language characteristics to generate weak labels. The labels correspond to different spatial relations that are critical to understanding radiology reports. These weak labels are then used to fine-tune a pretrained Bidirectional Encoder Representations from Transformers (BERT) model. Results Our weakly supervised BERT model provided satisfactory results in extracting spatial relations without manual annotations for training (spatial trigger F1: 72.89, relation F1: 52.47). When this model is further fine-tuned on manual annotations (relation F1: 68.76), performance surpasses the fully supervised state-of-the-art. Discussion To our knowledge, this is the first work to automatically create detailed weak labels corresponding to radiological information of clinical significance. Our data programming approach is (1) adaptable as the labeling functions can be updated with relatively little manual effort to incorporate more variations in radiology language reporting formats and (2) generalizable as these functions can be applied across multiple radiology subdomains in most cases. Conclusions We demonstrate a weakly supervision model performs sufficiently well in identifying a variety of relations from radiology text without manual annotations, while exceeding state-of-the-art results when annotated data are available.
Collapse
Affiliation(s)
- Surabhi Datta
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA
| | - Kirk Roberts
- Corresponding Author: Kirk Roberts, PhD, School of Biomedical Informatics, The University of Texas Health Science Center at Houston, 7000 Fannin St, Suite 600, Houston, TX 77030, USA;
| |
Collapse
|
11
|
Dong H, Suárez-Paniagua V, Zhang H, Wang M, Casey A, Davidson E, Chen J, Alex B, Whiteley W, Wu H. Ontology-driven and weakly supervised rare disease identification from clinical notes. BMC Med Inform Decis Mak 2023; 23:86. [PMID: 37147628 PMCID: PMC10162001 DOI: 10.1186/s12911-023-02181-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/09/2022] [Accepted: 04/21/2023] [Indexed: 05/07/2023] Open
Abstract
BACKGROUND Computational text phenotyping is the practice of identifying patients with certain disorders and traits from clinical notes. Rare diseases are challenging to be identified due to few cases available for machine learning and the need for data annotation from domain experts. METHODS We propose a method using ontologies and weak supervision, with recent pre-trained contextual representations from Bi-directional Transformers (e.g. BERT). The ontology-driven framework includes two steps: (i) Text-to-UMLS, extracting phenotypes by contextually linking mentions to concepts in Unified Medical Language System (UMLS), with a Named Entity Recognition and Linking (NER+L) tool, SemEHR, and weak supervision with customised rules and contextual mention representation; (ii) UMLS-to-ORDO, matching UMLS concepts to rare diseases in Orphanet Rare Disease Ontology (ORDO). The weakly supervised approach is proposed to learn a phenotype confirmation model to improve Text-to-UMLS linking, without annotated data from domain experts. We evaluated the approach on three clinical datasets, MIMIC-III discharge summaries, MIMIC-III radiology reports, and NHS Tayside brain imaging reports from two institutions in the US and the UK, with annotations. RESULTS The improvements in the precision were pronounced (by over 30% to 50% absolute score for Text-to-UMLS linking), with almost no loss of recall compared to the existing NER+L tool, SemEHR. Results on radiology reports from MIMIC-III and NHS Tayside were consistent with the discharge summaries. The overall pipeline processing clinical notes can extract rare disease cases, mostly uncaptured in structured data (manually assigned ICD codes). CONCLUSION The study provides empirical evidence for the task by applying a weakly supervised NLP pipeline on clinical notes. The proposed weak supervised deep learning approach requires no human annotation except for validation and testing, by leveraging ontologies, NER+L tools, and contextual representations. The study also demonstrates that Natural Language Processing (NLP) can complement traditional ICD-based approaches to better estimate rare diseases in clinical notes. We discuss the usefulness and limitations of the weak supervision approach and propose directions for future studies.
Collapse
Affiliation(s)
- Hang Dong
- Centre for Medical Informatics, Usher Institute of Population Health Sciences and Informatics, University of Edinburgh, Edinburgh, United Kingdom.
- Health Data Research UK, London, United Kingdom.
- Department of Computer Science, University of Oxford, Oxford, United Kingdom.
| | - Víctor Suárez-Paniagua
- Centre for Medical Informatics, Usher Institute of Population Health Sciences and Informatics, University of Edinburgh, Edinburgh, United Kingdom
- Health Data Research UK, London, United Kingdom
| | - Huayu Zhang
- Advanced Care Research Centre, Usher Institute, University of Edinburgh, Edinburgh, United Kingdom
| | - Minhong Wang
- Institute of Health Informatics, University College London, London, United Kingdom
| | - Arlene Casey
- Advanced Care Research Centre, Usher Institute, University of Edinburgh, Edinburgh, United Kingdom
| | - Emma Davidson
- Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, United Kingdom
| | - Jiaoyan Chen
- Department of Computer Science, The University of Manchester, Manchester, United Kingdom
| | - Beatrice Alex
- Edinburgh Futures Institute, University of Edinburgh, Edinburgh, United Kingdom
| | - William Whiteley
- Health Data Research UK, London, United Kingdom
- Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, United Kingdom
| | - Honghan Wu
- Health Data Research UK, London, United Kingdom.
- Institute of Health Informatics, University College London, London, United Kingdom.
| |
Collapse
|
12
|
A review of natural language processing in the identification of suicidal behavior. JOURNAL OF AFFECTIVE DISORDERS REPORTS 2023. [DOI: 10.1016/j.jadr.2023.100507] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/25/2023] Open
|
13
|
Broadbent M, Medina Grespan M, Axford K, Zhang X, Srikumar V, Kious B, Imel Z. A machine learning approach to identifying suicide risk among text-based crisis counseling encounters. Front Psychiatry 2023; 14:1110527. [PMID: 37032952 PMCID: PMC10076638 DOI: 10.3389/fpsyt.2023.1110527] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/28/2022] [Accepted: 02/23/2023] [Indexed: 04/11/2023] Open
Abstract
Introduction With the increasing utilization of text-based suicide crisis counseling, new means of identifying at risk clients must be explored. Natural language processing (NLP) holds promise for evaluating the content of crisis counseling; here we use a data-driven approach to evaluate NLP methods in identifying client suicide risk. Methods De-identified crisis counseling data from a regional text-based crisis encounter and mobile tipline application were used to evaluate two modeling approaches in classifying client suicide risk levels. A manual evaluation of model errors and system behavior was conducted. Results The neural model outperformed a term frequency-inverse document frequency (tf-idf) model in the false-negative rate. While 75% of the neural model's false negative encounters had some discussion of suicidality, 62.5% saw a resolution of the client's initial concerns. Similarly, the neural model detected signals of suicidality in 60.6% of false-positive encounters. Discussion The neural model demonstrated greater sensitivity in the detection of client suicide risk. A manual assessment of errors and model performance reflected these same findings, detecting higher levels of risk in many of the false-positive encounters and lower levels of risk in many of the false negatives. NLP-based models can detect the suicide risk of text-based crisis encounters from the encounter's content.
Collapse
Affiliation(s)
- Meghan Broadbent
- Educational Psychology, The University of Utah, Salt Lake City, UT, United States
| | - Mattia Medina Grespan
- Kahlert School of Computing, The University of Utah, Salt Lake City, UT, United States
| | - Katherine Axford
- Educational Psychology, The University of Utah, Salt Lake City, UT, United States
| | - Xinyao Zhang
- Educational Psychology, The University of Utah, Salt Lake City, UT, United States
| | - Vivek Srikumar
- Kahlert School of Computing, The University of Utah, Salt Lake City, UT, United States
| | - Brent Kious
- Department of Psychiatry, The University of Utah, Salt Lake City, UT, United States
| | - Zac Imel
- Educational Psychology, The University of Utah, Salt Lake City, UT, United States
| |
Collapse
|
14
|
Cusick M, Velupillai S, Downs J, Campion TR, Sholle ET, Dutta R, Pathak J. Portability of natural language processing methods to detect suicidality from clinical text in US and UK electronic health records. JOURNAL OF AFFECTIVE DISORDERS REPORTS 2022; 10:100430. [PMID: 36644339 PMCID: PMC9835770 DOI: 10.1016/j.jadr.2022.100430] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022] Open
Abstract
Background In the global effort to prevent death by suicide, many academic medical institutions are implementing natural language processing (NLP) approaches to detect suicidality from unstructured clinical text in electronic health records (EHRs), with the hope of targeting timely, preventative interventions to individuals most at risk of suicide. Despite the international need, the development of these NLP approaches in EHRs has been largely local and not shared across healthcare systems. Methods In this study, we developed a process to share NLP approaches that were individually developed at King's College London (KCL), UK and Weill Cornell Medicine (WCM), US - two academic medical centers based in different countries with vastly different healthcare systems. We tested and compared the algorithms' performance on manually annotated clinical notes (KCL: n = 4,911 and WCM = 837). Results After a successful technical porting of the NLP approaches, our quantitative evaluation determined that independently developed NLP approaches can detect suicidality at another healthcare organization with a different EHR system, clinical documentation processes, and culture, yet do not achieve the same level of success as at the institution where the NLP algorithm was developed (KCL approach: F1-score 0.85 vs. 0.68, WCM approach: F1-score 0.87 vs. 0.72). Limitations Independent NLP algorithm development and patient cohort selection at the two institutions comprised direct comparability. Conclusions Shared use of these NLP approaches is a critical step forward towards improving data-driven algorithms for early suicide risk identification and timely prevention.
Collapse
Affiliation(s)
- Marika Cusick
- WeiCornell Medicine, 402 E. 67th St., New York, NY 10065, USA, South London and Maudsley NHS Foundation Trust, London, UK, Corresponding author. (M. Cusick)
| | - Sumithra Velupillai
- IoPPN, King’s College London, London, UK, South London and Maudsley NHS Foundation Trust, London, UK
| | - Johnny Downs
- IoPPN, King’s College London, London, UK, South London and Maudsley NHS Foundation Trust, London, UK
| | - Thomas R. Campion
- WeiCornell Medicine, 402 E. 67th St., New York, NY 10065, USA, South London and Maudsley NHS Foundation Trust, London, UK
| | - Evan T. Sholle
- WeiCornell Medicine, 402 E. 67th St., New York, NY 10065, USA, South London and Maudsley NHS Foundation Trust, London, UK
| | - Rina Dutta
- IoPPN, King’s College London, London, UK, South London and Maudsley NHS Foundation Trust, London, UK
| | - Jyotishman Pathak
- WeiCornell Medicine, 402 E. 67th St., New York, NY 10065, USA, South London and Maudsley NHS Foundation Trust, London, UK
| |
Collapse
|
15
|
Comparisons of deep learning and machine learning while using text mining methods to identify suicide attempts of patients with mood disorders. J Affect Disord 2022; 317:107-113. [PMID: 36029873 DOI: 10.1016/j.jad.2022.08.054] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/27/2022] [Revised: 08/05/2022] [Accepted: 08/20/2022] [Indexed: 11/23/2022]
Abstract
BACKGROUND Suicide attempt is one of the most severe consequences for patients with mood disorders. This study aimed to perform deep learning and machine learning while using text mining to identify patients with suicide attempts and to compare their effectiveness. METHODS A total of 13,100 patients with mood disorders were selected. Two traditional text mining methods, logistic regression and Support vector machine (SVM), and one deep learning model (Convolutional neural network, CNN) were adopted to perform overall analysis and gender-specific subgroup analysis of patients to identify suicide attempts. The classification effectiveness of these models was evaluated by accuracy, F1-value, precision, recall, and the area under Receiver operator characteristic curve (ROC). RESULTS CNN's results were greater than the other two for all indicators except recall which was slightly smaller than SVM in male subgroup analysis. The accuracy values of the CNN were 98.4 %, 98.2 %, and 98.5 % in the overall analysis and the subgroup analysis for males and females, respectively. The results of McNemar's test showed that CNN and SVM models' predictions were statistically different from the logistic regression model's predictions in the overall analysis and the subgroup analysis for females (P < 0.050). LIMITATIONS A fixed number of features were selected based on document frequency to train models; this was a single-site study. CONCLUSIONS CNN model was a better way to detect suicide attempts in patients with mood disorders prior to hospital admission, saving time and resources in recognizing high-risk patients and preventing suicide.
Collapse
|
16
|
Nordin N, Zainol Z, Mohd Noor MH, Chan LF. Suicidal behaviour prediction models using machine learning techniques: A systematic review. Artif Intell Med 2022; 132:102395. [DOI: 10.1016/j.artmed.2022.102395] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2022] [Revised: 08/12/2022] [Accepted: 08/29/2022] [Indexed: 11/02/2022]
|
17
|
Improving ascertainment of suicidal ideation and suicide attempt with natural language processing. Sci Rep 2022; 12:15146. [PMID: 36071081 PMCID: PMC9452591 DOI: 10.1038/s41598-022-19358-3] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2022] [Accepted: 08/29/2022] [Indexed: 12/03/2022] Open
Abstract
Methods relying on diagnostic codes to identify suicidal ideation and suicide attempt in Electronic Health Records (EHRs) at scale are suboptimal because suicide-related outcomes are heavily under-coded. We propose to improve the ascertainment of suicidal outcomes using natural language processing (NLP). We developed information retrieval methodologies to search over 200 million notes from the Vanderbilt EHR. Suicide query terms were extracted using word2vec. A weakly supervised approach was designed to label cases of suicidal outcomes. The NLP validation of the top 200 retrieved patients showed high performance for suicidal ideation (area under the receiver operator curve [AUROC]: 98.6, 95% confidence interval [CI] 97.1–99.5) and suicide attempt (AUROC: 97.3, 95% CI 95.2–98.7). Case extraction produced the best performance when combining NLP and diagnostic codes and when accounting for negated suicide expressions in notes. Overall, we demonstrated that scalable and accurate NLP methods can be developed to identify suicidal behavior in EHRs to enhance prevention efforts, predictive models, and precision medicine.
Collapse
|
18
|
ScAN: Suicide Attempt and Ideation Events Dataset. PROCEEDINGS OF THE CONFERENCE. ASSOCIATION FOR COMPUTATIONAL LINGUISTICS. NORTH AMERICAN CHAPTER. MEETING 2022; 2022:1029-1040. [PMID: 36848299 PMCID: PMC9958515 DOI: 10.18653/v1/2022.naacl-main.75] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
Suicide is an important public health concern and one of the leading causes of death worldwide. Suicidal behaviors, including suicide attempts (SA) and suicide ideations (SI), are leading risk factors for death by suicide. Information related to patients' previous and current SA and SI are frequently documented in the electronic health record (EHR) notes. Accurate detection of such documentation may help improve surveillance and predictions of patients' suicidal behaviors and alert medical professionals for suicide prevention efforts. In this study, we first built Suicide Attempt and Ideation Events (ScAN) dataset, a subset of the publicly available MIMIC III dataset spanning over 12k+ EHR notes with 19k+ annotated SA and SI events information. The annotations also contain attributes such as method of suicide attempt. We also provide a strong baseline model ScANER (Suicide Attempt and Ideation Events Retreiver), a multi-task RoBERTa-based model with a retrieval module to extract all the relevant suicidal behavioral evidences from EHR notes of an hospital-stay and, and a prediction module to identify the type of suicidal behavior (SA and SI) concluded during the patient's stay at the hospital. ScANER achieved a macro-weighted F1-score of 0.83 for identifying suicidal behavioral evidences and a macro F1-score of 0.78 and 0.60 for classification of SA and SI for the patient's hospital-stay, respectively. ScAN and ScANER are publicly available.
Collapse
|
19
|
Zhong Z, Bao W, Wang J, Zhu X, Zhang X. FLEE: A Hierarchical Federated Learning Framework for Distributed Deep Neural Network over Cloud, Edge and End Device. ACM T INTEL SYST TEC 2022. [DOI: 10.1145/3514501] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/31/2022]
Abstract
With the development of smart devices, the computing capabilities of portable end devices such as mobile phones have been greatly enhanced. Meanwhile, traditional cloud computing faces great challenges caused by privacy-leakage and time-delay problems, there is a trend to push models down to edges and end devices. However, due to the limitation of computing resource, it is difficult for end devices to complete complex computing tasks alone. Therefore, this paper divides the model into two parts and deploys them on multiple end devices and edges respectively. Meanwhile, an early exit is set to reduce computing resource overhead, forming a hierarchical distributed architecture. In order to enable the distributed model to continuously evolve by using new data generated by end devices, we comprehensively consider various data distributions on end devices and edges, proposing a hierarchical federated learning framework
FLEE
, which can realize dynamical updates of models without redeploying them. Through image and sentence classification experiments, we verify that it can improve model performances under all kinds of data distributions, and prove that compared with other frameworks, the models trained by
FLEE
consume less global computing resource in the inference stage.
Collapse
Affiliation(s)
- Zhengyi Zhong
- College of Systems Engineering, National University of Defense Technology, China
| | - Weidong Bao
- College of Systems Engineering, National University of Defense Technology, China
| | - Ji Wang
- College of Systems Engineering, National University of Defense Technology, China
| | - Xiaomin Zhu
- College of Systems Engineering, National University of Defense Technology, China
| | - Xiongtao Zhang
- College of Systems Engineering, National University of Defense Technology, China
| |
Collapse
|
20
|
Machine learning for suicidal ideation identification: A systematic literature review. COMPUTERS IN HUMAN BEHAVIOR 2022. [DOI: 10.1016/j.chb.2021.107095] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
21
|
Campion TR, Sholle ET, Pathak J, Johnson SB, Leonard JP, Cole CL. An architecture for research computing in health to support clinical and translational investigators with electronic patient data. J Am Med Inform Assoc 2021; 29:677-685. [PMID: 34850911 PMCID: PMC8690260 DOI: 10.1093/jamia/ocab266] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2021] [Revised: 10/20/2021] [Accepted: 11/15/2021] [Indexed: 12/13/2022] Open
Abstract
Objective Obtaining electronic patient data, especially from electronic health record (EHR) systems, for clinical and translational research is difficult. Multiple research informatics systems exist but navigating the numerous applications can be challenging for scientists. This article describes Architecture for Research Computing in Health (ARCH), our institution’s approach for matching investigators with tools and services for obtaining electronic patient data. Materials and Methods Supporting the spectrum of studies from populations to individuals, ARCH delivers a breadth of scientific functions—including but not limited to cohort discovery, electronic data capture, and multi-institutional data sharing—that manifest in specific systems—such as i2b2, REDCap, and PCORnet. Through a consultative process, ARCH staff align investigators with tools with respect to study design, data sources, and cost. Although most ARCH services are available free of charge, advanced engagements require fee for service. Results Since 2016 at Weill Cornell Medicine, ARCH has supported over 1200 unique investigators through more than 4177 consultations. Notably, ARCH infrastructure enabled critical coronavirus disease 2019 response activities for research and patient care. Discussion ARCH has provided a technical, regulatory, financial, and educational framework to support the biomedical research enterprise with electronic patient data. Collaboration among informaticians, biostatisticians, and clinicians has been critical to rapid generation and analysis of EHR data. Conclusion A suite of tools and services, ARCH helps match investigators with informatics systems to reduce time to science. ARCH has facilitated research at Weill Cornell Medicine and may provide a model for informatics and research leaders to support scientists elsewhere.
Collapse
Affiliation(s)
- Thomas R Campion
- Department of Population Health Sciences, Weill Cornell Medicine, New York, New York, USA.,Department of Pediatrics, Weill Cornell Medicine, New York, New York, USA.,Information Technologies & Services Department, Weill Cornell Medicine, New York, New York, USA.,Clinical and Translational Science Center, Weill Cornell Medicine, New York, New York, USA
| | - Evan T Sholle
- Department of Population Health Sciences, Weill Cornell Medicine, New York, New York, USA.,Information Technologies & Services Department, Weill Cornell Medicine, New York, New York, USA.,Clinical and Translational Science Center, Weill Cornell Medicine, New York, New York, USA
| | - Jyotishman Pathak
- Department of Population Health Sciences, Weill Cornell Medicine, New York, New York, USA.,Clinical and Translational Science Center, Weill Cornell Medicine, New York, New York, USA
| | - Stephen B Johnson
- Department of Population Health, New York University Grossman School of Medicine, New York, New York, USA
| | - John P Leonard
- Department of Medicine, Weill Cornell Medicine, New York, New York, USA
| | - Curtis L Cole
- Department of Population Health Sciences, Weill Cornell Medicine, New York, New York, USA.,Information Technologies & Services Department, Weill Cornell Medicine, New York, New York, USA.,Clinical and Translational Science Center, Weill Cornell Medicine, New York, New York, USA.,Department of Medicine, Weill Cornell Medicine, New York, New York, USA
| |
Collapse
|
22
|
Dong H, Suarez-Paniagua V, Zhang H, Wang M, Whitfield E, Wu H. Rare Disease Identification from Clinical Notes with Ontologies and Weak Supervision. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021; 2021:2294-2298. [PMID: 34891745 DOI: 10.1109/embc46164.2021.9630043] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
The identification of rare diseases from clinical notes with Natural Language Processing (NLP) is challenging due to the few cases available for machine learning and the need of data annotation from clinical experts. We propose a method using ontologies and weak supervision. The approach includes two steps: (i) Text-to-UMLS, linking text mentions to concepts in Unified Medical Language System (UMLS), with a named entity linking tool (e.g. SemEHR) and weak supervision based on customised rules and Bidirectional Encoder Representations from Transformers (BERT) based contextual representations, and (ii) UMLS-to-ORDO, matching UMLS concepts to rare diseases in Orphanet Rare Disease Ontology (ORDO). Using MIMIC-III US intensive care discharge summaries as a case study, we show that the Text-to-UMLS process can be greatly improved with weak supervision, without any annotated data from domain experts. Our analysis shows that the overall pipeline processing discharge summaries can surface rare disease cases, which are mostly uncaptured in manual ICD codes of the hospital admissions.
Collapse
|