Integration of Face-to-Face Screening With Real-time Machine Learning to Predict Risk of Suicide Among Adults

Automatically extracting social determinants of health for suicide: a narrative literature review

Suicide is a complex phenomenon that is often not preceded by a diagnosed mental health condition, therefore making it difficult to study and mitigate. Artificial Intelligence has increasingly been used to better understand Social Determinants of Health factors that influence suicide outcomes. In this review we find that many studies use limited SDoH information and minority groups are often underrepresented, thereby omitting important factors that could influence risk of suicide.

Scalable incident detection via natural language processing and probabilistic language models

Post marketing safety surveillance depends in part on the ability to detect concerning clinical events at scale. Spontaneous reporting might be an effective component of safety surveillance, but it requires awareness and understanding among healthcare professionals to achieve its potential. Reliance on readily available structured data such as diagnostic codes risks under-coding and imprecision. Clinical textual data might bridge these gaps, and natural language processing (NLP) has been shown to aid in scalable phenotyping across healthcare records in multiple clinical domains. In this study, we developed and validated a novel incident phenotyping approach using unstructured clinical textual data agnostic to Electronic Health Record (EHR) and note type. It's based on a published, validated approach (PheRe) used to ascertain social determinants of health and suicidality across entire healthcare records. To demonstrate generalizability, we validated this approach on two separate phenotypes that share common challenges with respect to accurate ascertainment: (1) suicide attempt; (2) sleep-related behaviors. With samples of 89,428 records and 35,863 records for suicide attempt and sleep-related behaviors, respectively, we conducted silver standard (diagnostic coding) and gold standard (manual chart review) validation. We showed Area Under the Precision-Recall Curve of ~ 0.77 (95% CI 0.75-0.78) for suicide attempt and AUPR ~ 0.31 (95% CI 0.28-0.34) for sleep-related behaviors. We also evaluated performance by coded race and demonstrated differences in performance by race differed across phenotypes. Scalable phenotyping models, like most healthcare AI, require algorithmovigilance and debiasing prior to implementation.

Risk factors of passive suicidal ideation among outpatients in traditional medicine clinics: The case of Korean medicine in South Korea

Suicide is the leading cause of death worldwide, especially in South Korea. Individuals using Korean medicine (KM) clinics are common in this country, but KM doctor is not yet used in the country's suicide prevention policy. In this study, we investigated the prevalence and risk factors of passive suicidal ideation (SI) among outpatients at KM clinics, and attempted to develop a predictive model of SI through multivariate analysis. The Korea Health Panel Annual Data 2019, a nationally representative survey in South Korea, was analyzed. In this study, 1924 (weighted n:5,958,666) people aged 19 or older who used the KM outpatient service at least once in 2019 were set as study subjects. The authors calculated the weighted prevalence of passive SI and identified significant sociodemographic, physical, and mental factors using the Rao-Scott chi-square test and weighted logistic regression. A suicide risk scorecard was developed using the point to double the odds method to quantify individual risk, resulting in a score range of 0-100. We found high prevalence of passive SI among KM outpatients (7.5 %), and the annual prevalence of passive SI tended to be higher with increasing age (4.6 % in young adults, 6.7 % in middle-aged, and 12.4 % in old age). Also, the prevalence increased as stress worsened (2.0 % in hardly, 3.9 % in a little, 13.7 % in a lot, and 24.8 % in very much). By using multiple logistic regression analysis, SI risk prediction score model was developed, including basic livelihood security recipients, presence of chronic diseases, perceived stress, depression, and low quality of life. According to our findings, passive SI is not uncommon among KM outpatients. Based on current findings, the potential role of KMDs in the country's suicide prevention strategy can be discussed focusing on specific groups such as socioeconomically vulnerable groups, the elderly population, and those with physical illnesses.

Accuracy of ICD-10 codes for suicidal ideation and action in pediatric emergency department encounters

Objectives

According to the ideation-to-action framework of suicidality, suicidal ideation and suicidal action arise via distinct trajectories. Studying suicidality under this framework requires accurate identification of both ideation and action. We sought to assess the accuracy of ICD-10 codes for suicidal ideation and action in emergency department (ED) encounters.

Methods

Accuracy of ICD-10 coding for suicidality was assessed through chart review of clinical notes for 205 ED encounters among patients 6-18 years old at a large academic pediatric hospital between June 1, 2016, and June 1, 2022. Physician notes were reviewed for documentation of past or present suicidal ideation, suicidal action, or both. The study cohort consisted of 103 randomly selected "cases," or encounters assigned at least one ICD-10 code for suicidality, and 102 propensity-matched "non-cases" lacking ICD-10 codes. Accuracy of ICD-10 codes was assessed using sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV).

Results

Against a gold standard chart review, the PPV for ICD-10 suicidality codes was 86.9%, and the NPV was 76.2%. Nearly half of encounters involving suicidality were not captured by ICD-10 coding (sensitivity=53.4%). Sensitivity was higher for ideation-present (82.4%) than for action-present (33.7%) or action-past (20.4%).

Conclusions

Many cases of suicidality may be missed by relying on only ICD-10 codes. Accuracy of ICD-10 codes is high for suicidal ideation but low for action. To scale the ideation-to-action model for use in large populations, better data sources are needed to identify cases of suicidal action.

Machine Learning-Based Prediction of Suicidal Thinking in Adolescents by Derivation and Validation in 3 Independent Worldwide Cohorts: Algorithm Development and Validation Study

BACKGROUND

Suicide is the second-leading cause of death among adolescents and is associated with clusters of suicides. Despite numerous studies on this preventable cause of death, the focus has primarily been on single nations and traditional statistical methods.

OBJECTIVE

This study aims to develop a predictive model for adolescent suicidal thinking using multinational data sets and machine learning (ML).

METHODS

We used data from the Korea Youth Risk Behavior Web-based Survey with 566,875 adolescents aged between 13 and 18 years and conducted external validation using the Youth Risk Behavior Survey with 103,874 adolescents and Norway's University National General Survey with 19,574 adolescents. Several tree-based ML models were developed, and feature importance and Shapley additive explanations values were analyzed to identify risk factors for adolescent suicidal thinking.

RESULTS

When trained on the Korea Youth Risk Behavior Web-based Survey data from South Korea with a 95% CI, the XGBoost model reported an area under the receiver operating characteristic (AUROC) curve of 90.06% (95% CI 89.97-90.16), displaying superior performance compared to other models. For external validation using the Youth Risk Behavior Survey data from the United States and the University National General Survey from Norway, the XGBoost model achieved AUROCs of 83.09% and 81.27%, respectively. Across all data sets, XGBoost consistently outperformed the other models with the highest AUROC score, and was selected as the optimal model. In terms of predictors of suicidal thinking, feelings of sadness and despair were the most influential, accounting for 57.4% of the impact, followed by stress status at 19.8%. This was followed by age (5.7%), household income (4%), academic achievement (3.4%), sex (2.1%), and others, which contributed less than 2% each.

CONCLUSIONS

This study used ML by integrating diverse data sets from 3 countries to address adolescent suicide. The findings highlight the important role of emotional health indicators in predicting suicidal thinking among adolescents. Specifically, sadness and despair were identified as the most significant predictors, followed by stressful conditions and age. These findings emphasize the critical need for early diagnosis and prevention of mental health issues during adolescence.

Developing a clinical decision support system software prototype that assists in the management of patients with self-harm in the emergency department: protocol of the PERMANENS project

BACKGROUND

Self-harm presents a significant public health challenge. Emergency departments (EDs) are crucial healthcare settings in managing self-harm, but clinician uncertainty in risk assessment may contribute to ineffective care. Clinical Decision Support Systems (CDSSs) show promise in enhancing care processes, but their effective implementation in self-harm management remains unexplored.

METHODS

PERMANENS comprises a combination of methodologies and study designs aimed at developing a CDSS prototype that assists clinicians in the personalized assessment and management of ED patients presenting with self-harm. Ensemble prediction models will be constructed by applying machine learning techniques on electronic registry data from four sites, i.e., Catalonia (Spain), Ireland, Norway, and Sweden. These models will predict key adverse outcomes including self-harm repetition, suicide, premature death, and lack of post-discharge care. Available registry data include routinely collected electronic health record data, mortality data, and administrative data, and will be harmonized using the OMOP Common Data Model, ensuring consistency in terminologies, vocabularies and coding schemes. A clinical knowledge base of effective suicide prevention interventions will be developed rooted in a systematic review of clinical practice guidelines, including quality assessment of guidelines using the AGREE II tool. The CDSS software prototype will include a backend that integrates the prediction models and the clinical knowledge base to enable accurate patient risk stratification and subsequent intervention allocation. The CDSS frontend will enable personalized risk assessment and will provide tailored treatment plans, following a tiered evidence-based approach. Implementation research will ensure the CDSS' practical functionality and feasibility, and will include periodic meetings with user-advisory groups, mixed-methods research to identify currently unmet needs in self-harm risk assessment, and small-scale usability testing of the CDSS prototype software.

DISCUSSION

Through the development of the proposed CDSS software prototype, PERMANENS aims to standardize care, enhance clinician confidence, improve patient satisfaction, and increase treatment compliance. The routine integration of CDSS for self-harm risk assessment within healthcare systems holds significant potential in effectively reducing suicide mortality rates by facilitating personalized and timely delivery of effective interventions on a large scale for individuals at risk of suicide.

Randomized Controlled Comparative Effectiveness Trial of Risk Model-Guided Clinical Decision Support for Suicide Screening

Suicide prevention requires risk identification, appropriate intervention, and follow-up. Traditional risk identification relies on patient self-reporting, support network reporting, or face-to-face screening with validated instruments or history and physical exam. In the last decade, statistical risk models have been studied and more recently deployed to augment clinical judgment. Models have generally been found to be low precision or problematic at scale due to low incidence. Few have been tested in clinical practice, and none have been tested in clinical trials to our knowledge.

Methods

We report the results of a pragmatic randomized controlled trial (RCT) in three outpatient adult Neurology clinic settings. This two-arm trial compared the effectiveness of Interruptive and Non-Interruptive Clinical Decision Support (CDS) to prompt further screening of suicidal ideation for those predicted to be high risk using a real-time, validated statistical risk model of suicide attempt risk, with the decision to screen as the primary end point. Secondary outcomes included rates of suicidal ideation and attempts in both arms. Manual chart review of every trial encounter was used to determine if suicide risk assessment was subsequently documented.

Results

From August 16, 2022, through February 16, 2023, our study randomized 596 patient encounters across 561 patients for providers to receive either Interruptive or Non-Interruptive CDS in a 1:1 ratio. Adjusting for provider cluster effects, Interruptive CDS led to significantly higher numbers of decisions to screen (42%=121/289 encounters) compared to Non-Interruptive CDS (4%=12/307) (odds ratio=17.7, p-value <0.001). Secondarily, no documented episodes of suicidal ideation or attempts occurred in either arm. While the proportion of documented assessments among those noting the decision to screen was higher for providers in the Non-Interruptive arm (92%=11/12) than in the Interruptive arm (52%=63/121), the interruptive CDS was associated with more frequent documentation of suicide risk assessment (63/289 encounters compared to 11/307, p-value<0.001).

Conclusions

In this pragmatic RCT of real-time predictive CDS to guide suicide risk assessment, Interruptive CDS led to higher numbers of decisions to screen and documented suicide risk assessments. Well-powered large-scale trials randomizing this type of CDS compared to standard of care are indicated to measure effectiveness in reducing suicidal self-harm. ClinicalTrials.gov Identifier: NCT05312437.

Predicting suicide death after emergency department visits with mental health or self-harm diagnoses

OBJECTIVE

Use health records data to predict suicide death following emergency department visits.

METHODS

Electronic health records and insurance claims from seven health systems were used to: identify emergency department visits with mental health or self-harm diagnoses by members aged 11 or older; extract approximately 2500 potential predictors including demographic, historical, and baseline clinical characteristics; and ascertain subsequent deaths by self-harm. Logistic regression with lasso and random forest models predicted self-harm death over 90 days after each visit.

RESULTS

Records identified 2,069,170 eligible visits, 899 followed by suicide death within 90 days. The best-fitting logistic regression with lasso model yielded an area under the receiver operating curve of 0.823 (95% CI 0.810-0.836). Visits above the 95th percentile of predicted risk included 34.8% (95% CI 31.1-38.7) of subsequent suicide deaths and had a 0.303% (95% CI 0.261-0.346) suicide death rate over the following 90 days. Model performance was similar across subgroups defined by age, sex, race, and ethnicity.

CONCLUSIONS

Machine learning models using coded data from health records have moderate performance in predicting suicide death following emergency department visits for mental health or self-harm diagnosis and could be used to identify patients needing more systematic follow-up.

Machine Learning-Based Prediction of Suicidality in Adolescents With Allergic Rhinitis: Derivation and Validation in 2 Independent Nationwide Cohorts

BACKGROUND

Given the additional risk of suicide-related behaviors in adolescents with allergic rhinitis (AR), it is important to use the growing field of machine learning (ML) to evaluate this risk.

OBJECTIVE

This study aims to evaluate the validity and usefulness of an ML model for predicting suicide risk in patients with AR.

METHODS

We used data from 2 independent survey studies, Korea Youth Risk Behavior Web-based Survey (KYRBS; n=299,468) for the original data set and Korea National Health and Nutrition Examination Survey (KNHANES; n=833) for the external validation data set, to predict suicide risks of AR in adolescents aged 13 to 18 years, with 3.45% (10,341/299,468) and 1.4% (12/833) of the patients attempting suicide in the KYRBS and KNHANES studies, respectively. The outcome of interest was the suicide attempt risks. We selected various ML-based models with hyperparameter tuning in the discovery and performed an area under the receiver operating characteristic curve (AUROC) analysis in the train, test, and external validation data.

RESULTS

The study data set included 299,468 (KYRBS; original data set) and 833 (KNHANES; external validation data set) patients with AR recruited between 2005 and 2022. The best-performing ML model was the random forest model with a mean AUROC of 84.12% (95% CI 83.98%-84.27%) in the original data set. Applying this result to the external validation data set revealed the best performance among the models, with an AUROC of 89.87% (sensitivity 83.33%, specificity 82.58%, accuracy 82.59%, and balanced accuracy 82.96%). While looking at feature importance, the 5 most important features in predicting suicide attempts in adolescent patients with AR are depression, stress status, academic achievement, age, and alcohol consumption.

CONCLUSIONS

This study emphasizes the potential of ML models in predicting suicide risks in patients with AR, encouraging further application of these models in other conditions to enhance adolescent health and decrease suicide rates.

External Validation and Updating of a Statistical Civilian-Based Suicide Risk Model in US Naval Primary Care

Importance

Suicide remains an ongoing concern in the US military. Statistical models have not been broadly disseminated for US Navy service members.

Objective

To externally validate and update a statistical suicide risk model initially developed in a civilian setting with an emphasis on primary care.

Design, Setting, and Participants

This retrospective cohort study used data collected from 2007 through 2017 among active-duty US Navy service members. The external civilian model was applied to every visit at Naval Medical Center Portsmouth (NMCP), its NMCP Naval Branch Health Clinics (NBHCs), and TRICARE Prime Clinics (TPCs) that fall within the NMCP area. The model was retrained and recalibrated using visits to NBHCs and TPCs and updated using Department of Defense (DoD)-specific billing codes and demographic characteristics, including expanded race and ethnicity categories. Domain and temporal analyses were performed with bootstrap validation. Data analysis was performed from September 2020 to December 2022.

Exposure

Visit to US NMCP.

Main Outcomes and Measures

Recorded suicidal behavior on the day of or within 30 days of a visit. Performance was assessed using area under the receiver operating curve (AUROC), area under the precision recall curve (AUPRC), Brier score, and Spiegelhalter z-test statistic.

Results

Of the 260 583 service members, 6529 (2.5%) had a recorded suicidal behavior, 206 412 (79.2%) were male; 104 835 (40.2%) were aged 20 to 24 years; and 9458 (3.6%) were Asian, 56 715 (21.8%) were Black or African American, and 158 277 (60.7%) were White. Applying the civilian-trained model resulted in an AUROC of 0.77 (95% CI, 0.74-0.79) and an AUPRC of 0.004 (95% CI, 0.003-0.005) at NBHCs with poor calibration (Spiegelhalter P < .001). Retraining the algorithm improved AUROC to 0.92 (95% CI, 0.91-0.93) and AUPRC to 0.66 (95% CI, 0.63-0.68). Number needed to screen in the top risk tiers was 366 for the external model and 200 for the retrained model; the lower number indicates better performance. Domain validation showed AUROC of 0.90 (95% CI, 0.90-0.91) and AUPRC of 0.01 (95% CI, 0.01-0.01), and temporal validation showed AUROC of 0.75 (95% CI, 0.72-0.78) and AUPRC of 0.003 (95% CI, 0.003-0.005).

Conclusions and Relevance

In this cohort study of active-duty Navy service members, a civilian suicide attempt risk model was externally validated. Retraining and updating with DoD-specific variables improved performance. Domain and temporal validation results were similar to external validation, suggesting that implementing an external model in US Navy primary care clinics may bypass the need for costly internal development and expedite the automation of suicide prevention in these clinics.

Emergency department visits and boarding for pediatric patients with suicidality before and during the COVID-19 pandemic

OBJECTIVE

To quantify the increase in pediatric patients presenting to the emergency department with suicidality before and during the COVID-19 pandemic, and the subsequent impact on emergency department length of stay and boarding.

METHODS

This retrospective cohort study from June 1, 2016, to October 31, 2022, identified patients ages 6 to 21 presenting to the emergency department at a pediatric academic medical center with suicidality using ICD-10 codes. Number of emergency department encounters for suicidality, demographic characteristics of patients with suicidality, and emergency department length of stay were compared before and during the COVID-19 pandemic. Unobserved components models were used to describe monthly counts of emergency department encounters for suicidality.

RESULTS

There were 179,736 patient encounters to the emergency department during the study period, 6,215 (3.5%) for suicidality. There were, on average, more encounters for suicidality each month during the COVID-19 pandemic than before the COVID-19 pandemic. A time series unobserved components model demonstrated a temporary drop of 32.7 encounters for suicidality in April and May of 2020 (p<0.001), followed by a sustained increase of 31.2 encounters starting in July 2020 (p = 0.003). The average length of stay for patients that boarded in the emergency department with a diagnosis of suicidality was 37.4 hours longer during the COVID-19 pandemic compared to before the COVID-19 pandemic (p<0.001).

CONCLUSIONS

The number of encounters for suicidality among pediatric patients and the emergency department length of stay for psychiatry boarders has increased during the COVID-19 pandemic. There is a need for acute care mental health services and solutions to emergency department capacity issues.

Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings

AIMS AND OBJECTIVES

To examine the nature and use of automation in contemporary clinical information systems by reviewing studies reporting the implementation and evaluation of artificial intelligence (AI) technologies in healthcare settings.

METHOD

PubMed/MEDLINE, Web of Science, EMBASE, the tables of contents of major informatics journals, and the bibliographies of articles were searched for studies reporting evaluation of AI in clinical settings from January 2021 to December 2022. We documented the clinical application areas and tasks supported, and the level of system autonomy. Reported effects on user experience, decision-making, care delivery and outcomes were summarised.

RESULTS

AI technologies are being applied in a wide variety of clinical areas. Most contemporary systems utilise deep learning, use routinely collected data, support diagnosis and triage, are assistive (requiring users to confirm or approve AI provided information or decisions), and are used by doctors in acute care settings in high-income nations. AI systems are integrated and used within existing clinical information systems including electronic medical records. There is limited support for One Health goals. Evaluation is largely based on quantitative methods measuring effects on decision-making.

CONCLUSION

AI systems are being implemented and evaluated in many clinical areas. There remain many opportunities to understand patterns of routine use and evaluate effects on decision-making, care delivery and patient outcomes using mixed-methods. Support for One Health including integrating data about environmental factors and social determinants needs further exploration.

An efficient landmark model for prediction of suicide attempts in multiple clinical settings

Growing evidence has shown that applying machine learning models to large clinical data sources may exceed clinician performance in suicide risk stratification. However, many existing prediction models either suffer from "temporal bias" (a bias that stems from using case-control sampling) or require training on all available patient visit data. Here, we adopt a "landmark model" framework that aligns with clinical practice for prediction of suicide-related behaviors (SRBs) using a large electronic health record database. Using the landmark approach, we developed models for SRB prediction (regularized Cox regression and random survival forest) that establish a time-point (e.g., clinical visit) from which predictions are made over user-specified prediction windows using historical information up to that point. We applied this approach to cohorts from three clinical settings: general outpatient, psychiatric emergency department, and psychiatric inpatients, for varying prediction windows and lengths of historical data. Models achieved high discriminative performance (area under the Receiver Operating Characteristic curve 0.74-0.93 for the Cox model) across different prediction windows and settings, even with relatively short periods of historical data. In short, we developed accurate, dynamic SRB risk prediction models with the landmark approach that reduce bias and enhance the reliability and portability of suicide risk prediction models.

Suicide Risk Assessment, Management, and Mitigation in the Emergency Setting

Suicide is a serious public health issue and is a leading cause of death worldwide. Suicidal ideation is a common presentation in emergency department (ED) settings, with many nuanced complications. Therefore, understanding screening, assessment, and mitigation is paramount to successful encounters with individuals presenting to emergency settings in psychiatric crises. Screening helps to identify the few people at risk within a large group. Assessment seeks to decide whether a specific individual is at significant risk. Mitigation aims to reduce the risk of suicide or of a serious attempt for a person at risk. These aims cannot be achieved with perfect reliability, but some approaches are more effective than others. Suicide screening specifics are important, even to individual practitioners, because a positive screen triggers assessment. Most practitioners understand assessment well: beginning with early psychiatric training, they are taught signs and symptoms suggesting that a patient might be at risk of suicide. Mitigating suicide risk is increasingly important to reduce the misery of ED boarding for patients awaiting psychiatric admission. For many patients, hospital admission is unnecessary if support, monitoring, and contingency plans are workable. For any individual patient, there may be a complicated mix of findings, risks, and interventions. Evidence-based screening and assessment tools are inadequate for the possible complexities, making care of individual patients dependent on good clinical assessment. The authors review the available evidence and offer experienced recommendations for challenges not yet thoroughly researched.