Meta-analysis of the strength of exploratory suicide prediction models; from clinicians to computers

Role of machine learning algorithms in suicide risk prediction: a systematic review-meta analysis of clinical studies

OBJECTIVE

Suicide is a complex and multifactorial public health problem. Understanding and addressing the various factors associated with suicide is crucial for prevention and intervention efforts. Machine learning (ML) could enhance the prediction of suicide attempts.

METHOD

A systematic review was performed using PubMed, Scopus, Web of Science and SID databases. We aim to evaluate the performance of ML algorithms and summarize their effects, gather relevant and reliable information to synthesize existing evidence, identify knowledge gaps, and provide a comprehensive list of the suicide risk factors using mixed method approach.

RESULTS

Forty-one studies published between 2011 and 2022, which matched inclusion criteria, were chosen as suitable. We included studies aimed at predicting the suicide risk by machine learning algorithms except natural language processing (NLP) and image processing. The neural network (NN) algorithm exhibited the lowest accuracy at 0.70, whereas the random forest demonstrated the highest accuracy, reaching 0.94. The study assessed the COX and random forest models and observed a minimum area under the curve (AUC) value of 0.54. In contrast, the XGBoost classifier yielded the highest AUC value, reaching 0.97. These specific AUC values emphasize the algorithm-specific performance in capturing the trade-off between sensitivity and specificity for suicide risk prediction. Furthermore, our investigation identified several common suicide risk factors, including age, gender, substance abuse, depression, anxiety, alcohol consumption, marital status, income, education, and occupation. This comprehensive analysis contributes valuable insights into the multifaceted nature of suicide risk, providing a foundation for targeted preventive strategies and intervention efforts.

CONCLUSIONS

The effectiveness of ML algorithms and their application in predicting suicide risk has been controversial. There is a need for more studies on these algorithms in clinical settings, and the related ethical concerns require further clarification.

Developing a clinical decision support system software prototype that assists in the management of patients with self-harm in the emergency department: protocol of the PERMANENS project

BACKGROUND

Self-harm presents a significant public health challenge. Emergency departments (EDs) are crucial healthcare settings in managing self-harm, but clinician uncertainty in risk assessment may contribute to ineffective care. Clinical Decision Support Systems (CDSSs) show promise in enhancing care processes, but their effective implementation in self-harm management remains unexplored.

METHODS

PERMANENS comprises a combination of methodologies and study designs aimed at developing a CDSS prototype that assists clinicians in the personalized assessment and management of ED patients presenting with self-harm. Ensemble prediction models will be constructed by applying machine learning techniques on electronic registry data from four sites, i.e., Catalonia (Spain), Ireland, Norway, and Sweden. These models will predict key adverse outcomes including self-harm repetition, suicide, premature death, and lack of post-discharge care. Available registry data include routinely collected electronic health record data, mortality data, and administrative data, and will be harmonized using the OMOP Common Data Model, ensuring consistency in terminologies, vocabularies and coding schemes. A clinical knowledge base of effective suicide prevention interventions will be developed rooted in a systematic review of clinical practice guidelines, including quality assessment of guidelines using the AGREE II tool. The CDSS software prototype will include a backend that integrates the prediction models and the clinical knowledge base to enable accurate patient risk stratification and subsequent intervention allocation. The CDSS frontend will enable personalized risk assessment and will provide tailored treatment plans, following a tiered evidence-based approach. Implementation research will ensure the CDSS' practical functionality and feasibility, and will include periodic meetings with user-advisory groups, mixed-methods research to identify currently unmet needs in self-harm risk assessment, and small-scale usability testing of the CDSS prototype software.

DISCUSSION

Through the development of the proposed CDSS software prototype, PERMANENS aims to standardize care, enhance clinician confidence, improve patient satisfaction, and increase treatment compliance. The routine integration of CDSS for self-harm risk assessment within healthcare systems holds significant potential in effectively reducing suicide mortality rates by facilitating personalized and timely delivery of effective interventions on a large scale for individuals at risk of suicide.

Machine learning and the prediction of suicide in psychiatric populations: a systematic review

Machine learning (ML) has emerged as a promising tool to enhance suicidal prediction. However, as many large-sample studies mixed psychiatric and non-psychiatric populations, a formal psychiatric diagnosis emerged as a strong predictor of suicidal risk, overshadowing more subtle risk factors specific to distinct populations. To overcome this limitation, we conducted a systematic review of ML studies evaluating suicidal behaviors exclusively in psychiatric clinical populations. A systematic literature search was performed from inception through November 17, 2022 on PubMed, EMBASE, and Scopus following the PRISMA guidelines. Original research using ML techniques to assess the risk of suicide or predict suicide attempts in the psychiatric population were included. An assessment for bias risk was performed using the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guidelines. About 1032 studies were retrieved, and 81 satisfied the inclusion criteria and were included for qualitative synthesis. Clinical and demographic features were the most frequently employed and random forest, support vector machine, and convolutional neural network performed better in terms of accuracy than other algorithms when directly compared. Despite heterogeneity in procedures, most studies reported an accuracy of 70% or greater based on features such as previous attempts, severity of the disorder, and pharmacological treatments. Although the evidence reported is promising, ML algorithms for suicidal prediction still present limitations, including the lack of neurobiological and imaging data and the lack of external validation samples. Overcoming these issues may lead to the development of models to adopt in clinical practice. Further research is warranted to boost a field that holds the potential to critically impact suicide mortality.

The use of machine learning on administrative and survey data to predict suicidal thoughts and behaviors: a systematic review

Background

Machine learning is a promising tool in the area of suicide prevention due to its ability to combine the effects of multiple risk factors and complex interactions. The power of machine learning has led to an influx of studies on suicide prediction, as well as a few recent reviews. Our study distinguished between data sources and reported the most important predictors of suicide outcomes identified in the literature.

Objective

Our study aimed to identify studies that applied machine learning techniques to administrative and survey data, summarize performance metrics reported in those studies, and enumerate the important risk factors of suicidal thoughts and behaviors identified.

Methods

A systematic literature search of PubMed, Medline, Embase, PsycINFO, Web of Science, Cumulative Index to Nursing and Allied Health Literature (CINAHL), and Allied and Complementary Medicine Database (AMED) to identify all studies that have used machine learning to predict suicidal thoughts and behaviors using administrative and survey data was performed. The search was conducted for articles published between January 1, 2019 and May 11, 2022. In addition, all articles identified in three recently published systematic reviews (the last of which included studies up until January 1, 2019) were retained if they met our inclusion criteria. The predictive power of machine learning methods in predicting suicidal thoughts and behaviors was explored using box plots to summarize the distribution of the area under the receiver operating characteristic curve (AUC) values by machine learning method and suicide outcome (i.e., suicidal thoughts, suicide attempt, and death by suicide). Mean AUCs with 95% confidence intervals (CIs) were computed for each suicide outcome by study design, data source, total sample size, sample size of cases, and machine learning methods employed. The most important risk factors were listed.

Results

The search strategy identified 2,200 unique records, of which 104 articles met the inclusion criteria. Machine learning algorithms achieved good prediction of suicidal thoughts and behaviors (i.e., an AUC between 0.80 and 0.89); however, their predictive power appears to differ across suicide outcomes. The boosting algorithms achieved good prediction of suicidal thoughts, death by suicide, and all suicide outcomes combined, while neural network algorithms achieved good prediction of suicide attempts. The risk factors for suicidal thoughts and behaviors differed depending on the data source and the population under study.

Conclusion

The predictive utility of machine learning for suicidal thoughts and behaviors largely depends on the approach used. The findings of the current review should prove helpful in preparing future machine learning models using administrative and survey data.

Systematic review registration

https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022333454 identifier CRD42022333454.

Prediction models of suicide and non-fatal suicide attempt after discharge from a psychiatric inpatient stay: A machine learning approach on nationwide Danish registers

INTRODUCTION

To develop machine learning models capable of predicting suicide and non-fatal suicide attempt as separate outcomes in the first 30 days after discharge from a psychiatric inpatient stay.

METHODS

Prospective cohort study using nationwide Danish registry data. We included individuals who were 18 years or older, and all discharges from psychiatric hospitalizations in Denmark from 1995 to 2018. We trained predictive models using 10-fold cross validation on 80% of the data and did testing on the remaining 20%.

RESULTS

The best model for predicting non-fatal suicide attempt was an ensemble of predictions from gradient boosting (XGBoost) and categorical boosting (catBoost). The ROC-AUC for predicting suicide attempt was 0.85 (95% CI: 0.84-0.85). At a risk threshold of 4.36%, positive predictive value (PPV) was 11.0% and sensitivity was 47.2%. The best model for predicting suicide was an ensemble of predictions from random forest, XGBoost and catBoost. For suicide, the ROC-AUC was 0.71 (95% CI: 0.70-0.73). At a risk threshold of 0.15%, PPV was 0.34% and sensitivity was 56.0%. The most contributing predictors differed when predicting suicide and suicide attempt, indicating that separate models are needed. The ensemble model was fair across sex and age, and more so than the penalized logistic regression model.

CONCLUSIONS

We achieved good performance for predicting suicide attempts and demonstrated a clinical application of ensemble models. Our results indicate a difference in predictive performance for models predicting suicide and suicide attempt, respectively. Thus, we recommend that suicide and suicide attempt are treated as two separate endpoints, in particular for clinical application. We demonstrated that the ensemble model is fairer across sex and age compared with a penalized logistic regression, and therefore we recommend the use of well-tested ensembles despite a more complex explainability.

Meta-analysis of clinical risk factors for suicide among people presenting to emergency departments and general hospitals with suicidal thoughts and behaviours

BACKGROUND

Suicidal thoughts and behaviours (STB) are a common reason for presentation to emergency departments and general hospitals. A meta-analysis of the strength of clinical risk factors for subsequent suicide might aid understanding of suicidal behaviour and help suicide prevention.

METHODS

We conducted a meta-analysis of cohort and controlled studies on clinical risk factors and later suicide among people presenting to emergency departments and general hospitals with STB. Data were extracted from papers meeting inclusion criteria, published in Medline, PsycINFO, and Embase between 1 January 1960 and 10 October 2022 using papers located with the search terms ((suicide*).m_titl AND (emergency* OR accident and emergency OR casualty OR general hospital OR toxicology service).mp) or were indexed in PubMed and had titles located with the search terms (suicide* OR self-harm OR self-harm OR self-injury OR self-injury OR self-poisoning OR self-poisoning OR overdose OR para-suicide OR parasuicide [title/abstract]) AND (Emergency department OR emergency room OR Casualty OR general hospital OR toxicology OR accident and emergency [all fields]). Data about the association between clinical risk factors and suicide extracted from three or more studies were included in a random-effects meta-analysis of the odds of later death by suicide. The study was registered in PROSPERO and conducted according to MOOSE and PRISMA guidelines.

RESULTS

Seventy-five studies reported on 741,624 people, of which 19,649 died by suicide (2.65%). Male sex (odds ratio (OR) = 1.99) and age (OR = 2.01) were the most consistently reported risk factors. The strongest associations with subsequent death by suicide related to violent self-harm methods at the hospital presentation, including: unspecified violent method (OR = 4.97), any violent method (OR = 4.57) and the specific violent methods of drowning (OR = 4.32), hanging (OR = 4.26), and use of firearms (OR = 10.08). Patients categorised as higher risk using suicide prediction scales or any other method that combined risk factors had moderately increased odds of suicide (OR = 2.58). Younger age, Black and Hispanic ethnicity, overdose, a diagnosis of adjustment disorder, and the absence of any psychiatric diagnosis were protective against suicide.

CONCLUSIONS

Most risk factors for suicide among people who have presented with STB are not strongly associated with later suicide. The strongest risk factors relate to self-harm methods. In the absence of clear indicators of future suicide, all people presenting with suicidality warrant a thorough assessment of their needs, and further research is needed before we can meaningfully categorise people with STB according to suicide risk.

Building an Integrated Data Infrastructure to Examine the Spectrum of Suicide Risk Factors in Philadelphia Medicaid

While there are many data-driven approaches to identifying individuals at risk of suicide, they tend to focus on clinical risk factors, such as previous psychiatric hospitalizations, and rarely include risk factors that occur in nonclinical settings, such as jails or emergency shelters. A better understanding of system-level encounters by individuals at risk of suicide could help inform suicide prevention efforts. In Philadelphia, we built a community-level data infrastructure that encompassed suicide death records, behavioral health claims, incarceration episodes, emergency housing episodes, and involuntary commitment petitions to examine a broader spectrum of suicide risk factors. Here, we describe the development of the data infrastructure, present key trends in suicide deaths in Philadelphia, and, for the Medicaid-eligible population, determine whether suicide decedents were more likely to interact with the behavioral health, carceral, and housing service systems compared to Medicaid-eligible Philadelphians who did not die by suicide. Between 2003 and 2018, there was an increase in the number of annual suicide deaths among Medicaid-eligible individuals, in part due to changes in Medicaid eligibility. There were disproportionately more suicide deaths among Black and Hispanic individuals who were Medicaid-eligible, who were younger on average, compared to suicide decedents who were never Medicaid-eligible. However, when we accounted for the racial and ethnic composition of the Medicaid population at large, we found that White individuals were four times as likely to die by suicide, while Asian, Black, Hispanic, and individuals of other races were less likely to die by suicide. Overall, 58% of individuals who were Medicaid-eligible and died by suicide had at least one Medicaid-funded behavioral health claim, 10% had at least one emergency housing episode, 25% had at least one incarceration episode, and 22% had at least one involuntary commitment. By developing a data infrastructure that can incorporate a broader spectrum of risk factors for suicide, we demonstrate how communities can harness administrative data to inform suicide prevention efforts. Our findings point to the need for suicide prevention in nonclinical settings such as jails and emergency shelters, and demonstrate important trends in suicide deaths in the Medicaid population.

Machine learning-based prediction for self-harm and suicide attempts in adolescents

This study aimed to use machine learning (ML) models to predict the risk of self-harm and suicide attempts in adolescents. We conducted secondary analysis of cross-sectional data from the Longitudinal Study of Australian Children dataset. Several key variables at the age of 14-15 years were used to predict self-harm or suicide attempt at 16-17 years. Random forest classification models were used to select the optimal subset of predictors and subsequently make predictions. Among 2809 participants, 296 (10.54%) reported an act of self-harm and 145 (5.16%) reported attempting suicide at least once in the past 12 months. The area under the receiver operating curve was fair for self-harm (0.7397) and suicide attempt (0.7220), which outperformed the prediction strategy solely based on prior suicide or self-harm attempt (AUC: 0.6). The most important factors identified were similar, and included depressed feelings, strengths and difficulties questionnaire scores, perceptions of self, and school- and parent-related factors. The random forest classification algorithm, an ML technique, can effectively select the optimal subset of predictors from hundreds of variables to forecast the risks of suicide and self-harm among adolescents. Further research is needed to validate the utility and scalability of ML techniques in mental health research.

Implementing Evidence-Based Suicide Prevention Strategies for Greatest Impact

Suicide remains a leading cause of death in the United States and globally. In this review, epidemiological trends in mortality and suicide risk are presented, with consideration given to the impact of the COVID-19 pandemic. A public health model of suicide prevention with a community and clinical framework, along with advances in scientific discovery, offer new solutions that await widespread implementation. Actionable interventions with evidence for reducing risk for suicidal behavior are presented, including universal and targeted strategies at community, public policy, and clinical levels. Clinical interventions include screening and risk assessment; brief interventions (e.g., safety planning, education, and lethal means counseling) that can be done in primary care, emergency, and behavioral health settings; psychotherapies (cognitive-behavioral, dialectical behavior, mentalization therapy); pharmacotherapy; and systemwide procedures for health care organizations (training, policies, workflow, surveillance of suicide indicators, use of health records for screening, care steps). Suicide prevention strategies must be prioritized and implemented at scale for greatest impact.

The performance of machine learning models in predicting suicidal ideation, attempts, and deaths: A meta-analysis and systematic review

Research has posited that machine learning could improve suicide risk prediction models, which have traditionally performed poorly. This systematic review and meta-analysis evaluated the performance of machine learning models in predicting longitudinal outcomes of suicide-related outcomes of ideation, attempt, and death and examines outcome, data, and model types as potential covariates of model performance. Studies were extracted from PubMed, Web of Science, Embase, and PsycINFO. A bivariate mixed effects meta-analysis and meta-regression analyses were performed for studies using machine learning to predict future events of suicidal ideation, attempts, and/or deaths. Risk of bias was assessed for each study using an adaptation of the Prediction model Risk Of Bias Assessment Tool. Narrative review included 56 studies, and analyses examined 54 models from 35 studies. The models achieved a very good pooled AUC of 0.86, sensitivity of 0.66 (95% CI [0.60, 0.72)], and specificity of 0.87 (95% CI [0.84, 0.90]). Pooled AUCs for ideation, attempt, and death were similar at 0.88, 0.87, and 0.84 respectively. Model performance was highly varied; however, meta-regressions did not provide evidence that performance varied by outcome, data, or model types. Findings suggest that machine learning has the potential to improve suicide risk detection, with pooled estimates of machine learning performance comparing favourably to performance of traditional suicide prediction models. However, more studies with lower risk of bias are necessary to improve the application of machine learning in suicidology.

Structured data vs. unstructured data in machine learning prediction models for suicidal behaviors: A systematic review and meta-analysis

Suicide remains a leading cause of preventable death worldwide, despite advances in research and decreases in mental health stigma through government health campaigns. Machine learning (ML), a type of artificial intelligence (AI), is the use of algorithms to simulate and imitate human cognition. Given the lack of improvement in clinician-based suicide prediction over time, advancements in technology have allowed for novel approaches to predicting suicide risk. This systematic review and meta-analysis aimed to synthesize current research regarding data sources in ML prediction of suicide risk, incorporating and comparing outcomes between structured data (human interpretable such as psychometric instruments) and unstructured data (only machine interpretable such as electronic health records). Online databases and gray literature were searched for studies relating to ML and suicide risk prediction. There were 31 eligible studies. The outcome for all studies combined was AUC = 0.860, structured data showed AUC = 0.873, and unstructured data was calculated at AUC = 0.866. There was substantial heterogeneity between the studies, the sources of which were unable to be defined. The studies showed good accuracy levels in the prediction of suicide risk behavior overall. Structured data and unstructured data also showed similar outcome accuracy according to meta-analysis, despite different volumes and types of input data.

Use of Artificial Intelligence-Based Strategies for Assessing Suicidal Behavior and Mental Illness: A Literature Review

Mental illness leading to suicide attempts is prevalent in a large portion of the population especially in low and middle-income nations. There remains a significant social stigma associated with mental illness that can lead to stigmatization of patients. Hence, patients are reluctant to communicate their problems to health care providers. Physicians have difficulty in timely identification of patients at risk for suicide. Novel and rigorously designed strategies are needed to determine the population at risk for suicide. This would be the first step in overcoming the multitude of barriers in the management of mental illness. Clinical tools and the use of electronic medical records (EMR) are time intensive. Recently, several artificial intelligence (AI)-based predictive technologies have gained momentum. The aim of this review is to summarize the recent advances in this landscape.

The importance of suicide risk assessment in patients affected by neurofibromatosis

OBJECTIVE

Neurofibromatosis 1 (NF1) is a chronic medical disease that often presents with psychiatric disorders. We investigated suicidal ideation in NF1 patients compared to healthy controls. We also evaluated whether hopelessness, depressive symptoms and perceived disability may mediate suicidal ideation in patients with NF1.

METHODS

We enrolled 60 patients with NF1 and 50 healthy controls with no history of NF1. Patients underwent a full psychiatric evaluation. Psychiatric diagnosis was made according to Diagnostic and Statistical Manual of Mental Disorders, fifth edition (DSM-5) criteria. Patients and controls underwent a series of psychometric measures, namely the Columbia Suicide Severity Rating Scale, the Beck Hopelessness Scale, the Italian Perceived Disability Scale and the Beck Depression Inventory.

RESULTS

Suicidal ideation was significantly higher in patients with NF1 (45%) than in controls (10%). Patients also presented more severe perceived disability and hopelessness and more frequent psychiatric disorders than controls. Multivariable logistic regression analysis showed that perceived disability was independently associated with the presence of suicidal ideation in patients with NF1.

CONCLUSIONS

In conclusion, our results showed that suicidal ideation was present in almost half of patients with NF1, suggesting the importance of suicide assessment in these patients.Key pointsPatients with NF1 have an increased suicide ideation when compared to healthy controlsIncreased suicidal ideation correlates with perceived disability, but not with the presence of psychiatric disordersAssessment of suicidal ideation should be performed in patients with NF1.