Validation of an Electronic Health Record-Based Suicide Risk Prediction Modeling Approach Across Multiple Health Care Systems

Suicide Death Prediction Using the Maryland Suicide Data Warehouse: A Sensitivity Analysis

OBJECTIVE

Nearly 50,000 Americans die each year from suicide, despite suicide death being a rare event in the context of health risk assessment and modeling. Prior research has underscored the need for contextualizing suicide risk models in terms of their potential uses and generalizability. This sensitivity analysis makes use of the Maryland Suicide Data Warehouse (MSDW) and illustrates how results inform clinical decision support.

METHOD

A cohort of 1 million living control patients were extracted from the MSDW in addition to 1,667 patients who had died by suicide between the years 2016 and 2019 according to the Maryland Office of the Medical Examiner (OCME). Data were extracted and aggregated as part of a 4-year retrospective design. Binary logistic and two penalized regression models were deployed in a repeated fivefold cross-validation. Model performances were evaluated using sensitivity, positive predictive value (PPV), and F1, and model coefficients were ranked according to coefficient size.

RESULTS

Several features were significantly associated with patients having died by suicide, including male sex, depressive and anxiety disorder diagnoses, social needs, and prior suicidal ideation and suicide attempt. Cross-validated binary logistic regression outperformed either ridge or LASSO (least absolute shrinkage and selection operator) models but generally achieved low-to-moderate PPV and sensitivity across most thresholds and a peak F1 of 0.323.

CONCLUSIONS

Suicide death prediction is constrained by the context of use, which determines the best balance of precision and recall. Predictive models must be evaluated close to the level of intervention. They may not hold up to different needs at different levels of care.

Absence of nonfatal suicidal behavior preceding suicide death reveals differences in clinical risks

Nonfatal suicidality is the most robust predictor of suicide death. However, only ~10% of those who survive an attempt go on to die by suicide. Moreover, ~50% of suicide deaths occur in the absence of prior known attempts, suggesting risks other than nonfatal suicide attempt need to be identified. We studied data from 4,000 population-ascertained suicide deaths and 26,191 population controls to improve understanding of risks leading to suicide death. This study included 2,253 suicide deaths and 3,375 controls with evidence of nonfatal suicidality (SUI_SI/SB and CTL_SI/SB) from diagnostic codes and natural language processing of electronic health records notes. Characteristics of these groups were compared to 1,669 suicides with no prior nonfatal SI/SB (SUI_None) and 22,816 controls with no lifetime suicidality (CTL_None). The SUI_None and CTL_None groups had fewer diagnoses and were older than SUI_SI/SB and CTL_SI/SB. Mental health diagnoses were far less common in both the SUI_None and CTL_None groups; mental health problems were less associated with suicide death than with presence of SI/SB. Physical health diagnoses were conversely more often associated with risk of suicide death than with presence of SI/SB. Pending replication, results indicate highly significant clinical differences among suicide deaths with versus without prior nonfatal SI/SB.

Role of machine learning algorithms in suicide risk prediction: a systematic review-meta analysis of clinical studies

OBJECTIVE

Suicide is a complex and multifactorial public health problem. Understanding and addressing the various factors associated with suicide is crucial for prevention and intervention efforts. Machine learning (ML) could enhance the prediction of suicide attempts.

METHOD

A systematic review was performed using PubMed, Scopus, Web of Science and SID databases. We aim to evaluate the performance of ML algorithms and summarize their effects, gather relevant and reliable information to synthesize existing evidence, identify knowledge gaps, and provide a comprehensive list of the suicide risk factors using mixed method approach.

RESULTS

Forty-one studies published between 2011 and 2022, which matched inclusion criteria, were chosen as suitable. We included studies aimed at predicting the suicide risk by machine learning algorithms except natural language processing (NLP) and image processing. The neural network (NN) algorithm exhibited the lowest accuracy at 0.70, whereas the random forest demonstrated the highest accuracy, reaching 0.94. The study assessed the COX and random forest models and observed a minimum area under the curve (AUC) value of 0.54. In contrast, the XGBoost classifier yielded the highest AUC value, reaching 0.97. These specific AUC values emphasize the algorithm-specific performance in capturing the trade-off between sensitivity and specificity for suicide risk prediction. Furthermore, our investigation identified several common suicide risk factors, including age, gender, substance abuse, depression, anxiety, alcohol consumption, marital status, income, education, and occupation. This comprehensive analysis contributes valuable insights into the multifaceted nature of suicide risk, providing a foundation for targeted preventive strategies and intervention efforts.

CONCLUSIONS

The effectiveness of ML algorithms and their application in predicting suicide risk has been controversial. There is a need for more studies on these algorithms in clinical settings, and the related ethical concerns require further clarification.

Validation of an ICD-Code-Based Case Definition for Psychotic Illness Across Three Health Systems

BACKGROUND AND HYPOTHESIS

Psychosis-associated diagnostic codes are increasingly being utilized as case definitions for electronic health record (EHR)-based algorithms to predict and detect psychosis. However, data on the validity of psychosis-related diagnostic codes is limited. We evaluated the positive predictive value (PPV) of International Classification of Diseases (ICD) codes for psychosis.

STUDY DESIGN

Using EHRs at 3 health systems, ICD codes comprising primary psychotic disorders and mood disorders with psychosis were grouped into 5 higher-order groups. 1133 records were sampled for chart review using the full EHR. PPVs (the probability of chart-confirmed psychosis given ICD psychosis codes) were calculated across multiple treatment settings.

STUDY RESULTS

PPVs across all diagnostic groups and hospital systems exceeded 70%: Mass General Brigham 0.72 [95% CI 0.68-0.77], Boston Children's Hospital 0.80 [0.75-0.84], and Boston Medical Center 0.83 [0.79-0.86]. Schizoaffective disorder PPVs were consistently the highest across sites (0.80-0.92) and major depressive disorder with psychosis were the most variable (0.57-0.79). To determine if the first documented code captured first-episode psychosis (FEP), we excluded cases with prior chart evidence of a diagnosis of or treatment for a psychotic illness, yielding substantially lower PPVs (0.08-0.62).

CONCLUSIONS

We found that the first documented psychosis diagnostic code accurately captured true episodes of psychosis but was a poor index of FEP. These data have important implications for the case definitions used in the development of risk prediction models designed to predict or detect undiagnosed psychosis.

Developing a clinical decision support system software prototype that assists in the management of patients with self-harm in the emergency department: protocol of the PERMANENS project

BACKGROUND

Self-harm presents a significant public health challenge. Emergency departments (EDs) are crucial healthcare settings in managing self-harm, but clinician uncertainty in risk assessment may contribute to ineffective care. Clinical Decision Support Systems (CDSSs) show promise in enhancing care processes, but their effective implementation in self-harm management remains unexplored.

METHODS

PERMANENS comprises a combination of methodologies and study designs aimed at developing a CDSS prototype that assists clinicians in the personalized assessment and management of ED patients presenting with self-harm. Ensemble prediction models will be constructed by applying machine learning techniques on electronic registry data from four sites, i.e., Catalonia (Spain), Ireland, Norway, and Sweden. These models will predict key adverse outcomes including self-harm repetition, suicide, premature death, and lack of post-discharge care. Available registry data include routinely collected electronic health record data, mortality data, and administrative data, and will be harmonized using the OMOP Common Data Model, ensuring consistency in terminologies, vocabularies and coding schemes. A clinical knowledge base of effective suicide prevention interventions will be developed rooted in a systematic review of clinical practice guidelines, including quality assessment of guidelines using the AGREE II tool. The CDSS software prototype will include a backend that integrates the prediction models and the clinical knowledge base to enable accurate patient risk stratification and subsequent intervention allocation. The CDSS frontend will enable personalized risk assessment and will provide tailored treatment plans, following a tiered evidence-based approach. Implementation research will ensure the CDSS' practical functionality and feasibility, and will include periodic meetings with user-advisory groups, mixed-methods research to identify currently unmet needs in self-harm risk assessment, and small-scale usability testing of the CDSS prototype software.

DISCUSSION

Through the development of the proposed CDSS software prototype, PERMANENS aims to standardize care, enhance clinician confidence, improve patient satisfaction, and increase treatment compliance. The routine integration of CDSS for self-harm risk assessment within healthcare systems holds significant potential in effectively reducing suicide mortality rates by facilitating personalized and timely delivery of effective interventions on a large scale for individuals at risk of suicide.

Suicide Risk Screening for Head and Neck Cancer Patients: An Implementation Study

OBJECTIVES

 There is limited research on suicide risk screening (SRS) among head and neck cancer (HNC) patients, a population at increased risk for suicide. To address this gap, this single-site mixed methods study assessed oncology professionals' perspectives about the feasibility, acceptability, and appropriateness of an electronic SRS program that was implemented as a part of routine care for HNC patients.

METHODS

Staff who assisted with SRS implementation completed (e.g., nurses, medical assistants, advanced practice providers, physicians, social workers) a one-time survey (N = 29) and interview (N = 25). Quantitative outcomes were assessed using previously validated feasibility, acceptability, and appropriateness measures. Additional qualitative data were collected to provide context for interpreting the scores.

RESULTS

 Nurses and medical assistants, who were directly responsible for implementing SRS, reported low feasibility, acceptability, and appropriateness, compared with other team members (e.g., physicians, social workers, advanced practice providers). Team members identified potential improvements needed to optimize SRS, such as hiring additional staff, improving staff training, providing different modalities for screening completion among individuals with disabilities, and revising the patient-reported outcomes to improve suicide risk prediction.

CONCLUSION

 Staff perspectives about implementing SRS as a part of routine cancer care for HNC patients varied widely. Before screening can be implemented on a larger scale for HNC and other cancer patients, additional implementation strategies may be needed that optimize workflow and reduce staff burden, such as staff training, multiple modalities for completion, and refined tools for identifying which patients are at greatest risk for suicide.

Validation of an ICD-code-based case definition for psychotic illness across three health systems

Background and Hypothesis

Early detection of psychosis is critical for improving outcomes. Algorithms to predict or detect psychosis using electronic health record (EHR) data depend on the validity of the case definitions used, typically based on diagnostic codes. Data on the validity of psychosis-related diagnostic codes is limited. We evaluated the positive predictive value (PPV) of International Classification of Diseases (ICD) codes for psychosis.

Study Design

Using EHRs at three health systems, ICD codes comprising primary psychotic disorders and mood disorders with psychosis were grouped into five higher-order groups. 1,133 records were sampled for chart review using the full EHR. PPVs (the probability of chart-confirmed psychosis given ICD psychosis codes) were calculated across multiple treatment settings.

Study Results

PPVs across all diagnostic groups and hospital systems exceeded 70%: Massachusetts General Brigham 0.72 [95% CI 0.68-0.77], Boston Children's Hospital 0.80 [0.75-0.84], and Boston Medical Center 0.83 [0.79-0.86]. Schizoaffective disorder PPVs were consistently the highest across sites (0.80-0.92) and major depressive disorder with psychosis were the most variable (0.57-0.79). To determine if the first documented code captured first-episode psychosis (FEP), we excluded cases with prior chart evidence of a diagnosis of or treatment for a psychotic illness, yielding substantially lower PPVs (0.08-0.62).

Conclusions

We found that the first documented psychosis diagnostic code accurately captured true episodes of psychosis but was a poor index of FEP. These data have important implications for the development of risk prediction models designed to predict or detect undiagnosed psychosis.

Continuous-Time and Dynamic Suicide Attempt Risk Prediction with Neural Ordinary Differential Equations

Suicide is one of the leading causes of death in the US, and the number of attributable deaths continues to increase. Risk of suicide-related behaviors (SRBs) is dynamic, and SRBs can occur across a continuum of time and locations. However, current SRB risk assessment methods, whether conducted by clinicians or through machine learning models, treat SRB risk as static and are confined to specific times and locations, such as following a hospital visit. Such a paradigm is unrealistic as SRB risk fluctuates and creates time gaps in the availability of risk scores. Here, we develop two closely related model classes, Event-GRU-ODE and Event-GRU-Discretized, that can predict the dynamic risk of events as a continuous trajectory based on Neural ODEs, an advanced AI model class for time series prediction. As such, these models can estimate changes in risk across the continuum of future time points, even without new observations, and can update these estimations as new data becomes available. We train and validate these models for SRB prediction using a large electronic health records database. Both models demonstrated high discrimination performance for SRB prediction (e.g., AUROC > 0.92 in the full, general cohort), serving as an initial step toward developing novel and comprehensive suicide prevention strategies based on dynamic changes in risk.

Determining Distinct Suicide Attempts From Recurrent Electronic Health Record Codes: Classification Study

BACKGROUND

Prior suicide attempts are a relatively strong risk factor for future suicide attempts. There is growing interest in using longitudinal electronic health record (EHR) data to derive statistical risk prediction models for future suicide attempts and other suicidal behavior outcomes. However, model performance may be inflated by a largely unrecognized form of "data leakage" during model training: diagnostic codes for suicide attempt outcomes may refer to prior attempts that are also included in the model as predictors.

OBJECTIVE

We aimed to develop an automated rule for determining when documented suicide attempt diagnostic codes identify distinct suicide attempt events.

METHODS

From a large health care system's EHR, we randomly sampled suicide attempt codes for 300 patients with at least one pair of suicide attempt codes documented at least one but no more than 90 days apart. Supervised chart reviewers assigned the clinical settings (ie, emergency department [ED] versus non-ED), methods of suicide attempt, and intercode interval (number of days). The probability (or positive predictive value) that the second suicide attempt code in a given pair of codes referred to a distinct suicide attempt event from its preceding suicide attempt code was calculated by clinical setting, method, and intercode interval.

RESULTS

Of 1015 code pairs reviewed, 835 (82.3%) were nonindependent (ie, the 2 codes referred to the same suicide attempt event). When the second code in a pair was documented in a clinical setting other than the ED, it represented a distinct suicide attempt 3.3% of the time. The more time elapsed between codes, the more likely the second code in a pair referred to a distinct suicide attempt event from its preceding code. Code pairs in which the second suicide attempt code was assigned in an ED at least 5 days after its preceding suicide attempt code had a positive predictive value of 0.90.

CONCLUSIONS

EHR-based suicide risk prediction models that include International Classification of Diseases codes for prior suicide attempts as a predictor may be highly susceptible to bias due to data leakage in model training. We derived a simple rule to distinguish codes that reflect new, independent suicide attempts: suicide attempt codes documented in an ED setting at least 5 days after a preceding suicide attempt code can be confidently treated as new events in EHR-based suicide risk prediction models. This rule has the potential to minimize upward bias in model performance when prior suicide attempts are included as predictors in EHR-based suicide risk prediction models.

The role of substance use disorders in the transition from suicide attempt to suicide death: a record linkage study of a Swedish cohort

BACKGROUND

Suicidal behavior and substance use disorders (SUDs) are important public health concerns. Prior suicide attempts and SUDs are two of the most consistent predictors of suicide death, and clarifying the role of SUDs in the transition from suicide attempt to suicide death could inform prevention efforts.

METHODS

We used national Swedish registry data to identify individuals born 1960-1985, with an index suicide attempt in 1997-2017 (N = 74 873; 46.7% female). We assessed risk of suicide death as a function of registration for a range of individual SUDs. We further examined whether the impact of SUDs varied as a function of (i) aggregate genetic liability to suicidal behavior, or (ii) age at index suicide attempt.

RESULTS

In univariate models, risk of suicide death was higher among individuals with any SUD registration [hazard ratios (HRs) = 2.68-3.86]. In multivariate models, effects of specific SUDs were attenuated, but remained elevated for AUD (HR = 1.86 95% confidence intervals 1.68-2.05), opiates [HR = 1.58 (1.37-1.82)], sedatives [HR = 1.93 (1.70-2.18)], and multiple substances [HR = 2.09 (1.86-2.35)]. In secondary analyses, the effects of most, but not all, SUD were exacerbated by higher levels of genetic liability to suicide death, and among individuals who were younger at their index suicide attempt.

CONCLUSIONS

In the presence of a strong predictor of suicide death - a prior attempt - substantial predictive power is still attributable to SUDs. Individuals with SUDs may warrant additional suicide screening and prevention efforts, particularly in the context of a family history of suicidal behavior or early onset of suicide attempt.

Risk Factors for Suicide in Men

Suicide and the risk factors associated with it have been researched with increasing interest over the last 5 decades with respect to socioeconomic status, age, geographic location, and ethnic background. There has been less focus related to the risk factors specific to gender and how to incorporate clinical screening and interventions to reduce the mortality of suicide in males. With men accounting for a disproportionate number of deaths from suicide in the United States and worldwide, how gender could impact suicidal behavior and ideations remains a topic understudied and with great potential for significant improvement in clinical recognition and treatment.

Digital Application of Clinical Staging to Support Stratification in Youth Mental Health Services: Validity and Reliability Study

BACKGROUND

As the demand for youth mental health care continues to rise, managing wait times and reducing treatment delays are key challenges to delivering timely and quality care. Clinical staging is a heuristic model for youth mental health that can stratify care allocation according to individuals' risk of illness progression. The application of staging has been traditionally limited to trained clinicians yet leveraging digital technologies to apply clinical staging could increase the scalability and usability of this model in services.

OBJECTIVE

The aim of this study was to validate a digital algorithm to accurately differentiate young people at lower and higher risk of developing mental disorders.

METHODS

We conducted a study with a cohort comprising 131 young people, aged between 16 and 25 years, who presented to youth mental health services in Australia between November 2018 and March 2021. Expert psychiatrists independently assigned clinical stages (either stage 1a or stage 1b+), which were then compared to the digital algorithm's allocation based on a multidimensional self-report questionnaire.

RESULTS

Of the 131 participants, the mean age was 20.3 (SD 2.4) years, and 72% (94/131) of them were female. Ninety-one percent of clinical stage ratings were concordant between the digital algorithm and the experts' ratings, with a substantial interrater agreement (κ=0.67; P<.001). The algorithm demonstrated an accuracy of 91% (95% CI 86%-95%; P=.03), a sensitivity of 80%, a specificity of 93%, and an F1-score of 73%. Of the concordant ratings, 16 young people were allocated to stage 1a, while 103 were assigned to stage 1b+. Among the 12 discordant cases, the digital algorithm allocated a lower stage (stage 1a) to 8 participants compared to the experts. These individuals had significantly milder symptoms of depression (P<.001) and anxiety (P<.001) compared to those with concordant stage 1b+ ratings.

CONCLUSIONS

This novel digital algorithm is sufficiently robust to be used as an adjunctive decision support tool to stratify care and assist with demand management in youth mental health services. This work could transform care pathways and expedite care allocation for those in the early stages of common anxiety and depressive disorders. Between 11% and 27% of young people seeking care may benefit from low-intensity, self-directed, or brief interventions. Findings from this study suggest the possibility of redirecting clinical capacity to focus on individuals in stage 1b+ for further assessment and intervention.

A 50-state survey study of thoughts of suicide and social isolation among older adults in the United States

BACKGROUND

We aimed to characterize the prevalence of social disconnection and thoughts of suicide among older adults in the United States, and examine the association between them in a large naturalistic study.

METHODS

We analyzed data from 6 waves of a fifty-state non-probability survey among US adults conducted between February and December 2021. The internet-based survey collected the PHQ-9, as well as multiple measures of social connectedness. We applied multiple logistic regression to analyze the association between presence of thoughts of suicide and social disconnection. Exploratory analysis, using generalized random forests, examined heterogeneity of effects across sociodemographic groups.

RESULTS

Of 16,164 survey respondents age 65 and older, mean age was 70.9 (SD 5.0); the cohort was 61.4 % female and 29.6 % male; 2.0 % Asian, 6.7 % Black, 2.2 % Hispanic, and 86.8 % White. A total of 1144 (7.1 %) reported thoughts of suicide at least several days in the prior 2 week period. In models adjusted for sociodemographic features, households with 3 or more additional members (adjusted OR 1.73, 95 % CI 1.28-2.33) and lack of social supports, particularly emotional supports (adjusted OR 2.60, 95 % CI 2.09-3.23), were independently associated with greater likelihood of reporting such thoughts, as was greater reported loneliness (adjusted OR 1.75, 95 % CI 1.64-1.87). The effects of emotional support varied significantly across sociodemographic groups.

CONCLUSIONS

Thoughts of suicide are common among older adults in the US, and associated with lack of social support, but not with living alone.

TRIAL REGISTRATION

NA.

The use of advanced technology and statistical methods to predict and prevent suicide

In the past decade, two themes have emerged across suicide research. First, according to meta-analyses, the ability to predict and prevent suicidal thoughts and behaviours is weaker than would be expected for the size of the field. Second, review and commentary papers propose that technological and statistical methods (such as smartphones, wearables, digital phenotyping and machine learning) might become solutions to this problem. In this Review, we aim to strike a balance between the pessimistic picture presented by these meta-analyses and the optimistic picture presented by review and commentary papers about the promise of advanced technological and statistical methods to improve the ability to understand, predict and prevent suicide. We divide our discussion into two broad categories. First, we discuss the research aimed at assessment, with the goal of better understanding or more accurately predicting suicidal thoughts and behaviours. Second, we discuss the literature that focuses on prevention of suicidal thoughts and behaviours. Ecological momentary assessment, wearables and other technological and statistical advances hold great promise for predicting and preventing suicide, but there is much yet to do.

An efficient landmark model for prediction of suicide attempts in multiple clinical settings

Growing evidence has shown that applying machine learning models to large clinical data sources may exceed clinician performance in suicide risk stratification. However, many existing prediction models either suffer from "temporal bias" (a bias that stems from using case-control sampling) or require training on all available patient visit data. Here, we adopt a "landmark model" framework that aligns with clinical practice for prediction of suicide-related behaviors (SRBs) using a large electronic health record database. Using the landmark approach, we developed models for SRB prediction (regularized Cox regression and random survival forest) that establish a time-point (e.g., clinical visit) from which predictions are made over user-specified prediction windows using historical information up to that point. We applied this approach to cohorts from three clinical settings: general outpatient, psychiatric emergency department, and psychiatric inpatients, for varying prediction windows and lengths of historical data. Models achieved high discriminative performance (area under the Receiver Operating Characteristic curve 0.74-0.93 for the Cox model) across different prediction windows and settings, even with relatively short periods of historical data. In short, we developed accurate, dynamic SRB risk prediction models with the landmark approach that reduce bias and enhance the reliability and portability of suicide risk prediction models.

Shared genetic and environmental etiology between substance use disorders and suicidal behavior

BACKGROUND

Previous studies have demonstrated substantial associations between substance use disorders (SUD) and suicidal behavior. The current study empirically assesses the extent to which shared genetic and/or environmental factors contribute to associations between alcohol use disorders (AUD) or drug use disorders (DUD) and suicidal behavior, including attempts and death.

METHODS

The authors used Swedish national registry data, including medical, pharmacy, criminal, and death registrations, for a large cohort of twins, full siblings, and half siblings (N = 1 314 990) born 1960-1980 and followed through 2017. They conducted twin-sibling modeling of suicide attempt (SA) or suicide death (SD) with AUD and DUD to estimate genetic and environmental correlations between outcomes. Analyses were stratified by sex.

RESULTS

Genetic correlations between SA and SUD ranged from rA = 0.60-0.88; corresponding shared environmental correlations were rC = 0.42-0.89 but accounted for little overall variance; and unique environmental correlations were rE = 0.42-0.57. When replacing attempt with SD, genetic and shared environmental correlations with AUD and DUD were comparable (rA = 0.48-0.72, rC = 0.92-1.00), but were attenuated for unique environmental factors (rE = -0.01 to 0.31).

CONCLUSIONS

These findings indicate that shared genetic and unique environmental factors contribute to comorbidity of suicidal behavior and SUD, in conjunction with previously reported causal associations. Thus, each outcome should be considered an indicator of risk for the others. Opportunities for joint prevention and intervention, while limited by the polygenic nature of these outcomes, may be feasible considering moderate environmental correlations between SA and SUD.

Unsound sleep, wound-up mind: a longitudinal examination of acute suicidal affective disturbance features among an eating disorder sample

BACKGROUND

Suicide is one of the most commonly reported causes of death in individuals with eating disorders. However, the mechanisms underlying the suicide and disordered eating link are largely unknown, and current assessments are still unable to accurately predict future suicidal thoughts and behaviors. The purpose of this study is to test the utility of two promising proximal risk factors, sleep quality and agitation, in predicting suicidal ideation in a sample of individuals with elevated suicidal thoughts and behaviors, namely those with eating disorders.

METHODS

Women (N = 97) receiving treatment at an eating disorder treatment center completed weekly questionnaires assessing suicidal ideation, agitation, and sleep. General linear mixed models examined whether agitation and/or sleep quality were concurrently or prospectively associated with suicidal ideation across 12 weeks of treatment.

RESULTS

There was a significant interaction between within-person agitation and sleep quality on suicidal ideation [B(s.e.) = -0.02(0.01), p < 0.05], such that on weeks when an individual experienced both higher than their average agitation and lower than their average sleep quality, they also experienced their highest levels of suicidal ideation. However, neither agitation nor sleep quality prospectively predicted suicidal ideation.

CONCLUSIONS

This study was the first to examine dynamic associations between interpersonal constructs and suicidal ideation in individuals with eating disorders. Results suggest that ongoing assessment for overarousal symptoms, such as agitation and poor sleep quality, in individuals with eating disorders may be warranted in order to manage suicidal ideation among this vulnerable population.

Improving risk prediction for target subpopulations: Predicting suicidal behaviors among multiple sclerosis patients

Several recent studies have applied machine learning techniques to develop risk algorithms that predict subsequent suicidal behavior based on electronic health record data. In this study we used a retrospective cohort study design to test whether developing more tailored predictive models-within specific subpopulations of patients-would improve predictive accuracy. A retrospective cohort of 15,117 patients diagnosed with multiple sclerosis (MS), a diagnosis associated with increased risk of suicidal behavior, was used. The cohort was randomly divided into equal sized training and validation sets. Overall, suicidal behavior was identified among 191 (1.3%) of the patients with MS. A Naïve Bayes Classifier model was trained on the training set to predict future suicidal behavior. With 90% specificity, the model detected 37% of subjects who later demonstrated suicidal behavior, on average 4.6 years before the first suicide attempt. The performance of a model trained only on MS patients was better at predicting suicide in MS patients than that a model trained on a general patient sample of a similar size (AUC of 0.77 vs. 0.66). Unique risk factors for suicidal behavior among patients with MS included pain-related codes, gastroenteritis and colitis, and history of smoking. Future studies are needed to further test the value of developing population-specific risk models.

Using Machine Learning to Examine Suicidal Ideation After Traumatic Brain Injury: A Traumatic Brain Injury Model Systems National Database Study

OBJECTIVE

The aim of the study was to predict suicidal ideation 1 yr after moderate to severe traumatic brain injury.

DESIGN

This study used a cross-sectional design with data collected through the prospective, longitudinal Traumatic Brain Injury Model Systems network at hospitalization and 1 yr after injury. Participants who completed the Patient Health Questionnaire-9 suicide item at year 1 follow-up ( N = 4328) were included.

RESULTS

A gradient boosting machine algorithm demonstrated the best performance in predicting suicidal ideation 1 yr after traumatic brain injury. Predictors were Patient Health Questionnaire-9 items (except suicidality), Generalized Anxiety Disorder-7 items, and a measure of heavy drinking. Results of the 10-fold cross-validation gradient boosting machine analysis indicated excellent classification performance with an area under the curve of 0.882. Sensitivity was 0.85 and specificity was 0.77. Accuracy was 0.78 (95% confidence interval, 0.77-0.79). Feature importance analyses revealed that depressed mood and guilt were the most important predictors of suicidal ideation, followed by anhedonia, concentration difficulties, and psychomotor disturbance.

CONCLUSIONS

Overall, depression symptoms were most predictive of suicidal ideation. Despite the limited clinical impact of the present findings, machine learning has potential to improve prediction of suicidal behavior, leveraging electronic health record data, to identify individuals at greatest risk, thereby facilitating intervention and optimization of long-term outcomes after traumatic brain injury.

Use of an Agitation Measure to Screen for Suicide and Self-Harm Risk Among Emergency Department Patients

BACKGROUND

Suicidality alone is insensitive to suicide risk among emergency department (ED) patients.

OBJECTIVE

We describe the performance of adding an objective assessment of agitation to a suicide screening instrument for predicting suicide and self-harm after an ED encounter.

METHODS

We tested the performance of a novel screener combining the presence of suicidality or agitation for predicting suicide within 90 days or a repeat ED visit for self-harm within 30 days using retrospective data from all patients seen in an urban safety net ED over 27 months. Patients were assessed for suicidality using the Columbia-Suicide Severity Rating Scale-Clinical Practice Screener and for agitation using either the Behavioral Activity Rating Scale or Richmond Agitation Sedation Scale. We hypothesized that a screener based on the presence of either suicidality or agitation would be more sensitive to suicide risk than the Columbia-Suicide Severity Rating Scale-Clinical Practice Screener alone. The screener's performance is described, and multivariable regression evaluates the correlations between screening and outcomes.

RESULTS

The sample comprised 16,467 patients seen in the ED who had available suicide screening and agitation data. Thirteen patients (0.08%) died by suicide within 90 days after ED discharge. The sensitivity and specificity of the screener combining suicidality and agitation for predicting suicide was 0.69 (95% confidence interval, 0.44-0.94) and 0.74 (0.44-0.94), respectively. The sensitivity and specificity for agitation combined with positive suicide screening for self-harm within 30 days were 0.95 (0.89-1.00) and 0.73 (0.73-0.74). For both outcomes, augmenting the Columbia-Suicide Severity Rating Scale-Clinical Practice Screener with a measure of agitation improved both sensitivity and overall performance compared to historical performance of the Columbia-Suicide Severity Rating Scale-Clinical Practice Screener alone.

CONCLUSIONS

Combining a brief objective measure of agitation with a common suicide screening instrument improved sensitivity and predictive performance for suicide and self-harm risk after ED discharge. These findings speak to the importance of assessing agitation not only for imminent safety risk during the patient encounter but also for reducing the likelihood of future adverse events. This work can improve the detection and management of suicide risk in emergency settings.

Clinical implementation of suicide risk prediction models in healthcare: a qualitative study

BACKGROUND

Suicide risk prediction models derived from electronic health records (EHR) are a novel innovation in suicide prevention but there is little evidence to guide their implementation.

METHODS

In this qualitative study, 30 clinicians and 10 health care administrators were interviewed from one health system anticipating implementation of an automated EHR-derived suicide risk prediction model and two health systems piloting different implementation approaches. Site-tailored interview guides focused on respondents' expectations for and experiences with suicide risk prediction models in clinical practice, and suggestions for improving implementation. Interview prompts and content analysis were guided by Consolidated Framework for Implementation Research (CFIR) constructs.

RESULTS

Administrators and clinicians found use of the suicide risk prediction model and the two implementation approaches acceptable. Clinicians desired opportunities for early buy-in, implementation decision-making, and feedback. They wanted to better understand how this manner of risk identification enhanced existing suicide prevention efforts. They also wanted additional training to understand how the model determined risk, particularly after patients they expected to see identified by the model were not flagged at-risk and patients they did not expect to see identified were. Clinicians were concerned about having enough suicide prevention resources for potentially increased demand and about their personal liability; they wanted clear procedures for situations when they could not reach patients or when patients remained at-risk over a sustained period. Suggestions for making risk model workflows more efficient and less burdensome included consolidating suicide risk information in a dedicated module in the EHR and populating risk assessment scores and text in clinical notes.

CONCLUSION

Health systems considering suicide risk model implementation should engage clinicians early in the process to ensure they understand how risk models estimate risk and add value to existing workflows, clarify clinician role expectations, and summarize risk information in a convenient place in the EHR to support high-quality patient care.

Predicting Poor Outcomes Among Individuals Seeking Care for Major Depressive Disorder

Objective

To develop and validate algorithms to identify individuals with major depressive disorder (MDD) at elevated risk for suicidality or for an acute care event.

Methods

We conducted a retrospective cohort analysis among adults with MDD diagnosed between January 1, 2018 and February 28, 2019. Generalized estimating equation models were developed to predict emergency department (ED) visit, inpatient hospitalization, acute care visit (ED or inpatient), partial-day hospitalization, and suicidality in the year following diagnosis. Outcomes (per 1000 patients per month, PkPPM) were categorized as all-cause, psychiatric, or MDD-specific and combined into composite measures. Predictors included demographics, medical and pharmacy utilization, social determinants of health, and comorbid diagnoses as well as features indicative of clinically relevant changes in psychiatric health. Models were trained on data from 1.7M individuals, with sensitivity, positive predictive value, and area-under-the-curve (AUC) derived from a validation dataset of 0.7M.

Results

Event rates were 124.0 PkPPM (any outcome), 21.2 PkPPM (psychiatric utilization), and 7.6 PkPPM (suicidality). Among the composite models, the model predicting suicidality had the highest AUC (0.916) followed by any psychiatric acute care visit (0.891) and all-cause ED visit (0.790). Event-specific models all achieved an AUC >0.87, with the highest AUC noted for partial-day hospitalization (AUC = 0.938). Select predictors of all three outcomes included younger age, Medicaid insurance, past psychiatric ED visits, past suicidal ideation, and alcohol use disorder diagnoses, among others.

Conclusions

Analytical models derived from clinically-relevant features identify individuals with MDD at risk for poor outcomes and can be a practical tool for health care organizations to divert high-risk populations into comprehensive care models.

The performance of machine learning models in predicting suicidal ideation, attempts, and deaths: A meta-analysis and systematic review

Research has posited that machine learning could improve suicide risk prediction models, which have traditionally performed poorly. This systematic review and meta-analysis evaluated the performance of machine learning models in predicting longitudinal outcomes of suicide-related outcomes of ideation, attempt, and death and examines outcome, data, and model types as potential covariates of model performance. Studies were extracted from PubMed, Web of Science, Embase, and PsycINFO. A bivariate mixed effects meta-analysis and meta-regression analyses were performed for studies using machine learning to predict future events of suicidal ideation, attempts, and/or deaths. Risk of bias was assessed for each study using an adaptation of the Prediction model Risk Of Bias Assessment Tool. Narrative review included 56 studies, and analyses examined 54 models from 35 studies. The models achieved a very good pooled AUC of 0.86, sensitivity of 0.66 (95% CI [0.60, 0.72)], and specificity of 0.87 (95% CI [0.84, 0.90]). Pooled AUCs for ideation, attempt, and death were similar at 0.88, 0.87, and 0.84 respectively. Model performance was highly varied; however, meta-regressions did not provide evidence that performance varied by outcome, data, or model types. Findings suggest that machine learning has the potential to improve suicide risk detection, with pooled estimates of machine learning performance comparing favourably to performance of traditional suicide prediction models. However, more studies with lower risk of bias are necessary to improve the application of machine learning in suicidology.

Identifying Youth at Risk for Suicidal Thoughts and Behaviors Using the "p" factor in Primary Care: An Exploratory Study

Suicide is a major, preventable public health problem. The general factor of psychopathology ("p" factor) might help improve detection and prediction of individuals at risk for suicide. This cross-sectional proof-of-concept study tests whether the p-factor score is associated with suicidal thoughts and behaviors (STB) better than a depression scale alone. Youth (N = 841; mean age 18.02, SD = 3.36) in primary care were universally screened using the Behavioral Health Screen (BHS). Factor analysis and ROC results showed the BHS assesses the p-factor, and the p-factor score demonstrates higher classification accuracy of several types of STB than a depression scale. The p-factor could help clinicians in the identification of youths with STB.

Structured data vs. unstructured data in machine learning prediction models for suicidal behaviors: A systematic review and meta-analysis

Suicide remains a leading cause of preventable death worldwide, despite advances in research and decreases in mental health stigma through government health campaigns. Machine learning (ML), a type of artificial intelligence (AI), is the use of algorithms to simulate and imitate human cognition. Given the lack of improvement in clinician-based suicide prediction over time, advancements in technology have allowed for novel approaches to predicting suicide risk. This systematic review and meta-analysis aimed to synthesize current research regarding data sources in ML prediction of suicide risk, incorporating and comparing outcomes between structured data (human interpretable such as psychometric instruments) and unstructured data (only machine interpretable such as electronic health records). Online databases and gray literature were searched for studies relating to ML and suicide risk prediction. There were 31 eligible studies. The outcome for all studies combined was AUC = 0.860, structured data showed AUC = 0.873, and unstructured data was calculated at AUC = 0.866. There was substantial heterogeneity between the studies, the sources of which were unable to be defined. The studies showed good accuracy levels in the prediction of suicide risk behavior overall. Structured data and unstructured data also showed similar outcome accuracy according to meta-analysis, despite different volumes and types of input data.

Patient expectations of and experiences with a suicide risk identification algorithm in clinical practice

BACKGROUND

Suicide risk prediction models derived from electronic health records (EHR) and insurance claims are a novel innovation in suicide prevention but patient perspectives on their use have been understudied.

METHODS

In this qualitative study, between March and November 2020, 62 patients were interviewed from three health systems: one anticipating implementation of an EHR-derived suicide risk prediction model and two others piloting different implementation approaches. Site-tailored interview guides focused on patients' perceptions of this technology, concerns, and preferences for and experiences with suicide risk prediction model implementation in clinical practice. A constant comparative analytic approach was used to derive themes.

RESULTS

Interview participants were generally supportive of suicide risk prediction models derived from EHR data. Concerns included apprehension about inducing anxiety and suicidal thoughts, or triggering coercive treatment, particularly among those who reported prior negative experiences seeking mental health care. Participants who were engaged in mental health care or case management expected to be asked about their suicide risk and largely appreciated suicide risk conversations, particularly by clinicians comfortable discussing suicidality.

CONCLUSION

Most patients approved of suicide risk models that use EHR data to identify patients at-risk for suicide. As health systems proceed to implement such models, patient-centered care would involve dialogue initiated by clinicians experienced with assessing suicide risk during virtual or in person care encounters. Health systems should proactively monitor for negative consequences that result from risk model implementation to protect patient trust.

Detecting and distinguishing indicators of risk for suicide using clinical records

Health systems are essential for suicide risk detection. Most efforts target people with mental health (MH) diagnoses, but this only represents half of the people who die by suicide. This study seeks to discover and validate health indicators of suicide death among those with, and without, MH diagnoses. This case-control study used statistical modeling with health record data on diagnoses, procedures, and encounters. The study included 3,195 individuals who died by suicide from 2000 to 2015 and 249,092 randomly selected matched controls, who were age 18+ and affiliated with nine Mental Health Research Network affiliated health systems. Of the 202 indicators studied, 170 (84%) were associated with suicide in the discovery cohort, with 148 (86%) of those in the validation cohort. Malignant cancer diagnoses were risk factors for suicide in those without MH diagnoses, and multiple individual psychiatric-related indicators were unique to the MH subgroup. Protective effects across MH-stratified models included diagnoses of benign neoplasms, respiratory infections, and utilization of reproductive services. MH-stratified latent class models validated five subgroups with distinct patterns of indicators in both those with and without MH. The highest risk groups were characterized via high utilization with multiple healthcare concerns in both groups. The lowest risk groups were characterized as predominantly young, female, and high utilizers of preventive services. Healthcare data include many indicators of suicide risk for those with and without MH diagnoses, which may be used to support the identification and understanding of risk as well as targeting of prevention in health systems.

Machine Learning Analysis of Handgun Transactions to Predict Firearm Suicide Risk

IMPORTANCE

Evidence suggests that limiting access to firearms among individuals at high risk of suicide can be an effective means of suicide prevention, yet accurately identifying those at risk to intervene remains a key challenge. Firearm purchasing records may offer a large-scale and objective data source for the development of tools to predict firearm suicide risk.

OBJECTIVE

To test whether a statewide database of handgun transaction records, coupled with machine learning techniques, can be used to forecast firearm suicide risk.

DESIGN, SETTING, AND PARTICIPANTS

This prognostic study used the California database of 4 976 391 handgun transaction records from 1 951 006 individuals from January 1, 1996, to October 6, 2015. Transaction-level random forest classification was implemented to predict firearm suicide risk, and the relative predictive power of features in the algorithm was estimated via permutation importance. Analyses were performed from December 1, 2020, to May 19, 2022.

MAIN OUTCOMES AND MEASURES

The main outcome was firearm suicide within 1 year of a firearm transaction, derived from California death records (1996-2016). With the use of California's Dealer's Records of Sale (1996-2015), 41 handgun, transaction, purchaser, and community-level predictor variables were generated.

RESULTS

There are a total of 4 976 391 transactions in the California's Dealer's Record of Sale database representing 1 951 006 individuals (1 525 754 men [78.2% of individuals]; mean [SD] age, 43.4 [13.9] years). Firearm suicide within 1 year occurred in 0.07% of handgun transactions (3278 transactions among 2614 individuals). A total of 38.6% of observed firearm suicides were among transactions classified in the highest-risk ventile (379 of 983 transactions), with 95% specificity. Among the small number of transactions with a random forest score above 0.95, more than two-thirds (24 of 35 [68.6%]) were associated with a purchaser who died by firearm suicide within 1 year. Important features included known risk factors, such as older age at first purchase, and previously unreported predictors, including distance to firearms dealer and month of purchase.

CONCLUSIONS AND RELEVANCE

This prognostic study presented the first large-scale machine learning analysis of individual-level handgun transaction records. The results suggested the potential utility of such records in identifying high-risk individuals to aid suicide prevention efforts. It also identified handgun, individual, and community characteristics that have strong predictive relationships with firearm suicide and may warrant further study.

The association of prescription opioid use with suicide attempts: An analysis of statewide medical claims data

Background

Suicides and opioid overdose deaths are among the most pressing public health concerns in the US. However direct evidence for the association between opioid use and suicidal behavior is limited. The objective of this article is to examine the association between frequency and dose of prescription opioid use and subsequent suicide attempts.

Methods and findings

This retrospective cohort study analyzed 4 years of statewide medical claims data from the Connecticut All-Payer Claims Database. Commercially insured adult patients in Connecticut (n = 842,773) who had any medical claims beginning in January 2012 were followed through December 2015. The primary outcome was suicide attempt identified using International Classification of Diseases (ICD 9) diagnosis codes. Primary predictor variables included frequency of opioid use, which was defined as the number of months with claims for prescription opioids per year, and strength of opioid dose, which was standardized using morphine milligram equivalent (MME) units. We also controlled for psychiatric and medical comorbidities using ICD 9 codes. We used Cox proportional hazards regression to examine the association between frequency, dose, and suicide attempts, adjusting for medical and psychiatric comorbid conditions. Interactions among measures of opioid use and comorbid conditions were analyzed.

In this cohort study with follow-up time up to 4 years (range = 2–48 months, median = 46 months), the hazard ratios (HR) from the time-to-event analysis indicated that patients prescribed opioid medications for at least 6 months during the past year and at 20–50 MME levels or higher had 4.44 (95% CI: [3.71, 5.32]) to 7.23 (95% CI: [6.22, 8.41]) times the risk of attempted suicide compared to those not prescribed opioids. Risk of suicide attempt was sharply elevated among patients with psychiatric conditions other than anxiety who were prescribed more frequent and higher opioid doses. In contrast, more frequent and higher doses of prescription opioids were associated with lower risk of suicide attempts among patients with medical conditions necessitating pain management.

This study is limited by its exclusive focus on commercially insured patients and does not include patients covered by public insurance. It is also limited to patients’ receipt of prescription opioids and does not take into account opioids obtained through other means, nor does it include measures of actual patient opioid use.

Conclusions

This analysis provides evidence of a complex relationship among prescription opioids, mental health, pain and other medical comorbidities, and suicide risk. Findings indicate the need for proactive suicide surveillance among individuals diagnosed with affective or psychotic disorders who are receiving frequent and high doses of opioids. However, appropriate opioid treatment may have significant value in reducing suicide risk for those without psychiatric comorbidities.

Implementing Machine Learning Models for Suicide Risk Prediction in Clinical Practice: Focus Group Study With Hospital Providers

Background

Interest in developing machine learning models that use electronic health record data to predict patients’ risk of suicidal behavior has recently proliferated. However, whether and how such models might be implemented and useful in clinical practice remain unknown. To ultimately make automated suicide risk–prediction models useful in practice, and thus better prevent patient suicides, it is critical to partner with key stakeholders, including the frontline providers who will be using such tools, at each stage of the implementation process.

Objective

The aim of this focus group study is to inform ongoing and future efforts to deploy suicide risk–prediction models in clinical practice. The specific goals are to better understand hospital providers’ current practices for assessing and managing suicide risk; determine providers’ perspectives on using automated suicide risk–prediction models in practice; and identify barriers, facilitators, recommendations, and factors to consider.

Methods

We conducted 10 two-hour focus groups with a total of 40 providers from psychiatry, internal medicine and primary care, emergency medicine, and obstetrics and gynecology departments within an urban academic medical center. Audio recordings of open-ended group discussions were transcribed and coded for relevant and recurrent themes by 2 independent study staff members. All coded text was reviewed and discrepancies were resolved in consensus meetings with doctoral-level staff.

Results

Although most providers reported using standardized suicide risk assessment tools in their clinical practices, existing tools were commonly described as unhelpful and providers indicated dissatisfaction with current suicide risk assessment methods. Overall, providers’ general attitudes toward the practical use of automated suicide risk–prediction models and corresponding clinical decision support tools were positive. Providers were especially interested in the potential to identify high-risk patients who might be missed by traditional screening methods. Some expressed skepticism about the potential usefulness of these models in routine care; specific barriers included concerns about liability, alert fatigue, and increased demand on the health care system. Key facilitators included presenting specific patient-level features contributing to risk scores, emphasizing changes in risk over time, and developing systematic clinical workflows and provider training. Participants also recommended considering risk-prediction windows, timing of alerts, who will have access to model predictions, and variability across treatment settings.

Conclusions

Providers were dissatisfied with current suicide risk assessment methods and were open to the use of a machine learning–based risk-prediction system to inform clinical decision-making. They also raised multiple concerns about potential barriers to the usefulness of this approach and suggested several possible facilitators. Future efforts in this area will benefit from incorporating systematic qualitative feedback from providers, patients, administrators, and payers on the use of these new approaches in routine care, especially given the complex, sensitive, and unfortunately still stigmatized nature of suicide risk.

Translating promise into practice: a review of machine learning in suicide research and prevention

In ever more pressured health-care systems, technological solutions offering scalability of care and better resource targeting are appealing. Research on machine learning as a technique for identifying individuals at risk of suicidal ideation, suicide attempts, and death has grown rapidly. This research often places great emphasis on the promise of machine learning for preventing suicide, but overlooks the practical, clinical implementation issues that might preclude delivering on such a promise. In this Review, we synthesise the broad empirical and review literature on electronic health record-based machine learning in suicide research, and focus on matters of crucial importance for implementation of machine learning in clinical practice. The challenge of preventing statistically rare outcomes is well known; progress requires tackling data quality, transparency, and ethical issues. In the future, machine learning models might be explored as methods to enable targeting of interventions to specific individuals depending upon their level of need-ie, for precision medicine. Primarily, however, the promise of machine learning for suicide prevention is limited by the scarcity of high-quality scalable interventions available to individuals identified by machine learning as being at risk of suicide.

Predictive structured-unstructured interactions in EHR models: A case study of suicide prediction

Clinical risk prediction models powered by electronic health records (EHRs) are becoming increasingly widespread in clinical practice. With suicide-related mortality rates rising in recent years, it is becoming increasingly urgent to understand, predict, and prevent suicidal behavior. Here, we compare the predictive value of structured and unstructured EHR data for predicting suicide risk. We find that Naive Bayes Classifier (NBC) and Random Forest (RF) models trained on structured EHR data perform better than those based on unstructured EHR data. An NBC model trained on both structured and unstructured data yields similar performance (AUC = 0.743) to an NBC model trained on structured data alone (0.742, p = 0.668), while an RF model trained on both data types yields significantly better results (AUC = 0.903) than an RF model trained on structured data alone (0.887, p < 0.001), likely due to the RF model’s ability to capture interactions between the two data types. To investigate these interactions, we propose and implement a general framework for identifying specific structured-unstructured feature pairs whose interactions differ between case and non-case cohorts, and thus have the potential to improve predictive performance and increase understanding of clinical risk. We find that such feature pairs tend to capture heterogeneous pairs of general concepts, rather than homogeneous pairs of specific concepts. These findings and this framework can be used to improve current and future EHR-based clinical modeling efforts.

Prediction of Suicide Attempts Using Clinician Assessment, Patient Self-report, and Electronic Health Records

IMPORTANCE

Half of the people who die by suicide make a health care visit within 1 month of their death. However, clinicians lack the tools to identify these patients.

OBJECTIVE

To predict suicide attempts within 1 and 6 months of presentation at an emergency department (ED) for psychiatric problems.

DESIGN, SETTING, AND PARTICIPANTS

This prognostic study assessed the 1-month and 6-month risk of suicide attempts among 1818 patients presenting to an ED between February 4, 2015, and March 13, 2017, with psychiatric problems. Data analysis was performed from May 1, 2020, to November 19, 2021.

MAIN OUTCOMES AND MEASURES

Suicide attempts 1 and 6 months after presentation to the ED were defined by combining data from electronic health records (EHRs) with patient 1-month (n = 1102) and 6-month (n = 1220) follow-up surveys. Ensemble machine learning was used to develop predictive models and a risk score for suicide.

RESULTS

A total of 1818 patients participated in this study (1016 men [55.9%]; median age, 33 years [IQR, 24-46 years]; 266 Hispanic patients [14.6%]; 1221 non-Hispanic White patients [67.2%], 142 non-Hispanic Black patients [7.8%], 64 non-Hispanic Asian patients [3.5%], and 125 non-Hispanic patients of other race and ethnicity [6.9%]). A total of 137 of 1102 patients (12.9%; weighted prevalence) attempted suicide within 1 month, and a total of 268 of 1220 patients (22.0%; weighted prevalence) attempted suicide within 6 months. Clinicians' assessment alone was little better than chance at predicting suicide attempts, with externally validated area under the receiver operating characteristic curve (AUC) of 0.67 for the 1-month model and 0.60 for the 6-month model. Prediction accuracy was slightly higher for models based on EHR data (1-month model: AUC, 0.71; 6 month model: AUC, 0.65) and was best using patient self-reports (1-month model: AUC, 0.76; 6-month model: AUC, 0.77), especially when patient self-reports were combined with EHR and/or clinician data (1-month model: AUC, 0.77; and 6 month model: AUC, 0.79). A model that used only 20 patient self-report questions and an EHR-based risk score performed similarly well (1-month model: AUC, 0.77; 6 month model: AUC, 0.78). In the best 1-month model, 30.7% (positive predicted value) of the patients classified as having highest risk (top 25% of the sample) made a suicide attempt within 1 month of their ED visit, accounting for 64.8% (sensitivity) of all 1-month attempts. In the best 6-month model, 46.0% (positive predicted value) of the patients classified at highest risk made a suicide attempt within 6 months of their ED visit, accounting for 50.2% (sensitivity) of all 6-month attempts.

CONCLUSIONS AND RELEVANCE

This prognostic study suggests that the ability to identify patients at high risk of suicide attempt after an ED visit for psychiatric problems improved using a combination of patient self-reports and EHR data.

Biobanking in everyday clinical practice in psychiatry-The Munich Mental Health Biobank

Translational research on complex, multifactorial mental health disorders, such as bipolar disorder, major depressive disorder, schizophrenia, and substance use disorders requires databases with large-scale, harmonized, and integrated real-world and research data. The Munich Mental Health Biobank (MMHB) is a mental health-specific biobank that was established in 2019 to collect, store, connect, and supply such high-quality phenotypic data and biosamples from patients and study participants, including healthy controls, recruited at the Department of Psychiatry and Psychotherapy (DPP) and the Institute of Psychiatric Phenomics and Genomics (IPPG), University Hospital of the Ludwig-Maximilians-University (LMU), Munich, Germany. Participants are asked to complete a questionnaire that assesses sociodemographic and cross-diagnostic clinical information, provide blood samples, and grant access to their existing medical records. The generated data and biosamples are available to both academic and industry researchers. In this manuscript, we outline the workflow and infrastructure of the MMHB, describe the clinical characteristics and representativeness of the sample collected so far, and reveal future plans for expansion and application. As of 31 October 2021, the MMHB contains a continuously growing set of data from 578 patients and 104 healthy controls (46.37% women; median age, 38.31 years). The five most common mental health diagnoses in the MMHB are recurrent depressive disorder (38.78%; ICD-10: F33), alcohol-related disorders (19.88%; ICD-10: F10), schizophrenia (19.69%; ICD-10: F20), depressive episode (15.94%; ICD-10: F32), and personality disorders (13.78%; ICD-10: F60). Compared with the average patient treated at the recruiting hospitals, MMHB participants have significantly more mental health-related contacts, less severe symptoms, and a higher level of functioning. The distribution of diagnoses is also markedly different in MMHB participants compared with individuals who did not participate in the biobank. After establishing the necessary infrastructure and initiating recruitment, the major tasks for the next phase of the MMHB project are to improve the pace of participant enrollment, diversify the sociodemographic and diagnostic characteristics of the sample, and improve the utilization of real-world data generated in routine clinical practice.

Temporally informed random forests for suicide risk prediction

OBJECTIVE

Suicide is one of the leading causes of death worldwide, yet clinicians find it difficult to reliably identify individuals at high risk for suicide. Algorithmic approaches for suicide risk detection have been developed in recent years, mostly based on data from electronic health records (EHRs). Significant room for improvement remains in the way these models take advantage of temporal information to improve predictions.

MATERIALS AND METHODS

We propose a temporally enhanced variant of the random forest (RF) model-Omni-Temporal Balanced Random Forests (OT-BRFs)-that incorporates temporal information in every tree within the forest. We develop and validate this model using longitudinal EHRs and clinician notes from the Mass General Brigham Health System recorded between 1998 and 2018, and compare its performance to a baseline Naive Bayes Classifier and 2 standard versions of balanced RFs.

RESULTS

Temporal variables were found to be associated with suicide risk: Elevated suicide risk was observed in individuals with a higher total number of visits as well as those with a low rate of visits over time, while lower suicide risk was observed in individuals with a longer period of EHR coverage. RF models were more accurate than Naive Bayesian classifiers at predicting suicide risk in advance (area under the receiver operating curve = 0.824 vs. 0.754, respectively). The proposed OT-BRF model performed best among all RF approaches, yielding a sensitivity of 0.339 at 95% specificity, compared to 0.290 and 0.286 for the other 2 RF models. Temporal variables were assigned high importance by the models that incorporated them.

DISCUSSION

We demonstrate that temporal variables have an important role to play in suicide risk detection and that requiring their inclusion in all RF trees leads to increased predictive performance. Integrating temporal information into risk prediction models helps the models interpret patient data in temporal context, improving predictive performance.

Prediction across healthcare settings: a case study in predicting emergency department disposition

Several approaches exist today for developing predictive models across multiple clinical sites, yet there is a lack of comparative data on their performance, especially within the context of EHR-based prediction models. We set out to provide a framework for prediction across healthcare settings. As a case study, we examined an ED disposition prediction model across three geographically and demographically diverse sites. We conducted a 1-year retrospective study, including all visits in which the outcome was either discharge-to-home or hospitalization. Four modeling approaches were compared: a ready-made model trained at one site and validated at other sites, a centralized uniform model incorporating data from all sites, multiple site-specific models, and a hybrid approach of a ready-made model re-calibrated using site-specific data. Predictions were performed using XGBoost. The study included 288,962 visits with an overall admission rate of 16.8% (7.9–26.9%). Some risk factors for admission were prominent across all sites (e.g., high-acuity triage emergency severity index score, high prior admissions rate), while others were prominent at only some sites (multiple lab tests ordered at the pediatric sites, early use of ECG at the adult site). The XGBoost model achieved its best performance using the uniform and site-specific approaches (AUC = 0.9–0.93), followed by the calibrated-model approach (AUC = 0.87–0.92), and the ready-made approach (AUC = 0.62–0.85). Our results show that site-specific customization is a key driver of predictive model performance.

From everyday life predictions to suicide prevention: Clinical and ethical considerations in suicide predictive analytic tools

Advances in artificial intelligence and machine learning have fueled growing interest in the application of predictive analytics to identify high-risk suicidal patients. Such application will require the aggregation of large-scale, sensitive patient data to help inform complex and potentially stigmatizing health care decisions. This paper provides a description of how suicide prediction is uniquely difficult by comparing it to nonmedical (weather and traffic forecasting) and medical predictions (cancer and human immunodeficiency virus risk), followed by clinical and ethical challenges presented within a risk-benefit conceptual framework. Because the misidentification of suicide risk may be associated with unintended negative consequences, clinicians and policymakers need to carefully weigh the risks and benefits of using suicide predictive analytics across health care populations. Practical recommendations are provided to strengthen the protection of patient rights and enhance the clinical utility of suicide predictive analytics tools.

Design and Implementation of an Informatics Infrastructure for Standardized Data Acquisition, Transfer, Storage, and Export in Psychiatric Clinical Routine: Feasibility Study

BACKGROUND

Empirically driven personalized diagnostic applications and treatment stratification is widely perceived as a major hallmark in psychiatry. However, databased personalized decision making requires standardized data acquisition and data access, which are currently absent in psychiatric clinical routine.

OBJECTIVE

Here, we describe the informatics infrastructure implemented at the psychiatric Münster University Hospital, which allows standardized acquisition, transfer, storage, and export of clinical data for future real-time predictive modelling in psychiatric routine.

METHODS

We designed and implemented a technical architecture that includes an extension of the electronic health record (EHR) via scalable standardized data collection and data transfer between EHRs and research databases, thus allowing the pooling of EHRs and research data in a unified database and technical solutions for the visual presentation of collected data and analyses results in the EHR. The Single-source Metadata ARchitecture Transformation (SMA:T) was used as the software architecture. SMA:T is an extension of the EHR system and uses module-driven engineering to generate standardized applications and interfaces. The operational data model was used as the standard. Standardized data were entered on iPads via the Mobile Patient Survey (MoPat) and the web application Mopat@home, and the standardized transmission, processing, display, and export of data were realized via SMA:T.

RESULTS

The technical feasibility of the informatics infrastructure was demonstrated in the course of this study. We created 19 standardized documentation forms with 241 items. For 317 patients, 6451 instances were automatically transferred to the EHR system without errors. Moreover, 96,323 instances were automatically transferred from the EHR system to the research database for further analyses.

CONCLUSIONS

In this study, we present the successful implementation of the informatics infrastructure enabling standardized data acquisition and data access for future real-time predictive modelling in clinical routine in psychiatry. The technical solution presented here might guide similar initiatives at other sites and thus help to pave the way toward future application of predictive models in psychiatric clinical routine.

Characterization of time-variant and time-invariant assessment of suicidality on Reddit using C-SSRS

Suicide is the 10th leading cause of death in the U.S (1999-2019). However, predicting when someone will attempt suicide has been nearly impossible. In the modern world, many individuals suffering from mental illness seek emotional support and advice on well-known and easily-accessible social media platforms such as Reddit. While prior artificial intelligence research has demonstrated the ability to extract valuable information from social media on suicidal thoughts and behaviors, these efforts have not considered both severity and temporality of risk. The insights made possible by access to such data have enormous clinical potential-most dramatically envisioned as a trigger to employ timely and targeted interventions (i.e., voluntary and involuntary psychiatric hospitalization) to save lives. In this work, we address this knowledge gap by developing deep learning algorithms to assess suicide risk in terms of severity and temporality from Reddit data based on the Columbia Suicide Severity Rating Scale (C-SSRS). In particular, we employ two deep learning approaches: time-variant and time-invariant modeling, for user-level suicide risk assessment, and evaluate their performance against a clinician-adjudicated gold standard Reddit corpus annotated based on the C-SSRS. Our results suggest that the time-variant approach outperforms the time-invariant method in the assessment of suicide-related ideations and supportive behaviors (AUC:0.78), while the time-invariant model performed better in predicting suicide-related behaviors and suicide attempt (AUC:0.64). The proposed approach can be integrated with clinical diagnostic interviews for improving suicide risk assessments.

Reconciling Statistical and Clinicians' Predictions of Suicide Risk

Statistical models, including those based on electronic health records, can accurately identify patients at high risk for a suicide attempt or death, leading to implementation of risk prediction models for population-based suicide prevention in health systems. However, some have questioned whether statistical predictions can really inform clinical decisions. Appropriately reconciling statistical algorithms with traditional clinician assessment depends on whether predictions from these two methods are competing, complementary, or merely duplicative. In June 2019, the National Institute of Mental Health convened a meeting, "Identifying Research Priorities for Risk Algorithms Applications in Healthcare Settings to Improve Suicide Prevention." Here, participants of this meeting summarize key issues regarding the potential clinical application of suicide prediction models. The authors attempt to clarify the key conceptual and technical differences between traditional risk prediction by clinicians and predictions from statistical models, review the limited evidence regarding both the accuracy of and the concordance between these alternative methods of prediction, present a conceptual framework for understanding agreement and disagreement between statistical and clinician predictions, identify priorities for improving data regarding suicide risk, and propose priority questions for future research. Future suicide risk assessment will likely combine statistical prediction with traditional clinician assessment, but research is needed to determine the optimal combination of these two methods.

Patient perspectives on acceptability of, and implementation preferences for, use of electronic health records and machine learning to identify suicide risk

OBJECTIVE

Assess patient understanding of, potential concerns with, and implementation preferences related to automated suicide risk identification using electronic health record data and machine learning.

METHOD

Focus groups (n = 23 participants) informed a web-based survey sent to 11,486 Kaiser Permanente Northwest members in April 2020. Survey items assessed patient preferences using Likert and visual analog scales (means scored from -50 to 50). Descriptive statistics summarized findings.

RESULTS

1357 (12%) participants responded. Most (84%) found machine learning-derived suicide risk identification an acceptable use of electronic health record data; however, 67% objected to use of externally sourced data. Participants felt consent (or opt-out) should be required (mean = -14). The majority (69%) supported outreach to at-risk individuals by a trusted clinician through care messages (57%) or telephone calls (47-54%). Highest endorsements were for psychiatrists/therapists (99%) or a primary care clinician (75-96%); less than half (42%) supported outreach by any clinician and participants generally felt only trusted clinicians should have access to risk information (mean = -16).

CONCLUSION

Patients generally support use of EHR data (not externally sourced risk information) to inform automated suicide risk identification models but prefer to consent or opt-out; trusted clinicians should outreach by telephone or care message to at risk individuals.

Self-harm in the context of borderline personality disorder

The present article gives a selective overview of recent studies on the role of nonsuicidal self-injury (NSSI) and suicidal behavior in the context of borderline personality disorder (BPD). Previous research found self-harming behavior, particularly NSSI, to constitute an easily accessible marker in the early detection of individuals at risk of development of BPD. The review further summarizes studies that investigated inter-relations between BPD features and self-harming behavior over time. Mainly, affective instability has been shown to play a role in the maintenance of NSSI and the increased risk of suicidal behavior among individuals with BPD. Finally, results about the effectiveness of treatment programs on the reduction of self-harming behavior among individuals with BPD are presented.

A Feasibility Study Using a Machine Learning Suicide Risk Prediction Model Based on Open-Ended Interview Language in Adolescent Therapy Sessions

Background: As adolescent suicide rates continue to rise, innovation in risk identification is warranted. Machine learning can identify suicidal individuals based on their language samples. This feasibility pilot was conducted to explore this technology’s use in adolescent therapy sessions and assess machine learning model performance. Method: Natural language processing machine learning models to identify level of suicide risk using a smartphone app were tested in outpatient therapy sessions. Data collection included language samples, depression and suicidality standardized scale scores, and therapist impression of the client’s mental state. Previously developed models were used to predict suicidal risk. Results: 267 interviews were collected from 60 students in eight schools by ten therapists, with 29 students indicating suicide or self-harm risk. During external validation, models were trained on suicidal speech samples collected from two separate studies. We found that support vector machines (AUC: 0.75; 95% CI: 0.69–0.81) and logistic regression (AUC: 0.76; 95% CI: 0.70–0.82) lead to good discriminative ability, with an extreme gradient boosting model performing the best (AUC: 0.78; 95% CI: 0.72–0.84). Conclusion: Voice collection technology and associated procedures can be integrated into mental health therapists’ workflow. Collected language samples could be classified with good discrimination using machine learning methods.

Machine learning for suicidology: A practical review of exploratory and hypothesis-driven approaches

Machine learning is being used to discover models to predict the progression from suicidal ideation to action in clinical populations. While quantifiable improvements in prediction accuracy have been achieved over theory-driven efforts, models discovered through machine learning continue to fall short of clinical relevance. Thus, the value of machine learning for reaching this objective is hotly contested. We agree that machine learning, treated as a "black box" approach antithetical to theory-building, will not discover clinically relevant models of suicide. However, such models may be developed through deliberate synthesis of data- and theory-driven approaches. By providing an accessible overview of essential concepts and common methods, we highlight how generalizable models and scientific insight may be obtained by incorporating prior knowledge and expectations to machine learning research, drawing examples from suicidology. We then discuss challenges investigators will face when using machine learning to discover models of low prevalence outcomes, such as suicide.

Decomposing implicit associations about life and death improves our understanding of suicidal behavior

The Death/Suicide Implicit Association Test (IAT) is effective at detecting and prospectively predicting suicidal thoughts and behaviors. However, traditional IAT scoring procedures used in all prior studies (i.e., D-scores) provide an aggregate score that is inherently relative, obfuscating the separate associations (i.e., "Me = Death/Suicide," "Me = Life") that might be most relevant for understanding suicide-related implicit cognition. Here, we decompose the D-scores and validate a new analytic technique called the Decomposed D-scores ("DD-scores") that creates separate scores for each category ("Me," "Not Me") in the IAT. Across large online volunteer samples (N > 12,000), results consistently showed that a weakened association between "Me = Life" is more strongly predictive of having a history of suicidal attempts than is a stronger association between "Me = Death/Suicide." These findings replicated across three different versions of the IAT and were observed when calculated using both reaction times and error rates. However, among those who previously attempted suicide, a strengthened association between "Me = Death" is more strongly predictive of the recency of a suicide attempt. These results suggest that decomposing traditional IAT D-scores can offer new insights into the mental associations that may underlie clinical phenomena and may help to improve the prediction, and ultimately the prevention, of these clinical outcomes.

Illuminating the Black Box: Interpreting Deep Neural Network Models for Psychiatric Research

Psychiatric research is often confronted with complex abstractions and dynamics that are not readily accessible or well-defined to our perception and measurements, making data-driven methods an appealing approach. Deep neural networks (DNNs) are capable of automatically learning abstractions in the data that can be entirely novel and have demonstrated superior performance over classical machine learning models across a range of tasks and, therefore, serve as a promising tool for making new discoveries in psychiatry. A key concern for the wider application of DNNs is their reputation as a "black box" approach-i.e., they are said to lack transparency or interpretability of how input data are transformed to model outputs. In fact, several existing and emerging tools are providing improvements in interpretability. However, most reviews of interpretability for DNNs focus on theoretical and/or engineering perspectives. This article reviews approaches to DNN interpretability issues that may be relevant to their application in psychiatric research and practice. It describes a framework for understanding these methods, reviews the conceptual basis of specific methods and their potential limitations, and discusses prospects for their implementation and future directions.