Risk stratification using data from electronic medical records better predicts suicide risks than clinician assessments

Predictive Validity of the K-CAT-SS in High-Risk Adolescents and Young Adults

OBJECTIVE

Suicide is a leading cause of death in adolescents and young adults and has increased substantially in the past 15 years. Accurate suicide risk stratification based on rapid screening can help reverse these trends. This study aimed to assess the ability of the Kiddie Computerized Adaptive Test Suicide Scale (K-CAT-SS), a brief computerized adaptive test of suicidality, to predict suicide attempts (SAs) in high-risk youth.

METHOD

A total of 652 participants (age range, 12-24 years), 78% of whom presented with suicidal ideation or behavior, were recruited within 1 month of mental health care contact. The K-CAT-SS, scaled from 0 to 100, was administered at baseline, and participants were assessed at about 1, 3, and 6 months after intake. Weekly incidence of SAs was assessed using the Adolescent Longitudinal Interval Follow-up Evaluation and Columbia-Suicide Severity Rating Scale. A secondary outcome was suicidal behavior, including aborted, interrupted, and actual SAs.

RESULTS

The K-CAT-SS showed a 4.91-fold increase in SAs for every 25-point increase in the baseline score (95% CI 2.83-8.52) and a 3.51-fold increase in suicidal behaviors (95% CI 2.32-5.30). These relations persisted following adjustment for prior attempts; demographic variables including age, sex, gender identity, sexual orientation, and race/ethnicity; and other measures of psychopathology. No moderating effects were identified. At 3 months, area under the receiver operating characteristic curve was 0.83 (95% CI 0.72-0.93) for 1 or more SAs.

CONCLUSION

The K-CAT-SS is an excellent tool for suicide risk stratification, particularly in higher-risk populations where other measures have shown lower predictive validity.

Machine learning and the prediction of suicide in psychiatric populations: a systematic review

Machine learning (ML) has emerged as a promising tool to enhance suicidal prediction. However, as many large-sample studies mixed psychiatric and non-psychiatric populations, a formal psychiatric diagnosis emerged as a strong predictor of suicidal risk, overshadowing more subtle risk factors specific to distinct populations. To overcome this limitation, we conducted a systematic review of ML studies evaluating suicidal behaviors exclusively in psychiatric clinical populations. A systematic literature search was performed from inception through November 17, 2022 on PubMed, EMBASE, and Scopus following the PRISMA guidelines. Original research using ML techniques to assess the risk of suicide or predict suicide attempts in the psychiatric population were included. An assessment for bias risk was performed using the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guidelines. About 1032 studies were retrieved, and 81 satisfied the inclusion criteria and were included for qualitative synthesis. Clinical and demographic features were the most frequently employed and random forest, support vector machine, and convolutional neural network performed better in terms of accuracy than other algorithms when directly compared. Despite heterogeneity in procedures, most studies reported an accuracy of 70% or greater based on features such as previous attempts, severity of the disorder, and pharmacological treatments. Although the evidence reported is promising, ML algorithms for suicidal prediction still present limitations, including the lack of neurobiological and imaging data and the lack of external validation samples. Overcoming these issues may lead to the development of models to adopt in clinical practice. Further research is warranted to boost a field that holds the potential to critically impact suicide mortality.

Impact of Implementation Facilitation on the REACH VET Clinical Program for Veterans at Risk for Suicide

OBJECTIVE

In 2017, the Veterans Health Administration (VHA) implemented a national suicide prevention program, called Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment (REACH VET), that uses a predictive algorithm to identify, attempt to reach, assess, and care for patients at the highest risk for suicide. The authors aimed to evaluate whether facilitation enhanced implementation of REACH VET at VHA facilities not meeting target completion rates.

METHODS

In this hybrid effectiveness-implementation type 2 program evaluation, a quasi-experimental pre-post design was used to assess changes in implementation outcome measures evaluated 6 months before and 6 months after onset of facilitation of REACH VET implementation at 23 VHA facilities. Measures included percentages of patients with documented coordinator and provider acknowledgment of receipt, care evaluation, and outreach attempt. Generalized estimating equations were used to compare differences in REACH VET outcome measures before and after facilitation. Qualitative interviews were conducted with personnel and were explored via template analysis.

RESULTS

Time had a significant effect in all outcomes models (p<0.001). An effect of facilitation was significant only for the outcome of attempted outreach. Patients identified by REACH VET had significantly higher odds of having a documented outreach attempt after facilitation of REACH VET implementation, compared with before facilitation. Site personnel felt supported and reported that the external facilitators were helpful and responsive.

CONCLUSIONS

Facilitation of REACH VET implementation was associated with an improvement in outreach attempts to veterans identified as being at increased risk for suicide. Outreach is critical for engaging veterans in care.

The use of machine learning on administrative and survey data to predict suicidal thoughts and behaviors: a systematic review

Background

Machine learning is a promising tool in the area of suicide prevention due to its ability to combine the effects of multiple risk factors and complex interactions. The power of machine learning has led to an influx of studies on suicide prediction, as well as a few recent reviews. Our study distinguished between data sources and reported the most important predictors of suicide outcomes identified in the literature.

Objective

Our study aimed to identify studies that applied machine learning techniques to administrative and survey data, summarize performance metrics reported in those studies, and enumerate the important risk factors of suicidal thoughts and behaviors identified.

Methods

A systematic literature search of PubMed, Medline, Embase, PsycINFO, Web of Science, Cumulative Index to Nursing and Allied Health Literature (CINAHL), and Allied and Complementary Medicine Database (AMED) to identify all studies that have used machine learning to predict suicidal thoughts and behaviors using administrative and survey data was performed. The search was conducted for articles published between January 1, 2019 and May 11, 2022. In addition, all articles identified in three recently published systematic reviews (the last of which included studies up until January 1, 2019) were retained if they met our inclusion criteria. The predictive power of machine learning methods in predicting suicidal thoughts and behaviors was explored using box plots to summarize the distribution of the area under the receiver operating characteristic curve (AUC) values by machine learning method and suicide outcome (i.e., suicidal thoughts, suicide attempt, and death by suicide). Mean AUCs with 95% confidence intervals (CIs) were computed for each suicide outcome by study design, data source, total sample size, sample size of cases, and machine learning methods employed. The most important risk factors were listed.

Results

The search strategy identified 2,200 unique records, of which 104 articles met the inclusion criteria. Machine learning algorithms achieved good prediction of suicidal thoughts and behaviors (i.e., an AUC between 0.80 and 0.89); however, their predictive power appears to differ across suicide outcomes. The boosting algorithms achieved good prediction of suicidal thoughts, death by suicide, and all suicide outcomes combined, while neural network algorithms achieved good prediction of suicide attempts. The risk factors for suicidal thoughts and behaviors differed depending on the data source and the population under study.

Conclusion

The predictive utility of machine learning for suicidal thoughts and behaviors largely depends on the approach used. The findings of the current review should prove helpful in preparing future machine learning models using administrative and survey data.

Systematic review registration

https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022333454 identifier CRD42022333454.

The Adoption of AI in Mental Health Care-Perspectives From Mental Health Professionals: Qualitative Descriptive Study

BACKGROUND

Artificial intelligence (AI) is transforming the mental health care environment. AI tools are increasingly accessed by clients and service users. Mental health professionals must be prepared not only to use AI but also to have conversations about it when delivering care. Despite the potential for AI to enable more efficient and reliable and higher-quality care delivery, there is a persistent gap among mental health professionals in the adoption of AI.

OBJECTIVE

A needs assessment was conducted among mental health professionals to (1) understand the learning needs of the workforce and their attitudes toward AI and (2) inform the development of AI education curricula and knowledge translation products.

METHODS

A qualitative descriptive approach was taken to explore the needs of mental health professionals regarding their adoption of AI through semistructured interviews. To reach maximum variation sampling, mental health professionals (eg, psychiatrists, mental health nurses, educators, scientists, and social workers) in various settings across Ontario (eg, urban and rural, public and private sector, and clinical and research) were recruited.

RESULTS

A total of 20 individuals were recruited. Participants included practitioners (9/20, 45% social workers and 1/20, 5% mental health nurses), educator scientists (5/20, 25% with dual roles as professors/lecturers and researchers), and practitioner scientists (3/20, 15% with dual roles as researchers and psychiatrists and 2/20, 10% with dual roles as researchers and mental health nurses). Four major themes emerged: (1) fostering practice change and building self-efficacy to integrate AI into patient care; (2) promoting system-level change to accelerate the adoption of AI in mental health; (3) addressing the importance of organizational readiness as a catalyst for AI adoption; and (4) ensuring that mental health professionals have the education, knowledge, and skills to harness AI in optimizing patient care.

CONCLUSIONS

AI technologies are starting to emerge in mental health care. Although many digital tools, web-based services, and mobile apps are designed using AI algorithms, mental health professionals have generally been slower in the adoption of AI. As indicated by this study's findings, the implications are 3-fold. At the individual level, digital professionals must see the value in digitally compassionate tools that retain a humanistic approach to care. For mental health professionals, resistance toward AI adoption must be acknowledged through educational initiatives to raise awareness about the relevance, practicality, and benefits of AI. At the organizational level, digital professionals and leaders must collaborate on governance and funding structures to promote employee buy-in. At the societal level, digital and mental health professionals should collaborate in the creation of formal AI training programs specific to mental health to address knowledge gaps. This study promotes the design of relevant and sustainable education programs to support the adoption of AI within the mental health care sphere.

One-Week Suicide Risk Prediction Using Real-Time Smartphone Monitoring: Prospective Cohort Study

BACKGROUND

Suicide is a major global public health issue that is becoming increasingly common despite preventive efforts. Though current methods for predicting suicide risk are not sufficiently accurate, technological advances provide invaluable tools with which we may evolve toward a personalized, predictive approach.

OBJECTIVE

We aim to predict the short-term (1-week) risk of suicide by identifying changes in behavioral patterns characterized through real-time smartphone monitoring in a cohort of patients with suicidal ideation.

METHODS

We recruited 225 patients between February 2018 and March 2020 with a history of suicidal thoughts and behavior as part of the multicenter SmartCrisis study. Throughout 6 months of follow-up, we collected information on the risk of suicide or mental health crises. All participants underwent voluntary passive monitoring using data generated by their own smartphones, including distance walked and steps taken, time spent at home, and app usage. The algorithm constructs daily activity profiles for each patient according to these data and detects changes in the distribution of these profiles over time. Such changes are considered critical periods, and their relationship with suicide-risk events was tested.

RESULTS

During follow-up, 18 (8%) participants attempted suicide, and 14 (6.2%) presented to the emergency department for psychiatric care. The behavioral changes identified by the algorithm predicted suicide risk in a time frame of 1 week with an area under the curve of 0.78, indicating good accuracy.

CONCLUSIONS

We describe an innovative method to identify mental health crises based on passively collected information from patients' smartphones. This technology could be applied to homogeneous groups of patients to identify different types of crises.

Intentional Self-Harm Among US Veterans With Traumatic Brain Injury or Posttraumatic Stress Disorder: Retrospective Cohort Study From 2008 to 2017

BACKGROUND

Veterans with a history of traumatic brain injury (TBI) and/or posttraumatic stress disorder (PTSD) may be at increased risk of suicide attempts and other forms of intentional self-harm as compared to veterans without TBI or PTSD.

OBJECTIVE

Using administrative data from the US Veterans Health Administration (VHA), we studied associations between TBI and PTSD diagnoses, and subsequent diagnoses of intentional self-harm among US veterans who used VHA health care between 2008 and 2017.

METHODS

All veterans with encounters or hospitalizations for intentional self-harm were assigned "index dates" corresponding to the date of the first related visit; among those without intentional self-harm, we randomly selected a date from among the veteran's health care encounters to match the distribution of case index dates over the 10-year period. We then examined the prevalence of TBI and PTSD diagnoses within the 5-year period prior to veterans' index dates. TBI, PTSD, and intentional self-harm were identified using International Classification of Diseases diagnosis and external cause of injury codes from inpatient and outpatient VHA encounters. We stratified analyses by veterans' average yearly VHA utilization in the 5-year period before their index date (low, medium, or high). Variations in prevalence and odds of intentional self-harm diagnoses were compared by veterans' prior TBI and PTSD diagnosis status (TBI only, PTSD only, and comorbid TBI/PTSD) for each VHA utilization stratum. Multivariable models adjusted for age, sex, race, ethnicity, marital status, Department of Veterans Affairs service-connection status, and Charlson Comorbidity Index scores.

RESULTS

About 6.7 million veterans with at least two VHA visits in the 5-year period before their index dates were included in the analyses; 86,644 had at least one intentional self-harm diagnosis during the study period. During the periods prior to veterans' index dates, 93,866 were diagnosed with TBI only; 892,420 with PTSD only; and 102,549 with comorbid TBI/PTSD. Across all three VHA utilization strata, the prevalence of intentional self-harm diagnoses was higher among veterans diagnosed with TBI, PTSD, or TBI/PTSD than among veterans with neither diagnosis. The observed difference was most pronounced among veterans in the high VHA utilization stratum. The prevalence of intentional self-harm was six times higher among those with comorbid TBI/PTSD (6778/58,295, 11.63%) than among veterans with neither TBI nor PTSD (21,979/1,144,991, 1.92%). Adjusted odds ratios suggested that, after accounting for potential confounders, veterans with TBI, PTSD, or comorbid TBI/PTSD had higher odds of self-harm compared to veterans without these diagnoses. Among veterans with high VHA utilization, those with comorbid TBI/PTSD were 4.26 (95% CI 4.15-4.38) times more likely to receive diagnoses for intentional self-harm than veterans with neither diagnosis. This pattern was similar for veterans with low and medium VHA utilization.

CONCLUSIONS

Veterans with TBI and/or PTSD diagnoses, compared to those with neither diagnosis, were substantially more likely to be subsequently diagnosed with intentional self-harm between 2008 and 2017. These associations were most pronounced among veterans who used VHA health care most frequently. These findings suggest a need for suicide prevention efforts targeted at veterans with these diagnoses.

Text mining methods for the characterisation of suicidal thoughts and behaviour

Traditional research methods have shown low predictive value for suicidal risk assessments and limitations to be applied in clinical practice. The authors sought to evaluate natural language processing as a new tool for assessing self-injurious thoughts and behaviors and emotions related. We used MEmind project to assess 2838 psychiatric outpatients. Anonymous unstructured responses to the open-ended question "how are you feeling today?" were collected according to their emotional state. Natural language processing was used to process the patients' writings. The texts were automatically represented (corpus) and analyzed to determine their emotional content and degree of suicidal risk. Authors compared the patients' texts with a question used to assess lack of desire to live, as a suicidal risk assessment tool. Corpus consists of 5,489 short free-text documents containing 12,256 tokenized or unique words. The natural language processing showed an ROC-AUC score of 0.9638 when compared with the responses to lack of a desire to live question. Natural language processing shows encouraging results for classifying subjects according to their desire not to live as a measure of suicidal risk using patients' free texts. It is also easily applicable to clinical practice and facilitates real-time communication with patients, allowing better intervention strategies to be designed.

A Stakeholder-Informed Ethical Framework to Guide Implementation of Suicide Risk Prediction Models Derived from Electronic Health Records

OBJECTIVE

Develop a stakeholder-informed ethical framework to provide practical guidance to health systems considering implementation of suicide risk prediction models.

METHODS

In this multi-method study, patients and family members participating in formative focus groups (n = 4 focus groups, 23 participants), patient advisors, and a bioethics consultant collectively informed the development of a web-based survey; survey results (n = 1,357 respondents) and themes from interviews with stakeholders (patients, health system administrators, clinicians, suicide risk model developers, and a bioethicist) were used to draft the ethical framework.

RESULTS

Clinical, ethical, operational, and technical issues reiterated by multiple stakeholder groups and corresponding questions for risk prediction model adopters to consider prior to and during suicide risk model implementation are organized within six ethical principles in the resulting stakeholder-informed framework. Key themes include: patients' rights to informed consent and choice to conceal or reveal risk (autonomy); appropriate application of risk models, data and model limitations and consequences including ambiguous risk predictors in opaque models (explainability); selecting actionable risk thresholds (beneficence, distributive justice); access to risk information and stigma (privacy); unanticipated harms (non-maleficence); and planning for expertise and resources to continuously audit models, monitor harms, and redress grievances (stewardship).

CONCLUSIONS

Enthusiasm for risk prediction in the context of suicide is understandable given the escalating suicide rate in the U.S. Attention to ethical and practical concerns in advance of automated suicide risk prediction model implementation may help avoid unnecessary harms that could thwart the promise of this innovation in suicide prevention. HIGHLIGHTSPatients' desire to consent/opt out of suicide risk prediction models.Recursive ethical questioning should occur throughout risk model implementation.Risk modeling resources are needed to continuously audit models and monitor harms.

Clinical implementation of suicide risk prediction models in healthcare: a qualitative study

BACKGROUND

Suicide risk prediction models derived from electronic health records (EHR) are a novel innovation in suicide prevention but there is little evidence to guide their implementation.

METHODS

In this qualitative study, 30 clinicians and 10 health care administrators were interviewed from one health system anticipating implementation of an automated EHR-derived suicide risk prediction model and two health systems piloting different implementation approaches. Site-tailored interview guides focused on respondents' expectations for and experiences with suicide risk prediction models in clinical practice, and suggestions for improving implementation. Interview prompts and content analysis were guided by Consolidated Framework for Implementation Research (CFIR) constructs.

RESULTS

Administrators and clinicians found use of the suicide risk prediction model and the two implementation approaches acceptable. Clinicians desired opportunities for early buy-in, implementation decision-making, and feedback. They wanted to better understand how this manner of risk identification enhanced existing suicide prevention efforts. They also wanted additional training to understand how the model determined risk, particularly after patients they expected to see identified by the model were not flagged at-risk and patients they did not expect to see identified were. Clinicians were concerned about having enough suicide prevention resources for potentially increased demand and about their personal liability; they wanted clear procedures for situations when they could not reach patients or when patients remained at-risk over a sustained period. Suggestions for making risk model workflows more efficient and less burdensome included consolidating suicide risk information in a dedicated module in the EHR and populating risk assessment scores and text in clinical notes.

CONCLUSION

Health systems considering suicide risk model implementation should engage clinicians early in the process to ensure they understand how risk models estimate risk and add value to existing workflows, clarify clinician role expectations, and summarize risk information in a convenient place in the EHR to support high-quality patient care.

A risk calculator to predict suicide attempts among individuals with early-onset bipolar disorder

OBJECTIVES

To build a one-year risk calculator (RC) to predict individualized risk for suicide attempt in early-onset bipolar disorder.

METHODS

Youth numbering 394 with bipolar disorder who completed ≥2 follow-up assessments (median follow-up length = 13.1 years) in the longitudinal Course and Outcome of Bipolar Youth (COBY) study were included. Suicide attempt over follow-up was assessed via the A-LIFE Self-Injurious/Suicidal Behavior scale. Predictors from the literature on suicidal behavior in bipolar disorder that are readily assessed in clinical practice were selected and trichotomized as appropriate (presence past 6 months/lifetime history only/no lifetime history). The RC was trained via boosted multinomial classification trees; predictions were calibrated via Platt scaling. Half of the sample was used to train, and the other half to independently test the RC.

RESULTS

There were 249 suicide attempts among 106 individuals. Ten predictors accounted for >90% of the cross-validated relative influence in the model (AUC = 0.82; in order of relative influence): (1) age of mood disorder onset; (2) non-suicidal self-injurious behavior (trichotomized); (3) current age; (4) psychosis (trichotomized); (5) socioeconomic status; (6) most severe depressive symptoms in past 6 months (trichotomized none/subthreshold/threshold); (7) history of suicide attempt (trichotomized); (8) family history of suicidal behavior; (9) substance use disorder (trichotomized); (10) lifetime history of physical/sexual abuse. For all trichotomized variables, presence in the past 6 months reliably predicted higher risk than lifetime history.

CONCLUSIONS

This RC holds promise as a clinical and research tool for prospective identification of individualized high-risk periods for suicide attempt in early-onset bipolar disorder.

Assessing Suicidal Ideation in Young People With Depression: Factor Structure of the Suicidal Ideation Questionnaire

Evaluating suicidal ideation in young people seeking mental health treatment is an important component of clinical assessment and treatment planning. To reduce the burden of youth suicide, we need to improve our understanding of suicidal ideation, its underlying constructs, and how ideation translates into suicidal behaviour. Using exploratory factor analysis, we investigated the dimensionality of the Suicidal Ideation Questionnaire (SIQ) among 273 participants aged 15-25 with Major Depressive Disorder. Area under the receiver operating characteristic curve (AUROC) analysis was used to explore associations between latent factors and actual suicidal behaviour. Findings suggested that the SIQ assesses multiple factors underlying suicidal ideation. AUROC analyses demonstrated that latent factors relating to both active and passive suicidal ideation predicted past-month suicidal behaviour and suicide attempt. These findings contribute to an improved understanding of the complexities of suicidal ideation and relationships with suicidal behaviour in young people with depression.

Patient expectations of and experiences with a suicide risk identification algorithm in clinical practice

BACKGROUND

Suicide risk prediction models derived from electronic health records (EHR) and insurance claims are a novel innovation in suicide prevention but patient perspectives on their use have been understudied.

METHODS

In this qualitative study, between March and November 2020, 62 patients were interviewed from three health systems: one anticipating implementation of an EHR-derived suicide risk prediction model and two others piloting different implementation approaches. Site-tailored interview guides focused on patients' perceptions of this technology, concerns, and preferences for and experiences with suicide risk prediction model implementation in clinical practice. A constant comparative analytic approach was used to derive themes.

RESULTS

Interview participants were generally supportive of suicide risk prediction models derived from EHR data. Concerns included apprehension about inducing anxiety and suicidal thoughts, or triggering coercive treatment, particularly among those who reported prior negative experiences seeking mental health care. Participants who were engaged in mental health care or case management expected to be asked about their suicide risk and largely appreciated suicide risk conversations, particularly by clinicians comfortable discussing suicidality.

CONCLUSION

Most patients approved of suicide risk models that use EHR data to identify patients at-risk for suicide. As health systems proceed to implement such models, patient-centered care would involve dialogue initiated by clinicians experienced with assessing suicide risk during virtual or in person care encounters. Health systems should proactively monitor for negative consequences that result from risk model implementation to protect patient trust.

Artificial Intelligence Tools for Suicide Prevention in Adolescents and Young Adults

Background:

Artificial Intelligence is making a significant transformation in human lives. Its application in the medical and healthcare field has been also observed making an impact and improving overall outcomes. There has been a quest for similar processes in mental health due to the lack of observable changes in the areas of suicide prevention. In the last five years, there has been an emerging body of empirical research applying the technology of artificial intelligence (AI) and machine learning (ML) in mental health.

Objective:

To review the clinical applicability of the AI/ML-based tools in suicide prevention.

Methods:

The compelling question of predicting suicidality has been the focus of this research. We performed a broad literature search and then identified 36 articles relevant to meet the objectives of this review. We review the available evidence and provide a brief overview of the advances in this field.

Conclusion:

In the last five years, there has been more evidence supporting the implementation of these algorithms in clinical practice. Its current clinical utility is limited to using electronic health records and could be highly effective in conjunction with existing tools for suicide prevention. Other potential sources of relevant data include smart devices and social network sites. There are some serious questions about data privacy and ethics which need more attention while developing these new modalities in suicide research.

Suicide Screening in a Large Pediatric Emergency Department: Results, Feasibility, and Lessons Learned

OBJECTIVE

This study examined the feasibility of screening all patients entering the ED using the Columbia-Suicide Severity Rating Scale as well as examining the rates of suicide ideation and attempts endorsed by adolescents who present at the ED.

METHODS

This study used a sample of 12,113 patients between the ages of 11 and 19 years.

RESULTS

Results revealed that 13.5% of the participants endorsed passive suicide ideation in the month leading up to their ED visit and 11.3% of the participants reported active ideation in the prior month. Results also revealed that patients whose chief complaints were coded as psychiatric or medical trauma were more likely to endorse either active or passive suicidal ideation than other presenting problems. Patients with a psychiatric or medical trauma chief complaint were also more likely to report lifetime suicidal behavior and suicidal behavior 3 months before the ED visit.

CONCLUSIONS

In addition to findings, implications, feasibility, and lessons learned are discussed for other institutions or departments considering implementation of a widespread screening.Highlights:• Suicide screenings were implemented in a large pediatric emergency department.• One in 5 endorsed suicidal ideation or behavior regardless of presenting problem.• Feasibility and lessons learned are discussed for others hoping to implement a widespread screening.

Predictors of Intentional Self -Harm Among Medicaid Mental Health Clinic Clients In New York

BACKGROUND

Behavioral health outpatients are at risk for self-harm. Identifying individuals or combination of risk factors could discriminate those at elevated risk for self-harm.

METHODS

The study population (N = 248,491) included New York State Medicaid-enrolled individuals aged 10 to 64 with mental health clinic services between November 1, 2015 to November 1, 2016. Self-harm episodes were defined using ICD-10 codes from emergency department and inpatient visits. Multi-predictor logistic regression models were fit on a subsample of the data and compared to a testing sample based on discrimination performance (Area Under the Curve or AUC).

RESULTS

Of N = 248,491 patients, 4,224 (1.70%) had an episode of intentional self-harm. Factors associated with increased self-harm risk were age 17-25, being female and having recent diagnoses of depression (AOR=4.3, 95%CI: 3.6-5.0), personality disorder (AOR=4.2, 95%CI: 2.9-6.1), or substance use disorder (AOR=3.4, 95%CI: 2.7-4.3) within the last month. A multi-predictor logistic regression model including demographics and new psychiatric diagnoses within 90 days prior to index date had good discrimination and outperformed competitor models on a testing sample (AUC=0.86, 95%CI:0.85-0.87).

LIMITATIONS

New York State Medicaid data may not be generalizable to the entire U.S population. ICD-10 codes do not allow distinction between self-harm with and without intent to die.

CONCLUSIONS

Our results highlight the usefulness of recency of new psychiatric diagnoses, in predicting the magnitude and timing of intentional self-harm risk. An algorithm based on this finding could enhance clinical assessments support screening, intervention and outreach programs that are at the heart of a Zero Suicide prevention model.

Suicide prediction among men and women with depression: A population-based study

BACKGROUND

Accurate identification of persons at risk of suicide is challenging because suicide is a rare outcome with a multifactorial origin. The purpose of this study was to predict suicide among persons with depression using machine learning methods.

METHODS

A case-cohort study was conducted in Denmark between January 1, 1995 and December 31, 2015. Cases were all persons who died by suicide and had an incident depression diagnosis in Denmark (n = 2,774). The comparison subcohort was a 5% random sample of all individuals in Denmark at baseline, restricted to persons with an incident depression diagnosis during the study period (n = 11,963). Classification trees and random forests were used to predict suicide.

RESULTS

In men with depression, there was a high risk of suicide among those who were prescribed other analgesics and antipyretics (i.e., non-opioid analgesics such as acetaminophen), prescribed hypnotics and sedatives, and diagnosed with a poisoning (n = 96; risk = 81%). In women with depression, there was an elevated risk of suicide among those who were prescribed other analgesics and antipyretics, anxiolytics, and hypnotics and sedatives, but were not diagnosed with poisoning nor cerebrovascular diseases (n = 338; risk = 58%).

DISCUSSION

Psychiatric disorders and their associated medications were strongly indicative of suicide risk. Notably, anti-inflammatory medications (e.g., acetaminophen) prescriptions, which are used to treat chronic pain and illnesses, were associated with suicide risk in persons with depression. Machine learning may advance our ability to predict suicide deaths.

Evaluation of the Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment Suicide Risk Modeling Clinical Program in the Veterans Health Administration

IMPORTANCE

The Veterans Health Administration (VHA) implemented a national clinical program using a suicide risk prediction algorithm, Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment (REACH VET), in which clinicians facilitate care enhancements for individuals identified in local top 0.1% suicide risk tiers. Evaluation studies are needed.

OBJECTIVE

To determine associations with treatment engagement, health care utilization, suicide attempts, safety plan documentation, and 6-month mortality.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study used triple differences analyses comparing 6-month changes in outcomes after vs before program entry for individuals entering the REACH VET program (March 2017-December 2018) vs a similarly identified top 0.1% suicide risk tier cohort from prior to program initiation (March 2014-December 2015), adjusting for trends across subthreshold cohorts. Subcohort analyses (including individuals from March 2017-June 2018) evaluated difference-in-differences for cause-specific mortality using death certificate data. The subthreshold cohorts included individuals in the top 0.3% to 0.1% suicide risk tier, below the threshold for REACH VET eligibility, from the concurrent REACH VET period and from the pre-REACH VET period. Data were analyzed from December 2019 through September 2021.

EXPOSURES

REACH VET-designated clinicians treatment reevaluation and outreach for care enhancements, including safety planning, increased monitoring, and interventions to enhance coping.

MAIN OUTCOMES AND MEASURES

Process outcomes included VHA scheduled, completed, and missed appointments; mental health visits; and safety plan documentation and documentation within 6 months for individuals without plans within the prior 2 years. Clinical outcomes included mental health admissions, emergency department visits, nonfatal suicide attempts, and all-cause, suicide, and nonsuicide external-cause mortality.

RESULTS

A total of 173 313 individuals (mean [SD] age, 51.0 [14.7] years; 161 264 [93.1%] men and 12 049 [7.0%] women) were included in analyses, including 40 816 individuals eligible for REACH VET care and 36 604 individuals from the pre-REACH VET period in the top 0.1% of suicide risk. The REACH VET intervention was associated with significant increases in completed outpatient appointments (adjusted triple difference [ATD], 0.31; 95% CI, 0.06 to 0.55) and proportion of individuals with new safety plans (ATD, 0.08; 95% CI, 0.06 to 0.10) and reductions in mental health admissions (ATD, -0.08; 95% CI, -0.10 to -0.05), emergency department visits (ADT, -0.03; 95% CI, -0.06 to -0.01), and suicide attempts (ADT, -0.05; 95% CI, -0.06 to -0.03). Subcohort analyses did not identify differences in suicide or all-cause mortality (eg, age-and-sex-adjusted difference-in-difference for suicide mortality, 0.0007; 95% CI, -0.0006 to 0.0019).

CONCLUSIONS AND RELEVANCE

These findings suggest that REACH VET implementation was associated with greater treatment engagement and new safety plan documentation and fewer mental health admissions, emergency department visits, and suicide attempts. Clinical programs using risk modeling may be effective tools to support care enhancements and risk reduction.

Will Big Data and personalized medicine do the gender dimension justice?

Over the last decade, humans have produced each year as much data as were produced throughout the entire history of humankind. These data, in quantities that exceed current analytical capabilities, have been described as “the new oil,” an incomparable source of value. This is true for healthcare, as well. Conducting analyses of large, diverse, medical datasets promises the detection of previously unnoticed clinical correlations and new diagnostic or even therapeutic possibilities. However, using Big Data poses several problems, especially in terms of representing the uniqueness of each patient and expressing the differences between individuals, primarily gender and sex differences. The first two sections of the paper provide a definition of “Big Data” and illustrate the uses of Big Data in medicine. Subsequently, the paper explores the struggle to represent exhaustively the uniqueness of the patient through Big Data is highlighted prior to a deeper investigation of the digital representation of gender in personalized medicine. The final part of the paper put forward a series of recommendations for better approaching the complexity of gender in medical and clinical research involving Big Data for the creation or enhancement of personalized medicine services.

Reconciling Statistical and Clinicians' Predictions of Suicide Risk

Statistical models, including those based on electronic health records, can accurately identify patients at high risk for a suicide attempt or death, leading to implementation of risk prediction models for population-based suicide prevention in health systems. However, some have questioned whether statistical predictions can really inform clinical decisions. Appropriately reconciling statistical algorithms with traditional clinician assessment depends on whether predictions from these two methods are competing, complementary, or merely duplicative. In June 2019, the National Institute of Mental Health convened a meeting, "Identifying Research Priorities for Risk Algorithms Applications in Healthcare Settings to Improve Suicide Prevention." Here, participants of this meeting summarize key issues regarding the potential clinical application of suicide prediction models. The authors attempt to clarify the key conceptual and technical differences between traditional risk prediction by clinicians and predictions from statistical models, review the limited evidence regarding both the accuracy of and the concordance between these alternative methods of prediction, present a conceptual framework for understanding agreement and disagreement between statistical and clinician predictions, identify priorities for improving data regarding suicide risk, and propose priority questions for future research. Future suicide risk assessment will likely combine statistical prediction with traditional clinician assessment, but research is needed to determine the optimal combination of these two methods.

Artificial Intelligence for Mental Health Care: Clinical Applications, Barriers, Facilitators, and Artificial Wisdom

Artificial intelligence (AI) is increasingly employed in health care fields such as oncology, radiology, and dermatology. However, the use of AI in mental health care and neurobiological research has been modest. Given the high morbidity and mortality in people with psychiatric disorders, coupled with a worsening shortage of mental health care providers, there is an urgent need for AI to help identify high-risk individuals and provide interventions to prevent and treat mental illnesses. While published research on AI in neuropsychiatry is rather limited, there is a growing number of successful examples of AI's use with electronic health records, brain imaging, sensor-based monitoring systems, and social media platforms to predict, classify, or subgroup mental illnesses as well as problems such as suicidality. This article is the product of a study group held at the American College of Neuropsychopharmacology conference in 2019. It provides an overview of AI approaches in mental health care, seeking to help with clinical diagnosis, prognosis, and treatment, as well as clinical and technological challenges, focusing on multiple illustrative publications. Although AI could help redefine mental illnesses more objectively, identify them at a prodromal stage, personalize treatments, and empower patients in their own care, it must address issues of bias, privacy, transparency, and other ethical concerns. These aspirations reflect human wisdom, which is more strongly associated than intelligence with individual and societal well-being. Thus, the future AI or artificial wisdom could provide technology that enables more compassionate and ethically sound care to diverse groups of people.

A Brain Capital Grand Strategy: toward economic reimagination

‘I have not found among my possessions anything which I hold more dear than, or value so much as, my knowledge of the actions of great people, acquired by long experience in contemporary affairs, and a continual study of antiquity.’ The Prince, Machiavelli

Predicting post-discharge self-harm incidents using disease comorbidity networks: A retrospective machine learning study

BACKGROUND

Self-harm is preventable if the risk can be identified early. The co-occurrence of multiple diseases is related to self-harm risk. This study develops a comorbidity network-based deep learning framework to improve the prediction of individual self-harm.

METHODS

Between 01/01/2007-12/31/2010, we obtained 2,323 patients with self-harm records and 46,460 randomly sampled controls from 1,764,094 inpatients across 44 public hospitals in Hong Kong. 80% of the samples were randomly selected for model training, and the remaining 20% were set aside for model testing. We propose a novel patient embedding method, namely Dx2Vec (Diagnoses to Vector), based on the comorbidity network constructed by all historical diagnoses. Dx2Vec represents the comorbidity patterns among diseases and temporal patterns of historical admissions for each patient.

RESULTS

Experiments demonstrate that the Dx2Vec-based model outperforms the baseline deep learning model in identifying patients who would self-harm within 12 months (C-statistic: 0.89). The precision is 0.54 for positive cases and 0.98 for negative cases, whilst the recall is 0.72 for positive cases and 0.96 for negative cases. The model extracted the most predictive diagnoses, and pairwise comorbid diagnoses to help medical professionals identify patients with risk.

LIMITATIONS

The inpatient data does not contain lab test information.

CONCLUSIONS

Incorporation of a disease comorbidity network can significantly improve self-harm prediction performance, indicating that it is critical to consider comorbidity patterns in self-harm screening and prevention programs. The findings have the potential to be translated into effective self-harm screening systems.

Machine learning for suicidology: A practical review of exploratory and hypothesis-driven approaches

Machine learning is being used to discover models to predict the progression from suicidal ideation to action in clinical populations. While quantifiable improvements in prediction accuracy have been achieved over theory-driven efforts, models discovered through machine learning continue to fall short of clinical relevance. Thus, the value of machine learning for reaching this objective is hotly contested. We agree that machine learning, treated as a "black box" approach antithetical to theory-building, will not discover clinically relevant models of suicide. However, such models may be developed through deliberate synthesis of data- and theory-driven approaches. By providing an accessible overview of essential concepts and common methods, we highlight how generalizable models and scientific insight may be obtained by incorporating prior knowledge and expectations to machine learning research, drawing examples from suicidology. We then discuss challenges investigators will face when using machine learning to discover models of low prevalence outcomes, such as suicide.

Relative accuracy of social and medical determinants of suicide in electronic health records

OBJECTIVE

This paper compares the accuracy of predicting suicide from Social Determinants of Health (SDoH) or history of illness.

POPULATION STUDIED

5 313 965 Veterans who at least had two primary care visits between 2008 and 2016.

STUDY DESIGN

The dependent variable was suicide or intentional self-injury. The independent variables were 10 495 International Classification of Disease (ICD) Version 9 codes, age, and gender. The ICD codes included 40 V-codes used for measuring SDoH, such as family disruption, family history of substance abuse, lack of education, legal impediments, social isolation, unemployment, and homelessness. The sample was randomly divided into training (90 percent) and validation (10 percent) sets. Area under the receiver operating characteristic (AROC) was used to measure accuracy of predictions in the validation set.

PRINCIPAL FINDINGS

Separate analyses were done for inpatient and outpatient codes; the results were similar. In the hospitalized group, the mean age was 67.2 years, and 92.1 percent were male. The mean number of medical diagnostic codes during the study period was 37; and 12.9 percent had at least one SDoH V-code. At least one episode of suicide or intentional self-injury occurred in 1.89 percent of cases. SDoH V-codes, on average, elevated the risk of suicide or intentional self-injury by 24-fold (ranging from 4- to 86-fold). An index of 40 SDoH codes predicted suicide or intentional self-injury with an AROC of 0.64. An index of 10 445 medical diagnoses, without SDoH V-codes, had AROC of 0.77. The combined SDoH and medical diagnoses codes also had AROC of 0.77.

CONCLUSION

In predicting suicide or intentional self-harm, SDoH V-codes add negligible information beyond what is already available in medical diagnosis codes.

IMPLICATIONS FOR PRACTICE

Policies that affect SDoH (eg, housing policies, resilience training) may not have an impact on suicide rates, if they do not change the underlying medical causes of SDoH.

Identifying risk factors for mortality among patients previously hospitalized for a suicide attempt

Age-adjusted suicide rates in the US have increased over the past two decades across all age groups. The ability to identify risk factors for suicidal behavior is critical to selected and indicated prevention efforts among those at elevated risk of suicide. We used widely available statewide hospitalization data to identify and test the joint predictive power of clinical risk factors associated with death by suicide for patients previously hospitalized for a suicide attempt (N = 19,057). Twenty-eight clinical factors from the prior suicide attempt were found to be significantly associated with the hazard of subsequent suicide mortality. These risk factors and their two-way interactions were used to build a joint predictive model via stepwise regression, in which the predicted individual survival probability was found to be a valid measure of risk for later suicide death. A high-risk group with a four-fold increase in suicide mortality risk was identified based on the out-of-sample predicted survival probabilities. This study demonstrates that the combination of state-level hospital discharge and mortality data can be used to identify suicide attempters who are at high risk of subsequent suicide death.

Implementing Precision Psychiatry: A Systematic Review of Individualized Prediction Models for Clinical Practice

BACKGROUND

The impact of precision psychiatry for clinical practice has not been systematically appraised. This study aims to provide a comprehensive review of validated prediction models to estimate the individual risk of being affected with a condition (diagnostic), developing outcomes (prognostic), or responding to treatments (predictive) in mental disorders.

METHODS

PRISMA/RIGHT/CHARMS-compliant systematic review of the Web of Science, Cochrane Central Register of Reviews, and Ovid/PsycINFO databases from inception until July 21, 2019 (PROSPERO CRD42019155713) to identify diagnostic/prognostic/predictive prediction studies that reported individualized estimates in psychiatry and that were internally or externally validated or implemented. Random effect meta-regression analyses addressed the impact of several factors on the accuracy of prediction models.

FINDINGS

Literature search identified 584 prediction modeling studies, of which 89 were included. 10.4% of the total studies included prediction models internally validated (n = 61), 4.6% models externally validated (n = 27), and 0.2% (n = 1) models considered for implementation. Across validated prediction modeling studies (n = 88), 18.2% were diagnostic, 68.2% prognostic, and 13.6% predictive. The most frequently investigated condition was psychosis (36.4%), and the most frequently employed predictors clinical (69.5%). Unimodal compared to multimodal models (β = .29, P = .03) and diagnostic compared to prognostic (β = .84, p < .0001) and predictive (β = .87, P = .002) models were associated with increased accuracy.

INTERPRETATION

To date, several validated prediction models are available to support the diagnosis and prognosis of psychiatric conditions, in particular, psychosis, or to predict treatment response. Advancements of knowledge are limited by the lack of implementation research in real-world clinical practice. A new generation of implementation research is required to address this translational gap.

Artificial Intelligence and Suicide Prevention: A Systematic Review of Machine Learning Investigations

Suicide is a leading cause of death that defies prediction and challenges prevention efforts worldwide. Artificial intelligence (AI) and machine learning (ML) have emerged as a means of investigating large datasets to enhance risk detection. A systematic review of ML investigations evaluating suicidal behaviors was conducted using PubMed/MEDLINE, PsychInfo, Web-of-Science, and EMBASE, employing search strings and MeSH terms relevant to suicide and AI. Databases were supplemented by hand-search techniques and Google Scholar. Inclusion criteria: (1) journal article, available in English, (2) original investigation, (3) employment of AI/ML, (4) evaluation of a suicide risk outcome. N = 594 records were identified based on abstract search, and 25 hand-searched reports. N = 461 reports remained after duplicates were removed, n = 316 were excluded after abstract screening. Of n = 149 full-text articles assessed for eligibility, n = 87 were included for quantitative synthesis, grouped according to suicide behavior outcome. Reports varied widely in methodology and outcomes. Results suggest high levels of risk classification accuracy (>90%) and Area Under the Curve (AUC) in the prediction of suicidal behaviors. We report key findings and central limitations in the use of AI/ML frameworks to guide additional research, which hold the potential to impact suicide on broad scale.

Can machine-learning methods really help predict suicide?

PURPOSE OF REVIEW

In recent years there has been interest in the use of machine learning in suicide research in reaction to the failure of traditional statistical methods to produce clinically useful models of future suicide. The current review summarizes recent prediction studies in the suicide literature including those using machine learning approaches to understand what value these novel approaches add.

RECENT FINDINGS

Studies using machine learning to predict suicide deaths report area under the curve that are only modestly greater than, and sensitivities that are equal to, those reported in studies using more conventional predictive methods. Positive predictive value remains around 1% among the cohort studies with a base rate that was not inflated by case-control methodology.

SUMMARY

Machine learning or artificial intelligence may afford opportunities in mental health research and in the clinical care of suicidal patients. However, application of such techniques should be carefully considered to avoid repeating the mistakes of existing methodologies. Prediction studies using machine-learning methods have yet to make a major contribution to our understanding of the field and are unproven as clinically useful tools.

A day in the life of a psychiatrist in 2050: where will the algorithm take us?

Digital phenotyping (such as using live data from personal digital devices on sleep, activity and social media interactions) to monitor and interpret people's current mental state is a newly emerging development in psychiatry. This article offers an imaginary insight into its future potential for both psychiatrist and patient.

Suicide and suicide attempts in adults: exploring suicide risk 24 months after a psychiatric emergency room visit

Objective:

Suicide risk (including attempted and completed suicide) should be measured over short periods of time after contacting health services. The objective of this study was to identify the patterns of attempted and completed suicides within 24-months of a psychiatric emergency department visit, as well as to investigate predictive risk factors, including sociodemographic and clinical variables, previous suicidal behavior, and service utilization.

Method:

A convenience sample (n=147), recruited at a general hospital’s psychiatric emergency room, included patients with suicidal ideation, suicidal plans or previous suicide attempts. These patients were followed for 24 months, focusing on two main outcomes: attempted and completed suicides.

Results:

After six months there were no completed suicides and 36 suicide attempts, while after 24 months there were seven completed suicides and 69 suicide attempts. A final logistic regression model for suicide attempts at 24 months identified somatic pathology and the number of previous psychiatric hospitalizations as predictive factors, with a good area under the receiver operating characteristic curve.

Conclusions:

The findings showed distinct patterns of attempted and completed suicides over time, indicating the importance of a systematic multidisciplinary suicide risk evaluation in psychiatric emergency rooms.

Clinician-recalled quoted speech in electronic health records and risk of suicide attempt: a case-crossover study

OBJECTIVE

Clinician narrative style in electronic health records (EHR) has rarely been investigated. Clinicians sometimes record brief quotations from patients, possibly more frequently when higher risk is perceived. We investigated whether the frequency of quoted phrases in an EHR was higher in time periods closer to a suicide attempt.

DESIGN

A case-crossover study was conducted in a large mental health records database. A natural language processing tool was developed using regular expression matching to identify text occurring within quotation marks in the EHR.

SETTING

Electronic records from a large mental healthcare provider serving a geographic catchment of 1.3 million residents in South London were linked with hospitalisation data.

PARTICIPANTS

1503 individuals were identified as having a hospitalised suicide attempt from 1 April 2006 to 31 March 2017 with at least one document in both the case period (1-30 days prior to admission) and the control period (61-90 days prior to admission).

OUTCOME MEASURES

The number of quoted phrases in the control as compared with the case period.

RESULTS

Both attended (OR 1.05, 95% CI 1.02 to 1.08) and non-attended (OR 1.15, 95% CI 1.04 to 1.26) clinical appointments were independently higher in the case compared with control period, while there was no difference in mental healthcare hospitalisation (OR 0.99, 95% CI 0.98 to 1.01). In addition, there was no difference in the levels of quoted text between the comparison time periods (OR 1.09, 95% CI 0.91 to 1.30).

CONCLUSIONS

This study successfully developed an algorithm to identify quoted speech in text fields from routine mental healthcare records. Contrary to the hypothesis, no association between this exposure and proximity to a suicide attempt was found; however, further evaluation is warranted on the way in which clinician-perceived risk might be feasibly characterised from clinical text.

Reaching Those at Highest Risk for Suicide: Development of a Model Using Machine Learning Methods for use With Native American Communities

OBJECTIVE

Suicide prevention is a major priority in Native American communities. We used machine learning with community-based suicide surveillance data to better identify those most at risk.

METHOD

This study leverages data from the Celebrating Life program operated by the White Mountain Apache Tribe in Arizona and in partnership with Johns Hopkins University. We examined N = 2,390 individuals with a validated suicide-related event between 2006 and 2017. Predictors included 73 variables (e.g., demographics, educational history, past mental health, and substance use). The outcome was suicide attempt 6, 12, and 24 months after an initial event. We tested four algorithmic approaches using cross-validation.

RESULTS

Area under the curves ranged from AUC = 0.81 (95% CI ± 0.08) for the decision tree classifiers to AUC = 0.87 (95% CI ± 0.04) for the ridge regression, results that were considerably higher than a past suicide attempt (AUC = 0.57; 95% CI ± 0.08). Selecting a cutoff value based on risk concentration plots yielded 0.88 sensitivity, 0.72 specificity, and a positive predictive value of 0.12 for detecting an attempt 24 months postindex event.

CONCLUSION

These models substantially improved our ability to determine who was most at risk in this community. Further work is needed including developing clinical guidance and external validation.

Predicting death by suicide using administrative health care system data: Can recurrent neural network, one-dimensional convolutional neural network, and gradient boosted trees models improve prediction performance?

BACKGROUND

Suicide is a leading cause of death, particularly in younger persons, and this results in tremendous years of life lost.

OBJECTIVE

To compare the performance of recurrent neural networks, one-dimensional convolutional neural networks, and gradient boosted trees, with logistic regression and feedforward neural networks.

METHODS

The modeling dataset contained 3548 persons that died by suicide and 35,480 persons that did not die by suicide between 2000 and 2016. 101 predictors were selected, and these were assembled for each of the 40 quarters (10 years) prior to the quarter of death, resulting in 4040 predictors in total for each person. Model configurations were evaluated using 10-fold cross-validation.

RESULTS

The optimal recurrent neural network model configuration (AUC: 0.8407), one-dimensional convolutional neural network configuration (AUC: 0.8419), and XGB model configuration (AUC: 0.8493) all outperformed logistic regression (AUC: 0.8179). In addition to superior discrimination, the optimal XGB model configuration also achieved superior calibration.

CONCLUSIONS

Although the models developed in this study showed promise, further research is needed to determine the performance limits of statistical and machine learning models that quantify suicide risk, and to develop prediction models optimized for implementation in clinical settings. It appears that the XGB model class is the most promising in terms of discrimination, calibration, and computational expense.

LIMITATIONS

Many important predictors are not available in administrative data and this likely places a limit on how well prediction models developed with administrative data can perform.

Predicting death by suicide following an emergency department visit for parasuicide with administrative health care system data and machine learning

BACKGROUND

Suicide is a leading cause of death worldwide and results in a large number of person years of life lost. There is an opportunity to evaluate whether administrative health care system data and machine learning can quantify suicide risk in a clinical setting.

METHODS

The objective was to compare the performance of prediction models that quantify the risk of death by suicide within 90 days of an ED visit for parasuicide with predictors available in administrative health care system data.The modeling dataset was assembled from 5 administrative health care data systems. The data systems contained nearly all of the physician visits, ambulatory care visits, inpatient hospitalizations, and community pharmacy dispenses, of nearly the entire 4.07 million persons in Alberta, Canada. 101 predictors were selected, and these were assembled for each of the 8 quarters (2 years) prior to the quarter of death, resulting in 808 predictors in total for each person. Prediction model performance was validated with 10-fold cross-validation.

FINDINGS

The optimal gradient boosted trees prediction model achieved promising discrimination (AUC: 0.88) and calibration that could lead to clinical applications. The 5 most important predictors in the optimal gradient boosted trees model each came from a different administrative health care data system.

INTERPRETATION

The combination of predictors from multiple administrative data systems and the combination of personal and ecologic predictors resulted in promising prediction performance. Further research is needed to develop prediction models optimized for implementation in clinical settings.

FUNDING

There was no funding for this study.

A Framework for Applying Natural Language Processing in Digital Health Interventions

BACKGROUND

Digital health interventions (DHIs) are poised to reduce target symptoms in a scalable, affordable, and empirically supported way. DHIs that involve coaching or clinical support often collect text data from 2 sources: (1) open correspondence between users and the trained practitioners supporting them through a messaging system and (2) text data recorded during the intervention by users, such as diary entries. Natural language processing (NLP) offers methods for analyzing text, augmenting the understanding of intervention effects, and informing therapeutic decision making.

OBJECTIVE

This study aimed to present a technical framework that supports the automated analysis of both types of text data often present in DHIs. This framework generates text features and helps to build statistical models to predict target variables, including user engagement, symptom change, and therapeutic outcomes.

METHODS

We first discussed various NLP techniques and demonstrated how they are implemented in the presented framework. We then applied the framework in a case study of the Healthy Body Image Program, a Web-based intervention trial for eating disorders (EDs). A total of 372 participants who screened positive for an ED received a DHI aimed at reducing ED psychopathology (including binge eating and purging behaviors) and improving body image. These users generated 37,228 intervention text snippets and exchanged 4285 user-coach messages, which were analyzed using the proposed model.

RESULTS

We applied the framework to predict binge eating behavior, resulting in an area under the curve between 0.57 (when applied to new users) and 0.72 (when applied to new symptom reports of known users). In addition, initial evidence indicated that specific text features predicted the therapeutic outcome of reducing ED symptoms.

CONCLUSIONS

The case study demonstrates the usefulness of a structured approach to text data analytics. NLP techniques improve the prediction of symptom changes in DHIs. We present a technical framework that can be easily applied in other clinical trials and clinical presentations and encourage other groups to apply the framework in similar contexts.

What We Can Learn From Failure: An EHR-Based Child Protection Alert System

This study aimed to evaluate the efficacy of a newly implemented Child Protection Alert System (CPAS) that utilizes triggering diagnoses to identify children who have been confirmed/strongly suspected as maltreated. We retrospectively reviewed electronic health records (EHRs) of 666 patients evaluated by our institution's child protection team between 2009 and 2014. We examined each EHR for the presence of a pop-up alert, a persistent text-based visual alert, and diagnoses denoting child maltreatment. Diagnostic accuracy of the CPAS for child maltreatment identification was assessed. Of 323 patients for whom child maltreatment was confirmed/strongly suspected, 21.7% (70/323) had a qualifying longitudinal diagnosis listed. The pop-up alert fired in 14% of cases (45/323) with a sensitivity and specificity of 13.9% (95% CI [10.4%, 18.2%]) and 100% (95% CI [98.9%, 100.0%]), respectively. The text-based visual alert displayed in 44 of 45 cases. The CPAS is a novel simple way to support clinical decision-making to identify and protect children at risk of (re)abuse. This study highlights multiple barriers that must be overcome to effectively design and implement a CPAS to protect at-risk children.

Barriers for the Research, Prevention, and Treatment of Suicidal Behavior

Efforts in research, prevention, and treatment of suicidal behavior have produced mixed results. One of the main barriers to combating suicidal behavior lies in the very conceptualization of suicide, a phenomenon that is at once sociological, psychiatric, and even philosophical, and one that has not always been included in the field of health care. There are also many barriers at the social level, ranging from stigma against people with suicidal behavior to stigma towards psychiatric care, as well as the controversial role of the media. The media plays an important role in society and depending on its attitude it can be either beneficial or harmful in our fight against suicidal behavior. Differences between countries - in the provision of resources, in the way of understanding the phenomenon or in the manner of providing official figures - pose an additional challenge to suicide prevention on a global level. In the field of research, predicting suicidal behavior by identifying effective risk markers is severely hampered by the low occurrence of suicide in the population, which limits the statistical power of studies. The authors recommend combining various risk factors to build predictive models. This, in addition to employing increasingly precise machine learning techniques, is a step in the right direction, although there is still a long way to go before the expected results can be obtained. Finally, adequate training of health professionals, both specialized and non-specialized, as well as gatekeeper training, is crucial for implementing suicide prevention strategies in the population.

Using Administrative Data to Predict Suicide After Psychiatric Hospitalization in the Veterans Health Administration System

There is a very high suicide rate in the year after psychiatric hospital discharge. Intensive postdischarge case management programs can address this problem but are not cost-effective for all patients. This issue can be addressed by developing a risk model to predict which inpatients might need such a program. We developed such a model for the 391,018 short-term psychiatric hospital admissions of US veterans in Veterans Health Administration (VHA) hospitals 2010-2013. Records were linked with the National Death Index to determine suicide within 12 months of hospital discharge (n=771). The Super Learner ensemble machine learning method was used to predict these suicides for time horizon between 1 week and 12 months after discharge in a 70% training sample. Accuracy was validated in the remaining 30% holdout sample. Predictors included VHA administrative variables and small area geocode data linked to patient home addresses. The models had AUC=.79-.82 for time horizons between 1 week and 6 months and AUC=.74 for 12 months. An analysis of operating characteristics showed that 22.4%-32.2% of patients who died by suicide would have been reached if intensive case management was provided to the 5% of patients with highest predicted suicide risk. Positive predictive value (PPV) at this higher threshold ranged from 1.2% over 12 months to 3.8% per case manager year over 1 week. Focusing on the low end of the risk spectrum, the 40% of patients classified as having lowest risk account for 0%-9.7% of suicides across time horizons. Variable importance analysis shows that 51.1% of model performance is due to psychopathological risk factors accounted, 26.2% to social determinants of health, 14.8% to prior history of suicidal behaviors, and 6.6% to physical disorders. The paper closes with a discussion of next steps in refining the model and prospects for developing a parallel precision treatment model.

Differences in Health Care Utilization in the Year Before Suicide Death: A Population-Based Case-Control Study

OBJECTIVE

To compare health care usage between suicide decedents and living controls in the year before suicide in a large representative US population.

PATIENTS AND METHODS

Cases (n=1221) and controls (n=3663) belonged to an integrated health care system from January 1, 2009, through December 31, 2014. Cases and controls were matched for age and sex in a 1:3 ratio, with diagnostic and/or billing codes used to enumerate and classify health care visits in the year before the index suicide. Matched analysis via conditional logistic regression related odds of suicide to visit type. A generalized estimating equation model was used to compare timing and frequency of visits between cases and controls.

RESULTS

In the year before death, cases had an increased odds of both inpatient hospitalizations and emergency department nonmental health visits (odds ratio [OR], 1.55; 95% CI, 1.27-1.88; P<.001 and OR, 1.42; 95% CI, 1.26-1.60; P<.001) but not outpatient nonmental health visits (OR, 1.00; 95% CI, 0.99-1.01; P=.63). Decedents increased health care utilization closer to suicide death and had significantly more health care visits than did controls 3 months before suicide (6 vs 2; P=.01) but not 9 to 12 months before suicide (4 vs 2; P=.07). At all time points, cases used more mental health care services than did controls.

CONCLUSION

Compared with controls, suicide decedents had emergency department visits and more inpatient hospitalizations, both mental health and nonmental health related. As death approached, cases' frequency of health care usage increased. The only category in which cases and controls did not differ was in the frequency of outpatient nonmental health visits.

Predicting death by suicide using administrative health care system data: Can feedforward neural network models improve upon logistic regression models?

BACKGROUND

Suicide is a leading cause of death worldwide. With the increasing volume of administrative health care data, there is an opportunity to evaluate whether machine learning models can improve upon statistical models for quantifying suicide risk.

OBJECTIVE

To compare the relative performance of logistic regression and single hidden layer feedforward neural network models that quantify suicide risk with predictors available in administrative health care system data.

METHODS

The modeling dataset contained 3548 persons that died by suicide and 35,480 persons that did not die by suicide between 2000 and 2016. 101 predictors were selected, and these were assembled for each of the 40 quarters (10 years) prior to the quarter of death, resulting in 4040 predictors in total for each person. Logistic regression and single hidden layer feedforward neural network model configurations were evaluated using 10-fold cross-validation.

RESULTS

The optimal feedforward neural network model configuration (AUC: 0.8352) outperformed logistic regression (AUC: 0.8179).

LIMITATIONS

Many important predictors are not available in administrative data and this likely places a limit on how well prediction models developed with administrative data can perform.

CONCLUSIONS

Although the models developed in this study showed promise, further research is needed to determine the performance limits of statistical and machine learning models that quantify suicide risk, and to develop prediction models optimized for implementation in clinical settings.

Predicting Future Suicide: Clinician Opinion versus a Standardized Assessment Tool

OBJECTIVE

To compare the effectiveness of clinician prediction of risk to a standardized assessment of presentation status.

METHODS

All adult psychiatry emergency department consults in the two main hospitals in Winnipeg, Canada, were assessed using a standardized form (n = 5,376). This form includes two risk scales for a gestalt physician assessment of risk (Suicide Likelihood scale, suicide Attempt Likelihood scale) and the Columbia Classification Algorithm of Suicide Assessment (C-CASA). Regression determined whether assessments predicted future suicide attempts and deaths. The area under the curve (AUC) determined the prediction accuracy of these methods.

RESULTS

Although the regression results were significant, the AUCs were either moderate or poor. Clinician assessment was not effective at predicting deaths (AUC = .546, .36-.73), but moderately accurate at predicting future attempts (AUC = .728, .66-.79). C-CASA assessment was moderately accurate at predicting both attempts and deaths (AUC = .666 and .678).

CONCLUSIONS

Clinician assessment does not significantly outperform a simple assessment of the occurrence of suicidal thoughts and behaviors during presentation to the emergency department. Behavior-based standardized assessments should be further researched in this field. Assessment of suicidality at presentation using C-CASA or similar assessment should be standard for psychiatric patients assessed in the emergency department.

Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation

IMPORTANCE

Suicide prediction models have the potential to improve the identification of patients at heightened suicide risk by using predictive algorithms on large-scale data sources. Suicide prediction models are being developed for use across enterprise-level health care systems including the US Department of Defense, US Department of Veterans Affairs, and Kaiser Permanente.

OBJECTIVES

To evaluate the diagnostic accuracy of suicide prediction models in predicting suicide and suicide attempts and to simulate the effects of implementing suicide prediction models using population-level estimates of suicide rates.

EVIDENCE REVIEW

A systematic literature search was conducted in MEDLINE, PsycINFO, Embase, and the Cochrane Library to identify research evaluating the predictive accuracy of suicide prediction models in identifying patients at high risk for a suicide attempt or death by suicide. Each database was searched from inception to August 21, 2018. The search strategy included search terms for suicidal behavior, risk prediction, and predictive modeling. Reference lists of included studies were also screened. Two reviewers independently screened and evaluated eligible studies.

FINDINGS

From a total of 7306 abstracts reviewed, 17 cohort studies met the inclusion criteria, representing 64 unique prediction models across 5 countries with more than 14 million participants. The research quality of the included studies was generally high. Global classification accuracy was good (≥0.80 in most models), while the predictive validity associated with a positive result for suicide mortality was extremely low (≤0.01 in most models). Simulations of the results suggest very low positive predictive values across a variety of population assessment characteristics.

CONCLUSIONS AND RELEVANCE

To date, suicide prediction models produce accurate overall classification models, but their accuracy of predicting a future event is near 0. Several critical concerns remain unaddressed, precluding their readiness for clinical applications across health systems.

Severity and Variability of Depression Symptoms Predicting Suicide Attempt in High-Risk Individuals

IMPORTANCE

Predicting suicidal behavior continues to be among the most challenging tasks in psychiatry.

OBJECTIVES

To examine the trajectories of clinical predictors of suicide attempt (specifically, depression symptoms, hopelessness, impulsivity, aggression, impulsive aggression, and irritability) for their ability to predict suicide attempt and to compute a risk score for suicide attempts.

DESIGN, SETTING, AND PARTICIPANTS

This is a longitudinal study of the offspring of parents (or probands) with mood disorders who were recruited from inpatient units at Western Psychiatric Institute and Clinic (Pittsburgh) and New York State Psychiatric Institute. Participants were recruited from July 15, 1997, to September 6, 2005, and were followed up through January 21, 2014. Probands and offspring (n = 663) were interviewed at baseline and at yearly follow-ups for 12 years. Lifetime and current psychiatric disorders were assessed, and self-reported questionnaires were administered. Model evaluation used 10-fold cross-validation, which split the entire data set into 10 equal parts, fit the model to 90% of the data (training set), and assessed it on the remaining 10% (test set) and repeated that process 10 times. Preliminary analyses were performed from July 20, 2015, to October 5, 2016. Additional analyses were conducted from July 26, 2017, to July 24, 2018.

MAIN OUTCOMES AND MEASURES

The broad definition of suicide attempt included actual, interrupted, and aborted attempts as well as suicidal ideation that prompted emergency referrals during the study. The narrow definition referred to actual attempt only.

RESULTS

The sample of offspring (n = 663) was almost equally distributed by sex (316 female [47.7%]) and had a mean (SD) age of 23.8 (8.5) years at the time of censored observations. Among the 663 offspring, 71 (10.7%) had suicide attempts over the course of the study. The trajectory of depression symptoms with the highest mean scores and variability over time was the only trajectory to predict suicide attempt (odds ratio [OR], 4.72; 95% CI, 1.47-15.21; P = .01). In addition, we identified the following predictors: younger age (OR, 0.82; 95% CI, 0.74-0.90; P < .001), lifetime history of unipolar disorder (OR, 4.71; 95% CI, 1.63-13.58; P = .004), lifetime history of bipolar disorder (OR, 3.4; 95% CI, 0.96-12.04; P = .06), history of childhood abuse (OR, 2.98; 95% CI, 1.40-6.38; P = .01), and proband actual attempt (OR, 2.24; 95% CI, 1.06-4.75; P = .04). Endorsing a score of 3 or higher on the risk score tool resulted in high sensitivity (87.3%) and moderate specificity (63%; area under the curve = 0.80).

CONCLUSIONS AND RELEVANCE

The specific predictors of suicide attempt identified are those that clinicians already assess during routine psychiatric evaluations; monitoring and treating depression symptoms to reduce their severity and fluctuation may attenuate the risk for suicidal behavior.

Delayed Recognition of Acute Stroke by Emergency Department Staff Following Failure to Activate Stroke by Emergency Medical Services

Introduction

Early recognition and pre-notification by emergency medical services (EMS) improves the timeliness of emergency department (ED) stroke care; however, little is known regarding the effects on care should EMS providers fail to pre-notify. We sought to determine if potential stroke patients transported by EMS, but for whom EMS did not provide pre-notification, suffer delays in ED door-to-stroke-team activation (DTA) as compared to the other available cohort of patients for whom the ED is not pre-notified–those arriving by private vehicle.

Methods

We queried our prospective stroke registry to identify consecutive stroke team activation patients over 12 months and retrospectively reviewed the electronic health record for each patient to validate registry data and abstract other clinical and operational data. We compared patients arriving by private vehicle to those arriving by EMS without pre-notification, and we employed a multivariable, penalized regression model to assess the probability of meeting the national DTA goal of ≤15 minutes, controlling for a variety of clinical factors.

Results

Our inclusion criteria were met by 200 patients. Overall performance of the regression model was excellent (area under the curve 0.929). Arrival via EMS without pre-notification, compared to arrival by private vehicle, was associated with an adjusted risk ratio of 0.55 (95% confidence interval, 0.27–0.96) for achieving DTA ≤ 15 minutes.

Conclusion

Our single-center data demonstrate that potential stroke patients arriving via EMS without pre-notification are less likely to meet the national DTA goal than patients arriving via other means. These data suggest a negative, unintended consequence of otherwise highly successful EMS efforts to improve stroke care, the root of which may be ED staff over-reliance on EMS for stroke recognition.

Predictors of self-harm emergency department visits in adolescents: A statewide longitudinal study

OBJECTIVE

This study investigated patient- and area-level characteristics associated with adolescent emergency department (ED) patients' risk of subsequent ED visits for self-harm.

METHOD

Retrospective analysis of adolescent patients presenting to a California ED in 2010 (n = 480,706) was conducted using statewide, all-payer, individually linkable administrative data. We examined associations between multiple predictors of interest (patient sociodemographic factors, prior ED utilization, and residential mobility; and area-level characteristics) and odds of a self-harm ED visit in 2010. Patients with any self-harm in 2010 were followed up over several years to assess predictors of recurrent self-harm.

RESULTS

Self-harm patients (n = 5539) were significantly more likely than control patients (n = 16,617) to have prior histories of ED utilization, particularly for mental health problems, substance abuse, and injuries. Residential mobility also increased risk of self-harm, but racial/ethnic minority status and residence in a disadvantaged zipcode decreased risk. Five-year cumulative incidence of recurrent self-harm was 19.3%. Admission as an inpatient at index visit, Medicaid insurance, and prior ED utilization for psychiatric problems or injury all increased recurrent self-harm risk.

CONCLUSIONS

A range of patient- and area-level characteristics observable in ED settings are associated with risk for subsequent self-harm among adolescents, suggesting new targets for intervention in this clinical context.

Risk Assessment Tools and Data-Driven Approaches for Predicting and Preventing Suicidal Behavior

Risk assessment of suicidal behavior is a time-consuming but notoriously inaccurate activity for mental health services globally. In the last 50 years a large number of tools have been designed for suicide risk assessment, and tested in a wide variety of populations, but studies show that these tools suffer from low positive predictive values. More recently, advances in research fields such as machine learning and natural language processing applied on large datasets have shown promising results for health care, and may enable an important shift in advancing precision medicine. In this conceptual review, we discuss established risk assessment tools and examples of novel data-driven approaches that have been used for identification of suicidal behavior and risk. We provide a perspective on the strengths and weaknesses of these applications to mental health-related data, and suggest research directions to enable improvement in clinical practice.

Digital Interventions for Mental Disorders: Key Features, Efficacy, and Potential for Artificial Intelligence Applications

Mental disorders are highly prevalent and often remain untreated. Many limitations of conventional face-to-face psychological interventions could potentially be overcome through Internet-based and mobile-based interventions (IMIs). This chapter introduces core features of IMIs, describes areas of application, presents evidence on the efficacy of IMIs as well as potential effect mechanisms, and delineates how Artificial Intelligence combined with IMIs may improve current practices in the prevention and treatment of mental disorders in adults. Meta-analyses of randomized controlled trials clearly show that therapist-guided IMIs can be highly effective for a broad range of mental health problems. Whether the effects of unguided IMIs are also clinically relevant, particularly under routine care conditions, is less clear. First studies on IMIs for the prevention of mental disorders have shown promising results. Despite limitations and challenges, IMIs are increasingly implemented into routine care worldwide. IMIs are also well suited for applications of Artificial Intelligence and Machine Learning, which provides ample opportunities to improve the identification and treatment of mental disorders. Together with methodological innovations, these approaches may also deepen our understanding of how psychological interventions work, and why. Ethical and professional restraints as well as potential contraindications of IMIs, however, should also be considered. In sum, IMIs have a high potential for improving the prevention and treatment of mental health disorders across various indications, settings, and populations. Therefore, implementing IMIs into routine care as both adjunct and alternative to face-to-face treatment is highly desirable. Technological advancements may further enhance the variability and flexibility of IMIs, and thus even further increase their impact in people's lives in the future.

Applying Machine Learning to Linked Administrative and Clinical Data to Enhance the Detection of Homelessness among Vulnerable Veterans

U.S. military veterans who were discharged from service for misconduct are at high risk for homelessness. Stratifying homelessness risk based on both military service factors and clinical characteristics could facilitate targeted provision of preventive services to those at critical risk. Using administrative data from the Department of Defense and Veterans Health Administration for 25,821 misconduct-discharged Veterans, we developed a dataset that included demographic and clinical characteristics corresponding to 12-months, 3-months, and 1-month preceding the first documentation of homelessness (or a matched index encounter for those without homelessness). Clinical time-trend features were extracted and included as additional model inputs. We developed several random forest models to classify homelessness risk. Models based on 1- and 3-months of data performed roughly as well as those based on 12-months of data. In best-performing models, 70% of those identified as at high-risk became homeless; 30% identified as at moderate risk became homeless (AUC=0.80; recall=0.64, specificity=0.82). Findings suggest the viability of risk stratification for targeting resources.