Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans health Administration

Artificial Intelligence: Review of Current and Future Applications in Medicine

Background

The role of artificial intelligence (AI) in health care is expanding rapidly. Currently, there are at least 29 US Food and Drug Administration-approved AI health care devices that apply to numerous medical specialties and many more are in development.

Observations

With increasing expectations for all health care sectors to deliver timely, fiscally-responsible, high-quality health care, AI has potential utility in numerous areas, such as image analysis, improved workflow and efficiency, public health, and epidemiology, to aid in processing large volumes of patient and medical data. In this review, we describe basic terminology, principles, and general AI applications relating to health care. We then discuss current and future applications for a variety of medical specialties. Finally, we discuss the future potential of AI along with the potential risks and limitations of current AI technology.

Conclusions

AI can improve diagnostic accuracy, increase patient safety, assist with patient triage, monitor disease progression, and assist with treatment decisions.

Rates and Correlates of Suicidality in VA Intensive Case Management Programs

There has been extensive concern about suicide among veterans, but no study has examined rates and correlates of suicidality in the highly vulnerable group of veterans receiving Veterans Health Administration (VHA) intensive case management services. Veterans participating in a national program evaluation were surveyed at the time of program entry and 6 months later. Sociodemographic and clinical characteristics were documented along with elements of program service delivery. Chi square tests were used to compare rates of suicidality (defined as either having made or threatened an attempt) at baseline and at the 6-month follow-up. Analysis of variance was also used to compare suicidal and non-suicidal veterans at follow-up. Logistic regression analysis was then used to identify independent correlates of suicidality 6 months after program entry. Among the 9921 veterans who later completed follow-up assessments 989 (10.0%) had reported suicidal behavior at program entry as compared to only 250 (2.51%) at 6 months (p < 0.0001). Multivariable logistic regression analysis showed suicidality at 6 months to be associated with suicidality at admission, increased subjective distress on the Brief Symptom Inventory (especially on depression items), violent behavior and decreased quality of life since admission, along with a greater likelihood of receiving crisis intervention, but not other services. Among veterans receiving intensive case management services from VHA, suicidal behavior declined by 75% from admission to 6 months (10-2.5%) and was associated with suicidality prior to program entry, worsening subjective symptoms and greater receipt of crisis intervention services.

AI-Based Prediction and Prevention of Psychological and Behavioral Changes in Ex-COVID-19 Patients

The COVID-19 pandemic has adverse consequences on human psychology and behavior long after initial recovery from the virus. These COVID-19 health sequelae, if undetected and left untreated, may lead to more enduring mental health problems, and put vulnerable individuals at risk of developing more serious psychopathologies. Therefore, an early distinction of such vulnerable individuals from those who are more resilient is important to undertake timely preventive interventions. The main aim of this article is to present a comprehensive multimodal conceptual approach for addressing these potential psychological and behavioral mental health changes using state-of-the-art tools and means of artificial intelligence (AI). Mental health COVID-19 recovery programs at post-COVID clinics based on AI prediction and prevention strategies may significantly improve the global mental health of ex-COVID-19 patients. Most COVID-19 recovery programs currently involve specialists such as pulmonologists, cardiologists, and neurologists, but there is a lack of psychiatrist care. The focus of this article is on new tools which can enhance the current limited psychiatrist resources and capabilities in coping with the upcoming challenges related to widespread mental health disorders. Patients affected by COVID-19 are more vulnerable to psychological and behavioral changes than non-COVID populations and therefore they deserve careful clinical psychological screening in post-COVID clinics. However, despite significant advances in research, the pace of progress in prevention of psychiatric disorders in these patients is still insufficient. Current approaches for the diagnosis of psychiatric disorders largely rely on clinical rating scales, as well as self-rating questionnaires that are inadequate for comprehensive assessment of ex-COVID-19 patients' susceptibility to mental health deterioration. These limitations can presumably be overcome by applying state-of-the-art AI-based tools in diagnosis, prevention, and treatment of psychiatric disorders in acute phase of disease to prevent more chronic psychiatric consequences.

Comparing the predictive value of screening to the use of electronic health record data for detecting future suicidal thoughts and behavior in an urban pediatric emergency department: A preliminary analysis

AIM

Brief screening and predictive modeling have garnered attention for utility at identifying individuals at risk of suicide. Although previous research has investigated these methods, little is known about how these methods compare against each other or work in combination in the pediatric population.

METHODS

Patients were aged 8-18 years old who presented from January 1, 2017, to June 30, 2019, to a Pediatric Emergency Department (PED). All patients were screened with the Ask Suicide Questionnaire (ASQ) as part of a universal screening approach. For all models, we used 5-fold cross-validation. We compared four models: Model 1 only included the ASQ; Model 2 included the ASQ and EHR data gathered at the time of ED visit (EHR data); Model 3 only included EHR data; and Model 4 included EHR data and a single item from the ASQ that asked about a lifetime history of suicide attempt. The main outcome was subsequent PED visit with suicide-related presenting problem within a 3-month follow-up period.

RESULTS

Of the N = 13,420 individuals, n = 141 had a subsequent suicide-related PED visit. Approximately 63% identified as Black. Results showed that a model based only on EHR data (Model 3) had an area under the curve (AUC) of 0.775 compared to the ASQ alone (Model 1), which had an AUC of 0.754. Combining screening and EHR data (Model 4) resulted in a 17.4% (absolute difference = 3.6%) improvement in sensitivity and 13.4% increase in AUC (absolute difference = 6.6%) compared to screening alone (Model 1).

CONCLUSION

Our findings show that predictive modeling based on EHR data is helpful either in the absence or as an addition to brief suicide screening. This is the first study to compare brief suicide screening to EHR-based predictive modeling and adds to our understanding of how best to identify youth at risk of suicidal thoughts and behaviors in clinical care settings.

Suicide among Scottish military veterans: follow-up and trends

OBJECTIVES

The risk of suicide among UK military veterans remains unclear. Few recent studies have been undertaken, and most studies found no clear evidence of increased risk. We used data from the Trends in Scottish Veterans Health cohort to investigate suicides up to 2017 in order to examine whether there have been any changes in the long-term pattern of veteran suicides since our earlier study to 2012, and to compare trends in the risk of suicide among veterans with matched non-veterans.

METHOD

Retrospective cohort study of 78 000 veterans and 253 000 non-veterans born between 1945 and 1995, matched for age, sex and area of residence, using survival analysis to examine the risk of suicide in veterans in comparison with non-veterans overall and by subgroup, and to investigate associations with specific mental health conditions.

RESULTS

Up to 37 years of follow-up, 388 (0.5%) veterans and 1531 (0.6%) non-veterans died from suicide. The risk of suicide among veterans did not differ from non-veterans overall. Increased risk among early service leavers was explained by differences in deprivation, and the previously reported increased risk in female veterans is now confined to older women. Suicide was most common in the fifth decade of life, and around 20 years postservice. A history of mood disorder or post-traumatic stress disorder was non-significantly more common in veterans.

CONCLUSIONS

Veterans are not at increased risk of suicide overall. The highest risk for both men and women is in middle age, many years after leaving service.

Improving Suicide Risk Detection and Clinical Follow-up after Discharge from Nursing Homes

Objectives: Suicide Awareness for Veterans Exiting Community Living Centers (SAVE-CLC) is a brief intervention to standardize suicide-risk screening and clinical follow-up after VA nursing home discharge. This paper examines the outcomes of SAVE-CLC compared to care as usual.Methods: A quasi-experimental evaluation was conducted (N = 124) with SAVE-CLC patients (n = 62) matched 1:1 to a pre-implementation comparison group. Data were obtained through VA Corporate Data Warehouse resources and chart reviews. Outcomes examined (within 30/90 days of discharge) included mortality rates, frequency of outpatient mental health visits, emergency department visits, rehospitalizations, depression screens (PHQ-2), and the latency period for outpatient mental health care.Results: A greater portion of SAVE-CLC patients received a depression screen after discharge, n = 42, 67.7% versus n = 8, 12.9%, OR = 14.2 (5.7, 35.3), p < .001. The number of days between discharge and first mental health visit was also substantially shorter for SAVE-CLC patients, M = 8.9, SD = 8.2 versus M = 17.6, SD = 9.1; t = 2.47 (122), p = .02. Significant differences were not observed in emergency department visits, hospitalizations, or mortality.Conclusions: SAVE-CLC is a time-limited intervention for detecting risk and speeding engagement in mental health care in the immediate high-risk post-discharge period.Clinical Implications: Care transitions present an important opportunity for addressing older adults' suicide risk; brief telephone-based interventions like SAVE-CLC may provide needed support to individuals returning home.

Sex-Specific Risk Profiles for Suicide Among Persons with Substance Use Disorders in Denmark

BACKGROUND AND AIMS

Persons with substance use disorders (SUDs) are at elevated risk of suicide death. We identified novel risk factors and interactions that predict suicide among men and women with SUD using machine learning.

DESIGN

Case-cohort study.

SETTING

Denmark.

PARTICIPANTS

The sample was restricted to persons with their first SUD diagnosis during 1995 to 2015. Cases were persons who died by suicide in Denmark during 1995 to 2015 (n = 2774) and the comparison subcohort was a 5% random sample of individuals in Denmark on 1 January 1995 (n = 13 179).

MEASUREMENTS

Suicide death was recorded in the Danish Cause of Death Registry. Predictors included social and demographic information, mental and physical health diagnoses, surgeries, medications, and poisonings.

FINDINGS

Persons among the highest risk for suicide, as identified by the classification trees, were men prescribed antidepressants in the 4 years before suicide and had a poisoning diagnosis in the 4 years before suicide; and women who were 30+ years old and had a poisoning diagnosis 4 years before and 12 months before suicide. Among men with SUD, the random forest identified five variables that were most important in predicting suicide; reaction to severe stress and adjustment disorders, drugs used to treat addictive disorders, age 30+ years, antidepressant use, and poisoning in the 4 prior years. Among women with SUD, the random forest found that the most important predictors of suicide were prior poisonings and reaction to severe stress and adjustment disorders. Individuals in the top 5% of predicted risk accounted for 15% of all suicide deaths among men and 24% of all suicides among women.

CONCLUSIONS

In Denmark, prior poisoning and comorbid psychiatric disorders may be among the most important indicators of suicide risk among persons with substance use disorders, particularly among women.

Evaluation of the Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment Suicide Risk Modeling Clinical Program in the Veterans Health Administration

IMPORTANCE

The Veterans Health Administration (VHA) implemented a national clinical program using a suicide risk prediction algorithm, Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment (REACH VET), in which clinicians facilitate care enhancements for individuals identified in local top 0.1% suicide risk tiers. Evaluation studies are needed.

OBJECTIVE

To determine associations with treatment engagement, health care utilization, suicide attempts, safety plan documentation, and 6-month mortality.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study used triple differences analyses comparing 6-month changes in outcomes after vs before program entry for individuals entering the REACH VET program (March 2017-December 2018) vs a similarly identified top 0.1% suicide risk tier cohort from prior to program initiation (March 2014-December 2015), adjusting for trends across subthreshold cohorts. Subcohort analyses (including individuals from March 2017-June 2018) evaluated difference-in-differences for cause-specific mortality using death certificate data. The subthreshold cohorts included individuals in the top 0.3% to 0.1% suicide risk tier, below the threshold for REACH VET eligibility, from the concurrent REACH VET period and from the pre-REACH VET period. Data were analyzed from December 2019 through September 2021.

EXPOSURES

REACH VET-designated clinicians treatment reevaluation and outreach for care enhancements, including safety planning, increased monitoring, and interventions to enhance coping.

MAIN OUTCOMES AND MEASURES

Process outcomes included VHA scheduled, completed, and missed appointments; mental health visits; and safety plan documentation and documentation within 6 months for individuals without plans within the prior 2 years. Clinical outcomes included mental health admissions, emergency department visits, nonfatal suicide attempts, and all-cause, suicide, and nonsuicide external-cause mortality.

RESULTS

A total of 173 313 individuals (mean [SD] age, 51.0 [14.7] years; 161 264 [93.1%] men and 12 049 [7.0%] women) were included in analyses, including 40 816 individuals eligible for REACH VET care and 36 604 individuals from the pre-REACH VET period in the top 0.1% of suicide risk. The REACH VET intervention was associated with significant increases in completed outpatient appointments (adjusted triple difference [ATD], 0.31; 95% CI, 0.06 to 0.55) and proportion of individuals with new safety plans (ATD, 0.08; 95% CI, 0.06 to 0.10) and reductions in mental health admissions (ATD, -0.08; 95% CI, -0.10 to -0.05), emergency department visits (ADT, -0.03; 95% CI, -0.06 to -0.01), and suicide attempts (ADT, -0.05; 95% CI, -0.06 to -0.03). Subcohort analyses did not identify differences in suicide or all-cause mortality (eg, age-and-sex-adjusted difference-in-difference for suicide mortality, 0.0007; 95% CI, -0.0006 to 0.0019).

CONCLUSIONS AND RELEVANCE

These findings suggest that REACH VET implementation was associated with greater treatment engagement and new safety plan documentation and fewer mental health admissions, emergency department visits, and suicide attempts. Clinical programs using risk modeling may be effective tools to support care enhancements and risk reduction.

Highlighting psychological pain avoidance and decision-making bias as key predictors of suicide attempt in major depressive disorder-A novel investigative approach using machine learning

OBJECTIVE

Predicting suicide is notoriously difficult and complex, but a serious public health issue. An innovative approach utilizing machine learning (ML) that incorporates features of psychological mechanisms and decision-making characteristics related to suicidality could create an improved model for identifying suicide risk in patients with major depressive disorder (MDD).

METHOD

Forty-four patients with MDD and past suicide attempts (MDD_SA, N = 44); 48 patients with MDD but without past suicide attempts (MDD_NS, N = 48-42 of whom with suicide ideation [MDD_SI, N = 42]), and healthy controls (HCs, N = 51) completed seven psychometric assessments including the Three-dimensional Psychological Pain Scale (TDPPS), and one behavioral assessment, the Balloon Analogue Risk Task (BART). Descriptive statistics, group comparisons, logistic regressions, and ML were used to explore and compare the groups and generate predictors of suicidal acts.

RESULTS

MDD_SA and MDD_NS differed in TDPPS total score, pain arousal and avoidance subscale scores, suicidal ideation scores, and relevant decision-making indicators in BART. Logistic regression tests linked suicide attempts to psychological pain avoidance and a risk decision-making indicator. The resultant key ML model distinguished MDD_SA/MDD_NS with 88.2% accuracy. The model could also distinguish MDD_SA/MDD_SI with 81.25% accuracy. The ML model using hopelessness could classify MDD_SI/HC with 94.4% accuracy.

CONCLUSION

ML analyses showed that motivation to avoid intolerable psychological pain, coupled with impaired decision-making bias toward under-valuing life's worth are highly predictive of suicide attempts. Analyses also demonstrated that suicidal ideation and attempts differed in potential mechanisms, as suicidal ideation was more related to hopelessness. ML algorithms show useful promises as a predictive instrument.

Designing a Clinical Decision Support Tool That Leverages Machine Learning for Suicide Risk Prediction: Development Study in Partnership With Native American Care Providers

Background

Machine learning algorithms for suicide risk prediction have been developed with notable improvements in accuracy. Implementing these algorithms to enhance clinical care and reduce suicide has not been well studied.

Objective

This study aims to design a clinical decision support tool and appropriate care pathways for community-based suicide surveillance and case management systems operating on Native American reservations.

Methods

Participants included Native American case managers and supervisors (N=9) who worked on suicide surveillance and case management programs on 2 Native American reservations. We used in-depth interviews to understand how case managers think about and respond to suicide risk. The results from interviews informed a draft clinical decision support tool, which was then reviewed with supervisors and combined with appropriate care pathways.

Results

Case managers reported acceptance of risk flags based on a predictive algorithm in their surveillance system tools, particularly if the information was available in a timely manner and used in conjunction with their clinical judgment. Implementation of risk flags needed to be programmed on a dichotomous basis, so the algorithm could produce output indicating high versus low risk. To dichotomize the continuous predicted probabilities, we developed a cutoff point that favored specificity, with the understanding that case managers’ clinical judgment would help increase sensitivity.

Conclusions

Suicide risk prediction algorithms show promise, but implementation to guide clinical care remains relatively elusive. Our study demonstrates the utility of working with partners to develop and guide the operationalization of risk prediction algorithms to enhance clinical care in a community setting.

Applying Machine Learning Approaches to Suicide Prediction Using Healthcare Data: Overview and Future Directions

Objective: Early identification of individuals who are at risk for suicide is crucial in supporting suicide prevention. Machine learning is emerging as a promising approach to support this objective. Machine learning is broadly defined as a set of mathematical models and computational algorithms designed to automatically learn complex patterns between predictors and outcomes from example data, without being explicitly programmed to do so. The model's performance continuously improves over time by learning from newly available data. Method: This concept paper explores how machine learning approaches applied to healthcare data obtained from electronic health records, including billing and claims data, can advance our ability to accurately predict future suicidal behavior. Results: We provide a general overview of machine learning concepts, summarize exemplar studies, describe continued challenges, and propose innovative research directions. Conclusion: Machine learning has potential for improving estimation of suicide risk, yet important challenges and opportunities remain. Further research can focus on incorporating evolving methods for addressing data imbalances, understanding factors that affect generalizability across samples and healthcare systems, expanding the richness of the data, leveraging newer machine learning approaches, and developing automatic learning systems.

Using machine learning to predict suicide in the 30 days after discharge from psychiatric hospital in Denmark

BACKGROUND

Suicide risk is high in the 30 days after discharge from psychiatric hospital, but knowledge of the profiles of high-risk patients remains limited.

AIMS

To examine sex-specific risk profiles for suicide in the 30 days after discharge from psychiatric hospital, using machine learning and Danish registry data.

METHOD

We conducted a case-cohort study capturing all suicide cases occurring in the 30 days after psychiatric hospital discharge in Denmark from 1 January 1995 to 31 December 2015 (n = 1205). The comparison subcohort was a 5% random sample of all persons born or residing in Denmark on 1 January 1995, and who had a first psychiatric hospital admission between 1995 and 2015 (n = 24 559). Predictors included diagnoses, surgeries, prescribed medications and demographic information. The outcome was suicide death recorded in the Danish Cause of Death Registry.

RESULTS

For men, prescriptions for anxiolytics and drugs used in addictive disorders interacted with other characteristics in the risk profiles (e.g. alcohol-related disorders, hypnotics and sedatives) that led to higher risk of postdischarge suicide. In women, there was interaction between recurrent major depression and other characteristics (e.g. poisoning, low income) that led to increased risk of suicide. Random forests identified important suicide predictors: alcohol-related disorders and nicotine dependence in men and poisoning in women.

CONCLUSIONS

Our findings suggest that accurate prediction of suicide during the high-risk period immediately after psychiatric hospital discharge may require a complex evaluation of multiple factors for men and women.

Evaluation of Electronic Health Record-Based Suicide Risk Prediction Models on Contemporary Data

BACKGROUND

Suicide risk prediction models have been developed by using information from patients' electronic health records (EHR), but the time elapsed between model development and health system implementation is often substantial. Temporal changes in health systems and EHR coding practices necessitate the evaluation of such models in more contemporary data.

OBJECTIVES

A set of published suicide risk prediction models developed by using EHR data from 2009 to 2015 across seven health systems reported c-statistics of 0.85 for suicide attempt and 0.83 to 0.86 for suicide death. Our objective was to evaluate these models' performance with contemporary data (2014-2017) from these systems.

METHODS

We evaluated performance using mental health visits (6,832,439 to mental health specialty providers and 3,987,078 to general medical providers) from 2014 to 2017 made by 1,799,765 patients aged 13+ across the health systems. No visits in our evaluation were used in the previous model development. Outcomes were suicide attempt (health system records) and suicide death (state death certificates) within 90 days following a visit. We assessed calibration and computed c-statistics with 95% confidence intervals (CI) and cut-point specific estimates of sensitivity, specificity, and positive/negative predictive value.

RESULTS

Models were well calibrated; 46% of suicide attempts and 35% of suicide deaths in the mental health specialty sample were preceded by a visit (within 90 days) with a risk score in the top 5%. In the general medical sample, 53% of attempts and 35% of deaths were preceded by such a visit. Among these two samples, respectively, c-statistics were 0.862 (95% CI: 0.860-0.864) and 0.864 (95% CI: 0.860-0.869) for suicide attempt, and 0.806 (95% CI: 0.790-0.822) and 0.804 (95% CI: 0.782-0.829) for suicide death.

CONCLUSION

Performance of the risk prediction models in this contemporary sample was similar to historical estimates for suicide attempt but modestly lower for suicide death. These published models can inform clinical practice and patient care today.

From everyday life predictions to suicide prevention: Clinical and ethical considerations in suicide predictive analytic tools

Advances in artificial intelligence and machine learning have fueled growing interest in the application of predictive analytics to identify high-risk suicidal patients. Such application will require the aggregation of large-scale, sensitive patient data to help inform complex and potentially stigmatizing health care decisions. This paper provides a description of how suicide prediction is uniquely difficult by comparing it to nonmedical (weather and traffic forecasting) and medical predictions (cancer and human immunodeficiency virus risk), followed by clinical and ethical challenges presented within a risk-benefit conceptual framework. Because the misidentification of suicide risk may be associated with unintended negative consequences, clinicians and policymakers need to carefully weigh the risks and benefits of using suicide predictive analytics across health care populations. Practical recommendations are provided to strengthen the protection of patient rights and enhance the clinical utility of suicide predictive analytics tools.

Association Between Patterns of Alcohol Use and Short-Term Risk of Suicide Attempt Among Patients With and Without Reported Suicidal Ideation

OBJECTIVE

To evaluate the association between patterns of alcohol use and short-term risk of suicide attempt among patients with and without reported suicidal ideation.

METHODS

Kaiser Permanente Washington electronic health record data were used to identify mental health visits (1/1/2010-6/30/2015) with documented assessments for unhealthy alcohol use (AUDIT-C) and suicidal ideation (PHQ-9 ninth question). Logistic regression fit using generalized estimating equations were used to conduct visit-level analyses, accounting for correlation between individuals' assessments. Separate models evaluated the association between (1) level of alcohol consumption and (2) frequency of heavy episodic drinking (HED), in combination with suicidal ideation (any vs none), with suicide attempt within 90 days following each visit. Primary models adjusted for age, gender, race/ethnicity and visit year.

RESULTS

Of 59,705 visits (43,706 unique patients), 372 (0.62%) were followed by a suicide attempt within 90 days. The risk of suicide attempt was significantly higher for patients reporting suicidal ideation across all levels of alcohol consumption compared to patients reporting low-level alcohol use and no suicidal ideation, particularly high-level use (OR 9.77, 95% CI, 6.23-15.34). Similarly, risk of suicide attempt was higher for patients reporting suicidal ideation across all levels of HED relative to those reporting no HED or suicidal ideation, particularly HED monthly or more (OR 6.80, 95% CI 4.77-9.72). Among patients reporting no suicidal ideation, no associations were observed.

CONCLUSIONS

Findings underscore the potential value of offering alcohol-related care to patient reporting suicidal ideation. Additional strategies are needed to identify suicide risk among those reporting no suicidal ideation.

National cross-sectional cohort study of the relationship between quality of mental healthcare and death by suicide

Background

Patient safety-based interventions aimed at lethal means restriction are effective at reducing death by suicide in inpatient mental health settings but are more challenging in the outpatient arena. As an alternative approach, we examined the association between quality of mental healthcare and suicide in a national healthcare system.

Methods

We calculated regional suicide rates for Department of Veterans Affairs (VA) Healthcare users from 2013 to 2017. To control for underlying variation in suicide risk in each of our 115 mental health referral regions (MHRRs), we calculated standardised rate ratios (SRRs) for VA users compared with the general population. We calculated quality metrics for outpatient mental healthcare in each MHRR using individual metrics as well as an Overall Quality Index. We assessed the correlation between quality metrics and suicide rates.

Results

Among the 115 VA MHRRs, the age-adjusted, sex-adjusted and race-adjusted annual suicide rates varied from 6.8 to 92.9 per 100 000 VA users, and the SRRs varied between 0.7 and 5.7. Mean regional-level adherence to each of our quality metrics ranged from a low of 7.7% for subspecialty care access to a high of 58.9% for care transitions. While there was substantial regional variation in quality, there was no correlation between an overall index of mental healthcare quality and SRR.

Conclusion

There was no correlation between overall quality of outpatient mental healthcare and rates of suicide in a national healthcare system. Although it is possible that quality was not high enough anywhere to prevent suicide at the population level or that we were unable to adequately measure quality, this examination of core mental health services in a well-resourced system raises doubts that a quality-based approach alone can lower population-level suicide rates.

Natural language processing of clinical mental health notes may add predictive value to existing suicide risk models

BACKGROUND

This study evaluated whether natural language processing (NLP) of psychotherapy note text provides additional accuracy over and above currently used suicide prediction models.

METHODS

We used a cohort of Veterans Health Administration (VHA) users diagnosed with post-traumatic stress disorder (PTSD) between 2004-2013. Using a case-control design, cases (those that died by suicide during the year following diagnosis) were matched to controls (those that remained alive). After selecting conditional matches based on having shared mental health providers, we chose controls using a 5:1 nearest-neighbor propensity match based on the VHA's structured Electronic Medical Records (EMR)-based suicide prediction model. For cases, psychotherapist notes were collected from diagnosis until death. For controls, psychotherapist notes were collected from diagnosis until matched case's date of death. After ensuring similar numbers of notes, the final sample included 246 cases and 986 controls. Notes were analyzed using Sentiment Analysis and Cognition Engine, a Python-based NLP package. The output was evaluated using machine-learning algorithms. The area under the curve (AUC) was calculated to determine models' predictive accuracy.

RESULTS

NLP derived variables offered small but significant predictive improvement (AUC = 0.58) for patients that had longer treatment duration. A small sample size limited predictive accuracy.

CONCLUSIONS

Study identifies a novel method for measuring suicide risk over time and potentially categorizing patient subgroups with distinct risk sensitivities. Findings suggest leveraging NLP derived variables from psychotherapy notes offers an additional predictive value over and above the VHA's state-of-the-art structured EMR-based suicide prediction model. Replication with a larger non-PTSD specific sample is required.

Predicting the 9-year course of mood and anxiety disorders with automated machine learning: A comparison between auto-sklearn, naïve Bayes classifier, and traditional logistic regression

BACKGROUND

Predicting the onset and course of mood and anxiety disorders is of clinical importance but remains difficult. We compared the predictive performances of traditional logistic regression, basic probabilistic machine learning (ML) methods, and automated ML (Auto-sklearn).

METHODS

Data were derived from the Netherlands Study of Depression and Anxiety. We compared how well multinomial logistic regression, a naïve Bayes classifier, and Auto-sklearn predicted depression and anxiety diagnoses at a 2-, 4-, 6-, and 9-year follow up, operationalized as binary or categorical variables. Predictor sets included demographic and self-report data, which can be easily collected in clinical practice at two initial time points (baseline and 1-year follow up).

RESULTS

At baseline, participants were 42.2 years old, 66.5% were women, and 53.6% had a current mood or anxiety disorder. The three methods were similarly successful in predicting (mental) health status, with correct predictions for up to 79% (95% CI 75-81%). However, Auto-sklearn was superior when assessing a more complex dataset with individual item scores.

CONCLUSIONS

Automated ML methods added only limited value, compared to traditional data modelling when predicting the onset and course of depression and anxiety. However, they hold potential for automatization and may be better suited for complex datasets.

Reconciling Statistical and Clinicians' Predictions of Suicide Risk

Statistical models, including those based on electronic health records, can accurately identify patients at high risk for a suicide attempt or death, leading to implementation of risk prediction models for population-based suicide prevention in health systems. However, some have questioned whether statistical predictions can really inform clinical decisions. Appropriately reconciling statistical algorithms with traditional clinician assessment depends on whether predictions from these two methods are competing, complementary, or merely duplicative. In June 2019, the National Institute of Mental Health convened a meeting, "Identifying Research Priorities for Risk Algorithms Applications in Healthcare Settings to Improve Suicide Prevention." Here, participants of this meeting summarize key issues regarding the potential clinical application of suicide prediction models. The authors attempt to clarify the key conceptual and technical differences between traditional risk prediction by clinicians and predictions from statistical models, review the limited evidence regarding both the accuracy of and the concordance between these alternative methods of prediction, present a conceptual framework for understanding agreement and disagreement between statistical and clinician predictions, identify priorities for improving data regarding suicide risk, and propose priority questions for future research. Future suicide risk assessment will likely combine statistical prediction with traditional clinician assessment, but research is needed to determine the optimal combination of these two methods.

Identification of Suicide Attempt Risk Factors in a National US Survey Using Machine Learning

IMPORTANCE

Because more than one-third of people making nonfatal suicide attempts do not receive mental health treatment, it is essential to extend suicide attempt risk factors beyond high-risk clinical populations to the general adult population.

OBJECTIVE

To identify future suicide attempt risk factors in the general population using a data-driven machine learning approach including more than 2500 questions from a large, nationally representative survey of US adults.

DESIGN, SETTING, AND PARTICIPANTS

Data came from wave 1 (2001 to 2002) and wave 2 (2004 to 2005) of the National Epidemiologic Survey on Alcohol and Related Conditions (NESARC). NESARC is a face-to-face longitudinal survey conducted with a national representative sample of noninstitutionalized civilian population 18 years and older in the US. The cumulative response rate across both waves was 70.2% resulting in 34 653 wave 2 interviews. A balanced random forest was trained using cross-validation to develop a suicide attempt risk model. Out-of-fold model prediction was used to assess model performance, including the area under the receiver operator curve, sensitivity, and specificity. Survey design and nonresponse weights allowed estimates to be representative of the US civilian population based on the 2000 census. Analyses were performed between May 15, 2019, and June 10, 2020.

MAIN OUTCOMES AND MEASURES

Attempted suicide in the 3 years between wave 1 and wave 2 interviews.

RESULTS

Of 34 653 participants, 20 089 were female (weighted proportion, 52.1%). The weighted mean (SD) age was 45.1 (17.3) years at wave 1 and 48.2 (17.3) years at wave 2. Attempted suicide during the 3 years between wave 1 and wave 2 interviews was self-reported by 222 of 34 653 participants (0.6%). Using survey questions measured at wave 1, the suicide attempt risk model yielded a cross-validated area under the receiver operator characteristic curve of 0.857 with a sensitivity of 85.3% (95% CI, 79.8-89.7) and a specificity of 73.3% (95% CI, 72.8-73.8) at an optimized threshold. The model identified 1.8% of the US population to be at a 10% or greater risk of suicide attempt. The most important risk factors were 3 questions about previous suicidal ideation or behavior; 3 items from the 12-Item Short Form Health Survey, namely feeling downhearted, doing activities less carefully, or accomplishing less because of emotional problems; younger age; lower educational achievement; and recent financial crisis.

CONCLUSIONS AND RELEVANCE

In this study, after searching through more than 2500 survey questions, several well-known risk factors of suicide attempt were confirmed, such as previous suicidal behaviors and ideation, and new risks were identified, including functional impairment resulting from mental disorders and socioeconomic disadvantage. These results may help guide future clinical assessment and the development of new suicide risk scales.

Prospective Validation of an Electronic Health Record-Based, Real-Time Suicide Risk Model

IMPORTANCE

Numerous prognostic models of suicide risk have been published, but few have been implemented outside of integrated managed care systems.

OBJECTIVE

To evaluate performance of a suicide attempt risk prediction model implemented in a vendor-supplied electronic health record to predict subsequent (1) suicidal ideation and (2) suicide attempt.

DESIGN, SETTING, AND PARTICIPANTS

This observational cohort study evaluated implementation of a suicide attempt prediction model in live clinical systems without alerting. The cohort comprised patients seen for any reason in adult inpatient, emergency department, and ambulatory surgery settings at an academic medical center in the mid-South from June 2019 to April 2020.

MAIN OUTCOMES AND MEASURES

Primary measures assessed external, prospective, and concurrent validity. Manual medical record validation of coded suicide attempts confirmed incident behaviors with intent to die. Subgroup analyses were performed based on demographic characteristics, relevant clinical context/setting, and presence or absence of universal screening. Performance was evaluated using discrimination (number needed to screen, C statistics, positive/negative predictive values) and calibration (Spiegelhalter z statistic). Recalibration was performed with logistic calibration.

RESULTS

The system generated 115 905 predictions for 77 973 patients (42 490 [54%] men, 35 404 [45%] women, 60 586 [78%] White, 12 620 [16%] Black). Numbers needed to screen in highest risk quantiles were 23 and 271 for suicidal ideation and attempt, respectively. Performance was maintained across demographic subgroups. Numbers needed to screen for suicide attempt by sex were 256 for men and 323 for women; and by race: 373, 176, and 407 for White, Black, and non-White/non-Black patients, respectively. Model C statistics were, across the health system: 0.836 (95% CI, 0.836-0.837); adult hospital: 0.77 (95% CI, 0.77-0.772); emergency department: 0.778 (95% CI, 0.777-0.778); psychiatry inpatient settings: 0.634 (95% CI, 0.633-0.636). Predictions were initially miscalibrated (Spiegelhalter z = -3.1; P = .001) with improvement after recalibration (Spiegelhalter z = 1.1; P = .26).

CONCLUSIONS AND RELEVANCE

In this study, this real-time predictive model of suicide attempt risk showed reasonable numbers needed to screen in nonpsychiatric specialty settings in a large clinical system. Assuming that research-valid models will translate without performing this type of analysis risks inaccuracy in clinical practice, misclassification of risk, wasted effort, and missed opportunity to correct and prevent such problems. The next step is careful pairing with low-cost, low-harm preventive strategies in a pragmatic trial of effectiveness in preventing future suicidality.

Suicide and the agent-host-environment triad: leveraging surveillance sources to inform prevention

Suicide in the US has increased in the last decade, across virtually every age and demographic group. Parallel increases have occurred in non-fatal self-harm as well. Research on suicide across the world has consistently demonstrated that suicide shares many properties with a communicable disease, including person-to-person transmission and point-source outbreaks. This essay illustrates the communicable nature of suicide through analogy to basic infectious disease principles, including evidence for transmission and vulnerability through the agent-host-environment triad. We describe how mathematical modeling, a suite of epidemiological methods, which the COVID-19 pandemic has brought into renewed focus, can and should be applied to suicide in order to understand the dynamics of transmission and to forecast emerging risk areas. We describe how new and innovative sources of data, including social media and search engine data, can be used to augment traditional suicide surveillance, as well as the opportunities and challenges for modeling suicide as a communicable disease process in an effort to guide clinical and public health suicide prevention efforts.

What health records data are required for accurate prediction of suicidal behavior?

OBJECTIVE

The study sought to evaluate how availability of different types of health records data affect the accuracy of machine learning models predicting suicidal behavior.

MATERIALS AND METHODS

Records from 7 large health systems identified 19 061 056 outpatient visits to mental health specialty or general medical providers between 2009 and 2015. Machine learning models (logistic regression with penalized LASSO [least absolute shrinkage and selection operator] variable selection) were developed to predict suicide death (n = 1240) or probable suicide attempt (n = 24 133) in the following 90 days. Base models were used only historical insurance claims data and were then augmented with data regarding sociodemographic characteristics (race, ethnicity, and neighborhood characteristics), past patient-reported outcome questionnaires from electronic health records, and data (diagnoses and questionnaires) recorded during the visit.

RESULTS

For prediction of any attempt following mental health specialty visits, a model limited to historical insurance claims data performed approximately as well (C-statistic 0.843) as a model using all available data (C-statistic 0.850). For prediction of suicide attempt following a general medical visit, addition of data recorded during the visit yielded a meaningful improvement over a model using all data up to the prior day (C-statistic 0.853 vs 0.838).

DISCUSSION

Results may not generalize to setting with less comprehensive data or different patterns of care. Even the poorest-performing models were superior to brief self-report questionnaires or traditional clinical assessment.

CONCLUSIONS

Implementation of suicide risk prediction models in mental health specialty settings may be less technically demanding than expected. In general medical settings, however, delivery of optimal risk predictions at the point of care may require more sophisticated informatics capability.

How machine-learning recommendations influence clinician treatment selections: the example of the antidepressant selection

Decision support systems embodying machine learning models offer the promise of an improved standard of care for major depressive disorder, but little is known about how clinicians' treatment decisions will be influenced by machine learning recommendations and explanations. We used a within-subject factorial experiment to present 220 clinicians with patient vignettes, each with or without a machine-learning (ML) recommendation and one of the multiple forms of explanation. We found that interacting with ML recommendations did not significantly improve clinicians' treatment selection accuracy, assessed as concordance with expert psychopharmacologist consensus, compared to baseline scenarios in which clinicians made treatment decisions independently. Interacting with incorrect recommendations paired with explanations that included limited but easily interpretable information did lead to a significant reduction in treatment selection accuracy compared to baseline questions. These results suggest that incorrect ML recommendations may adversely impact clinician treatment selections and that explanations are insufficient for addressing overreliance on imperfect ML algorithms. More generally, our findings challenge the common assumption that clinicians interacting with ML tools will perform better than either clinicians or ML algorithms individually.

Gender Differences in Lifetime Prevalence and Onset Timing of Suicidal Ideation and Suicide Attempt Among Post-9/11 Veterans and Nonveterans

BACKGROUND

Rising US suicide rates are particularly notable among military veterans, especially women. It is unknown whether these differences extend to suicidal ideation (SI) and suicide attempts (SA), which are major predictors of suicide. Literature comparing SI and SA prevalence and timing of onset between veterans and nonveterans is limited.

OBJECTIVE

The objective of this study was to estimate and compare SI and SA prevalence and onset timing relative to age and military service between veterans and nonveterans, by gender.

RESEARCH DESIGN

Gender-stratified analysis of cross-sectional data from the Comparative Health Assessment Interview Study. Generalized estimating equations logistic regression was used to compare prevalence and onset of SI and SA between time periods and across groups, controlling for years at risk in each time period.

SUBJECTS

National sample of 15,082 post-9/11 veterans (36.7% women) and 4638 nonveterans (30.5% women).

MEASURES

Columbia-Suicide Severity Rating Scale adapted to assess SI and SA relative to age (less than 18 y, 18 y and above) and military service (pre-, during, and post-military).

RESULTS

Veteran men experienced significantly higher odds of lifetime SI compared with nonveteran men (odds ratio=1.13), whereas veteran women experienced significantly higher odds of lifetime SA compared with nonveteran women (odds ratio=1.35). SI and SA onset varied considerably for veterans and nonveterans and by gender within veteran groups.

CONCLUSIONS

Veterans and nonveterans appear to differ in periods of risk for SI and SA. Furthermore, gender differences in SI and SA onset for veterans highlight the need for gender-informed veteran suicide prevention strategies that target periods of highest risk.

Machine Learning to Differentiate Risk of Suicide Attempt and Self-harm After General Medical Hospitalization of Women With Mental Illness

BACKGROUND

Suicide prevention is a public health priority, but risk factors for suicide after medical hospitalization remain understudied. This problem is critical for women, for whom suicide rates in the United States are disproportionately increasing.

OBJECTIVE

To differentiate the risk of suicide attempt and self-harm following general medical hospitalization among women with depression, bipolar disorder, and chronic psychosis.

METHODS

We developed a machine learning algorithm that identified risk factors of suicide attempt and self-harm after general hospitalization using electronic health record data from 1628 women in the University of California Los Angeles Integrated Clinical and Research Data Repository. To assess replicability, we applied the algorithm to a larger sample of 140,848 women in the New York City Clinical Data Research Network.

RESULTS

The classification tree algorithm identified risk groups in University of California Los Angeles Integrated Clinical and Research Data Repository (area under the curve 0.73, sensitivity 73.4, specificity 84.1, accuracy 0.84), and predictor combinations characterizing key risk groups were replicated in New York City Clinical Data Research Network (area under the curve 0.71, sensitivity 83.3, specificity 82.2, and accuracy 0.84). Predictors included medical comorbidity, history of pregnancy-related mental illness, age, and history of suicide-related behavior. Women with antecedent medical illness and history of pregnancy-related mental illness were at high risk (6.9%-17.2% readmitted for suicide-related behavior), as were women below 55 years old without antecedent medical illness (4.0%-7.5% readmitted).

CONCLUSIONS

Prevention of suicide attempt and self-harm among women following acute medical illness may be improved by screening for sex-specific predictors including perinatal mental health history.

Patient Feedback on the Use of Predictive Analytics for Suicide Prevention

OBJECTIVE

There is significant debate about the feasibility of using predictive models for suicide prevention. Although statistical considerations have received careful attention, patient perspectives have not been examined. This study collected feedback from high-risk veterans about the U.S. Department of Veterans Affairs (VA) prevention program called Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment (REACH VET).

METHODS

Anonymous questionnaires were obtained from veterans during their stay at a psychiatric inpatient unit (N=102). The questionnaire included three vignettes (the standard VA script, a more statistical vignette, and a more collaborative vignette) that described a conversation a clinician might initiate to introduce REACH VET. Patients rated each vignette on several factors, selected their favorite vignette, and provided qualitative feedback, including recommendations for clinicians.

RESULTS

All three vignettes were rated as neutral to very caring by more than 80% of respondents (at least 69% of respondents rated all vignettes as somewhat caring to very caring). Similar positive feedback was obtained for several ratings (e.g., helpful vs. unhelpful, informative vs. uninformative, encouraging vs. discouraging). There were few differences in the ratings of the three vignettes, and each of the three scripts was preferred as the "favorite" by at least 28% of the sample. Few patients endorsed concerns that the discussion would increase their hopelessness, and privacy concerns were rare. Most of the advice for clinicians emphasized the importance of a patient-centered approach.

CONCLUSIONS

The results provide preliminary support for the acceptability of predictive models to identify patients at risk for suicide, but more stakeholder research is needed.

A learning algorithm for predicting mental health symptoms and substance use

Learning health systems use data to generate knowledge that informs clinical care, but few studies have evaluated how to leverage patient-reported mental health symptoms and substance use data to make patient-specific predictions. We developed a general Bayesian prediction algorithm that uses self-reported psychiatric symptoms and substance use within a population to predict future symptoms and substance use for individuals in that population. We validated our approach in 2444 participants from two clinical cohorts - the National Network of Depression Centers and the Johns Hopkins HIV Clinical Cohort - by predicting symptoms of depression, anxiety, and mania as well as alcohol, heroin, and cocaine use and comparing our predictions to observed symptoms and substance use. When we dichotomized mental health symptoms as moderate-severe vs. none-mild, individual predictions yielded areas under the ROC curve (AUCs) of 0.84 [95% confidence interval 0.80-0.88] and 0.85 [0.82-0.88] for symptoms of depression in the two cohorts, AUCs of 0.84 [0.79-0.88] and 0.85 [0.82-0.88] for symptoms of anxiety, and an AUC of 0.77 [0.72-0.82] for manic symptoms. Predictions of substance use yielded an AUC of 0.92 [0.88-0.97] for heroin use, 0.90 [0.82-0.97] for cocaine use, and 0.90 [0.88-092] for alcohol misuse. This rigorous, mathematically grounded approach could provide patient-specific predictions at the point of care. It can be applied to other psychiatric symptoms and substance use indicators, and is customizable to specific health systems. Such approaches can realize the potential of a learning health system to transform ever-increasing quantities of data into tangible guidance for patient care.

User-Centered Design of a Machine Learning Intervention for Suicide Risk Prediction in a Military Setting

Primary care represents a major opportunity for suicide prevention in the military. Significant advances have been made in using electronic health record data to predict suicide attempts in patient populations. With a user-centered design approach, we are developing an intervention that uses predictive analytics to inform care teams about their patients' risk of suicide attempt. We present our experience working with clinicians and staff in a military primary care setting to create preliminary designs and a context-specific usability testing plan for the deployment of the suicide risk indicator.

Meta-analysis of the strength of exploratory suicide prediction models; from clinicians to computers

BACKGROUND

Suicide prediction models have been formulated in a variety of ways and are heterogeneous in the strength of their predictions. Machine learning has been a proposed as a way of improving suicide predictions by incorporating more suicide risk factors.

AIMS

To determine whether machine learning and the number of suicide risk factors included in suicide prediction models are associated with the strength of the resulting predictions.

METHOD

Random-effect meta-analysis of exploratory suicide prediction models constructed by combining two or more suicide risk factors or using clinical judgement (Prospero Registration CRD42017059665). Studies were located by searching for papers indexed in PubMed before 15 August 2020 with the term suicid* in the title.

RESULTS

In total, 86 papers reported 102 suicide prediction models and included 20 210 411 people and 106 902 suicides. The pooled odds ratio was 7.7 (95% CI 6.7-8.8) with high between-study heterogeneity (I2 = 99.5). Machine learning was associated with a non-significantly higher odds ratio of 11.6 (95% CI 6.0-22.3) and clinical judgement with a non-significantly lower odds ratio of 4.7 (95% CI 2.1-10.9). Models including a larger number of suicide risk factors had a higher odds ratio when machine-learning studies were included (P = 0.02). Among non-machine-learning studies, suicide prediction models including fewer risk factors performed just as well as those including more risk factors.

CONCLUSIONS

Machine learning might have the potential to improve the performance of suicide prediction models by increasing the number of included suicide risk factors but its superiority over other methods is unproven.

An implementation evaluation of "Zero Suicide" using normalization process theory to support high-quality care for patients at risk of suicide

Background

Suicide rates continue to rise across the United States, galvanizing the need for increased suicide prevention and intervention efforts. The Zero Suicide (ZS) model was developed in response to this need and highlights four key clinical functions of high-quality health care for patients at risk of suicide. The goal of this quality improvement study was to understand how six large health care systems operationalized practices to support these functions-identification, engagement, treatment and care transitions.

Methods

Using a key informant interview guide and data collection template, researchers who were embedded in each health care system cataloged and summarized current and future practices supporting ZS, including, (1) the function addressed; (2) a description of practice intent and mechanism of intervention; (3) the target patient population and service setting; (4) when/how the practice was (or will be) implemented; and (5) whether/how the practice was documented and/or measured. Normalization process theory (NPT), an implementation evaluation framework, was applied to help understand how ZS had been operationalized in routine clinical practices and, specifically, what ZS practices were described by key informants (coherence), the current state of norms/conventions supporting these practices (cognitive participation), how health care teams performed these practices (collective action), and whether/how practices were measured when they occurred (reflexive monitoring).

Results

The most well-defined and consistently measured ZS practices (current and future) focused on the identification of patients at high risk of suicide. Stakeholders also described numerous engagement and treatment practices, and some practices intended to support care transitions. However, few engagement and transition practices were systematically measured, and few treatment practices were designed specifically for patients at risk of suicide.

Conclusions

The findings from this study will support large-scale evaluation of the effectiveness of ZS implementation and inform recommendations for implementation of high-quality suicide-related care in health care systems nationwide.

Natural language processing and machine learning of electronic health records for prediction of first-time suicide attempts

OBJECTIVE

Limited research exists in predicting first-time suicide attempts that account for two-thirds of suicide decedents. We aimed to predict first-time suicide attempts using a large data-driven approach that applies natural language processing (NLP) and machine learning (ML) to unstructured (narrative) clinical notes and structured electronic health record (EHR) data.

METHODS

This case-control study included patients aged 10-75 years who were seen between 2007 and 2016 from emergency departments and inpatient units. Cases were first-time suicide attempts from coded diagnosis; controls were randomly selected without suicide attempts regardless of demographics, following a ratio of nine controls per case. Four data-driven ML models were evaluated using 2-year historical EHR data prior to suicide attempt or control index visits, with prediction windows from 7 to 730 days. Patients without any historical notes were excluded. Model evaluation on accuracy and robustness was performed on a blind dataset (30% cohort).

RESULTS

The study cohort included 45 238 patients (5099 cases, 40 139 controls) comprising 54 651 variables from 5.7 million structured records and 798 665 notes. Using both unstructured and structured data resulted in significantly greater accuracy compared to structured data alone (area-under-the-curve [AUC]: 0.932 vs. 0.901 P < .001). The best-predicting model utilized 1726 variables with AUC = 0.932 (95% CI, 0.922-0.941). The model was robust across multiple prediction windows and subgroups by demographics, points of historical most recent clinical contact, and depression diagnosis history.

CONCLUSIONS

Our large data-driven approach using both structured and unstructured EHR data demonstrated accurate and robust first-time suicide attempt prediction, and has the potential to be deployed across various populations and clinical settings.

The association of lung ultrasound images with COVID-19 infection in an emergency room cohort

Lung ultrasound could facilitate the triage of patients with suspected COVID-19 infection admitted to the emergency room. We developed a predictive model for COVID-19 diagnosis based on lung ultrasound and clinical features. We used ultrasound to image the lung bilaterally at two anterior sites, one and two hands below each clavicle, and a posterolateral site that was the posterior transverse continuation from the lower anterior site. We studied 100 patients, 31 of whom had a COVID-19 positive reverse transcriptase polymerase chain reaction. A positive test was independently associated with: quick sequential organ failure assessment score ≥1; ≥3 B-lines at the upper site; consolidation and thickened pleura at the lower site; and thickened pleura line at the posterolateral site. The model discrimination was an area (95%CI) under the receiver operating characteristic curve of 0.82 (0.75-0.90). The characteristics (95%CI) of the model's diagnostic threshold, applied to the population from which it was derived, were: sensitivity, 97% (83-100%); specificity, 62% (50-74%); positive predictive value, 54% (41-98%); and negative predictive value, 98% (88-99%). This model may facilitate triage of patients with suspected COVID-19 infection admitted to the emergency room.

Addressing Suicide in the Veteran Population: Engaging a Public Health Approach

Suicide is a national public health issue in America, and it disproportionately affects those who are serving or who have served in the United States military. The US Department of Veterans Affairs (VA) has made suicide prevention its number one clinical priority. VA is committed to prevent suicide among the entire population of those who have served our country in the military, regardless of whether they make use of any VA services or benefits. Suicide can be prevented through the application of a public health strategy embracing partners at all levels. Following a national strategy, VA has embarked on an effort involving the application of a public health strategy combining both clinically-based and community-focused interventions. This paper describes several examples of these efforts and steps forward.

Relative accuracy of social and medical determinants of suicide in electronic health records

OBJECTIVE

This paper compares the accuracy of predicting suicide from Social Determinants of Health (SDoH) or history of illness.

POPULATION STUDIED

5 313 965 Veterans who at least had two primary care visits between 2008 and 2016.

STUDY DESIGN

The dependent variable was suicide or intentional self-injury. The independent variables were 10 495 International Classification of Disease (ICD) Version 9 codes, age, and gender. The ICD codes included 40 V-codes used for measuring SDoH, such as family disruption, family history of substance abuse, lack of education, legal impediments, social isolation, unemployment, and homelessness. The sample was randomly divided into training (90 percent) and validation (10 percent) sets. Area under the receiver operating characteristic (AROC) was used to measure accuracy of predictions in the validation set.

PRINCIPAL FINDINGS

Separate analyses were done for inpatient and outpatient codes; the results were similar. In the hospitalized group, the mean age was 67.2 years, and 92.1 percent were male. The mean number of medical diagnostic codes during the study period was 37; and 12.9 percent had at least one SDoH V-code. At least one episode of suicide or intentional self-injury occurred in 1.89 percent of cases. SDoH V-codes, on average, elevated the risk of suicide or intentional self-injury by 24-fold (ranging from 4- to 86-fold). An index of 40 SDoH codes predicted suicide or intentional self-injury with an AROC of 0.64. An index of 10 445 medical diagnoses, without SDoH V-codes, had AROC of 0.77. The combined SDoH and medical diagnoses codes also had AROC of 0.77.

CONCLUSION

In predicting suicide or intentional self-harm, SDoH V-codes add negligible information beyond what is already available in medical diagnosis codes.

IMPLICATIONS FOR PRACTICE

Policies that affect SDoH (eg, housing policies, resilience training) may not have an impact on suicide rates, if they do not change the underlying medical causes of SDoH.

Artificial Intelligence and Suicide Prevention: A Systematic Review of Machine Learning Investigations

Suicide is a leading cause of death that defies prediction and challenges prevention efforts worldwide. Artificial intelligence (AI) and machine learning (ML) have emerged as a means of investigating large datasets to enhance risk detection. A systematic review of ML investigations evaluating suicidal behaviors was conducted using PubMed/MEDLINE, PsychInfo, Web-of-Science, and EMBASE, employing search strings and MeSH terms relevant to suicide and AI. Databases were supplemented by hand-search techniques and Google Scholar. Inclusion criteria: (1) journal article, available in English, (2) original investigation, (3) employment of AI/ML, (4) evaluation of a suicide risk outcome. N = 594 records were identified based on abstract search, and 25 hand-searched reports. N = 461 reports remained after duplicates were removed, n = 316 were excluded after abstract screening. Of n = 149 full-text articles assessed for eligibility, n = 87 were included for quantitative synthesis, grouped according to suicide behavior outcome. Reports varied widely in methodology and outcomes. Results suggest high levels of risk classification accuracy (>90%) and Area Under the Curve (AUC) in the prediction of suicidal behaviors. We report key findings and central limitations in the use of AI/ML frameworks to guide additional research, which hold the potential to impact suicide on broad scale.

Mortality among U.S. military participants at eight aboveground nuclear weapons test series

BACKGROUND

Approximately 235,000 military personnel participated at one of 230 U.S. atmospheric nuclear weapons tests from 1945 through 1962. At the Nevada Test Site (NTS), the atomic veterans participated in military maneuvers, observed nuclear weapons tests, or provided technical support. At the Pacific Proving Ground (PPG), they served aboard ships or were stationed on islands during or after nuclear weapons tests.

MATERIAL AND METHODS

Participants at seven test series, previously studied with high-quality dosimetry and personnel records, and the first test at TRINITY formed the cohort of 114,270 male military participants traced for vital status from 1945 through 2010. Dose reconstructions were based on Nuclear Test Personnel Review records, Department of Defense. Standardized mortality ratios (SMR) and Cox and Poisson regression models were used in the analysis.

RESULTS

Most atomic veterans were enlisted men, served in the Navy at the PPG, and were born before 1930. Vital status was determined for 96.8% of the veterans; 60% had died. Enlisted men had significantly high all-causes mortality SMR (1.06); officers had significantly low all-causes mortality SMR (0.71). The pattern of risk over time showed a diminution of the 'healthy soldier effect': the all-causes mortality SMR after 50 years of follow-up was 1.00. The healthy soldier effect for all cancers also diminished over time. The all-cancer SMR was significantly high after 50 years (SMR 1.10) primarily from smoking-related cancers, attributed in part to the availability of cigarettes in military rations. The highest SMR was for mesothelioma (SMR 1.56) which was correlated with asbestos exposure in naval ships. Prostate cancer was significantly high (SMR 1.13). Ischemic heart disease was significantly low (SMR 0.84). Estimated mean doses varied by organ were low; e.g., the mean red bone marrow dose was 6 mGy (maximum 108 mGy). Internal cohort dose-response analyses provided no evidence for increasing trends with radiation dose for leukemia (excluding chronic lymphocytic leukemia (CLL)) [ERR (95% CI) per 100 mGy -0.37 (-1.08, 0.33); n = 710], CLL, myelodysplastic syndrome, multiple myeloma, ischemic heart disease, or cancers of the lung, prostate, breast, and brain.

CONCLUSION

No statistically significant radiation associations were observed among 114,270 nuclear weapons test participants followed for up to 65 years. The 95% confidence limits were narrow and excluded mortality risks per unit dose that are two to four times higher than those reported in other investigations. Significantly elevated SMRs were seen for mesothelioma and asbestosis, attributed to asbestos exposure aboard ships.

Can machine-learning methods really help predict suicide?

PURPOSE OF REVIEW

In recent years there has been interest in the use of machine learning in suicide research in reaction to the failure of traditional statistical methods to produce clinically useful models of future suicide. The current review summarizes recent prediction studies in the suicide literature including those using machine learning approaches to understand what value these novel approaches add.

RECENT FINDINGS

Studies using machine learning to predict suicide deaths report area under the curve that are only modestly greater than, and sensitivities that are equal to, those reported in studies using more conventional predictive methods. Positive predictive value remains around 1% among the cohort studies with a base rate that was not inflated by case-control methodology.

SUMMARY

Machine learning or artificial intelligence may afford opportunities in mental health research and in the clinical care of suicidal patients. However, application of such techniques should be carefully considered to avoid repeating the mistakes of existing methodologies. Prediction studies using machine-learning methods have yet to make a major contribution to our understanding of the field and are unproven as clinically useful tools.

Testing Suicide Risk Prediction Algorithms Using Phone Measurements With Patients in Acute Mental Health Settings: Feasibility Study

BACKGROUND

Digital phenotyping and machine learning are currently being used to augment or even replace traditional analytic procedures in many domains, including health care. Given the heavy reliance on smartphones and mobile devices around the world, this readily available source of data is an important and highly underutilized source that has the potential to improve mental health risk prediction and prevention and advance mental health globally.

OBJECTIVE

This study aimed to apply machine learning in an acute mental health setting for suicide risk prediction. This study uses a nascent approach, adding to existing knowledge by using data collected through a smartphone in place of clinical data, which have typically been collected from health care records.

METHODS

We created a smartphone app called Strength Within Me, which was linked to Fitbit, Apple Health kit, and Facebook, to collect salient clinical information such as sleep behavior and mood, step frequency and count, and engagement patterns with the phone from a cohort of inpatients with acute mental health (n=66). In addition, clinical research interviews were used to assess mood, sleep, and suicide risk. Multiple machine learning algorithms were tested to determine the best fit.

RESULTS

K-nearest neighbors (KNN; k=2) with uniform weighting and the Euclidean distance metric emerged as the most promising algorithm, with 68% mean accuracy (averaged over 10,000 simulations of splitting the training and testing data via 10-fold cross-validation) and an average area under the curve of 0.65. We applied a combined 5×2 F test to test the model performance of KNN against the baseline classifier that guesses training majority, random forest, support vector machine and logistic regression, and achieved F statistics of 10.7 (P=.009) and 17.6 (P=.003) for training majority and random forest, respectively, rejecting the null of performance being the same. Therefore, we have taken the first steps in prototyping a system that could continuously and accurately assess the risk of suicide via mobile devices.

CONCLUSIONS

Predicting for suicidality is an underaddressed area of research to which this paper makes a useful contribution. This is part of the first generation of studies to suggest that it is feasible to utilize smartphone-generated user input and passive sensor data to generate a risk algorithm among inpatients at suicide risk. The model reveals fair concordance between phone-derived and research-generated clinical data, and with iterative development, it has the potential for accurate discriminant risk prediction. However, although full automation and independence of clinical judgment or input would be a worthy development for those individuals who are less likely to access specialist mental health services, and for providing a timely response in a crisis situation, the ethical and legal implications of such advances in the field of psychiatry need to be acknowledged.

Evaluation of Prevention Efforts and Risk Factors Among Veteran Suicide Decedents Who Died by Firearm

OBJECTIVE

Veterans die by suicide at a higher rate than the U.S. population, and veterans more frequently use a firearm as the suicide method. Consequently, firearm accessibility and storage represent important prevention considerations. This project aimed to explore the implementation of suicide prevention efforts among veterans who went on to die by suicide, with and without the use of a firearm, and to identify factors that differentiated veteran suicide decedents to help inform suicide prevention efforts.

METHODS

Data from the Veteran Health Administration Behavior Health Autopsy Program was analyzed for 97 veteran suicide decedents.

RESULTS

Results demonstrated that veterans who used a firearm for suicide were less likely to have engaged in suicide prevention efforts overall and were less likely to have received lethal means safety counseling / safety planning. Veterans who died by firearm had lower levels of notable risk factors (e.g., prior suicide attempt, no-shows for appointments), however were more likely to have a documented unsecured firearm in their home.

CONCLUSION

These findings support the benefit of broadening the reach of suicide prevention efforts, especially for high-risk veterans with access to firearms.

Suicide in Belgian psychiatric inpatients. A matched case-control study in a Belgian teaching hospital

BACKGROUND AND OBJECTIVES

Patients admitted to a psychiatric hospital show an increased risk for suicide but specific risk factors are not well understood.

METHODS

In this case-control study we describe suicides (n = 37) that took place during admission in a Flemish psychiatric teaching hospital between 2007 and 2015 and investigate predictive factors for suicide.

RESULTS

Inpatient suicide is a rare condition (37 patients among 20,442 admission periods between 2007 and 2015). Most inpatients who completed suicide were diagnosed with a mood disorder (68%); 38% committed suicide in the first month of hospitalization and 19% in the first week following admission. The majority of suicides took place just before or during the weekend (57%), with hanging as the prominent method (41%). Multivariate analysis showed that hopelessness was the only significant risk factor for inpatient suicide.

CONCLUSIONS

Inpatient suicide remains a very rare event in inpatient care. Enquiring and managing hopelessness is essential in inpatient treatment of psychiatric patients.

Reaching Those at Highest Risk for Suicide: Development of a Model Using Machine Learning Methods for use With Native American Communities

OBJECTIVE

Suicide prevention is a major priority in Native American communities. We used machine learning with community-based suicide surveillance data to better identify those most at risk.

METHOD

This study leverages data from the Celebrating Life program operated by the White Mountain Apache Tribe in Arizona and in partnership with Johns Hopkins University. We examined N = 2,390 individuals with a validated suicide-related event between 2006 and 2017. Predictors included 73 variables (e.g., demographics, educational history, past mental health, and substance use). The outcome was suicide attempt 6, 12, and 24 months after an initial event. We tested four algorithmic approaches using cross-validation.

RESULTS

Area under the curves ranged from AUC = 0.81 (95% CI ± 0.08) for the decision tree classifiers to AUC = 0.87 (95% CI ± 0.04) for the ridge regression, results that were considerably higher than a past suicide attempt (AUC = 0.57; 95% CI ± 0.08). Selecting a cutoff value based on risk concentration plots yielded 0.88 sensitivity, 0.72 specificity, and a positive predictive value of 0.12 for detecting an attempt 24 months postindex event.

CONCLUSION

These models substantially improved our ability to determine who was most at risk in this community. Further work is needed including developing clinical guidance and external validation.

Using predictive analytics to improve pragmatic trial design

Clinical trials embedded in health systems can randomize large populations using automated data sources to determine trial eligibility and assess outcomes. The suicide prevention outreach trial used real-world data for trial design and randomized 18,868 individuals in four health systems using patient-reported thoughts of death or self-harm (Patient Health Questionnaire item 9). This took 3.5 years. We consider if using predictive analytics, that is, suicide risk estimates based on prediction models, could improve trial "efficiency." We used data on mental health outpatient visits between 1 January 2009 and 30 September 2017 in seven health systems (HealthPartners; Henry Ford Health System; and Colorado, Hawaii, Northwest, Southern California, and Washington Kaiser Permanente regions). We used a suicide risk prediction model developed in these same systems. We compared five trial designs with different eligibility criteria: a response of a 2 or 3 on Patient Health Questionnaire item 9, a response of a 3, suicide risk score above 90th, 95th, or 99th percentile. We compared the sample that met each criterion, 90-day suicide attempt rate following first eligible visit, and necessary sample sizes to detect a 15%, 25%, and 35% relative reduction in the suicide attempt rate, assuming 90% power, for each eligibility criterion. Our sample included 24,355,599 outpatient visits. Despite wide-spread use of Patient Health Questionnaire, 21,026,985 (86.3%) visits did not have a recorded Patient Health Questionnaire. Of the 2,928,927 individuals in our sample, 109,861 had a recorded Patient Health Questionnaire item 9 response of a 2 or 3 over the study years with a 1.40% 90-day suicide attempt rate and 50,047 had a response of a 3 (suicide attempt rate 1.98%). More patients met criteria requiring a certain risk score or higher: 331,273 had a 90th percentile risk score or higher (suicide attempt rate: 1.36%); 182,316 a 95th percentile or higher (suicide attempt rate 2.16%), and 78,655 a 99th percentile or higher (suicide attempt rate: 3.95%). Eligibility criterion of a Patient Health Questionnaire item 9 response of a 2 or 3 would require randomizing 44,081 individuals (40.2% of eligible population in our sample); eligibility criterion of a 3 would require 31,024 individuals (62.0% of eligible population). Eligibility criterion of a suicide risk score of 90th percentile or higher would require 45,675 individuals (13.8% of eligible population), 95th percentile 28,699 individuals (15.7% of eligible population), and 99th percentile 15,509 (19.7% of eligible population). A suicide risk prediction calculator could improve trial "efficiency"; identifying more individuals at increased suicide risk than relying on patient-report. It is an open scientific question if individuals identified using predictive analytics would respond differently to interventions than those identified by more traditional means.

Short-term risk of suicide attempt associated with patterns of patient-reported alcohol use determined by routine AUDIT-C among adults receiving mental healthcare

OBJECTIVE

To evaluate the association between alcohol use routinely reported during outpatient mental healthcare visits and short-term risk of subsequent suicide attempt.

METHODS

Using a longitudinal retrospective-cohort design, electronic health records identified adult outpatient visits to a mental health provider (1/1/2010-6/30/2015) at Kaiser Permanente Washington with a documented Alcohol Use Disorders Identification Test-Consumption [AUDIT-C]. Suicide attempts within 90 days of AUDIT-C documentation were defined using death certificate cause-of-death and diagnosis codes (non-lethal). Visit-level analyses used generalized estimating equations to account for correlation between multiple AUDIT-Cs for individuals. Separate models evaluated the association between (1) level of consumption and (2) frequency of heavy drinking episodes and suicide attempts, adjusted for visit year, demographics, depressive symptom, and suicidal ideation.

RESULTS

Of 59,382 patient visits, 0.62% (N = 371) were followed by a suicide attempt within 90 days. Patients reporting high-level alcohol use were 1.77 times (95% CI, 1.22-2.57) more likely to attempt suicide than those reporting low-level use. Patients reporting daily or almost daily heavy drinking episodes were 2.33 times (95% CI, 1.38-3.93) more likely to attempt suicide than those reporting none.

CONCLUSIONS AND RELEVANCE

The AUDIT-C is a valuable tool for assessing patterns of patient-reported alcohol use associated with subsequent suicide attempt.

Suicide prediction models: a critical review of recent research with recommendations for the way forward

Suicide is a leading cause of death. A substantial proportion of the people who die by suicide come into contact with the health care system in the year before their death. This observation has resulted in the development of numerous suicide prediction tools to help target patients for preventive interventions. However, low sensitivity and low positive predictive value have led critics to argue that these tools have no clinical value. We review these tools and critiques here. We conclude that existing tools are suboptimal and that improvements, if they can be made, will require developers to work with more comprehensive predictor sets, staged screening designs, and advanced statistical analysis methods. We also conclude that although existing suicide prediction tools currently have little clinical value, and in some cases might do more harm than good, an even-handed assessment of the potential value of refined tools of this sort cannot currently be made because such an assessment would depend on evidence that currently does not exist about the effectiveness of preventive interventions. We argue that the only way to resolve this uncertainty is to link future efforts to develop or evaluate suicide prediction tools with concrete questions about specific clinical decisions aimed at reducing suicides and to evaluate the clinical value of these tools in terms of net benefit rather than sensitivity or positive predictive value. We also argue for a focus on the development of individualized treatment rules to help select the right suicide-focused treatments for the right patients at the right times. Challenges will exist in doing this because of the rarity of suicide even among patients considered high-risk, but we offer practical suggestions for how these challenges can be addressed.

SAVE-CLC: An Intervention to Reduce Suicide Risk in Older Veterans following Discharge from VA Nursing Facilities

Objective: We describe the development and implementation of a telephonic intervention (SAVE-CLC) piloted at three VA sites for Veterans returning to the community from VA nursing facilities (Community Living Centers or "CLCs"). Care transitions present a known period of medical risk for older adults and may pose increased risk for suicide. Veterans discharging from CLCs are at elevated risk compared to age and gender matched controls.Methods: Using a quality improvement approach, input was gathered from key stakeholders to aid in the development of the intervention. Veterans were screened for depressive symptoms and need for additional support by phone.Results: Of the Veterans who received the SAVE-CLC intervention, 87.9% had at least one prior mental health diagnosis, though only 19.7% had an outpatient mental health appointment arranged at CLC discharge. Results suggest that the intervention is feasible across multiple outpatient settings and is generally well-received by Veterans and caregivers, with 97% of those contacted reporting that the telephone calls were helpful.Conclusion: This flexible, telephone-based intervention addresses the unmet need of integrating mental health care into discharge planning during care transitions.Clinical Implications: SAVE-CLC offers a feasible and acceptable solution to suicide risk in older Veterans exiting a CLC.

Artificial Intelligence for Mental Health and Mental Illnesses: an Overview

PURPOSE OF REVIEW

Artificial intelligence (AI) technology holds both great promise to transform mental healthcare and potential pitfalls. This article provides an overview of AI and current applications in healthcare, a review of recent original research on AI specific to mental health, and a discussion of how AI can supplement clinical practice while considering its current limitations, areas needing additional research, and ethical implications regarding AI technology.

RECENT FINDINGS

We reviewed 28 studies of AI and mental health that used electronic health records (EHRs), mood rating scales, brain imaging data, novel monitoring systems (e.g., smartphone, video), and social media platforms to predict, classify, or subgroup mental health illnesses including depression, schizophrenia or other psychiatric illnesses, and suicide ideation and attempts. Collectively, these studies revealed high accuracies and provided excellent examples of AI's potential in mental healthcare, but most should be considered early proof-of-concept works demonstrating the potential of using machine learning (ML) algorithms to address mental health questions, and which types of algorithms yield the best performance. As AI techniques continue to be refined and improved, it will be possible to help mental health practitioners re-define mental illnesses more objectively than currently done in the DSM-5, identify these illnesses at an earlier or prodromal stage when interventions may be more effective, and personalize treatments based on an individual's unique characteristics. However, caution is necessary in order to avoid over-interpreting preliminary results, and more work is required to bridge the gap between AI in mental health research and clinical care.

Comparison of National and Local Approaches to Detecting Suicides in Healthcare Settings

Introduction:

In order to address the problem of suicide, healthcare providers and researchers must be able to accurately identify suicide deaths. Common approaches to detecting suicide in the healthcare setting include the National Death Index (NDI) and Root-Cause Analysis (RCA) methodology. No study has directly compared these two methods.

Materials and Methods:

Suicide reporting was evaluated within the Veterans Affairs (VA) healthcare system. All suicides were included that occurred within 7 days of discharge from an inpatient mental health unit and were reported to the VA through the NDI record linkage and/or RCA database between 2002 and 2014. The proportion of suicide deaths that were identified by NDI and found in the RCA database were calculated. Potential misclassification by the NDI was evaluated, whereby the RCA database identified a suicide case, but the NDI classified the death as a non-suicide.

Results:

In the study period, the NDI identified 222 patients who died by suicide within 7 days of discharge, while the RCA database only detected 95 reports of suicide. A comparison of cases across the two methods indicated that the RCA database identified only 35% (N = 78) of NDI detected suicides (N = 222). Conversely, the RCA database detected 13 suicide cases that the NDI had coded as deaths due to accidental poisoning or other causes. Importantly, RCA accounted for 13% (N = 7) of overdose suicides identified in all databases (N = 52).

Conclusions:

Combining national and local approaches to detect suicide may help to improve the classification of suicide deaths in the healthcare setting.