Connecting Research and Practice: Implementation of Suicide Prevention Strategies in Learning Health Care Systems

Stability of Suicide Risk Prediction Models During Changes in Health Care Delivery

OBJECTIVE

The authors aimed to use health records data to examine how the accuracy of statistical models predicting self-harm or suicide changed between 2015 and 2019, as health systems implemented suicide prevention programs.

METHODS

Data from four large health systems were used to identify specialty mental health visits by patients ages ≥11 years, assess 311 potential predictors of self-harm (including demographic characteristics, historical risk factors, and index visit characteristics), and ascertain fatal or nonfatal self-harm events over 90 days after each visit. New prediction models were developed with logistic regression with LASSO (least absolute shrinkage and selection operator) in random samples of visits (65%) from each calendar year and were validated in the remaining portion of the sample (35%).

RESULTS

A model developed for visits from 2009 to mid-2015 showed similar classification performance and calibration accuracy in a new sample of about 13.1 million visits from late 2015 to 2019. Area under the receiver operating characteristic curve (AUC) ranged from 0.840 to 0.849 in the new sample, compared with 0.851 in the original sample. New models developed for each year for 2015-2019 had classification performance (AUC range 0.790-0.853), sensitivity, and positive predictive value similar to those of the previously developed model. Models selected similar predictors from 2015 to 2019, except for more frequent selection of depression questionnaire data in later years, when questionnaires were more frequently recorded.

CONCLUSIONS

A self-harm prediction model developed with 2009-2015 visit data performed similarly when applied to 2015-2019 visits. New models did not yield superior performance or identify different predictors.

Racial-Ethnic Differences in Receipt of Past-Year Health Care Services Among Suicide Decedents: A Case-Control Study

OBJECTIVE

Suicide remains an urgent public health crisis. Although some sociodemographic characteristics are associated with greater suicide risk in the general population, it is unclear whether individuals utilizing health care in the United States have similar suicide incidence patterns. The authors examined whether race-ethnicity is associated with suicide death among patients seeking health care and investigated health care utilization patterns.

METHODS

Data were collected from electronic health records and government mortality records for patients seeking health care across nine health care systems in the United States. Patients who died by suicide (N=1,935) were matched with patients in a control group (N=19,350) within each health care system.

RESULTS

Patients who died by suicide were significantly more likely to be White, older, male, living in low-education areas, living in rural areas, or diagnosed as having mental health conditions or were significantly less likely to have commercial insurance (p<0.05). Among most racial-ethnic groups, those who died by suicide had a higher number of past-year mental health, primary care, and total health care visits; for American Indian/Alaska Native patients, the number of health care visits tended to be lower among suicide decedents.

CONCLUSIONS

These findings suggest that higher past-year health care utilization was associated with increased likelihood of suicide death across several racial-ethnic groups. This observation underscores the need for identifying and managing suicide risk in health care settings, including outside of mental health visits, among most racial-ethnic groups.

Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction

Suicide risk prediction models can identify individuals for targeted intervention. Discussions of transparency, explainability, and transportability in machine learning presume complex prediction models with many variables outperform simpler models. We compared random forest, artificial neural network, and ensemble models with 1500 temporally defined predictors to logistic regression models. Data from 25,800,888 mental health visits made by 3,081,420 individuals in 7 health systems were used to train and evaluate suicidal behavior prediction models. Model performance was compared across several measures. All models performed well (area under the receiver operating curve [AUC]: 0.794-0.858). Ensemble models performed best, but improvements over a regression model with 100 predictors were minimal (AUC improvements: 0.006-0.020). Results are consistent across performance metrics and subgroups defined by race, ethnicity, and sex. Our results suggest simpler parametric models, which are easier to implement as part of routine clinical practice, perform comparably to more complex machine learning methods.

Use of an Agitation Measure to Screen for Suicide and Self-Harm Risk Among Emergency Department Patients

BACKGROUND

Suicidality alone is insensitive to suicide risk among emergency department (ED) patients.

OBJECTIVE

We describe the performance of adding an objective assessment of agitation to a suicide screening instrument for predicting suicide and self-harm after an ED encounter.

METHODS

We tested the performance of a novel screener combining the presence of suicidality or agitation for predicting suicide within 90 days or a repeat ED visit for self-harm within 30 days using retrospective data from all patients seen in an urban safety net ED over 27 months. Patients were assessed for suicidality using the Columbia-Suicide Severity Rating Scale-Clinical Practice Screener and for agitation using either the Behavioral Activity Rating Scale or Richmond Agitation Sedation Scale. We hypothesized that a screener based on the presence of either suicidality or agitation would be more sensitive to suicide risk than the Columbia-Suicide Severity Rating Scale-Clinical Practice Screener alone. The screener's performance is described, and multivariable regression evaluates the correlations between screening and outcomes.

RESULTS

The sample comprised 16,467 patients seen in the ED who had available suicide screening and agitation data. Thirteen patients (0.08%) died by suicide within 90 days after ED discharge. The sensitivity and specificity of the screener combining suicidality and agitation for predicting suicide was 0.69 (95% confidence interval, 0.44-0.94) and 0.74 (0.44-0.94), respectively. The sensitivity and specificity for agitation combined with positive suicide screening for self-harm within 30 days were 0.95 (0.89-1.00) and 0.73 (0.73-0.74). For both outcomes, augmenting the Columbia-Suicide Severity Rating Scale-Clinical Practice Screener with a measure of agitation improved both sensitivity and overall performance compared to historical performance of the Columbia-Suicide Severity Rating Scale-Clinical Practice Screener alone.

CONCLUSIONS

Combining a brief objective measure of agitation with a common suicide screening instrument improved sensitivity and predictive performance for suicide and self-harm risk after ED discharge. These findings speak to the importance of assessing agitation not only for imminent safety risk during the patient encounter but also for reducing the likelihood of future adverse events. This work can improve the detection and management of suicide risk in emergency settings.

The Paradox of Suicide Prevention

The recognition that we cannot use risk stratification (high, medium, low) to predict suicide or to allocate resources has led to a paradigm shift in suicide prevention efforts. There are challenges in adapting to these new paradigms, including reluctance of clinicians and services to move away from traditional risk categorisations; and conversely, the risk of a pendulum swing in which the focus of care swings from one approach to determining service priority and focus (e.g., diagnosis, formulation, risk and clinical care) to a new focus (e.g., suicide specific and non-clinical care), potentially supplanting the previous approach. This paper argues that the Prevention Paradox provides a useful mental model to support a shift in paradigm, whilst maintaining a balanced approach that incorporates new paradigms within the effective aspects of existing ones. The Prevention Paradox highlights the seemingly paradoxical situation where the greatest burden of disease or death is caused by those at low to moderate risk due their larger numbers. Current planning frameworks and resources do not support successful or sustainable adoption of these new approaches, leading to missed opportunities to prevent suicidal behaviours in healthcare. Adopting systems approaches to suicide prevention, such as the Zero Suicide Framework, implemented in a large mental health service in Australia and presented in this paper as a case study, can support a balanced approach of population- and individual-based suicide prevention efforts. Results demonstrate significant reductions in re-presentations with suicide attempts for consumers receiving this model of care; however, the increasing numbers of placements compromise the capacity of clinical teams to complete all components of standardised pathway of care. This highlights the need for review of resource planning frameworks and ongoing evaluations of the critical aspects of the interventions.

Evaluating the heterogeneous effect of a modifiable risk factor on suicide: The case of vitamin D deficiency

OBJECTIVES

To illustrate the use of machine learning methods to search for heterogeneous effects of a target modifiable risk factor on suicide in observational studies. The illustration focuses on secondary analysis of a matched case-control study of vitamin D deficiency predicting subsequent suicide.

METHODS

We describe a variety of machine learning methods to search for prescriptive predictors; that is, predictors of significant variation in the association between a target risk factor and subsequent suicide. In each case, the purpose is to evaluate the potential value of selective intervention on the target risk factor to prevent the outcome based on the provisional assumption that the target risk factor is causal. The approaches illustrated include risk modeling based on the super learner ensemble machine learning method, Least Absolute Shrinkage and Selection Operator (Lasso) penalized regression, and the causal forest algorithm.

RESULTS

The logic of estimating heterogeneous intervention effects is exposited along with the illustration of some widely used methods for implementing this logic.

CONCLUSIONS

In addition to describing best practices in using the machine learning methods considered here, we close with a discussion of broader design and analysis issues in planning an observational study to investigate heterogeneous effects of a modifiable risk factor.