Implementation of Suicide Risk Estimation Analytics to Support Mental Health Care for Quality Improvement

Acceptability of linking individual credit, financial, and public records data to healthcare records for suicide risk machine learning models

Objectives

Individual-level information about negative life events (NLE) such as bankruptcy, foreclosure, divorce, and criminal arrest might improve the accuracy of machine learning models for suicide risk prediction. Individual-level NLE data is routinely collected by vendors such as Equifax. However, little is known about the acceptability of linking this NLE data to healthcare data. Our objective was to assess preferences for linking external NLE data to healthcare records for suicide prevention.

Materials and Methods

We conducted a discrete choice experiment (DCE) among Kaiser Permanente Washington (KPWA) members. Patient partners assisted in the design and pretesting of the DCE survey. The DCE included 12 choice tasks involving 4 data linking program attributes and 3 levels within each attribute. We estimated latent class conditional logit models to derive preference weights.

Results

There were 743 participants. Willingness to link data varied by type of information to be linked, demographic characteristics, and experience with NLE. Overall, 65.1% of people were willing to link data and 34.9% were more private. Trust in KPWA to safeguard data was the strongest predictor of willingness to link data.

Discussion

Most respondents supported linking NLE data for suicide prevention. Contrary to expectations, People of Color and people who reported experience with NLEs were more likely to be willing to link their data.

Conclusions

A majority of participants were willing to have their credit and public records data linked to healthcare records provided that conditions are in place to protect privacy and autonomy.

Developing a suicide risk model for use in the Indian Health Service

We developed and evaluated an electronic health record (EHR)-based model for suicide risk specific to an American Indian patient population. Using EHR data for all patients over 18 with a visit between 1/1/2017 and 10/2/2021, we developed a model for the risk of a suicide attempt or death in the 90 days following a visit. Features included demographics, medications, diagnoses, and scores from relevant screening tools. We compared the predictive performance of logistic regression and random forest models against existing suicide screening, which was augmented to include the history of previous attempts or ideation. During the study, 16,835 patients had 331,588 visits, with 490 attempts and 37 deaths by suicide. The logistic regression and random forest models (area under the ROC (AUROC) 0.83 [0.80-0.86]; both models) performed better than enhanced screening (AUROC 0.64 [0.61-0.67]). These results suggest that an EHR-based suicide risk model can add value to existing practices at Indian Health Service clinics.

Effectiveness of Integrating Suicide Care in Primary Care : Secondary Analysis of a Stepped-Wedge, Cluster Randomized Implementation Trial

BACKGROUND

Primary care encounters are common among patients at risk for suicide.

OBJECTIVE

To evaluate the effectiveness of implementing population-based suicide care (SC) in primary care for suicide attempt prevention.

DESIGN

Secondary analysis of a stepped-wedge, cluster randomized implementation trial. (ClinicalTrials.gov: NCT02675777).

SETTING

19 primary care practices within a large health care system in Washington State, randomly assigned launch dates.

PATIENTS

Adult patients (aged ≥18 years) with primary care visits from January 2015 to July 2018.

INTERVENTION

Practice facilitators, electronic medical record (EMR) clinical decision support, and performance monitoring supported implementation of depression screening, suicide risk assessment, and safety planning.

MEASUREMENTS

Clinical practice and patient measures relied on EMR and insurance claims data to compare usual care (UC) and SC periods. Primary outcomes included documented safety planning after population-based screening and suicide risk assessment and suicide attempts or deaths (with self-harm intent) within 90 days of a visit. Mixed-effects logistic models regressed binary outcome indicators on UC versus SC, adjusted for randomization stratification and calendar time, accounting for repeated outcomes from the same site. Monthly outcome rates (percentage per 10 000 patients) were estimated by applying marginal standardization.

RESULTS

During UC, 255 789 patients made 953 402 primary care visits and 228 255 patients made 615 511 visits during the SC period. The rate of safety planning was higher in the SC group than in the UC group (38.3 vs. 32.8 per 10 000 patients; rate difference, 5.5 [95% CI, 2.3 to 8.7]). Suicide attempts within 90 days were lower in the SC group than in the UC group (4.5 vs. 6.0 per 10 000 patients; rate difference, -1.5 [CI, -2.6 to -0.4]).

LIMITATION

Suicide care was implemented in combination with care for depression and substance use.

CONCLUSION

Implementation of population-based SC concurrent with a substance use program resulted in a 25% reduction in the suicide attempt rate in the 90 days after primary care visits.

PRIMARY FUNDING SOURCE

National Institute of Mental Health.

Validation of a Multivariable Model to Predict Suicide Attempt in a Mental Health Intake Sample

Importance

Given that suicide rates have been increasing over the past decade and the demand for mental health care is at an all-time high, targeted prevention efforts are needed to identify individuals seeking to initiate mental health outpatient services who are at high risk for suicide. Suicide prediction models have been developed using outpatient mental health encounters, but their performance among intake appointments has not been directly examined.

Objective

To assess the performance of a predictive model of suicide attempts among individuals seeking to initiate an episode of outpatient mental health care.

Design, Setting, and Participants

This prognostic study tested the performance of a previously developed machine learning model designed to predict suicide attempts within 90 days of any mental health outpatient visit. All mental health intake appointments scheduled between January 1, 2012, and April 1, 2022, at Kaiser Permanente Northern California, a large integrated health care delivery system serving over 4.5 million patients, were included. Data were extracted and analyzed from August 9, 2022, to July 31, 2023.

Main Outcome and Measures

Suicide attempts (including completed suicides) within 90 days of the appointment, determined by diagnostic codes and government databases. All predictors were extracted from electronic health records.

Results

The study included 1 623 232 scheduled appointments from 835 616 unique patients. There were 2800 scheduled appointments (0.17%) followed by a suicide attempt within 90 days. The mean (SD) age across appointments was 39.7 (15.8) years, and most appointments were for women (1 103 184 [68.0%]). The model had an area under the receiver operating characteristic curve of 0.77 (95% CI, 0.76-0.78), an area under the precision-recall curve of 0.02 (95% CI, 0.02-0.02), an expected calibration error of 0.0012 (95% CI, 0.0011-0.0013), and sensitivities of 37.2% (95% CI, 35.5%-38.9%) and 18.8% (95% CI, 17.3%-20.2%) at specificities of 95% and 99%, respectively. The 10% of appointments at the highest risk level accounted for 48.8% (95% CI, 47.0%-50.6%) of the appointments followed by a suicide attempt.

Conclusions and Relevance

In this prognostic study involving mental health intakes, a previously developed machine learning model of suicide attempts showed good overall classification performance. Implementation research is needed to determine appropriate thresholds and interventions for applying the model in an intake setting to target high-risk cases in a manner that is acceptable to patients and clinicians.

Adolescent, Parent, and Provider Perceptions of a Predictive Algorithm to Identify Adolescent Suicide Risk in Primary Care

OBJECTIVE

To understand adolescent, parent, and provider perceptions of a machine learning algorithm for detecting adolescent suicide risk prior to its implementation primary care.

METHODS

We conducted semi-structured, qualitative interviews with adolescents (n = 9), parents (n = 12), and providers (n = 10; mixture of behavioral health and primary care providers) across two major health systems. Interviews were audio recorded and transcribed with analyses supported by use of NVivo. A codebook was developed combining codes derived inductively from interview transcripts and deductively from implementation science frameworks for content analysis.

RESULTS

Reactions to the algorithm were mixed. While many participants expressed privacy concerns, they believed the algorithm could be clinically useful for identifying adolescents at risk for suicide and facilitating follow-up. Parents' past experiences with their adolescents' suicidal thoughts and behaviors contributed to their openness to the algorithm. Results also aligned with several key Consolidated Framework for Implementation Research domains. For example, providers mentioned barriers inherent to the primary care setting such as time and resource constraints likely to impact algorithm implementation. Participants also cited a climate of mistrust of science and health care as potential barriers.

CONCLUSIONS

Findings shed light on factors that warrant consideration to promote successful implementation of suicide predictive algorithms in pediatric primary care. By attending to perspectives of potential end users prior to the development and testing of the algorithm, we can ensure that the risk prediction methods will be well-suited to the providers who would be interacting with them and the families who could benefit.

Randomized Controlled Comparative Effectiveness Trial of Risk Model-Guided Clinical Decision Support for Suicide Screening

Suicide prevention requires risk identification, appropriate intervention, and follow-up. Traditional risk identification relies on patient self-reporting, support network reporting, or face-to-face screening with validated instruments or history and physical exam. In the last decade, statistical risk models have been studied and more recently deployed to augment clinical judgment. Models have generally been found to be low precision or problematic at scale due to low incidence. Few have been tested in clinical practice, and none have been tested in clinical trials to our knowledge.

Methods

We report the results of a pragmatic randomized controlled trial (RCT) in three outpatient adult Neurology clinic settings. This two-arm trial compared the effectiveness of Interruptive and Non-Interruptive Clinical Decision Support (CDS) to prompt further screening of suicidal ideation for those predicted to be high risk using a real-time, validated statistical risk model of suicide attempt risk, with the decision to screen as the primary end point. Secondary outcomes included rates of suicidal ideation and attempts in both arms. Manual chart review of every trial encounter was used to determine if suicide risk assessment was subsequently documented.

Results

From August 16, 2022, through February 16, 2023, our study randomized 596 patient encounters across 561 patients for providers to receive either Interruptive or Non-Interruptive CDS in a 1:1 ratio. Adjusting for provider cluster effects, Interruptive CDS led to significantly higher numbers of decisions to screen (42%=121/289 encounters) compared to Non-Interruptive CDS (4%=12/307) (odds ratio=17.7, p-value <0.001). Secondarily, no documented episodes of suicidal ideation or attempts occurred in either arm. While the proportion of documented assessments among those noting the decision to screen was higher for providers in the Non-Interruptive arm (92%=11/12) than in the Interruptive arm (52%=63/121), the interruptive CDS was associated with more frequent documentation of suicide risk assessment (63/289 encounters compared to 11/307, p-value<0.001).

Conclusions

In this pragmatic RCT of real-time predictive CDS to guide suicide risk assessment, Interruptive CDS led to higher numbers of decisions to screen and documented suicide risk assessments. Well-powered large-scale trials randomizing this type of CDS compared to standard of care are indicated to measure effectiveness in reducing suicidal self-harm. ClinicalTrials.gov Identifier: NCT05312437.

Stability of Suicide Risk Prediction Models During Changes in Health Care Delivery

OBJECTIVE

The authors aimed to use health records data to examine how the accuracy of statistical models predicting self-harm or suicide changed between 2015 and 2019, as health systems implemented suicide prevention programs.

METHODS

Data from four large health systems were used to identify specialty mental health visits by patients ages ≥11 years, assess 311 potential predictors of self-harm (including demographic characteristics, historical risk factors, and index visit characteristics), and ascertain fatal or nonfatal self-harm events over 90 days after each visit. New prediction models were developed with logistic regression with LASSO (least absolute shrinkage and selection operator) in random samples of visits (65%) from each calendar year and were validated in the remaining portion of the sample (35%).

RESULTS

A model developed for visits from 2009 to mid-2015 showed similar classification performance and calibration accuracy in a new sample of about 13.1 million visits from late 2015 to 2019. Area under the receiver operating characteristic curve (AUC) ranged from 0.840 to 0.849 in the new sample, compared with 0.851 in the original sample. New models developed for each year for 2015-2019 had classification performance (AUC range 0.790-0.853), sensitivity, and positive predictive value similar to those of the previously developed model. Models selected similar predictors from 2015 to 2019, except for more frequent selection of depression questionnaire data in later years, when questionnaires were more frequently recorded.

CONCLUSIONS

A self-harm prediction model developed with 2009-2015 visit data performed similarly when applied to 2015-2019 visits. New models did not yield superior performance or identify different predictors.