What health records data are required for accurate prediction of suicidal behavior?

Suicide Death Prediction Using the Maryland Suicide Data Warehouse: A Sensitivity Analysis

OBJECTIVE

Nearly 50,000 Americans die each year from suicide, despite suicide death being a rare event in the context of health risk assessment and modeling. Prior research has underscored the need for contextualizing suicide risk models in terms of their potential uses and generalizability. This sensitivity analysis makes use of the Maryland Suicide Data Warehouse (MSDW) and illustrates how results inform clinical decision support.

METHOD

A cohort of 1 million living control patients were extracted from the MSDW in addition to 1,667 patients who had died by suicide between the years 2016 and 2019 according to the Maryland Office of the Medical Examiner (OCME). Data were extracted and aggregated as part of a 4-year retrospective design. Binary logistic and two penalized regression models were deployed in a repeated fivefold cross-validation. Model performances were evaluated using sensitivity, positive predictive value (PPV), and F1, and model coefficients were ranked according to coefficient size.

RESULTS

Several features were significantly associated with patients having died by suicide, including male sex, depressive and anxiety disorder diagnoses, social needs, and prior suicidal ideation and suicide attempt. Cross-validated binary logistic regression outperformed either ridge or LASSO (least absolute shrinkage and selection operator) models but generally achieved low-to-moderate PPV and sensitivity across most thresholds and a peak F1 of 0.323.

CONCLUSIONS

Suicide death prediction is constrained by the context of use, which determines the best balance of precision and recall. Predictive models must be evaluated close to the level of intervention. They may not hold up to different needs at different levels of care.

High-density lipoprotein (HDL) as an indicator for alcohol use in a psychiatrically ill population

AIMS

To examine the cross sectional and longitudinal associations between the Alcohol Use Disorders Identification Test-Concise (AUDIT-C) and differences in high-density lipoprotein (HDL) in a psychiatrically ill population.

METHODS

Retrospective observational study using electronic health record data from a large healthcare system, of patients hospitalized for a mental health/substance use disorder (MH/SUD) from 1 July 2016 to 31 May 2023, who had a proximal AUDIT-C and HDL (N = 15 915) and the subset who had a repeat AUDIT-C and HDL 1 year later (N = 2915). Linear regression models examined the association between cross-sectional and longitudinal AUDIT-C scores and HDL, adjusting for demographic and clinical characteristics that affect HDL.

RESULTS

Compared with AUDIT-C score = 0, HDL was higher among patients with greater AUDIT-C severity (e.g. moderate AUDIT-C score = 8.70[7.65, 9.75] mg/dl; severe AUDIT-C score = 13.02 [12.13, 13.90] mg/dL[95% confidence interval (CI)] mg/dl). The associations between cross-sectional HDL and AUDIT-C scores were similar with and without adjusting for patient demographic and clinical characteristics. HDL levels increased for patients with mild alcohol use at baseline and moderate or severe alcohol use at follow-up (15.06[2.77, 27.69] and 19.58[2.77, 36.39] mg/dL[95%CI] increase for moderate and severe, respectively).

CONCLUSIONS

HDL levels correlate with AUDIT-C scores among patients with MH/SUD. Longitudinally, there were some (but not consistent) increases in HDL associated with increases in AUDIT-C. The increases were within range of typical year-to-year variation in HDL across the population independent of alcohol use, limiting the ability to use HDL as a longitudinal clinical indicator for alcohol use in routine care.

Stability of Suicide Risk Prediction Models During Changes in Health Care Delivery

OBJECTIVE

The authors aimed to use health records data to examine how the accuracy of statistical models predicting self-harm or suicide changed between 2015 and 2019, as health systems implemented suicide prevention programs.

METHODS

Data from four large health systems were used to identify specialty mental health visits by patients ages ≥11 years, assess 311 potential predictors of self-harm (including demographic characteristics, historical risk factors, and index visit characteristics), and ascertain fatal or nonfatal self-harm events over 90 days after each visit. New prediction models were developed with logistic regression with LASSO (least absolute shrinkage and selection operator) in random samples of visits (65%) from each calendar year and were validated in the remaining portion of the sample (35%).

RESULTS

A model developed for visits from 2009 to mid-2015 showed similar classification performance and calibration accuracy in a new sample of about 13.1 million visits from late 2015 to 2019. Area under the receiver operating characteristic curve (AUC) ranged from 0.840 to 0.849 in the new sample, compared with 0.851 in the original sample. New models developed for each year for 2015-2019 had classification performance (AUC range 0.790-0.853), sensitivity, and positive predictive value similar to those of the previously developed model. Models selected similar predictors from 2015 to 2019, except for more frequent selection of depression questionnaire data in later years, when questionnaires were more frequently recorded.

CONCLUSIONS

A self-harm prediction model developed with 2009-2015 visit data performed similarly when applied to 2015-2019 visits. New models did not yield superior performance or identify different predictors.

Determining Distinct Suicide Attempts From Recurrent Electronic Health Record Codes: Classification Study

BACKGROUND

Prior suicide attempts are a relatively strong risk factor for future suicide attempts. There is growing interest in using longitudinal electronic health record (EHR) data to derive statistical risk prediction models for future suicide attempts and other suicidal behavior outcomes. However, model performance may be inflated by a largely unrecognized form of "data leakage" during model training: diagnostic codes for suicide attempt outcomes may refer to prior attempts that are also included in the model as predictors.

OBJECTIVE

We aimed to develop an automated rule for determining when documented suicide attempt diagnostic codes identify distinct suicide attempt events.

METHODS

From a large health care system's EHR, we randomly sampled suicide attempt codes for 300 patients with at least one pair of suicide attempt codes documented at least one but no more than 90 days apart. Supervised chart reviewers assigned the clinical settings (ie, emergency department [ED] versus non-ED), methods of suicide attempt, and intercode interval (number of days). The probability (or positive predictive value) that the second suicide attempt code in a given pair of codes referred to a distinct suicide attempt event from its preceding suicide attempt code was calculated by clinical setting, method, and intercode interval.

RESULTS

Of 1015 code pairs reviewed, 835 (82.3%) were nonindependent (ie, the 2 codes referred to the same suicide attempt event). When the second code in a pair was documented in a clinical setting other than the ED, it represented a distinct suicide attempt 3.3% of the time. The more time elapsed between codes, the more likely the second code in a pair referred to a distinct suicide attempt event from its preceding code. Code pairs in which the second suicide attempt code was assigned in an ED at least 5 days after its preceding suicide attempt code had a positive predictive value of 0.90.

CONCLUSIONS

EHR-based suicide risk prediction models that include International Classification of Diseases codes for prior suicide attempts as a predictor may be highly susceptible to bias due to data leakage in model training. We derived a simple rule to distinguish codes that reflect new, independent suicide attempts: suicide attempt codes documented in an ED setting at least 5 days after a preceding suicide attempt code can be confidently treated as new events in EHR-based suicide risk prediction models. This rule has the potential to minimize upward bias in model performance when prior suicide attempts are included as predictors in EHR-based suicide risk prediction models.

Predicting risk of suicidal behavior from insurance claims data vs. linked data from insurance claims and electronic health records

PURPOSE

Observational studies assessing effects of medical products on suicidal behavior often rely on health record data to account for pre-existing risk. We assess whether high-dimensional models predicting suicide risk using data derived from insurance claims and electronic health records (EHRs) are superior to models using data from insurance claims alone.

METHODS

Data were from seven large health systems identified outpatient mental health visits by patients aged 11 or older between 1/1/2009 and 9/30/2017. Data for the 5 years prior to each visit identified potential predictors of suicidal behavior typically available from insurance claims (e.g., mental health diagnoses, procedure codes, medication dispensings) and additional potential predictors available from EHRs (self-reported race and ethnicity, responses to Patient Health Questionnaire or PHQ-9 depression questionnaires). Nonfatal self-harm events following each visit were identified from insurance claims data and fatal self-harm events were identified by linkage to state mortality records. Random forest models predicting nonfatal or fatal self-harm over 90 days following each visit were developed in a 70% random sample of visits and validated in a held-out sample of 30%. Performance of models using linked claims and EHR data was compared to models using claims data only.

RESULTS

Among 15 845 047 encounters by 1 574 612 patients, 99 098 (0.6%) were followed by a self-harm event within 90 days. Overall classification performance did not differ between the best-fitting model using all data (area under the receiver operating curve or AUC = 0.846, 95% CI 0.839-0.854) and the best-fitting model limited to data available from insurance claims (AUC = 0.846, 95% CI 0.838-0.853). Competing models showed similar classification performance across a range of cut-points and similar calibration performance across a range of risk strata. Results were similar when the sample was limited to health systems and time periods where PHQ-9 depression questionnaires were recorded more frequently.

CONCLUSION

Investigators using health record data to account for pre-existing risk in observational studies of suicidal behavior need not limit that research to databases including linked EHR data.

Prediction models of suicide and non-fatal suicide attempt after discharge from a psychiatric inpatient stay: A machine learning approach on nationwide Danish registers

INTRODUCTION

To develop machine learning models capable of predicting suicide and non-fatal suicide attempt as separate outcomes in the first 30 days after discharge from a psychiatric inpatient stay.

METHODS

Prospective cohort study using nationwide Danish registry data. We included individuals who were 18 years or older, and all discharges from psychiatric hospitalizations in Denmark from 1995 to 2018. We trained predictive models using 10-fold cross validation on 80% of the data and did testing on the remaining 20%.

RESULTS

The best model for predicting non-fatal suicide attempt was an ensemble of predictions from gradient boosting (XGBoost) and categorical boosting (catBoost). The ROC-AUC for predicting suicide attempt was 0.85 (95% CI: 0.84-0.85). At a risk threshold of 4.36%, positive predictive value (PPV) was 11.0% and sensitivity was 47.2%. The best model for predicting suicide was an ensemble of predictions from random forest, XGBoost and catBoost. For suicide, the ROC-AUC was 0.71 (95% CI: 0.70-0.73). At a risk threshold of 0.15%, PPV was 0.34% and sensitivity was 56.0%. The most contributing predictors differed when predicting suicide and suicide attempt, indicating that separate models are needed. The ensemble model was fair across sex and age, and more so than the penalized logistic regression model.

CONCLUSIONS

We achieved good performance for predicting suicide attempts and demonstrated a clinical application of ensemble models. Our results indicate a difference in predictive performance for models predicting suicide and suicide attempt, respectively. Thus, we recommend that suicide and suicide attempt are treated as two separate endpoints, in particular for clinical application. We demonstrated that the ensemble model is fairer across sex and age compared with a penalized logistic regression, and therefore we recommend the use of well-tested ensembles despite a more complex explainability.

Managed care updates of subscriber jail release to prompt community suicide prevention: clinical trial protocol

BACKGROUND

Recent jail detention is a marker for trait and state suicide risk in community-based populations. However, healthcare providers are typically unaware that their client was in jail and few post-release suicide prevention efforts exist. This protocol paper describes an effectiveness-implementation trial evaluating community suicide prevention practices triggered by advances in informatics that alert CareSource, a large managed care organization (MCO), when a subscriber is released from jail.

METHODS

This randomized controlled trial investigates two evidence-based suicide prevention practices triggered by CareSource's jail detention/release notifications, in a partial factorial design. The first phase randomizes ~ 43,000 CareSource subscribers who pass through any Ohio jail to receive Caring Contact letters sent by CareSource or to Usual Care after jail release. The second phase (running simultaneously) involves a subset of ~ 6,000 of the 43,000 subscribers passing through jail who have been seen in one of 12 contracted behavioral health agencies in the 6 months prior to incarceration in a stepped-wedge design. Agencies will receive: (a) notifications of the client's jail detention/release, (b) instructions for re-engaging these clients, and (c) training in suicide risk assessment and the Safety Planning Intervention for use at re-engagement. We will track suicide-related and service linkage outcomes 6 months following jail release using claims data.

CONCLUSIONS

This design allows us to rigorously test two intervention main effects and their interaction. It also provides valuable information on the effects of system-level change and the scalability of interventions using big data from a MCO to flag jail release and suicide risk.

TRIAL REGISTRATION

The trial is registered at clinicaltrials.gov (NCT05579600). Registered 27 June, 2023.

Managed Care Updates of Subscriber Jail Release to Prompt Community Suicide Prevention: Clinical Trial Protocol

Background

Recent jail detention is a marker for trait and state suicide risk in community-based populations. However, healthcare providers are typically unaware that their client was in jail and few post-release suicide prevention efforts exist. This protocol paper describes an effectiveness-implementation trial evaluating community suicide prevention practices triggered by advances in informatics that alert CareSource, a large managed care organization (MCO), when a subscriber is released from jail.

Methods

This randomized controlled trial investigates two evidence-based suicide prevention practices triggered by CareSource's jail detention/release notifications, in a partial factorial design. The first phase randomizes ~43,000 CareSource subscribers who pass through any Ohio jail to receive Caring Contact letters sent by CareSource or to Usual Care after jail release. The second phase (running simultaneously) involves a subset of ~6,000 of the 43,000 subscribers passing through jail who have been seen in one of 12 contracted behavioral health agencies in the 6 months prior to incarceration in a stepped-wedge design. Agencies will receive: (a) notifications of the client's jail detention/release, (b) instructions for re-engaging these clients, and (c) training in suicide risk assessment and the Safety Planning Intervention for use at re-engagement. We will track suicide-related and service linkage outcomes 6 months following jail release using claims data.

Conclusions

This design allows us to rigorously test two intervention main effects and their interaction. It also provides valuable information on the effects of system-level change and the scalability of interventions using big data from a MCO to flag jail release and suicide risk.

Trial Registration

The trial is registered at clinicaltrials.gov (NCT05579600). Registered 27 June, 2023, https://beta.clinicaltrials.gov/study/NCT05579600?cond=Suicide&term=Managed%20Care&rank=1.

Health Service Protection vis-à-vis the Detection of Psychosocial Risks of Suicide during the Years 2019-2021

Health services are especially relevant in suicide prevention and intervention, representing a favourable environment in which to implement specific strategies to detect and address suicidal behaviours. Indeed, a significant proportion of people who die by suicide (DBS) present at primary care and mental health services during the last year, month, or even days before committing suicide. The objective of this descriptive and cross-sectional study of all registered cases of death by suicide (N = 265) in Extremadura (Spain) was to determine which of those people who died by suicide had mental health problems (MHP) and what type of assistance they had requested. Diagnoses, previous suicide attempts, type of health service, and last visit before death were explored with univariate analyses and logistic regressions. The proportion of people without MHP was found to be high, and these people had hardly visited the health services at all in their last year. People with MHP, between the ages of 40 and 69, and with previous suicide attempts were more likely to have visited the mental health service in the three months prior to their death. It is, thus, necessary to provide health professionals with tools and training in the prevention of and approach to suicide. Efforts must be directed towards effectively assessing mental health and the risk of suicide since a large proportion of people who die by suicide may go unnoticed.

Low Level of Agreement Between Self-Report and Clinical Assessment of Passive Suicidal Ideation

OBJECTIVE

Discrepancies between patient reports during clinical evaluations and self-reported suicide ideation are of vital importance. We study the agreement in passive suicidal ideation between reports made by clinicians and patients' self-reports.

METHOD

Wish of death in 648 outpatients was assessed by attending clinicians. Within 24 h after clinical evaluation, patients completed a self-report questionnaire in which they were asked whether they had no desire to live. We used cluster analysis to determine the clinical profile of a population of patients according to the concordance between reports made by clinicians and self-reported information.

RESULTS

A low level of agreement (kappa = 0.072) was found between clinicians and patients, as 56.4% (n = 366) of clinician reports classified as containing no death-related ideas although on self-report the patient did state that they had no desire to live. In this group containing discrepancies between the two reports, two clusters were found to have shared characteristics: female sex, middle age, cohabitation, active employment, no history of suicidal behavior, and diagnosis of neurotic, stress-related, and somatoform disorders. In a third, more severe cluster, patients self-reported sleep disturbances, less appetite, poor treatment adherence, and aggressiveness.

CONCLUSIONS

We found low agreement between self-reports and clinician assessments regarding the death wish. Self-reporting may be useful in assessing suicide risk. HIGHLIGHTSLow agreement was found between self-reports and clinician assessments regarding passive suicidal ideation.Most patients in whom the clinician underestimated the risk of suicide were women.Our results suggest that clinicians require adequate documentation of suicidal risk assessment to identify the high-risk population.

Machine Learning Prediction of Suicide Risk Does Not Identify Patients Without Traditional Risk Factors

Objective: To determine whether predictions of suicide risk from machine learning models identify unexpected patients or patients without medical record documentation of traditional risk factors.

Methods: The study sample included 27,091,382 outpatient mental health (MH) specialty or general medical visits with a MH diagnosis for patients aged 11 years or older from January 1, 2009, to September 30, 2017. We used predicted risk scores of suicide attempt and suicide death, separately, within 90 days of visits to classify visits into risk score percentile strata. For each stratum, we calculated counts and percentages of visits with traditional risk factors, including prior self-harm diagnoses and emergency department visits or hospitalizations with MH diagnoses, in the last 3, 12, and 60 months.

Results: Risk-factor percentages increased with predicted risk scores. Among MH specialty visits, 66%, 88%, and 99% of visits with suicide attempt risk scores in the top 3 strata (respectively, 90th-95th, 95th-98th, and ≥ 98th percentiles) and 60%, 77%, and 93% of visits with suicide risk scores in the top 3 strata represented patients who had at least one traditional risk factor documented in the prior 12 months. Among general medical visits, 52%, 66%, and 90% of visits with suicide attempt risk scores in the top 3 strata and 45%, 66%, and 79% of visits with suicide risk scores in the top 3 strata represented patients who had a history of traditional risk factors in the last 12 months.

Conclusions: Suicide risk alerts based on these machine learning models coincide with patients traditionally thought of as high-risk at their high-risk visits.

Health care utilization among individuals who die by suicide as compared to the general population: a population-based register study in Sweden

Background

Globally, 700 000 people die every year by suicide. Health care consultation patterns the period before suicide could be one potential way to identify people at risk for suicide. Therefore, this study examines health care patterns up to one year prior to the suicide by age, sex and prior diagnoses and specifically investigates if and how this differs from the general population of Skåne, Sweden.

Methods

This cohort study includes all individuals, aged 15 and older, that died by suicide in Region Skåne, Sweden from 2004 to 2015 (n = 1653). The individuals were identified through the Cause of death register and then linked to the Skåne healthcare register. Health care data was analyzed as proportions consulting different types of health care the month and year preceding the suicide, we also investigated the impact of age, sex and the occurrence of prior psychiatric and pain diagnoses. Additionally, we compared the proportion of consulting care among the suicide victims and the general population of Skåne.

Results

In the month before their death, 53% of the suicide victims had any health care consultation, compared with 20% in the general population of Skåne, a given month (p < 0.0001). The corresponding figures for the year prior to suicide was 86% among those who died by suicide, compared to 69% in the general population of Skåne, a given year (p < 0.0001). Women, and those having a documented history of psychiatric diagnosis were more likely to have health care consultations in the month and year preceding suicide (p < 0.001), compared to men and suicide victims without a history of psychiatric disease. Older adults that died by suicide, were less likely to consult psychiatric care compared to the younger suicide victims (p < 0.001).

Conclusion

A majority of the suicide victims consulted health care in the near time before death and the proportion of seeking health care was significantly higher than in the general population of Skåne and higher among female suicide victims as compared to males. Alternative preventive screening measures should be considered for individuals consulting health care, especially for older people and individuals outside the psychiatric care.

Structured data vs. unstructured data in machine learning prediction models for suicidal behaviors: A systematic review and meta-analysis

Suicide remains a leading cause of preventable death worldwide, despite advances in research and decreases in mental health stigma through government health campaigns. Machine learning (ML), a type of artificial intelligence (AI), is the use of algorithms to simulate and imitate human cognition. Given the lack of improvement in clinician-based suicide prediction over time, advancements in technology have allowed for novel approaches to predicting suicide risk. This systematic review and meta-analysis aimed to synthesize current research regarding data sources in ML prediction of suicide risk, incorporating and comparing outcomes between structured data (human interpretable such as psychometric instruments) and unstructured data (only machine interpretable such as electronic health records). Online databases and gray literature were searched for studies relating to ML and suicide risk prediction. There were 31 eligible studies. The outcome for all studies combined was AUC = 0.860, structured data showed AUC = 0.873, and unstructured data was calculated at AUC = 0.866. There was substantial heterogeneity between the studies, the sources of which were unable to be defined. The studies showed good accuracy levels in the prediction of suicide risk behavior overall. Structured data and unstructured data also showed similar outcome accuracy according to meta-analysis, despite different volumes and types of input data.

Validating a predictive algorithm for suicide risk with Alaska Native populations

INTRODUCTION

The American Indian/Alaska Native (AI/AN) suicide rate in Alaska is twice the state rate and four times the U.S. rate. Healthcare systems need innovative methods of suicide risk detection. The Mental Health Research Network (MHRN) developed suicide risk prediction algorithms in a general U.S.

PATIENT POPULATION

METHODS

We applied MHRN predictors and regression coefficients to electronic health records of AI/AN patients aged ≥13 years with behavioral health diagnoses and primary care visits between October 1, 2016, and March 30, 2018. Logistic regression assessed model accuracy for predicting and stratifying risk for suicide attempt within 90 days after a visit. We compared expected to observed risk and assessed model performance characteristics.

RESULTS

10,864 patients made 47,413 primary care visits. Suicide attempt occurred after 589 (1.2%) visits. Visits in the top 5% of predicted risk accounted for 40% of actual attempts. Among visits in the top 0.5% of predicted risk, 25.1% were followed by suicide attempt. The best fitting model had an AUC of 0.826 (95% CI: 0.809-0.843).

CONCLUSIONS

The MHRN model accurately predicted suicide attempts among AI/AN patients. Future work should develop clinical and operational guidance for effective implementation of the model with this population.

Translating promise into practice: a review of machine learning in suicide research and prevention

In ever more pressured health-care systems, technological solutions offering scalability of care and better resource targeting are appealing. Research on machine learning as a technique for identifying individuals at risk of suicidal ideation, suicide attempts, and death has grown rapidly. This research often places great emphasis on the promise of machine learning for preventing suicide, but overlooks the practical, clinical implementation issues that might preclude delivering on such a promise. In this Review, we synthesise the broad empirical and review literature on electronic health record-based machine learning in suicide research, and focus on matters of crucial importance for implementation of machine learning in clinical practice. The challenge of preventing statistically rare outcomes is well known; progress requires tackling data quality, transparency, and ethical issues. In the future, machine learning models might be explored as methods to enable targeting of interventions to specific individuals depending upon their level of need-ie, for precision medicine. Primarily, however, the promise of machine learning for suicide prevention is limited by the scarcity of high-quality scalable interventions available to individuals identified by machine learning as being at risk of suicide.

Artificial intelligence and suicide prevention: a systematic review

Background

Suicide is one of the main preventable causes of death. Artificial intelligence (AI) could improve methods for assessing suicide risk. The objective of this review is to assess the potential of AI in identifying patients who are at risk of attempting suicide.

Methods

A systematic review of the literature was conducted on PubMed, EMBASE, and SCOPUS databases, using relevant keywords.

Results

Thanks to this research, 296 studies were identified. Seventeen studies, published between 2014 and 2020 and matching inclusion criteria, were selected as relevant. Included studies aimed at predicting individual suicide risk or identifying at-risk individuals in a specific population. The AI performance was overall good, although variable across different algorithms and application settings.

Conclusions

AI appears to have a high potential for identifying patients at risk of suicide. The precise use of these algorithms in clinical situations, as well as the ethical issues it raises, remain to be clarified.

Clinical risk prediction models and informative cluster size: Assessing the performance of a suicide risk prediction algorithm

Clinical visit data are clustered within people, which complicates prediction modeling. Cluster size is often informative because people receiving more care are less healthy and at higher risk of poor outcomes. We used data from seven health systems on 1,518,968 outpatient mental health visits from January 1, 2012 to June 30, 2015 to predict suicide attempt within 90 days. We evaluated true performance of prediction models using a prospective validation set of 4,286,495 visits from October 1, 2015 to September 30, 2017. We examined dividing clustered data on the person or visit level for model training and cross-validation and considered a within cluster resampling approach for model estimation. We evaluated optimism by comparing estimated performance from a left-out testing dataset to performance in the prospective dataset. We used two prediction methods, logistic regression with least absolute shrinkage and selection operator (LASSO) and random forest. The random forest model using a visit-level split for model training and testing was optimistic; it overestimated discrimination (area under the curve, AUC = 0.95 in testing versus 0.84 in prospective validation) and classification accuracy (sensitivity = 0.48 in testing versus 0.19 in prospective validation, 95th percentile cut-off). Logistic regression and random forest models using a person-level split performed well, accurately estimating prospective discrimination and classification: estimated AUCs ranged from 0.85 to 0.87 in testing versus 0.85 in prospective validation, and sensitivity ranged from 0.15 to 0.20 in testing versus 0.17 to 0.19 in prospective validation. Within cluster resampling did not improve performance. We recommend dividing clustered data on the person level, rather than visit level, to ensure strong performance in prospective use and accurate estimation of future performance at the time of model development.

Evaluation of Electronic Health Record-Based Suicide Risk Prediction Models on Contemporary Data

BACKGROUND

Suicide risk prediction models have been developed by using information from patients' electronic health records (EHR), but the time elapsed between model development and health system implementation is often substantial. Temporal changes in health systems and EHR coding practices necessitate the evaluation of such models in more contemporary data.

OBJECTIVES

A set of published suicide risk prediction models developed by using EHR data from 2009 to 2015 across seven health systems reported c-statistics of 0.85 for suicide attempt and 0.83 to 0.86 for suicide death. Our objective was to evaluate these models' performance with contemporary data (2014-2017) from these systems.

METHODS

We evaluated performance using mental health visits (6,832,439 to mental health specialty providers and 3,987,078 to general medical providers) from 2014 to 2017 made by 1,799,765 patients aged 13+ across the health systems. No visits in our evaluation were used in the previous model development. Outcomes were suicide attempt (health system records) and suicide death (state death certificates) within 90 days following a visit. We assessed calibration and computed c-statistics with 95% confidence intervals (CI) and cut-point specific estimates of sensitivity, specificity, and positive/negative predictive value.

RESULTS

Models were well calibrated; 46% of suicide attempts and 35% of suicide deaths in the mental health specialty sample were preceded by a visit (within 90 days) with a risk score in the top 5%. In the general medical sample, 53% of attempts and 35% of deaths were preceded by such a visit. Among these two samples, respectively, c-statistics were 0.862 (95% CI: 0.860-0.864) and 0.864 (95% CI: 0.860-0.869) for suicide attempt, and 0.806 (95% CI: 0.790-0.822) and 0.804 (95% CI: 0.782-0.829) for suicide death.

CONCLUSION

Performance of the risk prediction models in this contemporary sample was similar to historical estimates for suicide attempt but modestly lower for suicide death. These published models can inform clinical practice and patient care today.

Reconciling Statistical and Clinicians' Predictions of Suicide Risk

Statistical models, including those based on electronic health records, can accurately identify patients at high risk for a suicide attempt or death, leading to implementation of risk prediction models for population-based suicide prevention in health systems. However, some have questioned whether statistical predictions can really inform clinical decisions. Appropriately reconciling statistical algorithms with traditional clinician assessment depends on whether predictions from these two methods are competing, complementary, or merely duplicative. In June 2019, the National Institute of Mental Health convened a meeting, "Identifying Research Priorities for Risk Algorithms Applications in Healthcare Settings to Improve Suicide Prevention." Here, participants of this meeting summarize key issues regarding the potential clinical application of suicide prediction models. The authors attempt to clarify the key conceptual and technical differences between traditional risk prediction by clinicians and predictions from statistical models, review the limited evidence regarding both the accuracy of and the concordance between these alternative methods of prediction, present a conceptual framework for understanding agreement and disagreement between statistical and clinician predictions, identify priorities for improving data regarding suicide risk, and propose priority questions for future research. Future suicide risk assessment will likely combine statistical prediction with traditional clinician assessment, but research is needed to determine the optimal combination of these two methods.

The bird's-eye view: A data-driven approach to understanding patient journeys from claims data

OBJECTIVE

In preference-sensitive conditions such as back pain, there can be high levels of variability in the trajectory of patient care. We sought to develop a methodology that extracts a realistic and comprehensive understanding of the patient journey using medical and pharmaceutical insurance claims data.

MATERIALS AND METHODS

We processed a sample of 10 000 patient episodes (comprised of 113 215 back pain-related claims) into strings of characters, where each letter corresponds to a distinct encounter with the healthcare system. We customized the Levenshtein edit distance algorithm to evaluate the level of similarity between each pair of episodes based on both their content (types of events) and ordering (sequence of events). We then used clustering to extract the main variations of the patient journey.

RESULTS

The algorithm resulted in 12 comprehensive and clinically distinct patterns (clusters) of patient journeys that represent the main ways patients are diagnosed and treated for back pain. We further characterized demographic and utilization metrics for each cluster and observed clear differentiation between the clusters in terms of both clinical content and patient characteristics.

DISCUSSION

Despite being a complex and often noisy data source, administrative claims provide a unique longitudinal overview of patient care across multiple service providers and locations. This methodology leverages claims to capture a data-driven understanding of how patients traverse the healthcare system.

CONCLUSIONS

When tailored to various conditions and patient settings, this methodology can provide accurate overviews of patient journeys and facilitate a shift toward high-quality practice patterns.

Systematic review of approaches to use of neighborhood-level risk factors with clinical data to predict clinical risk and recommend interventions

BACKGROUND

Despite a large body of literature investigating how the environment influences health outcomes, most published work to date includes only a limited subset of the rich clinical and environmental data that is available and does not address how these data might best be used to predict clinical risk or expected impact of clinical interventions.

OBJECTIVE

Identify existing approaches to inclusion of a broad set of neighborhood-level risk factors with clinical data to predict clinical risk and recommend interventions.

METHODS

A systematic review of scientific literature published and indexed in PubMed, Web of Science, Association of Computing Machinery (ACM) and SCOPUS from 2010 through October 2020 was performed. To be included, articles had to include search terms related to Electronic Health Record (EHR) data Neighborhood-Level Risk Factors (NLRFs), and Machine Learning (ML) Methods. Citations of relevant articles were also reviewed for additional articles for inclusion. Articles were reviewed and coded by two independent reviewers to capture key information including data sources, linkage of EHR to NRLFs, methods, and results. Articles were assessed for quality using a modified Quality Assessment Tool for Systematic Reviews of Observational Studies (QATSO).

RESULTS

A total of 334 articles were identified for abstract review. 36 articles were identified for full review with 19 articles included in the final analysis. All but two of the articles included socio-demographic data derived from the U.S. Census and we found great variability in sources of NLRFs beyond the Census. The majority or the articles (14 of 19) included broader clinical (e.g. medications, labs and co-morbidities) and demographic information about the individual from the EHR in addition to the clinical outcome variable. Half of the articles (10) had a stated goal to predict the outcome(s) of interest. While results of the studies reinforced the correlative association of NLRFs to clinical outcomes, only one article found that adding NLRFs into a model with other data added predictive power with the remainder concluding either that NLRFs were of mixed value depending on the model and outcome or that NLRFs added no predictive power over other data in the model. Only one article scored high on the quality assessment with 13 scoring moderate and 4 scoring low.

CONCLUSIONS

In spite of growing interest in combining NLRFs with EHR data for clinical prediction, we found limited evidence that NLRFs improve predictive power in clinical risk models. We found these data and methods are being used in four ways. First, early approaches to include broad NLRFs to predict clinical risk primarily focused on dimension reduction for feature selection or as a data preparation step to input into regression analysis. Second, more recent work incorporates NLRFs into more advanced predictive models, such as Neural Networks, Random Forest, and Penalized Lasso to predict clinical outcomes or predict value of interventions. Third, studies that test how inclusion of NLRFs predict clinical risk have shown mixed results regarding the value of these data over EHR or claims data alone and this review surfaced evidence of potential quality challenges and biases inherent to this approach. Finally, NLRFs were used with unsupervised learning to identify underlying patterns in patient populations to recommend targeted interventions. Further access to computable, high quality data is needed along with careful study design, including sub-group analysis, to better determine how these data and methods can be used to support decision making in a clinical setting.

Using weak supervision and deep learning to classify clinical notes for identification of current suicidal ideation

Mental health concerns, such as suicidal thoughts, are frequently documented by providers in clinical notes, as opposed to structured coded data. In this study, we evaluated weakly supervised methods for detecting "current" suicidal ideation from unstructured clinical notes in electronic health record (EHR) systems. Weakly supervised machine learning methods leverage imperfect labels for training, alleviating the burden of creating a large manually annotated dataset. After identifying a cohort of 600 patients at risk for suicidal ideation, we used a rule-based natural language processing approach (NLP) approach to label the training and validation notes (n = 17,978). Using this large corpus of clinical notes, we trained several statistical machine learning models-logistic classifier, support vector machines (SVM), Naive Bayes classifier-and one deep learning model, namely a text classification convolutional neural network (CNN), to be evaluated on a manually-reviewed test set (n = 837). The CNN model outperformed all other methods, achieving an overall accuracy of 94% and a F1-score of 0.82 on documents with "current" suicidal ideation. This algorithm correctly identified an additional 42 encounters and 9 patients indicative of suicidal ideation but missing a structured diagnosis code. When applied to a random subset of 5,000 clinical notes, the algorithm classified 0.46% (n = 23) for "current" suicidal ideation, of which 87% were truly indicative via manual review. Implementation of this approach for large-scale document screening may play an important role in point-of-care clinical information systems for targeted suicide prevention interventions and improve research on the pathways from ideation to attempt.