Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans health Administration

Predicting mechanical restraint of psychiatric inpatients by applying machine learning on electronic health data

OBJECTIVE

Mechanical restraint (MR) is used to prevent patients from harming themselves or others during inpatient treatment. The objective of this study was to investigate whether incident MR occurring in the first 3 days following admission could be predicted based on analysis of electronic health data available after the first hour of admission.

METHODS

The dataset consisted of clinical notes from electronic health records from the Central Denmark Region and data from the Danish Health Registers from patients admitted to a psychiatric department in the period from 2011 to 2015. Supervised machine learning algorithms were trained on a randomly selected subset of the data and validated using an independent test dataset.

RESULTS

A total of 5050 patients with 8869 admissions were included in the study. One hundred patients were mechanically restrained in the period between one hour and 3 days after the admission. A Random Forest algorithm predicted MR with an area under the curve of 0.87 (95% CI 0.79-0.93). At 94% specificity, the sensitivity was 56%. Among the ten strongest predictors, nine were derived from the clinical notes.

CONCLUSIONS

These findings open for the development of an early warning system that may guide interventions to reduce the use of MR.

Machine learning in mental health: a scoping review of methods and applications

BACKGROUND

This paper aims to synthesise the literature on machine learning (ML) and big data applications for mental health, highlighting current research and applications in practice.

METHODS

We employed a scoping review methodology to rapidly map the field of ML in mental health. Eight health and information technology research databases were searched for papers covering this domain. Articles were assessed by two reviewers, and data were extracted on the article's mental health application, ML technique, data type, and study results. Articles were then synthesised via narrative review.

RESULTS

Three hundred papers focusing on the application of ML to mental health were identified. Four main application domains emerged in the literature, including: (i) detection and diagnosis; (ii) prognosis, treatment and support; (iii) public health, and; (iv) research and clinical administration. The most common mental health conditions addressed included depression, schizophrenia, and Alzheimer's disease. ML techniques used included support vector machines, decision trees, neural networks, latent Dirichlet allocation, and clustering.

CONCLUSIONS

Overall, the application of ML to mental health has demonstrated a range of benefits across the areas of diagnosis, treatment and support, research, and clinical administration. With the majority of studies identified focusing on the detection and diagnosis of mental health conditions, it is evident that there is significant room for the application of ML to other areas of psychology and mental health. The challenges of using ML techniques are discussed, as well as opportunities to improve and advance the field.

Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation

IMPORTANCE

Suicide prediction models have the potential to improve the identification of patients at heightened suicide risk by using predictive algorithms on large-scale data sources. Suicide prediction models are being developed for use across enterprise-level health care systems including the US Department of Defense, US Department of Veterans Affairs, and Kaiser Permanente.

OBJECTIVES

To evaluate the diagnostic accuracy of suicide prediction models in predicting suicide and suicide attempts and to simulate the effects of implementing suicide prediction models using population-level estimates of suicide rates.

EVIDENCE REVIEW

A systematic literature search was conducted in MEDLINE, PsycINFO, Embase, and the Cochrane Library to identify research evaluating the predictive accuracy of suicide prediction models in identifying patients at high risk for a suicide attempt or death by suicide. Each database was searched from inception to August 21, 2018. The search strategy included search terms for suicidal behavior, risk prediction, and predictive modeling. Reference lists of included studies were also screened. Two reviewers independently screened and evaluated eligible studies.

FINDINGS

From a total of 7306 abstracts reviewed, 17 cohort studies met the inclusion criteria, representing 64 unique prediction models across 5 countries with more than 14 million participants. The research quality of the included studies was generally high. Global classification accuracy was good (≥0.80 in most models), while the predictive validity associated with a positive result for suicide mortality was extremely low (≤0.01 in most models). Simulations of the results suggest very low positive predictive values across a variety of population assessment characteristics.

CONCLUSIONS AND RELEVANCE

To date, suicide prediction models produce accurate overall classification models, but their accuracy of predicting a future event is near 0. Several critical concerns remain unaddressed, precluding their readiness for clinical applications across health systems.

A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models

OBJECTIVES

The objective of this study was to compare performance of logistic regression (LR) with machine learning (ML) for clinical prediction modeling in the literature.

STUDY DESIGN AND SETTING

We conducted a Medline literature search (1/2016 to 8/2017) and extracted comparisons between LR and ML models for binary outcomes.

RESULTS

We included 71 of 927 studies. The median sample size was 1,250 (range 72-3,994,872), with 19 predictors considered (range 5-563) and eight events per predictor (range 0.3-6,697). The most common ML methods were classification trees, random forests, artificial neural networks, and support vector machines. In 48 (68%) studies, we observed potential bias in the validation procedures. Sixty-four (90%) studies used the area under the receiver operating characteristic curve (AUC) to assess discrimination. Calibration was not addressed in 56 (79%) studies. We identified 282 comparisons between an LR and ML model (AUC range, 0.52-0.99). For 145 comparisons at low risk of bias, the difference in logit(AUC) between LR and ML was 0.00 (95% confidence interval, -0.18 to 0.18). For 137 comparisons at high risk of bias, logit(AUC) was 0.34 (0.20-0.47) higher for ML.

CONCLUSION

We found no evidence of superior performance of ML over LR. Improvements in methodology and reporting are needed for studies that compare modeling algorithms.

Challenges and Opportunities for Using Big Health Care Data to Advance Medical Science and Public Health

Methodological advancements in epidemiology, biostatistics, and data science have strengthened the research world's ability to use data captured from electronic health records (EHRs) to address pressing medical questions, but gaps remain. We describe methods investments that are needed to curate EHR data toward research quality and to integrate complementary data sources when EHR data alone are insufficient for research goals. We highlight new methods and directions for improving the integrity of medical evidence generated from pragmatic trials, observational studies, and predictive modeling. We also discuss needed methods contributions to further ease data sharing across multisite EHR data networks. Throughout, we identify opportunities for training and for bolstering collaboration among subject matter experts, methodologists, practicing clinicians, and health system leaders to help ensure that methods problems are identified and resulting advances are translated into mainstream research practice more quickly.

Suicide in Transgender Veterans: Prevalence, Prevention, and Implications of Current Policy

Transgender adults serve in the U.S. military at 2 to 3 times the rate of the general adult population. Unfortunately, transgender veterans die by suicide at twice the rate of their cisgender veteran peers and approximately 5.85 times the rate of the general population. This article reviews the literature regarding the prevalence of suicidal thoughts and behaviors in transgender veterans. Suicide risk and resilience factors are reviewed, and future areas of study are detailed that incorporate findings from the broader suicide-prevention literature and research on transgender mental-health disparities. Individual services and broader prevention considerations are discussed, including the adaptation of evidence-based suicide-specific psychological interventions, national transgender health-training resources, and relevant veteran suicide-prevention initiatives. Finally, U.S. Department of Defense and U.S. Department of Veterans Affairs policies regarding transgender service and health care are reviewed. State-level policies relevant to transgender veteran suicide such as firearm ownership and nondiscrimination laws are also reviewed, and their implications for suicide prevention are discussed. The aim of this article is to provide a broad review of research findings from multiple fields of study to assist health-care providers, researchers, and policymakers in their efforts to prevent transgender veteran suicide.

The use of machine learning in the study of suicidal and non-suicidal self-injurious thoughts and behaviors: A systematic review

BACKGROUND

Machine learning techniques offer promise to improve suicide risk prediction. In the current systematic review, we aimed to review the existing literature on the application of machine learning techniques to predict self-injurious thoughts and behaviors (SITBs).

METHOD

We systematically searched PsycINFO, PsycARTICLES, ERIC, CINAHL, and MEDLINE for articles published through February 2018.

RESULTS

Thirty-five articles met criteria to be included in the review. Included articles were reviewed by outcome: suicide death, suicide attempt, suicide plan, suicidal ideation, suicide risk, and non-suicidal self-injury. We observed three general aims in the use of SITB-focused machine learning analyses: (1) improving prediction accuracy, (2) identifying important model indicators (i.e., variable selection) and indicator interactions, and (3) modeling underlying subgroups. For studies with the aim of boosting predictive accuracy, we observed greater prediction accuracy of SITBs than in previous studies using traditional statistical methods. Studies using machine learning for variable selection purposes have both replicated findings of well-known SITB risk factors and identified novel variables that may augment model performance. Finally, some of these studies have allowed for subgroup identification, which in turn has helped to inform clinical cutoffs.

LIMITATIONS

Limitations of the current review include relatively low paper sample size, inconsistent reporting procedures resulting in an inability to compare model accuracy across studies, and lack of model validation on external samples.

CONCLUSIONS

We concluded that leveraging machine learning techniques to further predictive accuracy and identify novel indicators will aid in the prediction and prevention of suicide.

Digital Interventions for Mental Disorders: Key Features, Efficacy, and Potential for Artificial Intelligence Applications

Mental disorders are highly prevalent and often remain untreated. Many limitations of conventional face-to-face psychological interventions could potentially be overcome through Internet-based and mobile-based interventions (IMIs). This chapter introduces core features of IMIs, describes areas of application, presents evidence on the efficacy of IMIs as well as potential effect mechanisms, and delineates how Artificial Intelligence combined with IMIs may improve current practices in the prevention and treatment of mental disorders in adults. Meta-analyses of randomized controlled trials clearly show that therapist-guided IMIs can be highly effective for a broad range of mental health problems. Whether the effects of unguided IMIs are also clinically relevant, particularly under routine care conditions, is less clear. First studies on IMIs for the prevention of mental disorders have shown promising results. Despite limitations and challenges, IMIs are increasingly implemented into routine care worldwide. IMIs are also well suited for applications of Artificial Intelligence and Machine Learning, which provides ample opportunities to improve the identification and treatment of mental disorders. Together with methodological innovations, these approaches may also deepen our understanding of how psychological interventions work, and why. Ethical and professional restraints as well as potential contraindications of IMIs, however, should also be considered. In sum, IMIs have a high potential for improving the prevention and treatment of mental health disorders across various indications, settings, and populations. Therefore, implementing IMIs into routine care as both adjunct and alternative to face-to-face treatment is highly desirable. Technological advancements may further enhance the variability and flexibility of IMIs, and thus even further increase their impact in people's lives in the future.

The Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS): progress toward understanding suicide among soldiers

Responding to an unprecedented increase in the suicide rate among soldiers, in 2008 the US Army and US National Institute of Mental Health funded the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS), a multicomponent epidemiological and neurobiological study of risk and resilience factors for suicidal thoughts and behaviors, and their psychopathological correlates among Army personnel. Using a combination of administrative records, representative surveys, computerized neurocognitive tests, and blood samples, Army STARRS and its longitudinal follow-up study (STARRS-LS) are designed to identify potentially actionable findings to inform the Army's suicide prevention efforts. The current report presents a broad overview of Army STARRS and its findings to date on suicide deaths, attempts, and ideation, as well as other important outcomes that may increase suicide risk (e.g., mental disorders, sexual assault victimization). The findings highlight the complexity of environmental and genetic risk and protective factors in different settings and contexts, and the importance of life and career history in understanding suicidal thoughts and behaviors.

Protecting Life While Preserving Liberty: Ethical Recommendations for Suicide Prevention With Artificial Intelligence

In the United States, suicide increased by 24% in the past 20 years, and suicide risk identification at point-of-care remains a cornerstone of the effort to curb this epidemic (1). As risk identification is difficult because of symptom under-reporting, timing, or lack of screening, healthcare systems rely increasingly on risk scoring and now artificial intelligence (AI) to assess risk. AI remains the science of solving problems and accomplishing tasks, through automated or computational means, that normally require human intelligence. This science is decades-old and includes traditional predictive statistics and machine learning. Only in the last few years has it been applied rigorously in suicide risk prediction and prevention. Applying AI in this context raises significant ethical concern, particularly in balancing beneficence and respecting personal autonomy. To navigate the ethical issues raised by suicide risk prediction, we provide recommendations in three areas-communication, consent, and controls-for both providers and researchers (2).

Looking to the Future: A Synthesis of New Developments and Challenges in Suicide Research and Prevention

Suicide and attempted suicide are major public health concerns. In recent decades, there have been many welcome developments in understanding and preventing suicide, as well as good progress in intervening with those who have attempted suicide. Despite these developments, though, considerable challenges remain. In this article, we explore both the recent developments and the challenges ahead for the field of suicide research and prevention. To do so, we consulted 32 experts from 12 countries spanning four continents who had contributed to the International Handbook of Suicide Prevention (2nd edition). All contributors nominated, in their view, (i) the top 3 most exciting new developments in suicide research and prevention in recent years, and (ii) the top 3 challenges. We have synthesized their suggestions into new developments and challenges in research and practice, giving due attention to implications for psychosocial interventions. This Perspective article is not a review of the literature, although we did draw from the suicide research literature to obtain evidence to elucidate the responses from the contributors. Key new developments and challenges include: employing novel techniques to improve the prediction of suicidal behavior; testing and applying theoretical models of suicidal behavior; harnessing new technologies to monitor and intervene in suicide risk; expanding suicide prevention activities to low and middle-income countries; moving toward a more refined understanding of sub-groups of people at risk and developing tailored interventions. We also discuss the importance of multidisciplinary working and the challenges of implementing interventions in practice.

Predicting Suicide Attempts and Suicide Deaths Following Outpatient Visits Using Electronic Health Records

OBJECTIVE

The authors sought to develop and validate models using electronic health records to predict suicide attempt and suicide death following an outpatient visit.

METHOD

Across seven health systems, 2,960,929 patients age 13 or older (mean age, 46 years; 62% female) made 10,275,853 specialty mental health visits and 9,685,206 primary care visits with mental health diagnoses between Jan. 1, 2009, and June 30, 2015. Health system records and state death certificate data identified suicide attempts (N=24,133) and suicide deaths (N=1,240) over 90 days following each visit. Potential predictors included 313 demographic and clinical characteristics extracted from records for up to 5 years before each visit: prior suicide attempts, mental health and substance use diagnoses, medical diagnoses, psychiatric medications dispensed, inpatient or emergency department care, and routinely administered depression questionnaires. Logistic regression models predicting suicide attempt and death were developed using penalized LASSO (least absolute shrinkage and selection operator) variable selection in a random sample of 65% of the visits and validated in the remaining 35%.

RESULTS

Mental health specialty visits with risk scores in the top 5% accounted for 43% of subsequent suicide attempts and 48% of suicide deaths. Of patients scoring in the top 5%, 5.4% attempted suicide and 0.26% died by suicide within 90 days. C-statistics (equivalent to area under the curve) for prediction of suicide attempt and suicide death were 0.851 (95% CI=0.848, 0.853) and 0.861 (95% CI=0.848, 0.875), respectively. Primary care visits with scores in the top 5% accounted for 48% of subsequent suicide attempts and 43% of suicide deaths. C-statistics for prediction of suicide attempt and suicide death were 0.853 (95% CI=0.849, 0.857) and 0.833 (95% CI=0.813, 0.853), respectively.

CONCLUSIONS

Prediction models incorporating both health record data and responses to self-report questionnaires substantially outperform existing suicide risk prediction tools.

Predictive modeling of housing instability and homelessness in the Veterans Health Administration

OBJECTIVE

To develop and test predictive models of housing instability and homelessness based on responses to a brief screening instrument administered throughout the Veterans Health Administration (VHA).

DATA SOURCES/STUDY SETTING

Electronic medical record data from 5.8 million Veterans who responded to the VHA's Homelessness Screening Clinical Reminder (HSCR) between October 2012 and September 2015.

STUDY DESIGN

We randomly selected 80% of Veterans in our sample to develop predictive models. We evaluated the performance of both logistic regression and random forests-a machine learning algorithm-using the remaining 20% of cases.

DATA COLLECTION/EXTRACTION METHODS

Data were extracted from two sources: VHA's Corporate Data Warehouse and National Homeless Registry.

PRINCIPAL FINDINGS

Performance for all models was acceptable or better. Random forests models were more sensitive in predicting housing instability and homelessness than logistic regression, but less specific in predicting housing instability. Rates of positive screens for both outcomes were highest among Veterans in the top strata of model-predicted risk.

CONCLUSIONS

Predictive models based on medical record data can identify Veterans likely to report housing instability and homelessness, making the HSCR screening process more efficient and informing new engagement strategies. Our findings have implications for similar instruments in other health care systems.

Latent class cluster analysis of symptom ratings identifies distinct subgroups within the clinical high risk for psychosis syndrome

The clinical-high-risk for psychosis (CHR-P) syndrome is heterogeneous in terms of clinical presentation and outcomes. Identifying more homogenous subtypes of the syndrome may help clarify its etiology and improve the prediction of psychotic illness. This study applied latent class cluster analysis (LCCA) to symptom ratings from the North American Prodrome Longitudinal Studies 1 and 2 (NAPLS 1 and 2). These analyses produced evidence for three to five subgroups within the CHR-P syndrome. Differences in negative and disorganized symptoms distinguished among the subgroups. Subgroup membership was found to predict conversion to psychosis. The authors contrast the methods employed within this study with previous attempts to identify more homogenous subgroups of CHR-P individuals and discuss how these results could be tested in future samples of CHR-P individuals.

Smartphones, Sensors, and Machine Learning to Advance Real-Time Prediction and Interventions for Suicide Prevention: a Review of Current Progress and Next Steps

PURPOSE OF REVIEW

As rates of suicide continue to rise, there is urgent need for innovative approaches to better understand, predict, and care for those at high risk of suicide. Numerous mobile and sensor technology solutions have already been proposed, are in development, or are already available today. This review seeks to assess their clinical evidence and help the reader understand the current state of the field.

RECENT FINDINGS

Advances in smartphone sensing, machine learning methods, and mobile apps directed towards reducing suicide offer promising evidence; however, most of these innovative approaches are still nascent. Further replication and validation of preliminary results is needed. Whereas numerous promising mobile and sensor technology based solutions for real time understanding, predicting, and caring for those at highest risk of suicide are being studied today, their clinical utility remains largely unproven. However, given both the rapid pace and vast scale of current research efforts, we expect clinicians will soon see useful and impactful digital tools for this space within the next 2 to 5 years.

Applying KT Network Complexity to a Highly-Partnered Knowledge Transfer Effort Comment on "Using Complexity and Network Concepts to Inform Healthcare Knowledge Translation"

The re-conceptualization of knowledge translation (KT) in Kitson and colleagues’ manuscript "Using Complexity and Network Concepts to Inform Healthcare Knowledge Translation" is an advancement in how one can incorporate implementation into the KT process. Kitson notes that "the challenge is to explain how it might help in the healthcare policy, practice, and research communities." We propose that these concepts are well presented when considering highly-partnered research that includes all sectors. In this manuscript we provide an example of highly-partnered KT effort framed within the KT Complexity Network Theory. This effort is described by identifying the activities and sectors involved.

Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans health Administration

OBJECTIVES

The US Veterans Health Administration (VHA) has begun using predictive modeling to identify Veterans at high suicide risk to target care. Initial analyses are reported here.

METHODS

A penalized logistic regression model was compared with an earlier proof-of-concept logistic model. Exploratory analyses then considered commonly-used machine learning algorithms. Analyses were based on electronic medical records for all 6,360 individuals classified in the National Death Index as having died by suicide in fiscal years 2009-2011 who used VHA services the year of their death or prior year and a 1% probability sample of time-matched VHA service users alive at the index date (n = 2,112,008).

RESULTS

A penalized logistic model with 61 predictors had sensitivity comparable to the proof-of-concept model (which had 381 predictors) at target thresholds. The machine learning algorithms had relatively similar sensitivities, the highest being for Bayesian additive regression trees, with 10.7% of suicides occurred among the 1.0% of Veterans with highest predicted risk and 28.1% among the 5.0% of with highest predicted risk.

CONCLUSIONS

Based on these results, VHA is using penalized logistic regression in initial intervention implementation. The paper concludes with a discussion of other practical issues that might be explored to increase model performance.