1
|
Kitchen C, Zirikly A, Belouali A, Kharrazi H, Nestadt P, Wilcox HC. Suicide Death Prediction Using the Maryland Suicide Data Warehouse: A Sensitivity Analysis. Arch Suicide Res 2024:1-15. [PMID: 38945167 DOI: 10.1080/13811118.2024.2363227] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 07/02/2024]
Abstract
OBJECTIVE Nearly 50,000 Americans die each year from suicide, despite suicide death being a rare event in the context of health risk assessment and modeling. Prior research has underscored the need for contextualizing suicide risk models in terms of their potential uses and generalizability. This sensitivity analysis makes use of the Maryland Suicide Data Warehouse (MSDW) and illustrates how results inform clinical decision support. METHOD A cohort of 1 million living control patients were extracted from the MSDW in addition to 1,667 patients who had died by suicide between the years 2016 and 2019 according to the Maryland Office of the Medical Examiner (OCME). Data were extracted and aggregated as part of a 4-year retrospective design. Binary logistic and two penalized regression models were deployed in a repeated fivefold cross-validation. Model performances were evaluated using sensitivity, positive predictive value (PPV), and F1, and model coefficients were ranked according to coefficient size. RESULTS Several features were significantly associated with patients having died by suicide, including male sex, depressive and anxiety disorder diagnoses, social needs, and prior suicidal ideation and suicide attempt. Cross-validated binary logistic regression outperformed either ridge or LASSO (least absolute shrinkage and selection operator) models but generally achieved low-to-moderate PPV and sensitivity across most thresholds and a peak F1 of 0.323. CONCLUSIONS Suicide death prediction is constrained by the context of use, which determines the best balance of precision and recall. Predictive models must be evaluated close to the level of intervention. They may not hold up to different needs at different levels of care.
Collapse
|
2
|
Benson NM, Yakubu A, Ren B, Aboud C, Vargas V, Greenfield SF, Busch AB. High-density lipoprotein (HDL) as an indicator for alcohol use in a psychiatrically ill population. Alcohol Alcohol 2024; 59:agae028. [PMID: 38678370 PMCID: PMC11055959 DOI: 10.1093/alcalc/agae028] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2023] [Revised: 03/12/2024] [Accepted: 03/25/2024] [Indexed: 04/29/2024] Open
Abstract
AIMS To examine the cross sectional and longitudinal associations between the Alcohol Use Disorders Identification Test-Concise (AUDIT-C) and differences in high-density lipoprotein (HDL) in a psychiatrically ill population. METHODS Retrospective observational study using electronic health record data from a large healthcare system, of patients hospitalized for a mental health/substance use disorder (MH/SUD) from 1 July 2016 to 31 May 2023, who had a proximal AUDIT-C and HDL (N = 15 915) and the subset who had a repeat AUDIT-C and HDL 1 year later (N = 2915). Linear regression models examined the association between cross-sectional and longitudinal AUDIT-C scores and HDL, adjusting for demographic and clinical characteristics that affect HDL. RESULTS Compared with AUDIT-C score = 0, HDL was higher among patients with greater AUDIT-C severity (e.g. moderate AUDIT-C score = 8.70[7.65, 9.75] mg/dl; severe AUDIT-C score = 13.02 [12.13, 13.90] mg/dL[95% confidence interval (CI)] mg/dl). The associations between cross-sectional HDL and AUDIT-C scores were similar with and without adjusting for patient demographic and clinical characteristics. HDL levels increased for patients with mild alcohol use at baseline and moderate or severe alcohol use at follow-up (15.06[2.77, 27.69] and 19.58[2.77, 36.39] mg/dL[95%CI] increase for moderate and severe, respectively). CONCLUSIONS HDL levels correlate with AUDIT-C scores among patients with MH/SUD. Longitudinally, there were some (but not consistent) increases in HDL associated with increases in AUDIT-C. The increases were within range of typical year-to-year variation in HDL across the population independent of alcohol use, limiting the ability to use HDL as a longitudinal clinical indicator for alcohol use in routine care.
Collapse
Affiliation(s)
- Nicole M Benson
- McLean Hospital, Belmont, MA 02478, United States; Department of Psychiatry, Harvard Medical School, Boston, MA 02115, United States
| | - Amin Yakubu
- McLean Hospital, Belmont, MA 02478, United States
| | - Boyu Ren
- McLean Hospital, Belmont, MA 02478, United States; Department of Psychiatry, Harvard Medical School, Boston, MA 02115, United States
| | - Carol Aboud
- McLean Hospital, Belmont, MA 02478, United States
| | | | - Shelly F Greenfield
- McLean Hospital, Belmont, MA 02478, United States; Department of Psychiatry, Harvard Medical School, Boston, MA 02115, United States
| | - Alisa B Busch
- McLean Hospital, Belmont, MA 02478, United States; Departments of Psychiatry and Health Care Policy, Harvard Medical School, Boston, MA 02115, United States
| |
Collapse
|
3
|
Simon GE, Cruz M, Shortreed SM, Sterling SA, Coleman KJ, Ahmedani BK, Yaseen ZS, Mosholder AD. Stability of Suicide Risk Prediction Models During Changes in Health Care Delivery. Psychiatr Serv 2024; 75:139-147. [PMID: 37587793 DOI: 10.1176/appi.ps.20230172] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 08/18/2023]
Abstract
OBJECTIVE The authors aimed to use health records data to examine how the accuracy of statistical models predicting self-harm or suicide changed between 2015 and 2019, as health systems implemented suicide prevention programs. METHODS Data from four large health systems were used to identify specialty mental health visits by patients ages ≥11 years, assess 311 potential predictors of self-harm (including demographic characteristics, historical risk factors, and index visit characteristics), and ascertain fatal or nonfatal self-harm events over 90 days after each visit. New prediction models were developed with logistic regression with LASSO (least absolute shrinkage and selection operator) in random samples of visits (65%) from each calendar year and were validated in the remaining portion of the sample (35%). RESULTS A model developed for visits from 2009 to mid-2015 showed similar classification performance and calibration accuracy in a new sample of about 13.1 million visits from late 2015 to 2019. Area under the receiver operating characteristic curve (AUC) ranged from 0.840 to 0.849 in the new sample, compared with 0.851 in the original sample. New models developed for each year for 2015-2019 had classification performance (AUC range 0.790-0.853), sensitivity, and positive predictive value similar to those of the previously developed model. Models selected similar predictors from 2015 to 2019, except for more frequent selection of depression questionnaire data in later years, when questionnaires were more frequently recorded. CONCLUSIONS A self-harm prediction model developed with 2009-2015 visit data performed similarly when applied to 2015-2019 visits. New models did not yield superior performance or identify different predictors.
Collapse
Affiliation(s)
- Gregory E Simon
- Washington Health Research Institute, Kaiser Permanente, Seattle (Simon, Cruz, Shortreed); Bernard J. Tyson School of Medicine (Simon, Coleman) and Southern California Department of Research and Evaluation (Coleman), Kaiser Permanente, Pasadena; Department of Biostatistics, University of Washington, Seattle (Cruz, Shortreed); Northern California Division of Research, Kaiser Permanente, Oakland (Sterling); Henry Ford Health Center for Health Services Research, Detroit (Ahmedani); U.S. Food and Drug Administration (FDA), Silver Spring, Maryland (Yaseen, Mosholder)
| | - Maricela Cruz
- Washington Health Research Institute, Kaiser Permanente, Seattle (Simon, Cruz, Shortreed); Bernard J. Tyson School of Medicine (Simon, Coleman) and Southern California Department of Research and Evaluation (Coleman), Kaiser Permanente, Pasadena; Department of Biostatistics, University of Washington, Seattle (Cruz, Shortreed); Northern California Division of Research, Kaiser Permanente, Oakland (Sterling); Henry Ford Health Center for Health Services Research, Detroit (Ahmedani); U.S. Food and Drug Administration (FDA), Silver Spring, Maryland (Yaseen, Mosholder)
| | - Susan M Shortreed
- Washington Health Research Institute, Kaiser Permanente, Seattle (Simon, Cruz, Shortreed); Bernard J. Tyson School of Medicine (Simon, Coleman) and Southern California Department of Research and Evaluation (Coleman), Kaiser Permanente, Pasadena; Department of Biostatistics, University of Washington, Seattle (Cruz, Shortreed); Northern California Division of Research, Kaiser Permanente, Oakland (Sterling); Henry Ford Health Center for Health Services Research, Detroit (Ahmedani); U.S. Food and Drug Administration (FDA), Silver Spring, Maryland (Yaseen, Mosholder)
| | - Stacy A Sterling
- Washington Health Research Institute, Kaiser Permanente, Seattle (Simon, Cruz, Shortreed); Bernard J. Tyson School of Medicine (Simon, Coleman) and Southern California Department of Research and Evaluation (Coleman), Kaiser Permanente, Pasadena; Department of Biostatistics, University of Washington, Seattle (Cruz, Shortreed); Northern California Division of Research, Kaiser Permanente, Oakland (Sterling); Henry Ford Health Center for Health Services Research, Detroit (Ahmedani); U.S. Food and Drug Administration (FDA), Silver Spring, Maryland (Yaseen, Mosholder)
| | - Karen J Coleman
- Washington Health Research Institute, Kaiser Permanente, Seattle (Simon, Cruz, Shortreed); Bernard J. Tyson School of Medicine (Simon, Coleman) and Southern California Department of Research and Evaluation (Coleman), Kaiser Permanente, Pasadena; Department of Biostatistics, University of Washington, Seattle (Cruz, Shortreed); Northern California Division of Research, Kaiser Permanente, Oakland (Sterling); Henry Ford Health Center for Health Services Research, Detroit (Ahmedani); U.S. Food and Drug Administration (FDA), Silver Spring, Maryland (Yaseen, Mosholder)
| | - Brian K Ahmedani
- Washington Health Research Institute, Kaiser Permanente, Seattle (Simon, Cruz, Shortreed); Bernard J. Tyson School of Medicine (Simon, Coleman) and Southern California Department of Research and Evaluation (Coleman), Kaiser Permanente, Pasadena; Department of Biostatistics, University of Washington, Seattle (Cruz, Shortreed); Northern California Division of Research, Kaiser Permanente, Oakland (Sterling); Henry Ford Health Center for Health Services Research, Detroit (Ahmedani); U.S. Food and Drug Administration (FDA), Silver Spring, Maryland (Yaseen, Mosholder)
| | - Zimri S Yaseen
- Washington Health Research Institute, Kaiser Permanente, Seattle (Simon, Cruz, Shortreed); Bernard J. Tyson School of Medicine (Simon, Coleman) and Southern California Department of Research and Evaluation (Coleman), Kaiser Permanente, Pasadena; Department of Biostatistics, University of Washington, Seattle (Cruz, Shortreed); Northern California Division of Research, Kaiser Permanente, Oakland (Sterling); Henry Ford Health Center for Health Services Research, Detroit (Ahmedani); U.S. Food and Drug Administration (FDA), Silver Spring, Maryland (Yaseen, Mosholder)
| | - Andrew D Mosholder
- Washington Health Research Institute, Kaiser Permanente, Seattle (Simon, Cruz, Shortreed); Bernard J. Tyson School of Medicine (Simon, Coleman) and Southern California Department of Research and Evaluation (Coleman), Kaiser Permanente, Pasadena; Department of Biostatistics, University of Washington, Seattle (Cruz, Shortreed); Northern California Division of Research, Kaiser Permanente, Oakland (Sterling); Henry Ford Health Center for Health Services Research, Detroit (Ahmedani); U.S. Food and Drug Administration (FDA), Silver Spring, Maryland (Yaseen, Mosholder)
| |
Collapse
|
4
|
Bentley KH, Madsen EM, Song E, Zhou Y, Castro V, Lee H, Lee YH, Smoller JW. Determining Distinct Suicide Attempts From Recurrent Electronic Health Record Codes: Classification Study. JMIR Form Res 2024; 8:e46364. [PMID: 38190236 PMCID: PMC10804255 DOI: 10.2196/46364] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2023] [Revised: 09/15/2023] [Accepted: 09/27/2023] [Indexed: 01/09/2024] Open
Abstract
BACKGROUND Prior suicide attempts are a relatively strong risk factor for future suicide attempts. There is growing interest in using longitudinal electronic health record (EHR) data to derive statistical risk prediction models for future suicide attempts and other suicidal behavior outcomes. However, model performance may be inflated by a largely unrecognized form of "data leakage" during model training: diagnostic codes for suicide attempt outcomes may refer to prior attempts that are also included in the model as predictors. OBJECTIVE We aimed to develop an automated rule for determining when documented suicide attempt diagnostic codes identify distinct suicide attempt events. METHODS From a large health care system's EHR, we randomly sampled suicide attempt codes for 300 patients with at least one pair of suicide attempt codes documented at least one but no more than 90 days apart. Supervised chart reviewers assigned the clinical settings (ie, emergency department [ED] versus non-ED), methods of suicide attempt, and intercode interval (number of days). The probability (or positive predictive value) that the second suicide attempt code in a given pair of codes referred to a distinct suicide attempt event from its preceding suicide attempt code was calculated by clinical setting, method, and intercode interval. RESULTS Of 1015 code pairs reviewed, 835 (82.3%) were nonindependent (ie, the 2 codes referred to the same suicide attempt event). When the second code in a pair was documented in a clinical setting other than the ED, it represented a distinct suicide attempt 3.3% of the time. The more time elapsed between codes, the more likely the second code in a pair referred to a distinct suicide attempt event from its preceding code. Code pairs in which the second suicide attempt code was assigned in an ED at least 5 days after its preceding suicide attempt code had a positive predictive value of 0.90. CONCLUSIONS EHR-based suicide risk prediction models that include International Classification of Diseases codes for prior suicide attempts as a predictor may be highly susceptible to bias due to data leakage in model training. We derived a simple rule to distinguish codes that reflect new, independent suicide attempts: suicide attempt codes documented in an ED setting at least 5 days after a preceding suicide attempt code can be confidently treated as new events in EHR-based suicide risk prediction models. This rule has the potential to minimize upward bias in model performance when prior suicide attempts are included as predictors in EHR-based suicide risk prediction models.
Collapse
Affiliation(s)
- Kate H Bentley
- Center for Precision Psychiatry, Department of Psychiatry, Massachusetts General Hospital, Boston, MA, United States
- Department of Psychiatry, Harvard Medical School, Boston, MA, United States
| | - Emily M Madsen
- Center for Precision Psychiatry, Department of Psychiatry, Massachusetts General Hospital, Boston, MA, United States
- Psychiatric and Neurodevelopmental Genetics Unit, Center for Genomic Medicine, Massachusetts General Hospital, Boston, MA, United States
| | - Eugene Song
- Center for Precision Psychiatry, Department of Psychiatry, Massachusetts General Hospital, Boston, MA, United States
- Psychiatric and Neurodevelopmental Genetics Unit, Center for Genomic Medicine, Massachusetts General Hospital, Boston, MA, United States
| | - Yu Zhou
- Center for Precision Psychiatry, Department of Psychiatry, Massachusetts General Hospital, Boston, MA, United States
- Psychiatric and Neurodevelopmental Genetics Unit, Center for Genomic Medicine, Massachusetts General Hospital, Boston, MA, United States
| | - Victor Castro
- Mass General Brigham Research Information Science and Computing, Somerville, MA, United States
| | - Hyunjoon Lee
- Center for Precision Psychiatry, Department of Psychiatry, Massachusetts General Hospital, Boston, MA, United States
- Psychiatric and Neurodevelopmental Genetics Unit, Center for Genomic Medicine, Massachusetts General Hospital, Boston, MA, United States
| | - Younga H Lee
- Department of Psychiatry, Harvard Medical School, Boston, MA, United States
- Psychiatric and Neurodevelopmental Genetics Unit, Center for Genomic Medicine, Massachusetts General Hospital, Boston, MA, United States
| | - Jordan W Smoller
- Center for Precision Psychiatry, Department of Psychiatry, Massachusetts General Hospital, Boston, MA, United States
- Department of Psychiatry, Harvard Medical School, Boston, MA, United States
- Psychiatric and Neurodevelopmental Genetics Unit, Center for Genomic Medicine, Massachusetts General Hospital, Boston, MA, United States
| |
Collapse
|
5
|
Simon GE, Shortreed SM, Johnson E, Yaseen ZS, Stone M, Mosholder AD, Ahmedani BK, Coleman KJ, Coley RY, Penfold RB, Toh S. Predicting risk of suicidal behavior from insurance claims data vs. linked data from insurance claims and electronic health records. Pharmacoepidemiol Drug Saf 2024; 33:e5734. [PMID: 38112287 PMCID: PMC10843611 DOI: 10.1002/pds.5734] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2023] [Revised: 10/16/2023] [Accepted: 11/10/2023] [Indexed: 12/21/2023]
Abstract
PURPOSE Observational studies assessing effects of medical products on suicidal behavior often rely on health record data to account for pre-existing risk. We assess whether high-dimensional models predicting suicide risk using data derived from insurance claims and electronic health records (EHRs) are superior to models using data from insurance claims alone. METHODS Data were from seven large health systems identified outpatient mental health visits by patients aged 11 or older between 1/1/2009 and 9/30/2017. Data for the 5 years prior to each visit identified potential predictors of suicidal behavior typically available from insurance claims (e.g., mental health diagnoses, procedure codes, medication dispensings) and additional potential predictors available from EHRs (self-reported race and ethnicity, responses to Patient Health Questionnaire or PHQ-9 depression questionnaires). Nonfatal self-harm events following each visit were identified from insurance claims data and fatal self-harm events were identified by linkage to state mortality records. Random forest models predicting nonfatal or fatal self-harm over 90 days following each visit were developed in a 70% random sample of visits and validated in a held-out sample of 30%. Performance of models using linked claims and EHR data was compared to models using claims data only. RESULTS Among 15 845 047 encounters by 1 574 612 patients, 99 098 (0.6%) were followed by a self-harm event within 90 days. Overall classification performance did not differ between the best-fitting model using all data (area under the receiver operating curve or AUC = 0.846, 95% CI 0.839-0.854) and the best-fitting model limited to data available from insurance claims (AUC = 0.846, 95% CI 0.838-0.853). Competing models showed similar classification performance across a range of cut-points and similar calibration performance across a range of risk strata. Results were similar when the sample was limited to health systems and time periods where PHQ-9 depression questionnaires were recorded more frequently. CONCLUSION Investigators using health record data to account for pre-existing risk in observational studies of suicidal behavior need not limit that research to databases including linked EHR data.
Collapse
Affiliation(s)
- Gregory E Simon
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington, USA
- Department of Health Systems Science, Bernard J. Tyson Kaiser Permanente School of Medicine, Pasadena, California, USA
| | - Susan M Shortreed
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington, USA
- Department of Biostatistics, University of Washington, Seattle, Washington, USA
| | - Eric Johnson
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington, USA
| | - Zimri S Yaseen
- U.S. Food and Drug Administration, Silver Spring, Maryland, USA
| | - Marc Stone
- U.S. Food and Drug Administration, Silver Spring, Maryland, USA
| | | | - Brian K Ahmedani
- Center for Health Policy and Health Services Research, Henry Ford Health, Detroit, Michigan, USA
| | - Karen J Coleman
- Department of Health Systems Science, Bernard J. Tyson Kaiser Permanente School of Medicine, Pasadena, California, USA
- Department of Research and Evaluation, Kaiser Permanente Southern California, Pasadena, California, USA
| | - R Yates Coley
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington, USA
- Department of Biostatistics, University of Washington, Seattle, Washington, USA
| | - Robert B Penfold
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington, USA
| | - Sengwee Toh
- Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, Massachusetts, USA
| |
Collapse
|
6
|
Nielsen SD, Christensen RHB, Madsen T, Karstoft KI, Clemmensen L, Benros ME. Prediction models of suicide and non-fatal suicide attempt after discharge from a psychiatric inpatient stay: A machine learning approach on nationwide Danish registers. Acta Psychiatr Scand 2023; 148:525-537. [PMID: 37961014 DOI: 10.1111/acps.13629] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/22/2023] [Revised: 10/06/2023] [Accepted: 10/13/2023] [Indexed: 11/15/2023]
Abstract
INTRODUCTION To develop machine learning models capable of predicting suicide and non-fatal suicide attempt as separate outcomes in the first 30 days after discharge from a psychiatric inpatient stay. METHODS Prospective cohort study using nationwide Danish registry data. We included individuals who were 18 years or older, and all discharges from psychiatric hospitalizations in Denmark from 1995 to 2018. We trained predictive models using 10-fold cross validation on 80% of the data and did testing on the remaining 20%. RESULTS The best model for predicting non-fatal suicide attempt was an ensemble of predictions from gradient boosting (XGBoost) and categorical boosting (catBoost). The ROC-AUC for predicting suicide attempt was 0.85 (95% CI: 0.84-0.85). At a risk threshold of 4.36%, positive predictive value (PPV) was 11.0% and sensitivity was 47.2%. The best model for predicting suicide was an ensemble of predictions from random forest, XGBoost and catBoost. For suicide, the ROC-AUC was 0.71 (95% CI: 0.70-0.73). At a risk threshold of 0.15%, PPV was 0.34% and sensitivity was 56.0%. The most contributing predictors differed when predicting suicide and suicide attempt, indicating that separate models are needed. The ensemble model was fair across sex and age, and more so than the penalized logistic regression model. CONCLUSIONS We achieved good performance for predicting suicide attempts and demonstrated a clinical application of ensemble models. Our results indicate a difference in predictive performance for models predicting suicide and suicide attempt, respectively. Thus, we recommend that suicide and suicide attempt are treated as two separate endpoints, in particular for clinical application. We demonstrated that the ensemble model is fairer across sex and age compared with a penalized logistic regression, and therefore we recommend the use of well-tested ensembles despite a more complex explainability.
Collapse
Affiliation(s)
- Sara Dorthea Nielsen
- Department of Applied Mathematics and Computer Science, Technical University of Denmark, Kgs. Lyngby, Denmark
- Copenhagen Research Center for Biological and Precision Psychiatry, Mental Health Centre Copenhagen, Copenhagen University Hospital, Copenhagen, Denmark
| | - Rune H B Christensen
- Copenhagen Research Center for Biological and Precision Psychiatry, Mental Health Centre Copenhagen, Copenhagen University Hospital, Copenhagen, Denmark
| | - Trine Madsen
- Danish Research Institute of Suicide Prevention, Mental Health Center Copenhagen, Copenhagen, Denmark
- Section of Epidemiology, University of Copenhagen, Faculty of Health and Medical Sciences, Copenhagen, Denmark
| | - Karen-Inge Karstoft
- Department of Psychology, Faculty of Social Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Line Clemmensen
- Department of Applied Mathematics and Computer Science, Technical University of Denmark, Kgs. Lyngby, Denmark
| | - Michael E Benros
- Copenhagen Research Center for Biological and Precision Psychiatry, Mental Health Centre Copenhagen, Copenhagen University Hospital, Copenhagen, Denmark
- Department of Clinical Medicine, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| |
Collapse
|
7
|
Arias SA, Sperber K, Jones R, Taxman FS, Miller TR, Zylberfuden S, Weinstock LM, Brown GK, Ahmedani B, Johnson JE. Managed care updates of subscriber jail release to prompt community suicide prevention: clinical trial protocol. BMC Health Serv Res 2023; 23:1265. [PMID: 37974126 PMCID: PMC10655488 DOI: 10.1186/s12913-023-10249-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2023] [Accepted: 10/30/2023] [Indexed: 11/19/2023] Open
Abstract
BACKGROUND Recent jail detention is a marker for trait and state suicide risk in community-based populations. However, healthcare providers are typically unaware that their client was in jail and few post-release suicide prevention efforts exist. This protocol paper describes an effectiveness-implementation trial evaluating community suicide prevention practices triggered by advances in informatics that alert CareSource, a large managed care organization (MCO), when a subscriber is released from jail. METHODS This randomized controlled trial investigates two evidence-based suicide prevention practices triggered by CareSource's jail detention/release notifications, in a partial factorial design. The first phase randomizes ~ 43,000 CareSource subscribers who pass through any Ohio jail to receive Caring Contact letters sent by CareSource or to Usual Care after jail release. The second phase (running simultaneously) involves a subset of ~ 6,000 of the 43,000 subscribers passing through jail who have been seen in one of 12 contracted behavioral health agencies in the 6 months prior to incarceration in a stepped-wedge design. Agencies will receive: (a) notifications of the client's jail detention/release, (b) instructions for re-engaging these clients, and (c) training in suicide risk assessment and the Safety Planning Intervention for use at re-engagement. We will track suicide-related and service linkage outcomes 6 months following jail release using claims data. CONCLUSIONS This design allows us to rigorously test two intervention main effects and their interaction. It also provides valuable information on the effects of system-level change and the scalability of interventions using big data from a MCO to flag jail release and suicide risk. TRIAL REGISTRATION The trial is registered at clinicaltrials.gov (NCT05579600). Registered 27 June, 2023.
Collapse
Affiliation(s)
- Sarah A Arias
- Butler Hospital, Providence, RI, USA.
- Department of Psychiatry and Human Behavior, Brown University, Butler Hospital, 345 Blackstone Blvd., Providence, RI, 02906, USA.
| | | | - Richard Jones
- Department of Psychiatry and Human Behavior, Brown University, Butler Hospital, 345 Blackstone Blvd., Providence, RI, 02906, USA
| | - Faye S Taxman
- Center for Advancing Correctional Excellence!, George Mason University, Arlington, VA, USA
| | - Ted R Miller
- Pacific Institute for Research & Evaluation, Beltsville, MI, USA
- Curtin University School of Public Health, Perth, Australia
| | | | - Lauren M Weinstock
- Department of Psychiatry and Human Behavior, Brown University, Butler Hospital, 345 Blackstone Blvd., Providence, RI, 02906, USA
| | - Gregory K Brown
- Department of Psychiatry, University of Pennsylvania, Philadelphia, PA, USA
| | - Brian Ahmedani
- Department of Psychiatry, Henry Ford Health System, Detroit, MI, USA
| | - Jennifer E Johnson
- Charles Stewart Mott Department of Public Health, Michigan State University, Flint, MI, USA
| |
Collapse
|
8
|
Arias SA, Sperber K, Jones R, Taxman FS, Miller TR, Zylberfuden S, Weinstock LM, Brown GK, Ahmedani B, Johnson JE. Managed Care Updates of Subscriber Jail Release to Prompt Community Suicide Prevention: Clinical Trial Protocol. RESEARCH SQUARE 2023:rs.3.rs-3350204. [PMID: 37841869 PMCID: PMC10571633 DOI: 10.21203/rs.3.rs-3350204/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/17/2023]
Abstract
Background Recent jail detention is a marker for trait and state suicide risk in community-based populations. However, healthcare providers are typically unaware that their client was in jail and few post-release suicide prevention efforts exist. This protocol paper describes an effectiveness-implementation trial evaluating community suicide prevention practices triggered by advances in informatics that alert CareSource, a large managed care organization (MCO), when a subscriber is released from jail. Methods This randomized controlled trial investigates two evidence-based suicide prevention practices triggered by CareSource's jail detention/release notifications, in a partial factorial design. The first phase randomizes ~43,000 CareSource subscribers who pass through any Ohio jail to receive Caring Contact letters sent by CareSource or to Usual Care after jail release. The second phase (running simultaneously) involves a subset of ~6,000 of the 43,000 subscribers passing through jail who have been seen in one of 12 contracted behavioral health agencies in the 6 months prior to incarceration in a stepped-wedge design. Agencies will receive: (a) notifications of the client's jail detention/release, (b) instructions for re-engaging these clients, and (c) training in suicide risk assessment and the Safety Planning Intervention for use at re-engagement. We will track suicide-related and service linkage outcomes 6 months following jail release using claims data. Conclusions This design allows us to rigorously test two intervention main effects and their interaction. It also provides valuable information on the effects of system-level change and the scalability of interventions using big data from a MCO to flag jail release and suicide risk. Trial Registration The trial is registered at clinicaltrials.gov (NCT05579600). Registered 27 June, 2023, https://beta.clinicaltrials.gov/study/NCT05579600?cond=Suicide&term=Managed%20Care&rank=1.
Collapse
|
9
|
Puig-Amores I, Cuadrado-Gordillo I, Martín-Mora-Parra G. Health Service Protection vis-à-vis the Detection of Psychosocial Risks of Suicide during the Years 2019-2021. Healthcare (Basel) 2023; 11:healthcare11101505. [PMID: 37239791 DOI: 10.3390/healthcare11101505] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2023] [Revised: 05/18/2023] [Accepted: 05/21/2023] [Indexed: 05/28/2023] Open
Abstract
Health services are especially relevant in suicide prevention and intervention, representing a favourable environment in which to implement specific strategies to detect and address suicidal behaviours. Indeed, a significant proportion of people who die by suicide (DBS) present at primary care and mental health services during the last year, month, or even days before committing suicide. The objective of this descriptive and cross-sectional study of all registered cases of death by suicide (N = 265) in Extremadura (Spain) was to determine which of those people who died by suicide had mental health problems (MHP) and what type of assistance they had requested. Diagnoses, previous suicide attempts, type of health service, and last visit before death were explored with univariate analyses and logistic regressions. The proportion of people without MHP was found to be high, and these people had hardly visited the health services at all in their last year. People with MHP, between the ages of 40 and 69, and with previous suicide attempts were more likely to have visited the mental health service in the three months prior to their death. It is, thus, necessary to provide health professionals with tools and training in the prevention of and approach to suicide. Efforts must be directed towards effectively assessing mental health and the risk of suicide since a large proportion of people who die by suicide may go unnoticed.
Collapse
Affiliation(s)
- Ismael Puig-Amores
- Department of Psychology and Anthropology, Faculty of Education and Psychology, University of Extremadura, 06071 Badajoz, Spain
| | - Isabel Cuadrado-Gordillo
- Department of Psychology and Anthropology, Faculty of Education and Psychology, University of Extremadura, 06071 Badajoz, Spain
| | - Guadalupe Martín-Mora-Parra
- Department of Psychology and Anthropology, Faculty of Education and Psychology, University of Extremadura, 06071 Badajoz, Spain
| |
Collapse
|
10
|
Vera-Varela C, Manrique Mirón PC, Barrigón ML, Álvarez-García R, Portillo P, Chamorro J, Baca-García E. Low Level of Agreement Between Self-Report and Clinical Assessment of Passive Suicidal Ideation. Arch Suicide Res 2022; 26:1895-1910. [PMID: 34223799 DOI: 10.1080/13811118.2021.1945984] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
Abstract
OBJECTIVE Discrepancies between patient reports during clinical evaluations and self-reported suicide ideation are of vital importance. We study the agreement in passive suicidal ideation between reports made by clinicians and patients' self-reports. METHOD Wish of death in 648 outpatients was assessed by attending clinicians. Within 24 h after clinical evaluation, patients completed a self-report questionnaire in which they were asked whether they had no desire to live. We used cluster analysis to determine the clinical profile of a population of patients according to the concordance between reports made by clinicians and self-reported information. RESULTS A low level of agreement (kappa = 0.072) was found between clinicians and patients, as 56.4% (n = 366) of clinician reports classified as containing no death-related ideas although on self-report the patient did state that they had no desire to live. In this group containing discrepancies between the two reports, two clusters were found to have shared characteristics: female sex, middle age, cohabitation, active employment, no history of suicidal behavior, and diagnosis of neurotic, stress-related, and somatoform disorders. In a third, more severe cluster, patients self-reported sleep disturbances, less appetite, poor treatment adherence, and aggressiveness. CONCLUSIONS We found low agreement between self-reports and clinician assessments regarding the death wish. Self-reporting may be useful in assessing suicide risk. HIGHLIGHTSLow agreement was found between self-reports and clinician assessments regarding passive suicidal ideation.Most patients in whom the clinician underestimated the risk of suicide were women.Our results suggest that clinicians require adequate documentation of suicidal risk assessment to identify the high-risk population.
Collapse
|
11
|
Cruz M, Shortreed SM, Richards JE, Coley RY, Yarborough BJ, Walker RL, Johnson E, Ahmedani BK, Rossom R, Coleman KJ, Boggs JM, Beck AL, Simon GE. Machine Learning Prediction of Suicide Risk Does Not Identify Patients Without Traditional Risk Factors. J Clin Psychiatry 2022; 83:21m14178. [PMID: 36044603 PMCID: PMC10270326 DOI: 10.4088/jcp.21m14178] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/15/2022]
Abstract
Objective: To determine whether predictions of suicide risk from machine learning models identify unexpected patients or patients without medical record documentation of traditional risk factors. Methods: The study sample included 27,091,382 outpatient mental health (MH) specialty or general medical visits with a MH diagnosis for patients aged 11 years or older from January 1, 2009, to September 30, 2017. We used predicted risk scores of suicide attempt and suicide death, separately, within 90 days of visits to classify visits into risk score percentile strata. For each stratum, we calculated counts and percentages of visits with traditional risk factors, including prior self-harm diagnoses and emergency department visits or hospitalizations with MH diagnoses, in the last 3, 12, and 60 months. Results: Risk-factor percentages increased with predicted risk scores. Among MH specialty visits, 66%, 88%, and 99% of visits with suicide attempt risk scores in the top 3 strata (respectively, 90th-95th, 95th-98th, and ≥ 98th percentiles) and 60%, 77%, and 93% of visits with suicide risk scores in the top 3 strata represented patients who had at least one traditional risk factor documented in the prior 12 months. Among general medical visits, 52%, 66%, and 90% of visits with suicide attempt risk scores in the top 3 strata and 45%, 66%, and 79% of visits with suicide risk scores in the top 3 strata represented patients who had a history of traditional risk factors in the last 12 months. Conclusions: Suicide risk alerts based on these machine learning models coincide with patients traditionally thought of as high-risk at their high-risk visits.
Collapse
Affiliation(s)
- Maricela Cruz
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington
- Department of Biostatistics, School of Public Health, University of Washington, Seattle, Washington
- Corresponding author: Maricela Cruz, Kaiser Permanente Washington Health Research Institute, 1730 Minor Ave Ste 1600, Seattle, WA 98101
| | - Susan M Shortreed
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington
- Department of Biostatistics, School of Public Health, University of Washington, Seattle, Washington
| | - Julie E Richards
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington
- Department of Health Services, School of Public Health, University of Washington, Seattle, Washington
| | - R Yates Coley
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington
- Department of Biostatistics, School of Public Health, University of Washington, Seattle, Washington
| | | | - Rod L Walker
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington
| | - Eric Johnson
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington
| | - Brian K Ahmedani
- Henry Ford Health System, Center for Health Policy & Health Services Research, Detroit, Michigan
| | | | - Karen J Coleman
- Kaiser Permanente Southern California, Department of Research and Evaluation, Pasadena, California
| | - Jennifer M Boggs
- Kaiser Permanente Colorado Institute for Health Research, Aurora, Colorado
| | - Arne L Beck
- Kaiser Permanente Colorado Institute for Health Research, Aurora, Colorado
| | - Gregory E Simon
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington
| |
Collapse
|
12
|
Bondesson E, Alpar T, Petersson IF, Schelin MEC, Jöud A. Health care utilization among individuals who die by suicide as compared to the general population: a population-based register study in Sweden. BMC Public Health 2022; 22:1616. [PMID: 36008801 PMCID: PMC9404588 DOI: 10.1186/s12889-022-14006-x] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2022] [Accepted: 08/12/2022] [Indexed: 11/21/2022] Open
Abstract
Background Globally, 700 000 people die every year by suicide. Health care consultation patterns the period before suicide could be one potential way to identify people at risk for suicide. Therefore, this study examines health care patterns up to one year prior to the suicide by age, sex and prior diagnoses and specifically investigates if and how this differs from the general population of Skåne, Sweden. Methods This cohort study includes all individuals, aged 15 and older, that died by suicide in Region Skåne, Sweden from 2004 to 2015 (n = 1653). The individuals were identified through the Cause of death register and then linked to the Skåne healthcare register. Health care data was analyzed as proportions consulting different types of health care the month and year preceding the suicide, we also investigated the impact of age, sex and the occurrence of prior psychiatric and pain diagnoses. Additionally, we compared the proportion of consulting care among the suicide victims and the general population of Skåne. Results In the month before their death, 53% of the suicide victims had any health care consultation, compared with 20% in the general population of Skåne, a given month (p < 0.0001). The corresponding figures for the year prior to suicide was 86% among those who died by suicide, compared to 69% in the general population of Skåne, a given year (p < 0.0001). Women, and those having a documented history of psychiatric diagnosis were more likely to have health care consultations in the month and year preceding suicide (p < 0.001), compared to men and suicide victims without a history of psychiatric disease. Older adults that died by suicide, were less likely to consult psychiatric care compared to the younger suicide victims (p < 0.001). Conclusion A majority of the suicide victims consulted health care in the near time before death and the proportion of seeking health care was significantly higher than in the general population of Skåne and higher among female suicide victims as compared to males. Alternative preventive screening measures should be considered for individuals consulting health care, especially for older people and individuals outside the psychiatric care.
Collapse
Affiliation(s)
- Elisabeth Bondesson
- Faculty of Medicine, Department of Clinical Sciences Lund, Division of Orthopaedics, Lund University, Lund, Sweden. .,Department of Neurosurgery and Pain Rehabilitation, Skåne University Hospital, Lund, Sweden. .,Faculty of Medicine, Department of Laboratory Medicine, Division of Occupational and Environmental Medicine, Lund University, Lund, Sweden.
| | - Tori Alpar
- Faculty of Medicine, Department of Clinical Sciences Lund, Division of Orthopaedics, Lund University, Lund, Sweden
| | - Ingemar F Petersson
- Faculty of Medicine, Department of Clinical Sciences Lund, Division of Orthopaedics, Lund University, Lund, Sweden.,Department of Research and Education, Skåne University Hospital, Lund, Sweden
| | - Maria E C Schelin
- Faculty of Medicine, Department of Clinical Sciences Lund, Division of Orthopaedics, Lund University, Lund, Sweden.,Department of Research and Education, Skåne University Hospital, Lund, Sweden.,Institute for Palliative Care, Lund University and Region Skåne, Lund, Sweden
| | - Anna Jöud
- Faculty of Medicine, Department of Clinical Sciences Lund, Division of Orthopaedics, Lund University, Lund, Sweden.,Faculty of Medicine, Department of Laboratory Medicine, Division of Occupational and Environmental Medicine, Lund University, Lund, Sweden.,Department of Research and Education, Skåne University Hospital, Lund, Sweden
| |
Collapse
|
13
|
Hopkins D, Rickwood DJ, Hallford DJ, Watsford C. Structured data vs. unstructured data in machine learning prediction models for suicidal behaviors: A systematic review and meta-analysis. Front Digit Health 2022; 4:945006. [PMID: 35983407 PMCID: PMC9378826 DOI: 10.3389/fdgth.2022.945006] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2022] [Accepted: 06/29/2022] [Indexed: 11/23/2022] Open
Abstract
Suicide remains a leading cause of preventable death worldwide, despite advances in research and decreases in mental health stigma through government health campaigns. Machine learning (ML), a type of artificial intelligence (AI), is the use of algorithms to simulate and imitate human cognition. Given the lack of improvement in clinician-based suicide prediction over time, advancements in technology have allowed for novel approaches to predicting suicide risk. This systematic review and meta-analysis aimed to synthesize current research regarding data sources in ML prediction of suicide risk, incorporating and comparing outcomes between structured data (human interpretable such as psychometric instruments) and unstructured data (only machine interpretable such as electronic health records). Online databases and gray literature were searched for studies relating to ML and suicide risk prediction. There were 31 eligible studies. The outcome for all studies combined was AUC = 0.860, structured data showed AUC = 0.873, and unstructured data was calculated at AUC = 0.866. There was substantial heterogeneity between the studies, the sources of which were unable to be defined. The studies showed good accuracy levels in the prediction of suicide risk behavior overall. Structured data and unstructured data also showed similar outcome accuracy according to meta-analysis, despite different volumes and types of input data.
Collapse
Affiliation(s)
- Danielle Hopkins
- Faculty of Health, University of Canberra, Canberra, ACT, Australia
- *Correspondence: Danielle Hopkins
| | | | | | - Clare Watsford
- Faculty of Health, University of Canberra, Canberra, ACT, Australia
| |
Collapse
|
14
|
Shaw JL, Beans JA, Noonan C, Smith JJ, Mosley M, Lillie KM, Avey JP, Ziebell R, Simon G. Validating a predictive algorithm for suicide risk with Alaska Native populations. Suicide Life Threat Behav 2022; 52:696-704. [PMID: 35293010 PMCID: PMC9378560 DOI: 10.1111/sltb.12853] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/12/2021] [Revised: 12/09/2021] [Accepted: 01/11/2022] [Indexed: 11/29/2022]
Abstract
INTRODUCTION The American Indian/Alaska Native (AI/AN) suicide rate in Alaska is twice the state rate and four times the U.S. rate. Healthcare systems need innovative methods of suicide risk detection. The Mental Health Research Network (MHRN) developed suicide risk prediction algorithms in a general U.S. PATIENT POPULATION METHODS We applied MHRN predictors and regression coefficients to electronic health records of AI/AN patients aged ≥13 years with behavioral health diagnoses and primary care visits between October 1, 2016, and March 30, 2018. Logistic regression assessed model accuracy for predicting and stratifying risk for suicide attempt within 90 days after a visit. We compared expected to observed risk and assessed model performance characteristics. RESULTS 10,864 patients made 47,413 primary care visits. Suicide attempt occurred after 589 (1.2%) visits. Visits in the top 5% of predicted risk accounted for 40% of actual attempts. Among visits in the top 0.5% of predicted risk, 25.1% were followed by suicide attempt. The best fitting model had an AUC of 0.826 (95% CI: 0.809-0.843). CONCLUSIONS The MHRN model accurately predicted suicide attempts among AI/AN patients. Future work should develop clinical and operational guidance for effective implementation of the model with this population.
Collapse
Affiliation(s)
- Jennifer L Shaw
- Division of Organizational Development and Innovation, Research and Data Services Department, Southcentral Foundation, Anchorage, Alaska, USA
| | - Julie A Beans
- Division of Organizational Development and Innovation, Research and Data Services Department, Southcentral Foundation, Anchorage, Alaska, USA
| | - Carolyn Noonan
- Institute for Research and Education to Advance Community Health, Washington State University, Seattle, Washington, USA
| | - Julia J Smith
- Division of Organizational Development and Innovation, Research and Data Services Department, Southcentral Foundation, Anchorage, Alaska, USA
| | - Mike Mosley
- Division of Organizational Development and Innovation, Research and Data Services Department, Southcentral Foundation, Anchorage, Alaska, USA
| | - Kate M Lillie
- Division of Organizational Development and Innovation, Research and Data Services Department, Southcentral Foundation, Anchorage, Alaska, USA
| | - Jaedon P Avey
- Division of Organizational Development and Innovation, Research and Data Services Department, Southcentral Foundation, Anchorage, Alaska, USA
| | - Rebecca Ziebell
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington, USA
| | - Gregory Simon
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington, USA
| |
Collapse
|
15
|
Kirtley OJ, van Mens K, Hoogendoorn M, Kapur N, de Beurs D. Translating promise into practice: a review of machine learning in suicide research and prevention. Lancet Psychiatry 2022; 9:243-252. [PMID: 35183281 DOI: 10.1016/s2215-0366(21)00254-6] [Citation(s) in RCA: 21] [Impact Index Per Article: 10.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/10/2021] [Revised: 07/02/2021] [Accepted: 07/07/2021] [Indexed: 02/06/2023]
Abstract
In ever more pressured health-care systems, technological solutions offering scalability of care and better resource targeting are appealing. Research on machine learning as a technique for identifying individuals at risk of suicidal ideation, suicide attempts, and death has grown rapidly. This research often places great emphasis on the promise of machine learning for preventing suicide, but overlooks the practical, clinical implementation issues that might preclude delivering on such a promise. In this Review, we synthesise the broad empirical and review literature on electronic health record-based machine learning in suicide research, and focus on matters of crucial importance for implementation of machine learning in clinical practice. The challenge of preventing statistically rare outcomes is well known; progress requires tackling data quality, transparency, and ethical issues. In the future, machine learning models might be explored as methods to enable targeting of interventions to specific individuals depending upon their level of need-ie, for precision medicine. Primarily, however, the promise of machine learning for suicide prevention is limited by the scarcity of high-quality scalable interventions available to individuals identified by machine learning as being at risk of suicide.
Collapse
Affiliation(s)
| | | | - Mark Hoogendoorn
- Department of Computer Science, Vrij Universiteit Amsterdam, Amsterdam, Netherlands
| | - Navneet Kapur
- Centre for Mental Health and Safety and Greater Manchester National Institute for Health Research Patient Safety Translational Research Centre, University of Manchester, Manchester, UK; Greater Manchester Mental Health NHS Foundation Trust, Manchester, UK
| | - Derek de Beurs
- Department of Epidemiology, Trimbos Institute, Utrecht, Netherlands
| |
Collapse
|
16
|
Lejeune A, Le Glaz A, Perron PA, Sebti J, Baca-Garcia E, Walter M, Lemey C, Berrouiguet S. Artificial intelligence and suicide prevention: a systematic review. Eur Psychiatry 2022; 65:1-22. [PMID: 35166203 PMCID: PMC8988272 DOI: 10.1192/j.eurpsy.2022.8] [Citation(s) in RCA: 15] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/14/2021] [Revised: 12/13/2021] [Accepted: 12/20/2021] [Indexed: 11/23/2022] Open
Abstract
Background Suicide is one of the main preventable causes of death. Artificial intelligence (AI) could improve methods for assessing suicide risk. The objective of this review is to assess the potential of AI in identifying patients who are at risk of attempting suicide. Methods A systematic review of the literature was conducted on PubMed, EMBASE, and SCOPUS databases, using relevant keywords. Results Thanks to this research, 296 studies were identified. Seventeen studies, published between 2014 and 2020 and matching inclusion criteria, were selected as relevant. Included studies aimed at predicting individual suicide risk or identifying at-risk individuals in a specific population. The AI performance was overall good, although variable across different algorithms and application settings. Conclusions AI appears to have a high potential for identifying patients at risk of suicide. The precise use of these algorithms in clinical situations, as well as the ethical issues it raises, remain to be clarified.
Collapse
Affiliation(s)
- Alban Lejeune
- URCI Mental Health Department, Brest Medical University Hospital, Brest, France
| | - Aziliz Le Glaz
- URCI Mental Health Department, Brest Medical University Hospital, Brest, France
| | | | - Johan Sebti
- Mental Health Department, French Polynesia Hospital, FFC3+H9G, Pirae, French Polynesia
| | | | - Michel Walter
- URCI Mental Health Department, Brest Medical University Hospital, Brest, France
- EA 7479 SPURBO, Université de Bretagne Occidentale, Brest, France
| | - Christophe Lemey
- URCI Mental Health Department, Brest Medical University Hospital, Brest, France
- EA 7479 SPURBO, Université de Bretagne Occidentale, Brest, France
- SPURBO, IMT Atlantique, Lab-STICC, UMR CNRS 6285, F-29238, Brest, France
| | - Sofian Berrouiguet
- URCI Mental Health Department, Brest Medical University Hospital, Brest, France
- LaTIM, INSERM, UMR 1101, Brest, France
| |
Collapse
|
17
|
Coley RY, Walker RL, Cruz M, Simon GE, Shortreed SM. Clinical risk prediction models and informative cluster size: Assessing the performance of a suicide risk prediction algorithm. Biom J 2021; 63:1375-1388. [PMID: 34031916 PMCID: PMC9134927 DOI: 10.1002/bimj.202000199] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2020] [Revised: 02/01/2021] [Accepted: 02/04/2021] [Indexed: 11/11/2022]
Abstract
Clinical visit data are clustered within people, which complicates prediction modeling. Cluster size is often informative because people receiving more care are less healthy and at higher risk of poor outcomes. We used data from seven health systems on 1,518,968 outpatient mental health visits from January 1, 2012 to June 30, 2015 to predict suicide attempt within 90 days. We evaluated true performance of prediction models using a prospective validation set of 4,286,495 visits from October 1, 2015 to September 30, 2017. We examined dividing clustered data on the person or visit level for model training and cross-validation and considered a within cluster resampling approach for model estimation. We evaluated optimism by comparing estimated performance from a left-out testing dataset to performance in the prospective dataset. We used two prediction methods, logistic regression with least absolute shrinkage and selection operator (LASSO) and random forest. The random forest model using a visit-level split for model training and testing was optimistic; it overestimated discrimination (area under the curve, AUC = 0.95 in testing versus 0.84 in prospective validation) and classification accuracy (sensitivity = 0.48 in testing versus 0.19 in prospective validation, 95th percentile cut-off). Logistic regression and random forest models using a person-level split performed well, accurately estimating prospective discrimination and classification: estimated AUCs ranged from 0.85 to 0.87 in testing versus 0.85 in prospective validation, and sensitivity ranged from 0.15 to 0.20 in testing versus 0.17 to 0.19 in prospective validation. Within cluster resampling did not improve performance. We recommend dividing clustered data on the person level, rather than visit level, to ensure strong performance in prospective use and accurate estimation of future performance at the time of model development.
Collapse
Affiliation(s)
- R. Yates Coley
- Kaiser Permanente Washington Health Research Institute (1730 Minor Ave, Suite 1600, Seattle, WA 98101, USA)
- Department of Biostatistics, University of Washington (1705 NE Pacific St, Seattle, WA 98195, USA)
| | - Rod L. Walker
- Kaiser Permanente Washington Health Research Institute (1730 Minor Ave, Suite 1600, Seattle, WA 98101, USA)
| | - Maricela Cruz
- Kaiser Permanente Washington Health Research Institute (1730 Minor Ave, Suite 1600, Seattle, WA 98101, USA)
| | - Gregory E. Simon
- Kaiser Permanente Washington Health Research Institute (1730 Minor Ave, Suite 1600, Seattle, WA 98101, USA)
| | - Susan M. Shortreed
- Kaiser Permanente Washington Health Research Institute (1730 Minor Ave, Suite 1600, Seattle, WA 98101, USA)
- Department of Biostatistics, University of Washington (1705 NE Pacific St, Seattle, WA 98195, USA)
| |
Collapse
|
18
|
Walker RL, Shortreed SM, Ziebell RA, Johnson E, Boggs JM, Lynch FL, Daida YG, Ahmedani BK, Rossom R, Coleman KJ, Simon GE. Evaluation of Electronic Health Record-Based Suicide Risk Prediction Models on Contemporary Data. Appl Clin Inform 2021; 12:778-787. [PMID: 34407559 PMCID: PMC8373461 DOI: 10.1055/s-0041-1733908] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2021] [Accepted: 07/01/2021] [Indexed: 10/20/2022] Open
Abstract
BACKGROUND Suicide risk prediction models have been developed by using information from patients' electronic health records (EHR), but the time elapsed between model development and health system implementation is often substantial. Temporal changes in health systems and EHR coding practices necessitate the evaluation of such models in more contemporary data. OBJECTIVES A set of published suicide risk prediction models developed by using EHR data from 2009 to 2015 across seven health systems reported c-statistics of 0.85 for suicide attempt and 0.83 to 0.86 for suicide death. Our objective was to evaluate these models' performance with contemporary data (2014-2017) from these systems. METHODS We evaluated performance using mental health visits (6,832,439 to mental health specialty providers and 3,987,078 to general medical providers) from 2014 to 2017 made by 1,799,765 patients aged 13+ across the health systems. No visits in our evaluation were used in the previous model development. Outcomes were suicide attempt (health system records) and suicide death (state death certificates) within 90 days following a visit. We assessed calibration and computed c-statistics with 95% confidence intervals (CI) and cut-point specific estimates of sensitivity, specificity, and positive/negative predictive value. RESULTS Models were well calibrated; 46% of suicide attempts and 35% of suicide deaths in the mental health specialty sample were preceded by a visit (within 90 days) with a risk score in the top 5%. In the general medical sample, 53% of attempts and 35% of deaths were preceded by such a visit. Among these two samples, respectively, c-statistics were 0.862 (95% CI: 0.860-0.864) and 0.864 (95% CI: 0.860-0.869) for suicide attempt, and 0.806 (95% CI: 0.790-0.822) and 0.804 (95% CI: 0.782-0.829) for suicide death. CONCLUSION Performance of the risk prediction models in this contemporary sample was similar to historical estimates for suicide attempt but modestly lower for suicide death. These published models can inform clinical practice and patient care today.
Collapse
Affiliation(s)
- Rod L. Walker
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington, United States
| | - Susan M. Shortreed
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington, United States
| | - Rebecca A. Ziebell
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington, United States
| | - Eric Johnson
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington, United States
| | - Jennifer M. Boggs
- Kaiser Permanente Colorado, Institute for Health Research, Aurora, Colorado, United States
| | - Frances L. Lynch
- Kaiser Permanente Northwest, Center for Health Research, Portland, Oregon, United States
| | - Yihe G. Daida
- Kaiser Permanente Hawaii, Center for Integrated Health Care Research, Honolulu, Hawaii, United States
| | - Brian K. Ahmedani
- Henry Ford Health System, Center for Health Policy & Health Services Research, Detroit, Michigan, United States
| | - Rebecca Rossom
- Department of Research, HealthPartners Institute, Minneapolis, Minnesota, United States
| | - Karen J. Coleman
- Department of Research and Evaluation, Kaiser Permanente Southern California, Pasadena, California, United States
| | - Gregory E. Simon
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington, United States
| |
Collapse
|
19
|
Simon GE, Matarazzo BB, Walsh CG, Smoller JW, Boudreaux ED, Yarborough BJH, Shortreed SM, Coley RY, Ahmedani BK, Doshi RP, Harris LI, Schoenbaum M. Reconciling Statistical and Clinicians' Predictions of Suicide Risk. Psychiatr Serv 2021; 72:555-562. [PMID: 33691491 DOI: 10.1176/appi.ps.202000214] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
Statistical models, including those based on electronic health records, can accurately identify patients at high risk for a suicide attempt or death, leading to implementation of risk prediction models for population-based suicide prevention in health systems. However, some have questioned whether statistical predictions can really inform clinical decisions. Appropriately reconciling statistical algorithms with traditional clinician assessment depends on whether predictions from these two methods are competing, complementary, or merely duplicative. In June 2019, the National Institute of Mental Health convened a meeting, "Identifying Research Priorities for Risk Algorithms Applications in Healthcare Settings to Improve Suicide Prevention." Here, participants of this meeting summarize key issues regarding the potential clinical application of suicide prediction models. The authors attempt to clarify the key conceptual and technical differences between traditional risk prediction by clinicians and predictions from statistical models, review the limited evidence regarding both the accuracy of and the concordance between these alternative methods of prediction, present a conceptual framework for understanding agreement and disagreement between statistical and clinician predictions, identify priorities for improving data regarding suicide risk, and propose priority questions for future research. Future suicide risk assessment will likely combine statistical prediction with traditional clinician assessment, but research is needed to determine the optimal combination of these two methods.
Collapse
Affiliation(s)
- Gregory E Simon
- Kaiser Permanente Washington Health Research Institute, Seattle (Simon, Shortreed, Coley); Department of Veterans Affairs Rocky Mountain Mental Illness Research, Education and Clinical Center, and Department of Psychiatry, University of Colorado School of Medicine, Aurora (Matarazzo); Department of Medicine and Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee (Walsh); Department of Psychiatry, Massachusetts General Hospital, Boston (Smoller); Department of Emergency Medicine and Department of Psychiatry, University of Massachusetts Medical School, Worcester (Boudreaux); Kaiser Permanente Northwest Center for Health Research, Portland, Oregon (Yarborough); Department of Biostatistics, University of Washington, Seattle (Shortreed, Coley); Center for Health Policy and Health Services Research, Henry Ford Health System, Detroit (Ahmedani); Department of Community Medicine and Healthcare, University of Connecticut, Farmington (Doshi); Shifa Consulting, Arlington, Virginia (Harris); Division of Services and Intervention Research, National Institute of Mental Health, Bethesda, Maryland (Schoenbaum)
| | - Bridget B Matarazzo
- Kaiser Permanente Washington Health Research Institute, Seattle (Simon, Shortreed, Coley); Department of Veterans Affairs Rocky Mountain Mental Illness Research, Education and Clinical Center, and Department of Psychiatry, University of Colorado School of Medicine, Aurora (Matarazzo); Department of Medicine and Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee (Walsh); Department of Psychiatry, Massachusetts General Hospital, Boston (Smoller); Department of Emergency Medicine and Department of Psychiatry, University of Massachusetts Medical School, Worcester (Boudreaux); Kaiser Permanente Northwest Center for Health Research, Portland, Oregon (Yarborough); Department of Biostatistics, University of Washington, Seattle (Shortreed, Coley); Center for Health Policy and Health Services Research, Henry Ford Health System, Detroit (Ahmedani); Department of Community Medicine and Healthcare, University of Connecticut, Farmington (Doshi); Shifa Consulting, Arlington, Virginia (Harris); Division of Services and Intervention Research, National Institute of Mental Health, Bethesda, Maryland (Schoenbaum)
| | - Colin G Walsh
- Kaiser Permanente Washington Health Research Institute, Seattle (Simon, Shortreed, Coley); Department of Veterans Affairs Rocky Mountain Mental Illness Research, Education and Clinical Center, and Department of Psychiatry, University of Colorado School of Medicine, Aurora (Matarazzo); Department of Medicine and Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee (Walsh); Department of Psychiatry, Massachusetts General Hospital, Boston (Smoller); Department of Emergency Medicine and Department of Psychiatry, University of Massachusetts Medical School, Worcester (Boudreaux); Kaiser Permanente Northwest Center for Health Research, Portland, Oregon (Yarborough); Department of Biostatistics, University of Washington, Seattle (Shortreed, Coley); Center for Health Policy and Health Services Research, Henry Ford Health System, Detroit (Ahmedani); Department of Community Medicine and Healthcare, University of Connecticut, Farmington (Doshi); Shifa Consulting, Arlington, Virginia (Harris); Division of Services and Intervention Research, National Institute of Mental Health, Bethesda, Maryland (Schoenbaum)
| | - Jordan W Smoller
- Kaiser Permanente Washington Health Research Institute, Seattle (Simon, Shortreed, Coley); Department of Veterans Affairs Rocky Mountain Mental Illness Research, Education and Clinical Center, and Department of Psychiatry, University of Colorado School of Medicine, Aurora (Matarazzo); Department of Medicine and Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee (Walsh); Department of Psychiatry, Massachusetts General Hospital, Boston (Smoller); Department of Emergency Medicine and Department of Psychiatry, University of Massachusetts Medical School, Worcester (Boudreaux); Kaiser Permanente Northwest Center for Health Research, Portland, Oregon (Yarborough); Department of Biostatistics, University of Washington, Seattle (Shortreed, Coley); Center for Health Policy and Health Services Research, Henry Ford Health System, Detroit (Ahmedani); Department of Community Medicine and Healthcare, University of Connecticut, Farmington (Doshi); Shifa Consulting, Arlington, Virginia (Harris); Division of Services and Intervention Research, National Institute of Mental Health, Bethesda, Maryland (Schoenbaum)
| | - Edwin D Boudreaux
- Kaiser Permanente Washington Health Research Institute, Seattle (Simon, Shortreed, Coley); Department of Veterans Affairs Rocky Mountain Mental Illness Research, Education and Clinical Center, and Department of Psychiatry, University of Colorado School of Medicine, Aurora (Matarazzo); Department of Medicine and Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee (Walsh); Department of Psychiatry, Massachusetts General Hospital, Boston (Smoller); Department of Emergency Medicine and Department of Psychiatry, University of Massachusetts Medical School, Worcester (Boudreaux); Kaiser Permanente Northwest Center for Health Research, Portland, Oregon (Yarborough); Department of Biostatistics, University of Washington, Seattle (Shortreed, Coley); Center for Health Policy and Health Services Research, Henry Ford Health System, Detroit (Ahmedani); Department of Community Medicine and Healthcare, University of Connecticut, Farmington (Doshi); Shifa Consulting, Arlington, Virginia (Harris); Division of Services and Intervention Research, National Institute of Mental Health, Bethesda, Maryland (Schoenbaum)
| | - Bobbi Jo H Yarborough
- Kaiser Permanente Washington Health Research Institute, Seattle (Simon, Shortreed, Coley); Department of Veterans Affairs Rocky Mountain Mental Illness Research, Education and Clinical Center, and Department of Psychiatry, University of Colorado School of Medicine, Aurora (Matarazzo); Department of Medicine and Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee (Walsh); Department of Psychiatry, Massachusetts General Hospital, Boston (Smoller); Department of Emergency Medicine and Department of Psychiatry, University of Massachusetts Medical School, Worcester (Boudreaux); Kaiser Permanente Northwest Center for Health Research, Portland, Oregon (Yarborough); Department of Biostatistics, University of Washington, Seattle (Shortreed, Coley); Center for Health Policy and Health Services Research, Henry Ford Health System, Detroit (Ahmedani); Department of Community Medicine and Healthcare, University of Connecticut, Farmington (Doshi); Shifa Consulting, Arlington, Virginia (Harris); Division of Services and Intervention Research, National Institute of Mental Health, Bethesda, Maryland (Schoenbaum)
| | - Susan M Shortreed
- Kaiser Permanente Washington Health Research Institute, Seattle (Simon, Shortreed, Coley); Department of Veterans Affairs Rocky Mountain Mental Illness Research, Education and Clinical Center, and Department of Psychiatry, University of Colorado School of Medicine, Aurora (Matarazzo); Department of Medicine and Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee (Walsh); Department of Psychiatry, Massachusetts General Hospital, Boston (Smoller); Department of Emergency Medicine and Department of Psychiatry, University of Massachusetts Medical School, Worcester (Boudreaux); Kaiser Permanente Northwest Center for Health Research, Portland, Oregon (Yarborough); Department of Biostatistics, University of Washington, Seattle (Shortreed, Coley); Center for Health Policy and Health Services Research, Henry Ford Health System, Detroit (Ahmedani); Department of Community Medicine and Healthcare, University of Connecticut, Farmington (Doshi); Shifa Consulting, Arlington, Virginia (Harris); Division of Services and Intervention Research, National Institute of Mental Health, Bethesda, Maryland (Schoenbaum)
| | - R Yates Coley
- Kaiser Permanente Washington Health Research Institute, Seattle (Simon, Shortreed, Coley); Department of Veterans Affairs Rocky Mountain Mental Illness Research, Education and Clinical Center, and Department of Psychiatry, University of Colorado School of Medicine, Aurora (Matarazzo); Department of Medicine and Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee (Walsh); Department of Psychiatry, Massachusetts General Hospital, Boston (Smoller); Department of Emergency Medicine and Department of Psychiatry, University of Massachusetts Medical School, Worcester (Boudreaux); Kaiser Permanente Northwest Center for Health Research, Portland, Oregon (Yarborough); Department of Biostatistics, University of Washington, Seattle (Shortreed, Coley); Center for Health Policy and Health Services Research, Henry Ford Health System, Detroit (Ahmedani); Department of Community Medicine and Healthcare, University of Connecticut, Farmington (Doshi); Shifa Consulting, Arlington, Virginia (Harris); Division of Services and Intervention Research, National Institute of Mental Health, Bethesda, Maryland (Schoenbaum)
| | - Brian K Ahmedani
- Kaiser Permanente Washington Health Research Institute, Seattle (Simon, Shortreed, Coley); Department of Veterans Affairs Rocky Mountain Mental Illness Research, Education and Clinical Center, and Department of Psychiatry, University of Colorado School of Medicine, Aurora (Matarazzo); Department of Medicine and Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee (Walsh); Department of Psychiatry, Massachusetts General Hospital, Boston (Smoller); Department of Emergency Medicine and Department of Psychiatry, University of Massachusetts Medical School, Worcester (Boudreaux); Kaiser Permanente Northwest Center for Health Research, Portland, Oregon (Yarborough); Department of Biostatistics, University of Washington, Seattle (Shortreed, Coley); Center for Health Policy and Health Services Research, Henry Ford Health System, Detroit (Ahmedani); Department of Community Medicine and Healthcare, University of Connecticut, Farmington (Doshi); Shifa Consulting, Arlington, Virginia (Harris); Division of Services and Intervention Research, National Institute of Mental Health, Bethesda, Maryland (Schoenbaum)
| | - Riddhi P Doshi
- Kaiser Permanente Washington Health Research Institute, Seattle (Simon, Shortreed, Coley); Department of Veterans Affairs Rocky Mountain Mental Illness Research, Education and Clinical Center, and Department of Psychiatry, University of Colorado School of Medicine, Aurora (Matarazzo); Department of Medicine and Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee (Walsh); Department of Psychiatry, Massachusetts General Hospital, Boston (Smoller); Department of Emergency Medicine and Department of Psychiatry, University of Massachusetts Medical School, Worcester (Boudreaux); Kaiser Permanente Northwest Center for Health Research, Portland, Oregon (Yarborough); Department of Biostatistics, University of Washington, Seattle (Shortreed, Coley); Center for Health Policy and Health Services Research, Henry Ford Health System, Detroit (Ahmedani); Department of Community Medicine and Healthcare, University of Connecticut, Farmington (Doshi); Shifa Consulting, Arlington, Virginia (Harris); Division of Services and Intervention Research, National Institute of Mental Health, Bethesda, Maryland (Schoenbaum)
| | - Leah I Harris
- Kaiser Permanente Washington Health Research Institute, Seattle (Simon, Shortreed, Coley); Department of Veterans Affairs Rocky Mountain Mental Illness Research, Education and Clinical Center, and Department of Psychiatry, University of Colorado School of Medicine, Aurora (Matarazzo); Department of Medicine and Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee (Walsh); Department of Psychiatry, Massachusetts General Hospital, Boston (Smoller); Department of Emergency Medicine and Department of Psychiatry, University of Massachusetts Medical School, Worcester (Boudreaux); Kaiser Permanente Northwest Center for Health Research, Portland, Oregon (Yarborough); Department of Biostatistics, University of Washington, Seattle (Shortreed, Coley); Center for Health Policy and Health Services Research, Henry Ford Health System, Detroit (Ahmedani); Department of Community Medicine and Healthcare, University of Connecticut, Farmington (Doshi); Shifa Consulting, Arlington, Virginia (Harris); Division of Services and Intervention Research, National Institute of Mental Health, Bethesda, Maryland (Schoenbaum)
| | - Michael Schoenbaum
- Kaiser Permanente Washington Health Research Institute, Seattle (Simon, Shortreed, Coley); Department of Veterans Affairs Rocky Mountain Mental Illness Research, Education and Clinical Center, and Department of Psychiatry, University of Colorado School of Medicine, Aurora (Matarazzo); Department of Medicine and Department of Biomedical Informatics, Vanderbilt University, Nashville, Tennessee (Walsh); Department of Psychiatry, Massachusetts General Hospital, Boston (Smoller); Department of Emergency Medicine and Department of Psychiatry, University of Massachusetts Medical School, Worcester (Boudreaux); Kaiser Permanente Northwest Center for Health Research, Portland, Oregon (Yarborough); Department of Biostatistics, University of Washington, Seattle (Shortreed, Coley); Center for Health Policy and Health Services Research, Henry Ford Health System, Detroit (Ahmedani); Department of Community Medicine and Healthcare, University of Connecticut, Farmington (Doshi); Shifa Consulting, Arlington, Virginia (Harris); Division of Services and Intervention Research, National Institute of Mental Health, Bethesda, Maryland (Schoenbaum)
| |
Collapse
|
20
|
Bobroske K, Larish C, Cattrell A, Bjarnadóttir MV, Huan L. The bird's-eye view: A data-driven approach to understanding patient journeys from claims data. J Am Med Inform Assoc 2021; 27:1037-1045. [PMID: 32521006 DOI: 10.1093/jamia/ocaa052] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2020] [Revised: 03/31/2020] [Accepted: 04/09/2020] [Indexed: 12/29/2022] Open
Abstract
OBJECTIVE In preference-sensitive conditions such as back pain, there can be high levels of variability in the trajectory of patient care. We sought to develop a methodology that extracts a realistic and comprehensive understanding of the patient journey using medical and pharmaceutical insurance claims data. MATERIALS AND METHODS We processed a sample of 10 000 patient episodes (comprised of 113 215 back pain-related claims) into strings of characters, where each letter corresponds to a distinct encounter with the healthcare system. We customized the Levenshtein edit distance algorithm to evaluate the level of similarity between each pair of episodes based on both their content (types of events) and ordering (sequence of events). We then used clustering to extract the main variations of the patient journey. RESULTS The algorithm resulted in 12 comprehensive and clinically distinct patterns (clusters) of patient journeys that represent the main ways patients are diagnosed and treated for back pain. We further characterized demographic and utilization metrics for each cluster and observed clear differentiation between the clusters in terms of both clinical content and patient characteristics. DISCUSSION Despite being a complex and often noisy data source, administrative claims provide a unique longitudinal overview of patient care across multiple service providers and locations. This methodology leverages claims to capture a data-driven understanding of how patients traverse the healthcare system. CONCLUSIONS When tailored to various conditions and patient settings, this methodology can provide accurate overviews of patient journeys and facilitate a shift toward high-quality practice patterns.
Collapse
Affiliation(s)
- Katherine Bobroske
- Cambridge Centre for Health and Leadership Enterprise, University of Cambridge, Cambridge, United Kingdom
| | - Christine Larish
- Research and Development, Evolent Health, Arlington, Virginia, USA
| | - Anita Cattrell
- Research and Development, Evolent Health, Arlington, Virginia, USA
| | | | - Lawrence Huan
- Cambridge Centre for Health and Leadership Enterprise, University of Cambridge, Cambridge, United Kingdom
| |
Collapse
|
21
|
Wilkinson K, Sheets L, Fitch D, Popejoy L. Systematic review of approaches to use of neighborhood-level risk factors with clinical data to predict clinical risk and recommend interventions. J Biomed Inform 2021; 116:103713. [PMID: 33610880 DOI: 10.1016/j.jbi.2021.103713] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2020] [Revised: 02/06/2021] [Accepted: 02/10/2021] [Indexed: 11/24/2022]
Abstract
BACKGROUND Despite a large body of literature investigating how the environment influences health outcomes, most published work to date includes only a limited subset of the rich clinical and environmental data that is available and does not address how these data might best be used to predict clinical risk or expected impact of clinical interventions. OBJECTIVE Identify existing approaches to inclusion of a broad set of neighborhood-level risk factors with clinical data to predict clinical risk and recommend interventions. METHODS A systematic review of scientific literature published and indexed in PubMed, Web of Science, Association of Computing Machinery (ACM) and SCOPUS from 2010 through October 2020 was performed. To be included, articles had to include search terms related to Electronic Health Record (EHR) data Neighborhood-Level Risk Factors (NLRFs), and Machine Learning (ML) Methods. Citations of relevant articles were also reviewed for additional articles for inclusion. Articles were reviewed and coded by two independent reviewers to capture key information including data sources, linkage of EHR to NRLFs, methods, and results. Articles were assessed for quality using a modified Quality Assessment Tool for Systematic Reviews of Observational Studies (QATSO). RESULTS A total of 334 articles were identified for abstract review. 36 articles were identified for full review with 19 articles included in the final analysis. All but two of the articles included socio-demographic data derived from the U.S. Census and we found great variability in sources of NLRFs beyond the Census. The majority or the articles (14 of 19) included broader clinical (e.g. medications, labs and co-morbidities) and demographic information about the individual from the EHR in addition to the clinical outcome variable. Half of the articles (10) had a stated goal to predict the outcome(s) of interest. While results of the studies reinforced the correlative association of NLRFs to clinical outcomes, only one article found that adding NLRFs into a model with other data added predictive power with the remainder concluding either that NLRFs were of mixed value depending on the model and outcome or that NLRFs added no predictive power over other data in the model. Only one article scored high on the quality assessment with 13 scoring moderate and 4 scoring low. CONCLUSIONS In spite of growing interest in combining NLRFs with EHR data for clinical prediction, we found limited evidence that NLRFs improve predictive power in clinical risk models. We found these data and methods are being used in four ways. First, early approaches to include broad NLRFs to predict clinical risk primarily focused on dimension reduction for feature selection or as a data preparation step to input into regression analysis. Second, more recent work incorporates NLRFs into more advanced predictive models, such as Neural Networks, Random Forest, and Penalized Lasso to predict clinical outcomes or predict value of interventions. Third, studies that test how inclusion of NLRFs predict clinical risk have shown mixed results regarding the value of these data over EHR or claims data alone and this review surfaced evidence of potential quality challenges and biases inherent to this approach. Finally, NLRFs were used with unsupervised learning to identify underlying patterns in patient populations to recommend targeted interventions. Further access to computable, high quality data is needed along with careful study design, including sub-group analysis, to better determine how these data and methods can be used to support decision making in a clinical setting.
Collapse
Affiliation(s)
- Katie Wilkinson
- Institute for Data Science and Informatics, University of Missouri, Columbia, MO 65212, United States; School of Medicine, University of Missouri, Columbia, MO 65212, United States.
| | - Lincoln Sheets
- Institute for Data Science and Informatics, University of Missouri, Columbia, MO 65212, United States; School of Medicine, University of Missouri, Columbia, MO 65212, United States
| | - Dale Fitch
- Institute for Data Science and Informatics, University of Missouri, Columbia, MO 65212, United States; School of Social Work, University of Missouri, Columbia, MO 65212, United States
| | - Lori Popejoy
- Institute for Data Science and Informatics, University of Missouri, Columbia, MO 65212, United States; School of Nursing, University of Missouri, Columbia, MO 65212, United States
| |
Collapse
|
22
|
Cusick M, Adekkanattu P, Campion TR, Sholle ET, Myers A, Banerjee S, Alexopoulos G, Wang Y, Pathak J. Using weak supervision and deep learning to classify clinical notes for identification of current suicidal ideation. J Psychiatr Res 2021; 136:95-102. [PMID: 33581461 DOI: 10.1016/j.jpsychires.2021.01.052] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/12/2020] [Revised: 01/17/2021] [Accepted: 01/29/2021] [Indexed: 10/22/2022]
Abstract
Mental health concerns, such as suicidal thoughts, are frequently documented by providers in clinical notes, as opposed to structured coded data. In this study, we evaluated weakly supervised methods for detecting "current" suicidal ideation from unstructured clinical notes in electronic health record (EHR) systems. Weakly supervised machine learning methods leverage imperfect labels for training, alleviating the burden of creating a large manually annotated dataset. After identifying a cohort of 600 patients at risk for suicidal ideation, we used a rule-based natural language processing approach (NLP) approach to label the training and validation notes (n = 17,978). Using this large corpus of clinical notes, we trained several statistical machine learning models-logistic classifier, support vector machines (SVM), Naive Bayes classifier-and one deep learning model, namely a text classification convolutional neural network (CNN), to be evaluated on a manually-reviewed test set (n = 837). The CNN model outperformed all other methods, achieving an overall accuracy of 94% and a F1-score of 0.82 on documents with "current" suicidal ideation. This algorithm correctly identified an additional 42 encounters and 9 patients indicative of suicidal ideation but missing a structured diagnosis code. When applied to a random subset of 5,000 clinical notes, the algorithm classified 0.46% (n = 23) for "current" suicidal ideation, of which 87% were truly indicative via manual review. Implementation of this approach for large-scale document screening may play an important role in point-of-care clinical information systems for targeted suicide prevention interventions and improve research on the pathways from ideation to attempt.
Collapse
Affiliation(s)
- Marika Cusick
- Department of Information and Technology Services, Weill Cornell Medicine, New York, USA; Department Population Health Sciences, Weill Cornell Medicine, New York, USA.
| | - Prakash Adekkanattu
- Department of Information and Technology Services, Weill Cornell Medicine, New York, USA.
| | - Thomas R Campion
- Department of Information and Technology Services, Weill Cornell Medicine, New York, USA; Department Population Health Sciences, Weill Cornell Medicine, New York, USA.
| | - Evan T Sholle
- Department of Information and Technology Services, Weill Cornell Medicine, New York, USA.
| | - Annie Myers
- Department Population Health Sciences, Weill Cornell Medicine, New York, USA.
| | - Samprit Banerjee
- Department Population Health Sciences, Weill Cornell Medicine, New York, USA.
| | | | - Yanshan Wang
- Division of Digital Health Sciences, Mayo Clinic, MN, USA; Department of Health Sciences Research, Mayo Clinic, MN, USA.
| | - Jyotishman Pathak
- Department Population Health Sciences, Weill Cornell Medicine, New York, USA; Department of Psychiatry, Weill Cornell Medicine, New York, USA.
| |
Collapse
|
23
|
Lenert L. The science of informatics and predictive analytics. J Am Med Inform Assoc 2019; 26:1425-1426. [PMID: 31730703 DOI: 10.1093/jamia/ocz202] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Affiliation(s)
- Leslie Lenert
- Institution of Medical University of South Carolina, Charleston, SC
| |
Collapse
|