Measurement of predictive validity in violence risk assessment studies: a second-order systematic review

The predictive validity of the V-RISK-10 and BVC among involuntarily admitted patients

Although endangerment towards others is a criterion for an involuntary admission in many countries, research on risk assessment of endangerment among involuntarily admitted individuals is limited. In this retrospective case-control study, we calculated scores for a German-translated version of the Violence Risk Screening-10 (V-RISK-10) and the Brøset Violence Checklist (BVC) in a sample of 111 people undergoing an involuntary admission in Reinickendorf, Berlin. Outcomes were violence, coercive measures, and readmission. In line with our hypotheses, the BVC demonstrated stronger predictive validities for short-term, and V-RISK-10 for long-term events. There was an incremental validity for both instruments for restraint 24 hours after admission and any violence until discharge. These findings support the evidence that structured risk assessment instruments may be useful for individuals undergoing an involuntary admission. Ethical considerations about screening procedures are discussed.

Evidence-based sentencing and scientific evidence

Evidence-based sentencing (EBS) is a new name for an aspiration that has deep roots in criminal law: to apply the sentence most appropriate to each offender's risk of reoffending, in order to reduce that risk as far as possible. This modern version of the traditional sentencing goals of rehabilitation and incapacitation fits into the broader approach of so-called "evidence-based public policy." It takes the view that the best existing evidence for reducing reoffending are modern structured risk assessment tools and claims to be able to achieve several goals at once: reducing reoffending, maintaining high levels of public safety, making more efficient use of public resources, and moving criminal policy away from ideological battles by basing it on the objective knowledge provided by the best available scientific evidence. However, despite the success of this approach in recent years, it is not clear to what extent it succeeds in correctly assessing the risk of individual offenders, nor whether it achieves its intended effect of reducing recidivism. This paper aims to critically examine these two issues: the quality of the scientific evidence on which EBS is based, and the available data on the extent to which it achieves (or does not achieve) its intended goals.

Violence risk assessment instruments in forensic psychiatric populations: a systematic review and meta-analysis

BACKGROUND

Although structured tools have been widely used to predict violence risk in specialist mental health settings, there is uncertainty about the extent and quality of evidence of their predictive performance. We aimed to systematically review the predictive performance of tools used to assess violence risk in forensic mental health, where they are routinely administered.

METHODS

In our systematic review and meta-analysis, we followed PRISMA guidelines and searched four databases (PsycINFO, Embase, Medline, and Global Health) from database inception to Nov 1, 2022, to identify studies examining the predictive performance of risk assessment tools in people discharged from forensic (secure) mental health hospitals. Systematic and narrative reviews were excluded from the review. Performance measures and descriptive statistics were extracted from published reports. A quality assessment was performed for each study using the Prediction Model Risk of Bias Assessment Tool. Meta-analysis was conducted on the performance of instruments that were independently externally validated with a sample size greater than 100. The study was registered with PROSPERO, CRD42022304716.

FINDINGS

We conducted a systematic review of 50 eligible publications, assessing the predictive performance of 36 tools, providing data for 10 460 participants (88% men, 12% women; median age [from 47 studies] was 35 years, IQR 33-38) from 12 different countries. Post-discharge interpersonal violence and crime was most often measured by new criminal offences or recidivism (47 [94%] of 50 studies); only three studies used informant or self-report data on physical aggression or violent behaviour. Overall, the predictive performance of risk assessment tools was mixed. Most studies reported one discrimination metric, the area under the receiver operating characteristic curve (AUC); other key performance measures such as calibration, sensitivity, and specificity were not presented. Most studies had a high risk of bias (49 [98%] of 50), partly due to poor analytical approaches. A meta-analysis was conducted for violent recidivism on 29 independent external validations from 19 studies with at least 100 patients. Pooled AUCs for predicting violent outcomes ranged from 0·72 (0·65-0·79; I2=0%) for H10, to 0·69 for the Historical Clinical Risk Management-20 version 2 (95% CI 0·65-0·72; I2=0%) and Violence Risk Appraisal Guide (0·63-0·75; I2=0%), to 0·64 for the Static-99 (0·53-0·73; I2=45%).

INTERPRETATION

Current violence risk assessment tools in forensic mental health have mixed evidence of predictive performance. Forensic mental health services should review their use of current risk assessment tools and consider implementing those with higher-quality evidence in support.

FUNDING

Wellcome Trust.

Validation of the YLS/CMI on an Australian Juvenile Offending Population

There is a wealth of research that shows juvenile justice systems that utilize structured and validated assessment tools, such as the YLS/CMI, are far more effective at reducing rates of recidivism than those who do not. In line with this research, the Department of Justice (DoJ) in Western Australia adopted the YLS/CMI as the standard risk assessment tool for evaluating the criminogenic risk and needs of youth entering the justice system. While there is evidence supporting the utility of the YLS/CMI in predicting recidivism, there is little research demonstrating its effectiveness in Australian juvenile populations and no such research in a West Australian population. There is also a lack of research on the utility of the tool with young Indigenous offenders, which is particularly concerning given the significant overrepresentation of Indigenous people in the Australian criminal justice system. Our study was the first to examine the reliability and predictive validity of the YLS/CMI on a sample of West Australian juvenile offenders. In this paper, we present the results of two analyses. The first examines the properties of the YLS/CMI in a cohort of 4,653 juvenile offenders in Western Australia, including factor structure, internal consistency, and differences between male and female youth and between Indigenous and non-Indigenous youth. Consistent with our hypotheses, the tool demonstrated excellent internal consistency (α = .91) and cross-validation analyses identified significant differences between groups on total YLS scores and risk domains. The second analysis examines the predictive validity of the YLS/CMI in a subsample of 921 youth with a minimum follow-up period of 2 years. The overall recidivism rate was 74.8% and there were differences in scores and recidivism rates for Indigenous compared to non-Indigenous youth, although the predictive accuracies in terms of AUC were similar (c. = 0.65 and 0.66, respectively).

Calibrating violence risk assessments for uncertainty

Psychiatrists and other mental health clinicians are often tasked with assessing patients' risk of violence. Approaches to this vary and include both unstructured (based on individual clinicians' judgement) and structured methods (based on formalised scoring and algorithms with varying scope for clinicians' judgement). The end result is usually a categorisation of risk, which may, in turn, reference a probability estimate of violence over a certain time period. Research over recent decades has made considerable improvements in refining structured approaches and categorising patients' risk classifications at a group level. The ability, however, to apply these findings clinically to predict the outcomes of individual patients remains contested. In this article, we review methods of assessing violence risk and empirical findings on their predictive validity. We note, in particular, limitations in calibration (accuracy at predicting absolute risk) as distinct from discrimination (accuracy at separating patients by outcome). We also consider clinical applications of these findings, including challenges applying statistics to individual patients, and broader conceptual issues in distinguishing risk and uncertainty. Based on this, we argue that there remain significant limits to assessing violence risk for individuals and that this requires careful consideration in clinical and legal contexts.

The predictive performance of criminal risk assessment tools used at sentencing: Systematic review of validation studies

Although risk assessment tools have been widely used to inform sentencing decisions, there is uncertainty about the extent and quality of evidence of their predictive performance when validated in new samples. Following PRISMA guidelines, we conducted a systematic review of validation studies of 11 commonly used risk assessment tools for sentencing. We identified 36 studies with 597,665 participants, among which were 27 independent validation studies with 177,711 individuals. Overall, the predictive performance of the included risk assessment tools was mixed, and ranged from poor to moderate. Tool performance was typically overestimated in studies with smaller sample sizes or studies in which tool developers were co-authors. Most studies only reported area under the curve (AUC), which ranged from 0.57 to 0.75 in independent studies with more than 500 participants. The majority did not report key performance measures, such as calibration and rates of false positives and negatives. In addition, most validation studies had a high risk of bias, partly due to inappropriate analytical approach used. We conclude that the research priority is for future investigations to address the key methodological shortcomings identified in this review, and policy makers should enable this research. More sufficiently powered independent validation studies are necessary.

Criminogenic risk assessment: A meta-review and critical analysis

A vast body of research underlies the ascendancy of criminogenic risk assessment, which was developed to predict recidivism. It is unclear, however, whether the empirical evidence supports its expansion across the criminal legal system. This meta-review thus attempts to answer the following questions: 1) How well does criminogenic risk assessment differentiate people who are at high risk of recidivism from those at low risk of recidivism? 2) How well do researchers' conclusions about (1) match the empirical evidence? 3) Does the empirical evidence support the theory, policy, and practice recommendations that researchers make based on their conclusions? A systematic literature search identified 39 meta-analyses and systematic reviews that met inclusion criteria. Findings from these meta-analyses and systematic reviews are summarized and synthesized, and their interpretations are critically assessed. We find that criminogenic risk assessment's predictive performance is based on inappropriate statistics, and that conclusions about the evidence are inconsistent and often overstated. Three thematic areas of inferential overreach are identified: contestable inferences from criminalization to criminality, from prediction to explanation, and from prediction to intervention. We conclude by exploring possible reasons for the mismatch between proponents' conclusions and the evidence, and discuss implications for policy and practice.

A machine learning approach for predicting suicidal thoughts and behaviours among college students

Suicidal thoughts and behaviours are prevalent among college students. Yet little is known about screening tools to identify students at higher risk. We aimed to develop a risk algorithm to identify the main predictors of suicidal thoughts and behaviours among college students within one-year of baseline assessment. We used data collected in 2013–2019 from the French i-Share cohort, a longitudinal population-based study including 5066 volunteer students. To predict suicidal thoughts and behaviours at follow-up, we used random forests models with 70 potential predictors measured at baseline, including sociodemographic and familial characteristics, mental health and substance use. Model performance was measured using the area under the receiver operating curve (AUC), sensitivity, and positive predictive value. At follow-up, 17.4% of girls and 16.8% of boys reported suicidal thoughts and behaviours. The models achieved good predictive performance: AUC, 0.8; sensitivity, 79% for girls, 81% for boys; and positive predictive value, 40% for girls and 36% for boys. Among the 70 potential predictors, four showed the highest predictive power: 12-month suicidal thoughts, trait anxiety, depression symptoms, and self-esteem. We identified a parsimonious set of mental health indicators that accurately predicted one-year suicidal thoughts and behaviours in a community sample of college students.

Machine Learning Assessment of Early Life Factors Predicting Suicide Attempt in Adolescence or Young Adulthood

IMPORTANCE

Although longitudinal studies have reported associations between early life factors (ie, in-utero/perinatal/infancy) and long-term suicidal behavior, they have concentrated on 1 or few selected factors, and established associations, but did not investigate if early-life factors predict suicidal behavior.

OBJECTIVE

To identify and evaluate the ability of early-life factors to predict suicide attempt in adolescents and young adults from the general population.

DESIGN, SETTING, AND PARTICIPANTS

This prognostic study used data from the Québec Longitudinal Study of Child Development, a population-based longitudinal study from Québec province, Canada. Participants were followed-up from birth to age 20 years. Random forest classification algorithms were developed to predict suicide attempt. To avoid overfitting, prediction performance indices were assessed across 50 randomly split subsamples, and then the mean was calculated. Data were analyzed from November 2019 to June 2020.

EXPOSURES

Factors considered in the analysis included 150 variables, spanning virtually all early life domains, including pregnancy and birth information; child, parents, and neighborhood characteristics; parenting and family functioning; parents' mental health; and child temperament, as assessed by mothers, fathers, and hospital birth records.

MAIN OUTCOMES AND MEASURES

The main outcome was self-reported suicide attempt by age 20 years.

RESULTS

Among 1623 included youths aged 20 years, 845 (52.1%) were female and 778 (47.9%) were male. Models show moderate prediction performance. The areas under the curve for the prediction of suicide attempt were 0.72 (95% CI, 0.71-0.73) for females and 0.62 (95% CI, 0.60-0.62) for males. The models showed low sensitivity (females, 0.50; males, 0.32), moderate positive predictive values (females, 0.60; males, 0.62), and good specificity (females, 0.76; males, 0.82) and negative predicted values (females, 0.75; males, 0.71). The most important factors contributing to the prediction included socioeconomic and demographic characteristics of the family (eg, mother and father education and age, socioeconomic status, neighborhood characteristics), parents' psychological state (specifically parents' antisocial behaviors) and parenting practices. Birth-related variables also contributed to the prediction of suicidal behavior (eg, prematurity). Sex differences were also identified, with family-related socioeconomic and demographic characteristics being the top factors for females and parents' antisocial behavior being the top factor for males.

CONCLUSIONS AND RELEVANCE

These findings suggest that early life factors contributed modestly to the prediction of suicidal behavior in adolescence and young adulthood. Although these factors may inform the understanding of the etiological processes of suicide, their utility in the long-term prediction of suicide attempt was limited.

Persistence and Potential Lethality in Intimate Partner Violence: Evaluating the Concurrent and Predictive Validity of a Dual Risk Assessment Protocol

This study seeks to determine the concurrent and predictive validity of a dual risk assessment protocol. It combines the risk of persistence in intimate partner violence (IPV) measured via the Domestic Violence Screening Instrument-Revised (DVSI-R) with supplemental items from the Danger Risk Assessment (DRA) bearing on the risk of potential lethality. We further test whether this assessment protocol reproduces disparities by race and ethnicity found in the larger population. Using a sample of 4,665 IPV male defendants with a female victim, analyses support both types of criterion validity. The DRA risk score is associated with felony charges, incarceration at the initial arrest, and the frequency of subsequent dangerous behavior. Results also suggest minimal predictive bias or disparate impact by race and ethnicity. Incorporating supplemental items bearing on potential lethality risk adds important information concerning the risk management strategies of those involved in IPV.

The Validity and Reliability of Available Intimate Partner Homicide and Reassault Risk Assessment Tools: A Systematic Review

At least one in seven homicides around the world is perpetrated by intimate partners. The danger of intimate partner homicide (IPH) associated with intimate partner violence (IPV) has led to the development of numerous IPV reassault and IPH risk assessment tools. Using 18 electronic databases and research repositories, we conducted a systematic review of IPH or IPV reassault risk assessment instruments. After review, 43 studies reported in 42 articles met inclusion criteria. We systematically extracted, analyzed, and synthesized data on tools studied, sample details, data collection location, study design, analysis methods, validity, reliability, and feasibility of use. Findings indicate that researchers in eight countries have tested 18 distinct IPH or IPV reassault risk assessment tools. The tools are designed for various professionals including law enforcement, first responders, and social workers. Twenty-six studies focused on assessing the risk of male perpetrators, although eight included female perpetrators. Eighteen studies tested tools with people in mixed-sex relationships, though many studies did not explicitly report the gender of both the perpetrators and victims/survivors. The majority of studies were administered or coded by researchers rather than administered in real-world settings. Reliable and valid instruments that accurately and feasibly assess the risk of IPH and IPV reassault in community settings are necessary for improving public safety and reducing violent deaths. Although researchers have developed several instruments assessing different risk factors, systematic research on the feasibility of using these instruments in practice settings is lacking.

Therapeutic Risk Management for Violence: Augmenting Clinical Risk Assessment With Structured Instruments

Violence risk assessment is a requisite component of mental health treatment. Adhering to standards of care and ethical and legal requirements necessitates a cogent process for conducting, and then documenting, other-directed violence risk screening, assessment, and management. In this 5-part series, we describe a model for achieving therapeutic risk management of the potentially violent patient, with essential elements involving: clinical interview augmented by structured screening or assessment tools; risk stratification in terms of temporality and severity; chain analysis to intervene on the functions of violent ideation and behavior; and a personalized safety plan to mitigate/manage risk. This second column in the series describes the advantages of, and offers suggestions for, incorporating structured tools into violence risk assessment.

Assessing Risk Among Correctional Community Probation Populations: Predicting Reoffense With Mobile Neurocognitive Assessment Software

We seek to address current limitations of forensic risk assessments by introducing the first mobile, self-scoring, risk assessment software that relies on neurocognitive testing to predict reoffense. This assessment, run entirely on a tablet, measures decision-making via a suite of neurocognitive tests in less than 30 minutes. The software measures several cognitive and decision-making traits of the user, including impulsivity, empathy, aggression, and several other traits linked to reoffending. Our analysis measured whether this assessment successfully predicted recidivism by testing probationers in a large urban city (Houston, TX, United States) from 2017 to 2019. To determine predictive validity, we used machine learning to yield cross-validated receiver–operator characteristics. Results gave a recidivism prediction value of 0.70, making it comparable to commonly used risk assessments. This novel approach diverges from traditional self-reporting, interview-based, and criminal-records-based approaches, and can also add a protective layer against bias, while strengthening model accuracy in predicting reoffense. In addition, subjectivity is eliminated and time-consuming administrative efforts are reduced. With continued data collection, this approach opens the possibility of identifying different levels of recidivism risk, by crime type, for any age, or gender, and seeks to steer individuals appropriately toward rehabilitative programs. Suggestions for future research directions are provided.

Violence risk-assessment screening tools for acute care mental health settings: Literature review

BACKGROUND

Violence is a large concern for mental health professionals: 90% of physicians and nurses working in mental health areas have been subject to violence from patients. Approximately 80% of violent acts from patients are directed toward nurses.

OBJECTIVE

The purpose of this integrative literature review was to identify violence risk-assessment screening tools that could be used in acute care mental health settings.

DESIGN

The Stetler model of evidence-based practice guided the literature search, in which 8 violence risk-assessment tools were identified, 4 of which were used for further examination.

RESULTS

The Brøset Violence Checklist and Violence Risk Screening-10 provided the best assessment for violence in the acute care mental health setting.

CONCLUSIONS

Using a violence risk assessment screening tool helps identify patients at risk for violence allowing for quick intervention to prevent violent episodes.

The Operational Development and Empirical Testing of the Terrorist Radicalization Assessment Protocol (TRAP-18)

The Terrorist Radicalization Assessment Protocol (TRAP-18) is a structured professional judgment instrument for threat assessment of the individual terrorist. It is a rationally derived theoretical model comprising eight proximal warning behaviors and 10 distal characteristics. Empirical research on the TRAP-18 is reviewed, including both nomothetic and idiographic studies of individual terrorists in both the United States and Europe. Mean interrater reliability is 0.895 (Cohen's kappa), ranging from 0.69 to 1.0. Evidence of criterion validity has been demonstrated, including usefulness of the instrument across various extremist ideologies (jihadism, ethnic nationalism, and single-issue), and its ability to discriminate between thwarted and successful attackers. The instrument appears to advance the domain recommendations of Monahan (2012, 2016) for the risk assessment of the individual terrorist. The TRAP-18 is further discussed as a threat assessment instrument for mental health clinicians. The limitations of the current research provide direction for further studies to assess its reliability and construct, discriminant, and predictive validity.

Self-reported violent ideation and its link to interpersonal violence among offenders with mental disorders and general psychiatric patients

This study aims at comparing mentally disordered offenders and general psychiatric patients regarding violent ideation and at exploring its association with interpersonal violence. We recruited 200 detainees undergoing forensic psychiatric evaluation and 390 general psychiatric patients at discharge. At baseline, they were asked about violent ideation; at the 20-week follow-up, information about violent acts was gathered from crime conviction registry, interviews, and records. The lifetime prevalence of violent ideation was 32.5% for offenders and 35.6% for patients; the corresponding two-month prevalence was 22.5% and 21.0%, respectively. For the both samples combined, those with violent ideation in their lifetime were significantly more prone to commit violent acts during follow-up than those without such ideation, OR = 2.65. The same applied to the patient sample, OR = 3.41. In terms of positive predictive values, fewer than 25% of those with violent ideation committed violent acts. Contrary to our hypothesis, the prevalence of violent ideation did not differ significantly between offenders and patients. However, there was support for the hypothesized association between violent ideation and violent acts on a group level. On an individual level, the clinician should consider additional factors when assessing the risk for violent acts.

Identification of low risk of violent crime in severe mental illness with a clinical prediction tool (Oxford Mental Illness and Violence tool [OxMIV]): a derivation and validation study

BACKGROUND

Current approaches to stratify patients with psychiatric disorders into groups on the basis of violence risk are limited by inconsistency, variable accuracy, and unscalability. To address the need for a scalable and valid tool to assess violence risk in patients with schizophrenia spectrum or bipolar disorder, we describe the derivation of a score based on routinely collected factors and present findings from external validation.

METHODS

On the basis of a national cohort of 75 158 Swedish individuals aged 15-65 years with a diagnosis of severe mental illness (schizophrenia spectrum or bipolar disorder) with 574 018 patient episodes between Jan 1, 2001, and Dec 31, 2008, we developed predictive models for violent offending (primary outcome) within 1 year of hospital discharge for inpatients or clinical contact with psychiatric services for outpatients (patient episode) through linkage of population-based registers. We developed a derivation model to determine the relative influence of prespecified criminal history and sociodemographic and clinical risk factors, which are mostly routinely collected, and then tested it in an external validation. We measured discrimination and calibration for prediction of violent offending at 1 year using specified risk cutoffs.

FINDINGS

Of the cohort of 75 158 patients with schizophrenia spectrum or bipolar disorder, we assigned 58 771 (78%) to the derivation sample and 16 387 (22%) to the validation sample. In the derivation sample, 830 (1%) individuals committed a violent offence within 12 months of their patient episode. We developed a 16-item model. The strongest predictors of violent offending within 12 months were conviction for previous violent crime (adjusted odds ratio 5·03 [95% CI 4·23-5·98]; p<0·0001), male sex (2·32 [1·91-2·81]; p<0·0001), and age (0·63 per 10 years of age [0·58-0·67]; p<0·0001). In external validation, the model showed good measures of discrimination (c-index 0·89 [0·85-0·93]) and calibration. For risk of violent offending at 1 year, with a 5% cutoff, sensitivity was 62% (95% CI 55-68) and specificity was 94% (93-94). The positive predictive value was 11% and the negative predictive value was more than 99%. We used the model to generate a simple web-based risk calculator (Oxford Mental Illness and Violence tool [OxMIV]).

INTERPRETATION

We have developed a prediction score in a national cohort of patients with schizophrenia spectrum or bipolar disorder, which can be used as an adjunct to decision making in clinical practice by identifying those who are at low risk of violent offending. The low positive predictive value suggests that further clinical assessment in individuals at high risk of violent offending is required to establish who might benefit from additional risk management. Further validation in other countries is needed.

FUNDING

Wellcome Trust and Swedish Research Council.

The Predictive Validity of the Structured Assessment of Violence Risk in Youth for Young Spanish Offenders

The present study examined the predictive validity of the Structured Assessment of Violence Risk in Youth (SAVRY) in a group of young Spanish offenders. The sample is made up of 594 minors from the Juvenile Court, between the ages of 14 and 18 at the time they committed the delinquent act. The SAVRY was able to differentiate between low and high-risk younger offenders. Mean scores on risk factor are greater in the group of recidivist offenders, the group of non-recidivist shows higher mean scores in Protective domain. The accuracy of the instrument is high (AUC_{RiskTotalScore} = 0.737 and AUC_{SummaryRiskRating} = 0.748). An approximation of the predictive validity study of the SAVRY in Spanish younger offenders is presented. The results obtained support the SAVRY good functioning with not English samples.

The Short- to Medium-Term Predictive Validity of Static and Dynamic Risk-of-Violence Measures in Medium- to Low-Secure Forensic and Civil Inpatients

The prediction and subsequent management of aggression by psychiatric inpatients is a crucial role of the mental health professional. This retrospective cohort study examines the predictive validity of 10 static and dynamic risk-of-violence measures and subscales in 37 forensic and 37 civil inpatients residing in a medium- to-low security psychiatric facility for a period of up to 6 months. Retrospective file records were sourced to conduct an AUC analysis of the ROC curve for short- and medium-term follow-up periods. The hypothesis that dynamic measures would be better predictors than static measures over the short term was supported. Albeit to a lesser extent, dynamic measures were still better predictors than static measures over the medium term. This result was seen in both civil and forensic groups. Three previously untested measures were found to predict aggression within the sample. It is recommended that mental health services employ the use of dynamic measures when making short-term risk-of-violence predictions for civil and/or forensic inpatients.

The Structured Assessment of Violence Risk in Adults with Intellectual Disability: A Systematic Review

BACKGROUND

While structured professional judgement approaches to assessing and managing the risk of violence have been extensively examined in mental health/forensic settings, the application of the findings to people with an intellectual disability is less extensively researched and reviewed. This review aimed to assess whether risk assessment tools have adequate predictive validity for violence in adults with an intellectual disability.

METHODS

Standard systematic review methodology was used to identify and synthesize appropriate studies.

RESULTS

A total of 14 studies were identified as meeting the inclusion criteria. These studies assessed the predictive validity of 18 different risk assessment tools, mainly in forensic settings. All studies concluded that the tools assessed were successful in predicting violence. Studies were generally of a high quality.

CONCLUSIONS

There is good quality evidence that risk assessment tools are valid for people with intellectual disability who offend but further research is required to validate tools for use with people with intellectual disability who offend.

Testing a machine-learning algorithm to predict the persistence and severity of major depressive disorder from baseline self-reports

Heterogeneity of major depressive disorder (MDD) illness course complicates clinical decision-making. Although efforts to use symptom profiles or biomarkers to develop clinically useful prognostic subtypes have had limited success, a recent report showed that machine-learning (ML) models developed from self-reports about incident episode characteristics and comorbidities among respondents with lifetime MDD in the World Health Organization World Mental Health (WMH) Surveys predicted MDD persistence, chronicity and severity with good accuracy. We report results of model validation in an independent prospective national household sample of 1056 respondents with lifetime MDD at baseline. The WMH ML models were applied to these baseline data to generate predicted outcome scores that were compared with observed scores assessed 10-12 years after baseline. ML model prediction accuracy was also compared with that of conventional logistic regression models. Area under the receiver operating characteristic curve based on ML (0.63 for high chronicity and 0.71-0.76 for the other prospective outcomes) was consistently higher than for the logistic models (0.62-0.70) despite the latter models including more predictors. A total of 34.6-38.1% of respondents with subsequent high persistence chronicity and 40.8-55.8% with the severity indicators were in the top 20% of the baseline ML-predicted risk distribution, while only 0.9% of respondents with subsequent hospitalizations and 1.5% with suicide attempts were in the lowest 20% of the ML-predicted risk distribution. These results confirm that clinically useful MDD risk-stratification models can be generated from baseline patient self-reports and that ML methods improve on conventional methods in developing such models.

Covert and Implicit Influences on the Interpretation of Violence Risk Instruments

Forensic mental health practitioners are frequently asked to estimate the risk of future violence. Legal decisions concerning the sentencing, management and disposition of offenders often rely on the advice of such testimony. The burgeoning use of violence risk instruments in these settings undoubtedly injects a level of scientific rigour into forensic evaluations for courts and tribunals. Yet scrutiny of the inherent limitations of both risk instruments and the inferences and formulations drawn from them are often veiled by the discipline's endorsement for such approaches. Misconceptions about the validity and dependability of present-day risk assessments and expert infallibility persist. The furtive influences that shape both the (mis)interpretation and miscommunication of risk instruments in legal settings necessitate discussion.

Prediction of violent reoffending on release from prison: derivation and external validation of a scalable tool

BACKGROUND

More than 30 million people are released from prison worldwide every year, who include a group at high risk of perpetrating interpersonal violence. Because there is considerable inconsistency and inefficiency in identifying those who would benefit from interventions to reduce this risk, we developed and validated a clinical prediction rule to determine the risk of violent offending in released prisoners.

METHODS

We did a cohort study of a population of released prisoners in Sweden. Through linkage of population-based registers, we developed predictive models for violent reoffending for the cohort. First, we developed a derivation model to determine the strength of prespecified, routinely obtained criminal history, sociodemographic, and clinical risk factors using multivariable Cox proportional hazard regression, and then tested them in an external validation. We measured discrimination and calibration for prediction of our primary outcome of violent reoffending at 1 and 2 years using cutoffs of 10% for 1-year risk and 20% for 2-year risk.

FINDINGS

We identified a cohort of 47 326 prisoners released in Sweden between 2001 and 2009, with 11 263 incidents of violent reoffending during this period. We developed a 14-item derivation model to predict violent reoffending and tested it in an external validation (assigning 37 100 individuals to the derivation sample and 10 226 to the validation sample). The model showed good measures of discrimination (Harrell's c-index 0·74) and calibration. For risk of violent reoffending at 1 year, sensitivity was 76% (95% CI 73-79) and specificity was 61% (95% CI 60-62). Positive and negative predictive values were 21% (95% CI 19-22) and 95% (95% CI 94-96), respectively. At 2 years, sensitivity was 67% (95% CI 64-69) and specificity was 70% (95% CI 69-72). Positive and negative predictive values were 37% (95% CI 35-39) and 89% (95% CI 88-90), respectively. Of individuals with a predicted risk of violent reoffending of 50% or more, 88% had drug and alcohol use disorders. We used the model to generate a simple, web-based, risk calculator (OxRec) that is free to use.

INTERPRETATION

We have developed a prediction model in a Swedish prison population that can assist with decision making on release by identifying those who are at low risk of future violent offending, and those at high risk of violent reoffending who might benefit from drug and alcohol treatment. Further assessments in other populations and countries are needed.

FUNDING

Wellcome Trust, the Swedish Research Council, and the Swedish Research Council for Health, Working Life and Welfare.

Couples Counseling for Aboriginal Clients Following Intimate Partner Violence: Service Providers' Perceptions of Risk

Interventions for family violence in Aboriginal communities should take a culture-based approach and focus on healing for the whole family. The purpose of this research was to identify risk issues from the perspective of service providers for couples counseling with Aboriginal clients following intimate partner violence. A total of 25 service providers participated in over the phone interviews concerning risk with Aboriginal men in couple counseling. Five concepts emerged including (a) collaterals, (b) commitment to change, (c) violence, (d) mind-set, and (e) mental health. It was concluded that culturally competent interventions should involve the entire community and have a restorative approach. The concepts were compared and contrasted with the available literature.

A Prospective Cohort Study of Absconsion Incidents in Forensic Psychiatric Settings: Can We Identify Those at High-Risk?

Background

Incidents of absconsion in forensic psychiatric units can have potentially serious consequences, yet surprisingly little is known about the characteristics of patients who abscond from these settings. The few previous studies conducted to date have employed retrospective designs, and no attempt has been made to develop an empirically-derived risk assessment scale. In this prospective study, we aimed to identify predictors of absconsion over a two-year period and investigate the feasibility of developing a brief risk assessment scale.

Methods

The study examined a representative sample of 135 patients treated in forensic medium- and low-secure wards. At baseline, demographic, clinical, treatment-related, and offending/behavioural factors were ascertained from electronic medical records and the treating teams. Incidents of absconsion (i.e., failure to return from leave, incidents of escape, and absconding whilst on escorted leave) were assessed at a two-year follow-up. Logistic regression analyses were used to determine the strongest predictors of absconsion which were then weighted according to their ability to discriminate absconders and non-absconders. The predictive utility of a brief risk assessment scale based on these weighted items was evaluated using receiver operator characteristics (ROC).

Results

During the two-year follow-up period, 27 patients (20%) absconded, accounting for 56 separate incidents. In multivariate analyses, four factors relating to offending and behaviour emerged as the strongest predictors of absconsion: history of sexual offending, previous absconsion, recent inpatient verbal aggression, and recent inpatient substance use. The weighted risk scale derived from these factors had moderate-to-good predictive accuracy (ROC area under the curve: 0.80; sensitivity: 067; specificity: 0.71), a high negative predictive value (0.91), but a low positive predictive value (0.34).

Conclusion

Potentially-targetable recent behaviours, such as inpatient verbal aggression and substance use, are strong predictors of absconsion in forensic settings; the absence of these factors may enable clinical teams to identify unnecessarily restricted low-risk individuals.

Replicating the violence risk appraisal guide: a total forensic cohort study

INTRODUCTION

The performance of violence risk assessment instruments can be primarily investigated by analysing two psychometric properties: discrimination and calibration. Although many studies have examined the discrimination capacity of the Violence Risk Appraisal Guide (VRAG) and other actuarial risk assessment tools, few have evaluated how well calibrated these instruments are. The aim of the present investigation was to replicate the development study of the VRAG in Europe including measurements of discrimination and calibration.

METHOD

Using a prospective study design, we assessed a total cohort of violent offenders in the Zurich Canton of Switzerland using the VRAG prior to discharge from prisons, secure facilities, and outpatient clinics. Assessors adhered strictly to the assessment protocol set out in the instrument's manual. After controlling for attrition, 206 offenders were followed in the community for a fixed period of 7 years. We used charges and convictions for subsequent violent offenses as the outcomes. Receiver operating characteristic analysis was conducted to measure discrimination, and Sanders' decomposition of the Brier score as well as Bayesian credible intervals were calculated to measure calibration.

RESULTS

The discrimination of the VRAG's risk bins was modest (area under the curve = 0.72, 95% CI = 0.63-0.81, p<0.05). However, the calibration of the tool was poor, with Sanders' calibration score suggesting an average assessment error of 21% in the probabilistic estimates associated with each bin. The Bayesian credible intervals revealed that in five out of nine risk bins the intervals did not contain the expected risk rates.

DISCUSSION

Measurement of the calibration validity of risk assessment instruments needs to be improved, as has been done with respect to discrimination. Additional replication studies that focus on the calibration of actuarial risk assessment instruments are needed. Meanwhile, we recommend caution when using the VRAG probabilistic risk estimates in practice.

Predictive validity performance indicators in violence risk assessment: a methodological primer

The predictive validity of violence risk assessments can be divided into two components: calibration and discrimination. The most common performance indicator used to measure the predictive validity of structured risk assessments, the area under the receiver operating characteristic curve (AUC), measures the latter component but not the former. As it does not capture how well a risk assessment tool's predictions of risk agree with actual observed risk, the AUC provides an incomplete portrayal of predictive validity. This primer provides an overview of calibration and discrimination performance indicators that measure global performance, performance in identifying higher-risk groups, and performance in identifying lower-risk groups. It is recommended that future research into the predictive validity of violence risk assessment tools includes a number of performance indicators that measure different facets of predictive validity and that the limitations of reported indicators be routinely explicated.