Current obstacles in replicating risk assessment findings: a systematic review of commonly used actuarial instruments

Evidence-based sentencing and scientific evidence

Evidence-based sentencing (EBS) is a new name for an aspiration that has deep roots in criminal law: to apply the sentence most appropriate to each offender's risk of reoffending, in order to reduce that risk as far as possible. This modern version of the traditional sentencing goals of rehabilitation and incapacitation fits into the broader approach of so-called "evidence-based public policy." It takes the view that the best existing evidence for reducing reoffending are modern structured risk assessment tools and claims to be able to achieve several goals at once: reducing reoffending, maintaining high levels of public safety, making more efficient use of public resources, and moving criminal policy away from ideological battles by basing it on the objective knowledge provided by the best available scientific evidence. However, despite the success of this approach in recent years, it is not clear to what extent it succeeds in correctly assessing the risk of individual offenders, nor whether it achieves its intended effect of reducing recidivism. This paper aims to critically examine these two issues: the quality of the scientific evidence on which EBS is based, and the available data on the extent to which it achieves (or does not achieve) its intended goals.

A Static-99R Validation Study on Individuals With Mental Disorders: 5 to 20 Years of Fixed Follow-Up After Sexual Offenses

'The Static-99R is one of the most commonly used risk assessment instruments for individuals convicted of sexual offenses. It has been validated for use on many populations, but few studies specifically target and describe individuals with mental disorders. Additionally, research on the discriminative properties (how well the instrument separates recidivists from non-recidivists) of the instrument over longer follow-up periods is scarce. This article evaluated the validity of the Static-99R using a cohort of individuals with mental disorders convicted of sexual offenses in Sweden (N = 146) with fixed 5-year (n = 100), 10-year (n = 91), 15-year (n = 79), and 20-year (n = 36) follow-up periods. A Static-99R cut score of 6 demonstrated the highest Youden index, maximizing sensitivity (72.7%) and specificity (74.2%), with 25.8% of recidivists correctly assumed to reoffend sexually and 95.7% of non-recidivists correctly assumed not to. The Static-99R instrument demonstrated adequate discrimination (AUC = 0.79, CI 95% = 0.70-0.87, and OR = 1.45, CI 95% = 1.14-1.84, p < 0.001, 5-year fixed follow-up), with only marginal differences for 10-, 15-, and 20-year fixed follow-up (AUC = 0.73, 0.74, and 0.74 and OR = 1.31, 1.36, and 1.40, respectively). Calibration (quantifying risk and correspondence with the instrument's norms) was acceptable (Brier = 0.088, P/E = 0.70, E/O = 1.43), with the routine sample norms displaying a decisively better fit to the study cohort compared to the high-risk/high-need sample norms. The results affirm the recommendation that, when in doubt and where there is no recent local norm group large enough available, the Static-99R routine sample found in the evaluators' workbook should be used.

Predictive Validity of the Static-99 and Static-99R in Switzerland

This study evaluated the validity of the Static-99 and Static-99R in assessing sexual recidivism in Switzerland, based on a sample of 142 male sex offenders. Both tools showed predictive validity, but the Static-99R had better discrimination (OR = 1.82, AUC = .81) and calibration (Brier = .078, P/E = 0.96) than the Static-99. A cut score of four on the Static-99R maximized sensitivity (92.9%) and specificity (60.2%). However, although most offenders (98.7%) with a score < 4 did not commit sexual offenses in the 5-year follow-up period, only one in five (20.3%) offenders with a score ≥ 4 actually recidivated. Furthermore, the predicted number of recidivists in the well above average risk category (Static-99R ≥ 6) was 24% higher than expected in routine samples. The results suggest that the Static-99R may be a useful screening tool to identify low-risk individuals but offenders with scores ≥ 4 should be subjected to a more thorough assessment.

Assessing violence risk with Aboriginal and Torres Strait Islander offenders: considerations for forensic practice

Forensic professionals and courts have frequently expressed concern about the susceptibility of contemporary risk assessment tools to cultural bias. Furthermore, progress in the development of valid methods of assessment for offenders who identify from Aboriginal and Torres Strait Islander cultural backgrounds has been slow. This paper considers how cultural perspectives on risk are essential to the development of assessment methods that have greater validity and acceptance by both courts and the community. This will involve considering the social, cultural and political determinants of risk in each cultural group and the identification of those risk factors that are most relevant to forensic decision-making.

Risk factors related to intimate partner violence police recidivism in Spain

Background/Objective: Some professionals, such as police officers, are required to prevent violent behavior, such as intimate partner violence (IPV). For this task they use actuarial tools designed to estimate the risk of occurrence of further violence after a previous complaint (police recidivism), taking into account risk and protective indicators which they can observe, in spite of they are not behavioral assessment experts. Method: To try to refine the police risk assessments carried out in Spain since 2007 and to improve the two tools available on the Spanish VioGén System, Police Risk Assessment and Risk Evolution (VPR_3.1 and VPER_3.0), this paper, using an epidemiological design, in a sample of 6,613 new cases of IPV of Spain, studies empirical relationships among 65 indicators (56 risk and 9 protection) and IPV police recidivism up to six months. Results: It resulted in a recidivism rate of 7.4%, finding statistically significant associations of 46 indicators. Conclusions: Empirical evidence about static indicators and new relevant dynamic indicators in the victims’ police protection management is presented. Practical implications for future police risk assessments are discussed.

Predicting violence and recidivism in a large sample of males on probation or parole

This study evaluated the utility of items and scales from the Iowa Violence and Victimization Instrument in a sample of 1961 males from the state of Iowa who were on probation or released from prison to parole supervision. This is the first study to examine the potential of the Iowa Violence and Victimization Instrument to predict criminal offenses. The males were followed for 30months immediately following their admission to probation or parole. AUC analyses indicated fair to good predictive power for the Iowa Violence and Victimization Instrument for charges of violence and victimization, but chance predictive power for drug offenses. Notably, both scales of the instrument performed equally well at the 30-month follow-up. Items on the Iowa Violence and Victimization Instrument not only predicted violence, but are straightforward to score. Violence management strategies are discussed as they relate to the current findings, including the potential to expand the measure to other jurisdictions and populations.

Covert and Implicit Influences on the Interpretation of Violence Risk Instruments

Forensic mental health practitioners are frequently asked to estimate the risk of future violence. Legal decisions concerning the sentencing, management and disposition of offenders often rely on the advice of such testimony. The burgeoning use of violence risk instruments in these settings undoubtedly injects a level of scientific rigour into forensic evaluations for courts and tribunals. Yet scrutiny of the inherent limitations of both risk instruments and the inferences and formulations drawn from them are often veiled by the discipline's endorsement for such approaches. Misconceptions about the validity and dependability of present-day risk assessments and expert infallibility persist. The furtive influences that shape both the (mis)interpretation and miscommunication of risk instruments in legal settings necessitate discussion.

Predicting general criminal recidivism in mentally disordered offenders using a random forest approach

BACKGROUND

Psychiatric expert opinions are supposed to assess the accused individual's risk of reoffending based on a valid scientific foundation. In contrast to specific recidivism, general recidivism has only been poorly considered in Continental Europe; we therefore aimed to develop a valid instrument for assessing the risk of general criminal recidivism of mentally ill offenders.

METHOD

Data of 259 mentally ill offenders with a median time at risk of 107 months were analyzed and combined with the individuals' criminal records. We derived risk factors for general criminal recidivism and classified re-offences by using a random forest approach.

RESULTS

In our sample of mentally ill offenders, 51% were reconvicted. The most important predictive factors for general criminal recidivism were: number of prior convictions, age, type of index offence, diversity of criminal history, and substance abuse. With our statistical approach we were able to correctly identify 58-95% of all reoffenders and 65-97% of all committed offences (AUC = .90).

CONCLUSIONS

Our study presents a new statistical approach to forensic-psychiatric risk-assessment, allowing experts to evaluate general risk of reoffending in mentally disordered individuals, with a special focus on high-risk groups. This approach might serve not only for expert opinions in court, but also for risk management strategies and therapeutic interventions.

Evidence for risk estimate precision: implications for individual risk communication

Actuarial risk assessment instruments using well-established predictor variables measured at the individual level (e.g., age, criminal history, psychopathy) discriminate well between recidivists and non-recidivists across diverse samples. Data indicating the relative risk of recidivism can inform policy decisions about allocating resources according to risk within a correctional system, consistent with the first of the risk-need-responsivity (RNR) principles. Evidence for the precision of absolute risk as applied to an individual based on scores from many samples, however, has proven challenging. In this paper, we present a study examining the association of actuarial risk estimate precision with sample size using the Post Conviction Risk Assessment (PCRA; Lowenkamp et al., 2013), in samples of up to 26,642 offenders. Results indicate that the precision of individual estimates can be demonstrated with sufficient sample size. We believe that the implications of absolute risk for the communication of an individual offender's risks have been poorly understood. We argue that the purpose of individual-level risk communication is to ensure the effective application of policy, which requires matching a new case to aggregate data. We illustrate how an offender's risk might thus be communicated, and conclude that this function is distinct from management of an individual's criminogenic needs and identification of effective and suitable treatments.

Replicating the violence risk appraisal guide: a total forensic cohort study

INTRODUCTION

The performance of violence risk assessment instruments can be primarily investigated by analysing two psychometric properties: discrimination and calibration. Although many studies have examined the discrimination capacity of the Violence Risk Appraisal Guide (VRAG) and other actuarial risk assessment tools, few have evaluated how well calibrated these instruments are. The aim of the present investigation was to replicate the development study of the VRAG in Europe including measurements of discrimination and calibration.

METHOD

Using a prospective study design, we assessed a total cohort of violent offenders in the Zurich Canton of Switzerland using the VRAG prior to discharge from prisons, secure facilities, and outpatient clinics. Assessors adhered strictly to the assessment protocol set out in the instrument's manual. After controlling for attrition, 206 offenders were followed in the community for a fixed period of 7 years. We used charges and convictions for subsequent violent offenses as the outcomes. Receiver operating characteristic analysis was conducted to measure discrimination, and Sanders' decomposition of the Brier score as well as Bayesian credible intervals were calculated to measure calibration.

RESULTS

The discrimination of the VRAG's risk bins was modest (area under the curve = 0.72, 95% CI = 0.63-0.81, p<0.05). However, the calibration of the tool was poor, with Sanders' calibration score suggesting an average assessment error of 21% in the probabilistic estimates associated with each bin. The Bayesian credible intervals revealed that in five out of nine risk bins the intervals did not contain the expected risk rates.

DISCUSSION

Measurement of the calibration validity of risk assessment instruments needs to be improved, as has been done with respect to discrimination. Additional replication studies that focus on the calibration of actuarial risk assessment instruments are needed. Meanwhile, we recommend caution when using the VRAG probabilistic risk estimates in practice.