The case against categorical risk estimates

Consistency and Construct Validity of the Five-Level System for Risk Communication Using Static and Dynamic Tools: An Investigation Using the Static-99R and VRS-SO

This study examined the Council of State Governments' five-level system for risk communication, as applied to the Static-99R and Violence Risk Scale-Sexual Offense Version (VRS-SO). Aims of the system include increasing consistency in risk communication and linking risk categories to psychologically meaningful constructs. We investigated concordance between risk levels assigned by the instruments, and distributions of VRS-SO dynamic needs associated with Static-99R risk levels, among a multisite sample (n = 1,404) of persons who have sexually offended. Concordant categorical risk ratings were assigned in just over a third of cases, suggesting that consistency remains a concern with the system, particularly when conceptually disparate tools are applied. Densities of criminogenic needs varied widely among persons assigned the same risk level by the Static-99R and diverged from the descriptions ascribed by the system. These findings can inform clinical assessments and further refinement of the system.

Calibrating violence risk assessments for uncertainty

Psychiatrists and other mental health clinicians are often tasked with assessing patients' risk of violence. Approaches to this vary and include both unstructured (based on individual clinicians' judgement) and structured methods (based on formalised scoring and algorithms with varying scope for clinicians' judgement). The end result is usually a categorisation of risk, which may, in turn, reference a probability estimate of violence over a certain time period. Research over recent decades has made considerable improvements in refining structured approaches and categorising patients' risk classifications at a group level. The ability, however, to apply these findings clinically to predict the outcomes of individual patients remains contested. In this article, we review methods of assessing violence risk and empirical findings on their predictive validity. We note, in particular, limitations in calibration (accuracy at predicting absolute risk) as distinct from discrimination (accuracy at separating patients by outcome). We also consider clinical applications of these findings, including challenges applying statistics to individual patients, and broader conceptual issues in distinguishing risk and uncertainty. Based on this, we argue that there remain significant limits to assessing violence risk for individuals and that this requires careful consideration in clinical and legal contexts.

Improving Our Risk Communication: Standardized Risk Levels for Brief Assessment of Recidivism Risk-2002R

A Five-Level Risk and Needs system has been proposed as a common language for standardizing the meaning of risk levels across risk/need tools used in corrections. Study 1 examined whether the Five-Levels could be applied to BARR-2002R (N = 2,390), an actuarial tool for general recidivism. Study 2 examined the construct validity of BARR-2002R risk levels in two samples of individuals with a history of sexual offending (N = 1,081). Study 1 found reasonable correspondence between BARR-2002R scores and four of the five standardized risk levels (no Level V). Study 2 found that the profiles of individuals in Levels II, III, and IV were mostly consistent with expectations; however, individuals in the lowest risk level (Level I) had more criminogenic needs than expected based on the original descriptions of the Five-Levels. The Five-Level system was mostly successful when applied to BARR-2002R. Revisions to this system, or the inclusion of putatively dynamic risk factors and protective factors, may be required to improve alignment with the information provided by certain risk tools.

Police are Influenced by Anchoring and Risk When Allocating Resources for Scenario-based Intimate Partner Violence Cases

Sixty-six police officers were given four intimate partner violence (IPV) scenarios to rate for risk of future violence. At the start of the experiment, participants were provided with either a low-risk or high-risk "anchor" scenario of police attending an IPV incident. Next, participants were given three counterbalanced scenarios: high, medium, and low risk. Half the participants were given a structured professional judgment tool to guide their decisions. Participants given the low-risk anchor rated the following scenarios as being of greater risk than those given the high-risk anchor. Participants were consistent in identifying high-, medium-, and low-risk scenarios and the tool made no difference to these ratings. Participants were more confident in their higher risk judgments than their lower risk judgments. Officers distributed a disproportionately high amount of resources to the high-risk offenders and the results suggest that police officers can make decisions consistent with Risk-Need-Responsivity principles. However, anchoring effects and working in a context where violence is more severe and frequent has the potential to bias perceptions and make officers less sensitive to risk.

California Sexually Violent Predator (SVP) Evaluations in the Field: Static-99R and Diagnostic Field Reliability

Tests and diagnoses used in sexually violent predator (SVP) evaluations must be reliable, as reliability is foundational to validity. The current study contained a stratified sample of evaluations of 395 individuals referred as potential SVPs between 2012 and 2017. Each individual was initially evaluated by at least two experts. The sample included three groups: individuals not meeting SVP criteria (N = 200, or 400 evaluations), individuals meeting SVP criteria (N = 95, with 190 evaluations), and individuals where evaluators disagreed (N = 100, with 200 evaluations). The sample also included 200 subsequent independent evaluations on these "disagree" cases. Static-99R score intraclass coefficient (ICC) interrater reliability was good to excellent within each group and overall. Evaluators scored the Static-99R within one point of each other 87% of the time. Cohen's kappa diagnostic agreement for Pedophilic Disorder was substantial. ASPD and substance abuse kappa were in the "fair" range, while OSPD diagnoses in the positive group were at the "moderate" level of agreement. Ethnic differences in diagnoses were consistent with other studies, with equivalent Static-99R ICC values across ethnic groups. There were no significant differences between state civil servants versus contracted experts in Static-99R ratings or final determinations. The results suggest that Static-99R scores have acceptable reliability in these evaluations, and Pedophilic Disorder (the most common paraphilic disorder in our study) and OSPD can be reliably diagnosed. We discuss limitations of the study, as well as the need for care in high-stakes evaluations given the imperfect reliability of psychological measurements.

Improving Risk Communication: Developing Risk Ratios for the VRAG-R

We developed a set of risk ratios for the Violence Risk Appraisal Guide-Revised (VRAG-R) to broaden the range of risk communication options available when using this tool and to provide information needed for future efforts to apply The Council of State Governments Justice Center's standardized five-level risk framework to the scale. A slightly reduced version of the VRAG-R normative data set was used for the analyses (N = 1,238). Contrary to previous research developing risk ratios, logistic regression provided a more accurate estimate of observed violent recidivism rates than Cox regression for both total VRAG-R scores and VRAG-R decile bins. Further analyses indicated the relationship between the VRAG-R and violent recidivism was consistent over a 15-year follow-up period. Due to the difficulties with interpreting odds ratios, the final risk ratios were computed using rate ratios derived from a logistic regression model using a 5-year fixed follow-up period. These risk ratios, and templates for how the ratios might be used in an assessment report, are presented in the appendices.

Critical considerations in the development and interpretation of common risk language

Existing risk communication procedures are marred by various well-documented problems and inconsistencies. The Council of State Governments' Justice Center (United States) developed a five-level system for risk and needs communication, to standardize these procedures and to provide a common risk language. Introduction of a common language could constitute a dramatic shift in criminal justice processes, with wide-ranging impacts. This article provides a critical review of the system and its suitability for application to various risk assessment functions. Issues discussed include: applicability to specialist and generalist offending behavior, the characteristics of suitable instruments, statistical and conceptual priorities, barriers to precision in language, and conceptual issues related to changes in risk level. A thorough understanding of each of these issues is necessary to apply the system to new contexts and populations, and facilitate straightforward and precise risk communication. Absent further elaboration of the system, many problems with risk communication will persist.

The dark figure of sexual recidivism

Empirical studies of sexual offender recidivism have proliferated in recent decades. Virtually all of the studies define recidivism as a new legal charge or conviction for a sexual crime, and these studies tend to find recidivism rates of the order of 5-15% after 5 years and 10-25% after 10+ years. It is uncontroversial that such a definition of recidivism underestimates the true rate of sexual recidivism because most sexual crime is not reported to legal authorities, a principle known as the "dark figure of crime." To estimate the magnitude of the dark figure of sexual recidivism, this paper uses a probabilistic simulation approach in conjunction with the following: (i) victim self-report survey data about the rate of reporting sexual crime to legal authorities; (ii) offender self-report data about the number of victims per offender; and (iii) different assumptions about the chances of being convicted of a new sexual offense given that it is reported. Under any configuration of assumptions, the dark figure is substantial, and as a consequence the disparity between recidivism defined as a new legal charge or conviction for a sex crime and recidivism defined as actually committing a new sexual crime is large. These findings call into question the utility of recidivism studies that rely exclusively on official crime statistics to define sexual recidivism, and highlight the need for additional, long-term studies that use a variety of different measures to assess whether or not sexual recidivism has occurred.

Risk assessment communication difficulties: An empirical examination of the effects of categorical versus probabilistic risk communication in sexually violent predator decisions

Expert testimony concerning risk and its communication to the trier of fact has important implications for some of the most significant legal decisions. In a simulated sexual violent predator hearing, we examined how mock jurors interpret and use recidivism risk expert testimony communicated either categorically, using verbal labels, or probabilistically, using numeric values. Based upon the STATIC-99R, we compared mock jurors' decision-making and verdicts when we manipulated the style of risk communication across four different risk levels. In terms of verdict decisions, we found that higher risk levels were associated with more commitment decisions, but that this relationship only existed for the categorical risk-communication format. We also replicated previous research demonstrating that participants overestimate recidivism risk in general, especially when higher risk is communicated categorically. Finally, our participants did not differentiate well between the four levels of risk offered, instead apparently employing a more simplistic dichotomy between "low" or "high" risk for both their verdict decisions and their thresholds for commitment. The legal and policy implications of our findings are discussed, as well as suggestions for more effective presentation of expert risk testimony.