The decisionalization of individualization

Planning, design and logistics of a decision analysis study: The FBI/Ames study involving forensic firearms examiners

This paper describes design and logistical aspects of a decision analysis study to assess the performance of qualified firearms examiners working in accredited laboratories in the United States in terms of accuracy (error rate), repeatability, and reproducibility of decisions involving comparisons of fired bullets and cartridge cases. The purpose of the study was to validate current practice of the forensic discipline of firearms/toolmarks (F/T) examination. It elicited error rate data by counting the number of false positive and false negative conclusions. Preceded by the experimental design, decisions, and logistics described herein, testing was ultimately administered 173 qualified, practicing F/T examiners in public and private crime laboratories. The first round of testing evaluated accuracy, while two subsequent rounds evaluated repeatability and reproducibility of examiner conclusions. This project expands on previous studies by involving many F/T examiners in challenging comparisons and by executing the study in the recommended double-blind format.

The strange persistence of (source) "identification" claims in forensic literature through descriptivism, diagnosticism and machinism

Many forensic scientists consider that identification (individualisation) - in the sense of statements of the kind "the questioned item and the known item come from the same source" - is a concept that is central to their discipline. This is so despite decade-long, fundamental critiques levelled by both practitioners and academics against the conceptual and practical feasibility of forensic identification. Oddly, there is a constant stream of publications in (peer-reviewed) forensic science journals that treat forensic identification axiomatically as a valid object of study, sidestepping the fundamental critiques. This paper reviews and discusses three exemplary strands of publications that exemplify this persistent trend. These strands are called descriptivism, diagnosticism and machinism. The latter term refers to methods borrowed from the now increasingly popular approaches used in the field of machine learning. In turn, descriptivism and diagnosticism refer to general design aspects of mainstream research methods, illustrated here through a critical review of two recent papers on, respectively, forensic odontology and a framework for interpreting fingerprint evidence. The critique of the use of 'identification' in these strands of publication includes, but goes beyond, semantic details and the reiteration of long-known shortcomings of obsolete technical language such as 'match' and 'matching'. Specifically, this paper exposes deeper problems such as the subtle and argumentatively unfounded carrying-over of source conclusions to ultimate issues and the use probability concepts for questions that require more than the mere quantification of uncertainty. This paper submits that in order to foster trust in an era of continually expanding publishing activities, it should be a vital interest to forensic science journals to better examine what identification-related research can and cannot legitimately purport to achieve.

Forensic science and the principle of excluded middle: "Inconclusive" decisions and the structure of error rate studies

In a paper published recently in this journal, Dror and Scurich (2020) [20] critically discuss the notions of "inconclusive evidence" (i.e., test items for which it is difficult to render a categorical response) and "inconclusive decisions" (i.e., experts' conclusions or responses) in the context of forensic science error rate studies. They expose several ways in which the understanding and use of "inconclusives" in current forensic science research and practice can adversely affect the outcomes of error rate studies. A main cause of distortion, according to Dror and Scurich, is what they call "erroneous inconclusive" decisions, in particular the lack of acknowledgment of this type of erroneous conclusion in the computation of error rates. To overcome this complication, Dror and Scurich call for a more explicit monitoring of "inconclusives" using a modified error rate study design. Whilst we agree with several well-argued points raised by the authors, we disagree with their framing of "inconclusive decisions" as potential errors. In this paper, we argue that referring to an "inconclusive decision" as an error is a contradiction in terms, runs counter to an analysis based on decision logic and, hence, is questionable as a concept. We also reiterate that the very term "inconclusive decision" disregards the procedural architecture of the criminal justice system across modern jurisdictions, especially the fact that forensic experts have no decisional rights in the criminal process. These positions do not ignore the possibility that "inconclusives" - if used excessively - do raise problems in forensic expert reporting, in particular limited assertiveness (or, overcautiousness). However, these drawbacks derive from inherent limitations of experts rather than from the seemingly erroneous nature of "inconclusives" that needs to be fixed. More fundamentally, we argue that attempts to score "inconclusives" as errors amount to philosophical claims disguised as forensic methodology. Specifically, these attempts interfere with the metaphysical substrate underpinning empirical research. We point this out on the basis of the law of the excluded middle, i.e. the principle of "no third possibility being given" (tertium non datur).

Minor or adult? Introducing decision analysis in forensic age estimation

Nowadays, forensic age estimation takes an important role in worldwide forensic and medico-legal institutes that are solicited by judicial or administrative authorities for providing an expert report on the age of individuals. The authorities' ultimate issue of interest is often the probability that the person is younger or older than a given age threshold, which is usually the age of majority. Such information is fundamental for deciding whether a person being judged falls under the legal category of an adult. This is a decision that may have important consequences for the individual, depending on the legal framework in which the decision is made. The aim of this paper is to introduce a normative approach for assisting the authority in the decision-making process given knowledge from available findings reported by means of probabilities. The normative approach proposed here has been acknowledged in the forensic framework, and represents a promising structure for reasoning that can support the decision-making process in forensic age estimation. The paper introduces the fundamental elements of decision theory applied to the specific case of age estimation, and provides some examples to illustrate its practical application.

Are Inconclusive Decisions in Forensic Science as Deficient as They Are Said to Be?

Many quarters of forensic science use reporting formats such as “identification,” “inconclusive,” and “exclusion.” These types of conclusions express opinions as to whether or not a particular person or object is the source of the material or traces of unknown source that is of interest in a given case. Rendering an “inconclusive” conclusion is sometimes criticized as being inadequate because—supposedly—it does not provide recipients of expert information with helpful directions. In this paper, we critically examine this claim using decision theory. We present and defend the viewpoint according to which deciding to render an “inconclusive” conclusion is, on a formal account, not as inadequate as may commonly be thought. Using elements of decision theory from existing accounts on the topic, we show that inconclusive conclusions can actually be viable alternatives with respect to other types of conclusions, such as “identification.”

Decisional Dimensions in Expert Witness Testimony A Structural Analysis

The relationship between forensic science and legal adjudication is intricate mainly because the need to inform fact-finders on issues going beyond the layman's knowledge poses challenges both on empirical and normative dimensions, in particular with regards to the specific role and duties of the different participants in the legal process. While rationality is widely upheld as one of the aspirations of the legal process across many modern jurisdictions, a pending question is how to remedy the uneasy relationship between general propositions (and knowledge claims) conditioning expert witness testimony, and individualized decisions taken by fact-finders. The focus has hitherto been put on the utilization of model-based and formal methods of reasoning while, regrettably, the concepts of judgment and decision-making have not received equal attention. A first aspiration of our paper will thus be to further clarify the nature of this systemic relationship in the particular area of the legal process involving scientific experts, by conducting a critical transversal analysis of current empirical, normative and doctrinal understandings of expert witness testimony. As a second aim, we will use this insight to argue in favor of the view that structural features of expert witness testimony are embedded in a decision-making process, and that the understanding of this decisional dimension is important for clarifying the respective roles of expert witnesses and fact-finders, and for favoring their mutual understanding thereof. To substantiate this perspective, and attest to its growing recognition as a frontier understanding, we will provide real-world examples from forensic science reporting practice and policy documents of professional bodies.

Big data uncertainties

Big data-the idea that an always-larger volume of information is being constantly recorded-suggests that new problems can now be subjected to scientific scrutiny. However, can classical statistical methods be used directly on big data? We analyze the problem by looking at two known pitfalls of big datasets. First, that they are biased, in the sense that they do not offer a complete view of the populations under consideration. Second, that they present a weak but pervasive level of dependence between all their components. In both cases we observe that the uncertainty of the conclusion obtained by statistical methods is increased when used on big data, either because of a systematic error (bias), or because of a larger degree of randomness (increased variance). We argue that the key challenge raised by big data is not only how to use big data to tackle new problems, but to develop tools and methods able to rigorously articulate the new risks therein.

Finding the way forward for forensic science in the US-A commentary on the PCAST report

A recent report by the US President's Council of Advisors on Science and Technology (PCAST), (2016) has made a number of recommendations for the future development of forensic science. Whereas we all agree that there is much need for change, we find that the PCAST report recommendations are founded on serious misunderstandings. We explain the traditional forensic paradigms of match and identification and the more recent foundation of the logical approach to evidence evaluation. This forms the groundwork for exposing many sources of confusion in the PCAST report. We explain how the notion of treating the scientist as a black box and the assignment of evidential weight through error rates is overly restrictive and misconceived. Our own view sees inferential logic, the development of calibrated knowledge and understanding of scientists as the core of the advance of the profession.