Hidden multiple comparisons increase forensic error rates

When wires are cut, the tool produces striations on the cut surface; as in other forms of forensic analysis, these striation marks are used to connect the evidence to the source that created them. Here, we argue that the practice of comparing two wire cut surfaces introduces complexities not present in better-investigated forensic examination of toolmarks such as those observed on bullets, as wire comparisons inherently require multiple distinct comparisons, increasing the expected false discovery rate. We call attention to the multiple comparison problem in wire examination and relate it to other situations in forensics that involve multiple comparisons, such as database searches.

Polygraph-based deception detection and Machine Learning. Combining the Worst of Both Worlds?

At a time when developments in computational approaches, often associated with the now much-vaunted terms Machine Learning (ML) and Artificial Intelligence (AI), face increasing challenges in terms of fairness, transparency and accountability, the temptation for researchers to apply mainstream ML methods to virtually any type of data seems to remain irresistible. In this paper we critically examine a recent proposal to apply ML to polygraph screening results (where human interviewers have made a conclusion about deception), which raises several questions about the purpose and the design of the research, particularly given the vacuous scientific status of polygraph-based procedures themselves. We argue that in high-stake environments such as criminal justice and employment practice, where fundamental rights and principles of justice are at stake, the legal and ethical considerations for scientific research are heightened. Specifically, we argue that the combination of ambiguously labelled data and ad hoc ML models does not meet this requirement. Worse, such research can inappropriately legitimise otherwise scientifically invalid, indeed pseudo-scientific methods such as polygraph-based deception detection, especially when presented in a reputable scientific journal. We conclude that methodological concerns, such as those highlighted in this paper, should be addressed before research can be said to contribute to resolving any of the fundamental validity issues that underlie methods and techniques used in legal proceedings.

Machine learning applications in forensic DNA profiling: A critical review

Machine learning (ML) is a range of powerful computational algorithms capable of generating predictive models via intelligent autonomous analysis of relatively large and often unstructured data. ML has become an integral part of our daily lives with a plethora of applications, including web, business, automotive industry, clinical diagnostics, scientific research, and more recently, forensic science. In the field of forensic DNA, the manual analysis of complex data can be challenging, time-consuming, and error-prone. The integration of novel ML-based methods may aid in streamlining this process while maintaining the high accuracy and reproducibility required for forensic tools. Due to the relative novelty of such applications, the forensic community is largely unaware of ML capabilities and limitations. Furthermore, computer science and ML professionals are often unfamiliar with the forensic science field and its specific requirements. This manuscript offers a brief introduction to the capabilities of machine learning methods and their applications in the context of forensic DNA analysis and offers a critical review of the current literature in this rapidly developing field.

The strange persistence of (source) "identification" claims in forensic literature through descriptivism, diagnosticism and machinism

Many forensic scientists consider that identification (individualisation) - in the sense of statements of the kind "the questioned item and the known item come from the same source" - is a concept that is central to their discipline. This is so despite decade-long, fundamental critiques levelled by both practitioners and academics against the conceptual and practical feasibility of forensic identification. Oddly, there is a constant stream of publications in (peer-reviewed) forensic science journals that treat forensic identification axiomatically as a valid object of study, sidestepping the fundamental critiques. This paper reviews and discusses three exemplary strands of publications that exemplify this persistent trend. These strands are called descriptivism, diagnosticism and machinism. The latter term refers to methods borrowed from the now increasingly popular approaches used in the field of machine learning. In turn, descriptivism and diagnosticism refer to general design aspects of mainstream research methods, illustrated here through a critical review of two recent papers on, respectively, forensic odontology and a framework for interpreting fingerprint evidence. The critique of the use of 'identification' in these strands of publication includes, but goes beyond, semantic details and the reiteration of long-known shortcomings of obsolete technical language such as 'match' and 'matching'. Specifically, this paper exposes deeper problems such as the subtle and argumentatively unfounded carrying-over of source conclusions to ultimate issues and the use probability concepts for questions that require more than the mere quantification of uncertainty. This paper submits that in order to foster trust in an era of continually expanding publishing activities, it should be a vital interest to forensic science journals to better examine what identification-related research can and cannot legitimately purport to achieve.

Establishment of a national DNA database in the Republic of Serbia: Legal aspects and implications for the future

Forensic DNA databases have been established in the vast majority of European countries and represent an essential personal identification instrument in the criminal justice system. The implementation of suitable legislation and accompanying legal practice is an imperative to ensure sensible use of the DNA repository, without interfering with the rights, freedoms and privacy of individuals and their families. Thus, the complex matter of DNA databank effectiveness and intrusiveness is a delicate balancing act, which has resulted in diverse database governing regimes among different countries. In 2018, the National Assembly of the Republic of Serbia passed the first Law on the National DNA Register, which was created by the Ministry of Interior. This short and imprecise regulation was announced without prior consultation with professional stakeholders or the public, resulting in controversy. In this article, we examine essential questions pertinent to launching a forensic DNA database through the lens of the new Law on the National DNA Register in the Republic of Serbia and in comparison to the other European approaches and guidelines to regulating forensic DNA repositories. We further discuss the resulting legal, social and ethical implications and concerns and propose future actions intended to improve the legal framework. We believe that stimulating such discussions throughout the scientific and professional community will result in the advancement of DNA profiling and databasing in a future European Union member state and, more generally, the manner in which biological data are managed in various countries, particularly those that are developing.

Performance of forensic facial comparison by morphological analysis across optimal and suboptimal CCTV settings

Facial comparison is an important yet understudied discipline in forensics. The recommended method for facial comparison in a forensic setting involves morphological analysis (MA) with the use of a facial feature list. The performance of this approach has not been tested across various closed-circuit television (CCTV) conditions. This is of particular concern as video and image data available to law enforcement is often varied and of subpar conditions. The present study aimed at testing MA across two types of CCTV data, representing ideal and less than ideal settings, also assessing which particular shortcomings arose from less-than-ideal settings. The study was conducted on a subset of the Wits Face Database arranged in a total of 225 face pools. Each face pool consisted of a target image obtained from either a high-definition digital CCTV camera or a low-definition analogue CCTV camera in monochrome, contrasted to 10 possible matches. The face pools were analysed and scored using MA and confusion matrices were used to analyse the outcomes. A notably high chance corrected accuracy (CCA) (97.3%) and reliability (0.969) was identified across the digital CCTV sample, while in the analogue CCTV sample MA appeared to underperform both in accuracy (CCA: 33.1%) and reliability (0.529). The majority of the errors in scoring resulted in false negatives in the analogue sample (75.2%), while across both CCTV conditions false positives were low (digital: 0.3%; analogue: 1.2%). Even though hit rates appeared deceptively high in the analogue sample, the various measures of performance used and particularly the chance corrected accuracy highlighted its shortfalls. Overall, CCTV recording quality appears closely associated to MA performance, despite the favourable error rates when using the Facial Identification Scientific Working Group feature list.

The development of forensic DNA analysis: New debates on the issue of fundamental human rights

Before the advent of forensic DNA profiling, forensic techniques such as fingerprint examination and blood type comparison were used in the identification of suspects. DNA profiling has since become the gold standard of forensic science, and forensic DNA analysis techniques continue to evolve. Recent developments such as familial searching and phenotyping have raised ethical questions and concerns reflecting those expressed in the late 1980s when forensic DNA analysis was first introduced. At that time, attempts to use DNA evidence in criminal trials were met with challenges to its evidential value and admissibility. A common concern was whether the probative value of the evidence would outweigh its potentially prejudicial effect. This gave rise to a complex three-way debate, which revolved around first, the admissibility of the scientific principles in criminal courts; second, the scientific process involved in analysing DNA samples; and third, the impact that forensic DNA analysis may have on fundamental human rights. Ultimately, debates about the scientific process and the admissibility of such evidence in criminal trials overshadowed the debate about potential infringements of fundamental human rights. This resulted in a lack of critical discussion around the erosion of civil liberties through the use of scientific technologies. This paper revisits the early debates on the development of forensic DNA analysis. It draws parallels with current developments and analyses the potential for current and future human rights infringements, highlighting that the libertarian model offers a necessary counterbalance to the other arguments, due to its concern for maintaining fundamental rights.

A generalised Bayes' factor formula for evidence evaluation under activity level propositions: Variations around a fibres scenario

Generalised Bayes' factors and associated Bayesian networks are developed for the transfer of extrinsic evidence at the activity level, developments that extend previous work on activity level evaluation. A strategy for the assessment of extrinsic evidence is developed in stages with progressive increases in complexity. The final development is illustrated with an example involving fibres from clothing. This provides a list of factors involved in the consideration of a transfer case with activity level propositions and their roles in the determination of evidential value.

Validation of forensic facial comparison by morphological analysis in photographic and CCTV samples

Between the ever-increasing availability of surveillance evidence and expert-based forensic facial comparison being considered admissible in court, confirming its validity is paramount. Facial comparison is most commonly conducted using morphological analysis (MA), a largely untested feature-based approach. This study aimed at validating the current recommended practice of MA in both standardised and suboptimal surveillance samples. Face pools of 175 South African males were compiled with a series of facial photographs, using images from the Wits Face Database. The first 75 face pools consisted of wildtype (unstandardised) high-quality target photographs, while the remaining 100 face pools consisted of suboptimal closed-circuit television (CCTV) target images. Target images were compared to high-quality standardised photographs. Face pools were analysed using the Facial Identification Scientific Working Group's guidelines and feature list. Confusion matrices were used to determine the performance of MA in each cohort. MA was found highly accurate (chance-corrected accuracy (CCA): 99.1%) and reliable (κ = 0.921) in the photographic sample and less accurate (CCA: 82.6%) and reliable (κ = 0.743), in the CCTV sample. Higher false-positive and false-negative rates were noted for the CCTV sample, with the majority of errors resulting in false-negative outcomes. The decreased performance in the CCTV sample was attributed to various factors including image quality, angle of recording and lighting. Other studies testing facial comparison identified lower accuracies and reliability across various conditions. Better performance was found here and in other studies that included some form of facial feature list, reinforcing the importance of using a systematic facial feature list.

Rethinking the Confident Eyewitness: A Reply to Wixted, Mickes, and Fisher

In the current issue, Wixted, Mickes, and Fisher make the claim that eyewitness memory is not inherently unreliable. They also describe specific conditions under which an eyewitness's confidence can be a reliable indicator of accuracy in the context of both recall and recognition. We argue, however, that calculating the probative value of eyewitness evidence is more complicated than the authors acknowledge. In this commentary, we raise several concerns about the collection and assessment of eyewitness confidence in the real world. We also discuss how frequently the conditions specified by Wixted et al. are met in real cases. The potential for memory contamination is commonplace and can likely be outside the control of investigators. Moreover, there is reason to doubt that legal decision makers are sensitive to the myriad ways that eyewitness memory can be influenced. Because of this, we think eyewitness-memory scientists would do well to further research the intricacies of eyewitness confidence in the real world.

DNA commission of the International society for forensic genetics: Assessing the value of forensic biological evidence - Guidelines highlighting the importance of propositions: Part I: evaluation of DNA profiling comparisons given (sub-) source propositions

The interpretation of evidence continues to be one of the biggest challenges facing the forensic community. This is the first of two papers intended to provide advice on difficult aspects of evaluation and in particular on the formulation of propositions. The scientist has a dual role: investigator (crime-focused), where often there is no suspect available and a database search may be required; evaluator (suspect-focused), where the strength of evidence is assessed in the context of the case. In investigative mode, generally the aim is to produce leads regarding the source of the DNA. Sub-source level propositions will be adequate to help identify potential suspects who can be further investigated by the authorities. Once in evaluative mode, given the defence version of events of the person of interest, it may become necessary to consider alternatives that go beyond the source of the DNA (i.e., to consider activity level propositions). In the evaluation phase, it is crucial that formulation of propositions allows the assessment of all the results that will help with the issue at hand. Propositions should therefore be precise (indication of the number of contributors, information on the relevant population etc.), be about causes, not effects (e.g. a 'matching' DNA profile) and to avoid bias, must not be findings-led. This means that ideally, propositions should be decided based on the case information and before the results of the comparisons are known. This paper primarily reflects upon what has been coined as "sub-source level propositions". These are restricted to the evaluation of the DNA profiles themselves, and help answer the issue regarding the source of the DNA. It is to be emphasised that likelihood ratios given sub-source level propositions cannot be carried over to a different level - for example, activity level propositions, where the DNA evidence is put into the context of the alleged activities. This would be highly misleading and could give rise to miscarriages of justice; this will be discussed in a second paper. The value of forensic results depends not only on propositions, but also on the type of results (e.g. allelic designations, peak heights, replicates) and upon the model used: it is therefore important to discuss these aspects. Finally, since communication is key to help understanding by courts, we will explore how to convey the value of the results and explain the importance of avoiding the practice of transposing the conditional.

Bayes and the Law

Although the last forty years has seen considerable growth in the use of statistics in legal proceedings, it is primarily classical statistical methods rather than Bayesian methods that have been used. Yet the Bayesian approach avoids many of the problems of classical statistics and is also well suited to a broader range of problems. This paper reviews the potential and actual use of Bayes in the law and explains the main reasons for its lack of impact on legal practice. These include misconceptions by the legal community about Bayes' theorem, over-reliance on the use of the likelihood ratio and the lack of adoption of modern computational methods. We argue that Bayesian Networks (BNs), which automatically produce the necessary Bayesian calculations, provide an opportunity to address most concerns about using Bayes in the law.

Identifying personal microbiomes using metagenomic codes

Community composition within the human microbiome varies across individuals, but it remains unknown if this variation is sufficient to uniquely identify individuals within large populations or stable enough to identify them over time. We investigated this by developing a hitting set-based coding algorithm and applying it to the Human Microbiome Project population. Our approach defined body site-specific metagenomic codes: sets of microbial taxa or genes prioritized to uniquely and stably identify individuals. Codes capturing strain variation in clade-specific marker genes were able to distinguish among 100s of individuals at an initial sampling time point. In comparisons with follow-up samples collected 30-300 d later, ∼30% of individuals could still be uniquely pinpointed using metagenomic codes from a typical body site; coincidental (false positive) matches were rare. Codes based on the gut microbiome were exceptionally stable and pinpointed >80% of individuals. The failure of a code to match its owner at a later time point was largely explained by the loss of specific microbial strains (at current limits of detection) and was only weakly associated with the length of the sampling interval. In addition to highlighting patterns of temporal variation in the ecology of the human microbiome, this work demonstrates the feasibility of microbiome-based identifiability-a result with important ethical implications for microbiome study design. The datasets and code used in this work are available for download from huttenhower.sph.harvard.edu/idability.

Informing the judgments of fingerprint analysts using quality metric and statistical assessment tools

The aim of this research was to evaluate how fingerprint analysts would incorporate information from newly developed tools into their decision making processes. Specifically, we assessed effects using the following: (1) a quality tool to aid in the assessment of the clarity of the friction ridge details, (2) a statistical tool to provide likelihood ratios representing the strength of the corresponding features between compared fingerprints, and (3) consensus information from a group of trained fingerprint experts. The measured variables for the effect on examiner performance were the accuracy and reproducibility of the conclusions against the ground truth (including the impact on error rates) and the analyst accuracy and variation for feature selection and comparison. The results showed that participants using the consensus information from other fingerprint experts demonstrated more consistency and accuracy in minutiae selection. They also demonstrated higher accuracy, sensitivity, and specificity in the decisions reported. The quality tool also affected minutiae selection (which, in turn, had limited influence on the reported decisions); the statistical tool did not appear to influence the reported decisions.

A perspective on errors, bias, and interpretation in the forensic sciences and direction for continuing advancement

The forensic sciences are under review more so than ever before. Such review is necessary and healthy and should be a continuous process. It identifies areas for improvement in quality practices and services. The issues surrounding error, i.e., measurement error, human error, contextual bias, and confirmatory bias, and interpretation are discussed. Infrastructure is already in place to support reliability. However, more definition and clarity of terms and interpretation would facilitate communication and understanding. Material improvement across the disciplines should be sought through national programs in education and training, focused on science, the scientific method, statistics, and ethics. To provide direction for advancing the forensic sciences a list of recommendations ranging from further documentation to new research and validation to education and to accreditation is provided for consideration. The list is a starting point for discussion that could foster further thought and input in developing an overarching strategic plan for enhancing the forensic sciences.

Phylogeographic investigations: the role of trees in forensic genetics

The human mitochondrial DNA (mtDNA) genome is commonly analyzed in various disciplines, such as population, medical, and forensic genetics, but conceptual and scientific exchange between them is still limited. Here we review several aspects of the mtDNA phylogeny that are particularly--but not exclusively--of interest to the forensic community. Among the issues that arise, we emphasize the importance of integrating evolutionary concepts into the forensic routine. We also discuss topics such as mtDNA mutation-rate heterogeneity and the weight of evidence, ethnic affiliations of mtDNA profiles, and the abuse of reference databases. Finally, we show the usefulness of coding-region variation in a forensic context.

Bayesian networks and probabilistic reasoning about scientific evidence when there is a lack of data

Bayesian networks (BNs) are a kind of graphical model that formally combines elements of graph and probability theory. BNs are a mathematically and statistically rigorous technique allowing their user to define a pictorial representation of assumed dependencies and influences among a set of variables deemed to be relevant for a particular inferential problem. The formalism allows one to process newly acquired evidence according to the rules of probability calculus. Applications of BNs have been reported in various forensic disciplines. However, there seems to be some reluctance to consider BNs as a more general framework for representing and evaluating sources of uncertainties associated with scientific evidence. Notably, BNs are widely thought of as an essentially numerical method, requiring "exact" numbers with a high "accuracy". The present paper aims to draw the reader's attention to the point that the availability of hard numerical data is not a necessary requirement for using BNs in forensic science. An abstraction of quantitative BNs, known as qualitative probabilistic networks (QPNs), and sensitivity analyses are presented and their potential applications discussed. As a main difference to their quantitative counterpart, QPNs contain qualitative probabilistic relationships instead of numerical relations. Sensitivity analyses consist of varying the probabilities assigned to one or more variables and evaluating the effect on one or more other variables of interest. Both QPNs and sensitivity analyses appear to be useful concepts that permit one to work in contexts with acute lack of numerical data and where reasoning consistent with the laws of probability should nevertheless be performed.

California's Proposition 69: a dangerous precedent for criminal DNA databases

On November 2, 2004, California voters approved Proposition 69, “The DNA Fingerprint, Unsolved Crime, and Innocence Protection Act” by a margin of approximately 60 to 40 percent. Given the limited amount of information provided to voters during the initiative process, it is unclear how many of the yea-sayers were apprised of the full implications of this measure. Indeed, by voting “yes” on Proposition 69, California has elected to house the most radical and costly state criminal DNA database in the country. This dangerous expansion of California's database poses tremendous threats to civil liberties and social justice while offering little, if anything, by way of increasing the safety of its citizens.

Prior to November 2, California law required the permanent retention of DNA samples from felons convicted of serious, violent crimes.

Dangerous excursions: the case against expanding forensic DNA databases to innocent persons

Over the past fifteen years, the United States has witnessed an extraordinary expansion in the banking and mining of DNA for law enforcement purposes. While the earliest state laws governing forensic DNA limited collection and retention of DNA samples to sexual offenders – on the theory that these persons were especially prone to recidivism and most likely to leave behind biological evidence – today forty-three states collect DNA from all felons, twenty-eight from juvenile offenders, and thirty-eight from those who commit certain categories of misdemeanors.

A few states have expanded their databases beyond convicted criminals. Virginia, Louisiana, Texas, and California have authorized DNA retention from persons merely arrested for various offenses, although to date only Virginia has implemented such a program. At the federal level, an ill-considered statute that allows for the seizure and storage of DNA from anyone arrested and from non-U.S. citizens detained under federal authorities was recently signed into law.

California's Proposition 69: a dangerous precedent for criminal DNA databases

On November 2, 2004, California voters approved Proposition 69, “The DNA Fingerprint, Unsolved Crime, and Innocence Protection Act” by a margin of approximately 60 to 40 percent. Given the limited amount of information provided to voters during the initiative process, it is unclear how many of the yea-sayers were apprised of the full implications of this measure. Indeed, by voting “yes” on Proposition 69, California has elected to house the most radical and costly state criminal DNA database in the country. This dangerous expansion of California's database poses tremendous threats to civil liberties and social justice while offering little, if anything, by way of increasing the safety of its citizens.

Prior to November 2, California law required the permanent retention of DNA samples from felons convicted of serious, violent crimes. The new law expands the database to include DNA samples from all felons and individuals with past felony convictions - including juveniles - and, beginning in five years, all adults arrested for any felony offense.

Forensic hair morphology comparison – a dying art or junk science?

There has been debate in both the judicial and forensic fields concerning the admissibility and reliability of the so-called forensic comparison sciences such as handwriting, tool mark analyses, and hair analysis. In particular, there has been increasing controversy over the use and interpretation of hair comparison evidence and it has been held partly responsible for miscarriages of justice. There has also been a perceived devaluation of the worth of microscopic human hair analysis particularly since the advent of DNA profiling. This article will attempt to initiate discussion on the past, current and future role of forensic human hair analysis and comparison.

A general approach to Bayesian networks for the interpretation of evidence

Bayesian networks (BNs) are mathematically and statistically rigorous techniques for handling uncertainty. The field of forensic science has recently attributed increased attention to the many advantages of this graphical method for assisting the evaluation of scientific evidence. However, the majority of contributions that relate to this topic restrict themselves to the presentation of already "constructed" BNs, and often, only a few explanations are given as to how one obtains a specific BN structure for a given problem. Based on several examples, the present paper will therefore attempt to explain in more detail some guiding considerations that might be helpful for the elicitation of appropriate structures for BNs.

A graphical model for the evaluation of cross-transfer evidence in DNA profiles

The role of graphical models in the assessment of transfer evidence is described with particular reference to the role of cross-transfer evidence. The issues involved in the determination of factors (nodes), associations (links) and probabilities to be included are discussed. Four types of subjective probabilities are of particular interest: those for transfer, persistence and recovery; innocent acquisition; relevance; innocent presence. Examples are given to illustrate the roles of various aspects of the suspect's and victim's lifestyle and the investigation of the evidence found on the suspect and victim in assessing the probability of ultimate issue, that the suspect committed the crime.