An illustration of the effect of various sources of uncertainty on DNA likelihood ratio calculations

A collaborative study on the precision of the Markov chain Monte Carlo algorithms used for DNA profile interpretation

Several fully continuous probabilistic genotyping software (PGS) use Markov chain Monte Carlo algorithms (MCMC) to assign weights to different proposed genotype combinations at a locus. Replicate interpretations of the same profile in these software are expected not to produce identical weights and likelihood ratio (LR) values due to the Monte Carlo aspect. This paper reports a detailed precision study under reproducibility conditions conducted as a collaborative exercise across the National Institute of Standards and Technology (NIST), Federal Bureau of Investigation (FBI), and Institute of Environmental Science and Research (ESR). Replicate interpretations generated across the three laboratories used the same input files, software version, and settings but different random number seed and different computers. This work demonstrates that using different computers to analyze replicate interpretations does not contribute to any variations in LR values. The study quantifies the magnitude of differences in the assigned LRs that is only due to run-to-run MCMC variability and addresses the potential explanations for the observed differences.

Extending the discussion on inconsistency in forensic decisions and results

The subject of inter- and intra-laboratory inconsistency was recently raised in a commentary by Itiel Dror. We re-visit an inter-laboratory trial, with which some of the authors of this current discussion were associated, to diagnose the causes of any differences in the likelihood ratios (LRs) assigned using probabilistic genotyping software. Some of the variation was due to different decisions that would be made on a case-by-case basis, some due to laboratory policy and would hence differ between laboratories, and the final and smallest part was the run-to-run difference caused by the Monte Carlo aspect of the software used. However, the net variation in LRs was considerable. We believe that most laboratories will self-diagnose the cause of their difference from the majority answer and in some, but not all instances will take corrective action. An inter-laboratory exercise consisting of raw data files for relatively straightforward mixtures, such as two mixtures of three or four persons, would allow laboratories to calibrate their procedures and findings.

A practical treatment of sensitivity analyses in activity level evaluations

Evaluations of forensic observations considering activity level propositions are becoming more common place in forensic institutions. A measure that can be taken to interrogate the evaluation for robustness is called sensitivity analysis. A sensitivity analysis explores the sensitivity of the evaluation to the data used when assigning probabilities, or to the level of uncertainty surrounding a probability assignment, or to the choice of various assumptions within the model. There have been a number of publications that describe sensitivity analysis in technical terms, and demonstrate their use, but limited literature on how that theory can be applied in practice. In this work we provide some simplified examples of how sensitivity analyses can be carried out, when they are likely to show that the evaluation is sensitive to underlying data, knowledge or assumptions, how to interpret the results of sensitivity analysis, and how the outcome can be reported. We also provide access to an application to conduct sensitivity analysis.

Developmental validation of STRmix™ NGS, a probabilistic genotyping tool for the interpretation of autosomal STRs from forensic profiles generated using NGS

We describe the developmental validation of the probabilistic genotyping software - STRmix™ NGS - developed for the interpretation of forensic DNA profiles containing autosomal STRs generated using next generation sequencing (NGS) also known as massively parallel sequencing (MPS) technologies. Developmental validation was carried out in accordance with the Scientific Working Group on DNA Analysis Methods (SWGDAM) Guidelines for the Validation of Probabilistic Genotyping Systems and the International Society for Forensic Genetics (ISFG) recommendations and included sensitivity and specificity testing, accuracy, precision, and the interpretation of case-types samples. The results of developmental validation demonstrate the appropriateness of the software for the interpretation of profiles developed using NGS technology.

Internal Validation of MaSTR™ Probabilistic Genotyping Software for the Interpretation of 2–5 Person Mixed DNA Profiles

Mixed human deoxyribonucleic acid (DNA) samples present one of the most challenging pieces of evidence that a forensic analyst can encounter. When multiple contributors, stochastic amplification, and allele drop-out further complicate the mixture profile, interpretation by hand becomes unreliable and statistical analysis problematic. Probabilistic genotyping software has provided a tool to address complex mixture interpretation and provide likelihood ratios for defined sets of propositions. The MaSTR™ software is a fully continuous probabilistic system that considers a wide range of STR profile data to provide likelihood ratios on DNA mixtures. Mixtures with two to five contributors and a range of component ratios and allele peak heights were created to test the validity of MaSTR™ with data similar to real casework. Over 280 different mixed DNA profiles were used to perform more than 2600 analyses using different sets of propositions and numbers of contributors. The results of the analyses demonstrated that MaSTR™ provided accurate and precise statistical data on DNA mixtures with up to five contributors, including minor contributors with stochastic amplification effects. Tests for both Type I and Type II errors were performed. The findings in this study support that MaSTR™ is a robust tool that meets the current standards for probabilistic genotyping.

Hamiltonian Monte Carlo with strict convergence criteria reduces run-to-run variability in forensic DNA mixture deconvolution

MOTIVATION

Analysing mixed DNA profiles is a common task in forensic genetics. Due to the complexity of the data, such analysis is often performed using Markov Chain Monte Carlo (MCMC)-based genotyping algorithms. These trade off precision against execution time. When default settings (including default chain lengths) are used, as large as a 10-fold changes in inferred log-likelihood ratios (LR) are observed when the software is run twice on the same case. So far, this uncertainty has been attributed to the stochasticity of MCMC algorithms. Since LRs translate directly to strength of the evidence in a criminal trial, forensic laboratories desire LR with small run-to-run variability.

RESULTS

We present the use of a Hamiltonian Monte Carlo (HMC) algorithm that reduces run-to-run variability in forensic DNA mixture deconvolution by around an order of magnitude without increased runtime. We achieve this by enforcing strict convergence criteria. We show that the choice of convergence metric strongly influences precision. We validate our method by reproducing previously published results for benchmark DNA mixtures (MIX05, MIX13, and ProvedIt). We also present a complete software implementation of our algorithm that is able to leverage GPU acceleration for the inference process. In the benchmark mixtures, on consumer-grade hardware, the runtime is less than 7 min for 3 contributors, less than 35 min for 4 contributors, and less than an hour for 5 contributors with one known contributor.

A Review of Probabilistic Genotyping Systems: EuroForMix, DNAStatistX and STRmix™

Probabilistic genotyping has become widespread. EuroForMix and DNAStatistX are both based upon maximum likelihood estimation using a γ model, whereas STRmix™ is a Bayesian approach that specifies prior distributions on the unknown model parameters. A general overview is provided of the historical development of probabilistic genotyping. Some general principles of interpretation are described, including: the application to investigative vs. evaluative reporting; detection of contamination events; inter and intra laboratory studies; numbers of contributors; proposition setting and validation of software and its performance. This is followed by details of the evolution, utility, practice and adoption of the software discussed.

When evaluating DNA evidence within a likelihood ratio framework, should the propositions be exhaustive?

We seek to develop a rational approach to forming propositions when little information is available from the outset, as this often happens in casework. If propositions used when evaluating evidence are not exhaustive (in the context of the case), then there is a theoretical risk that an LR greater than one may be associated with a proposition in the numerator that - if all meaningful propositions had been considered - would in fact have a lower posterior probability after consideration of the evidence. Ideally, all propositions should be considered. However, with multiple propositions, some terms will be larger than others and for simplification very small terms can be neglected without changing the order of magnitude of the value of the evidence (i.e. LR). Our analysis shows that mathematically a contributor's DNA can be assumed to be present under both prosecution and alternative propositions (H_p and H_a) if there is a reasonable prior probability of their DNA being present and their inclusion is supported by the profile. This is because the terms associated to these sub-propositions will dominate our LR. For example, in the absence of specific information, when considering two persons of interest (POI) as potential contributors to a mixed DNA profile we suggest the assumption of one when examining the presence of the other, after checking that both collectively explain the profile well. This represents more meaningful propositions and allows better discrimination. Slooten and Caliebe have shown that the overall LR is the weighted average of LRs with the same number of contributors (NoC) under both propositions. The weights involve both an assessment of the probability of the crime scene DNA profile and the probability of this NoC given the background information.

How can courts take into account the uncertainty in a likelihood ratio?

As legal practitioners and courts become more aware of scientific methods and evidence evaluation, they are demanding measures of the reliability of expert opinion. In particular, there are calls for error rates to accompany opinion evidence in comparative forensic sciences. While error rates or confidence intervals can be useful for those disciplines that claim to identify the source of a trace, the call for these statistical tools has extended to sciences that present opinions in the form of a likelihood ratio. In this article we argue against presenting both a likelihood ratio and numerical measures of its uncertainty. We explain how the LR already encapsulates uncertainty. Instead we consider how sensitivity analyses can be used to guide the presentation of LRs that are informative to the court and not unfair to defendants.

A multivariate statistical approach for the estimation of the ethnic origin of unknown genetic profiles in forensic genetics

DNA typing and genetic profile data interpretation are among the most relevant topics in forensic science; among other applications, genetic profile's capability to distinguish biogeographic information about population groups, subgroups and affiliations have been largely explored in the last decade. In fact, for investigative and intelligence purposes, it is extremely useful to identify subjects and estimate their biogeographic origins by examining the recovered DNA profiles from evidence on a crime scene. Current approaches for BiogeoGraphic Ancestry (BGA) estimation using STRs profiles are usually based on Bayesian methods, which quantify the evidence in terms of likelihood ratio, supporting or not the hypothesis that a certain profile belongs to a specific ethnic group. The present study provides an alternative approach to the likelihood ratio method that involves multivariate data analysis strategies for the estimation of multiple populations. Starting from the well-known NIST US autosomal STRs dataset involving African-American, Asian, and Caucasian individuals, and moving towards further and more geographically restricted populations (such as Northern Africans vs sub-Saharan Africans, Afghans vs Iraqis and Italians vs Romanians), powerful multivariate techniques such as Sparse and Logistic Principal Component Analysis (SL-PCA), Sparse Partial Least Squares-Discriminant Analysis (sPLS-DA) and Support Vector Machines (SVM) were employed and their discriminating power was also compared. Both sPLS-DA and SVM techniques provided robust classifications, yielding high sensitivity and specificity models capable of discriminating populations on ethnic basis. This application may represent a powerful and dynamic tool for law enforcement agencies whenever a standard autosomal STR profile is obtained from the biological evidence collected at a crime scene or recovered during mass-disaster and missing person investigations.

Must the random man be unrelated? A lingering misconception in forensic genetics

A nearly universal practice among forensic DNA scientists includes mentioning an unrelated person as the possible alternative source of a DNA stain, when one in fact refers to an unknown person. Hence, experts typically express their conclusions with statements like: “The probability of the DNA evidence is X times higher if the suspect is the source of the trace than if another person unrelated to the suspect is the source of the trace.” Published forensic guidelines encourage such allusions to the unrelated person. However, as the authors show here, rational reasoning and population genetic principles do not require the conditioning of the evidential value on the unrelatedness between the unknown individual and the person of interest (e.g., a suspect). Surprisingly, this important semantic issue has been overlooked for decades, despite its potential to mislead the interpretation of DNA evidence by criminal justice system stakeholders.

Systematic evaluation of STRmix™ performance on degraded DNA profile data

This study examined the DNA degradation modeling capacity of STRmix™, a widely implemented DNA interpretation software program. As a part of the CAL DOJ STRmix™ v2.4 validation, a large volume of STR profile data was generated from intact template DNA exposed to DNase I for a series of increasing time intervals. The resulting degraded profile data was analyzed with STRmix™ v2.4, and the efficacy of the analysis was assessed, both in terms of how the degradation modeling parameter values from the STRmix™ analysis compared to ground truth values, and how the weight-of-evidence statistics calculated for degraded profiles compared to those calculated for corresponding intact profiles. An additional set of differentially degraded mixture data was generated in silico to further challenge the STRmix™ degradation model, as well as to determine the extent to which end-user adjustment of the model's application can assist in resolving analysis problems that arise when high levels of degradation are observed in a profile. This work demonstrates that the degradation model in STRmix™ is capable of addressing a wide range of degraded STR profile data. The assessment expands the range of samples that have been rigorously examined using probabilistic genotyping approaches, as called for by forensic advisory bodies such as the United States President's Council of Advisors on Science and Technology.

Four model variants within a continuous forensic DNA mixture interpretation framework: Effects on evidential inference and reporting

Continuous mixture interpretation methods that employ probabilistic genotyping to compute the Likelihood Ratio (LR) utilize more information than threshold-based systems. The continuous interpretation schemes described in the literature, however, do not all use the same underlying probabilistic model and standards outlining which probabilistic models may or may not be implemented into casework do not exist; thus, it is the individual forensic laboratory or expert that decides which model and corresponding software program to implement. For countries, such as the United States, with an adversarial legal system, one can envision a scenario where two probabilistic models are used to present the weight of evidence, and two LRs are presented by two experts. Conversely, if no independent review of the evidence is requested, one expert using one model may present one LR as there is no standard or guideline requiring the uncertainty in the LR estimate be presented. The choice of model determines the underlying probability calculation, and changes to it can result in non-negligible differences in the reported LR or corresponding verbal categorization presented to the trier-of-fact. In this paper, we study the impact of model differences on the LR and on the corresponding verbal expression computed using four variants of a continuous mixture interpretation method. The four models were tested five times each on 101, 1-, 2- and 3-person experimental samples with known contributors. For each sample, LRs were computed using the known contributor as the person of interest. In all four models, intra-model variability increased with an increase in the number of contributors and with a decrease in the contributor’s template mass. Inter-model variability in the associated verbal expression of the LR was observed in 32 of the 195 LRs used for comparison. Moreover, in 11 of these profiles there was a change from LR > 1 to LR < 1. These results indicate that modifications to existing continuous models do have the potential to significantly impact the final statistic, justifying the continuation of broad-based, large-scale, independent studies to quantify the limits of reliability and variability of existing forensically relevant systems.

The Probabilistic Genotyping Software STRmix: Utility and Evidence for its Validity

Forensic DNA interpretation is transitioning from manual interpretation based usually on binary decision-making toward computer-based systems that model the probability of the profile given different explanations for it, termed probabilistic genotyping (PG). Decision-making by laboratories to implement probability-based interpretation should be based on scientific principles for validity and information that supports its utility, such as criteria to support admissibility. The principles behind STRmix™ are outlined in this study and include standard mathematics and modeling of peak heights and variability in those heights. All PG methods generate a likelihood ratio (LR) and require the formulation of propositions. Principles underpinning formulations of propositions include the identification of reasonably assumed contributors. Substantial data have been produced that support precision, error rate, and reliability of PG, and in particular, STRmix™. A current issue is access to the code and quality processes used while coding. There are substantial data that describe the performance, strengths, and limitations of STRmix™, one of the available PG software.

Special issue on measuring and reporting the precision of forensic likelihood ratios: Introduction to the debate

The present paper introduces the Science & Justice virtual special issue on measuring and reporting the precision of forensic likelihood ratios - whether this should be done, and if so how. The focus is on precision (aka reliability) as opposed to accuracy (aka validity). The topic is controversial and different authors are expected to express a range of nuanced opinions. The present paper frames the debate, explaining the underlying problem and referencing classes of solutions proposed in the existing literature. The special issue will consist of a number of position papers, responses to those position papers, and replies to the responses.

Evaluating forensic biology results given source level propositions

The evaluation of forensic evidence can occur at any level within the hierarchy of propositions depending on the question being asked and the amount and type of information that is taken into account within the evaluation. Commonly DNA evidence is reported given propositions that deal with the sub-source level in the hierarchy, which deals only with the possibility that a nominated individual is a source of DNA in a trace (or contributor to the DNA in the case of a mixed DNA trace). We explore the use of information obtained from examinations, presumptive and discriminating tests for body fluids, DNA concentrations and some case circumstances within a Bayesian network in order to provide assistance to the Courts that have to consider propositions at source level. We use a scenario in which the presence of blood is of interest as an exemplar and consider how DNA profiling results and the potential for laboratory error can be taken into account. We finish with examples of how the results of these reports could be presented in court using either numerical values or verbal descriptions of the results.