GHEP-ISFG collaborative exercise on mixture profiles (GHEP-MIX06). Reporting conclusions: Results and evaluation

A tale of two PG systems: A comparison of the two most widely used continuous probabilistic genotyping systems in the United States

The development of probabilistic genotyping (PG) systems to quantitatively analyze DNA mixture samples has been transformative in forensic science. TrueAllele® Casework (TA) and STRmix™ (STRmix) are the two most widely used PG systems in the United States. The two systems were challenged with 48 two-, three-, and four-person mock casework samples, for a total of 152 likelihood ratio (LR) comparisons. TA and STRmix converged on the same result (supportive, non-supportive, or inconclusive) for ~91% of contributor-specific comparisons. Where moderate or substantial differences in log(LR) values were observed, 9% affected the conclusion of the reference association to the mixture. The PG systems exhibited high correlations for estimated contributor-specific template quantities (~92%) and log(LR)s produced (>88%). When the log(LR)s for only low-template contributors (<100 pg) were compared, the R2 value dropped to ~68% and the difference became statistically significant. Of the 14 contributor comparisons where the conclusion differed, two were contradictory (supportive vs. non-supportive) and 12 were either inconclusive versus non-supportive or inconclusive versus supportive. The differing results were likely due to dissimilarities in the mixture input file as STRmix uses a lab-defined analytical threshold (AT) and TA models to 10 RFUs for each electropherogram. When 7 of the 14 mixtures were reanalyzed by STRmix using a 10 RFU AT, the log(LR)s for the low-template contributors became more similar to TAs. This study shows that while both systems may produce accurate and calibrated LRs, their results can deviate, especially for low-template, degraded contributors, and the deviation is generally predictable.

The most consistent finding in forensic science is inconsistency

The most consistent finding in many forensic science domains is inconsistency (i.e., lack of reliability, reproducibility, repeatability, and replicability). The lack of consistency is a major problem, both from a scientific and a criminal justice point of view. Examining forensic conclusion data, from across many forensic domains, highlights the underlying cognitive issues and offers a better understanding of the issues and challenges. Such insights enable the development of ways to minimize these inconsistencies and move forward. The aim is to highlight the problem, so that it can be minimized and the reliability of forensic science evidence can be improved.

Internal Validation of MaSTR™ Probabilistic Genotyping Software for the Interpretation of 2–5 Person Mixed DNA Profiles

Mixed human deoxyribonucleic acid (DNA) samples present one of the most challenging pieces of evidence that a forensic analyst can encounter. When multiple contributors, stochastic amplification, and allele drop-out further complicate the mixture profile, interpretation by hand becomes unreliable and statistical analysis problematic. Probabilistic genotyping software has provided a tool to address complex mixture interpretation and provide likelihood ratios for defined sets of propositions. The MaSTR™ software is a fully continuous probabilistic system that considers a wide range of STR profile data to provide likelihood ratios on DNA mixtures. Mixtures with two to five contributors and a range of component ratios and allele peak heights were created to test the validity of MaSTR™ with data similar to real casework. Over 280 different mixed DNA profiles were used to perform more than 2600 analyses using different sets of propositions and numbers of contributors. The results of the analyses demonstrated that MaSTR™ provided accurate and precise statistical data on DNA mixtures with up to five contributors, including minor contributors with stochastic amplification effects. Tests for both Type I and Type II errors were performed. The findings in this study support that MaSTR™ is a robust tool that meets the current standards for probabilistic genotyping.

A Review of Probabilistic Genotyping Systems: EuroForMix, DNAStatistX and STRmix™

Probabilistic genotyping has become widespread. EuroForMix and DNAStatistX are both based upon maximum likelihood estimation using a γ model, whereas STRmix™ is a Bayesian approach that specifies prior distributions on the unknown model parameters. A general overview is provided of the historical development of probabilistic genotyping. Some general principles of interpretation are described, including: the application to investigative vs. evaluative reporting; detection of contamination events; inter and intra laboratory studies; numbers of contributors; proposition setting and validation of software and its performance. This is followed by details of the evolution, utility, practice and adoption of the software discussed.

Examining performance and likelihood ratios for two likelihood ratio systems using the PROVEDIt dataset

A likelihood ratio (LR) system is defined as the entire pipeline of the measurement and interpretation processes where probabilistic genotyping software (PGS) is a piece of the whole LR system. To gain understanding on how two LR systems perform, a total of 154 two-person, 147 three-person, and 127 four-person mixture profiles of varying DNA quality, DNA quantity, and mixture ratios were obtained from the filtered (.CSV) files of the GlobalFiler 29 cycles 15s PROVEDIt dataset and deconvolved in two independently developed fully continuous programs, STRmix v2.6 and EuroForMix v2.1.0. Various parameters were set in each software and LR computations obtained from the two software were based on same/fixed EPG features, same pair of propositions, number of contributors, theta, and population allele frequencies. The ability of each LR system to discriminate between contributor (H1-true) and non-contributor (H2-true) scenarios was evaluated qualitatively and quantitatively. Differences in the numeric LR values and their corresponding verbal classifications between the two LR systems were compared. The magnitude of the differences in the assigned LRs and the potential explanations for the observed differences greater than or equal to 3 on the log10 scale were described. Cases of LR < 1 for H1-true tests and LR > 1 for H2-true tests were also discussed. Our intent is to demonstrate the value of using a publicly available ground truth known mixture dataset to assess discrimination performance of any LR system and show the steps used to understand similarities and differences between different LR systems. We share our observations with the forensic community and describe how examining more than one PGS with similar discrimination power can be beneficial, help analysts compare interpretation especially with low-template profiles or minor contributor cases, and be a potential additional diagnostic check even if software in use does contain certain diagnostic statistics as part of the output.

Statistefix 4.0: A novel probabilistic software tool

Latest innovations indicate that continuous tools are promising DNA trace assessment methods. In this study, we present the continuous software solution Statistefix 4.0. The software supports DNA experts in deducing DNA profiles for database queries and can help to preselect DNA samples suitable for further processing using advanced probabilistic search engines. The novel tool weights genotype contributions and deduces major contributors from high- and low-quality DNA traces. Peak height, degradation, stutter as well as allelic drop-in/-out events are incorporated in the statistical model. We analyzed reference and casework samples as well as artificially generated mixture samples for software evaluation. The tool offers the completely automated assessment of reference and mixture samples. Deconvolution outcomes of mixtures are compared with EuroForMix, GenoProof Mixture 3 and STRmix™. Data show that Statistefix 4.0 is as successful as analogously tested and implemented software. Deduced DNA profiles from casework samples highlight the potential benefit in routine casework. Statistefix 4.0 is freely available, works with replicates of different autosomal kits and enables bulk sample processing. This inter-laboratory study includes a variety of sample types and indicates a timesaving, robust and easily implemented software that supports DNA analysts in evaluating DNA traces.

Towards developing forensically relevant single-cell pipelines by incorporating direct-to-PCR extraction: compatibility, signal quality, and allele detection

Current analysis of forensic DNA stains relies on the probabilistic interpretation of bulk-processed samples that represent mixed profiles consisting of an unknown number of potentially partial representations of each contributor. Single-cell methods, in contrast, offer a solution to the forensic DNA mixture problem by incorporating a step that separates cells before extraction. A forensically relevant single-cell pipeline relies on efficient direct-to-PCR extractions that are compatible with standard downstream forensic reagents. Here we demonstrate the feasibility of implementing single-cell pipelines into the forensic process by exploring four metrics of electropherogram (EPG) signal quality-i.e., allele detection rates, peak heights, peak height ratios, and peak height balance across low- to high-molecular-weight short tandem repeat (STR) markers-obtained with four direct-to-PCR extraction treatments and a common post-PCR laboratory procedure. Each treatment was used to extract DNA from 102 single buccal cells, whereupon the amplification reagents were immediately added to the tube and the DNA was amplified/injected using post-PCR conditions known to elicit a limit of detection (LoD) of one DNA molecule. The results show that most cells, regardless of extraction treatment, rendered EPGs with at least a 50% true positive allele detection rate and that allele drop-out was not cell independent. Statistical tests demonstrated that extraction treatments significantly impacted all metrics of EPG quality, where the Arcturus® PicoPure™ extraction method resulted in the lowest median allele drop-out rate, highest median average peak height, highest median average peak height ratio, and least negative median values of EPG sloping for GlobalFiler™ STR loci amplified at half volume. We, therefore, conclude the feasibility of implementing single-cell pipelines for casework purposes and demonstrate that inferential systems assuming cell independence will not be appropriate in the probabilistic interpretation of a collection of single-cell EPGs.

Cognitive and Human Factors in Expert Decision Making: Six Fallacies and the Eight Sources of Bias

Fallacies about the nature of biases have shadowed a proper cognitive understanding of biases and their sources, which in turn lead to ways that minimize their impact. Six such fallacies are presented: it is an ethical issue, only applies to "bad apples", experts are impartial and immune, technology eliminates bias, blind spot, and the illusion of control. Then, eight sources of bias are discussed and conceptualized within three categories: (A) factors that relate to the specific case and analysis, which include the data, reference materials, and contextual information, (B) factors that relate to the specific person doing the analysis, which include past experience base rates, organizational factors, education and training, and personal factors, and lastly, (C) cognitive architecture and human nature that impacts all of us. These factors can impact what the data are (e.g., how data are sampled and collected, or what is considered as noise and therefore disregarded), the actual results (e.g., decisions on testing strategies, how analysis is conducted, and when to stop testing), and the conclusions (e.g., interpretation of the results). The paper concludes with specific measures that can minimize these biases.

The Error in "Error Rate": Why Error Rates Are So Needed, Yet So Elusive

Establishing error rates is crucial for knowing how well one is performing, determining whether improvement is needed, measuring whether interventions are effective, as well as for providing transparency. However, the flurry of activities in establishing error rates for the forensic sciences has largely overlooked some fundamental issues that make error rates a problematic construct and limit the ability to obtain a meaningful error rate. These include knowing the ground truth, establishing appropriate databases, determining what counts as an error, characterizing what is an acceptable error rate, ecological validity, and transparency within the adversarial legal system. Without addressing these practical and theoretical challenges, the very notion of a meaningful error rate is limited.

Interpol review of forensic biology and forensic DNA typing 2016-2019

This review paper covers the forensic-relevant literature in biological sciences from 2016 to 2019 as a part of the 19th Interpol International Forensic Science Managers Symposium. The review papers are also available at the Interpol website at: https://www.interpol.int/content/download/14458/file/Interpol%20Review%20Papers%202019.pdf.

Perceptions and estimates of error rates in forensic science: A survey of forensic analysts

Every scientific technique features some error, and legal standards for the admissibility of scientific evidence (e.g., Daubert v. Merrill Dow Pharmaceuticals, Inc., 1993; Kumho Tire Co v. Carmichael, 1999) guide trial courts to consider known error rates. However, recent reviews of forensic science conclude that error rates for some common techniques are not well-documented or even established (e.g., NAS, 2009; PCAST, 2016). Furthermore, many forensic analysts have historically denied the presence of error in their field. Therefore, it is important to establish what forensic scientists actually know or believe about errors rates in their disciplines. We surveyed 183 practicing forensic analysts to examine what they think and estimate about error rates in their various disciplines. Results revealed that analysts perceive all types of errors to be rare, with false positive errors even more rare than false negatives. Likewise, analysts typically reported that they prefer to minimize the risk of false positives over false negatives. Most analysts could not specify where error rates for their discipline were documented or published. Their estimates of error in their fields were widely divergent - with some estimates unrealistically low.

STRmix™ collaborative exercise on DNA mixture interpretation

An intra and inter-laboratory study using the probabilistic genotyping (PG) software STRmix™ is reported. Two complex mixtures from the PROVEDIt set, analysed on an Applied Biosystems™ 3500 Series Genetic Analyzer, were selected. 174 participants responded. For Sample 1 (low template, in the order of 200 rfu for major contributors) five participants described the comparison as inconclusive with respect to the POI or excluded him. Where LRs were assigned, the point estimates ranging from 2 × 10⁴ to 8 × 10⁶. For Sample 2 (in the order of 2000 rfu for major contributors), LRs ranged from 2 × 10²⁸ to 2 × 10²⁹. Where LRs were calculated, the differences between participants can be attributed to (from largest to smallest impact): This study demonstrates a high level of repeatability and reproducibility among the participants. For those results that differed from the mode, the differences in LR were almost always minor or conservative.

Probabilistic genotyping software: An overview

The interpretation of mixed profiles from DNA evidentiary material is one of the more challenging duties of the forensic scientist. Traditionally, analysts have used a "binary" approach to interpretation where inferred genotypes are either included or excluded from the mixture using a stochastic threshold and other biological parameters such as heterozygote balance, mixture ratio, and stutter ratios. As the sensitivity of STR multiplexes and capillary electrophoresis instrumentation improved over the past 25 years, coupled with the change in the type of evidence being submitted for analysis (from high quality and quantity (often single-source) stains to low quality and quantity (often mixed) "touch" samples), the complexity of DNA profile interpretation has equally increased. This review provides a historical perspective on the movement from binary methods of interpretation to probabilistic methods of interpretation. We describe the two approaches to probabilistic genotyping (semi-continuous and fully continuous) and address issues such as validation and court acceptance. Areas of future needs for probabilistic software are discussed.

The Probabilistic Genotyping Software STRmix: Utility and Evidence for its Validity

Forensic DNA interpretation is transitioning from manual interpretation based usually on binary decision-making toward computer-based systems that model the probability of the profile given different explanations for it, termed probabilistic genotyping (PG). Decision-making by laboratories to implement probability-based interpretation should be based on scientific principles for validity and information that supports its utility, such as criteria to support admissibility. The principles behind STRmix™ are outlined in this study and include standard mathematics and modeling of peak heights and variability in those heights. All PG methods generate a likelihood ratio (LR) and require the formulation of propositions. Principles underpinning formulations of propositions include the identification of reasonably assumed contributors. Substantial data have been produced that support precision, error rate, and reliability of PG, and in particular, STRmix™. A current issue is access to the code and quality processes used while coding. There are substantial data that describe the performance, strengths, and limitations of STRmix™, one of the available PG software.