A series of recommended tests when validating probabilistic DNA profile interpretation software

'Low' LRs obtained from DNA mixtures: On calibration and discrimination performance of probabilistic genotyping software

The validity of a probabilistic genotyping (PG) system is typically demonstrated by following international guidelines for the developmental and internal validation of PG software. These guidelines mainly focus on discriminatory power. Very few studies have reported with metrics that depend on calibration of likelihood ratio (LR) systems. In this study, discriminatory power as well as various calibration metrics, such as Empirical Cross-Entropy (ECE) plots, pool adjacent violator (PAV) plots, log likelihood ratio cost (Cllr and Cllrcal), fiducial calibration discrepancy plots, and Turing' expectation were examined using the publicly-available PROVEDIt dataset. The aim was to gain deeper insight into the performance of a variety of PG software in the 'lower' LR ranges (∼LR 1-10,000), with focus on DNAStatistX and EuroForMix which use maximum likelihood estimation (MLE). This may be a driving force for the end users to reconsider current LR thresholds for reporting. In previous studies, overstated 'low' LRs were observed for these PG software. However, applying (arbitrarily) high LR thresholds for reporting wastes relevant evidential value. This study demonstrates, based on calibration performance, that previously reported LR thresholds can be lowered or even discarded. Considering LRs >1, there was no evidence for miscalibration performance above LR ∼1000 when using Fst 0.01. Below this LR value, miscalibration was observed. Calibration performance generally improved with the use of Fst 0.03, but the extent of this was dependent on the dataset: results ranged from miscalibration up to LR ∼100 to no evidence of miscalibration alike PG software using different methods to model peak height, HMC and STRmix. This study demonstrates that practitioners using MLE-based models should be careful when low LR ranges are reported, though applying arbitrarily high LR thresholds is discouraged. This study also highlights various calibration metrics that are useful in understanding the performance of a PG system.

A collaborative study on the precision of the Markov chain Monte Carlo algorithms used for DNA profile interpretation

Several fully continuous probabilistic genotyping software (PGS) use Markov chain Monte Carlo algorithms (MCMC) to assign weights to different proposed genotype combinations at a locus. Replicate interpretations of the same profile in these software are expected not to produce identical weights and likelihood ratio (LR) values due to the Monte Carlo aspect. This paper reports a detailed precision study under reproducibility conditions conducted as a collaborative exercise across the National Institute of Standards and Technology (NIST), Federal Bureau of Investigation (FBI), and Institute of Environmental Science and Research (ESR). Replicate interpretations generated across the three laboratories used the same input files, software version, and settings but different random number seed and different computers. This work demonstrates that using different computers to analyze replicate interpretations does not contribute to any variations in LR values. The study quantifies the magnitude of differences in the assigned LRs that is only due to run-to-run MCMC variability and addresses the potential explanations for the observed differences.

Decreased accuracy of forensic DNA mixture analysis for groups with lower genetic diversity

Forensic investigation of DNA samples from multiple contributors has become commonplace. These complex analyses use statistical frameworks accounting for multiple levels of uncertainty in allelic contributions from different individuals, particularly for samples containing few molecules of DNA. These methods have been thoroughly tested along some axes of variation, but less attention has been paid to accuracy across human genetic variation. Here, we quantify the accuracy of DNA mixture analysis over 244 human groups. We find higher false inclusion rates for mixtures with more contributors, and for groups with lower genetic diversity. Even for two-contributor mixtures where one contributor is known and the reference group is correctly specified, false inclusion rates are 1e-5 or higher for 56 out of 244 groups. This means that, depending on multiple testing, some false inclusions may be expected. These false positives could be lessened with more selective and conservative use of DNA mixture analysis.

Gemeinsame Empfehlungen der Projektgruppe „Biostatistische DNA-Berechnungen“ und der Spurenkommission zur biostatistischen Bewertung forensischer DNA-analytischer Befunde mit vollkontinuierlichen Modellen (VKM)

Die biostatistische Bewertung DNA-analytischer Befunde unterstützt Gerichte bei der Einschätzung des Beweiswertes hinsichtlich einer möglichen Spurenbeteiligung durch eine zu betrachtende Person (engl. „Person Of Interest“; POI). Um die Vergleichbarkeit derartiger Berechnungen auf Grundlage etablierter wissenschaftlicher Standards zu gewährleisten, wurden bereits in der Vergangenheit entsprechende Empfehlungen im nationalen Konsens formuliert.

Mit Einführung sog. vollkontinuierlicher Modelle (VKM) für die probabilistische Genotypisierung, die u. a. die Signalintensitäten eines Elektropherogramms berücksichtigen, wurde eine Ergänzung zu den damaligen Empfehlungen erforderlich. VKM erlauben eine biostatistische Bewertung von Spuren mit möglichen Drop-in- und Drop-out-Ereignissen und wahrscheinlichkeitsbasierte Prognosen der zu einer Mischspur beitragenden Genotypen („Deconvolution“).

Die vorliegende Veröffentlichung enthält Empfehlungen zum Einsatz VKM-basierter Software und zur Berichterstattung vollkontinuierlicher LR-Werte (engl. „Fully Continuous Likelihood Ratios“; LRfc). Sie empfiehlt bei schwierig zu interpretierenden Befunden eine VKM-Berechnung zur Bewertung einer Spurenlegerschaft. Die VKM-Berechnung ersetzt die bisher in Ausnahmefällen als hinnehmbar erachtete Vorgehensweise einer binären Berechnung unter Ausklammern einzelner Merkmalssysteme. Der Einsatz von VKM erfordert eine umfassende Anwenderschulung sowie eine Validierung und Verifizierung gemäß den Vorgaben der Programmanbieter. Mit der Empfehlung von LRfc-Schwellenwerten soll eine sichere, vergleichbare Anwendung von VKM gewährleistet werden.

A Review of Probabilistic Genotyping Systems: EuroForMix, DNAStatistX and STRmix™

Probabilistic genotyping has become widespread. EuroForMix and DNAStatistX are both based upon maximum likelihood estimation using a γ model, whereas STRmix™ is a Bayesian approach that specifies prior distributions on the unknown model parameters. A general overview is provided of the historical development of probabilistic genotyping. Some general principles of interpretation are described, including: the application to investigative vs. evaluative reporting; detection of contamination events; inter and intra laboratory studies; numbers of contributors; proposition setting and validation of software and its performance. This is followed by details of the evolution, utility, practice and adoption of the software discussed.

A comparison of likelihood ratios obtained from EuroForMix and STRmix™

Likelihood ratios (LR) differences between the probabilistic genotyping software EuroForMix and STRmix™ are examined. After considering differences in the allele probabilities, the LRs from both software for an unambiguous single-source profile were identical (four significant figures). LRs from both software for an unambiguous single-source profile with alleles previously unseen in the allele frequency database (rare alleles) were the same (three significant figures) for θ = 0.01. Due to differences in the minimum allele frequencies, the LRs differed by three orders of magnitude when θ = 0. For both software, the LRs for a single-source dilution series decreased as the input amount decreased. The LRs from both software were within an order of magnitude for known contributors. The largest difference was where the target input amount was 0.0156 ng: The LR_EuroForMix was 2.1 × 10²⁵ and the LR_STRmix was 8.0 × 10²⁴ . Both software show similar LR behavior with respect to mixture ratio. For two person mixtures the LR increases for both the major and the minor as the ratio moves away from 1:1. The LR for the major stabilizes at about 3:1 whereas the LR for the minor reaches its maximum at about 3:1 and then declines. Greater differences in LR were observed between EuroForMix and STRmix™ for mixtures. One-hundred and twenty-nine mixtures from the PROVEDIt dataset were compared. LRs for 84% of the comparisons for known contributors without rare alleles were within two orders of magnitude. Five divergent results were investigated, and a manual intervention approach was applied where appropriate.

Examining performance and likelihood ratios for two likelihood ratio systems using the PROVEDIt dataset

A likelihood ratio (LR) system is defined as the entire pipeline of the measurement and interpretation processes where probabilistic genotyping software (PGS) is a piece of the whole LR system. To gain understanding on how two LR systems perform, a total of 154 two-person, 147 three-person, and 127 four-person mixture profiles of varying DNA quality, DNA quantity, and mixture ratios were obtained from the filtered (.CSV) files of the GlobalFiler 29 cycles 15s PROVEDIt dataset and deconvolved in two independently developed fully continuous programs, STRmix v2.6 and EuroForMix v2.1.0. Various parameters were set in each software and LR computations obtained from the two software were based on same/fixed EPG features, same pair of propositions, number of contributors, theta, and population allele frequencies. The ability of each LR system to discriminate between contributor (H1-true) and non-contributor (H2-true) scenarios was evaluated qualitatively and quantitatively. Differences in the numeric LR values and their corresponding verbal classifications between the two LR systems were compared. The magnitude of the differences in the assigned LRs and the potential explanations for the observed differences greater than or equal to 3 on the log10 scale were described. Cases of LR < 1 for H1-true tests and LR > 1 for H2-true tests were also discussed. Our intent is to demonstrate the value of using a publicly available ground truth known mixture dataset to assess discrimination performance of any LR system and show the steps used to understand similarities and differences between different LR systems. We share our observations with the forensic community and describe how examining more than one PGS with similar discrimination power can be beneficial, help analysts compare interpretation especially with low-template profiles or minor contributor cases, and be a potential additional diagnostic check even if software in use does contain certain diagnostic statistics as part of the output.

The analogy between DNA kinship and DNA mixture evaluation, with applications for the interpretation of likelihood ratios produced by possibly imperfect models

Two main applications of forensic DNA analysis are the investigation of possible relatedness and the investigation whether a person left DNA in a trace. Both of these are usually carried out by the calculation of likelihood ratios. In the kinship case, it is standard to let the likelihood ratio express the support in favour of the investigated relatedness versus no relatedness, and in the investigation of traces, one by default compares the hypothesis that the person of interest contributed DNA, versus that he is unrelated to any of the actual contributors. In both cases however, we can also view the probabilistic procedure as an inference of the profile of the person we look for: in other words, in both cases we carry out probabilistic genotyping. In this article we use this general analogy to develop various more specific analogies between kinship and mixture likelihood ratios. These analogies help to understand the concepts that play a role, and also to understand the importance of the statistical modeling needed for DNA mixtures. In this article, we apply our findings to consider what we can and cannot conclude from a likelihood ratio in favour of contribution to a mixed DNA profile, if that is computed by a model whose specifics are not entirely known to us, or where we do not know whether they provide a good description of the stochastic effects involved in the generation of DNA trace profiles. We show that, if unrelated individuals are adequately modeled, we can give bounds on how often LR's coming from certain types of black box models may arise, both for persons who are actual contributors and who are unrelated. In particular we show that no model, provided it satisfies basic requirements, can overestimate the evidence found for actual contributors both often and strongly.

Multi-laboratory validation of DNAxs including the statistical library DNAStatistX

This study describes a multi-laboratory validation of DNAxs, a DNA eXpert System for the data management and probabilistic interpretation of DNA profiles [1], and its statistical library DNAStatistX to which, besides the organising laboratory, four laboratories participated. The software was modified to read multiple data formats and the study was performed prior to the release of the software to the forensic community. The first exercise explored all main functionalities of DNAxs with feedback on user-friendliness, installation and general performance. Next, every laboratory performed likelihood ratio (LR) calculations using their own dataset and a dataset provided by the organising laboratory. The organising laboratory performed LR calculations using all datasets. The datasets were generated with different STR typing kits or analysis systems and consisted of samples varying in DNA amounts, mixture ratios, number of contributors and drop-out level. Hypothesis sets had the correct, under- and over-assigned number of contributors and true and false donors as person of interest. When comparing the results between laboratories, the LRs were foremost within one unit on log10 scale. The few LR results that deviated more had differences for the parameters estimated by the optimizer within DNAStatistX. Some of these were indicated by failed iteration results, others by a failed model validation, since unrealistic hypotheses were included. When these results that do not meet the quality criteria were excluded, as is in accordance with interpretation guidelines, none of the analyses in the different laboratories yielded a different statement in the casework report. Nonetheless, changes in software parameters were sought that minimized differences in outcomes, which made the DNAStatistX module more robust. Overall, the software was found intuitive, user-friendly and valid for use in multiple laboratories.

Systematic evaluation of STRmix™ performance on degraded DNA profile data

This study examined the DNA degradation modeling capacity of STRmix™, a widely implemented DNA interpretation software program. As a part of the CAL DOJ STRmix™ v2.4 validation, a large volume of STR profile data was generated from intact template DNA exposed to DNase I for a series of increasing time intervals. The resulting degraded profile data was analyzed with STRmix™ v2.4, and the efficacy of the analysis was assessed, both in terms of how the degradation modeling parameter values from the STRmix™ analysis compared to ground truth values, and how the weight-of-evidence statistics calculated for degraded profiles compared to those calculated for corresponding intact profiles. An additional set of differentially degraded mixture data was generated in silico to further challenge the STRmix™ degradation model, as well as to determine the extent to which end-user adjustment of the model's application can assist in resolving analysis problems that arise when high levels of degradation are observed in a profile. This work demonstrates that the degradation model in STRmix™ is capable of addressing a wide range of degraded STR profile data. The assessment expands the range of samples that have been rigorously examined using probabilistic genotyping approaches, as called for by forensic advisory bodies such as the United States President's Council of Advisors on Science and Technology.

Probabilistic approaches to interpreting two-person DNA mixtures from post-coital specimens

Biological samples submitted for sexual assault investigation typically involve mixtures of DNA from the victim and the assailant/s. Providing a statistical weight to such evidence may be mathematically complex and may be affected by subjective judgment of a human analyst. Software tools have been developed to address these issues. To contribute towards improving the system for routine DNA testing of sexual assault cases, we evaluated two likelihood ratio (LR) approaches: a semi-continuous model using LRmix Studio and a fully continuous approach employed in STRmix™ for interpreting two-person DNA mixtures. LRs conditioned on the presence of the receptive partner's DNA were calculated for a total of 102 two-person DNA samples from simulated mixtures and various post-coital samples. Our results highlight the importance of maximising information provided into the LR calculation to generate strong support for the true hypothesis. This can be achieved by recovering sufficient DNA from a sample to minimise risk of drop-out and increase peak intensities and by implementing a statistical model that utilises as much of the electropherogram information as possible. LRmix is open-source and can handle profiles with allelic drop-out and drop-ins, however stuttering is not modelled and requires manual removal by a DNA analyst especially for mixtures with low template components. STRmix™ makes effective use of all available information by incorporating into its biological model complicating aspects of a DNA profile such as degradation, allele drop-out and drop-in, stutters, and peak height variability.

STRmix™ collaborative exercise on DNA mixture interpretation

An intra and inter-laboratory study using the probabilistic genotyping (PG) software STRmix™ is reported. Two complex mixtures from the PROVEDIt set, analysed on an Applied Biosystems™ 3500 Series Genetic Analyzer, were selected. 174 participants responded. For Sample 1 (low template, in the order of 200 rfu for major contributors) five participants described the comparison as inconclusive with respect to the POI or excluded him. Where LRs were assigned, the point estimates ranging from 2 × 10⁴ to 8 × 10⁶. For Sample 2 (in the order of 2000 rfu for major contributors), LRs ranged from 2 × 10²⁸ to 2 × 10²⁹. Where LRs were calculated, the differences between participants can be attributed to (from largest to smallest impact): This study demonstrates a high level of repeatability and reproducibility among the participants. For those results that differed from the mode, the differences in LR were almost always minor or conservative.

Probabilistic genotyping software: An overview

The interpretation of mixed profiles from DNA evidentiary material is one of the more challenging duties of the forensic scientist. Traditionally, analysts have used a "binary" approach to interpretation where inferred genotypes are either included or excluded from the mixture using a stochastic threshold and other biological parameters such as heterozygote balance, mixture ratio, and stutter ratios. As the sensitivity of STR multiplexes and capillary electrophoresis instrumentation improved over the past 25 years, coupled with the change in the type of evidence being submitted for analysis (from high quality and quantity (often single-source) stains to low quality and quantity (often mixed) "touch" samples), the complexity of DNA profile interpretation has equally increased. This review provides a historical perspective on the movement from binary methods of interpretation to probabilistic methods of interpretation. We describe the two approaches to probabilistic genotyping (semi-continuous and fully continuous) and address issues such as validation and court acceptance. Areas of future needs for probabilistic software are discussed.

The Probabilistic Genotyping Software STRmix: Utility and Evidence for its Validity

Forensic DNA interpretation is transitioning from manual interpretation based usually on binary decision-making toward computer-based systems that model the probability of the profile given different explanations for it, termed probabilistic genotyping (PG). Decision-making by laboratories to implement probability-based interpretation should be based on scientific principles for validity and information that supports its utility, such as criteria to support admissibility. The principles behind STRmix™ are outlined in this study and include standard mathematics and modeling of peak heights and variability in those heights. All PG methods generate a likelihood ratio (LR) and require the formulation of propositions. Principles underpinning formulations of propositions include the identification of reasonably assumed contributors. Substantial data have been produced that support precision, error rate, and reliability of PG, and in particular, STRmix™. A current issue is access to the code and quality processes used while coding. There are substantial data that describe the performance, strengths, and limitations of STRmix™, one of the available PG software.

Accurate assessment of the weight of evidence for DNA mixtures by integrating the likelihood ratio

Several methods exist for weight of evidence calculations on DNA mixtures. Especially if dropout is a possibility, it may be difficult to estimate mixture specific parameters needed for the evaluation. For semi-continuous models, the LR for a person to have contributed to a mixture depends on the specified number of contributors and the probability of dropout for each. We show here that, for the semi-continuous model that we consider, the weight of evidence can be accurately obtained by applying the standard statistical technique of integrating the likelihood ratio against the parameter likelihoods obtained from the mixture data. This method takes into account all likelihood ratios belonging to every choice of parameters, but LR's belonging to parameters that provide a better explanation to the mixture data put in more weight into the final result. We therefore avoid having to estimate the number of contributors or their probabilities of dropout, and let the whole evaluation depend on the mixture data and the allele frequencies, which is a practical advantage as well as a gain in objectivity. Using simulated mixtures, we compare the LR obtained in this way with the best informed LR, i.e., the LR using the parameters that were used to generate the data, and show that results obtained by integration of the LR approximate closely these ideal values. We investigate both contributors and non-contributors for mixtures with various numbers of contributors. For contributors we always obtain a result close to the best informed LR whereas non-contributors are excluded more strongly if a smaller dropout probability is imposed for them. The results therefore naturally lead us to reconsider what we mean by a contributor, or by the number of contributors.

The future of forensic DNA analysis

The author's thoughts and opinions on where the field of forensic DNA testing is headed for the next decade are provided in the context of where the field has come over the past 30 years. Similar to the Olympic motto of 'faster, higher, stronger', forensic DNA protocols can be expected to become more rapid and sensitive and provide stronger investigative potential. New short tandem repeat (STR) loci have expanded the core set of genetic markers used for human identification in Europe and the USA. Rapid DNA testing is on the verge of enabling new applications. Next-generation sequencing has the potential to provide greater depth of coverage for information on STR alleles. Familial DNA searching has expanded capabilities of DNA databases in parts of the world where it is allowed. Challenges and opportunities that will impact the future of forensic DNA are explored including the need for education and training to improve interpretation of complex DNA profiles.

Evaluation of low-template DNA profiles using peak heights

In recent years statistical models for the analysis of complex (low-template and/or mixed) DNA profiles have moved from using only presence/absence information about allelic peaks in an electropherogram, to quantitative use of peak heights. This is challenging because peak heights are very variable and affected by a number of factors. We present a new peak-height model with important novel features, including over- and double-stutter, and a new approach to dropin. Our model is incorporated in open-source

Lab Retriever: a software tool for calculating likelihood ratios incorporating a probability of drop-out for forensic DNA profiles

Background

Technological advances have enabled the analysis of very small amounts of DNA in forensic cases. However, the DNA profiles from such evidence are frequently incomplete and can contain contributions from multiple individuals. The complexity of such samples confounds the assessment of the statistical weight of such evidence. One approach to account for this uncertainty is to use a likelihood ratio framework to compare the probability of the evidence profile under different scenarios. While researchers favor the likelihood ratio framework, few open-source software solutions with a graphical user interface implementing these calculations are available for practicing forensic scientists.

Results

To address this need, we developed Lab Retriever, an open-source, freely available program that forensic scientists can use to calculate likelihood ratios for complex DNA profiles. Lab Retriever adds a graphical user interface, written primarily in JavaScript, on top of a C++ implementation of the previously published R code of Balding. We redesigned parts of the original Balding algorithm to improve computational speed. In addition to incorporating a probability of allelic drop-out and other critical parameters, Lab Retriever computes likelihood ratios for hypotheses that can include up to four unknown contributors to a mixed sample. These computations are completed nearly instantaneously on a modern PC or Mac computer.

Conclusions

Lab Retriever provides a practical software solution to forensic scientists who wish to assess the statistical weight of evidence for complex DNA profiles. Executable versions of the program are freely available for Mac OSX and Windows operating systems.

Electronic supplementary material

The online version of this article (doi:10.1186/s12859-015-0740-8) contains supplementary material, which is available to authorized users.