An assessment of the performance of the probabilistic genotyping software EuroForMix: Trends in likelihood ratios and analysis of Type I & II errors

A collaborative study on the precision of the Markov chain Monte Carlo algorithms used for DNA profile interpretation

Several fully continuous probabilistic genotyping software (PGS) use Markov chain Monte Carlo algorithms (MCMC) to assign weights to different proposed genotype combinations at a locus. Replicate interpretations of the same profile in these software are expected not to produce identical weights and likelihood ratio (LR) values due to the Monte Carlo aspect. This paper reports a detailed precision study under reproducibility conditions conducted as a collaborative exercise across the National Institute of Standards and Technology (NIST), Federal Bureau of Investigation (FBI), and Institute of Environmental Science and Research (ESR). Replicate interpretations generated across the three laboratories used the same input files, software version, and settings but different random number seed and different computers. This work demonstrates that using different computers to analyze replicate interpretations does not contribute to any variations in LR values. The study quantifies the magnitude of differences in the assigned LRs that is only due to run-to-run MCMC variability and addresses the potential explanations for the observed differences.

'Low' LRs obtained from DNA mixtures: On calibration and discrimination performance of probabilistic genotyping software

The validity of a probabilistic genotyping (PG) system is typically demonstrated by following international guidelines for the developmental and internal validation of PG software. These guidelines mainly focus on discriminatory power. Very few studies have reported with metrics that depend on calibration of likelihood ratio (LR) systems. In this study, discriminatory power as well as various calibration metrics, such as Empirical Cross-Entropy (ECE) plots, pool adjacent violator (PAV) plots, log likelihood ratio cost (Cllr and Cllrcal), fiducial calibration discrepancy plots, and Turing' expectation were examined using the publicly-available PROVEDIt dataset. The aim was to gain deeper insight into the performance of a variety of PG software in the 'lower' LR ranges (∼LR 1-10,000), with focus on DNAStatistX and EuroForMix which use maximum likelihood estimation (MLE). This may be a driving force for the end users to reconsider current LR thresholds for reporting. In previous studies, overstated 'low' LRs were observed for these PG software. However, applying (arbitrarily) high LR thresholds for reporting wastes relevant evidential value. This study demonstrates, based on calibration performance, that previously reported LR thresholds can be lowered or even discarded. Considering LRs >1, there was no evidence for miscalibration performance above LR ∼1000 when using Fst 0.01. Below this LR value, miscalibration was observed. Calibration performance generally improved with the use of Fst 0.03, but the extent of this was dependent on the dataset: results ranged from miscalibration up to LR ∼100 to no evidence of miscalibration alike PG software using different methods to model peak height, HMC and STRmix. This study demonstrates that practitioners using MLE-based models should be careful when low LR ranges are reported, though applying arbitrarily high LR thresholds is discouraged. This study also highlights various calibration metrics that are useful in understanding the performance of a PG system.

An overview of log likelihood ratio cost in forensic science - Where is it used and what values can we expect?

There is increasing support for reporting evidential strength as a likelihood ratio (LR) and increasing interest in (semi-)automated LR systems. The log-likelihood ratio cost (Cllr) is a popular metric for such systems, penalizing misleading LRs further from 1 more. Cllr = 0 indicates perfection while Cllr = 1 indicates an uninformative system. However, beyond this, what constitutes a "good" Cllr is unclear. Aiming to provide handles on when a Cllr is "good", we studied 136 publications on (semi-)automated LR systems. Results show Cllr use heavily depends on the field, e.g., being absent in DNA analysis. Despite more publications on automated LR systems over time, the proportion reporting Cllr remains stable. Noticeably, Cllr values lack clear patterns and depend on the area, analysis and dataset. As LR systems become more prevalent, comparing them becomes crucial. This is hampered by different studies using different datasets. We advocate using public benchmark datasets to advance the field.

Single cell genomics applications in forensic science: Current state and future directions

Standard methods of mixture analysis involve subjecting a dried crime scene sample to a "bulk" DNA extraction method such that the resulting isolate compromises a homogenized DNA mixture from the individual donors. If, however, instead of bulk DNA extraction, a sufficient number of individual cells from the mixed stain are subsampled prior to genetic analysis then it should be possible to recover highly probative single source, non-mixed scDNA profiles from each of the donors. This approach can detect low DNA level minor donors to a mixture that otherwise would not be identified using standard methods and can also resolve rare mixtures comprising first degree relatives and thereby also prevent the false inclusion of non-donor relatives. This literature landscape review and associated commentary reports on the history and increasing interest in current and potential future applications of scDNA in forensic genomics, and critically evaluates opportunities and impediments to further progress.

ProbRank: An efficient DNA database search method for complex mixtures per a quantitative likelihood ratio model

Searching a DNA Database with a DNA profile from an evidentiary trace can provide investigative leads in a forensic case. Various searching approaches exist such as conventional methods based on matching alleles or more advanced methods computing likelihood ratios (LR) while considering drop-in and drop-out. Here we examine the potential of using a quantitative LR model (EuroForMix model incorporated in ProbRank method) that takes peak heights into account in comparison to a qualitative LR model (LRmix model implemented in SmartRank method). Both methods present DNA database candidates in order of decreasing LR. Especially regarding minor contributors in complex mixtures, the method using the quantitative model outperforms the method using the qualitative model in terms of sensitivity and specificity as more true donors and less adventitious matches are retrieved. ProbRank is to be implemented in DNAStatistX and is sufficiently fast for daily use.

Likelihood Ratio Calculation Using LRmix Studio

LRmix Studio performs statistical analyses on forensic casework samples by calculating a likelihood ratio (LR) following a semi-continuous, unrestricted approach. The software utilizes a basic probabilistic model allowing the comparison of two alternative hypotheses regarding the evidence profile to include known and/or unknown contributors, for a maximum of a 4-person mixture. Other statistical factors that are included in this model are the incorporation of multiple probability of drop-out values, probability of drop-in, a correction factor for population substructure, assumed contributor inclusion, and inclusion of an unknown relative in the defense hypothesis. A range of plausible probability of drop-out values can be calculated for various contributors and hypotheses based on a Monte Carlo probability method and included in the likelihood ratio calculation. The software also includes several ways to test the validity and robustness of the probabilistic model. A sensitivity analysis can be performed by calculating likelihood ratios for the given profile against a range of drop-out values. Additionally, a non-contributor test can be performed on the crime scene sample and the chosen LR parameters to test the robustness of the model. This can give a point of comparison of the likelihood ratio generated for the person of interest (POI) compared to "random man" profiles generated from uploaded allelic frequencies. Finally, the analysis can be printed in a well-structured and user-friendly report that includes all analysis parameters. Within this chapter, the reader will learn the steps to calculate a likelihood ratio using the semi-continuous software, LRmix Studio. Additional tools supplied through the software will also be explained and demonstrated.

Recent advances in forensic biology and forensic DNA typing: INTERPOL review 2019-2022

This review paper covers the forensic-relevant literature in biological sciences from 2019 to 2022 as a part of the 20th INTERPOL International Forensic Science Managers Symposium. Topics reviewed include rapid DNA testing, using law enforcement DNA databases plus investigative genetic genealogy DNA databases along with privacy/ethical issues, forensic biology and body fluid identification, DNA extraction and typing methods, mixture interpretation involving probabilistic genotyping software (PGS), DNA transfer and activity-level evaluations, next-generation sequencing (NGS), DNA phenotyping, lineage markers (Y-chromosome, mitochondrial DNA, X-chromosome), new markers and approaches (microhaplotypes, proteomics, and microbial DNA), kinship analysis and human identification with disaster victim identification (DVI), and non-human DNA testing including wildlife forensics. Available books and review articles are summarized as well as 70 guidance documents to assist in quality control that were published in the past three years by various groups within the United States and around the world.

Probabilistic Genotyping of Single Cell Replicates from Mixtures Involving First-Degree Relatives Prevents the False Inclusions of Non-Donor Relatives

Analysis of complex DNA mixtures comprised of related individuals requires a great degree of care due to the increased risk of falsely including non-donor first-degree relatives. Although alternative likelihood ratio (LR) propositions that may aid in the analysis of these difficult cases can be employed, the prior information required for their use is not always known, nor do these alternative propositions always prevent false inclusions. For example, with a father/mother/child mixture, conditioning the mixture on the presence of one of the parents is recommended. However, the definitive presence of the parent(s) is not always known and an assumption of their presence in the mixture may not be objectively justifiable. Additionally, the high level of allele sharing seen with familial mixtures leads to an increased risk of underestimating the number of contributors (NOC) to a mixture. Therefore, fully resolving and identifying each of the individuals present in familial mixtures and excluding related non-donors is an important goal of the mixture deconvolution process and can be of great investigative value. Here, firstly, we further investigated and confirmed the problems encountered with standard bulk analysis of familial mixtures and demonstrated the ability of single cell analysis to fully distinguish first-degree relatives (FDR). Then, separation of each of the individual donors via single cell analysis was carried out by a combination of direct single cell subsampling (DSCS), enhanced DNA typing, and probabilistic genotyping, and applied to three complex familial 4-person mixtures resulting in a probative gain of LR for all donors and an accurate determination of the NOC. Significantly, non-donor first-degree relatives that were falsely included (LRs > 102−108) by a standard bulk sampling and analysis approach were no longer falsely included using DSCS.

Internal Validation of MaSTR™ Probabilistic Genotyping Software for the Interpretation of 2–5 Person Mixed DNA Profiles

Mixed human deoxyribonucleic acid (DNA) samples present one of the most challenging pieces of evidence that a forensic analyst can encounter. When multiple contributors, stochastic amplification, and allele drop-out further complicate the mixture profile, interpretation by hand becomes unreliable and statistical analysis problematic. Probabilistic genotyping software has provided a tool to address complex mixture interpretation and provide likelihood ratios for defined sets of propositions. The MaSTR™ software is a fully continuous probabilistic system that considers a wide range of STR profile data to provide likelihood ratios on DNA mixtures. Mixtures with two to five contributors and a range of component ratios and allele peak heights were created to test the validity of MaSTR™ with data similar to real casework. Over 280 different mixed DNA profiles were used to perform more than 2600 analyses using different sets of propositions and numbers of contributors. The results of the analyses demonstrated that MaSTR™ provided accurate and precise statistical data on DNA mixtures with up to five contributors, including minor contributors with stochastic amplification effects. Tests for both Type I and Type II errors were performed. The findings in this study support that MaSTR™ is a robust tool that meets the current standards for probabilistic genotyping.

A tool for simulating single source and mixed DNA profiles

Simulation studies play an important role in the study of probabilistic genotyping systems, as a low cost and fast alternative to in vitro studies. With ongoing calls for further study of the behaviour of probabilistic genotyping systems, there is a continuous need for such studies. In most cases, researchers use simplified models, for example ignoring complexities such as peak height variability due to lack of availability of advanced tools. We fill this void and describe a tool that can simulate DNA profiles in silico for the validation and investigation of probabilistic genotyping software. Contributor genotypes are simulated by randomly sampling alleles from selected allele frequencies. Some or all contributors may be related to a pedigree and the genotypes of non-founders are obtained by random gene dropping. The number of contributors per profile, and ranges for parameters such as DNA template amount and degradation parameters can be configured. Peak height variability is modelled using a lognormal distribution or a gamma distribution. Profile behaviour of simulated profiles is shown to be broadly similar to laboratory generated profiles though the latter shows more variation. Simulation studies do not remove the need for experimental data. The tool has been made available as an R-package named simDNAmixtures.

Development and validation of a fast and automated DNA identification line

The importance of DNA evidence for gaining investigative leads demands a fast workflow for forensic DNA profiling performed in large volumes. Therefore, we developed software solutions for automated DNA profile analysis, contamination check, major donor inference, DNA database (DDB) comparison and reporting of the conclusions. This represents the Fast DNA IDentification Line (FIDL) and this study describes its development, validation and implementation in criminal casework at the authors' institute. This first implementation regards single donor profiles and major contributors to mixtures. The validation included testing of the software components on their own and examination of the performance of different DDB search strategies. Furthermore, end-to-end testing was performed under three conditions: (1) testing of scenarios that can occur in DNA casework practice, (2) tests using three months of previous casework data, and (3) testing in a casework production environment in parallel to standard casework practices. The same DNA database candidates were retrieved by this automated line as by the manual workflow. The data flow was correct, results were reproducible and robust, results requiring manual analysis were correctly flagged, and reported results were as expected. Overall, we found FIDL valid for use in casework practice in our institute. The results from FIDL are automatically reported within three working days from receiving the trace sample. This includes the time needed for registration of the case, DNA extraction, quantification, polymerase chain reaction and capillary electrophoresis. FIDL itself takes less than two hours from intake of the raw CE data to reporting. Reported conclusions are one of five options: (1) candidate retrieved from DDB, (2) no candidate retrieved from DDB, (3) high evidential value with regards to reference within the case, (4) results require examination of expert, or (5) insufficient amount of DNA obtained to generate a DNA profile. In our current process, the automated report is sent within three working days and a complete report, with confirmation of the FIDL results, and signed by a reporting officer is sent at a later time. The signed report may include additional analyses regarding e.g. minor contributors. The automated report with first case results is quickly available to the police enabling them to act upon the DNA results prior to receiving the full DNA report. This line enables a uniform and efficient manner of handling large numbers of traces and cases and provides high value investigative leads in the early stages of the investigation.

Exploring likelihood ratios assigned for siblings of the true mixture contributor as an alternate contributor

Relatives tend to have more DNA in common than unrelated people. The closer the biological relationship, the higher the chance of alleles being identical by descent between the individuals. Therefore, when considering a mixed DNA profile, close relatives of the true contributor may not always be excluded as a possible contributor to a mixture due to allele sharing. In these situations, it might be more appropriate under the alternate proposition to consider that the DNA could have originated from a relative of the person of interest rather than an unrelated individual. The probabilistic genotyping software STRmix™ automatically provides LRs considering close biological relatives as alternate sources of the DNA. In this paper, we investigate the support for siblings of the true contributor to a mixture (who are not present in the mixture themselves). We interpret the mixtures and assign LRs using STRmix™ and investigate whether the resulting LRs could be used to indicate whether the true contributor could be a sibling of the POI. Most siblings will have one or more alleles that are not observed in the mixture profile. Support for siblings to have contributed can only occur when allelic dropout is a possibility at the loci where the siblings have alleles that are not observed in the profile. In these data, that was only observed in components with assigned template of 588 rfu or less.

Explainable artificial intelligence in forensics: Realistic explanations for number of contributor predictions of DNA profiles

Machine learning obtains good accuracy in determining the number of contributors (NOC) in short tandem repeat (STR) mixture DNA profiles. However, the models used so far are not understandable to users as they only output a prediction without any reasoning for that conclusion. Therefore, we leverage techniques from the field of explainable artificial intelligence (XAI) to help users understand why specific predictions are made. Where previous attempts at explainability for NOC estimation have relied upon using simpler, more understandable models that achieve lower accuracy, we use techniques that can be applied to any machine learning model. Our explanations incorporate SHAP values and counterfactual examples for each prediction into a single visualization. Existing methods for generating counterfactuals focus on uncorrelated features. This makes them inappropriate for the highly correlated features derived from STR data for NOC estimation, as these techniques simulate combinations of features that could not have resulted from an STR profile. For this reason, we have constructed a new counterfactual method, Realistic Counterfactuals (ReCo), which generates realistic counterfactual explanations for correlated data. We show that ReCo outperforms state-of-the-art methods on traditional metrics, as well as on a novel realism score. A user evaluation of the visualization shows positive opinions of end-users, which is ultimately the most appropriate metric in assessing explanations for real-world settings.

A Review of Probabilistic Genotyping Systems: EuroForMix, DNAStatistX and STRmix™

Probabilistic genotyping has become widespread. EuroForMix and DNAStatistX are both based upon maximum likelihood estimation using a γ model, whereas STRmix™ is a Bayesian approach that specifies prior distributions on the unknown model parameters. A general overview is provided of the historical development of probabilistic genotyping. Some general principles of interpretation are described, including: the application to investigative vs. evaluative reporting; detection of contamination events; inter and intra laboratory studies; numbers of contributors; proposition setting and validation of software and its performance. This is followed by details of the evolution, utility, practice and adoption of the software discussed.

A comparison of likelihood ratios obtained from EuroForMix and STRmix™

Likelihood ratios (LR) differences between the probabilistic genotyping software EuroForMix and STRmix™ are examined. After considering differences in the allele probabilities, the LRs from both software for an unambiguous single-source profile were identical (four significant figures). LRs from both software for an unambiguous single-source profile with alleles previously unseen in the allele frequency database (rare alleles) were the same (three significant figures) for θ = 0.01. Due to differences in the minimum allele frequencies, the LRs differed by three orders of magnitude when θ = 0. For both software, the LRs for a single-source dilution series decreased as the input amount decreased. The LRs from both software were within an order of magnitude for known contributors. The largest difference was where the target input amount was 0.0156 ng: The LR_EuroForMix was 2.1 × 10²⁵ and the LR_STRmix was 8.0 × 10²⁴ . Both software show similar LR behavior with respect to mixture ratio. For two person mixtures the LR increases for both the major and the minor as the ratio moves away from 1:1. The LR for the major stabilizes at about 3:1 whereas the LR for the minor reaches its maximum at about 3:1 and then declines. Greater differences in LR were observed between EuroForMix and STRmix™ for mixtures. One-hundred and twenty-nine mixtures from the PROVEDIt dataset were compared. LRs for 84% of the comparisons for known contributors without rare alleles were within two orders of magnitude. Five divergent results were investigated, and a manual intervention approach was applied where appropriate.

Examining performance and likelihood ratios for two likelihood ratio systems using the PROVEDIt dataset

A likelihood ratio (LR) system is defined as the entire pipeline of the measurement and interpretation processes where probabilistic genotyping software (PGS) is a piece of the whole LR system. To gain understanding on how two LR systems perform, a total of 154 two-person, 147 three-person, and 127 four-person mixture profiles of varying DNA quality, DNA quantity, and mixture ratios were obtained from the filtered (.CSV) files of the GlobalFiler 29 cycles 15s PROVEDIt dataset and deconvolved in two independently developed fully continuous programs, STRmix v2.6 and EuroForMix v2.1.0. Various parameters were set in each software and LR computations obtained from the two software were based on same/fixed EPG features, same pair of propositions, number of contributors, theta, and population allele frequencies. The ability of each LR system to discriminate between contributor (H1-true) and non-contributor (H2-true) scenarios was evaluated qualitatively and quantitatively. Differences in the numeric LR values and their corresponding verbal classifications between the two LR systems were compared. The magnitude of the differences in the assigned LRs and the potential explanations for the observed differences greater than or equal to 3 on the log10 scale were described. Cases of LR < 1 for H1-true tests and LR > 1 for H2-true tests were also discussed. Our intent is to demonstrate the value of using a publicly available ground truth known mixture dataset to assess discrimination performance of any LR system and show the steps used to understand similarities and differences between different LR systems. We share our observations with the forensic community and describe how examining more than one PGS with similar discrimination power can be beneficial, help analysts compare interpretation especially with low-template profiles or minor contributor cases, and be a potential additional diagnostic check even if software in use does contain certain diagnostic statistics as part of the output.

Statistefix 4.0: A novel probabilistic software tool

Latest innovations indicate that continuous tools are promising DNA trace assessment methods. In this study, we present the continuous software solution Statistefix 4.0. The software supports DNA experts in deducing DNA profiles for database queries and can help to preselect DNA samples suitable for further processing using advanced probabilistic search engines. The novel tool weights genotype contributions and deduces major contributors from high- and low-quality DNA traces. Peak height, degradation, stutter as well as allelic drop-in/-out events are incorporated in the statistical model. We analyzed reference and casework samples as well as artificially generated mixture samples for software evaluation. The tool offers the completely automated assessment of reference and mixture samples. Deconvolution outcomes of mixtures are compared with EuroForMix, GenoProof Mixture 3 and STRmix™. Data show that Statistefix 4.0 is as successful as analogously tested and implemented software. Deduced DNA profiles from casework samples highlight the potential benefit in routine casework. Statistefix 4.0 is freely available, works with replicates of different autosomal kits and enables bulk sample processing. This inter-laboratory study includes a variety of sample types and indicates a timesaving, robust and easily implemented software that supports DNA analysts in evaluating DNA traces.

Application of a probabilistic genotyping software to MPS mixture STR data is supported by similar trends in LRs compared with CE data

The interpretation of short tandem repeat (STR) profiles can be challenging when, for example, alleles are masked due to allele sharing among contributors and/or when they are subject to drop-out, for instance from sample degradation. Mixture interpretation can be improved by increasing the number of STRs and/or loci with a higher discriminatory power. Both capillary electrophoresis (CE, 6-dye) and massively parallel sequencing (MPS) provide a platform for analysing relatively large numbers of autosomal STRs. In addition, MPS enables distinguishing between sequence variants, resulting in enlarged discriminatory power. Also, MPS allows for small amplicon sizes for all loci as spacing is not an issue, which is beneficial with degraded DNA. Altogether, MPS has the potential to increase the weights of evidence for true contributors to (complex) DNA profiles. In this study, likelihood ratio (LR) calculations were performed using STR profiles obtained with two different MPS systems and analysed using different settings: 1) MPS PowerSeq™ Auto System profiles analysed using FDSTools equipped with optimized settings such as noise correction, 2) ForenSeq™ DNA Signature Prep Kit profiles analysed using the default settings in the Universal Analysis Software (UAS), and 3) ForenSeq™ DNA Signature Prep Kit profiles analysed using FDSTools empirically adapted to cope with one-directional reads and provisional, basic settings. The LR calculations used genotyping data for two- to four-person mixtures varying for mixture proportion, level of drop-out and allele sharing and were generated with the continuous model EuroForMix. The LR results for the over 2000 sets of propositions were affected by the variation for the number of markers and analysis settings used in the three approaches. Nevertheless, trends for true and non-contributors, effects of replicates, assigned number of contributors, and model validation results were comparable for the three MPS approaches and alike the trends known for CE data. Based on this analogy, we regard the probabilistic interpretation of MPS STR data fit for forensic DNA casework. In addition, guidelines were derived on when to apply LR calculations to MPS autosomal STR data and report the corresponding results.

Multi-laboratory validation of DNAxs including the statistical library DNAStatistX

This study describes a multi-laboratory validation of DNAxs, a DNA eXpert System for the data management and probabilistic interpretation of DNA profiles [1], and its statistical library DNAStatistX to which, besides the organising laboratory, four laboratories participated. The software was modified to read multiple data formats and the study was performed prior to the release of the software to the forensic community. The first exercise explored all main functionalities of DNAxs with feedback on user-friendliness, installation and general performance. Next, every laboratory performed likelihood ratio (LR) calculations using their own dataset and a dataset provided by the organising laboratory. The organising laboratory performed LR calculations using all datasets. The datasets were generated with different STR typing kits or analysis systems and consisted of samples varying in DNA amounts, mixture ratios, number of contributors and drop-out level. Hypothesis sets had the correct, under- and over-assigned number of contributors and true and false donors as person of interest. When comparing the results between laboratories, the LRs were foremost within one unit on log10 scale. The few LR results that deviated more had differences for the parameters estimated by the optimizer within DNAStatistX. Some of these were indicated by failed iteration results, others by a failed model validation, since unrealistic hypotheses were included. When these results that do not meet the quality criteria were excluded, as is in accordance with interpretation guidelines, none of the analyses in the different laboratories yielded a different statement in the casework report. Nonetheless, changes in software parameters were sought that minimized differences in outcomes, which made the DNAStatistX module more robust. Overall, the software was found intuitive, user-friendly and valid for use in multiple laboratories.