Are low LRs reliable?

Extending the discussion on inconsistency in forensic decisions and results

The subject of inter- and intra-laboratory inconsistency was recently raised in a commentary by Itiel Dror. We re-visit an inter-laboratory trial, with which some of the authors of this current discussion were associated, to diagnose the causes of any differences in the likelihood ratios (LRs) assigned using probabilistic genotyping software. Some of the variation was due to different decisions that would be made on a case-by-case basis, some due to laboratory policy and would hence differ between laboratories, and the final and smallest part was the run-to-run difference caused by the Monte Carlo aspect of the software used. However, the net variation in LRs was considerable. We believe that most laboratories will self-diagnose the cause of their difference from the majority answer and in some, but not all instances will take corrective action. An inter-laboratory exercise consisting of raw data files for relatively straightforward mixtures, such as two mixtures of three or four persons, would allow laboratories to calibrate their procedures and findings.

An inter-laboratory comparison of probabilistic genotyping parameters and evaluation of performance on DNA mixtures from different laboratories

Probabilistic genotyping (PG) is becoming the preferred standard for evidence interpretation, amongst forensic DNA laboratories, especially those in the United States. Various groups have expressed concern about reliability of PG systems, especially for mixtures beyond two contributors. Studies involving interlaboratory testing of known mixtures have been identified as ways to evaluate the reliability of PG systems. Reliability means different things in different contexts. However, it suffices here to think about it as a mixture of precision and accuracy. We might also consider whether a system is prone to producing misleading results - for example large likelihood ratios (LRs) when the POI is truly not a contributor, or small LRs when the POI is a truly a contributor. In this paper we show that the PG system STRmix™ is relatively unaffected by differences in parameter settings. That is, a DNA mixture that is analyzed in different laboratories using STRmix™ will result in different LRs, but less than 0.05% of these LRs would result in a different, or misleading conclusion as long as the LR is greater than 50. For the purposes of this study, we define LRs assigned using different parameters for the same mixtures as similar if the LR of the true POI is greater than the LRs generated for 99.9% of the general population. These findings are based on an interlaboratory study involving eight laboratories that provided twenty known DNA mixtures of two to four contributors and their individual laboratory STRmix™ parameters. The eight sets of laboratory parameters included differences in STR kits and PCR cycles as well as the peak, stutter, and locus specific amplification efficiency variances.

Einsatz vollkontinuierlicher Modelle zur biostatistischen Bewertung forensischer DNA-analytischer Befunde

Die biostatistische Bewertung DNA-analytischer Befunde unterstützt Gerichte bei der Einschätzung des Beweiswertes einer Spur. In der Praxis werden dabei zunehmend Spuren mit minimaler DNA-Menge und möglichen „Drop-in“- und „Drop-out“-Ereignissen sowie komplexe Mischspuren analysiert. Solche Spuren sind mit einer klassischen „binären“ Berechnung biostatistisch häufig nicht oder nur eingeschränkt bewertbar.

Die Entwicklung vollkontinuierlicher Modelle (VKM) macht eine Vielzahl dieser bisher nicht berechenbaren Spuren einer biostatistischen Bewertung zugänglich. Dabei werden nahezu sämtliche verfügbaren Informationen einer DNA-Spur in die Berechnung einbezogen. Während diese probabilistischen Verfahren international bereits vielfach zum Einsatz kommen, liegen hierzu im deutschsprachigen Raum nur wenige Erfahrungen vor.

Um Funktionsweise, Möglichkeiten und Grenzen von VKM-Berechnungen zu erfassen, wurden Mischspuren bekannter Zusammensetzung mit 4 aktuell verfügbaren VKM-Programmen vergleichend analysiert. Bei der Auswertung wurden zentrale Aspekte betrachtet, wie beispielsweise die Konkordanz von Berechnungsergebnissen, der Einfluss von Drop-in- und Drop-out-Ereignissen auf die berechneten vollkontinuierlichen LR-Werte (LRfc) sowie die Ableitung recherchefähiger DNA-Profile mithilfe wahrscheinlichkeitsbasierter Prognosen (Deconvolution).

Die im Rahmen dieser Arbeit gewonnenen Erfahrungen bilden, zusammen mit weiteren bereits international publizierten Studien, eine Basis für Empfehlungen zum Einsatz von VKM-basierter Software bei der biostatistischen Bewertung DNA-analytischer Befunde.

Gemeinsame Empfehlungen der Projektgruppe „Biostatistische DNA-Berechnungen“ und der Spurenkommission zur biostatistischen Bewertung forensischer DNA-analytischer Befunde mit vollkontinuierlichen Modellen (VKM)

Die biostatistische Bewertung DNA-analytischer Befunde unterstützt Gerichte bei der Einschätzung des Beweiswertes hinsichtlich einer möglichen Spurenbeteiligung durch eine zu betrachtende Person (engl. „Person Of Interest“; POI). Um die Vergleichbarkeit derartiger Berechnungen auf Grundlage etablierter wissenschaftlicher Standards zu gewährleisten, wurden bereits in der Vergangenheit entsprechende Empfehlungen im nationalen Konsens formuliert.

Mit Einführung sog. vollkontinuierlicher Modelle (VKM) für die probabilistische Genotypisierung, die u. a. die Signalintensitäten eines Elektropherogramms berücksichtigen, wurde eine Ergänzung zu den damaligen Empfehlungen erforderlich. VKM erlauben eine biostatistische Bewertung von Spuren mit möglichen Drop-in- und Drop-out-Ereignissen und wahrscheinlichkeitsbasierte Prognosen der zu einer Mischspur beitragenden Genotypen („Deconvolution“).

Die vorliegende Veröffentlichung enthält Empfehlungen zum Einsatz VKM-basierter Software und zur Berichterstattung vollkontinuierlicher LR-Werte (engl. „Fully Continuous Likelihood Ratios“; LRfc). Sie empfiehlt bei schwierig zu interpretierenden Befunden eine VKM-Berechnung zur Bewertung einer Spurenlegerschaft. Die VKM-Berechnung ersetzt die bisher in Ausnahmefällen als hinnehmbar erachtete Vorgehensweise einer binären Berechnung unter Ausklammern einzelner Merkmalssysteme. Der Einsatz von VKM erfordert eine umfassende Anwenderschulung sowie eine Validierung und Verifizierung gemäß den Vorgaben der Programmanbieter. Mit der Empfehlung von LRfc-Schwellenwerten soll eine sichere, vergleichbare Anwendung von VKM gewährleistet werden.

A tool for simulating single source and mixed DNA profiles

Simulation studies play an important role in the study of probabilistic genotyping systems, as a low cost and fast alternative to in vitro studies. With ongoing calls for further study of the behaviour of probabilistic genotyping systems, there is a continuous need for such studies. In most cases, researchers use simplified models, for example ignoring complexities such as peak height variability due to lack of availability of advanced tools. We fill this void and describe a tool that can simulate DNA profiles in silico for the validation and investigation of probabilistic genotyping software. Contributor genotypes are simulated by randomly sampling alleles from selected allele frequencies. Some or all contributors may be related to a pedigree and the genotypes of non-founders are obtained by random gene dropping. The number of contributors per profile, and ranges for parameters such as DNA template amount and degradation parameters can be configured. Peak height variability is modelled using a lognormal distribution or a gamma distribution. Profile behaviour of simulated profiles is shown to be broadly similar to laboratory generated profiles though the latter shows more variation. Simulation studies do not remove the need for experimental data. The tool has been made available as an R-package named simDNAmixtures.

High-quality data from a forensically relevant single-cell pipeline enabled by low PBS and proteinase K concentrations

Interpreting forensic DNA signal is arduous since the total intensity is a cacophony of signal from noise, artifact, and allele from an unknown number of contributors (NOC). An alternate to traditional bulk-processing pipelines is a single-cell one, where the sample is collected, and each cell is sequestered resulting in n single-source, single-cell EPGs (scEPG) that must be interpreted using applicable strategies. As with all forensic DNA interpretation strategies, high quality electropherograms are required; thus, to enhance the credibility of single-cell forensics, it is necessary to produce an efficient direct-to-PCR treatment that is compatible with prevailing downstream laboratory processes. We incorporated the semi-automated micro-fluidic DEPArray™ technology into the single-cell laboratory and optimized its implementation by testing the effects of four laboratory treatments on single-cell profiles. We focused on testing effects of phosphate buffer saline (PBS) since it is an important reagent that mitigates cell rupture but is also a PCR inhibitor. Specifically, we explored the effect of decreasing PBS concentrations on five electropherogram-quality metrics from 241 leukocytes: profile drop-out, allele drop-out, allele peak heights, peak height ratios, and scEPG sloping. In an effort to improve reagent use, we also assessed two concentrations of proteinase K. The results indicate that decreasing PBS concentrations to 0.5X or 0.25X improves scEPG quality, while modest modifications to proteinase K concentrations did not significantly impact it. We, therefore, conclude that a lower than recommended proteinase K concentration coupled with a lower than recommended PBS concentration results in enhanced scEPGs within the semi-automated single-cell pipeline.

Using big data from probabilistic genotyping to solve crime

Forensic Science South Australia (FSSA) has been using STRmix™ software to deconvolute all reported DNA mixtures since 2012. Almost a decade of deconvolutions had led to a substantial repository of analysed profile data that can be interrogated to observe trends in case type, location or occurrence. In addition, deconvolutions can be compared in order to identify common DNA donors and reveal new intelligence information in cases where DNA profiling has previously provided no investigative information. As a proof of concept all samples deconvoluted as part of criminal casework (suspect or no-suspect) were interrogated and compared to each other using the mixture-to-mixture comparison feature in STRmix™. Within the Adelaide region there were 32 groups of cases that had evidence samples linked by a common DNA donor with LR > 1 million which was in addition to direct links and mixture searching links identified previously. These groups of cases can then be interrogated to reveal additional information to inform Police intelligence gathering. Our paper reports on the findings of this proof-of-concept study.

A Review of Probabilistic Genotyping Systems: EuroForMix, DNAStatistX and STRmix™

Probabilistic genotyping has become widespread. EuroForMix and DNAStatistX are both based upon maximum likelihood estimation using a γ model, whereas STRmix™ is a Bayesian approach that specifies prior distributions on the unknown model parameters. A general overview is provided of the historical development of probabilistic genotyping. Some general principles of interpretation are described, including: the application to investigative vs. evaluative reporting; detection of contamination events; inter and intra laboratory studies; numbers of contributors; proposition setting and validation of software and its performance. This is followed by details of the evolution, utility, practice and adoption of the software discussed.

Evaluating DNA evidence possibly involving multiple (mixed) samples, common donors and related contributors

Forensic DNA profiling is used in various circumstances to evaluate support for two competing propositions with the assignment of a likelihood ratio. Many software implementations exist that tackle a range of inference problems spanning identification and relationship testing. We propose a flexible likelihood ratio framework that caters to inference problems in forensic genetics. The framework allows for investigation of the degree of support for the contribution of multiple persons to multiple samples allowing for persons to be related according to a pedigree, including inbred relationships. We explain how a number of routine as well as more complex problems can be treated within this framework.

Measuring calibration of likelihood-ratio systems: A comparison of four metrics, including a new metric devPAV

Numerical likelihood-ratio (LR) systems aim to calculate evidential strength for forensic evidence evaluation. Calibration of such LR-systems is essential: one does not want to over- or understate the strength of the evidence. Metrics that measure calibration differ in sensitivity to errors in calibration of such systems. In this paper we compare four calibration metrics by a simulation study based on Gaussian Log LR-distributions. Three calibration metrics are taken from the literature (Good, 1985; Royall, 1997; Ramos and Gonzalez-Rodriguez, 2013) [1-3], and a fourth metric is proposed by us. We evaluated these metrics by two performance criteria: differentiation (between well- and ill-calibrated LR-systems) and stability (of the value of the metric for a variety of well-calibrated LR-systems). Two metrics from the literature (the expected values of LR and of 1/LR, and the rate of misleading evidence stronger than 2) do not behave as desired in many simulated conditions. The third one (C_llr^cal) performs better, but our newly proposed method (which we coin devPAV) is shown to behave equally well to clearly better under almost all simulated conditions. On the basis of this work, we recommend to use both devPAV and C_llr^cal to measure calibration of LR-systems, where the current results indicate that devPAV is the preferred metric. In the future external validity of this comparison study can be extended by simulating non-Gaussian LR-distributions.

The analogy between DNA kinship and DNA mixture evaluation, with applications for the interpretation of likelihood ratios produced by possibly imperfect models

Two main applications of forensic DNA analysis are the investigation of possible relatedness and the investigation whether a person left DNA in a trace. Both of these are usually carried out by the calculation of likelihood ratios. In the kinship case, it is standard to let the likelihood ratio express the support in favour of the investigated relatedness versus no relatedness, and in the investigation of traces, one by default compares the hypothesis that the person of interest contributed DNA, versus that he is unrelated to any of the actual contributors. In both cases however, we can also view the probabilistic procedure as an inference of the profile of the person we look for: in other words, in both cases we carry out probabilistic genotyping. In this article we use this general analogy to develop various more specific analogies between kinship and mixture likelihood ratios. These analogies help to understand the concepts that play a role, and also to understand the importance of the statistical modeling needed for DNA mixtures. In this article, we apply our findings to consider what we can and cannot conclude from a likelihood ratio in favour of contribution to a mixed DNA profile, if that is computed by a model whose specifics are not entirely known to us, or where we do not know whether they provide a good description of the stochastic effects involved in the generation of DNA trace profiles. We show that, if unrelated individuals are adequately modeled, we can give bounds on how often LR's coming from certain types of black box models may arise, both for persons who are actual contributors and who are unrelated. In particular we show that no model, provided it satisfies basic requirements, can overestimate the evidence found for actual contributors both often and strongly.