Perceptions and estimates of error rates in forensic science: A survey of forensic analysts

Identification issues in bodies maimed by train fatalities: The role of the forensic pathologist

Traffic accident injuries are relevant causes of mortality in industrialized countries. Among traffic accidents, the most destructive effect on the victim's body is produced by trains, which in a very short time provoke a complete distortion of the normal anatomical profiles and extensive amputations of the body. Deaths by severe body mutilation and/or decapitation require complex forensic investigation, as the victims' identification and the recognition of their body are not often feasible. In this study, a retrospective analysis of a series of cases of railway accidents is presented, with the aim of showing potential identification strategies, including a systematic physical re-assembling of the body. Among the railway fatalities assessed at the University Institute of Legal Medicine of Milan from 2016 until 2020, 61 cases were analyzed, focusing on the collection of postmortem data to improve accuracy for the identification of the victims. The results showed that the identification could be based on elements traditionally called "secondary identifiers" which, however, are becoming more and more important for positive identification, especially when dealing with migrants, vagrants, and homeless people. However, several issues need to be considered such as the limitations of the public prosecutors to perform expensive analyses (i.e., genetic testing), and the lack of antemortem data to be compared with postmortem findings, both odontological and fingerprint. In cases where it is not possible to proceed with identification or formal recognition of victims of railroad accidents with severe mutilations, a systematic physical re-assembling of the body is critical to reaching positive identification.

Surveying practicing firearm examiners

A sample (n = 79) of practicing firearm and toolmark examiners was queried about casework as well as their views about the potential role that statistics might play in future firearm examinations and expert witness testimony. Principal findings include: The modal response for time spent conducting bullet examinations is 2–4 hours, and the modal response for cartridge casings is 1–2 hours. The average participant (median) makes an identification in 65% of casework, makes an elimination in 12% of casework, and reports that the examination was inconclusive in 20% of casework calls. The vast majority of examiners work at laboratories that permit eliminations when class characteristics agree. The reported industry-wide false positive error rate is 1%, though very few participants could name a study or give a citation for their reported estimate. Qualitative responses about the potential role of statistics were mixed.

Gas Chromatographic Fingerprint Analysis for the Comparison of Seized Cannabis Samples

Cannabis sativa L. is widely used as recreational illegal drugs. Illicit Cannabis profiling, comparing seized samples, is challenging due to natural Cannabis heterogeneity. The aim of this study was to use GC–FID and GC–MS herbal fingerprints for intra (within)- and inter (between)-location variability evaluation. This study focused on finding an acceptable threshold to link seized samples. Through Pearson correlation-coefficient calculations between intra-location samples, ‘linked’ thresholds were derived using 95% and 99% confidence limits. False negative (FN) and false positive (FP) error rate calculations, aiming at obtaining the lowest possible FP value, were performed for different data pre-treatments. Fingerprint-alignment parameters were optimized using Automated Correlation-Optimized Warping (ACOW) or Design of Experiments (DoE), which presented similar results. Hence, ACOW data, as reference, showed 54% and 65% FP values (95 and 99% confidence, respectively). An additional fourth root normalization pre-treatment provided the best results for both the GC–FID and GC–MS datasets. For GC–FID, which showed the best improved FP error rate, 54 and 65% FP for the reference data decreased to 24 and 32%, respectively, after fourth root transformation. Cross-validation showed FP values similar as the entire calibration set, indicating the representativeness of the thresholds. A noteworthy improvement in discrimination between seized Cannabis samples could be concluded.

Metric forensic anthropology decisions: Reliability and biasability of sectioning-point-based sex estimates

Subjective decisions make human cognitive processes more susceptible to bias and error. Specifically, research indicates that additional context biases forensic anthropologists' morphological analyses. To address whether metric analyses are also subject to bias, we conducted a pilot study in which 52 experienced osteologists measured a difficult-to-classify human femur, with or without additional contextual information. Using a metric sectioning-point sex-estimation method, participants provided a sex estimate for individual skeletal element(s) and, when given multiple elements, the combined skeletal assemblage. Control group participants (n = 24) measured only the femur. In addition to the femur, bias group participants (n = 28) either measured a female humerus and viewed a female-biasing photograph (n = 14) or measured a male humerus and viewed a male-biasing photograph (n = 14). We explored whether the experts in the different groups would differ in: (1) femoral measurements; (2) femoral sex-estimation conclusions; and (3) final sex-estimation conclusions for the skeletal assemblage. Although the femoral measurements and femoral sex estimates were comparable across groups, the overall sex estimates in the female-biased group were impacted by contextual information-differing from both the control and male-biased groups (p < 0.001). Our results demonstrate that cognitive bias can occur even in metric sex-estimation conclusions. Specifically, this occurred when the metric data and single-element sex estimates were synthesized into an overall estimate. Thus, our results suggest that metric methods are most vulnerable to bias when data are synthesized into an overall conclusion, highlighting the need for bias countermeasures and comprehensive statistical frameworks for synthesizing metric data to mitigate the effects of cognitive bias.

Evaluation of non-invasive bioforensic techniques for determining the age of hot-iron brand burn scars in cattle

Hot-iron branding is a traditional form of permanent cattle identification in the United States. There is a need for science-based determination of cattle brand age. Near infrared reflectance spectroscopy (NIRS) has been used to obtain information about animal tissues and healing processes. Height-width allometry and NIRS were applied to hot-iron cattle brand scars to determine if either or both of these methods can be used to non-invasively establish the interval sincethe application of hot-iron cattle brands. Length and width of a brand routinely applied to calves (~30–60 d old) were established and then the same measurements were recorded on 378 calfhood branded cattle of known age ranging from 0.5 to > 6.5 yr-of-age. Brand width and height increased over the original measurements by > 100% between calfhood application and 2.5 yr-of-age (P < 0.001). Brand size did not change dramatically between 2.5 and > 6.5 yr, however, both width and height were (P < 0.05) greater at maturity than at weaning. Near infrared spectra were collected from a) branded skin b) non-clipped (hair), non-branded skin, and c) hair clipped, non-branded skin on Bos taurus cross calves. Individual trial calibrations yielded high R² and low SE of calibration values as well as similar cross validation performance (P < 0.001). Numerically lower but still strong performance (P < 0.001) resulted from combined data set calibrations. Cross-trial prediction of brand age was unsuccessful. One single year calibration underpredicted (P < 0.001) brand age of an independent validation set by 2.83 d, and another single year calibration underpredicted (P < 0.001) the same validation set by 9.91 d. When combined, these two datasets resulted in a calibration that overpredicted brand age in the validation set by 6.9 d (P < 0.02). Discriminant analyses for identification of skin surface type yielded success rates of 90% for branded, 99% for non-clipped, non-branded, and 96% for clipped, non-branded (P < 0.01). Discriminant analyses were also performed on samples grouped into a) less than 33 d, b) 141–153 d, and c) 169 d categories. All group membership identifications were successful at greater than 90% (P < 0.01). Preliminary results indicate that brand size could be used to indicate brand age and that NIRS can predict brand age as well as discriminate between broad brand age groups in cattle. More work will need to be done before these techniques can be used in real-world forensic applications.

Evaluating population affinity estimates in forensic anthropology: Insights from the forensic anthropology database for assessing methods accuracy (FADAMA)

Using a sample of anonymized U.S. forensic anthropology cases (n = 251) from the FADAMA database, we assess the degree of concordance between decedents' social identifiers and anthropologists' continental-based classifications. We report high success rates (>90%) that generally support previous findings, yet we acknowledge the limitations of assessing "ancestry" accuracy based on resolved cases and draw attention to situations in which our methods fail. For example, forensic anthropologists achieve just 20% accuracy when classifying individuals as "other" or "mixed"-problematic categories that we argue should be rejected. Leveraging our findings, we ask: what are we really estimating when we perform a skeletal assessment of "ancestry" in the US context? We argue that the "ancestry estimates" historically and routinely produced in forensic anthropology instead give information on population affinity: a measure of how similar a given case is to one among several socially relevant groups of interest. Distancing forensic anthropology from genetics and other disciplines that estimate ancestry, the approach of population affinity assesses similarities to both social and biological groupings, potentially at a fine-grained level, attempting to account for the complex histories, shared biologies, and wide ranges of diversity that characterize our communities and our casework. Population affinity is a flexible and inclusive approach that more accurately describes current forensic anthropological analyses of human variation. Going forward, we must acknowledge and build on the contributions of previous scholars as we work together toward our shared goal of theoretically grounded analyses of human variation that accurately and equitably serve all casework decedents.

Judges and forensic science education: A national survey

In criminal cases, forensic science reports and expert testimony play an increasingly important role in adjudication. More states now follow a federal reliability standard, which calls upon judges to assess the reliability and validity of scientific evidence. Little is known about how judges view their own background in forensic scientific evidence, and what types of specialized training they receive on it. In this study, we surveyed 164 judges from 39 different U.S. states, who attended past trainings at the National Judicial College. We asked these judges about their background in forensic science, their views concerning the reliability of common forensic disciplines, and their needs to better evaluate forensic science evidence. We discovered that judges held views regarding the scientific support for different forensic science disciplines that were fairly consistent with available literature; their error rate estimates were more supported by research than many estimates by laypersons, who often assume forensic methods are nearly infallible. We did not find any association between how judges rate forensic reliability and prior training. We did, however, find that training corresponded with judges' views that they should, and do in fact, take on a more active gatekeeping role regarding forensics. Regarding the tools judges need to vet forensic experts and properly evaluate forensic science evidence, they reported having very different backgrounds in relevant scientific concepts and having forensic science education needs. Judges reported needs in accessing better material concerning reliability of forensic science methods. These results support new efforts to expand scientific evidence education in the judiciary.

Implementing blind proficiency testing in forensic laboratories: Motivation, obstacles, and recommendations

Regular proficiency testing of forensic examiners is required at accredited laboratories and widely accepted as an important component of a functioning quality assurance program. Yet, unlike in other testing industries, the majority of forensic laboratories testing programs rely entirely on declared proficiency tests. Some laboratories, primarily federal forensic facilities, have adopted blind proficiency tests, which are also used in the medical and drug testing industries. Blind tests offer advantages. They must resemble actual cases, can test the entire laboratory pipeline, avoid changes in behavior from an examiner knowing they are being tested, and are one of the only methods that can detect misconduct. However, the forensic context present both logistical and cultural obstacles to the implementation of blind proficiency tests. In November 2018, we convened a meeting of directors and quality assurance managers of local and state laboratories to discuss obstacles to the adoption of blind testing and assess successful and potential strategies to overcome them. Here, we compare the situation in forensic science to other testing disciplines, identifying obstacles to the implementation of blind proficiency testing in forensic contexts, and proposing ways to address those issues and increase the ecological validity of proficiency tests at forensic laboratories.

Microscopical discrimination of human head hairs sharing a mitochondrial haplogroup

In forensic analyses, determining the level of consensus among examiners for hair comparison conclusions and ancestry identifications is important for assessing the scientific validity of microscopical hair examinations. Here, we present data from an interlaboratory study on the accuracy of microscopical hair comparisons among a subset of experienced hair examiners currently analyzing hair in forensic laboratories across the United States. We examined how well microscopical analysis of hair can reliably be used to differentiate hair samples, many of which were macroscopically similar. Using cut hair samples, many sharing similar macroscopic and microscopic features, collected from individuals who share the same mitochondrial haplogroup as an indication of genetic relatedness, we tested multiple aspects that could impact hair comparisons. This research tested the extent to which morphological features related to ancestry and hair length influence conclusions. Microscopical hair examinations yielded accurate assessments of inclusion/exclusion relative to the reference samples among 85% of the pairwise comparisons. We found shorter hairs had reduced levels of accuracy and hairs from populations examiners were not familiar with may have impacted their ability to resolve features. The reliability of ancestry determinations is not yet clear, but we found indications that the existing categories are only somewhat related to current ethnic and genetic variation. Our results provide support for the continued utility of microscopical comparison of hairs within forensic laboratories and to advocate for a combined analytical approach using both microscopical analysis and mtDNA data on all forensic analyses of hair.

The Error in "Error Rate": Why Error Rates Are So Needed, Yet So Elusive

Establishing error rates is crucial for knowing how well one is performing, determining whether improvement is needed, measuring whether interventions are effective, as well as for providing transparency. However, the flurry of activities in establishing error rates for the forensic sciences has largely overlooked some fundamental issues that make error rates a problematic construct and limit the ability to obtain a meaningful error rate. These include knowing the ground truth, establishing appropriate databases, determining what counts as an error, characterizing what is an acceptable error rate, ecological validity, and transparency within the adversarial legal system. Without addressing these practical and theoretical challenges, the very notion of a meaningful error rate is limited.

Assessing the reliability of a clothing-based forensic identification

Our justice system relies critically on the use of forensic science. More than a decade ago, a highly critical report raised significant concerns as to the reliability of many forensic techniques. These concerns persist today. Of particular concern to us is the use of photographic pattern analysis that attempts to identify an individual from purportedly distinct features. Such techniques have been used extensively in the courts over the past half century without, in our opinion, proper validation. We propose, therefore, that a large class of these forensic techniques should be subjected to rigorous analysis to determine their efficacy and appropriateness in the identification of individuals.

A 2009 report by the National Academy of Sciences was highly critical of many forensic practices. This report concluded that significant changes and advances were required to ensure the reliability across the forensic sciences. We examine the reliability of one such forensic technique used for identification based on purported distinct patterns on the seams of denim pants. Although first proposed more than 20 years ago, no thorough analysis of reliability or reproducibility of this forensic technique has previously been reported. We performed a detailed analysis of this forensic technique to determine its reliability and efficacy.

(Mis)use of scientific measurements in forensic science

Forensic science error rate studies have not given sufficient attention or weight to inconclusive evidence and inconclusive decisions. Inconclusive decisions can be correct decisions, but they can also be incorrect decisions. Errors can occur when inconclusive evidence is determined as an identification or exclusion, or conversely, when same- or different-source evidence is incorrectly determined as inconclusive. We present four common flaws in error rate studies: 1. Not including test items which are more prone to error; 2. Excluding inconclusive decisions from error rate calculations; 3. Counting inconclusive decisions as correct in error rate calculations; and 4. Examiners resorting to more inconclusive decisions during error rate studies than they do in casework. These flaws seriously undermine the credibility and accuracy of error rates reported in studies. To remedy these shortcomings, we present the problems and show the way forward by providing a corrected experimental design that quantifies error rates more accurately.