Applying 3D measurements and computer matching algorithms to two firearm examination proficiency tests

Automated interpretation of comparison scores for firearm toolmarks on cartridge case primers

An automated approach for evaluating the strength of the evidence of firearm toolmark comparison results is presented for a common source scenario. First, comparison scores are derived describing the similarity of marks typically encountered on the primer of fired cartridge cases: aperture shear striations as well as breechface and firing pin impressions. Subsequently, these scores are interpreted using reference distributions of comparison scores obtained for representative known matching (KM) and known non-matching (KNM) ballistic samples in a common source, score-based likelihood ratio (LR) system. We study various alternatives to set up such an LR system and compare them using qualitative and quantitative criteria known from the literature. As an example, results are applied to establish a system suitable for a firearm-ammunition combination often encountered in casework: Glock firearms with Fiocchi nickel primer ammunition. The system outputs an LR and a measure of LR uncertainty. The range of possible LR-values is limited to a minimum and maximum value in areas of the score domain with little reference data. Finally, the feasibility of combining LRs of different mark types into one LR for the entire primer is assessed. For the distribution models considered in this paper, different modeling approaches are optimal for different types of similarity scores. For the chosen firearm-ammunition combination, non-parametric Kernel Density Estimation (KDE) models perform best for similarity scores based on the correlation coefficient, whereas parametric models perform best for the Congruent Matching Cells (CMC) scores, assuming binomial and beta-binomial models for KM and KNM score distributions respectively. Finally, it is demonstrated that individual LRs of different mark types can be combined into one LR, to interpret a set of different marks on the primer as a whole.

Accuracy of comparison decisions by forensic firearms examiners

This black box study assessed the performance of forensic firearms examiners in the United States. It involved three different types of firearms and 173 volunteers who performed a total of 8640 comparisons of both bullets and cartridge cases. The overall false-positive error rate was estimated as 0.656% and 0.933% for bullets and cartridge cases, respectively, while the rate of false negatives was estimated as 2.87% and 1.87% for bullets and cartridge cases, respectively. The majority of errors were made by a limited number of examiners. Because chi-square tests of independence strongly suggest that error probabilities are not the same for each examiner, these are maximum-likelihood estimates based on the beta-binomial probability model and do not depend on an assumption of equal examiner-specific error rates. Corresponding 95% confidence intervals are (0.305%, 1.42%) and (0.548%, 1.57%) for false positives for bullets and cartridge cases, respectively, and (1.89%, 4.26%) and (1.16%, 2.99%) for false negatives for bullets and cartridge cases, respectively. The results of this study are consistent with prior studies, despite its comprehensive design and challenging specimens.

Forensic comparison of fired cartridge cases: Feature-extraction methods for feature-based calculation of likelihood ratios

We describe and validate a feature-based system for calculation of likelihood ratios from 3D digital images of fired cartridge cases. The system includes a database of 3D digital images of the bases of 10 cartridges fired per firearm from approximately 300 firearms of the same class (semi-automatic pistols that fire 9 mm diameter centre-fire Luger-type ammunition, and that have hemispherical firing pins and parallel breech-face marks). The images were captured using Evofinder®, an imaging system that is commonly used by operational forensic laboratories. A key component of the research reported is the comparison of different feature-extraction methods. Feature sets compared include those previously proposed in the literature, plus Zernike-moment based features. Comparisons are also made of using feature sets extracted from the firing-pin impression, from the breech-face region, and from the whole region of interest (firing-pin impression + breech-face region + flowback if present). Likelihood ratios are calculated using a statistical modelling pipeline that is standard in forensic voice comparison. Validation is conducted and results are assessed using validation procedures and validation metrics and graphics that are standard in forensic voice comparison.

Human and machine similarity judgments in forensic firearm comparisons

It is unclear whether humans assess similarity differently than automated algorithms in firearms comparisons. Human participants (untrained in firearm examination) were asked to assess the similarity of pairs of images (from 0 to 100). A sample of 40 pairs of cartridge casing 2D-images was used. The images were divided into 4 groups according to their similarity as determined by an algorithm. Humans were able to distinguish between matches and non-matches (both when shown the 2 middle groups, as well as when shown all 4 groups). Thus, humans are able to make high-quality similarity judgments in firearm comparisons based on two images. The humans' similarity scores were superior to the algorithms' scores at distinguishing matches and non-matches, but inferior in assessing similarity within groups. This suggests that humans do not have the same group thresholds as the algorithm, and that a hybrid human-machine approach could provide better identification results than humans or algorithms alone.

Evaluating firearm examiner conclusion variability using cartridge case reproductions

The forensic science pattern comparison areas, including fingerprints, footwear, and firearms, have been criticized for their subjective nature. While much research has attempted to move these disciplines to more objective methods, examiners are still coming to conclusions based on their own training and experience. To complement this subjectivity, black box studies are necessary to establish the accuracy of these feature-comparison methods. However, when cartridges are fired by a firearm to create cartridge case test sets there may be significant variability within the resulting impressions. This can result in different participants receiving test sets with varying levels of difficulty based on differences in impression quality. Therefore, comparison of accuracy between examiners is not straightforward. To compare accuracy between examiners, a method called double-casting was used to create plastic cartridge case reproductions. Double-casts of twenty-one test sets of master cartridge cases were created and mailed to firearm examiners. The double-casts ensured that all participants were comparing exhibits with the same level of detail. The examiners were tasked with determining if the unknown cartridge case in each set was fired by the same firearm as the three knowns. Automated comparisons were also used to compare the cartridge cases within each set. The results from this study showed that there are differences in examiner conclusions when examining the same evidence. Furthermore, it was shown that automated comparison metrics would benefit examiners as a quality control measure to correct any potential errors and strengthen conclusions.

Convergence-improved congruent matching cells (CMC) method for firing pin impression comparison

A firing pin impression is usually concave in shape with a small textured area, which makes it difficult to perform automated algorithm-based comparison. The congruent matching cells (CMC) method was invented for accurate breech face impression comparison, in which a reference impression is divided into correlation cells. Each cell is registered to a cell-sized area of the comparison impression that has maximum similarity in surface topography. Four parameters are used to quantify the congruent matching pattern of the registration position and orientation. This paper aims to further develop the cell-division-matching method based on a convergence feature and to develop practical convergence-improved algorithms for firing pin impression comparison. The convergence feature refers to the tendency of the x-y registration positions of correlated cell pairs to converge at the correct registration angle when comparing same-source samples at different orientations. The areal Gaussian filter is employed to extract high-frequency micro-features; the least-squares matching method is used to improve each cross-correlation precision and reach convergence in the registration positions of correlated cell pairs; and a density-based clustering algorithm is introduced to collect the registration positions of dense cell pairs relative to a virtual common center and to remove outliers. Improvements are achieved in the reliability and accuracy of the number of congruent matching cell pairs (CMCs) collected, which represents the quantification of the degree of pairwise impression similarity. Experiments in this report used 40 firing pin impression samples on cartridge cases fired from 10 pistols. The results included no false identifications or false exclusions.

Firearm examination: Examiner judgments and computer-based comparisons

Forensic firearm examination provides the court of law with information about the source of fired cartridge cases. We assessed the validity of source decisions of a computer-based method and of 73 firearm examiners who compared breechface and firing pin impressions of 48 comparison sets. We also compared the computer-based method's comparison scores with the examiners' degree-of-support judgments and assessed the validity of the latter. The true-positive rate (sensitivity) and true-negative rate (specificity) of the computer-based method (for the comparison of both the breechface and firing pin impressions) were 94.4% and at least 91.7%, respectively. For the examiners, the true-positive rate was at least 95.3% and the true-negative rate was at least 86.2%. The validity of the source decisions improved when the evaluations of breechface and firing pin impressions were combined and for the examiners also when the perceived difficulty of the comparison decreased. The examiners were reluctant to provide source decisions for "difficult" comparisons even though their source decisions were mostly correct. The correlation between the computer-based method's comparison scores and the examiners' degree-of-support judgments was low for the same-source comparisons to negligible for the different-source comparisons. Combining the outcomes of computer-based methods with the judgments of examiners could increase the validity of firearm examinations. The examiners' numerical degree-of-support judgments for their source decisions were not well-calibrated and showed clear signs of overconfidence. We suggest studying the merits of performance feedback to calibrate these judgments.

Interpol review of forensic firearm examination 2016-2019

This review paper covers the relevant literature on forensic firearm examination from 2016 to 2019 as a part of the 19th Interpol International Forensic Science Managers Symposium. The review papers are also available at the Interpol website at: https://www.interpol.int/content/download/14458/file/Interpol%20Review%20Papers%202019.pdf.

Objective Identification of Bullets Based on 3D Pattern Matching and Line Counting Scores

In firearm identification, a firearm examiner looks at a pair of fired bullets or cartridge cases using a comparison microscope and determines from this visual analysis if they were both fired from the same firearm. In the particular case of fired bullets, the individual firearm signature takes the form of a striated pattern. Over the time, the firearm examiner’s community developed two distinct approaches for bullet identification: pattern matching and line counting. More recently, the emergence of technology enabling the capture of surface topographies down to a submicron depth resolution has been a catalyst for the field of computerized objective ballistic identification. Objectiveness is achieved through the statistical analysis of various scores of known matches and known nonmatches exhibit pair comparison, which in turn implies the capture of large quantities of bullets and cartridge cases topographies. The main goal of this study was to develop an objective identification method for bullets fired from conventionally rifled barrels, and to test this method on public and proprietary bullet 3D image datasets captured at different lateral resolutions. Two newly developed bullet identification scores, the Line Counting Score (LCS) and the Pattern Matching Score, computed on 3D topographies yielded perfect match versus nonmatch separation for three different sets used in the standard Hamby–Brundage Test. A similar analysis performed using a larger, more-realistic set, enabled us to define a discriminative line at a false match rate of 1/10[Formula: see text]000 on a 2D plot that shows both identification scores for matches and nonmatches. The LCS is shown to produce a better sensitivity than the standard consecutive matching striae criteria for the more-realistic dataset. A likelihood function was also computed from a linear combination of both scores, and a conservative approach based on extreme value theory is proposed to extrapolate this function in the score domain where nonmatch data are not available. This study also provides a better understanding of the limitations of studies that involve very few firearms.