Photogrammetry using visible, infrared, hyperspectral and thermal imaging of crime scenes

A review of the contributions of forensic archaeology and anthropology to the process of disaster victim identification

Forensic archaeology and anthropology have developed significantly over past decades and now provide considerable assistance to the investigation process of disaster victim recovery and identification. In what are often chaotic death and crime scenes, the formal process of utilizing archaeological methods can bring control, order, and ensure systematic search. Procedures assist in defining scene extent, locating victims and evidence, rule out areas for consideration, and provide standardized recording and quality assurance through dedicated use of standardized forms (pro formas). Combined archaeological and anthropological search methods maximize opportunities to recovery the missing through identifying remains, mapping distributions, and providing accounting of victims at the scene. Anthropological assistance in examinations contributes to individual assessment, resolving commingling and fragmentation issues, and utilizing DNA sampling methods and matching data to reassociate and account for the missing. Utilization of archaeology, anthropology, and DNA matching data provides scope to review crime scene recovery and determine requirements and potential for further survey and retrieval. Adopting the most suitable methods for a particular context can maximize recovery, efficiency, and resource use. Case studies demonstrate the utility of archaeological methods in a range of scenarios. They exemplify the success of multidisciplinary analysis in providing evidence of the sequence of events, the timing of events, the impact of taphonomic processes, the location and accounting of victims, and the demonstration of systematic scene search. The considerations provided in this article, utilizing archaeology and anthropology processes, may assist investigators in planning and implementing responses to mass fatalities.

Fusion of Multimodal Imaging and 3D Digitization Using Photogrammetry

Multimodal sensors capture and integrate diverse characteristics of a scene to maximize information gain. In optics, this may involve capturing intensity in specific spectra or polarization states to determine factors such as material properties or an individual's health conditions. Combining multimodal camera data with shape data from 3D sensors is a challenging issue. Multimodal cameras, e.g., hyperspectral cameras, or cameras outside the visible light spectrum, e.g., thermal cameras, lack strongly in terms of resolution and image quality compared with state-of-the-art photo cameras. In this article, a new method is demonstrated to superimpose multimodal image data onto a 3D model created by multi-view photogrammetry. While a high-resolution photo camera captures a set of images from varying view angles to reconstruct a detailed 3D model of the scene, low-resolution multimodal camera(s) simultaneously record the scene. All cameras are pre-calibrated and rigidly mounted on a rig, i.e., their imaging properties and relative positions are known. The method was realized in a laboratory setup consisting of a professional photo camera, a thermal camera, and a 12-channel multispectral camera. In our experiments, an accuracy better than one pixel was achieved for the data fusion using multimodal superimposition. Finally, application examples of multimodal 3D digitization are demonstrated, and further steps to system realization are discussed.

Hyperspectral imaging in forensic science: an overview of major application areas

Analysis of evidence is a challenge. Crime scene materials are complex, diverse, sometimes of an unknown nature. Forensic science provides the most critical applications for their examination. Chemical tests, analytical methods, and techniques to process the evidence must be carefully selected by the forensic scientist. Ideally, it may be interpreted, analyzed, and judged in the original context of the crime scene. In this sense, hyperspectral imaging (HSI) has been employed as an analytical tool that maintains the integrity of the samples/objects for multiple and sequential analysis and for counter-proof exams. This paper is an overview of forensic science trends for the application of HSI techniques in the last ten years (2011-2021). The examination of documents was the main area of exploration, followed by bloodstain analysis aging process; trace analysis of explosives and gunshot residue. Chemometric tools were also addressed since they are crucial to obtain the most important information from the samples. There are great challenges in applying HSI in forensic science, but there have been clear technological and scientific advances, and a solid foundation has been built for the use of HSI in real-life cases.

Hyperspectral imaging-based cutaneous wound classification using neighbourhood extraction 3D convolutional neural network

OBJECTIVES

Hyperspectral imaging is an emerging imaging modality that beginning to gain attention for medical research and has an important potential in clinical applications. Nowadays, spectral imaging modalities such as multispectral and hyperspectral have proven their ability to provide important information that can help to better characterize the wound. Oxygenation changes in the wounded tissue differ from normal tissue. This causes the spectral characteristics to be different. In this study, it is classified cutaneous wounds with neighbourhood extraction 3-dimensional convolutional neural network method.

METHODS

The methodology of hyperspectral imaging performed to obtain the most useful information about the wounded and normal tissue is explained in detail. When the hyperspectral signatures of wounded and normal tissues are compared on the hyperspectral image, it is revealed that there is a relative difference between them. By taking advantage of these differences, cuboids that also consider neighbouring pixels are generated, and a uniquely designed 3-dimensional convolutional neural network model is trained with the cuboids to extract both spatial and spectral information.

RESULTS

The effectiveness of the proposed method was evaluated for different cuboid spatial dimensions and training/testing rates. The best result with 99.69% was achieved when the training/testing rate was 0.9/0.1 and the cuboid spatial dimension was 17. It is observed that the proposed method outperforms the 2-dimensional convolutional neural network method and achieves high accuracy even with much less training data. The obtained results using the neighbourhood extraction 3-dimensional convolutional neural network method show that the proposed method highly classifies the wounded area. In addition, the classification performance and the2computation time of the neighbourhood extraction 3-dimensional convolutional neural network methodology were analyzed and compared with existing 2-dimensional convolutional neural network.

CONCLUSIONS

As a clinical diagnostic tool, hyperspectral imaging, with neighbourhood extraction 3-dimensional convolutional neural network, has yielded remarkable results for the classification of wounded and normal tissues. Skin color does not play any role in the success of the proposed method. Since only the reflectance values of the spectral signatures are different for various skin colors. For different ethnic groups, The spectral signatures of wounded tissue and the spectral signatures of normal tissue show similar spectral characteristics among themselves.

Integration of Thermal and RGB Data Obtained by Means of a Drone for Interdisciplinary Inventory

Thermal infrared imagery is very much gaining in importance in the diagnosis of energy losses in cultural heritage through non-destructive measurement methods. Hence, owing to the fact that it is a very innovative and, above all, safe solution, it is possible to determine the condition of the building, locate places exposed to thermal escape, and plan actions to improve the condition of the facility. The presented work is devoted to the technology of creating a dense point cloud and a 3D model, based on data obtained from UAV. It has been shown that it is possible to build a 3D point model based on thermograms with the specified accuracy by using thermal measurement marks and the dense matching method. The results achieved in this way were compared and, as the result of this work, the model obtained from color photos was integrated with the point cloud created on the basis of the thermal images. The discussed approach exploits measurement data obtained with three independent devices (tools/appliances): a Matrice 300 RTK drone (courtesy of NaviGate); a Phantom 4 PRO drone; and a KT-165 thermal imaging camera. A stone church located in the southern part of Poland was chosen as the measuring object.

Next-generation time of death estimation: combining surrogate model-based parameter optimization and numerical thermodynamics

The postmortem interval (PMI), i.e. the time since death, plays a key role in forensic investigations, as it aids in the reconstruction of the timeline of events. Currently, the standard method for PMI estimation empirically correlates rectal temperatures and PMIs, frequently necessitating subjective correction factors. To address this shortcoming, numerical thermodynamic algorithms have recently been developed, providing rigorous methods to simulate postmortem body temperatures. Comparing these with measured body temperatures then allows non-subjective PMI determination. This approach, however, hinges on knowledge of two thermodynamic input parameters, which are often irretrievable in forensic practice: the ambient temperature prior to discovery of the body and the body temperature at the time of death (perimortem). Here, we overcome this critical limitation by combining numerical thermodynamic modelling with surrogate model-based parameter optimization. This hybrid computational framework predicts the two unknown parameters directly from the measured postmortem body temperatures. Moreover, by substantially reducing computation times (compared with conventional optimization algorithms), this powerful approach is uniquely suited for use directly at the crime scene. Crucially, we validated this method on deceased human bodies and achieved the lowest PMI estimation errors to date (0.18 h ± 0.77 h). Together, these aspects fundamentally expand the applicability of numerical thermodynamic PMI estimation.

Individualised and non-contact post-mortem interval determination of human bodies using visible and thermal 3D imaging

Determining the time since death, i.e., post-mortem interval (PMI), often plays a key role in forensic investigations. The current standard PMI-estimation method empirically correlates rectal temperatures and PMIs, frequently necessitating subjective correction factors. To overcome this, we previously developed a thermodynamic finite-difference (TFD) algorithm, providing a rigorous method to simulate post-mortem temperatures of bodies assuming a straight posture. However, in forensic practice, bodies are often found in non-straight postures, potentially limiting applicability of this algorithm in these cases. Here, we develop an individualised approach, enabling PMI reconstruction for bodies in arbitrary postures, by combining photogrammetry and TFD modelling. Utilising thermal photogrammetry, this approach also represents the first non-contact method for PMI reconstruction. The performed lab and crime scene validations reveal PMI reconstruction accuracies of 0.26 h ± 1.38 h for true PMIs between 2 h and 35 h and total procedural durations of ~15 min. Together, these findings broaden the potential applicability of TFD-based PMI reconstruction.

Establishing the time since death (TSD) is vital in many forensic investigations. By combining thermometry, photogrammetry and numerical thermodynamic modelling, the TSD can be determined non-invasively for bodies of arbitrary shape and posture with an unprecedented accuracy of 0.26 h ± 1.38 h.

Beyond the visible spectrum - applying 3D multispectral full-body imaging to the VirtoScan system

Multispectral photography offers a wide range of applications for forensic investigations. It is commonly used to detect latent evidence and to enhance the visibility of findings. Additionally, three-dimensional (3D) full-body documentation has become much easier and more affordable in recent years. However, the benefits of performing 3D imaging beyond the visible (VIS) spectrum are not well known, and the technique has not been widely used in forensic medical investigations. A multicamera setup was used to employ multispectral photogrammetry between 365 and 960 nm in postmortem investigations. The multicamera setup included four modified digital cameras, ultraviolet (UV) and near-infrared (NIR) light sources and supplemental lens filters. Full-body documentation was performed in conjunction with the use of a medical X-ray computed tomography (CT) scanner to automate the imaging procedure. Textured 3D models based on multispectral datasets from four example cases were reconstructed successfully. The level of detail and overall quality of the 3D reconstructions varied depending on the spectral range of the image data. Generally, the NIR datasets showed enhanced visibility of vein patterns and specific injuries, whereas the UV-induced datasets highlighted foreign substances on the skin. Three-dimensional multispectral full-body imaging enables the detection of latent evidence that is invisible to the naked eye and allows visualization, documentation and analysis of evidence beyond the VIS spectrum.

TaijiGNN: A New Cycle-Consistent Generative Neural Network for High-Quality Bidirectional Transformation between RGB and Multispectral Domains

Since multispectral images (MSIs) and RGB images (RGBs) have significantly different definitions and severely imbalanced information entropies, the spectrum transformation between them, especially reconstructing MSIs from RGBs, is a big challenge. We propose a new approach, the Taiji Generative Neural Network (TaijiGNN), to address the above-mentioned problems. TaijiGNN consists of two generators, G_MSI, and G_RGB. These two generators establish two cycles by connecting one generator's output with the other's input. One cycle translates the RGBs into the MSIs and converts the MSIs back to the RGBs. The other cycle does the reverse. The cycles can turn the problem of comparing two different domain images into comparing the same domain images. In the same domain, there are neither different domain definition problems nor severely underconstrained challenges, such as reconstructing MSIs from RGBs. Moreover, according to several investigations and validations, we effectively designed a multilayer perceptron neural network (MLP) to substitute the convolutional neural network (CNN) when implementing the generators to make them simple and high performance. Furthermore, we cut off the two traditional CycleGAN's identity losses to fit the spectral image translation. We also added two consistent losses of comparing paired images to improve the two generators' training effectiveness. In addition, during the training process, similar to the ancient Chinese philosophy Taiji's polarity Yang and polarity Yin, the two generators update their neural network parameters by interacting with and complementing each other until they all converge and the system reaches a dynamic balance. Furthermore, several qualitative and quantitative experiments were conducted on the two classical datasets, CAVE and ICVL, to evaluate the performance of our proposed approach. Promising results were obtained with a well-designed simplistic MLP requiring a minimal amount of training data. Specifically, in the CAVE dataset, to achieve comparable state-of-the-art results, we only need half of the dataset for training; for the ICVL dataset, we used only one-fifth of the dataset to train the model, but obtained state-of-the-art results.

Technological innovation in the recovery and analysis of 3D forensic footwear evidence: Structure from motion (SfM) photogrammetry

The recovery of three-dimensional footwear impressions at crime scenes can be a challenge but can also yield important investigative data. Traditional methods involve casting 3D impressions but these methods have limitations: the trace is usually destroyed during capture; the process can be time consuming, with a risk of failure; and the resultant cast is bulky and therefore difficult to share and store. The use of Structure from Motion (SfM) photogrammetry has been used widely to capture fossil footprints in the geological record and while there is a small body of work advocating its use in forensic practice the full potential of this technique has yet to be realised in an operational context. The availability of affordable software is one limiting factor and here we report the availability of a bespoke freeware for SfM recovery and subsequent analysis of for footwear evidence (DigTrace). Our aim here is not to provide a rigorous comparison of SfM methods to other recovery methods, but more to illustrate the potential while also documenting the typical workflows and potential errors associated with an SfM based approach. By doing so we hope to encourage further research, experimentation and ultimately adoption by practitioners.

Virtual anthropology? Reliability of three-dimensional photogrammetry as a forensic anthropology measurement and documentation technique

Osseous remains provide forensic anthropologists with morphological and osteometric information that can be used in building a biological profile. By conducting a visual and physical examination, an anthropologist can infer information such as the sex and age of the deceased. Traditionally, morphological and osteometric information is gathered by physically handling remains for analysis. With the advancement of digital technology, there has been a shift from direct to indirect methods of analysis by utilizing models generated from three-dimensional (3D) imaging, which includes computed tomography (CT) scanning and 3D photogrammetry. Although CT scanning is more common, photogrammetry has found application in a range of fields such as architecture, geography and road accident reconstruction. The application of modern-day photogrammetry for forensic anthropology purposes, however, has not been discussed extensively. The aim of this research was to validate the accuracy of 3D models generated by photogrammetry by comparing them to both 3D models generated by CT scanning and the actual physical models. In this study, six 3D models were created using photogrammetry (n = 3) and CT scanning (n = 3). The 3D models were generated from three different Bone Clone® human skulls. A mobile phone camera was used to capture images, which were then processed in Agisoft Metashape®. Intrarater, interrater, and intermethod reliability tests gave correlation coefficients of at least 0.9980, 0.9871, and 0.9862, respectively; rTEM results ranged from 0.250 to 6.55%; and an analysis of variance (ANOVA) yielded P values under 0.05 for all measurements except one. Statistical tests therefore showed photogrammetry to be a reliable and accurate alternative to more expensive CT scanning approaches.

Analysis of the Thermal Environment in Pedestrian Space Using 3D Thermal Imaging

Pedestrian space is an important place for people’s outdoor activities. Its thermal environment affects pedestrian walking experience, route selection and physical health. This study presents a 3D thermography-based method to analyze and evaluate the spatial distribution of thermal comfort. The proposed 3D thermal image was generated using 3D city models captured by an unmanned aerial vehicle (UAV) and thermal images gathered by an infrared camera. It can visualize construction elements, but also simply output surface temperatures at selected points. This paper described the process of using 3D thermal images to analyze the built environment, and selected two pedestrian spaces as case study objects. Their thermal images and mean radiant temperatures (MRT) were obtained from field measurement data collected by a drone and infrared camera. The following findings were obtained: (a) the MRT difference in the pedestrian space between sunlit and shaded areas was more than 3 °C; (b) the MRT values at the measurement points near vegetation were lower; (c) when the ratio of street height to width (H/W) was larger, the MRT values at all measurement points varied slightly. These findings can be used for the designers to evaluate and improve the thermal environment in pedestrian space.

Reconstructing the time since death using noninvasive thermometry and numerical analysis

The early postmortem interval (PMI), i.e., the time shortly after death, can aid in the temporal reconstruction of a suspected crime and therefore provides crucial information in forensic investigations. Currently, this information is often derived from an empirical model (Henssge's nomogram) describing posthumous body cooling under standard conditions. However, nonstandard conditions necessitate the use of subjective correction factors or preclude the use of Henssge's nomogram altogether. To address this, we developed a powerful method for early PMI reconstruction using skin thermometry in conjunction with a comprehensive thermodynamic finite-difference model, which we validated using deceased human bodies. PMIs reconstructed using this approach, on average, deviated no more than ±38 minutes from their corresponding true PMIs (which ranged from 5 to 50 hours), significantly improving on the ±3 to ±7 hours uncertainty of the gold standard. Together, these aspects render this approach a widely applicable, i.e., forensically relevant, method for thermometric early PMI reconstruction.

Communicating 3D data-interactive 3D PDF documents for expert reports and scientific publications in the field of forensic medicine

INTRODUCTION

Modern forensic investigations increasingly revert to 3D imaging techniques, such as computed tomography, magnetic resonance imaging, and 3D surface imaging. Findings are therefore often based on 3D data sets; however, this information is commonly reported and communicated within 2D imagery. The use of interactive 3D PDFs is already established in the scientific community but has yet to be implemented in the field of forensic medicine.

METHODS AND MATERIALS

Three example cases were chosen to serve as exemplary data for the most commonly applied imaging techniques in postmortem imaging. 3D surface models were created from postmortem magnetic resonance imaging (PMMR), postmortem computed tomography (PMCT), and 3D surface imaging data sets.

RESULTS

PMMR revealed a space-occupying subdural hemorrhage that led to ipsilateral compression of the brain tissue of the right hemisphere. PMCT displayed a defect in the skull on the left side of the temporal bone. 3D surface imaging data displayed a patterned discoloration on the inside of the left forearm.

DISCUSSION

Interactive 3D PDFs offer the possibility to communicate 3D information to the reader while maintaining all the benefits of a regular 2D PDF. With Adobe Acrobat, the reader can interactively navigate through 3D data sets and create sufficient depth cues to generate a realistic 3D perception of the data.

CONCLUSION

The interactive 3D PDF is a useful extension of standard 2D PDFs and has the potential to communicate 3D data to the reader in a more complete, more comprehensible, and less subjective manner than 2D PDFs.

Reflective spectroscopy investigations of clothing items to support law enforcement, search and rescue, and war crime investigations

Clothing articles are important pieces of evidence in criminal, search and rescue, and search and recovery investigations. Hyperspectral remote sensing of clothing will be an important tool for supporting such investigations in the near future. This study investigated over 300 items of clothing that varied in fabric type, texture, color, and pattern. Clothing items were analyzed using an ASD FieldSpec 4 High Resolution spectroradiometer with a contact probe attachment. Of the clothing items analyzed, there were 141 having endmember fabrics (100% single fabric type composition): 89 were cotton, 39 were polyester, 5 were wool, 1 was cashmere, 3 were acrylic, 1 was leather, and 3 were rayon. The remaining 164 clothing items were various fabric blends. Spectral features relating to different fabric types exhibit sufficient differences that allow them to be discriminated from the surrounding environment, as well as from one another in many, but not all, cases. Cotton and polyester, in particular, two of the most widely-used fabrics, exhibit numerous features in the near infrared (NIR) and shortwave infrared (SWIR) that would allow them to easily be distinguished from geologic materials in the environment such as rocks and soil. Plant based fibers, especially cotton, possess similar reflectance features to vegetation owing to their cellulose content. Outdoor aging experiments were conducted for 19 weeks on selected fabrics. Although significant changes were observed in aged garments, the variability observed in the reflectance of the aged garments does not support the derivation of a metric for aging, at least over the relatively short time scale of this effort. Results from this study should support numerous forensic efforts globally for non-destructive investigation of clothing items in the field and in lab settings with a spectroradiometer, enhance the potential for remote sensing searches, and in the future, potentially documenting crime scenes with hyperspectral imaging.

VirtoScan-on-Rails - an automated 3D imaging system for fast post-mortem whole-body surface documentation at autopsy tables

Two-dimensional photographic documentation is a substantial part of post-mortem examinations for legal investigations. Additional three-dimensional surface documentation has been shown to assist in the visualization of findings and contribute to the reconstruction of the sequence of events. However, 2D photo documentation and, especially, 3D surface documentation, are time-consuming procedures that require specially trained personnel. In this study a 3D imaging system, called VirtoScan-on-Rails, was developed to automate and facilitate 3D surface documentation for photo documentation in autopsy suites. The imaging system was built to quickly acquire photogrammetric image sets of whole bodies during different stages of external and internal examinations. VirtoScan-on-Rails was set up in the autopsy suite of the Zurich Institute of Forensic Medicine at the University of Zurich (Zurich, Switzerland). The imaging system is based on a movable frame that carries a multi-camera array. Data quality and the applicability of the system were analyzed and evaluated within two test series. Up to 200 overlapping photographic images were acquired at consecutive image-capturing positions over a distance of approximately 2000 mm. The image-capturing process took 1 min and 23 s to acquire a set of 200 images for one side of the body. During test series one and two, 53 photogrammetric image sets taken from 31 forensic cases were successfully reconstructed. VirtoScan-on-Rails is an automated, fast and easy-to-use 3D imaging setup for autopsy suits. It facilitates documenting bodies during different stages of forensic examinations and allows standardizing the procedure of photo documentation.