Digital restoration of fragmentary human skeletal remains: Testing the feasibility of virtual reality

Utilizing augmented reality for reconstruction of fractured, fragmented and damaged craniofacial remains in forensic anthropology

Forensic anthropologists are often confronted with human remains that have been damaged due to trauma, fire, or postmortem taphonomic alteration, frequently resulting in the fracture and fragmentation of skeletal elements. The augmented reality (AR) technology introduced in this paper builds on familiar 3D visualization methods and utilizes them to make three dimensional holographic meshes of skeletal fragments that can be manipulated, tagged, and examined by the user. Here, CT scans, neural radiance fields (NeRF) artificial intelligence software, and Unreal Engine production software are utilized to construct a three-dimensional holographic image that can be manipulated with HoloLens™ technology to analyze the fracture margin and reconstruct craniofacial elements without causing damage to fragile remains via excessive handling. This allows forensic anthropologists a means of assessing aspects of the biological profile and traumatic injuries without risking further damage to the skeleton. It can also be utilized by students and professional anthropologists to practice refitting before reconstructing craniofacial fragments if refitting is necessary. Additionally, the holographic images can be used to explain complicated concepts in a courtroom without the emotional response related to using bony elements as courtroom exhibits.

PuzzleFixer: A Visual Reassembly System for Immersive Fragments Restoration

We present PuzzleFixer, an immersive interactive system for experts to rectify defective reassembled 3D objects. Reassembling the fragments of a broken object to restore its original state is the prerequisite of many analytical tasks such as cultural relics analysis and forensics reasoning. While existing computer-aided methods can automatically reassemble fragments, they often derive incorrect objects due to the complex and ambiguous fragment shapes. Thus, experts usually need to refine the object manually. Prior advances in immersive technologies provide benefits for realistic perception and direct interactions to visualize and interact with 3D fragments. However, few studies have investigated the reassembled object refinement. The specific challenges include: 1) the fragment combination set is too large to determine the correct matches, and 2) the geometry of the fragments is too complex to align them properly. To tackle the first challenge, PuzzleFixer leverages dimensionality reduction and clustering techniques, allowing users to review possible match categories, select the matches with reasonable shapes, and drill down to shapes to correct the corresponding faces. For the second challenge, PuzzleFixer embeds the object with node-link networks to augment the perception of match relations. Specifically, it instantly visualizes matches with graph edges and provides force feedback to facilitate the efficiency of alignment interactions. To demonstrate the effectiveness of PuzzleFixer, we conducted an expert evaluation based on two cases on real-world artifacts and collected feedback through post-study interviews. The results suggest that our system is suitable and efficient for experts to refine incorrect reassembled objects.

Digital Reconstructions Using Linear Regression: How Well Can It Estimate Missing Shape Data from Small Damaged Areas?

Skeletal remains analyzed by anthropologists, paleontologists and forensic scientists are usually found fragmented or incomplete. Accurate estimations of the original morphologies are a challenge for which several digital reconstruction methods have been proposed. In this study, the accuracy of reconstructing bones based on multiple linear regression (RM) was tested. A total of 150 digital models from complete zygomatics from recent past populations (European and African American) were studied using high-density geometric morphometrics. Some landmarks (i.e., 2, 3 and 6) were coded as missing to simulate incomplete zygomatics and the missing landmarks were estimated with RM. In the zygomatics, this simulated damage affects a few square centimeters or less. Finally, the predicted and original shape data were compared. The results indicate that the predicted landmark coordinates were significantly different from the original ones, although this difference was less than the difference between the original zygomatic and the mean zygomatic in the sample. The performance of the method was affected by the location and the number of missing landmarks, with decreasing accuracy with increasing damaged area. We conclude that RM can accurately estimate the original appearance of the zygomatics when the damage is small.

The big puzzle: A critical review of virtual re-association methods for fragmented human remains in a DVI context'

During a Disaster Victims Identification (DVI) mission, international protocols rely on interdisciplinary work, especially between specialists from forensic imaging and anthropology. In case of air crashes or explosions, DVI units may face thousands of fragmented human remains (FHRs). The physical re-association of FHRs and the identification process is very complex and challenging, and relies upon expensive and destructive DNA analysis. A virtual re-association (VRA) of these fragments, using Multidetector Computed Tomography (MDCT), could be a helpful tool in forensic anthropology analysis, as it could assist in reducing the number of DNA samples. However, there is no standardized protocol for including such an approach into a DVI procedure. The aim of this study was to summarize and analyze existing techniques through a systematic review and to develop a protocol for virtual re-association of FHRs, adapted to the DVI context. A keyword-based literature search was conducted, focusing on the VRA methods using MDCT imaging and 3D surface scan methodology. Reviews and primary articles, published between 2005 and 2020 in the fields of forensic anthropology, paleoanthropology, archeology, and fracture reduction surgery were sorted out. A total of 45 publications were selected and analyzed based on their content and relevance. The results show that research on the re-association of FHRs increased significantly during the last five years. Seven steps regarding the MDCT-based method for the virtual re-association of FHRs could be identified: acquisition of 3D-images, segmentation of the MDCT-data, post-processing and surface generation, identification of intact and fracture surfaces, identification and registration of matching fragments, and validation of the re-association. The literature is surprisingly sparse regarding the FHRs re-association as a forensic tool, and mainly consists in case reports, whereas validated methods were presented in archeology and surgery publications. However, we were able to adapt the MDCT-based approach for the virtual re-association of the FHRs and propose an innovative protocol for DVI missions. This protocol includes the needed details, from the acquisition of MDCT imaging to the virtual re-association of 3D models and its validation. Each step has to be fully tested, adapted and validated in future studies.

Virtual reality and its transformation in forensic education and research practices

Documentation and evidence analysis are major components in forensic investigation; hence two-dimensional (2D) photographs along with three-dimensional (3D) models and data are used to accomplish this task. Data generated through 3D scanning and photogrammetry are generally visualised on a computer screen. However, spatial details are lost on the visualisation of 3D data on 2D computer screens. Virtual reality (VR) is an immersive technology that allows a user to visualise 3D information by immersing oneself into the scene. In forensics, VR was particularly introduced for the visualising and plotting distances of crime scenes; however, this technology has wider applications in the field of forensics and for court presentation. This short communication outlines the concept of VR and its potential in the field of forensics.