AutoImplant 2020-First MICCAI Challenge on Automatic Cranial Implant Design

MedShapeNet - a large-scale dataset of 3D medical shapes for computer vision

OBJECTIVES

The shape is commonly used to describe the objects. State-of-the-art algorithms in medical imaging are predominantly diverging from computer vision, where voxel grids, meshes, point clouds, and implicit surface models are used. This is seen from the growing popularity of ShapeNet (51,300 models) and Princeton ModelNet (127,915 models). However, a large collection of anatomical shapes (e.g., bones, organs, vessels) and 3D models of surgical instruments is missing.

METHODS

We present MedShapeNet to translate data-driven vision algorithms to medical applications and to adapt state-of-the-art vision algorithms to medical problems. As a unique feature, we directly model the majority of shapes on the imaging data of real patients. We present use cases in classifying brain tumors, skull reconstructions, multi-class anatomy completion, education, and 3D printing.

RESULTS

By now, MedShapeNet includes 23 datasets with more than 100,000 shapes that are paired with annotations (ground truth). Our data is freely accessible via a web interface and a Python application programming interface and can be used for discriminative, reconstructive, and variational benchmarks as well as various applications in virtual, augmented, or mixed reality, and 3D printing.

CONCLUSIONS

MedShapeNet contains medical shapes from anatomy and surgical instruments and will continue to collect data for benchmarks and applications. The project page is: https://medshapenet.ikim.nrw/.

Intelligent surgical planning for automatic reconstruction of orbital blowout fracture using a prior adversarial generative network

Orbital blowout fracture (OBF) is a disease that can result in herniation of orbital soft tissue, enophthalmos, and even severe visual dysfunction. Given the complex and diverse types of orbital wall fractures, reconstructing the orbital wall presents a significant challenge in OBF repair surgery. Accurate surgical planning is crucial in addressing this issue. However, there is currently a lack of efficient and precise surgical planning methods. Therefore, we propose an intelligent surgical planning method for automatic OBF reconstruction based on a prior adversarial generative network (GAN). Firstly, an automatic generation method of symmetric prior anatomical knowledge (SPAK) based on spatial transformation is proposed to guide the reconstruction of fractured orbital wall. Secondly, a reconstruction network based on SPAK-guided GAN is proposed to achieve accurate and automatic reconstruction of fractured orbital wall. Building upon this, a new surgical planning workflow based on the proposed reconstruction network and 3D Slicer software is developed to simplify the operational steps. Finally, the proposed surgical planning method is successfully applied in OBF repair surgery, verifying its reliability. Experimental results demonstrate that the proposed reconstruction network achieves relatively accurate automatic reconstruction of the orbital wall, with an average DSC of 92.35 ± 2.13% and a 95% Hausdorff distance of 0.59 ± 0.23 mm, markedly outperforming the compared state-of-the-art networks. Additionally, the proposed surgical planning workflow reduces the traditional planning time from an average of 25 min and 17.8 s to just 1 min and 35.1 s, greatly enhancing planning efficiency. In the future, the proposed surgical planning method will have good application prospects in OBF repair surgery.

Improving deep learning-based automatic cranial defect reconstruction by heavy data augmentation: From image registration to latent diffusion models

Modeling and manufacturing of personalized cranial implants are important research areas that may decrease the waiting time for patients suffering from cranial damage. The modeling of personalized implants may be partially automated by the use of deep learning-based methods. However, this task suffers from difficulties with generalizability into data from previously unseen distributions that make it difficult to use the research outcomes in real clinical settings. Due to difficulties with acquiring ground-truth annotations, different techniques to improve the heterogeneity of datasets used for training the deep networks have to be considered and introduced. In this work, we present a large-scale study of several augmentation techniques, varying from classical geometric transformations, image registration, variational autoencoders, and generative adversarial networks, to the most recent advances in latent diffusion models. We show that the use of heavy data augmentation significantly increases both the quantitative and qualitative outcomes, resulting in an average Dice Score above 0.94 for the SkullBreak and above 0.96 for the SkullFix datasets. The results show that latent diffusion models combined with vector quantized variational autoencoder outperform other generative augmentation strategies. Moreover, we show that the synthetically augmented network successfully reconstructs real clinical defects, without the need to acquire costly and time-consuming annotations. The findings of the work will lead to easier, faster, and less expensive modeling of personalized cranial implants. This is beneficial to numerous people suffering from cranial injuries. The work constitutes a considerable contribution to the field of artificial intelligence in the automatic modeling of personalized cranial implants.

A Stage-Wise Residual Attention Generation Adversarial Network for Mandibular Defect Repairing and Reconstruction

Surgical reconstruction of mandibular defects is a clinical routine manner for the rehabilitation of patients with deformities. The mandible plays a crucial role in maintaining the facial contour and ensuring the speech and mastication functions. The repairing and reconstruction of mandible defects is a significant yet challenging task in oral-maxillofacial surgery. Currently, the mainly available methods are traditional digitalized design methods that suffer from substantial artificial operations, limited applicability and high reconstruction error rates. An automated, precise, and individualized method is imperative for maxillofacial surgeons. In this paper, we propose a Stage-wise Residual Attention Generative Adversarial Network (SRA-GAN) for mandibular defect reconstruction. Specifically, we design a stage-wise residual attention mechanism for generator to enhance the extraction capability of mandibular remote spatial information, making it adaptable to various defects. For the discriminator, we propose a multi-field perceptual network, consisting of two parallel discriminators with different perceptual fields, to reduce the cumulative reconstruction errors. Furthermore, we design a self-encoder perceptual loss function to ensure the correctness of mandibular anatomical structures. The experimental results on a novel custom-built mandibular defect dataset demonstrate that our method has a promising prospect in clinical application, achieving the best Dice Similarity Coefficient (DSC) of 94.238% and 95% Hausdorff Distance (HD95) of 4.787.

Sex classification of 3D skull images using deep neural networks

Determining the fundamental characteristics that define a face as "feminine" or "masculine" has long fascinated anatomists and plastic surgeons, particularly those involved in aesthetic and gender-affirming surgery. Previous studies in this area have relied on manual measurements, comparative anatomy, and heuristic landmark-based feature extraction. In this study, we collected retrospectively at Cedars Sinai Medical Center (CSMC) a dataset of 98 skull samples, which is the first dataset of this kind of 3D medical imaging. We then evaluated the accuracy of multiple deep learning neural network architectures on sex classification with this dataset. Specifically, we evaluated methods representing three different 3D data modeling approaches: Resnet3D, PointNet++, and MeshNet. Despite the limited number of imaging samples, our testing results show that all three approaches achieve AUC scores above 0.9 after convergence. PointNet++ exhibits the highest accuracy, while MeshNet has the lowest. Our findings suggest that accuracy is not solely dependent on the sparsity of data representation but also on the architecture design, with MeshNet's lower accuracy likely due to the lack of a hierarchical structure for progressive data abstraction. Furthermore, we studied a problem related to sex determination, which is the analysis of the various morphological features that affect sex classification. We proposed and developed a new method based on morphological gradients to visualize features that influence model decision making. The method based on morphological gradients is an alternative to the standard saliency map, and the new method provides better visualization of feature importance. Our study is the first to develop and evaluate deep learning models for analyzing 3D facial skull images to identify imaging feature differences between individuals assigned male or female at birth. These findings may be useful for planning and evaluating craniofacial surgery, particularly gender-affirming procedures, such as facial feminization surgery.

Back to the Roots: Reconstructing Large and Complex Cranial Defects using an Image-based Statistical Shape Model

Designing implants for large and complex cranial defects is a challenging task, even for professional designers. Current efforts on automating the design process focused mainly on convolutional neural networks (CNN), which have produced state-of-the-art results on reconstructing synthetic defects. However, existing CNN-based methods have been difficult to translate to clinical practice in cranioplasty, as their performance on large and complex cranial defects remains unsatisfactory. In this paper, we present a statistical shape model (SSM) built directly on the segmentation masks of the skulls represented as binary voxel occupancy grids and evaluate it on several cranial implant design datasets. Results show that, while CNN-based approaches outperform the SSM on synthetic defects, they are inferior to SSM when it comes to large, complex and real-world defects. Experienced neurosurgeons evaluate the implants generated by the SSM to be feasible for clinical use after minor manual corrections. Datasets and the SSM model are publicly available at https://github.com/Jianningli/ssm .

Learning continuous shape priors from sparse data with neural implicit functions

Statistical shape models are an essential tool for various tasks in medical image analysis, including shape generation, reconstruction and classification. Shape models are learned from a population of example shapes, which are typically obtained through segmentation of volumetric medical images. In clinical practice, highly anisotropic volumetric scans with large slice distances are prevalent, e.g., to reduce radiation exposure in CT or image acquisition time in MR imaging. For existing shape modeling approaches, the resolution of the emerging model is limited to the resolution of the training shapes. Therefore, any missing information between slices prohibits existing methods from learning a high-resolution shape prior. We propose a novel shape modeling approach that can be trained on sparse, binary segmentation masks with large slice distances. This is achieved through employing continuous shape representations based on neural implicit functions. After training, our model can reconstruct shapes from various sparse inputs at high target resolutions beyond the resolution of individual training examples. We successfully reconstruct high-resolution shapes from as few as three orthogonal slices. Furthermore, our shape model allows us to embed various sparse segmentation masks into a common, low-dimensional latent space - independent of the acquisition direction, resolution, spacing, and field of view. We show that the emerging latent representation discriminates between healthy and pathological shapes, even when provided with sparse segmentation masks. Lastly, we qualitatively demonstrate that the emerging latent space is smooth and captures characteristic modes of shape variation. We evaluate our shape model on two anatomical structures: the lumbar vertebra and the distal femur, both from publicly available datasets.

Thickness and design features of clinical cranial implants-what should automated methods strive to replicate?

PURPOSE

New deep learning and statistical shape modelling approaches aim to automate the design process for patient-specific cranial implants, as highlighted by the MICCAI AutoImplant Challenges. To ensure applicability, it is important to determine if the training data used in developing these algorithms represent the geometry of implants designed for clinical use.

METHODS

Calavera Surgical Design provided a dataset of 206 post-craniectomy skull geometries and their clinically used implants. The MUG500+ dataset includes 29 post-craniectomy skull geometries and implants designed for automating design. For both implant and skull shapes, the inner and outer cortical surfaces were segmented, and the thickness between them was measured. For the implants, a 'rim' was defined that transitions from the repaired defect to the surrounding skull. For unilateral defect cases, skull implants were mirrored to the contra-lateral side and thickness differences were quantified.

RESULTS

The average thickness of the clinically used implants was 6.0 ± 0.5 mm, which approximates the thickness on the contra-lateral side of the skull (relative difference of -0.3 ± 1.4 mm). The average thickness of the MUG500+ implants was 2.9 ± 1.0 mm, significantly thinner than the intact skull thickness (relative difference of 2.9 ± 1.2 mm). Rim transitions in the clinical implants (average width of 8.3 ± 3.4 mm) were used to cap and create a smooth boundary with the skull.

CONCLUSIONS

For implant modelers or manufacturers, this shape analysis quantified differences of cranial implants (thickness, rim width, surface area, and volume) to help guide future automated design algorithms. After skull completion, a thicker implant can be more versatile for cases involving muscle hollowing or thin skulls, and wider rims can smooth over the defect margins to provide more stability. For clinicians, the differing measurements and implant designs can help inform the options available for their patient specific treatment.

Creating high-resolution 3D cranial implant geometry using deep learning techniques

Creating a personalized implant for cranioplasty can be costly and aesthetically challenging, particularly for comminuted fractures that affect a wide area. Despite significant advances in deep learning techniques for 2D image completion, generating a 3D shape inpainting remains challenging due to the higher dimensionality and computational demands for 3D skull models. Here, we present a practical deep-learning approach to generate implant geometry from defective 3D skull models created from CT scans. Our proposed 3D reconstruction system comprises two neural networks that produce high-quality implant models suitable for clinical use while reducing training time. The first network repairs low-resolution defective models, while the second network enhances the volumetric resolution of the repaired model. We have tested our method in simulations and real-life surgical practices, producing implants that fit naturally and precisely match defect boundaries, particularly for skull defects above the Frankfort horizontal plane.

Sparse convolutional neural network for high-resolution skull shape completion and shape super-resolution

Traditional convolutional neural network (CNN) methods rely on dense tensors, which makes them suboptimal for spatially sparse data. In this paper, we propose a CNN model based on sparse tensors for efficient processing of high-resolution shapes represented as binary voxel occupancy grids. In contrast to a dense CNN that takes the entire voxel grid as input, a sparse CNN processes only on the non-empty voxels, thus reducing the memory and computation overhead caused by the sparse input data. We evaluate our method on two clinically relevant skull reconstruction tasks: (1) given a defective skull, reconstruct the complete skull (i.e., skull shape completion), and (2) given a coarse skull, reconstruct a high-resolution skull with fine geometric details (shape super-resolution). Our method outperforms its dense CNN-based counterparts in the skull reconstruction task quantitatively and qualitatively, while requiring substantially less memory for training and inference. We observed that, on the 3D skull data, the overall memory consumption of the sparse CNN grows approximately linearly during inference with respect to the image resolutions. During training, the memory usage remains clearly below increases in image resolution-an [Formula: see text] increase in voxel number leads to less than [Formula: see text] increase in memory requirements. Our study demonstrates the effectiveness of using a sparse CNN for skull reconstruction tasks, and our findings can be applied to other spatially sparse problems. We prove this by additional experimental results on other sparse medical datasets, like the aorta and the heart. Project page at https://github.com/Jianningli/SparseCNN .

Optimization of Fixations for Additively Manufactured Cranial Implants: Insights from Finite Element Analysis

With the emergence of additive manufacturing technology, patient-specific cranial implants using 3D printing have massively influenced the field. These implants offer improved surgical outcomes and aesthetic preservation. However, as additive manufacturing in cranial implants is still emerging, ongoing research is investigating their reliability and sustainability. The long-term biomechanical performance of these implants is critically influenced by factors such as implant material, anticipated loads, implant-skull interface geometry, and structural constraints, among others. The efficacy of cranial implants involves an intricate interplay of these factors, with fixation playing a pivotal role. This study addresses two critical concerns: determining the ideal number of fixation points for cranial implants and the optimal curvilinear distance between those points, thereby establishing a minimum threshold. Employing finite element analysis, the research incorporates variables such as implant shapes, sizes, materials, the number of fixation points, and their relative positions. The study reveals that the optimal number of fixation points ranges from four to five, accounting for defect size and shape. Moreover, the optimal curvilinear distance between two screws is approximately 40 mm for smaller implants and 60 mm for larger implants. Optimal fixation placement away from the center mitigates higher deflection due to overhangs. Notably, a symmetric screw orientation reduces deflection, enhancing implant stability. The findings offer crucial insights into optimizing fixation strategies for cranial implants, thereby aiding surgical decision-making guidelines.

Next-generation personalized cranioplasty treatment

Decompressive craniectomy (DC) is a surgical procedure, that is followed by cranioplasty surgery. DC is usually performed to treat patients with traumatic brain injury, intracranial hemorrhage, cerebral infarction, brain edema, skull fractures, etc. In many published clinical case studies and systematic reviews, cranioplasty surgery is reported to restore cranial symmetry with good cosmetic outcomes and neurophysiologically relevant functional outcomes in hundreds of patients. In this review article, we present a number of key issues related to the manufacturing of patient-specific implants, clinical complications, cosmetic outcomes, and newer alternative therapies. While discussing alternative therapeutic treatments for cranioplasty, biomolecules and cellular-based approaches have been emphasized. The current clinical practices in the restoration of cranial defects involve 3D printing to produce patient-specific prefabricated cranial implants, that provide better cosmetic outcomes. Regardless of the advancements in image processing and 3D printing, the complete clinical procedure is time-consuming and requires significant costs. To reduce manual intervention and to address unmet clinical demands, it has been highlighted that automated implant fabrication by data-driven methods can accelerate the design and manufacturing of patient-specific cranial implants. The data-driven approaches, encompassing artificial intelligence (machine learning/deep learning) and E-platforms, such as publicly accessible clinical databases will lead to the development of the next generation of patient-specific cranial implants, which can provide predictable clinical outcomes. STATEMENT OF SIGNIFICANCE: Cranioplasty is performed to reconstruct cranial defects of patients who have undergone decompressive craniectomy. Cranioplasty surgery improves the aesthetic and functional outcomes of those patients. To meet the clinical demands of cranioplasty surgery, accelerated designing and manufacturing of 3D cranial implants are required. This review provides an overview of biomaterial implants and bone flap manufacturing methods for cranioplasty surgery. In addition, tissue engineering and regenerative medicine-based approaches to reduce clinical complications are also highlighted. The potential use of data-driven computer applications and data-driven artificial intelligence-based approaches are emphasized to accelerate the clinical protocols of cranioplasty treatment with less manual intervention and shorter intraoperative time.

Deep learning-based framework for automatic cranial defect reconstruction and implant modeling

BACKGROUND AND OBJECTIVE

This article presents a robust, fast, and fully automatic method for personalized cranial defect reconstruction and implant modeling.

METHODS

We propose a two-step deep learning-based method using a modified U-Net architecture to perform the defect reconstruction, and a dedicated iterative procedure to improve the implant geometry, followed by an automatic generation of models ready for 3-D printing. We propose a cross-case augmentation based on imperfect image registration combining cases from different datasets. Additional ablation studies compare different augmentation strategies and other state-of-the-art methods.

RESULTS

We evaluate the method on three datasets introduced during the AutoImplant 2021 challenge, organized jointly with the MICCAI conference. We perform the quantitative evaluation using the Dice and boundary Dice coefficients, and the Hausdorff distance. The Dice coefficient, boundary Dice coefficient, and the 95th percentile of Hausdorff distance averaged across all test sets, are 0.91, 0.94, and 1.53 mm respectively. We perform an additional qualitative evaluation by 3-D printing and visualization in mixed reality to confirm the implant's usefulness.

CONCLUSION

The article proposes a complete pipeline that enables one to create the cranial implant model ready for 3-D printing. The described method is a greatly extended version of the method that scored 1st place in all AutoImplant 2021 challenge tasks. We freely release the source code, which together with the open datasets, makes the results fully reproducible. The automatic reconstruction of cranial defects may enable manufacturing personalized implants in a significantly shorter time, possibly allowing one to perform the 3-D printing process directly during a given intervention. Moreover, we show the usability of the defect reconstruction in a mixed reality that may further reduce the surgery time.

CranGAN: Adversarial Point Cloud Reconstruction for patient-specific Cranial Implant Design

Automatizing cranial implant design has become an increasingly important avenue in biomedical research. Benefits in terms of financial resources, time and patient safety necessitate the formulation of an efficient and accurate procedure for the same. This paper attempts to provide a new research direction to this problem, through an adversarial deep learning solution. Specifically, in this work, we present CranGAN - a 3D Conditional Generative Adversarial Network designed to reconstruct a 3D representation of a complete skull given its defective counterpart. A novel solution of employing point cloud representations instead of conventional 3D meshes and voxel grids is proposed. We provide both qualitative and quantitative analysis of our experiments with three separate GAN objectives, and compare the utility of two 3D reconstruction loss functions viz. Hausdorff Distance and Chamfer Distance. We hope that our work inspires further research in this direction. Clinical relevance- This paper establishes a new research direction to assist in automated implant design for cranioplasty.

Studierfenster: an Open Science Cloud-Based Medical Imaging Analysis Platform

Imaging modalities such as computed tomography (CT) and magnetic resonance imaging (MRI) are widely used in diagnostics, clinical studies, and treatment planning. Automatic algorithms for image analysis have thus become an invaluable tool in medicine. Examples of this are two- and three-dimensional visualizations, image segmentation, and the registration of all anatomical structure and pathology types. In this context, we introduce Studierfenster ( www.studierfenster.at ): a free, non-commercial open science client-server framework for (bio-)medical image analysis. Studierfenster offers a wide range of capabilities, including the visualization of medical data (CT, MRI, etc.) in two-dimensional (2D) and three-dimensional (3D) space in common web browsers, such as Google Chrome, Mozilla Firefox, Safari, or Microsoft Edge. Other functionalities are the calculation of medical metrics (dice score and Hausdorff distance), manual slice-by-slice outlining of structures in medical images, manual placing of (anatomical) landmarks in medical imaging data, visualization of medical data in virtual reality (VR), and a facial reconstruction and registration of medical data for augmented reality (AR). More sophisticated features include the automatic cranial implant design with a convolutional neural network (CNN), the inpainting of aortic dissections with a generative adversarial network, and a CNN for automatic aortic landmark detection in CT angiography images. A user study with medical and non-medical experts in medical image analysis was performed, to evaluate the usability and the manual functionalities of Studierfenster. When participants were asked about their overall impression of Studierfenster in an ISO standard (ISO-Norm) questionnaire, a mean of 6.3 out of 7.0 possible points were achieved. The evaluation also provided insights into the results achievable with Studierfenster in practice, by comparing these with two ground truth segmentations performed by a physician of the Medical University of Graz in Austria. In this contribution, we presented an online environment for (bio-)medical image analysis. In doing so, we established a client-server-based architecture, which is able to process medical data, especially 3D volumes. Our online environment is not limited to medical applications for humans. Rather, its underlying concept could be interesting for researchers from other fields, in applying the already existing functionalities or future additional implementations of further image processing applications. An example could be the processing of medical acquisitions like CT or MRI from animals [Clinical Pharmacology & Therapeutics, 84(4):448-456, 68], which get more and more common, as veterinary clinics and centers get more and more equipped with such imaging devices. Furthermore, applications in entirely non-medical research in which images/volumes need to be processed are also thinkable, such as those in optical measuring techniques, astronomy, or archaeology.

MUG500+: Database of 500 high-resolution healthy human skulls and 29 craniotomy skulls and implants

In this article, we present a skull database containing 500 healthy skulls segmented from high-resolution head computed-tomography (CT) scans and 29 defective skulls segmented from craniotomy head CTs. Each healthy skull contains the complete anatomical structures of human skulls, including the cranial bones, facial bones and other subtle structures. For each craniotomy skull, a part of the cranial bone is missing, leaving a defect on the skull. The defects have various sizes, shapes and positions, depending on the specific pathological conditions of each patient. Along with each craniotomy skull, a cranial implant, which is designed manually by an expert and can fit with the defect, is provided. Considering the large volume of the healthy skull collection, the dataset can be used to study the geometry/shape variabilities of human skulls and create a robust statistical model of the shape of human skulls, which can be used for various tasks such as cranial implant design. The craniotomy collection can serve as an evaluation set for automatic cranial implant design algorithms.

Deep learning-a first meta-survey of selected reviews across scientific disciplines, their commonalities, challenges and research impact

Deep learning belongs to the field of artificial intelligence, where machines perform tasks that typically require some kind of human intelligence. Deep learning tries to achieve this by drawing inspiration from the learning of a human brain. Similar to the basic structure of a brain, which consists of (billions of) neurons and connections between them, a deep learning algorithm consists of an artificial neural network, which resembles the biological brain structure. Mimicking the learning process of humans with their senses, deep learning networks are fed with (sensory) data, like texts, images, videos or sounds. These networks outperform the state-of-the-art methods in different tasks and, because of this, the whole field saw an exponential growth during the last years. This growth resulted in way over 10,000 publications per year in the last years. For example, the search engine PubMed alone, which covers only a sub-set of all publications in the medical field, provides already over 11,000 results in Q3 2020 for the search term 'deep learning', and around 90% of these results are from the last three years. Consequently, a complete overview over the field of deep learning is already impossible to obtain and, in the near future, it will potentially become difficult to obtain an overview over a subfield. However, there are several review articles about deep learning, which are focused on specific scientific fields or applications, for example deep learning advances in computer vision or in specific tasks like object detection. With these surveys as a foundation, the aim of this contribution is to provide a first high-level, categorized meta-survey of selected reviews on deep learning across different scientific disciplines and outline the research impact that they already have during a short period of time. The categories (computer vision, language processing, medical informatics and additional works) have been chosen according to the underlying data sources (image, language, medical, mixed). In addition, we review the common architectures, methods, pros, cons, evaluations, challenges and future directions for every sub-category.