Automated Computer-aided Design of Cranial Implants Using a Deep Volumetric Convolutional Denoising Autoencoder

MedShapeNet - a large-scale dataset of 3D medical shapes for computer vision

OBJECTIVES

The shape is commonly used to describe the objects. State-of-the-art algorithms in medical imaging are predominantly diverging from computer vision, where voxel grids, meshes, point clouds, and implicit surface models are used. This is seen from the growing popularity of ShapeNet (51,300 models) and Princeton ModelNet (127,915 models). However, a large collection of anatomical shapes (e.g., bones, organs, vessels) and 3D models of surgical instruments is missing.

METHODS

We present MedShapeNet to translate data-driven vision algorithms to medical applications and to adapt state-of-the-art vision algorithms to medical problems. As a unique feature, we directly model the majority of shapes on the imaging data of real patients. We present use cases in classifying brain tumors, skull reconstructions, multi-class anatomy completion, education, and 3D printing.

RESULTS

By now, MedShapeNet includes 23 datasets with more than 100,000 shapes that are paired with annotations (ground truth). Our data is freely accessible via a web interface and a Python application programming interface and can be used for discriminative, reconstructive, and variational benchmarks as well as various applications in virtual, augmented, or mixed reality, and 3D printing.

CONCLUSIONS

MedShapeNet contains medical shapes from anatomy and surgical instruments and will continue to collect data for benchmarks and applications. The project page is: https://medshapenet.ikim.nrw/.

A Stage-Wise Residual Attention Generation Adversarial Network for Mandibular Defect Repairing and Reconstruction

Surgical reconstruction of mandibular defects is a clinical routine manner for the rehabilitation of patients with deformities. The mandible plays a crucial role in maintaining the facial contour and ensuring the speech and mastication functions. The repairing and reconstruction of mandible defects is a significant yet challenging task in oral-maxillofacial surgery. Currently, the mainly available methods are traditional digitalized design methods that suffer from substantial artificial operations, limited applicability and high reconstruction error rates. An automated, precise, and individualized method is imperative for maxillofacial surgeons. In this paper, we propose a Stage-wise Residual Attention Generative Adversarial Network (SRA-GAN) for mandibular defect reconstruction. Specifically, we design a stage-wise residual attention mechanism for generator to enhance the extraction capability of mandibular remote spatial information, making it adaptable to various defects. For the discriminator, we propose a multi-field perceptual network, consisting of two parallel discriminators with different perceptual fields, to reduce the cumulative reconstruction errors. Furthermore, we design a self-encoder perceptual loss function to ensure the correctness of mandibular anatomical structures. The experimental results on a novel custom-built mandibular defect dataset demonstrate that our method has a promising prospect in clinical application, achieving the best Dice Similarity Coefficient (DSC) of 94.238% and 95% Hausdorff Distance (HD95) of 4.787.

Adaptable cascaded registration for personalized maxilla completion and cleft defect volume estimation

BACKGROUND

Cone-beam computed tomography (CBCT) images provide high-resolution insights into the underlying craniofacial anomaly in patients with cleft lip and palate (CLP), requiring non-negligible annotation costs to measure the cleft defect for the guidance of the clinical secondary alveolar bone graft procedures. Considering the cumbersome volumetric image acquisition, there is a lack of paired CLP CBCTs and normal CBCTs for learning-based anatomical structure restoration models. Nowadays, the registration-based method relieves the annotation burden, though one-shot registration and the regular mask are limited to handling fine-grained shape variations and harmony between restored bony tissues and the defected maxilla.

PURPOSE

This study aimed to design and evaluate a novel method for deformable partial registration of the CLP CBCTs and normal CBCTs, enabling personalized maxilla completion and cleft defect volume prediction from CLP CBCTs.

METHODS

We proposed an adaptable deep registration framework for personalized maxilla completion and cleft defect volume prediction from CLP CBCTs. The key ingredient was a cascaded partial registration to exploit the maxillary morphology prior and attribute transfer. Cascaded registration with coarse-to-fine registration fields handled morphological variations of cleft defects and fine-grained maxillary restoration. We designed an adaptable cleft defect mask and volumetric Boolean operators for reliable voxel filling of the defected maxilla. A total of 36 clinically obtained CLP CBCTs were used to train and validate the proposed model, among which 22 CLP CBCTs were used to generate a training dataset with 440 synthetic CBCTs by B-spline deformation-based data augmentation and the remaining for testing. The proposed model was evaluated on maxilla completion and cleft defect volume prediction from clinically obtained unilateral and bilateral CLP CBCTs.

RESULTS

Extensive experiments demonstrated the effectiveness of the adaptable cleft defect mask and the cascaded partial registration on maxilla completion and cleft defect volume prediction. The proposed method achieved state-of-the-art performances with the Dice similarity coefficient of 0.90 ± $\pm$ 0.02 on the restored maxilla and 0.84 ± $\pm$ 0.04 on the estimated cleft defect, respectively. The average Hausdorff distance between the estimated cleft defect and the manually annotated ground truth was 0.30 ± $\pm$ 0.08 mm. The relative volume error of the cleft defect was0.09 ± $0.09\pm$ 0.08. The proposed model allowed for the prediction of cleft defect maps that were in line with the ground truth in the challenging unilateral and bilateral CLP CBCTs.

CONCLUSIONS

The results suggest that the proposed adaptable deep registration model enables patient-specific maxilla completion and automatic annotation of cleft defects, relieving tedious voxel-wise annotation and image acquisition burdens.

Back to the Roots: Reconstructing Large and Complex Cranial Defects using an Image-based Statistical Shape Model

Designing implants for large and complex cranial defects is a challenging task, even for professional designers. Current efforts on automating the design process focused mainly on convolutional neural networks (CNN), which have produced state-of-the-art results on reconstructing synthetic defects. However, existing CNN-based methods have been difficult to translate to clinical practice in cranioplasty, as their performance on large and complex cranial defects remains unsatisfactory. In this paper, we present a statistical shape model (SSM) built directly on the segmentation masks of the skulls represented as binary voxel occupancy grids and evaluate it on several cranial implant design datasets. Results show that, while CNN-based approaches outperform the SSM on synthetic defects, they are inferior to SSM when it comes to large, complex and real-world defects. Experienced neurosurgeons evaluate the implants generated by the SSM to be feasible for clinical use after minor manual corrections. Datasets and the SSM model are publicly available at https://github.com/Jianningli/ssm .

Deep Learning-Based Computer-Aided Diagnosis (CAD): Applications for Medical Image Datasets

Computer-aided diagnosis (CAD) has proved to be an effective and accurate method for diagnostic prediction over the years. This article focuses on the development of an automated CAD system with the intent to perform diagnosis as accurately as possible. Deep learning methods have been able to produce impressive results on medical image datasets. This study employs deep learning methods in conjunction with meta-heuristic algorithms and supervised machine-learning algorithms to perform an accurate diagnosis. Pre-trained convolutional neural networks (CNNs) or auto-encoder are used for feature extraction, whereas feature selection is performed using an ant colony optimization (ACO) algorithm. Ant colony optimization helps to search for the best optimal features while reducing the amount of data. Lastly, diagnosis prediction (classification) is achieved using learnable classifiers. The novel framework for the extraction and selection of features is based on deep learning, auto-encoder, and ACO. The performance of the proposed approach is evaluated using two medical image datasets: chest X-ray (CXR) and magnetic resonance imaging (MRI) for the prediction of the existence of COVID-19 and brain tumors. Accuracy is used as the main measure to compare the performance of the proposed approach with existing state-of-the-art methods. The proposed system achieves an average accuracy of 99.61% and 99.18%, outperforming all other methods in diagnosing the presence of COVID-19 and brain tumors, respectively. Based on the achieved results, it can be claimed that physicians or radiologists can confidently utilize the proposed approach for diagnosing COVID-19 patients and patients with specific brain tumors.

A review on the application of deep learning for CT reconstruction, bone segmentation and surgical planning in oral and maxillofacial surgery

Computer-assisted surgery (CAS) allows clinicians to personalize treatments and surgical interventions and has therefore become an increasingly popular treatment modality in maxillofacial surgery. The current maxillofacial CAS consists of three main steps: (1) CT image reconstruction, (2) bone segmentation, and (3) surgical planning. However, each of these three steps can introduce errors that can heavily affect the treatment outcome. As a consequence, tedious and time-consuming manual post-processing is often necessary to ensure that each step is performed adequately. One way to overcome this issue is by developing and implementing neural networks (NNs) within the maxillofacial CAS workflow. These learning algorithms can be trained to perform specific tasks without the need for explicitly defined rules. In recent years, an extremely large number of novel NN approaches have been proposed for a wide variety of applications, which makes it a difficult task to keep up with all relevant developments. This study therefore aimed to summarize and review all relevant NN approaches applied for CT image reconstruction, bone segmentation, and surgical planning. After full text screening, 76 publications were identified: 32 focusing on CT image reconstruction, 33 focusing on bone segmentation and 11 focusing on surgical planning. Generally, convolutional NNs were most widely used in the identified studies, although the multilayer perceptron was most commonly applied in surgical planning tasks. Moreover, the drawbacks of current approaches and promising research avenues are discussed.

CranGAN: Adversarial Point Cloud Reconstruction for patient-specific Cranial Implant Design

Automatizing cranial implant design has become an increasingly important avenue in biomedical research. Benefits in terms of financial resources, time and patient safety necessitate the formulation of an efficient and accurate procedure for the same. This paper attempts to provide a new research direction to this problem, through an adversarial deep learning solution. Specifically, in this work, we present CranGAN - a 3D Conditional Generative Adversarial Network designed to reconstruct a 3D representation of a complete skull given its defective counterpart. A novel solution of employing point cloud representations instead of conventional 3D meshes and voxel grids is proposed. We provide both qualitative and quantitative analysis of our experiments with three separate GAN objectives, and compare the utility of two 3D reconstruction loss functions viz. Hausdorff Distance and Chamfer Distance. We hope that our work inspires further research in this direction. Clinical relevance- This paper establishes a new research direction to assist in automated implant design for cranioplasty.

Studierfenster: an Open Science Cloud-Based Medical Imaging Analysis Platform

Imaging modalities such as computed tomography (CT) and magnetic resonance imaging (MRI) are widely used in diagnostics, clinical studies, and treatment planning. Automatic algorithms for image analysis have thus become an invaluable tool in medicine. Examples of this are two- and three-dimensional visualizations, image segmentation, and the registration of all anatomical structure and pathology types. In this context, we introduce Studierfenster ( www.studierfenster.at ): a free, non-commercial open science client-server framework for (bio-)medical image analysis. Studierfenster offers a wide range of capabilities, including the visualization of medical data (CT, MRI, etc.) in two-dimensional (2D) and three-dimensional (3D) space in common web browsers, such as Google Chrome, Mozilla Firefox, Safari, or Microsoft Edge. Other functionalities are the calculation of medical metrics (dice score and Hausdorff distance), manual slice-by-slice outlining of structures in medical images, manual placing of (anatomical) landmarks in medical imaging data, visualization of medical data in virtual reality (VR), and a facial reconstruction and registration of medical data for augmented reality (AR). More sophisticated features include the automatic cranial implant design with a convolutional neural network (CNN), the inpainting of aortic dissections with a generative adversarial network, and a CNN for automatic aortic landmark detection in CT angiography images. A user study with medical and non-medical experts in medical image analysis was performed, to evaluate the usability and the manual functionalities of Studierfenster. When participants were asked about their overall impression of Studierfenster in an ISO standard (ISO-Norm) questionnaire, a mean of 6.3 out of 7.0 possible points were achieved. The evaluation also provided insights into the results achievable with Studierfenster in practice, by comparing these with two ground truth segmentations performed by a physician of the Medical University of Graz in Austria. In this contribution, we presented an online environment for (bio-)medical image analysis. In doing so, we established a client-server-based architecture, which is able to process medical data, especially 3D volumes. Our online environment is not limited to medical applications for humans. Rather, its underlying concept could be interesting for researchers from other fields, in applying the already existing functionalities or future additional implementations of further image processing applications. An example could be the processing of medical acquisitions like CT or MRI from animals [Clinical Pharmacology & Therapeutics, 84(4):448-456, 68], which get more and more common, as veterinary clinics and centers get more and more equipped with such imaging devices. Furthermore, applications in entirely non-medical research in which images/volumes need to be processed are also thinkable, such as those in optical measuring techniques, astronomy, or archaeology.

Three-dimensional deep learning to automatically generate cranial implant geometry

We present a 3D deep learning framework that can generate a complete cranial model using a defective one. The Boolean subtraction between these two models generates the geometry of the implant required for surgical reconstruction. There is little or no need for post-processing to eliminate noise in the implant model generated by the proposed approach. The framework can be used to meet the repair needs of cranial imperfections caused by trauma, congenital defects, plastic surgery, or tumor resection. Traditional implant design methods for skull reconstruction rely on the mirror operation. However, these approaches have great limitations when the defect crosses the plane of symmetry or the patient's skull is asymmetrical. The proposed deep learning framework is based on an enhanced three-dimensional autoencoder. Each training sample for the framework is a pair consisting of a cranial model converted from CT images and a corresponding model with simulated defects on it. Our approach can learn the spatial distribution of the upper part of normal cranial bones and use flawed cranial data to predict its complete geometry. Empirical research on simulated defects and actual clinical applications shows that our framework can meet most of the requirements of cranioplasty.

AutoImplant 2020-First MICCAI Challenge on Automatic Cranial Implant Design

The aim of this paper is to provide a comprehensive overview of the MICCAI 2020 AutoImplant Challenge. The approaches and publications submitted and accepted within the challenge will be summarized and reported, highlighting common algorithmic trends and algorithmic diversity. Furthermore, the evaluation results will be presented, compared and discussed in regard to the challenge aim: seeking for low cost, fast and fully automated solutions for cranial implant design. Based on feedback from collaborating neurosurgeons, this paper concludes by stating open issues and post-challenge requirements for intra-operative use. The codes can be found at https://github.com/Jianningli/tmi.

Deep learning for cranioplasty in clinical practice: Going from synthetic to real patient data

Correct virtual reconstruction of a defective skull is a prerequisite for successful cranioplasty and its automatization has the potential for accelerating and standardizing the clinical workflow. This work provides a deep learning-based method for the reconstruction of a skull shape and cranial implant design on clinical data of patients indicated for cranioplasty. The method is based on a cascade of multi-branch volumetric CNNs that enables simultaneous training on two different types of cranioplasty ground-truth data: the skull patch, which represents the exact shape of the missing part of the original skull, and which can be easily created artificially from healthy skulls, and expert-designed cranial implant shapes that are much harder to acquire. The proposed method reaches an average surface distance of the reconstructed skull patches of 0.67 mm on a clinical test set of 75 defective skulls. It also achieves a 12% reduction of a newly proposed defect border Gaussian curvature error metric, compared to a baseline model trained on synthetic data only. Additionally, it produces directly 3D printable cranial implant shapes with a Dice coefficient 0.88 and a surface error of 0.65 mm. The outputs of the proposed skull reconstruction method reach good quality and can be considered for use in semi- or fully automatic clinical cranial implant design workflows.

Automatic skull defect restoration and cranial implant generation for cranioplasty

A fast and fully automatic design of 3D printed patient-specific cranial implants is highly desired in cranioplasty - the process to restore a defect on the skull. We formulate skull defect restoration as a 3D volumetric shape completion task, where a partial skull volume is completed automatically. The difference between the completed skull and the partial skull is the restored defect; in other words, the implant that can be used in cranioplasty. To fulfill the task of volumetric shape completion, a fully data-driven approach is proposed. Supervised skull shape learning is performed on a database containing 167 high-resolution healthy skulls. In these skulls, synthetic defects are injected to create training and evaluation data pairs. We propose a patch-based training scheme tailored for dealing with high-resolution and spatially sparse data, which overcomes the disadvantages of conventional patch-based training methods in high-resolution volumetric shape completion tasks. In particular, the conventional patch-based training is applied to images of high resolution and proves to be effective in tasks such as segmentation. However, we demonstrate the limitations of conventional patch-based training for shape completion tasks, where the overall shape distribution of the target has to be learnt, since it cannot be captured efficiently by a sub-volume cropped from the target. Additionally, the standard dense implementation of a convolutional neural network tends to perform poorly on sparse data, such as the skull, which has a low voxel occupancy rate. Our proposed training scheme encourages a convolutional neural network to learn from the high-resolution and spatially sparse data. In our study, we show that our deep learning models, trained on healthy skulls with synthetic defects, can be transferred directly to craniotomy skulls with real defects of greater irregularity, and the results show promise for clinical use. Project page: https://github.com/Jianningli/MIA.

Synthetic skull bone defects for automatic patient-specific craniofacial implant design

Patient-specific craniofacial implants are used to repair skull bone defects after trauma or surgery. Currently, cranial implants are designed and produced by third-party suppliers, which is usually time-consuming and expensive. Recent advances in additive manufacturing made the in-hospital or in-operation-room fabrication of personalized implants feasible. However, the implants are still manufactured by external companies. To facilitate an optimized workflow, fast and automatic implant manufacturing is highly desirable. Data-driven approaches, such as deep learning, show currently great potential towards automatic implant design. However, a considerable amount of data is needed to train such algorithms, which is, especially in the medical domain, often a bottleneck. Therefore, we present CT-imaging data of the craniofacial complex from 24 patients, in which we injected various artificial cranial defects, resulting in 240 data pairs and 240 corresponding implants. Based on this work, automatic implant design and manufacturing processes can be trained. Additionally, the data of this work build a solid base for researchers to work on automatic cranial implant designs.

Measurement(s)	Image Acquisition Matrix Size • Image Slice Thickness • craniofacial region
Technology Type(s)	imaging technique • computed tomography
Sample Characteristic - Organism	Homo sapiens

Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.13265225

Skull shape reconstruction using cascaded convolutional networks

Designing a cranial implant to restore the protective and aesthetic function of the patient's skull is a challenging process that requires a substantial amount of manual work, even for an experienced clinician. While computer-assisted approaches with various levels of required user interaction exist to aid this process, they are usually only validated on either a single type of simple synthetic defect or a very limited sample of real defects. The work presented in this paper aims to address two challenges: (i) design a fully automatic 3D shape reconstruction method that can address diverse shapes of real skull defects in various stages of healing and (ii) to provide an open dataset for optimization and validation of anatomical reconstruction methods on a set of synthetically broken skull shapes. We propose an application of the multi-scale cascade architecture of convolutional neural networks to the reconstruction task. Such an architecture is able to tackle the issue of trade-off between the output resolution and the receptive field of the model imposed by GPU memory limitations. Furthermore, we experiment with both generative and discriminative models and study their behavior during the task of anatomical reconstruction. The proposed method achieves an average surface error of 0.59mm for our synthetic test dataset with as low as 0.48mm for unilateral defects of parietal and temporal bone, matching state-of-the-art performance while being completely automatic. We also show that the model trained on our synthetic dataset is able to reconstruct real patient defects.

A review on computer-aided design and manufacturing of patient-specific maxillofacial implants

Introduction: Various prefabricated maxillofacial implants are used in the clinical routine for the surgical treatment of patients. In addition to these prefabricated implants, customized CAD/CAM implants become increasingly important for a more precise replacement of damaged anatomical structures. This paper reviews the design and manufacturing of patient-specific implants for the maxillofacial area.Areas covered: The contribution of this publication is to give a state-of-the-art overview in the usage of customized facial implants. Moreover, it provides future perspectives, including 3D printing technologies, for the manufacturing of patient-individual facial implants that are based on patient's data acquisitions, like Computed Tomography (CT) or Magnetic Resonance Imaging (MRI).Expert opinion: The main target of this review is to present various designing software and 3D manufacturing technologies that have been applied to fabricate facial implants. In doing so, different CAD designing software's are discussed, which are based on various methods and have been implemented and evaluated by researchers. Finally, recent 3D printing technologies that have been applied to manufacture patient-individual implants will be introduced and discussed.

A Client/Server based Online Environment for the Calculation of Medical Segmentation Scores

Image segmentation plays a major role in medical imaging. Especially in radiology, the detection and development of tumors and other diseases can be supported by image segmentation applications. Tools that provide image segmentation and calculation of segmentation scores are not available at any time for every device due to the size and scope of functionalities they offer. These tools need huge periodic updates and do not properly work on old or weak systems. However, medical use-cases often require fast and accurate results. A complex and slow software can lead to additional stress and thus unnecessary errors. The aim of this contribution is the development of a cross-platform tool for medical segmentation use-cases. The goal is a device-independent and always available possibility for medical imaging including manual segmentation and metric calculation. The result is Studierfenster (studierfenster.at), a web-tool for manual segmentation and segmentation metric calculation. In this contribution, the focus lies on the segmentation metric calculation part of the tool. It provides the functionalities of calculating directed and undirected Hausdorff Distance (HD) and Dice Similarity Coefficient (DSC) scores for two uploaded volumes, filtering for specific values, searching for specific values in the calculated metrics and exporting filtered metric lists in different file formats.