Registration-Free Infant Cortical Surface Parcellation using Deep Convolutional Neural Networks

A joint parcellation and boundary network with multi-rate-shared dilated graph attention for cortical surface parcellation

Cortical surface parcellation aims to segment the surface into anatomically and functionally significant regions, which are crucial for diagnosing and treating numerous neurological diseases. However, existing methods generally ignore the difficulty in learning labeling patterns of boundaries, hindering the performance of parcellation. To this end, this paper proposes a joint parcellation and boundary network (JPBNet) to promote the effectiveness of cortical surface parcellation. Its core is developing a multi-rate-shared dilated graph attention (MDGA) module and incorporating boundary learning into the parcellation process. The former, in particular, constructs a dilated graph attention strategy, extending the dilated convolution from regular data to irregular graph data. We fuse it with different dilated rates to extract context information in various scales by devising a shared graph attention layer. After that, a boundary enhancement module and a parcellation enhancement module based on graph attention mechanisms are built in each layer, forcing MDGA to capture informative and valuable features for boundary detection and parcellation tasks. Integrating MDGA, the boundary enhancement module, and the parcellation enhancement module at each layer to supervise boundary and parcellation information, an effective JPBNet is formed by stacking multiple layers. Experiments on the public dataset reveal that the proposed method outperforms comparison methods and performs well on boundaries for cortical surface parcellation.

SPHERICAL TRANSFORMER FOR QUALITY ASSESSMENT OF PEDIATRIC CORTICAL SURFACES

Brain cortical surfaces, which have an intrinsic spherical topology, are typically represented by triangular meshes and mapped onto a spherical manifold in neuroimaging analysis. Inspired by the strong capability of feature learning in Convolutional Neural Networks (CNNs), spherical CNNs have been developed accordingly and achieved many successes in cortical surface analysis. Motivated by the recent success of the transformer, in this paper, for the first of time, we extend the transformer into the spherical space and propose the spherical transformer, which can better learn contextual and structural features than spherical CNNs. We applied the spherical transformer in the important task of automatic quality assessment of infant cortical surfaces, which is a necessary procedure to identify problematic cases due to extremely low tissue contrast and strong motion effects in pediatric brain MRI studies. Experiments on 1,860 infant cortical surfaces validated its superior effectiveness and efficiency in comparison with spherical CNNs.

Research on the Characteristics of Food Impaction with Tight Proximal Contacts Based on Deep Learning

Objective

Based on deep learning, the characteristics of food impaction with tight proximal contacts were studied to guide the subsequent clinical treatment of occlusal adjustment. At the same time, digital model building, software measurement, and statistical correlation analysis were used to explore the cause of tooth impaction and to provide evidence for clinical treatment.

Methods

Volunteers with (n = 250) and without (n = 250) tooth impaction were recruited, respectively, to conduct a questionnaire survey. Meanwhile, models were made and perfused by skilled clinical physicians for these patients, and characteristics such as adjacent line length, adjacent surface area, tongue abduction gap angle, buccal abduction gap angle, and occlusal abduction gap angle were measured. A normality test, differential analysis, correlation analysis of pathological characteristics of the impaction group, principal component analysis (PCA), and binary logistic regression analysis were performed.

Results

The adjacent line length, adjacent surface area, tongue abduction gap angle, buccal abduction gap angle, and occlusal abduction gap angle all met normal distribution. There were statistically significant differences in adjacent line length (p < 0.001), adjacent surface area (p < 0.001), and occlusal abduction gap angle (p < 0.001) between the two groups. After dimensionality reduction by PCA on characteristics, adjacent line length, adjacent surface area, buccal abduction gap angle, and occlusal abduction gap angle had a strong correlation with the principal components. Binary logistic regression analysis showed that adjacent line length and adjacent surface area had positive effects on impaction. The buccal abduction gap angle and occlusal abduction gap angle had a significant negative influence on impaction.

Conclusion

Adjacent line length, adjacent surface area, buccal abduction gap angle, and occlusal abduction gap angle are independent factors influencing food impaction.

A Deep Network for Joint Registration and Parcellation of Cortical Surfaces

Cortical surface registration and parcellation are two essential steps in neuroimaging analysis. Conventionally, they are performed independently as two tasks, ignoring the inherent connections of these two closely-related tasks. Essentially, both tasks rely on meaningful cortical feature representations, so they can be jointly optimized by learning shared useful cortical features. To this end, we propose a deep learning framework for joint cortical surface registration and parcellation. Specifically, our approach leverages the spherical topology of cortical surfaces and uses a spherical network as the shared encoder to first learn shared features for both tasks. Then we train two task-specific decoders for registration and parcellation, respectively. We further exploit the more explicit connection between them by incorporating the novel parcellation map similarity loss to enforce the boundary consistency of regions, thereby providing extra supervision for the registration task. Conversely, parcellation network training also benefits from the registration, which provides a large amount of augmented data by warping one surface with manual parcellation map to another surface, especially when only few manually-labeled surfaces are available. Experiments on a dataset with more than 600 cortical surfaces show that our approach achieves large improvements on both parcellation and registration accuracy (over separately trained networks) and enables training high-quality parcellation and registration models using much fewer labeled data.

S3Reg: Superfast Spherical Surface Registration Based on Deep Learning

Cortical surface registration is an essential step and prerequisite for surface-based neuroimaging analysis. It aligns cortical surfaces across individuals and time points to establish cross-sectional and longitudinal cortical correspondences to facilitate neuroimaging studies. Though achieving good performance, available methods are either time consuming or not flexible to extend to multiple or high dimensional features. Considering the explosive availability of large-scale and multimodal brain MRI data, fast surface registration methods that can flexibly handle multimodal features are desired. In this study, we develop a Superfast Spherical Surface Registration (S3Reg) framework for the cerebral cortex. Leveraging an end-to-end unsupervised learning strategy, S3Reg offers great flexibility in the choice of input feature sets and output similarity measures for registration, and meanwhile reduces the registration time significantly. Specifically, we exploit the powerful learning capability of spherical Convolutional Neural Network (CNN) to directly learn the deformation fields in spherical space and implement diffeomorphic design with "scaling and squaring" layers to guarantee topology-preserving deformations. To handle the polar-distortion issue, we construct a novel spherical CNN model using three orthogonal Spherical U-Nets. Experiments are performed on two different datasets to align both adult and infant multimodal cortical features. Results demonstrate that our S3Reg shows superior or comparable performance with state-of-the-art methods, while improving the registration time from 1 min to 10 sec.

Spherical Deformable U-Net: Application to Cortical Surface Parcellation and Development Prediction

Convolutional Neural Networks (CNNs) have achieved overwhelming success in learning-related problems for 2D/3D images in the Euclidean space. However, unlike in the Euclidean space, the shapes of many structures in medical imaging have an inherent spherical topology in a manifold space, e.g., the convoluted brain cortical surfaces represented by triangular meshes. There is no consistent neighborhood definition and thus no straightforward convolution/pooling operations for such cortical surface data. In this paper, leveraging the regular and hierarchical geometric structure of the resampled spherical cortical surfaces, we create the 1-ring filter on spherical cortical triangular meshes and accordingly develop convolution/pooling operations for constructing Spherical U-Net for cortical surface data. However, the regular nature of the 1-ring filter makes it inherently limited to model fixed geometric transformations. To further enhance the transformation modeling capability of Spherical U-Net, we introduce the deformable convolution and deformable pooling to cortical surface data and accordingly propose the Spherical Deformable U-Net (SDU-Net). Specifically, spherical offsets are learned to freely deform the 1-ring filter on the sphere to adaptively localize cortical structures with different sizes and shapes. We then apply the SDU-Net to two challenging and scientifically important tasks in neuroimaging: cortical surface parcellation and cortical attribute map prediction. Both applications validate the competitive performance of our approach in accuracy and computational efficiency in comparison with state-of-the-art methods.

Computer-Aided Cervical Cancer Diagnosis Using Time-Lapsed Colposcopic Images

Cervical cancer causes the fourth most cancer-related deaths of women worldwide. Early detection of cervical intraepithelial neoplasia (CIN) can significantly increase the survival rate of patients. In this paper, we propose a deep learning framework for the accurate identification of LSIL+ (including CIN and cervical cancer) using time-lapsed colposcopic images. The proposed framework involves two main components, i.e., key-frame feature encoding networks and feature fusion network. The features of the original (pre-acetic-acid) image and the colposcopic images captured at around 60s, 90s, 120s and 150s during the acetic acid test are encoded by the feature encoding networks. Several fusion approaches are compared, all of which outperform the existing automated cervical cancer diagnosis systems using a single time slot. A graph convolutional network with edge features (E-GCN) is found to be the most suitable fusion approach in our study, due to its excellent explainability consistent with the clinical practice. A large-scale dataset, containing time-lapsed colposcopic images from 7,668 patients, is collected from the collaborative hospital to train and validate our deep learning framework. Colposcopists are invited to compete with our computer-aided diagnosis system. The proposed deep learning framework achieves a classification accuracy of 78.33%-comparable to that of an in-service colposcopist-which demonstrates its potential to provide assistance in the realistic clinical scenario.

Deep Multi-Scale Mesh Feature Learning for Automated Labeling of Raw Dental Surfaces From 3D Intraoral Scanners

Precisely labeling teeth on digitalized 3D dental surface models is the precondition for tooth position rearrangements in orthodontic treatment planning. However, it is a challenging task primarily due to the abnormal and varying appearance of patients' teeth. The emerging utilization of intraoral scanners (IOSs) in clinics further increases the difficulty in automated tooth labeling, as the raw surfaces acquired by IOS are typically low-quality at gingival and deep intraoral regions. In recent years, some pioneering end-to-end methods (e.g., PointNet) have been proposed in the communities of computer vision and graphics to consume directly raw surface for 3D shape segmentation. Although these methods are potentially applicable to our task, most of them fail to capture fine-grained local geometric context that is critical to the identification of small teeth with varying shapes and appearances. In this paper, we propose an end-to-end deep-learning method, called MeshSegNet, for automated tooth labeling on raw dental surfaces. Using multiple raw surface attributes as inputs, MeshSegNet integrates a series of graph-constrained learning modules along its forward path to hierarchically extract multi-scale local contextual features. Then, a dense fusion strategy is applied to combine local-to-global geometric features for the learning of higher-level features for mesh cell annotation. The predictions produced by our MeshSegNet are further post-processed by a graph-cut refinement step for final segmentation. We evaluated MeshSegNet using a real-patient dataset consisting of raw maxillary surfaces acquired by 3D IOS. Experimental results, performed 5-fold cross-validation, demonstrate that MeshSegNet significantly outperforms state-of-the-art deep learning methods for 3D shape segmentation.

Geometric Brain Surface Network For Brain Cortical Parcellation

A large number of surface-based analyses on brain imaging data adopt some specific brain atlases to better assess structural and functional changes in one or more brain regions. In these analyses, it is necessary to obtain an anatomically correct surface parcellation scheme in an individual brain by referring to the given atlas. Traditional ways to accomplish this goal are through a designed surface-based registration or hand-crafted surface features, although both of them are time-consuming. A recent deep learning approach depends on a regular spherical parameterization of the mesh, which is computationally prohibitive in some cases and may also demand further post-processing to refine the network output. Therefore, an accurate and fully-automatic cortical surface parcellation scheme directly working on the original brain surfaces would be highly advantageous. In this study, we propose an end-to-end deep brain cortical parcellation network, called DBPN. Through intrinsic and extrinsic graph convolution kernels, DBPN dynamically deciphers neighborhood graph topology around each vertex and encodes the deciphered knowledge into node features. Eventually, a non-linear mapping between the node features and parcellation labels is constructed. Our model is a two-stage deep network which contains a coarse parcellation network with a U-shape structure and a refinement network to fine-tune the coarse results. We evaluate our model in a large public dataset and our work achieves superior performance than state-of-the-art baseline methods in both accuracy and efficiency.

Intrinsic Patch-Based Cortical Anatomical Parcellation Using Graph Convolutional Neural Network on Surface Manifold

Automatic parcellation of cortical surfaces into anatomically meaningful regions of interest (ROIs) is of great importance in brain analysis. Due to the complex shape of the convoluted cerebral cortex, conventional methods generally require three steps to obtain the parcellations. First, the original cortical surface is iteratively inflated and mapped onto a spherical surface with minimal metric distortion, for providing a simpler shape for analysis. Then, a registration or learning-based labeling method is adopted to parcellate ROIs on the mapped spherical surface. Finally, parcellation labels on the spherical surface are mapped back to the original cortical surface. Despite great success, spherical mapping of the original cortical surface is inherently sensitive to topological noise and cannot deal with the impaired brains that violate spherical topology. To address these issues, in this paper, we propose to directly parcellate the cerebral cortex on the original cortical surface manifold without requiring spherical mapping, by leveraging the strong learning ability of the graph convolutional neural networks. Also, we extend the convolution to the surface manifold using the kernel strategy, which enables us to over-come the notorious shape difference issue (e.g., different vertex number and connections) across different subjects. Our method aims to learn the highly nonlinear mapping between cortical attribute patterns (on local intrinsic surface patches) and parcellation labels. We have validated our method on a normal cortical surface dataset and a synthetic dataset with impaired brains, which shows that our method achieves comparable accuracy to the methods using spherical mapping, and works well on cortical surfaces violating the spherical topology.

Cortical Surface Parcellation using Spherical Convolutional Neural Networks

We present cortical surface parcellation using spherical deep convolutional neural networks. Traditional multi-atlas cortical surface parcellation requires inter-subject surface registration using geometric features with slow processing speed on a single subject (2-3 hours). Moreover, even optimal surface registration does not necessarily produce optimal cortical parcellation as parcel boundaries are not fully matched to the geometric features. In this context, a choice of training features is important for accurate cortical parcellation. To utilize the networks efficiently, we propose cortical parcellation-specific input data from an irregular and complicated structure of cortical surfaces. To this end, we align ground-truth cortical parcel boundaries and use their resulting deformation fields to generate new pairs of deformed geometric features and parcellation maps. To extend the capability of the networks, we then smoothly morph cortical geometric features and parcellation maps using the intermediate deformation fields. We validate our method on 427 adult brains for 49 labels. The experimental results show that our method outperforms traditional multi-atlas and naive spherical U-Net approaches, while achieving full cortical parcellation in less than a minute.

Spherical U-Net on Cortical Surfaces: Methods and Applications

Convolutional Neural Networks (CNNs) have been providing the state-of-the-art performance for learning-related problems involving 2D/3D images in Euclidean space. However, unlike in the Euclidean space, the shapes of many structures in medical imaging have a spherical topology in a manifold space, e.g., brain cortical or subcortical surfaces represented by triangular meshes, with large inter-subject and intra-subject variations in vertex number and local connectivity. Hence, there is no consistent neighborhood definition and thus no straightforward convolution/transposed convolution operations for cortical/subcortical surface data. In this paper, by leveraging the regular and consistent geometric structure of the resampled cortical surface mapped onto the spherical space, we propose a novel convolution filter analogous to the standard convolution on the image grid. Accordingly, we develop corresponding operations for convolution, pooling, and transposed convolution for spherical surface data and thus construct spherical CNNs. Specifically, we propose the Spherical U-Net architecture by replacing all operations in the standard U-Net with their spherical operation counterparts. We then apply the Spherical U-Net to two challenging and neuroscientifically important tasks in infant brains: cortical surface parcellation and cortical attribute map development prediction. Both applications demonstrate the competitive performance in the accuracy, computational efficiency, and effectiveness of our proposed Spherical U-Net, in comparison with the state-of-the-art methods.

SPHERICAL U-NET FOR INFANT CORTICAL SURFACE PARCELLATION

In human brain MRI studies, it is of great importance to accurately parcellate cortical surfaces into anatomically and functionally meaningful regions. In this paper, we propose a novel end-to-end deep learning method by formulating surface parcellation as a semantic segmentation task on the sphere. To extend the convolutional neural networks (CNNs) to the spherical space, corresponding operations of surface convolution, pooling and upsampling are first developed to deal with data representation on spherical surface meshes, and then spherical CNNs are constructed accordingly. Specifically, the U-Net and SegNet architectures are transformed to the spherical representation for neonatal cortical surface parcellation. Experimental results on 90 neonates indicate the effectiveness and efficiency of our proposed spherical U-Net, in comparison with the spherical SegNet and the previous patch-wise classification method.