Feature-based multi-resolution registration of immunostained serial sections

Towards Realistic 3D Models of Tumor Vascular Networks

For reliable in silico or in vitro investigations in, for example, biosensing and drug delivery applications, accurate models of tumor vascular networks down to the capillary size are essential. Compared to images acquired with conventional medical imaging techniques, digitalized histological tumor slices have a higher resolution, enabling the delineation of capillaries. Volume rendering procedures can then be used to generate a 3D model. However, the preparation of such slices leads to misalignments in relative slice orientation between consecutive slices. Thus, image registration algorithms are necessary to re-align the slices. Here, we present an algorithm for the registration and reconstruction of a vascular network from histologic slices applied to 169 tumor slices. The registration includes two steps. First, consecutive images are incrementally pre-aligned using feature- and area-based transformations. Second, using the previous transformations, parallel registration for all images is enabled. Combining intensity- and color-based thresholds along with heuristic analysis, vascular structures are segmented. A 3D interpolation technique is used for volume rendering. This results in a 3D vascular network with approximately 400-450 vessels with diameters down to 25-30 µm. A delineation of vessel structures with close distance was limited in areas of high structural density. Improvement can be achieved by using images with higher resolution and or machine learning techniques.

Vessel Delineation Using U-Net: A Sparse Labeled Deep Learning Approach for Semantic Segmentation of Histological Images

Semantic segmentation is an important imaging analysis method enabling the identification of tissue structures. Histological image segmentation is particularly challenging, having large structural information while providing only limited training data. Additionally, labeling these structures to generate training data is time consuming. Here, we demonstrate the feasibility of a semantic segmentation using U-Net with a novel sparse labeling technique. The basic U-Net architecture was extended by attention gates, residual and recurrent links, and dropout regularization. To overcome the high class imbalance, which is intrinsic to histological data, under- and oversampling and data augmentation were used. In an ablation study, various architectures were evaluated, and the best performing model was identified. This model contains attention gates, residual links, and a dropout regularization of 0.125. The segmented images show accurate delineations of the vascular structures (with a precision of 0.9088 and an AUC-ROC score of 0.9717), and the segmentation algorithm is robust to images containing staining variations and damaged tissue. These results demonstrate the feasibility of sparse labeling in combination with the modified U-Net architecture.

The human splenic microcirculation is entirely open as shown by 3D models in virtual reality

The human spleen is equipped with an organ-specific microcirculation. The initial part of the venous circulation is formed by spleen-specific large microvessels, the sinuses. Sinuses eventually fuse to form venules and veins. For more than 170 years there have been debates, whether splenic red pulp capillaries join sinuses, i.e., whether the microcirculation is closed or open-or even simultaneously closed and open. We have now solved this question by three-dimensional reconstruction of a limited number of immunostained serial sections of red and white pulp areas, which were visualized in virtual reality. Splenic capillaries have special end structures exhibiting multiple small diverging endothelial cell processes, which always keep a certain distance to the walls of sinuses. Only very few capillary ends were difficult to diagnose. Positive identification of these end structures permits to conclude that the human splenic microcirculation is entirely open. This is also true for the perifollicular capillary network and for capillaries close to red pulp venules. Follicles are supplied by a relatively dense open perifollicular capillary net, which is primarily, but not exclusively, fed by sheathed and few non-sheathed capillaries from the surrounding red pulp network.

Unsupervised Histological Image Registration Using Structural Feature Guided Convolutional Neural Network

Registration of multiple stained images is a fundamental task in histological image analysis. In supervised methods, obtaining ground-truth data with known correspondences is laborious and time-consuming. Thus, unsupervised methods are expected. Unsupervised methods ease the burden of manual annotation but often at the cost of inferior results. In addition, registration of histological images suffers from appearance variance due to multiple staining, repetitive texture, and section missing during making tissue sections. To deal with these challenges, we propose an unsupervised structural feature guided convolutional neural network (SFG). Structural features are robust to multiple staining. The combination of low-resolution rough structural features and high-resolution fine structural features can overcome repetitive texture and section missing, respectively. SFG consists of two components of structural consistency constraints according to the formations of structural features, i.e., dense structural component and sparse structural component. The dense structural component uses structural feature maps of the whole image as structural consistency constraints, which represent local contextual information. The sparse structural component utilizes the distance of automatically obtained matched key points as structural consistency constraints because the matched key points in an image pair emphasize the matching of significant structures, which imply global information. In addition, a multi-scale strategy is used in both dense and sparse structural components to make full use of the structural information at low resolution and high resolution to overcome repetitive texture and section missing. The proposed method was evaluated on a public histological dataset (ANHIR) and ranked first as of Jan 18th, 2022.

Research on Panoramic Stitching Algorithm of Lateral Cranial Sequence Images in Dental Multifunctional Cone Beam Computed Tomography

In the design of dental multifunctional Cone Beam Computed Tomography, the linear scanning strategy not only saves equipment cost, but also avoids the demand for patients to be repositioned when acquiring lateral cranial sequence images. In order to obtain panoramic images, we propose a local normalized cross-correlation stitching algorithm based on Gaussian Mixture Model. Firstly, the Block-Matching and 3D filtering algorithm is used to remove quantum and impulse noises according to the characteristics of X-ray images; Then, the segmentation of the irrelevant region and the extraction of the region of interest are performed by Gaussian Mixture Model; The locally normalized cross-relation is used to complete the registration with the multi-resolution strategy based on wavelet transform and Particle Swarm Optimization algorithm; Finally, image fusion is achieved by the weighted smoothing fusion algorithm. The experimental results show that the panoramic image obtained by this method has significant performance in both subjective vision and objective quality evaluation and can be applied to preoperative diagnosis of clinical dental deformity and postoperative effect evaluation.

Exploring human splenic red pulp vasculature in virtual reality: details of sheathed capillaries and the open capillary network

We reconstructed serial sections of a representative adult human spleen to clarify the unknown arrangement of the splenic microvasculature, such as terminal arterioles, sheathed capillaries, the red pulp capillary network and venules. The resulting 3D model was evaluated in virtual reality (VR). Capillary sheaths often occurred after the second or third branching of a terminal arteriole and covered its capillary side or end branches. The sheaths started directly after the final smooth muscle cells of the arteriole and consisted of cuboidal CD271++ stromal sheath cells surrounded and infiltrated by B lymphocytes and macrophages. Some sheaths covered up to four sequential capillary bifurcations thus forming bizarre elongated structures. Each sheath had a unique form. Apart from symmetric dichotomous branchings inside the sheath, sheathed capillaries also gave off side branches, which crossed the sheath and freely ended at its surface. These side branches are likely to distribute materials from the incoming blood to sheath-associated B lymphocytes and macrophages and thus represent the first location for recognition of blood-borne antigens in the spleen. A few non-sheathed bypasses from terminal arterioles to the red pulp capillary network also exist. Red pulp venules are primarily supplied by sinuses, but they also exhibit a few connections to the capillary network. Thus, the human splenic red pulp harbors a primarily open microcirculation with a very minor closed part.

ANHIR: Automatic Non-Rigid Histological Image Registration Challenge

Automatic Non-rigid Histological Image Registration (ANHIR) challenge was organized to compare the performance of image registration algorithms on several kinds of microscopy histology images in a fair and independent manner. We have assembled 8 datasets, containing 355 images with 18 different stains, resulting in 481 image pairs to be registered. Registration accuracy was evaluated using manually placed landmarks. In total, 256 teams registered for the challenge, 10 submitted the results, and 6 participated in the workshop. Here, we present the results of 7 well-performing methods from the challenge together with 6 well-known existing methods. The best methods used coarse but robust initial alignment, followed by non-rigid registration, used multiresolution, and were carefully tuned for the data at hand. They outperformed off-the-shelf methods, mostly by being more robust. The best methods could successfully register over 98% of all landmarks and their mean landmark registration accuracy (TRE) was 0.44% of the image diagonal. The challenge remains open to submissions and all images are available for download.

Three-Dimensional Reconstruction of Erythrocytes Using the Novel Method For Corrective Realignment of the Transmission Electron Microscopy Cross-Section Images

The detailed kinetics study of erythrocyte deformability is useful for the early diagnosis of blood diseases and for monitoring the blood rheology. Present solutions for a three-dimensional (3D) reconstruction of erythrocytes has a limited potential. This study aimed to use erythrocyte transmission electron images (ETIs) to evaluate the morphological relationship between adjacent ETIs and generate erythrocytes 3D model. First, ultrathin serial sections of skeletal muscle tissue were obtained using an ultramicrotome. Further, the set of ETIs in a capillary were captured by transmission electron microscopy. The images were aligned by translations and rotations using custom software to optimize the morphological relationship between adjacent ETIs. These coordinate transformations exploit the unique principal axis of inertia of each image to define the body coordinate system and hence provide the means to accurately reconnect the adjacent ETIs. The sum of the distances between the corresponding points on the boundary of adjacent ETIs was minimized and, further, was optimized by using physiological relationship between the adjacent ETIs. The analysis allowed to define precise virtual relationship between the adjacent erythrocytes. Finally, extracted erythrocytes' cross-section images allowed to generate 3D model of the erythrocytes.

Locating human splenic capillary sheaths in virtual reality

Stromal capillary sheath cells in human spleens strongly express CD271, the low affinity nerve growth factor receptor p75. Serial sections of a representative adult human spleen were double-stained for CD271 versus smooth muscle alpha actin (SMA) plus CD34 to visualise capillary sheaths, the arterial tree and endothelial cells by transmitted light. Preliminary three-dimensional (3D) reconstructions of single regions were inspected in virtual reality (VR). This method showed that a large number of CD271+ sheaths occur in a post-arteriolar position often surrounding capillaries located close to divisions of arterioles. The length and diameter of capillary sheaths are rather heterogeneous. Long sheaths were observed to accompany one or two generations of capillary branches. We hypothesise that human splenic capillary sheaths may attract recirculating B-lymphocytes from the open circulation of the red pulp to start their migration into white pulp follicles along branches of the arterial tree. In addition, they may provide sites of interaction among sheath macrophages and B-lymphocytes. Our innovative approach allows stringent quality control by inserting the original immunostained serial sections into the 3D model for viewing and annotation in VR. Longer series of sections will allow to unequivocally localise most of the capillary sheaths in a given volume.

Capillary networks and follicular marginal zones in human spleens. Three-dimensional models based on immunostained serial sections

We have reconstructed small parts of capillary networks in the human splenic white pulp using serial sections immunostained for CD34 alone or for CD34 and CD271. The three-dimensional (3D) models show three types of interconnected networks: a network with very few long capillaries inside the white pulp originating from central arteries, a denser network surrounding follicles plus periarterial T-cell regions and a network in the red pulp. Capillaries of the perifollicular network and the red pulp network have open ends. Perifollicular capillaries form an arrangement similar to a basketball net located in the outer marginal zone. The marginal zone is defined by MAdCAM-1+ marginal reticular stromal cells. Perifollicular capillaries are connected to red pulp capillaries surrounded by CD271+ stromal capillary sheath cells. The scarcity of capillaries inside the splenic white pulp is astonishing, as non-polarised germinal centres with proliferating B-cells occur in adult human spleens. We suggest that specialized stromal marginal reticular cells form a barrier inside the splenic marginal zone, which together with the scarcity of capillaries guarantees the maintenance of gradients necessary for positioning of migratory B- and T-lymphocytes in the human splenic white pulp.

Three-Dimensional Arrangement of Human Bone Marrow Microvessels Revealed by Immunohistology in Undecalcified Sections

The arrangement of microvessels in human bone marrow is so far unknown. We combined monoclonal antibodies against CD34 and against CD141 to visualise all microvessel endothelia in 21 serial sections of about 1 cm² size derived from a human iliac crest. The specimen was not decalcified and embedded in Technovit^® 9100. In different regions of interest, the microvasculature was reconstructed in three dimensions using automatic methods. The three-dimensional models were subject to a rigid semiautomatic and manual quality control. In iliac crest bone marrow, the adipose tissue harbours irregularly distributed haematopoietic areas. These are fed by networks of large sinuses, which are loosely connected to networks of small capillaries prevailing in areas of pure adipose tissue. Our findings are compatible with the hypothesis that capillaries and sinuses in human iliac crest bone marrow are partially arranged in parallel.