Segmenting images using localized histograms and region merging

A deep learning-based framework (Co-ReTr) for auto-segmentation of non-small cell-lung cancer in computed tomography images

PURPOSE

Deep learning-based auto-segmentation algorithms can improve clinical workflow by defining accurate regions of interest while reducing manual labor. Over the past decade, convolutional neural networks (CNNs) have become prominent in medical image segmentation applications. However, CNNs have limitations in learning long-range spatial dependencies due to the locality of the convolutional layers. Transformers were introduced to address this challenge. In transformers with self-attention mechanism, even the first layer of information processing makes connections between distant image locations. Our paper presents a novel framework that bridges these two unique techniques, CNNs and transformers, to segment the gross tumor volume (GTV) accurately and efficiently in computed tomography (CT) images of non-small cell-lung cancer (NSCLC) patients.

METHODS

Under this framework, input of multiple resolution images was used with multi-depth backbones to retain the benefits of high-resolution and low-resolution images in the deep learning architecture. Furthermore, a deformable transformer was utilized to learn the long-range dependency on the extracted features. To reduce computational complexity and to efficiently process multi-scale, multi-depth, high-resolution 3D images, this transformer pays attention to small key positions, which were identified by a self-attention mechanism. We evaluated the performance of the proposed framework on a NSCLC dataset which contains 563 training images and 113 test images. Our novel deep learning algorithm was benchmarked against five other similar deep learning models.

RESULTS

The experimental results indicate that our proposed framework outperforms other CNN-based, transformer-based, and hybrid methods in terms of Dice score (0.92) and Hausdorff Distance (1.33). Therefore, our proposed model could potentially improve the efficiency of auto-segmentation of early-stage NSCLC during the clinical workflow. This type of framework may potentially facilitate online adaptive radiotherapy, where an efficient auto-segmentation workflow is required.

CONCLUSIONS

Our deep learning framework, based on CNN and transformer, performs auto-segmentation efficiently and could potentially assist clinical radiotherapy workflow.

Temporal activity patterns of layer II and IV rat barrel cortex neurons in healthy and injured conditions

Neurons are known to encode information not just by how frequently they fire, but also at what times they fire. However, characterizations of temporal encoding in sensory cortices under conditions of health and injury are limited. Here we characterized and compared the stimulus-evoked activity of 1210 online-sorted units in layers II and IV of rat barrel cortex under healthy and diffuse traumatic brain injury (TBI) (caused by a weight-drop model) conditions across three timepoints post-injury: four days, two weeks, and eight weeks. Temporal activity patterns in the first 50 ms post-stimulus recording showed four categories of responses: no response or 1, 2, or 3 temporally-distinct response components, that is, periods of high unit activity separated by silence. The relative proportions of unit response categories were similar between layers II and IV in healthy conditions but not in early post-TBI conditions. For units with multiple response components, inter-component timings were reliable in healthy and late post-TBI conditions but disrupted by injury. Response component times typically shifted earlier with increasing stimulus intensity and this was more pronounced in layer IV than layer II. Surprisingly, injury caused a reversal of this trend and in the late post-TBI condition no stimulus intensity-dependence differences were observed between layers II and IV. We speculate this indicates a potential compensatory mechanism in response to injury. These results demonstrate how temporal encoding features maladapt or functionally recover differently in sensory cortex after TBI. Such maladaptation or functional recovery is layer-dependent, perhaps due to differences in thalamic input or local inhibitory neuronal makeup.

Review of Deep Learning Based Automatic Segmentation for Lung Cancer Radiotherapy

Lung cancer is the leading cause of cancer-related mortality for males and females. Radiation therapy (RT) is one of the primary treatment modalities for lung cancer. While delivering the prescribed dose to tumor targets, it is essential to spare the tissues near the targets-the so-called organs-at-risk (OARs). An optimal RT planning benefits from the accurate segmentation of the gross tumor volume and surrounding OARs. Manual segmentation is a time-consuming and tedious task for radiation oncologists. Therefore, it is crucial to develop automatic image segmentation to relieve radiation oncologists of the tedious contouring work. Currently, the atlas-based automatic segmentation technique is commonly used in clinical routines. However, this technique depends heavily on the similarity between the atlas and the image segmented. With significant advances made in computer vision, deep learning as a part of artificial intelligence attracts increasing attention in medical image automatic segmentation. In this article, we reviewed deep learning based automatic segmentation techniques related to lung cancer and compared them with the atlas-based automatic segmentation technique. At present, the auto-segmentation of OARs with relatively large volume such as lung and heart etc. outperforms the organs with small volume such as esophagus. The average Dice similarity coefficient (DSC) of lung, heart and liver are over 0.9, and the best DSC of spinal cord reaches 0.9. However, the DSC of esophagus ranges between 0.71 and 0.87 with a ragged performance. In terms of the gross tumor volume, the average DSC is below 0.8. Although deep learning based automatic segmentation techniques indicate significant superiority in many aspects compared to manual segmentation, various issues still need to be solved. We discussed the potential issues in deep learning based automatic segmentation including low contrast, dataset size, consensus guidelines, and network design. Clinical limitations and future research directions of deep learning based automatic segmentation were discussed as well.

Unsupervised segmentation of noisy electron microscopy images using salient watersheds and region merging

BACKGROUND

Segmenting electron microscopy (EM) images of cellular and subcellular processes in the nervous system is a key step in many bioimaging pipelines involving classification and labeling of ultrastructures. However, fully automated techniques to segment images are often susceptible to noise and heterogeneity in EM images (e.g. different histological preparations, different organisms, different brain regions, etc.). Supervised techniques to address this problem are often helpful but require large sets of training data, which are often difficult to obtain in practice, especially across many conditions.

RESULTS

We propose a new, principled unsupervised algorithm to segment EM images using a two-step approach: edge detection via salient watersheds following by robust region merging. We performed experiments to gather EM neuroimages of two organisms (mouse and fruit fly) using different histological preparations and generated manually curated ground-truth segmentations. We compared our algorithm against several state-of-the-art unsupervised segmentation algorithms and found superior performance using two standard measures of under-and over-segmentation error.

CONCLUSIONS

Our algorithm is general and may be applicable to other large-scale segmentation problems for bioimages.

A divide and conquer deformable contour method with a model based searching algorithm

A divide and conquer deformable contour method is presented with an initial inside closed contour being divided into arbitrary segments, and these segments are allowed to deform separately preserving the segments' connectivity. A maximum area threshold, A/sub max/, is used to stop these outward contour segments' marching. Clear and blur contour points are then identified to partition the whole contour into clear and blur segments. A bi-directional searching method is then recursively applied to each blur segment including a search for contour-within-contour segment to reach a final close contour. Further improvements are provided by a model based searching algorithm. It is a two-step process with step 1 being a linked contour model matching operation where landmarks are extracted, and step 2 being a posteriori probability model matching and correction operation where large error segments are fine tuned to obtain the final results. The experiments include ultrasound images of pig heart, MRI brain images, MRI knee images having complex shapes with or without gaps, and inhomogeneous interior and contour region brightness distributions. These experiments have shown that the method has the capability of moving a contour into the neighboring region of the desired boundary by overcoming inhomogeneous interior, and by adapting each contour segment searching operation to different local difficulties, through a contour partition and repartition scheme in searching for a final solution.

An estimation/correction algorithm for detecting bone edges in CT images

The normal direction of the bone contour in computed tomography (CT) images provides important anatomical information and can guide segmentation algorithms. Since various bones in CT images have different sizes, and the intensity values of bone pixels are generally nonuniform and noisy, estimation of the normal direction using a single scale is not reliable. We propose a multiscale approach to estimate the normal direction of bone edges. The reliability of the estimation is calculated from the estimated results and, after re-scaling, the reliability is used to further correct the normal direction. The optimal scale at each point is obtained while estimating the normal direction; this scale is then used in a simple edge detector. Our experimental results have shown that use of this estimated/corrected normal direction improves the segmentation quality by decreasing the number of unexpected edges and discontinuities (gaps) of real contours. The corrected normal direction could also be used in postprocessing to delete false edges. Our segmentation algorithm is automatic, and its performance is evaluated on CT images of the human pelvis, leg, and wrist.

Segmentation of carpal bones from CT images using skeletally coupled deformable models

The in vivo investigation of joint kinematics in normal and injured wrist requires the segmentation of carpal bones from 3D (CT) images, and their registration over time. The non-uniformity of bone tissue, ranging from dense cortical bone to textured spongy bone, the irregular shape of closely packed carpal bones, small inter-bone spaces compared to the resolution of CT images, along with the presence of blood vessels, and the inherent blurring of CT imaging render the segmentation of carpal bones a challenging task. We review the performance of statistical classification, deformable models (active contours), region growing, region competition, and morphological operations for this application. We then propose a model which combines several of these approaches in a unified framework. Specifically, our approach is to use a curve evolution implementation of region growing from initialized seeds, where growth is modulated by a skeletally-mediated competition between neighboring regions. The inter-seed skeleton, which we interpret as the predicted boundary of collision between two regions, is used to couple the growth of seeds and to mediate long-range competition between them. The implementation requires subpixel representations of each growing region as well as the inter-region skeleton. This method combines the advantages of active contour models, region growing, and both local and global region competition methods. We demonstrate the effectiveness of this approach for our application where many of the difficulties presented above are overcome as illustrated by synthetic and real examples. Since this segmentation method does not rely on domain-specific knowledge, it should be applicable to a range of other medical imaging segmentation tasks.

Automated morphometric analysis in peripheral neuropathies

We describe a three-step algorithm for the morphometric analysis of color images of nerve specimens, currently used in the diagnosis of peripheral neuropathies. The algorithm first segments the images by applying a clustering method in the color space. It then identifies and eliminates irrelevant regions and, in the final step, calculates the diagnostic parameters required for clinical analysis. The results obtained on 25 images are reported and compared with corresponding measurements made by neurologists.

Hybrid image segmentation using watersheds and fast region merging

A hybrid multidimensional image segmentation algorithm is proposed, which combines edge and region-based techniques through the morphological algorithm of watersheds. An edge-preserving statistical noise reduction approach is used as a preprocessing stage in order to compute an accurate estimate of the image gradient. Then, an initial partitioning of the image into primitive regions is produced by applying the watershed transform on the image gradient magnitude. This initial segmentation is the input to a computationally efficient hierarchical (bottom-up) region merging process that produces the final segmentation. The latter process uses the region adjacency graph (RAG) representation of the image regions. At each step, the most similar pair of regions is determined (minimum cost RAG edge), the regions are merged and the RAG is updated. Traditionally, the above is implemented by storing all RAG edges in a priority queue. We propose a significantly faster algorithm, which additionally maintains the so-called nearest neighbor graph, due to which the priority queue size and processing time are drastically reduced. The final segmentation provides, due to the RAG, one-pixel wide, closed, and accurately localized contours/surfaces. Experimental results obtained with two-dimensional/three-dimensional (2-D/3-D) magnetic resonance images are presented.

Multimodality cardiovascular image segmentation using a deformable contour model

An automatic segmentation method has been developed for cardiovascular multimodality imaging. A "snake" model based on a curve shaping and an energy-minimizing process is used to detect blood-wall interfaces on Cine-CT, MRI and ultrasound images. Deformation of a reduced set of contour points was made according to a discretized global, regional and local minimum energy criterion. A continuous regional optimization process was also integrated into the deformation model, it takes into account a cubic spline interpolation and adaptive regularity constraints. The constraints provided rapid convergence toward a final contour position by successively stopping spline segments.

Precise matching of 3-D target models to multisensor data

This paper presents a three-dimensional (3-D) model-based ATR algorithm that operates simultaneously on imagery from three heterogeneous, approximately boresight aligned sensors. An iterative search matches models to range and optical imagery by repeatedly predicting detectable features, measuring support for these features in the imagery, and adjusting the transformations relating the target to the sensors in order to improve the match. The result is a locally optimal and globally consistent set of 3-D transformations that precisely relate the best matching target features to combined range, IR, and color images. Results show the multisensor algorithm recovers 3-D target pose more accurately than does a traditional single-sensor algorithm. Errors in registration between images are also corrected during matching. The intended application is imaging from semiautonomous military scout vehicles.

Segmenting radiographs of the hand and wrist

The work presented in this paper concerns the development of computer-based techniques for the segmentation of hand-wrist radiographs and in particular those obtained for the TW2 method for the assessment of skeletal maturity (bone age). The segmentation method is based on the concept of regions and it consists of region growing and region merging stages. A bone extraction stage follows, which labels regions as either bone or background using heuristic rules based on the grey level properties of the scene. Finally, a technique is proposed for the segmentation of bone outlines which helps in identifying conjugated bones.