1
|
Liu J, Zheng Y, Lin L, Guo J, Lv Y, Yuan J, Zhai H, Chen X, Shen L, Li L, Bai S, Han H. A robust transformer-based pipeline of 3D cell alignment, denoise and instance segmentation on electron microscopy sequence images. JOURNAL OF PLANT PHYSIOLOGY 2024; 297:154236. [PMID: 38621330 DOI: 10.1016/j.jplph.2024.154236] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/01/2024] [Revised: 03/15/2024] [Accepted: 03/20/2024] [Indexed: 04/17/2024]
Abstract
Germline cells are critical for transmitting genetic information to subsequent generations in biological organisms. While their differentiation from somatic cells during embryonic development is well-documented in most animals, the regulatory mechanisms initiating plant germline cells are not well understood. To thoroughly investigate the complex morphological transformations of their ultrastructure over developmental time, nanoscale 3D reconstruction of entire plant tissues is necessary, achievable exclusively through electron microscopy imaging. This paper presents a full-process framework designed for reconstructing large-volume plant tissue from serial electron microscopy images. The framework ensures end-to-end direct output of reconstruction results, including topological networks and morphological analysis. The proposed 3D cell alignment, denoise, and instance segmentation pipeline (3DCADS) leverages deep learning to provide a cell instance segmentation workflow for electron microscopy image series, ensuring accurate and robust 3D cell reconstructions with high computational efficiency. The pipeline involves five stages: the registration of electron microscopy serial images; image enhancement and denoising; semantic segmentation using a Transformer-based neural network; instance segmentation through a supervoxel-based clustering algorithm; and an automated analysis and statistical assessment of the reconstruction results, with the mapping of topological connections. The 3DCADS model's precision was validated on a plant tissue ground-truth dataset, outperforming traditional baseline models and deep learning baselines in overall accuracy. The framework was applied to the reconstruction of early meiosis stages in the anthers of Arabidopsis thaliana, resulting in a topological connectivity network and analysis of morphological parameters and characteristics of cell distribution. The experiment underscores the 3DCADS model's potential for biological tissue identification and its significance in quantitative analysis of plant cell development, crucial for examining samples across different genetic phenotypes and mutations in plant development. Additionally, the paper discusses the regulatory mechanisms of Arabidopsis thaliana's germline cells and the development of stamen cells before meiosis, offering new insights into the transition from somatic to germline cell fate in plants.
Collapse
Affiliation(s)
- Jiazheng Liu
- School of Future Technology, University of Chinese Academy of Sciences, Beijing 101408, China; Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; Team of Microscale Reconstruction and Intelligent Analysis, Laboratory of Brain-AI, Institute of Automation, Chinese Academy of Sciences, Beijing 101499, China
| | - Yafeng Zheng
- College of Life Sciences, Peking University, Beijing 100871, China; State Key Laboratory of Protein and Plant Gene Research, Beijing 100871, China
| | - Limei Lin
- Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; Team of Microscale Reconstruction and Intelligent Analysis, Laboratory of Brain-AI, Institute of Automation, Chinese Academy of Sciences, Beijing 101499, China
| | - Jingyue Guo
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 101408, China; Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; Team of Microscale Reconstruction and Intelligent Analysis, Laboratory of Brain-AI, Institute of Automation, Chinese Academy of Sciences, Beijing 101499, China
| | - Yanan Lv
- Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; Team of Microscale Reconstruction and Intelligent Analysis, Laboratory of Brain-AI, Institute of Automation, Chinese Academy of Sciences, Beijing 101499, China
| | - Jingbin Yuan
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 101408, China; Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; Team of Microscale Reconstruction and Intelligent Analysis, Laboratory of Brain-AI, Institute of Automation, Chinese Academy of Sciences, Beijing 101499, China
| | - Hao Zhai
- School of Future Technology, University of Chinese Academy of Sciences, Beijing 101408, China; Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; Team of Microscale Reconstruction and Intelligent Analysis, Laboratory of Brain-AI, Institute of Automation, Chinese Academy of Sciences, Beijing 101499, China
| | - Xi Chen
- Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; Team of Microscale Reconstruction and Intelligent Analysis, Laboratory of Brain-AI, Institute of Automation, Chinese Academy of Sciences, Beijing 101499, China
| | - Lijun Shen
- Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; Team of Microscale Reconstruction and Intelligent Analysis, Laboratory of Brain-AI, Institute of Automation, Chinese Academy of Sciences, Beijing 101499, China
| | - LinLin Li
- Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; Team of Microscale Reconstruction and Intelligent Analysis, Laboratory of Brain-AI, Institute of Automation, Chinese Academy of Sciences, Beijing 101499, China.
| | - Shunong Bai
- College of Life Sciences, Peking University, Beijing 100871, China; State Key Laboratory of Protein and Plant Gene Research, Beijing 100871, China.
| | - Hua Han
- School of Future Technology, University of Chinese Academy of Sciences, Beijing 101408, China; Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; Team of Microscale Reconstruction and Intelligent Analysis, Laboratory of Brain-AI, Institute of Automation, Chinese Academy of Sciences, Beijing 101499, China.
| |
Collapse
|
2
|
Li X, Long M, Huang J, Wu J, Shen H, Zhou F, Hou J, Xu Y, Wang D, Mei L, Liu Y, Hu T, Lei C. An orientation-free ring feature descriptor with stain-variability normalization for pathology image matching. Comput Biol Med 2023; 167:107675. [PMID: 37976825 DOI: 10.1016/j.compbiomed.2023.107675] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2023] [Revised: 10/08/2023] [Accepted: 11/06/2023] [Indexed: 11/19/2023]
Abstract
Comprehensively analyzing the corresponding regions in the images of serial slices stained using different methods is a common but important operation in pathological diagnosis. To help increase the efficiency of the analysis, various image registration methods are proposed to match the corresponding regions in different images, but their performance is highly influenced by the rotations, deformations, and variations of staining between the serial pathology images. In this work, we propose an orientation-free ring feature descriptor with stain-variability normalization for pathology image matching. Specifically, we normalize image staining to similar levels to minimize the impact of staining differences on pathology image matching. To overcome the rotation and deformation issues, we propose a rotation-invariance orientation-free ring feature descriptor that generates novel adaptive bins from ring features to build feature vectors. We measure the Euclidean distance of the feature vectors to evaluate keypoint similarity to achieve pathology image matching. A total of 46 pairs of clinical pathology images in hematoxylin-eosin and immunohistochemistry straining to verify the performance of our method. Experimental results indicate that our method meets the pathology image matching accuracy requirements (error ¡ 300μm), especially competent for large-angle rotation cases common in clinical practice.
Collapse
Affiliation(s)
- Xiaoxiao Li
- The Institute of Technological Sciences, Wuhan University, Wuhan 430072, China
| | - Mengping Long
- The Institute of Technological Sciences, Wuhan University, Wuhan 430072, China; Department of Pathology, Peking University Cancer Hospital, Beijing 100142, China
| | - Jin Huang
- The Institute of Technological Sciences, Wuhan University, Wuhan 430072, China
| | - Jianghua Wu
- Department of Pathology, Peking University Cancer Hospital, Beijing 100142, China
| | - Hui Shen
- Department of Hematology, Zhongnan Hospital of Wuhan University, Wuhan, 430071, China
| | - Fuling Zhou
- Department of Hematology, Zhongnan Hospital of Wuhan University, Wuhan, 430071, China
| | - Jinxuan Hou
- Department of Thyroid and Breast Surgery, Zhongnan Hospital of Wuhan University, Wuhan, 430071, China
| | - Yu Xu
- Department of Radiation and Medical Oncology, Zhongnan Hospital of Wuhan University, Wuhan, 430071, China
| | - Du Wang
- The Institute of Technological Sciences, Wuhan University, Wuhan 430072, China
| | - Liye Mei
- The Institute of Technological Sciences, Wuhan University, Wuhan 430072, China; School of Computer Science, Hubei University of Technology, Wuhan, 430068, China.
| | - Yiqiang Liu
- Department of Pathology, Peking University Cancer Hospital, Beijing 100142, China
| | - Taobo Hu
- The Institute of Technological Sciences, Wuhan University, Wuhan 430072, China; Department of Breast Surgery, Peking University People's Hospital, Beijing, 100044, China
| | - Cheng Lei
- The Institute of Technological Sciences, Wuhan University, Wuhan 430072, China; Suzhou Institute of Wuhan University, Suzhou, 215000, China; Shenzhen Institute of Wuhan University, Shenzhen, 518057, China.
| |
Collapse
|
3
|
Lin Y, Liang Z, He Y, Huang W, Guan T. End-to-end affine registration framework for histopathological images with weak annotations. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 241:107763. [PMID: 37634308 DOI: 10.1016/j.cmpb.2023.107763] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/28/2023] [Revised: 08/12/2023] [Accepted: 08/12/2023] [Indexed: 08/29/2023]
Abstract
BACKGROUND AND OBJECTIVE Histopathological image registration is an essential component in digital pathology and biomedical image analysis. Deep-learning-based algorithms have been proposed to achieve fast and accurate affine registration. Some previous studies assume that the pairs are free from sizeable initial position misalignment and large rotation angles before performing the affine transformation. However, large-rotation angles are often introduced into image pairs during the production process in real-world pathology images. Reliable initial alignment is important for registration performance. The existing deep-learning-based approaches often use a two-step affine registration pipeline because convolutional neural networks (CNNs) cannot correct large-angle rotations. METHODS In this manuscript, a general framework ARoNet is developed to achieve end-to-end affine registration for histopathological images. We use CNNs to extract global features of images and fuse them to construct correspondent information for affine transformation. In ARoNet, a rotation recognition network is implemented to eliminate great rotation misalignment. In addition, a self-supervised learning task is proposed to assist the learning of image representations in an unsupervised manner. RESULTS We applied our model to four datasets, and the results indicate that ARoNet surpasses existing affine registration algorithms in alignment accuracy when large angular misalignments (e.g., 180 rotation) are present, providing accurate affine initialization for subsequent non-rigid alignments. Besides, ARoNet shows advantages in execution time (0.05 per pair), registration accuracy, and robustness. CONCLUSION We believe that the proposed general framework promises to simplify and speed up the registration process and has the potential for clinical applications.
Collapse
Affiliation(s)
- Yuanhua Lin
- Shenzhen International Graduate School, Tsinghua University, 518055, Shenzhen, China
| | - Zhendong Liang
- Shenzhen International Graduate School, Tsinghua University, 518055, Shenzhen, China
| | - Yonghong He
- Shenzhen International Graduate School, Tsinghua University, 518055, Shenzhen, China
| | - Wenting Huang
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital & Shenzhen Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, 518116, Shenzhen, China
| | - Tian Guan
- Shenzhen International Graduate School, Tsinghua University, 518055, Shenzhen, China.
| |
Collapse
|
4
|
Gao Z, Tian Z, Pu B, Li S, Li K. Deep endpoints focusing network under geometric constraints for end-to-end biometric measurement in fetal ultrasound images. Comput Biol Med 2023; 165:107399. [PMID: 37683530 DOI: 10.1016/j.compbiomed.2023.107399] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/27/2023] [Revised: 07/25/2023] [Accepted: 08/26/2023] [Indexed: 09/10/2023]
Abstract
Biometric measurements in fetal ultrasound images are one of the most highly demanding medical image analysis tasks that can directly contribute to diagnosing fetal diseases. However, the natural high-speckle noise and shadows in ultrasound data present big challenges for automatic biometric measurement. Almost all the existing dominant automatic methods are two-stage models, where the key anatomical structures are segmented first and then measured, thus bringing segmentation and fitting errors. What is worse, the results of the second-stage fitting are completely dependent on the good performance of first-stage segmentation, i.e., the segmentation error will lead to a larger fitting error. To this end, we propose a novel end-to-end biometric measurement network, abbreviated as E2EBM-Net, that directly fits the measurement parameters. E2EBM-Net includes a cross-level feature fusion module to extract multi-scale texture information, a hard-soft attention module to improve position sensitivity, and center-focused detectors jointly to achieve accurate localizing and regressing of the measurement endpoints, as well as a loss function with geometric cues to enhance the correlations. To our knowledge, this is the first AI-based application to address the biometric measurement of irregular anatomical structures in fetal ultrasound images with an end-to-end approach. Experiment results showed that E2EBM-Net outperformed the existing methods and achieved the state-of-the-art performance.
Collapse
Affiliation(s)
- Zhan Gao
- College of Computer Science and Electronic Engineering, Hunan University, Changsha 410000, China
| | - Zean Tian
- College of Computer Science and Electronic Engineering, Hunan University, Changsha 410000, China
| | - Bin Pu
- College of Computer Science and Electronic Engineering, Hunan University, Changsha 410000, China
| | - Shengli Li
- Department of Ultrasound, Shenzhen Maternal & Child Healthcare Hospital, Southern Medical University, Shenzhen, 518028, China
| | - Kenli Li
- College of Computer Science and Electronic Engineering, Hunan University, Changsha 410000, China.
| |
Collapse
|
5
|
Hoque MZ, Keskinarkaus A, Nyberg P, Xu H, Seppänen T. Invasion depth estimation of carcinoma cells using adaptive stain normalization to improve epidermis segmentation accuracy. Comput Med Imaging Graph 2023; 108:102276. [PMID: 37611486 DOI: 10.1016/j.compmedimag.2023.102276] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Revised: 07/25/2023] [Accepted: 07/26/2023] [Indexed: 08/25/2023]
Abstract
Submucosal invasion depth is a significant prognostic factor when assessing lymph node metastasis and cancer itself to plan proper treatment for the patient. Conventionally, oncologists measure the invasion depth by hand which is a laborious, subjective, and time-consuming process. The manual pathological examination by measuring accurate carcinoma cell invasion with considerable inter-observer and intra-observer variations is still challenging. The increasing use of medical imaging and artificial intelligence reveals a significant role in clinical medicine and pathology. In this paper, we propose an approach to study invasive behavior and measure the invasion depth of carcinoma from stained histopathology images. Specifically, our model includes adaptive stain normalization, color decomposition, and morphological reconstruction with adaptive thresholding to separate the epithelium with blue ratio image. Our method splits the image into multiple non-overlapping meaningful segments and successfully finds the homogeneous segments to measure accurate invasion depth. The invasion depths are measured from the inner epithelium edge to outermost pixels of the deepest part of particles in image. We conduct our experiments on skin melanoma tissue samples as well as on organotypic invasion model utilizing myoma tissue and oral squamous cell carcinoma. The performance is experimentally compared to three closely related reference methods and our method provides a superior result in measuring invasion depth. This computational technique will be beneficial for the segmentation of epithelium and other particles for the development of novel computer-aided diagnostic tools in biobank applications.
Collapse
Affiliation(s)
- Md Ziaul Hoque
- Center for Machine Vision and Signal Analysis, Faculty of Information Technology and Electrical Engineering, University of Oulu, Finland; Division of Nephrology and Intelligent Critical Care, Department of Medicine, University of Florida, Gainesville, USA.
| | - Anja Keskinarkaus
- Center for Machine Vision and Signal Analysis, Faculty of Information Technology and Electrical Engineering, University of Oulu, Finland
| | - Pia Nyberg
- Biobank Borealis of Northern Finland, Oulu University Hospital, Finland; Translational Medicine Research Unit, Medical Research Center Oulu, Faculty of Medicine, University of Oulu, Finland
| | - Hongming Xu
- Department of Electrical and Computer Engineering, University of Alberta, Canada; School of Biomedical Engineering, Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian, China
| | - Tapio Seppänen
- Center for Machine Vision and Signal Analysis, Faculty of Information Technology and Electrical Engineering, University of Oulu, Finland
| |
Collapse
|
6
|
Roy M, Wang F, Teodoro G, Bhattarai S, Bhargava M, Rekha TS, Aneja R, Kong J. Deep learning based registration of serial whole-slide histopathology images in different stains. J Pathol Inform 2023; 14:100311. [PMID: 37214150 PMCID: PMC10193019 DOI: 10.1016/j.jpi.2023.100311] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Revised: 04/11/2023] [Accepted: 04/12/2023] [Indexed: 05/24/2023] Open
Abstract
For routine pathology diagnosis and imaging-based biomedical research, Whole-slide image (WSI) analyses have been largely limited to a 2D tissue image space. For a more definitive tissue representation to support fine-resolution spatial and integrative analyses, it is critical to extend such tissue-based investigations to a 3D tissue space with spatially aligned serial tissue WSIs in different stains, such as Hematoxylin and Eosin (H&E) and Immunohistochemistry (IHC) biomarkers. However, such WSI registration is technically challenged by the overwhelming image scale, the complex histology structure change, and the significant difference in tissue appearances in different stains. The goal of this study is to register serial sections from multi-stain histopathology whole-slide image blocks. We propose a novel translation-based deep learning registration network CGNReg that spatially aligns serial WSIs stained in H&E and by IHC biomarkers without prior deformation information for the model training. First, synthetic IHC images are produced from H&E slides through a robust image synthesis algorithm. Next, the synthetic and the real IHC images are registered through a Fully Convolutional Network with multi-scaled deformable vector fields and a joint loss optimization. We perform the registration at the full image resolution, retaining the tissue details in the results. Evaluated with a dataset of 76 breast cancer patients with 1 H&E and 2 IHC serial WSIs for each patient, CGNReg presents promising performance as compared with multiple state-of-the-art systems in our evaluation. Our results suggest that CGNReg can produce promising registration results with serial WSIs in different stains, enabling integrative 3D tissue-based biomedical investigations.
Collapse
Affiliation(s)
- Mousumi Roy
- Department of Computer Science, Stony Brook University, NY 11794, USA
| | - Fusheng Wang
- Department of Computer Science, Stony Brook University, NY 11794, USA
- Department of Biomedical Informatics, Stony Brook University, NY 11794, USA
| | - George Teodoro
- Department of Computer Science, Federal University of Minas Gerais, Belo Horizonte 31270-901, Brazil
| | - Shristi Bhattarai
- Department of Clinical and Diagnostic Sciences, School of Health Profession, University of Alabama at Birmingham, Birmingham, AL 35233, USA
| | - Mahak Bhargava
- Department of Clinical and Diagnostic Sciences, School of Health Profession, University of Alabama at Birmingham, Birmingham, AL 35233, USA
| | - T. Subbanna Rekha
- Department of Pathology, JSS Medical College, JSS Academy of Higher Education and Research, Mysuru, Karnataka 570009, India
| | - Ritu Aneja
- Department of Clinical and Diagnostic Sciences, School of Health Profession, University of Alabama at Birmingham, Birmingham, AL 35233, USA
| | - Jun Kong
- Department of Mathematics and Statistics, Georgia State University, Atlanta, GA 30303, USA
- Department of Computer Science and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA
| |
Collapse
|
7
|
Rahman MM, Khan MSI, Babu HMH. BreastMultiNet: A multi-scale feature fusion method using deep neural network to detect breast cancer. ARRAY 2022. [DOI: 10.1016/j.array.2022.100256] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022] Open
|