1
|
Liu Y, Gargesha M, Scott B, Tchilibou Wane AO, Wilson DL. Deep learning multi-organ segmentation for whole mouse cryo-images including a comparison of 2D and 3D deep networks. Sci Rep 2022; 12:15161. [PMID: 36071089 PMCID: PMC9452525 DOI: 10.1038/s41598-022-19037-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2021] [Accepted: 08/23/2022] [Indexed: 11/25/2022] Open
Abstract
Cryo-imaging provided 3D whole-mouse microscopic color anatomy and fluorescence images that enables biotechnology applications (e.g., stem cells and metastatic cancer). In this report, we compared three methods of organ segmentation: 2D U-Net with 2D-slices and 3D U-Net with either 3D-whole-mouse or 3D-patches. We evaluated the brain, thymus, lung, heart, liver, stomach, spleen, left and right kidney, and bladder. Training with 63 mice, 2D-slices had the best performance, with median Dice scores of > 0.9 and median Hausdorff distances of < 1.2 mm in eightfold cross validation for all organs, except bladder, which is a problem organ due to variable filling and poor contrast. Results were comparable to those for a second analyst on the same data. Regression analyses were performed to fit learning curves, which showed that 2D-slices can succeed with fewer samples. Review and editing of 2D-slices segmentation results reduced human operator time from ~ 2-h to ~ 25-min, with reduced inter-observer variability. As demonstrations, we used organ segmentation to evaluate size changes in liver disease and to quantify the distribution of therapeutic mesenchymal stem cells in organs. With a 48-GB GPU, we determined that extra GPU RAM improved the performance of 3D deep learning because we could train at a higher resolution.
Collapse
Affiliation(s)
- Yiqiao Liu
- Department of Biomedical Engineering, Case Western Reserve University, 10900 Euclid Avenue, Cleveland, OH, 44106, USA
| | | | - Bryan Scott
- BioInVision Inc, Suite E 781 Beta Drive, Cleveland, OH, 44143, USA
| | - Arthure Olivia Tchilibou Wane
- Department of Biomedical Engineering, Case Western Reserve University, 10900 Euclid Avenue, Cleveland, OH, 44106, USA
| | - David L Wilson
- Department of Biomedical Engineering, Case Western Reserve University, 10900 Euclid Avenue, Cleveland, OH, 44106, USA. .,BioInVision Inc, Suite E 781 Beta Drive, Cleveland, OH, 44143, USA. .,Department of Radiology, Case Western Reserve University, 10900 Euclid Avenue, Cleveland, OH, 44106, USA.
| |
Collapse
|
2
|
Qiu Z, Xu T, Langerman J, Das W, Wang C, Nair N, Aristizabal O, Mamou J, Turnbull DH, Ketterling JA, Wang Y. A Deep Learning Approach for Segmentation, Classification, and Visualization of 3-D High-Frequency Ultrasound Images of Mouse Embryos. IEEE TRANSACTIONS ON ULTRASONICS, FERROELECTRICS, AND FREQUENCY CONTROL 2021; 68:2460-2471. [PMID: 33755564 PMCID: PMC8274381 DOI: 10.1109/tuffc.2021.3068156] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/07/2023]
Abstract
Segmentation and mutant classification of high-frequency ultrasound (HFU) mouse embryo brain ventricle (BV) and body images can provide valuable information for developmental biologists. However, manual segmentation and identification of BV and body requires substantial time and expertise. This article proposes an accurate, efficient and explainable deep learning pipeline for automatic segmentation and classification of the BV and body. For segmentation, a two-stage framework is implemented. The first stage produces a low-resolution segmentation map, which is then used to crop a region of interest (ROI) around the target object and serve as the probability map of the autocontext input for the second-stage fine-resolution refinement network. The segmentation then becomes tractable on high-resolution 3-D images without time-consuming sliding windows. The proposed segmentation method significantly reduces inference time (102.36-0.09 s/volume ≈ 1000× faster) while maintaining high accuracy comparable to previous sliding-window approaches. Based on the BV and body segmentation map, a volumetric convolutional neural network (CNN) is trained to perform a mutant classification task. Through backpropagating the gradients of the predictions to the input BV and body segmentation map, the trained classifier is found to largely focus on the region where the Engrailed-1 (En1) mutation phenotype is known to manifest itself. This suggests that gradient backpropagation of deep learning classifiers may provide a powerful tool for automatically detecting unknown phenotypes associated with a known genetic mutation.
Collapse
|
3
|
Xu T, Qiu Z, Das W, Wang C, Langerman J, Nair N, Aristizábal O, Mamou J, Turnbull DH, Ketterling JA, Wang Y. DEEP MOUSE: AN END-TO-END AUTO-CONTEXT REFINEMENT FRAMEWORK FOR BRAIN VENTRICLE & BODY SEGMENTATION IN EMBRYONIC MICE ULTRASOUND VOLUMES. PROCEEDINGS. IEEE INTERNATIONAL SYMPOSIUM ON BIOMEDICAL IMAGING 2020; 2020:122-126. [PMID: 33381278 PMCID: PMC7768981 DOI: 10.1109/isbi45749.2020.9098387] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
The segmentation of the brain ventricle (BV) and body in embryonic mice high-frequency ultrasound (HFU) volumes can provide useful information for biological researchers. However, manual segmentation of the BV and body requires substantial time and expertise. This work proposes a novel deep learning based end-to-end auto-context refinement framework, consisting of two stages. The first stage produces a low resolution segmentation of the BV and body simultaneously. The resulting probability map for each object (BV or body) is then used to crop a region of interest (ROI) around the target object in both the original image and the probability map to provide context to the refinement segmentation network. Joint training of the two stages provides significant improvement in Dice Similarity Coefficient (DSC) over using only the first stage (0.818 to 0.906 for the BV, and 0.919 to 0.934 for the body). The proposed method significantly reduces the inference time (102.36 to 0.09 s/volume ≈1000x faster) while slightly improves the segmentation accuracy over the previous methods using slide-window approaches.
Collapse
Affiliation(s)
- Tongda Xu
- Department of Electrical and Computer Engineering, New York University, New York, USA
| | - Ziming Qiu
- Department of Electrical and Computer Engineering, New York University, New York, USA
| | | | - Chuiyu Wang
- School of Electronic and Information Engineering, Beihang University, Beijing, China
| | - Jack Langerman
- Department of Computer Science, New York University, New York, USA
| | - Nitin Nair
- Department of Electrical and Computer Engineering, New York University, New York, USA
| | - Orlando Aristizábal
- F. L. Lizzi Center for Biomedical Engineering, Riverside Research, New York, USA
- Skirball Institute of Biomolecular Medicine, New York University, New York, USA
| | - Jonathan Mamou
- F. L. Lizzi Center for Biomedical Engineering, Riverside Research, New York, USA
| | - Daniel H Turnbull
- Skirball Institute of Biomolecular Medicine, New York University, New York, USA
| | - Jeffrey A Ketterling
- F. L. Lizzi Center for Biomedical Engineering, Riverside Research, New York, USA
| | - Yao Wang
- Department of Electrical and Computer Engineering, New York University, New York, USA
| |
Collapse
|
4
|
Qiu Z, Langerman J, Nair N, Aristizabal O, Mamou J, Turnbull DH, Ketterling J, Wang Y. DEEP BV: A FULLY AUTOMATED SYSTEM FOR BRAIN VENTRICLE LOCALIZATION AND SEGMENTATION IN 3D ULTRASOUND IMAGES OF EMBRYONIC MICE. ... IEEE SIGNAL PROCESSING IN MEDICINE AND BIOLOGY SYMPOSIUM (SPMB). IEEE SIGNAL PROCESSING IN MEDICINE AND BIOLOGY SYMPOSIUM 2018; 2018:10.1109/SPMB.2018.8615610. [PMID: 30911672 PMCID: PMC6429562 DOI: 10.1109/spmb.2018.8615610] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/09/2023]
Abstract
Volumetric analysis of brain ventricle (BV) structure is a key tool in the study of central nervous system development in embryonic mice. High-frequency ultrasound (HFU) is the only non-invasive, real-time modality available for rapid volumetric imaging of embryos in utero. However, manual segmentation of the BV from HFU volumes is tedious, time-consuming, and requires specialized expertise. In this paper, we propose a novel deep learning based BV segmentation system for whole-body HFU images of mouse embryos. Our fully automated system consists of two modules: localization and segmentation. It first applies a volumetric convolutional neural network on a 3D sliding window over the entire volume to identify a 3D bounding box containing the entire BV. It then employs a fully convolutional network to segment the detected bounding box into BV and background. The system achieves a Dice Similarity Coefficient (DSC) of 0.8956 for BV segmentation on an unseen 111 HFU volume test set surpassing the previous state-of-the-art method (DSC of 0.7119) by a margin of 25%.
Collapse
Affiliation(s)
- Ziming Qiu
- Electrical and Computer Engineering, Tandon School of Engineering, New York University, Brooklyn, USA
| | - Jack Langerman
- Computer Science, Tandon School of Engineering, New York University, Brooklyn, USA
| | - Nitin Nair
- Electrical and Computer Engineering, Tandon School of Engineering, New York University, Brooklyn, USA
| | - Orlando Aristizabal
- F. L. Lizzi Center for Biomedical Engineering, Riverside Research, New York, USA
- Skirball Institute of Biomolecular Medicine, New York University School of Medicine, New York, USA
| | - Jonathan Mamou
- F. L. Lizzi Center for Biomedical Engineering, Riverside Research, New York, USA
| | - Daniel H Turnbull
- Skirball Institute of Biomolecular Medicine, New York University School of Medicine, New York, USA
| | - Jeffrey Ketterling
- F. L. Lizzi Center for Biomedical Engineering, Riverside Research, New York, USA
| | - Yao Wang
- Electrical and Computer Engineering, Tandon School of Engineering, New York University, Brooklyn, USA
| |
Collapse
|