1
|
Xu MC, Zhou Y, Jin C, De Groot M, Alexander DC, Oxtoby NP, Jacob J. MisMatch: Calibrated Segmentation via Consistency on Differential Morphological Feature Perturbations With Limited Labels. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:2988-2999. [PMID: 37155408 PMCID: PMC7615173 DOI: 10.1109/tmi.2023.3273158] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/27/2022] [Revised: 03/14/2023] [Accepted: 05/01/2023] [Indexed: 05/10/2023]
Abstract
Semi-supervised learning (SSL) is a promising machine learning paradigm to address the ubiquitous issue of label scarcity in medical imaging. The state-of-the-art SSL methods in image classification utilise consistency regularisation to learn unlabelled predictions which are invariant to input level perturbations. However, image level perturbations violate the cluster assumption in the setting of segmentation. Moreover, existing image level perturbations are hand-crafted which could be sub-optimal. In this paper, we propose MisMatch, a semi-supervised segmentation framework based on the consistency between paired predictions which are derived from two differently learnt morphological feature perturbations. MisMatch consists of an encoder and two decoders. One decoder learns positive attention for foreground on unlabelled data thereby generating dilated features of foreground. The other decoder learns negative attention for foreground on the same unlabelled data thereby generating eroded features of foreground. We normalise the paired predictions of the decoders, along the batch dimension. A consistency regularisation is then applied between the normalised paired predictions of the decoders. We evaluate MisMatch on four different tasks. Firstly, we develop a 2D U-net based MisMatch framework and perform extensive cross-validation on a CT-based pulmonary vessel segmentation task and show that MisMatch statistically outperforms state-of-the-art semi-supervised methods. Secondly, we show that 2D MisMatch outperforms state-of-the-art methods on an MRI-based brain tumour segmentation task. We then further confirm that 3D V-net based MisMatch outperforms its 3D counterpart based on consistency regularisation with input level perturbations, on two different tasks including, left atrium segmentation from 3D CT images and whole brain tumour segmentation from 3D MRI images. Lastly, we find that the performance improvement of MisMatch over the baseline might originate from its better calibration. This also implies that our proposed AI system makes safer decisions than the previous methods.
Collapse
Affiliation(s)
- Mou-Cheng Xu
- Centre for Medical Image ComputingUCLWC1V 6LJLondonU.K.
| | - Yukun Zhou
- Centre for Medical Image ComputingUCLWC1V 6LJLondonU.K.
| | - Chen Jin
- Centre for Medical Image ComputingUCLWC1V 6LJLondonU.K.
| | | | | | | | - Joseph Jacob
- Centre for Medical Image ComputingUCLWC1V 6LJLondonU.K.
| |
Collapse
|
2
|
Xu J, Moyer D, Gagoski B, Iglesias JE, Grant PE, Golland P, Adalsteinsson E. NeSVoR: Implicit Neural Representation for Slice-to-Volume Reconstruction in MRI. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:1707-1719. [PMID: 37018704 PMCID: PMC10287191 DOI: 10.1109/tmi.2023.3236216] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/02/2023]
Abstract
Reconstructing 3D MR volumes from multiple motion-corrupted stacks of 2D slices has shown promise in imaging of moving subjects, e. g., fetal MRI. However, existing slice-to-volume reconstruction methods are time-consuming, especially when a high-resolution volume is desired. Moreover, they are still vulnerable to severe subject motion and when image artifacts are present in acquired slices. In this work, we present NeSVoR, a resolution-agnostic slice-to-volume reconstruction method, which models the underlying volume as a continuous function of spatial coordinates with implicit neural representation. To improve robustness to subject motion and other image artifacts, we adopt a continuous and comprehensive slice acquisition model that takes into account rigid inter-slice motion, point spread function, and bias fields. NeSVoR also estimates pixel-wise and slice-wise variances of image noise and enables removal of outliers during reconstruction and visualization of uncertainty. Extensive experiments are performed on both simulated and in vivo data to evaluate the proposed method. Results show that NeSVoR achieves state-of-the-art reconstruction quality while providing two to ten-fold acceleration in reconstruction times over the state-of-the-art algorithms.
Collapse
|
3
|
Qin Y, Hu J, Han J. A 2OURSR: Adaptive adjustment based real MRI super-resolution via opinion-unaware measurements. Comput Med Imaging Graph 2023; 107:102247. [PMID: 37224741 DOI: 10.1016/j.compmedimag.2023.102247] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2023] [Revised: 05/11/2023] [Accepted: 05/11/2023] [Indexed: 05/26/2023]
Abstract
High-quality and high-resolution magnetic resonance (MR) images can provide more details for diagnosis and analyses. Recently, MR images guided neurosurgery has become an emerging technique in clinics. Unlike other medical imaging techniques, it is impossible to achieve both real-time imaging and high image quality in MR imaging. The real-time performance is closely related to the nuclear magnetic equipment itself as well as the collection strategy of the k space data. Optimizing the imaging time cost via the corresponding algorithm is harder than enhancing image quality. Further, in reconstructing low-resolution and noise-rich MR images, getting relatively high-definition and resolution MR images as references are difficult or impossible. In addition, the existing methods are restricted in learning the controllable functions under the supervision of known degradation types and levels. As a result, severely bad results are inevitable when the modeling assumptions are far apart from the actual situation. To address these problems, we propose a novel adaptive adjustment method based on real MR images via opinion-unaware measurements for real super-resolution (A2OURSR). It can estimate the degree of blur and noise from the test image itself using two scores. These two scores can be considered pseudo labels to train the adaptive adjustable degradation estimation module. Then, the outputs of the above model are used as the inputs of the conditional network to tweak the generated results. Thus, the results can be automatically adjusted via the whole dynamic model. Extensive experimental results show that the proposed A2OURSR is superior to state-of-the-art methods on benchmarks quantitatively and visually.
Collapse
Affiliation(s)
- Yanding Qin
- College of Artificial Intelligence, Nankai University, Tianjin 300350, China; Institute of Intelligence Technology and Robotic Systems, Shenzhen Research Institute of Nankai University, Shenzhen 518083, China
| | - Jinbin Hu
- College of Artificial Intelligence, Nankai University, Tianjin 300350, China; Institute of Intelligence Technology and Robotic Systems, Shenzhen Research Institute of Nankai University, Shenzhen 518083, China
| | - Jianda Han
- College of Artificial Intelligence, Nankai University, Tianjin 300350, China; Institute of Intelligence Technology and Robotic Systems, Shenzhen Research Institute of Nankai University, Shenzhen 518083, China.
| |
Collapse
|
4
|
Gagoski B, Xu J, Wighton P, Tisdall MD, Frost R, Lo WC, Golland P, van der Kouwe A, Adalsteinsson E, Grant PE. Automated detection and reacquisition of motion-degraded images in fetal HASTE imaging at 3 T. Magn Reson Med 2022; 87:1914-1922. [PMID: 34888942 PMCID: PMC8810713 DOI: 10.1002/mrm.29106] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2021] [Revised: 10/19/2021] [Accepted: 11/12/2021] [Indexed: 11/07/2022]
Abstract
PURPOSE Fetal brain Magnetic Resonance Imaging suffers from unpredictable and unconstrained fetal motion that causes severe image artifacts even with half-Fourier single-shot fast spin echo (HASTE) readouts. This work presents the implementation of a closed-loop pipeline that automatically detects and reacquires HASTE images that were degraded by fetal motion without any human interaction. METHODS A convolutional neural network that performs automatic image quality assessment (IQA) was run on an external GPU-equipped computer that was connected to the internal network of the MRI scanner. The modified HASTE pulse sequence sent each image to the external computer, where the IQA convolutional neural network evaluated it, and then the IQA score was sent back to the sequence. At the end of the HASTE stack, the IQA scores from all the slices were sorted, and only slices with the lowest scores (corresponding to the slices with worst image quality) were reacquired. RESULTS The closed-loop HASTE acquisition framework was tested on 10 pregnant mothers, for a total of 73 acquisitions of our modified HASTE sequence. The IQA convolutional neural network, which was successfully employed by our modified sequence in real time, achieved an accuracy of 85.2% and area under the receiver operator characteristic of 0.899. CONCLUSION The proposed acquisition/reconstruction pipeline was shown to successfully identify and automatically reacquire only the motion degraded fetal brain HASTE slices in the prescribed stack. This minimizes the overall time spent on HASTE acquisitions by avoiding the need to repeat the entire stack if only few slices in the stack are motion-degraded.
Collapse
Affiliation(s)
- Borjan Gagoski
- Fetal Neonatal Neuroimaging and Developmental Science Center, Boston Children's Hospital, Boston, Massachusetts, USA
- Department of Radiology, Harvard Medical School, Boston, Massachusetts, USA
| | - Junshen Xu
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
| | - Paul Wighton
- Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, Massachusetts, USA
| | - M. Dylan Tisdall
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Robert Frost
- Department of Radiology, Harvard Medical School, Boston, Massachusetts, USA
- Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, Massachusetts, USA
| | - Wei-Ching Lo
- Siemens Medical Solutions USA, Inc, Charlestown, Massachusetts, USA
| | - Polina Golland
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
- Computer Science and Artificial Intelligence Laboratory (CSAIL), Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
| | - Andre van der Kouwe
- Department of Radiology, Harvard Medical School, Boston, Massachusetts, USA
- Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, Massachusetts, USA
| | - Elfar Adalsteinsson
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
- Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
| | - P. Ellen Grant
- Fetal Neonatal Neuroimaging and Developmental Science Center, Boston Children's Hospital, Boston, Massachusetts, USA
- Department of Radiology, Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
5
|
Largent A, Kapse K, Barnett SD, De Asis-Cruz J, Whitehead M, Murnick J, Zhao L, Andersen N, Quistorff J, Lopez C, Limperopoulos C. Image Quality Assessment of Fetal Brain MRI Using Multi-Instance Deep Learning Methods. J Magn Reson Imaging 2021; 54:818-829. [PMID: 33891778 DOI: 10.1002/jmri.27649] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2020] [Revised: 04/09/2021] [Accepted: 04/12/2021] [Indexed: 11/06/2022] Open
Abstract
BACKGROUND Due to random motion of fetuses and maternal respirations, image quality of fetal brain MRIs varies considerably. To address this issue, visual inspection of the images is performed during acquisition phase and after 3D-reconstruction, and the images are re-acquired if they are deemed to be of insufficient quality. However, this process is time-consuming and subjective. Multi-instance (MI) deep learning methods (DLMs) may perform this task automatically. PURPOSE To propose an MI count-based DLM (MI-CB-DLM), an MI vote-based DLM (MI-VB-DLM), and an MI feature-embedding DLM (MI-FE-DLM) for automatic assessment of 3D fetal-brain MR image quality. To quantify influence of fetal gestational age (GA) on DLM performance. STUDY TYPE Retrospective. SUBJECTS Two hundred and seventy-one MR exams from 211 fetuses (mean GA ± SD = 30.9 ± 5.5 weeks). FIELD STRENGTH/SEQUENCE T2 -weighted single-shot fast spin-echo acquired at 1.5 T. ASSESSMENT The T2 -weighted images were reconstructed in 3D. Then, two fetal neuroradiologists, a clinical neuroscientist, and a fetal MRI technician independently labeled the reconstructed images as 1 or 0 based on image quality (1 = high; 0 = low). These labels were fused and served as ground truth. The proposed DLMs were trained and evaluated using three repeated 10-fold cross-validations (training and validation sets of 244 and 27 scans). To quantify GA influence, this variable was included as an input of the DLMs. STATISTICAL TESTS DLM performance was evaluated using precision, recall, F-score, accuracy, and AUC values. RESULTS Precision, recall, F-score, accuracy, and AUC averaged over the three cross validations were 0.85 ± 0.01, 0.85 ± 0.01, 0.85 ± 0.01, 0.85 ± 0.01, 0.93 ± 0.01, for MI-CB-DLM (without GA); 0.75 ± 0.03, 0.75 ± 0.03, 0.75 ± 0.03, 0.75 ± 0.03, 0.81 ± 0.03, for MI-VB-DLM (without GA); 0.81 ± 0.01, 0.81 ± 0.01, 0.81 ± 0.01, 0.81 ± 0.01, 0.89 ± 0.01, for MI-FE-DLM (without GA); and 0.86 ± 0.01, 0.86 ± 0.01, 0.86 ± 0.01, 0.86 ± 0.01, 0.93 ± 0.01, for MI-CB-DLM with GA. DATA CONCLUSION MI-CB-DLM performed better than other DLMs. Including GA as an input of MI-CB-DLM improved its performance. MI-CB-DLM may potentially be used to objectively and rapidly assess fetal MR image quality. EVIDENCE LEVEL 4 TECHNICAL EFFICACY: Stage 3.
Collapse
Affiliation(s)
- Axel Largent
- Developing Brain Institute, Division of Diagnostic Imaging and Radiology, Children's National Hospital, Washington, District of Columbia, USA
| | - Kushal Kapse
- Developing Brain Institute, Division of Diagnostic Imaging and Radiology, Children's National Hospital, Washington, District of Columbia, USA
| | - Scott D Barnett
- Developing Brain Institute, Division of Diagnostic Imaging and Radiology, Children's National Hospital, Washington, District of Columbia, USA
| | - Josepheen De Asis-Cruz
- Developing Brain Institute, Division of Diagnostic Imaging and Radiology, Children's National Hospital, Washington, District of Columbia, USA
| | - Matthew Whitehead
- Developing Brain Institute, Division of Diagnostic Imaging and Radiology, Children's National Hospital, Washington, District of Columbia, USA.,Department of Neurology, Children's National Hospital, Washington, District of Columbia, USA
| | - Jonathan Murnick
- Developing Brain Institute, Division of Diagnostic Imaging and Radiology, Children's National Hospital, Washington, District of Columbia, USA
| | - Li Zhao
- Developing Brain Institute, Division of Diagnostic Imaging and Radiology, Children's National Hospital, Washington, District of Columbia, USA
| | - Nicole Andersen
- Developing Brain Institute, Division of Diagnostic Imaging and Radiology, Children's National Hospital, Washington, District of Columbia, USA
| | - Jessica Quistorff
- Developing Brain Institute, Division of Diagnostic Imaging and Radiology, Children's National Hospital, Washington, District of Columbia, USA
| | - Catherine Lopez
- Developing Brain Institute, Division of Diagnostic Imaging and Radiology, Children's National Hospital, Washington, District of Columbia, USA
| | - Catherine Limperopoulos
- Developing Brain Institute, Division of Diagnostic Imaging and Radiology, Children's National Hospital, Washington, District of Columbia, USA.,Department of Radiology, Pediatrics, George Washington University, Washington, District of Columbia, USA.,Neurology School of Medicine and Health Sciences, George Washington University, Washington, District of Columbia, USA
| |
Collapse
|