Estimating the 4D respiratory lung motion by spatiotemporal registration and super-resolution image reconstruction

A model that predicts a real-time tumour surface using intra-treatment skin surface and end-of-expiration and end-of-inhalation planning CT images

OBJECTIVES

To develop a mapping model between skin surface motion and internal tumour motion and deformation using end-of-exhalation (EOE) and end-of-inhalation (EOI) 3D CT images for tracking lung tumours during respiration.

METHODS

Before treatment, skin and tumour surfaces were segmented and reconstructed from the EOE and the EOI 3D CT images. A non-rigid registration algorithm was used to register the EOE skin and tumour surfaces to the EOI, resulting in a displacement vector field that was then used to construct a mapping model. During treatment, the EOE skin surface was registered to the real-time, yielding a real-time skin surface displacement vector field. Using the mapping model generated, the input of a real-time skin surface can be used to calculate the real-time tumour surface. The proposed method was validated with and without simulated noise on 4D CT images from 15 patients at Léon Bérard Cancer Center and the 4D-lung dataset.

RESULTS

The average centre position error, dice similarity coefficient (DSC), 95%-Hausdorff distance and mean distance to agreement of the tumour surfaces were 1.29 mm, 0.924, 2.76 mm, and 1.13 mm without simulated noise, respectively. With simulated noise, these values were 1.33 mm, 0.920, 2.79 mm, and 1.15 mm, respectively.

CONCLUSIONS

A patient-specific model was proposed and validated that was constructed using only EOE and EOI 3D CT images and real-time skin surface images to predict internal tumour motion and deformation during respiratory motion.

ADVANCES IN KNOWLEDGE

The proposed method achieves comparable accuracy to state-of-the-art methods with fewer pre-treatment planning CT images, which holds potential for application in precise image-guided radiation therapy.

End-to-End Deep Learning of Non-rigid Groupwise Registration and Reconstruction of Dynamic MRI

Temporal correlation has been exploited for accelerated dynamic MRI reconstruction. Some methods have modeled inter-frame motion into the reconstruction process to produce temporally aligned image series and higher reconstruction quality. However, traditional motion-compensated approaches requiring iterative optimization of registration and reconstruction are time-consuming, while most deep learning-based methods neglect motion in the reconstruction process. We propose an unrolled deep learning framework with each iteration consisting of a groupwise diffeomorphic registration network (GRN) and a motion-augmented reconstruction network. Specifically, the whole dynamic sequence is registered at once to an implicit template which is used to generate a new set of dynamic images to efficiently exploit the full temporal information of the acquired data via the GRN. The generated dynamic sequence is then incorporated into the reconstruction network to augment the reconstruction performance. The registration and reconstruction networks are optimized in an end-to-end fashion for simultaneous motion estimation and reconstruction of dynamic images. The effectiveness of the proposed method is validated in highly accelerated cardiac cine MRI by comparing with other state-of-the-art approaches.

Vector-field dynamic x-ray (VF-DXR) using optical flow method in patients with chronic obstructive pulmonary disease

BACKGROUND

We assessed the difference in lung motion during inspiration/expiration between chronic obstructive pulmonary disease (COPD) patients and healthy volunteers using vector-field dynamic x-ray (VF-DXR) with optical flow method (OFM).

METHODS

We enrolled 36 COPD patients and 47 healthy volunteers, classified according to pulmonary function into: normal, COPD mild, and COPD severe. Contrast gradient was obtained from sequential dynamic x-ray (DXR) and converted to motion vector using OFM. VF-DXR images were created by projection of the vertical component of lung motion vectors onto DXR images. The maximum magnitude of lung motion vectors in tidal inspiration/expiration, forced inspiration/expiration were selected and defined as lung motion velocity (LMV). Correlations between LMV with demographics and pulmonary function and differences in LMV between COPD patients and healthy volunteers were investigated.

RESULTS

Negative correlations were confirmed between LMV and % forced expiratory volume in one second (%FEV1) in the tidal inspiration in the right lung (Spearman's rank correlation coefficient, rs = -0.47, p < 0.001) and the left lung (rs = -0.32, p = 0.033). A positive correlation between LMV and %FEV1 in the tidal expiration was observed only in the right lung (rs = 0.25, p = 0.024). LMVs among normal, COPD mild and COPD severe groups were different in the tidal respiration. COPD mild group showed a significantly larger magnitude of LMV compared with the normal group.

CONCLUSIONS

In the tidal inspiration, the lung parenchyma moved faster in COPD patients compared with healthy volunteers. VF-DXR was feasible for the assessment of lung parenchyma using LMV.

Few-shot learning for deformable image registration in 4DCT images

OBJECTIVES

To develop a rapid and accurate 4D deformable image registration (DIR) approach for online adaptive radiotherapy.

METHODS

We propose a deep learning (DL)-based few-shot registration network (FR-Net) to generate deformation vector fields from each respiratory phase to an implicit reference image, thereby mitigating the bias introduced by the selection of reference images. The proposed FR-Net is pretrained with limited unlabeled 4D data and further optimized by maximizing the intensity similarity of one specific four-dimensional computed tomography (4DCT) scan. Because of the learning ability of DL models, the few-shot learning strategy facilitates the generalization of the model to other 4D data sets and the acceleration of the optimization process.

RESULTS

The proposed FR-Net is evaluated for 4D groupwise and 3D pairwise registration on thoracic 4DCT data sets DIR-Lab and POPI. FR-Net displays an averaged target registration error of 1.48 mm and 1.16 mm between the maximum inhalation and exhalation phases in the 4DCT of DIR-Lab and POPI, respectively, with approximately 2 min required to optimize one 4DCT. Overall, FR-Net outperforms state-of-the-art methods in terms of registration accuracy and exhibits a low computational time.

CONCLUSION

We develop a few-shot groupwise DIR algorithm for 4DCT images. The promising registration performance and computational efficiency demonstrate the prospective applications of this approach in registration tasks for online adaptive radiotherapy.

ADVANCES IN KNOWLEDGE

This work exploits DL models to solve the optimization problem in registering 4DCT scans while combining groupwise registration and few-shot learning strategy to solve the problem of consuming computational time and inferior registration accuracy.

Lung Respiratory Motion Estimation Based on Fast Kalman Filtering and 4D CT Image Registration

Respiratory motion estimation is an important part in image-guided radiation therapy and clinical diagnosis. However, most of the respiratory motion estimation methods rely on indirect measurements of external breathing indicators, which will not only introduce great estimation errors, but also bring invasive injury for patients. In this paper, we propose a method of lung respiratory motion estimation based on fast Kalman filtering and 4D CT image registration (LRME-4DCT). In order to perform dynamic motion estimation for continuous phases, a motion estimation model is constructed by combining two kinds of GPU-accelerated 4D CT image registration methods with fast Kalman filtering method. To address the high computational requirements of 4D CT image sequences, a multi-level processing strategy is adopted in the 4D CT image registration methods, and respiratory motion states are predicted from three independent directions. In the DIR-lab dataset and POPI dataset with 4D CT images, the average target registration error (TRE) of the LRME-4DCT method can reach 0.91 mm and 0.85 mm respectively. Compared with traditional estimation methods based on pair-wise image registration, the proposed LRME-4DCT method can estimate the physiological respiratory motion more accurately and quickly. Our proposed LRME-4DCT method fully meets the practical clinical requirements for rapid dynamic estimation of lung respiratory motion.

GroupRegNet: a groupwise one-shot deep learning-based 4D image registration method

Accurate deformable four-dimensional (4D) (three-dimensional in space and time) medical images registration is essential in a variety of medical applications. Deep learning-based methods have recently gained popularity in this area for the significantly lower inference time. However, they suffer from drawbacks of non-optimal accuracy and the requirement of a large amount of training data. A new method named GroupRegNet is proposed to address both limitations. The deformation fields to warp all images in the group into a common template is obtained through one-shot learning. The use of the implicit template reduces bias and accumulated error associated with the specified reference image. The one-shot learning strategy is similar to the conventional iterative optimization method but the motion model and parameters are replaced with a convolutional neural network and the weights of the network. GroupRegNet also features a simpler network design and a more straightforward registration process, which eliminates the need to break up the input image into patches. The proposed method was quantitatively evaluated on two public respiratory-binned 4D-computed tomography datasets. The results suggest that GroupRegNet outperforms the latest published deep learning-based methods and is comparable to the top conventional method pTVreg. To facilitate future research, the source code is available at https://github.com/vincentme/GroupRegNet.

One-Shot Learning for Deformable Medical Image Registration and Periodic Motion Tracking

Deformable image registration is a very important field of research in medical imaging. Recently multiple deep learning approaches were published in this area showing promising results. However, drawbacks of deep learning methods are the need for a large amount of training datasets and their inability to register unseen images different from the training datasets. One shot learning comes without the need of large training datasets and has already been proven to be applicable to 3D data. In this work we present a one shot registration approach for periodic motion tracking in 3D and 4D datasets. When applied to a 3D dataset the algorithm calculates the inverse of the registration vector field simultaneously. For registration we employed a U-Net combined with a coarse to fine approach and a differential spatial transformer module. The algorithm was thoroughly tested with multiple 4D and 3D datasets publicly available. The results show that the presented approach is able to track periodic motion and to yield a competitive registration accuracy. Possible applications are the use as a stand-alone algorithm for 3D and 4D motion tracking or in the beginning of studies until enough datasets for a separate training phase are available.

Discrete Heat Kernel Smoothing in Irregular Image Domains

We present the discrete version of heat kernel smoothing on graph data structure. The method is used to smooth data in an irregularly shaped domains in 3D images. New statistical properties of heat kernel smoothing are derived. As an application, we show how to filter out noisy data in the lung blood vessel trees obtained from computed tomography. The method can be further used in representing the complex vessel trees parametrically as a linear combination of basis functions and extracting the skeleton representation of the trees.

Contour propagation for lung tumor delineation in 4D-CT using tensor-product surface of uniform and non-uniform closed cubic B-splines

A robust contour propagation method is proposed to help physicians delineate lung tumors on all phase images of four-dimensional computed tomography (4D-CT) by only manually delineating the contours on a reference phase. The proposed method models the trajectory surface swept by a contour in a respiratory cycle as a tensor-product surface of two closed cubic B-spline curves: a non-uniform B-spline curve which models the contour and a uniform B-spline curve which models the trajectory of a point on the contour. The surface is treated as a deformable entity, and is optimized from an initial surface by moving its control vertices such that the sum of the intensity similarities between the sampling points on the manually delineated contour and their corresponding ones on different phases is maximized. The initial surface is constructed by fitting the manually delineated contour on the reference phase with a closed B-spline curve. In this way, the proposed method can focus the registration on the contour instead of the entire image to prevent the deformation of the contour from being smoothed by its surrounding tissues, and greatly reduce the time consumption while keeping the accuracy of the contour propagation as well as the temporal consistency of the estimated respiratory motions across all phases in 4D-CT. Eighteen 4D-CT cases with 235 gross tumor volume (GTV) contours on the maximal inhale phase and 209 GTV contours on the maximal exhale phase are manually delineated slice by slice. The maximal inhale phase is used as the reference phase, which provides the initial contours. On the maximal exhale phase, the Jaccard similarity coefficient between the propagated GTV and the manually delineated GTV is 0.881 [Formula: see text] 0.026, and the Hausdorff distance is 3.07 [Formula: see text] 1.08 mm. The time for propagating the GTV to all phases is 5.55 [Formula: see text] 6.21 min. The results are better than those of the fast adaptive stochastic gradient descent B-spline method, the 3D  +  t B-spline method and the diffeomorphic demons method. The proposed method is useful for helping physicians delineate target volumes efficiently and accurately.

Diffeomorphic Multi-Frame Non-Rigid Registration of Cell Nuclei in 2D and 3D Live Cell Images

To gain a better understanding of cellular and molecular processes, it is important to quantitatively analyze the motion of subcellular particles in live cell microscopy image sequences. Since, generally, the subcellular particles move and cell nuclei move as well as deform, it is important to decouple the movement of particles from that of the cell nuclei using non-rigid registration methods. We have developed a diffeomorphic multi-frame approach for non-rigid registration of cell nuclei in 2D and 3D live cell fluorescence microscopy images. Our non-rigid registration approach is based on local optic flow estimation, exploits information from multiple consecutive image frames, and determines diffeomorphic transformations in the log-domain, which allows efficient computation of the inverse transformations. To register single images of an image sequence to a reference image, we use a temporally weighted mean image, which is constructed based on inverse transformations and multiple consecutive frames. Using multiple consecutive frames improves the registration accuracy compared to pairwise registration, and using a temporally weighted mean image significantly reduces the computation time compared with previous work. In addition, we use a flow boundary preserving method for regularization of computed deformation vector fields, which prevents from over-smoothing compared to standard Gaussian filtering. Our approach has been successfully applied to 2D and 3D synthetic as well as real live cell microscopy image sequences, and an experimental comparison with non-rigid pairwise, multi-frame, and temporal groupwise registration has been carried out.

Lung motion estimation by robust point matching and spatiotemporal tracking for 4D CT

We propose a deformable registration approach to estimate patient-specific lung motion during free breathing for four-dimensional (4D) computed tomography (CT) based on point matching and tracking between images in different phases. First, a robust point matching (RPM) algorithm coarsely aligns the source phase image onto all other target phase images of 4D CT. Scale-invariant feature transform (SIFT) is introduced into the cost function in order to accelerate and stabilize the convergence of the point matching. Next, the temporal consistency of the estimated lung motion model is preserved by fitting the trajectories of the points in the respiratory phase using L₁ norm regularization. Then, the fitted positions of a point along the trajectory are used as the initial positions for the point tracking. Spatial mean-shift iteration is employed to track points in all phase images. The tracked positions in all phases are used to perform RPM again. These steps are repeated until the number of updated points is smaller than a given threshold σ. With this method, the correspondence between the source phase image and other target phase image is established more accurately. Trajectory fitting ensures the estimated trajectory does not fluctuate violently. We evaluated our method by using the public DIR-lab, POPI-model, CREATIS and COPDgene lung datasets. In the experimental results, the proposed method achieved satisfied accuracy for image registration. Our method also preserved the topology of the deformation fields well for image registration with large deformation.

Lung motion estimation using dynamic point shifting: An innovative model based on a robust point matching algorithm

PURPOSE

Image-guided radiotherapy is an advanced 4D radiotherapy technique that has been developed in recent years. However, respiratory motion causes significant uncertainties in image-guided radiotherapy procedures. To address these issues, an innovative lung motion estimation model based on a robust point matching is proposed in this paper.

METHODS

An innovative robust point matching algorithm using dynamic point shifting is proposed to estimate patient-specific lung motion during free breathing from 4D computed tomography data. The correspondence of the landmark points is determined from the Euclidean distance between the landmark points and the similarity between the local images that are centered at points at the same time. To ensure that the points in the source image correspond to the points in the target image during other phases, the virtual target points are first created and shifted based on the similarity between the local image centered at the source point and the local image centered at the virtual target point. Second, the target points are shifted by the constrained inverse function mapping the target points to the virtual target points. The source point set and shifted target point set are used to estimate the transformation function between the source image and target image.

RESULTS

The performances of the authors' method are evaluated on two publicly available DIR-lab and POPI-model lung datasets. For computing target registration errors on 750 landmark points in six phases of the DIR-lab dataset and 37 landmark points in ten phases of the POPI-model dataset, the mean and standard deviation by the authors' method are 1.11 and 1.11 mm, but they are 2.33 and 2.32 mm without considering image intensity, and 1.17 and 1.19 mm with sliding conditions. For the two phases of maximum inhalation and maximum exhalation in the DIR-lab dataset with 300 landmark points of each case, the mean and standard deviation of target registration errors on the 3000 landmark points of ten cases by the authors' method are 1.21 and 1.04 mm. In the EMPIRE10 lung registration challenge, the authors' method ranks 24 of 39. According to the index of the maximum shear stretch, the authors' method is also efficient to describe the discontinuous motion at the lung boundaries.

CONCLUSIONS

By establishing the correspondence of the landmark points in the source phase and the other target phases combining shape matching and image intensity matching together, the mismatching issue in the robust point matching algorithm is adequately addressed. The target registration errors are statistically reduced by shifting the virtual target points and target points. The authors' method with consideration of sliding conditions can effectively estimate the discontinuous motion, and the estimated motion is natural. The primary limitation of the proposed method is that the temporal constraints of the trajectories of voxels are not introduced into the motion model. However, the proposed method provides satisfactory motion information, which results in precise tumor coverage by the radiation dose during radiotherapy.

LROC Investigation of Three Strategies for Reducing the Impact of Respiratory Motion on the Detection of Solitary Pulmonary Nodules in SPECT

The objective of this investigation was to determine the effectiveness of three motion reducing strategies in diminishing the degrading impact of respiratory motion on the detection of small solitary pulmonary nodules (SPN) in single photon emission computed tomographic (SPECT) imaging in comparison to a standard clinical acquisition and the ideal case of imaging in the absence of respiratory motion. To do this non-uniform rational B-spline cardiac-torso (NCAT) phantoms based on human-volunteer CT studies were generated spanning the respiratory cycle for a normal background distribution of Tc-99m NeoTect. Similarly, spherical phantoms of 1.0 cm diameter were generated to model small SPN for each of 150 uniquely located sites within the lungs whose respiratory motion was based on the motion of normal structures in the volunteer CT studies. The SIMIND Monte Carlo program was used to produce SPECT projection data from these. Normal and single-lesion containing SPECT projection sets with a clinically realistic Poisson noise level were created for the cases of: 1) the end-expiration (EE) frame with all counts, 2) respiration-averaged motion with all counts, 3) one-fourth of the 32 frames centered around EE (Quarter-Binning), 4) one-half of the 32 frames centered around EE (Half-Binning), and 5) eight temporally binned frames spanning the respiratory cycle. Each of the sets of combined projection data were reconstructed with RBI-EM with system spatial-resolution compensation (RC). Based on the known motion for each of the 150 different lesions, the reconstructed volumes of respiratory bins were shifted so as to superimpose the locations of the SPN onto that in the first bin (Reconstruct and Shift). Five human-observers performed localization receiver operating characteristics (LROC) studies of SPN detection. The observer results were analyzed for statistical significance differences in SPN detection accuracy among the three correction strategies, the standard acquisition, and the ideal case of the absence of respiratory motion. Our human-observer LROC determined that Quarter-Binning and Half-Binning strategies resulted in SPN detection accuracy statistically significantly below (P < 0.05) that of standard clinical acquisition, whereas the Reconstruct and Shift strategy resulted in a detection accuracy not statistically significantly different from that of the ideal case. This investigation demonstrates that tumor detection based on acquisitions associated with less than all the counts which could potentially be employed may result in poorer detection despite limiting the motion of the lesion. The Reconstruct and Shift method results in tumor detection that is equivalent to ideal motion correction.

Tensor-based dictionary learning for dynamic tomographic reconstruction

In dynamic computed tomography (CT) reconstruction, the data acquisition speed limits the spatio-temporal resolution. Recently, compressed sensing theory has been instrumental in improving CT reconstruction from far few-view projections. In this paper, we present an adaptive method to train a tensor-based spatio-temporal dictionary for sparse representation of an image sequence during the reconstruction process. The correlations among atoms and across phases are considered to capture the characteristics of an object. The reconstruction problem is solved by the alternating direction method of multipliers. To recover fine or sharp structures such as edges, the nonlocal total variation is incorporated into the algorithmic framework. Preclinical examples including a sheep lung perfusion study and a dynamic mouse cardiac imaging demonstrate that the proposed approach outperforms the vectorized dictionary-based CT reconstruction in the case of few-view reconstruction.