Deep learning-based motion tracking using ultrasound images

Artificial intelligence-based motion tracking in cancer radiotherapy: A review

Radiotherapy aims to deliver a prescribed dose to the tumor while sparing neighboring organs at risk (OARs). Increasingly complex treatment techniques such as volumetric modulated arc therapy (VMAT), stereotactic radiosurgery (SRS), stereotactic body radiotherapy (SBRT), and proton therapy have been developed to deliver doses more precisely to the target. While such technologies have improved dose delivery, the implementation of intra-fraction motion management to verify tumor position at the time of treatment has become increasingly relevant. Artificial intelligence (AI) has recently demonstrated great potential for real-time tracking of tumors during treatment. However, AI-based motion management faces several challenges, including bias in training data, poor transparency, difficult data collection, complex workflows and quality assurance, and limited sample sizes. This review presents the AI algorithms used for chest, abdomen, and pelvic tumor motion management/tracking for radiotherapy and provides a literature summary on the topic. We will also discuss the limitations of these AI-based studies and propose potential improvements.

Artificial intelligence for treatment delivery: image-guided radiotherapy

Radiation therapy (RT) is a highly digitized field relying heavily on computational methods and, as such, has a high affinity for the automation potential afforded by modern artificial intelligence (AI). This is particularly relevant where imaging is concerned and is especially so during image-guided RT (IGRT). With the advent of online adaptive RT (ART) workflows at magnetic resonance (MR) linear accelerators (linacs) and at cone-beam computed tomography (CBCT) linacs, the need for automation is further increased. AI as applied to modern IGRT is thus one area of RT where we can expect important developments in the near future. In this review article, after outlining modern IGRT and online ART workflows, we cover the role of AI in CBCT and MRI correction for dose calculation, auto-segmentation on IGRT imaging, motion management, and response assessment based on in-room imaging.

Landmark tracking in liver US images using cascade convolutional neural networks with long short-term memory

Accurate tracking of anatomic landmarks is critical for motion management in liver radiation therapy. Ultrasound (US) is a safe, low-cost technology that is broadly available and offer real-time imaging capability. This study proposed a deep learning-based tracking method for the US image-guided radiation therapy. The proposed cascade deep learning model is composed of an attention network, a mask region-based convolutional neural network (mask R-CNN), and a long short-term memory (LSTM) network. The attention network learns a mapping from an US image to a suspected area of landmark motion in order to reduce the search region. The mask R-CNN then produces multiple region-of-interest proposals in the reduced region and identifies the proposed landmark via three network heads: bounding box regression, proposal classification, and landmark segmentation. The LSTM network models the temporal relationship among the successive image frames for bounding box regression and proposal classification. To consolidate the final proposal, a selection method is designed according to the similarities between sequential frames. The proposed method was tested on the liver US tracking datasets used in the medical image computing and computer assisted interventions 2015 challenges, where the landmarks were annotated by three experienced observers to obtain their mean positions. Five-fold cross validation on the 24 given US sequences with ground truths shows that the mean tracking error for all landmarks is 0.65 ± 0.56 mm, and the errors of all landmarks are within 2 mm. We further tested the proposed model on 69 landmarks from the testing dataset that have the similar image pattern with the training pattern, resulting in a mean tracking error of 0.94 ± 0.83 mm. The proposed deep-learning model was implemented on a graphics processing unit (GPU), tracking 47-81 frames s-1. Our experimental results have demonstrated the feasibility and accuracy of our proposed method in tracking liver anatomic landmarks using US images, providing a potential solution for real-time liver tracking for active motion management during radiation therapy.

Robust and Realtime Large Deformation Ultrasound Registration Using End-to-End Differentiable Displacement Optimisation

Image registration for temporal ultrasound sequences can be very beneficial for image-guided diagnostics and interventions. Cooperative human-machine systems that enable seamless assistance for both inexperienced and expert users during ultrasound examinations rely on robust, realtime motion estimation. Yet rapid and irregular motion patterns, varying image contrast and domain shifts in imaging devices pose a severe challenge to conventional realtime registration approaches. While learning-based registration networks have the promise of abstracting relevant features and delivering very fast inference times, they come at the potential risk of limited generalisation and robustness for unseen data; in particular, when trained with limited supervision. In this work, we demonstrate that these issues can be overcome by using end-to-end differentiable displacement optimisation. Our method involves a trainable feature backbone, a correlation layer that evaluates a large range of displacement options simultaneously and a differentiable regularisation module that ensures smooth and plausible deformation. In extensive experiments on public and private ultrasound datasets with very sparse ground truth annotation the method showed better generalisation abilities and overall accuracy than a VoxelMorph network with the same feature backbone, while being two times faster at inference.

Co-attention spatial transformer network for unsupervised motion tracking and cardiac strain analysis in 3D echocardiography

Myocardial ischemia/infarction causes wall-motion abnormalities in the left ventricle. Therefore, reliable motion estimation and strain analysis using 3D+time echocardiography for localization and characterization of myocardial injury is valuable for early detection and targeted interventions. Previous unsupervised cardiac motion tracking methods rely on heavily-weighted regularization functions to smooth out the noisy displacement fields in echocardiography. In this work, we present a Co-Attention Spatial Transformer Network (STN) for improved motion tracking and strain analysis in 3D echocardiography. Co-Attention STN aims to extract inter-frame dependent features between frames to improve the motion tracking in otherwise noisy 3D echocardiography images. We also propose a novel temporal constraint to further regularize the motion field to produce smooth and realistic cardiac displacement paths over time without prior assumptions on cardiac motion. Our experimental results on both synthetic and in vivo 3D echocardiography datasets demonstrate that our Co-Attention STN provides superior performance compared to existing methods. Strain analysis from Co-Attention STNs also correspond well with the matched SPECT perfusion maps, demonstrating the clinical utility for using 3D echocardiography for infarct localization.

Therapeutic Ultrasound Effects on Human Induced Pluripotent Stem Cell Cardiomyocytes Measured Optically and with Spectral Ultrasound

To the best of our knowledge, therapeutic ultrasound (TUS) is thus far an unexplored means of delivering mechanical stimulation to cardiomyocyte cultures, which is necessary to engineer a more mature cardiomyocyte phenotype in vitro. Spectral ultrasound (SUS) may provide a way to non-invasively, non-disruptively and inexpensively monitor growth and change in cell cultures over long periods. Compared with other measurement methods, SUS as an acoustic measurement tool will not be affected by an acoustic therapy, unlike electrical measurement methods, in which motion caused by acoustic therapy can affect measurements. Further SUS has the potential to provide functional as well as morphological information in cell cultures. Human induced pluripotent stem cell cardiomyocytes (iPS-CMs) were imaged with calcium fluorescence microscopy while TUS was being applied. TUS was applied at 600 kHz and 1, 3.4 and 6 W/cm² for a continuous 1 s pulse. Measures of the instantaneous beat frequency, repolarization rate and calcium spike amplitude were calculated from the fluorescence data. At 600 kHz, TUS at 1 and 6 W/cm² had significant effects on the shortening of both the repolarization rate and instantaneous beat rate of the iPS-CMs (p < 0.05), while TUS at 3.4 and 6 W/cm² had significant effects on the shortening of the calcium spike amplitude (p < 0.05). Three SUS measures and one gray-level measure were captured from the iPS-CM monolayers while they were simultaneously being imaged with calcium-labeled confocal microscopy. The gray-level measure performed the best of all SUS measures; however, it was not reliable enough to produce a consistent determination of the beat rate of the cell. Finally, SUS measures were captured using three different transducers while simultaneously applying TUS. A center-of-mass (COM) measure calculated from the wavelet transform scalogram of the time-averaged radiofrequency data revealed that SUS was able to detect a change in the frequency content of the reflected ultrasound at 1 and 6 W/cm² before and after ultrasound application (p < 0.05), showing promise for the ability of SUS to measure changes in the beating behavior of iPS-CMs. Overall, SUS is promising as a method for constant monitoring of dynamic cell and tissue culture and growth.