Deformable registration of the inflated and deflated lung in cone-beam CT-guided thoracic surgery: initial investigation of a combined model- and image-driven approach

Volumetric tumor tracking from a single cone-beam X-ray projection image enabled by deep learning

Radiotherapy serves as a pivotal treatment modality for malignant tumors. However, the accuracy of radiotherapy is significantly compromised due to respiratory-induced fluctuations in the size, shape, and position of the tumor. To address this challenge, we introduce a deep learning-anchored, volumetric tumor tracking methodology that employs single-angle X-ray projection images. This process involves aligning the intraoperative two-dimensional (2D) X-ray images with the pre-treatment three-dimensional (3D) planning Computed Tomography (CT) scans, enabling the extraction of the 3D tumor position and segmentation. Prior to therapy, a bespoke patient-specific tumor tracking model is formulated, leveraging a hybrid data augmentation, style correction, and registration network to create a mapping from single-angle 2D X-ray images to the corresponding 3D tumors. During the treatment phase, real-time X-ray images are fed into the trained model, producing the respective 3D tumor positioning. Rigorous validation conducted on actual patient lung data and lung phantoms attests to the high localization precision of our method at lowered radiation doses, thus heralding promising strides towards enhancing the precision of radiotherapy.

Deep learning-based lung image registration: A review

Lung image registration can effectively describe the relative motion of lung tissues, thereby helping to solve series problems in clinical applications. Since the lungs are soft and fairly passive organs, they are influenced by respiration and heartbeat, resulting in discontinuity of lung motion and large deformation of anatomic features. This poses great challenges for accurate registration of lung image and its applications. The recent application of deep learning (DL) methods in the field of medical image registration has brought promising results. However, a versatile registration framework has not yet emerged due to diverse challenges of registration for different regions of interest (ROI). DL-based image registration methods used for other ROI cannot achieve satisfactory results in lungs. In addition, there are few review articles available on DL-based lung image registration. In this review, the development of conventional methods for lung image registration is briefly described and a more comprehensive survey of DL-based methods for lung image registration is illustrated. The DL-based methods are classified according to different supervision types, including fully-supervised, weakly-supervised and unsupervised. The contributions of researchers in addressing various challenges are described, as well as the limitations of these approaches. This review also presents a comprehensive statistical analysis of the cited papers in terms of evaluation metrics and loss functions. In addition, publicly available datasets for lung image registration are also summarized. Finally, the remaining challenges and potential trends in DL-based lung image registration are discussed.

Deformation analysis of surface and bronchial structures in intraoperative pneumothorax using deformable mesh registration

The positions of nodules can change because of intraoperative lung deflation, and the modeling of pneumothorax-associated deformation remains a challenging issue for intraoperative tumor localization. In this study, we introduce spatial and geometric analysis methods for inflated/deflated lungs and discuss heterogeneity in pneumothorax-associated lung deformation. Contrast-enhanced CT images simulating intraoperative conditions were acquired from live Beagle dogs. The images contain the overall shape of the lungs, including all lobes and internal bronchial structures, and were analyzed to provide a statistical deformation model that could be used as prior knowledge to predict pneumothorax. To address the difficulties of mapping pneumothorax CT images with topological changes and CT intensity shifts, we designed deformable mesh registration techniques for mixed data structures including the lobe surfaces and the bronchial centerlines. Three global-to-local registration steps were performed under the constraint that the deformation was spatially continuous and smooth, while matching visible bronchial tree structures as much as possible. The developed framework achieved stable registration with a Hausdorff distance of less than 1 mm and a target registration error of less than 5 mm, and visualized deformation fields that demonstrate per-lobe contractions and rotations with high variability between subjects. The deformation analysis results show that the strain of lung parenchyma was 35% higher than that of bronchi, and that deformation in the deflated lung is heterogeneous.

A hybrid, image-based and biomechanics-based registration approach to markerless intraoperative nodule localization during video-assisted thoracoscopic surgery

The resection of small, low-dense or deep lung nodules during video-assisted thoracoscopic surgery (VATS) is surgically challenging. Nodule localization methods in clinical practice typically rely on the preoperative placement of markers, which may lead to clinical complications. We propose a markerless lung nodule localization framework for VATS based on a hybrid method combining intraoperative cone-beam CT (CBCT) imaging, free-form deformation image registration, and a poroelastic lung model with allowance for air evacuation. The difficult problem of estimating intraoperative lung deformations is decomposed into two more tractable sub-problems: (i) estimating the deformation due the change of patient pose from preoperative CT (supine) to intraoperative CBCT (lateral decubitus); and (ii) estimating the pneumothorax deformation, i.e. a collapse of the lung within the thoracic cage. We were able to demonstrate the feasibility of our localization framework with a retrospective validation study on 5 VATS clinical cases. Average initial errors in the range of 22 to 38 mm were reduced to the range of 4 to 14 mm, corresponding to an error correction in the range of 63 to 85%. To our knowledge, this is the first markerless lung deformation compensation method dedicated to VATS and validated on actual clinical data.

Model-based registration for pneumothorax deformation analysis using intraoperative cone-beam CT images

Because the lung deforms during surgery because of pneumothorax, it is important to be able to track the location of a tumor. Deformation of the whole lung can be estimated using intraoperative cone-beam CT (CBCT) images. In this study, we used deformable mesh registration methods for paired CBCT images in the inflated and deflated states, and analyzed their deformation. We proposed a deformable mesh registration framework for deformations of partial organ shapes involving large deformation and rotation. Experimental results showed that the proposed methods reduced errors in point-to-point correspondence. As a result of registration using surgical clips placed on the lung surface during imaging, it was confirmed that an average error of 3.9 mm occurred in eight cases. The result of analysis showed that both tissue rotation and contraction had large effects on displacement.

Devising the guidelines: the techniques of pulmonary nodule localization in uniportal video-assisted thoracic surgery-hybrid operating room in the future

Pulmonary nodules beneath the pleura can be hard to visualize or palpate, especially during the uniportal thoracoscopic surgery. Conventionally, thoracic surgeons would use adjuvant modalities to localize the lesion preoperatively, of which computed tomography-guided hookwire implantation has been adopted most widely due to its feasibility and high success rate. However, procedure-associated complications such as pneumothorax and wire dislodgement can cause patient discomfort or localization failure. Occasionally more healthy tissue is resected than needed to guarantee the lesion is removed and with an adequate margin. A thoracotomy is necessary for specific scenario. With the development of imaging technology, it is now possible to replace the traditional workflow carried out in the radiology suite by centralizing the hookwire placement and uniportal minimally-invasive pulmonary resection inside the hybrid theater which equipped with advanced imaging devices. Theoretically, the advanced intra-operative imaging-guided techniques help to precisely locate and resection pulmonary lesion in a potentially tissue-sparing and quick paradigm.

Surface deformation analysis of collapsed lungs using model-based shape matching

Purpose

To facilitate intraoperative localization of lung nodules, this study used model-based shape matching techniques to analyze the inter-subject three-dimensional surface deformation induced by pneumothorax. Methods: Contrast- enhanced computed tomography (CT) images of the left lungs of 11 live beagle dogs were acquired at two bronchial pressures (14 and 2 cm\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\,\hbox {H}_2\hbox {O}$$\end{document}H2O). To address shape matching problems for largely deformed lung images with pixel intensity shift, a complete Laplacian-based shape matching solution that optimizes the differential displacement field was introduced.

Results

Experiments were performed to confirm the methods’ registration accuracy using CT images of lungs. Shape similarity and target displacement errors in the registered models were improved compared with those from existing shape matching methods. Spatial displacement of the whole lung’s surface was visualized with an average error of within 5 mm.

Conclusion

The proposed methods address problems with the matching of surfaces with large curvatures and deformations and achieved smaller registration errors than existing shape matching methods, even at the tip and ridge regions. The findings and inter-subject statistical representation are directly available for further research on pneumothorax deformation modeling.

Electronic supplementary material

The online version of this article (10.1007/s11548-019-02013-0) contains supplementary material, which is available to authorized users.

Devices for image-guided lung interventions: State-of-the-art review

Lung cancer is the leading cause of cancer-related death. According to the American Cancer Society, there were an estimated 222,500 new cases of lung cancer and 155,870 deaths from lung cancer in the United States in 2017. Accurate localization in lung interventions is one of the keys to reducing the death rate from lung cancer. In this study, a total of 217 publications from 2006 to 2017 about designs of medical devices for localization in lung interventions were screened, shortlisted, and categorized by localization principle and reviewed for functionality. Each study was analyzed for engineering characteristics and clinical significance. Research regarding interventional imaging equipment, navigation systems, and surgical devices was reviewed, and both research prototypes and commercial products were discussed. Finally, the future directions and existing challenges were summarized, including real-time intra-procedure guidance, accuracy of localization, clinical application, clinical adoptability, and clinical regulatory issues.

3D Printing Endobronchial Models for Surgical Training and Simulation

Lung cancer is the leading cause of cancer-related deaths. Many methods and devices help acquire more accurate clinical and localization information during lung interventions and may impact the death rate for lung cancer. However, there is a learning curve for operating these tools due to the complex structure of the airway. In this study, we first discuss the creation of a lung phantom model from medical images, which is followed by a comparison of 3D printing in terms of quality and consistency. Two tests were conducted to test the performance of the developed phantom, which was designed for training simulations of the target and ablation processes in endochonchial interventions. The target test was conducted through an electromagnetic tracking catheter with navigation software. An ablation catheter with a recently developed thermochromic ablation gel conducted the ablation test. The results of two tests show that the phantom was very useful for target and ablation simulation. In addition, the thermochromic gel allowed doctors to visualize the ablation zone. Many lung interventions may benefit from custom training or accuracy with the proposed low-cost and patient-specific phantom.

Hybrid Theater and Uniportal Video-Assisted Thoracic Surgery: The Perfect Match for Lung Nodule Localization

Cone-beam computed tomography provides unparalleled real-time imaging of the patient within the hybrid theater, which can be used for simultaneously diagnosing and localizing small pulmonary lesions for resection. Hybrid theater can guide more precise placement of electromagnetic navigation bronchoscopy tools, thereby increasing the diagnostic yield in biopsy procedures while reducing diffusion artifact in dye marking for nodule localization. Furthermore, hook-wires implantation that is widely used to assist in lesion localization for uniportal thoracoscopic surgery can take place in the hybrid suite, eliminating the common complications and discomfort associated with the conventional workflow carried out in the radiology suite.

Comprehensive evaluation of ten deformable image registration algorithms for contour propagation between CT and cone-beam CT images in adaptive head & neck radiotherapy

Deformable image registration (DIR) is a critical technic in adaptive radiotherapy (ART) for propagating contours between planning computerized tomography (CT) images and treatment CT/cone-beam CT (CBCT) images to account for organ deformation for treatment re-planning. To validate the ability and accuracy of DIR algorithms in organ at risk (OAR) contour mapping, ten intensity-based DIR strategies, which were classified into four categories—optical flow-based, demons-based, level-set-based and spline-based—were tested on planning CT and fractional CBCT images acquired from twenty-one head & neck (H&N) cancer patients who underwent 6~7-week intensity-modulated radiation therapy (IMRT). Three similarity metrics, i.e., the Dice similarity coefficient (DSC), the percentage error (PE) and the Hausdorff distance (HD), were employed to measure the agreement between the propagated contours and the physician-delineated ground truths of four OARs, including the vertebra (VTB), the vertebral foramen (VF), the parotid gland (PG) and the submandibular gland (SMG). It was found that the evaluated DIRs in this work did not necessarily outperform rigid registration. DIR performed better for bony structures than soft-tissue organs, and the DIR performance tended to vary for different ROIs with different degrees of deformation as the treatment proceeded. Generally, the optical flow-based DIR performed best, while the demons-based DIR usually ranked last except for a modified demons-based DISC used for CT-CBCT DIR. These experimental results suggest that the choice of a specific DIR algorithm depends on the image modality, anatomic site, magnitude of deformation and application. Therefore, careful examinations and modifications are required before accepting the auto-propagated contours, especially for automatic re-planning ART systems.

Integration of free-hand 3D ultrasound and mobile C-arm cone-beam CT: Feasibility and characterization for real-time guidance of needle insertion

This work presents development of an integrated ultrasound (US)-cone-beam CT (CBCT) system for image-guided needle interventions, combining a low-cost ultrasound system (Interson VC 7.5MHz, Pleasanton, CA) with a mobile C-arm for fluoroscopy and CBCT via use of a surgical tracker. Imaging performance of the ultrasound system was characterized in terms of depth-dependent contrast-to-noise ratio (CNR) and spatial resolution. US-CBCT system was evaluated in phantom studies simulating three needle-based procedures: drug delivery, tumor ablation, and lumbar puncture. Low-cost ultrasound provided flexibility but exhibited modest CNR and spatial resolution that is likely limited to fairly superficial applications within a ∼10cm depth of view. Needle tip localization demonstrated target registration error 2.1-3.0mm using fiducial-based registration.

DCE-MRI Perfusion and Permeability Parameters as predictors of tumor response to CCRT in Patients with locally advanced NSCLC

In this prospective study, 36 patients with stage III non-small cell lung cancers (NSCLC), who underwent dynamic contrast-enhanced MRI (DCE-MRI) before concurrent chemo-radiotherapy (CCRT) were enrolled. Pharmacokinetic analysis was carried out after non-rigid motion registration. The perfusion parameters [including Blood Flow (BF), Blood Volume (BV), Mean Transit Time (MTT)] and permeability parameters [including endothelial transfer constant (K^trans), reflux rate (K_ep), fractional extravascular extracellular space volume (V_e), fractional plasma volume (V_p)] were calculated, and their relationship with tumor regression was evaluated. The value of these parameters on predicting responders were calculated by receiver operating characteristic (ROC) curve. Multivariate logistic regression analysis was conducted to find the independent variables. Tumor regression rate is negatively correlated with V_e and its standard variation V_e_SD and positively correlated with K^trans and K_ep. Significant differences between responders and non-responders existed in K^trans, K_ep, V_e, V_e_SD, MTT, BV_SD and MTT_SD (P < 0.05). ROC indicated that V_e < 0.24 gave the largest area under curve of 0.865 to predict responders. Multivariate logistic regression analysis also showed V_e was a significant predictor. Baseline perfusion and permeability parameters calculated from DCE-MRI were seen to be a viable tool for predicting the early treatment response after CCRT of NSCLC.

Image-guided localization of small lung nodules in video-assisted thoracic surgery

The advancement of imaging technology has recently facilitated single port minimally-invasive thoracic surgery techniques. Cone-beam computed tomography (CBCT) shows promising results in visualizing the target lesion and its surrounding critical anatomy, with an error of less than 2 mm. The integration of CBCT with the operating room (OR) to form the hybrid OR, provides unparalleled real-time imaging of the patient, which can be used with electromagnetic navigation bronchoscopy to confirm successful navigation and increase procedural accuracy particularly for small peripheral pulmonary targets. Furthermore, implantation of hookwires or microcoils that are widely used to localize the lesion can take place in the hybrid suite, eliminating the common complications and discomfort associated with the conventional workflow carried out in the radiology suite. Displacement leading to localization failure can be also reduced, and sublobar resection will be performed without resecting a larger area of parenchyma than desired. This one-stop, paradigm-shifting concept for simultaneously diagnosing and managing small pulmonary lesions in the hybrid OR can lead to reduced invasiveness and improved patient care.

Minimally interactive segmentation of 4D dynamic upper airway MR images via fuzzy connectedness

PURPOSE

There are several disease conditions that lead to upper airway restrictive disorders. In the study of these conditions, it is important to take into account the dynamic nature of the upper airway. Currently, dynamic magnetic resonance imaging is the modality of choice for studying these diseases. Unfortunately, the contrast resolution obtainable in the images poses many challenges for an effective segmentation of the upper airway structures. No viable methods have been developed to date to solve this problem. In this paper, the authors demonstrate a practical solution by employing an iterative relative fuzzy connectedness delineation algorithm as a tool.

METHODS

3D dynamic images were collected at ten equally spaced instances over the respiratory cycle (i.e., 4D) in 20 female subjects with obstructive sleep apnea syndrome. The proposed segmentation approach consists of the following steps. First, image background nonuniformities are corrected which is then followed by a process to correct for the nonstandardness of MR image intensities. Next, standardized image intensity statistics are gathered for the nasopharynx and oropharynx portions of the upper airway as well as the surrounding soft tissue structures including air outside the body region, hard palate, soft palate, tongue, and other soft structures around the airway including tonsils (left and right) and adenoid. The affinity functions needed for fuzzy connectedness computation are derived based on these tissue intensity statistics. In the next step, seeds for fuzzy connectedness computation are specified for the airway and the background tissue components. Seed specification is needed in only the 3D image corresponding to the first time instance of the 4D volume; from this information, the 3D volume corresponding to the first time point is segmented. Seeds are automatically generated for the next time point from the segmentation of the 3D volume corresponding to the previous time point, and the process continues and runs without human interaction and completes in 10 s for segmenting the airway structure in the whole 4D volume.

RESULTS

Qualitative evaluations performed to examine smoothness and continuity of motions of the entire upper airway as well as its transverse sections at critical anatomic locations indicate that the segmentations are consistent. Quantitative evaluations of the separate 200 3D volumes and the 20 4D volumes yielded true positive and false positive volume fractions around 95% and 0.1%, respectively, and mean boundary placement errors under 0.5 mm. The method is robust to variations in the subjective action of seed specification. Compared with a segmentation approach based on a registration technique to propagate segmentations, the proposed method is more efficient, accurate, and less prone to error propagation from one respiratory time point to the next.

CONCLUSIONS

The proposed method is the first demonstration of a viable and practical approach for segmenting the upper airway structures in dynamic MR images. Compared to registration-based methods, it effectively reduces error propagation and consequently achieves not only more accurate segmentations but also more consistent motion representation in the segmentations. The method is practical, requiring minimal user interaction and computational time.

Case Report: Simultaneous Localization and Removal of Lung Nodules Through Extended Use of the Hybrid Suite

The ability to attain high-definition imaging for preoperative planning, intraoperative execution, and postoperative evaluation is instrumental in surgical practice. Hybrid room computed tomography (CT) allows for faster, less invasive diagnostic and therapeutic options for patients. We present our diagnostic workup and therapeutic intervention with hybrid CT imaging in a 71-year-old female with a growing lung nodule after previous lobectomy for lung cancer.

3D-2D image registration for target localization in spine surgery: investigation of similarity metrics providing robustness to content mismatch

In image-guided spine surgery, robust three-dimensional to two-dimensional (3D-2D) registration of preoperative computed tomography (CT) and intraoperative radiographs can be challenged by the image content mismatch associated with the presence of surgical instrumentation and implants as well as soft-tissue resection or deformation. This work investigates image similarity metrics in 3D-2D registration offering improved robustness against mismatch, thereby improving performance and reducing or eliminating the need for manual masking. The performance of four gradient-based image similarity metrics (gradient information (GI), gradient correlation (GC), gradient information with linear scaling (GS), and gradient orientation (GO)) with a multi-start optimization strategy was evaluated in an institutional review board-approved retrospective clinical study using 51 preoperative CT images and 115 intraoperative mobile radiographs. Registrations were tested with and without polygonal masks as a function of the number of multistarts employed during optimization. Registration accuracy was evaluated in terms of the projection distance error (PDE) and assessment of failure modes (PDE  >  30 mm) that could impede reliable vertebral level localization. With manual polygonal masking and 200 multistarts, the GC and GO metrics exhibited robust performance with 0% gross failures and median PDE < 6.4 mm (±4.4 mm interquartile range (IQR)) and a median runtime of 84 s (plus upwards of 1-2 min for manual masking). Excluding manual polygonal masks and decreasing the number of multistarts to 50 caused the GC-based registration to fail at a rate of >14%; however, GO maintained robustness with a 0% gross failure rate. Overall, the GI, GC, and GS metrics were susceptible to registration errors associated with content mismatch, but GO provided robust registration (median PDE  =  5.5 mm, 2.6 mm IQR) without manual masking and with an improved runtime (29.3 s). The GO metric improved the registration accuracy and robustness in the presence of strong image content mismatch. This capability could offer valuable assistance and decision support in spine level localization in a manner consistent with clinical workflow.

dPIRPLE: a joint estimation framework for deformable registration and penalized-likelihood CT image reconstruction using prior images

Sequential imaging studies are conducted in many clinical scenarios. Prior images from previous studies contain a great deal of patient-specific anatomical information and can be used in conjunction with subsequent imaging acquisitions to maintain image quality while enabling radiation dose reduction (e.g., through sparse angular sampling, reduction in fluence, etc). However, patient motion between images in such sequences results in misregistration between the prior image and current anatomy. Existing prior-image-based approaches often include only a simple rigid registration step that can be insufficient for capturing complex anatomical motion, introducing detrimental effects in subsequent image reconstruction. In this work, we propose a joint framework that estimates the 3D deformation between an unregistered prior image and the current anatomy (based on a subsequent data acquisition) and reconstructs the current anatomical image using a model-based reconstruction approach that includes regularization based on the deformed prior image. This framework is referred to as deformable prior image registration, penalized-likelihood estimation (dPIRPLE). Central to this framework is the inclusion of a 3D B-spline-based free-form-deformation model into the joint registration-reconstruction objective function. The proposed framework is solved using a maximization strategy whereby alternating updates to the registration parameters and image estimates are applied allowing for improvements in both the registration and reconstruction throughout the optimization process. Cadaver experiments were conducted on a cone-beam CT testbench emulating a lung nodule surveillance scenario. Superior reconstruction accuracy and image quality were demonstrated using the dPIRPLE algorithm as compared to more traditional reconstruction methods including filtered backprojection, penalized-likelihood estimation (PLE), prior image penalized-likelihood estimation (PIPLE) without registration, and prior image penalized-likelihood estimation with rigid registration of a prior image (PIRPLE) over a wide range of sampling sparsity and exposure levels.

Deformable image registration for cone-beam CT guided transoral robotic base-of-tongue surgery

Transoral robotic surgery (TORS) offers a minimally invasive approach to resection of base-of-tongue tumors. However, precise localization of the surgical target and adjacent critical structures can be challenged by the highly deformed intraoperative setup. We propose a deformable registration method using intraoperative cone-beam computed tomography (CBCT) to accurately align preoperative CT or MR images with the intraoperative scene. The registration method combines a Gaussian mixture (GM) model followed by a variation of the Demons algorithm. First, following segmentation of the volume of interest (i.e. volume of the tongue extending to the hyoid), a GM model is applied to surface point clouds for rigid initialization (GM rigid) followed by nonrigid deformation (GM nonrigid). Second, the registration is refined using the Demons algorithm applied to distance map transforms of the (GM-registered) preoperative image and intraoperative CBCT. Performance was evaluated in repeat cadaver studies (25 image pairs) in terms of target registration error (TRE), entropy correlation coefficient (ECC) and normalized pointwise mutual information (NPMI). Retraction of the tongue in the TORS operative setup induced gross deformation >30 mm. The mean TRE following the GM rigid, GM nonrigid and Demons steps was 4.6, 2.1 and 1.7 mm, respectively. The respective ECC was 0.57, 0.70 and 0.73, and NPMI was 0.46, 0.57 and 0.60. Registration accuracy was best across the superior aspect of the tongue and in proximity to the hyoid (by virtue of GM registration of surface points on these structures). The Demons step refined registration primarily in deeper portions of the tongue further from the surface and hyoid bone. Since the method does not use image intensities directly, it is suitable to multi-modality registration of preoperative CT or MR with intraoperative CBCT. Extending the 3D image registration to the fusion of image and planning data in stereo-endoscopic video is anticipated to support safer, high-precision base-of-tongue robotic surgery.