Registration and Fusion of Thermographic and Visual-Light Images in Neurosurgery

Registration of IRT and visible light images in neurosurgery: analysis and comparison of automatic intensity-based registration approaches

PURPOSE

The purpose of this study is to analyze and compare six automatic intensity-based registration methods for intraoperative infrared thermography (IRT) and visible light imaging (VIS/RGB). The practical requirement is to get a good performance of Euclidean distance between manually set landmarks in reference and target images as well as to achieve a high structural similarity index metric (SSIM) and peak signal-to-noise ratio (PSNR) with respect to the reference image.

METHODS

In this study, preprocessing is applied to bring both image types to a similar intensity. Similarity transformation is employed to align roughly IRT and visible light images. Two optimizers and two measures are used in this process. Thereafter, due to locally different displacement of the brain surface through respiration and heartbeat, two non-rigid transformations are applied, and finally, a bicubic interpolation is carried out to compensate for the resulting estimated transformation. Performance was assessed using eleven image datasets. The registration accuracy of the different computational approaches was assessed based on SSIM and PSNR. Additionally, five concise landmarks for each dataset were selected manually in reference and target images and the Euclidean distance between the corresponding landmarks was compared.

RESULTS

The results are showing that the combination of normalized intensity, mutual information measure with one-plus-one evolutionary optimizer in combination with Demon registration results in improved accuracy and performance as compared to all other methods tested here. Furthermore, the obtained results led to [Formula: see text], [Formula: see text], [Formula: see text], [Formula: see text], and [Formula: see text] registrations for datasets 1, 2, 5, 7, and 8 with respect to the second best result by calculating the mean Euclidean distance of five landmarks.

CONCLUSIONS

We conclude that the mutual information measure with one-plus-one evolutionary optimizer in combination with Demon registration can achieve better accuracy and performance to those other methods mentioned here for automatic registration of IRT and visible light images in neurosurgery.

Infrared Thermographic Imaging of Chest Wall Perfusion in Patients Undergoing Coronary Artery Bypass Grafting

Coronary artery disease represents a leading cause of death worldwide, to which the coronary artery bypass graft (CABG) is the main method of treatment in advanced multiple vessel disease. The use of the internal mammary artery (IMA) as a graft insures an improved long-term survival, but impairment of chest wall perfusion often leads to surgical site infection and increased morbidity and mortality. Infrared thermography (IRT) has established itself in the past decades as a non-invasive diagnostic technique. The applications vary from veterinary to human medicine and from head to toe. In this study we used IRT in 42 patients receiving CABG to determine the changes in skin surface temperature preoperatively, two hours, 24 h and 6 days after surgery. The results showed a significant and independent drop of surface temperature 2 h after surgery on the whole surface of the chest wall, as well as a further reduction on the left side after harvesting the IMA. The temperature returned to normal after 24 h and remained so after 6 days. The study has shown that IRT is sufficiently sensitive to demonstrate the known, subtle reduction in chest wall perfusion associated with IMA harvesting.

Assessment of Registration Methods for Thermal Infrared and Visible Images for Diabetic Foot Monitoring

This work presents a revision of four different registration methods for thermal infrared and visible images captured by a camera-based prototype for the remote monitoring of diabetic foot. This prototype uses low cost and off-the-shelf available sensors in thermal infrared and visible spectra. Four different methods (Geometric Optical Translation, Homography, Iterative Closest Point, and Affine transform with Gradient Descent) have been implemented and analyzed for the registration of images obtained from both sensors. All four algorithms' performances were evaluated using the Simultaneous Truth and Performance Level Estimation (STAPLE) together with several overlap benchmarks as the Dice coefficient and the Jaccard index. The performance of the four methods has been analyzed with the subject at a fixed focal plane and also in the vicinity of this plane. The four registration algorithms provide suitable results both at the focal plane as well as outside of it within 50 mm margin. The obtained Dice coefficients are greater than 0.950 in all scenarios, well within the margins required for the application at hand. A discussion of the obtained results under different distances is presented along with an evaluation of its robustness under changing conditions.

Segmentation Approaches for Diabetic Foot Disorders

Thermography enables non-invasive, accessible, and easily repeated foot temperature measurements for diabetic patients, promoting early detection and regular monitoring protocols, that limit the incidence of disabling conditions associated with diabetic foot disorders. The establishment of this application into standard diabetic care protocols requires to overcome technical issues, particularly the foot sole segmentation. In this work we implemented and evaluated several segmentation approaches which include conventional and Deep Learning methods. Multimodal images, constituted by registered visual-light, infrared and depth images, were acquired for 37 healthy subjects. The segmentation methods explored were based on both visual-light as well as infrared images, and optimization was achieved using the spatial information provided by the depth images. Furthermore, a ground truth was established from the manual segmentation performed by two independent researchers. Overall, the performance level of all the implemented approaches was satisfactory. Although the best performance, in terms of spatial overlap, accuracy, and precision, was found for the Skin and U-Net approaches optimized by the spatial information. However, the robustness of the U-Net approach is preferred.

Motion Correction in Multimodal Intraoperative Imaging

Thermographic imaging accompanied with time-resolved analysis is a promising technique for intraoperative imaging in neurosurgery. However, motion due to breathing and pulse of the patient introduces large inaccuracies to the demarcation of normal and pathological brain tissue. Since movements and physiological processes are both manifested as temperature variations, we employ co-registered visual-light images to unambiguously detect motion. In this article, we propose a feature-based approach which is selected from four best-known methods after thorough performance comparison. Complementing our previous work, we evaluate the performance of our methods by applying a frequency analysis and similarity measurements. Our approach enables an accurate motion correction without affecting physiological temperature shifts. Furthermore, real-time performance of the implementation is enabled by serial acceleration and parallelization methods.

Characterization and Performance of a Thermal Camera Communication System

This work presents a novel communications technology named Thermal Camera Communication (TCC), which is analogous to Optical Camera Communication (OCC). Thermographic cameras and Peltier cells are proposed as receiver and transmitter, respectively, changing completely their usual field of application. Furthermore, a comprehensive characterization of the Peltier-Thermal camera pair is carried out, presenting their bandwidth, achievable data rate under On-Off-Keying (OOK) modulation, noise characteristics, and energy efficiency. A comparison against the current state-of-the-art OCC technology is also provided, showing that TCC is a promising technology suitable for sensor networks. The thorough analysis of TCC performed in this work shows that commercial Peltier cells can be re-thought under a communications viewpoint in order to improve their performance. This novel communication technology can be applied in environments such as the access to public transportation or buildings due to the new health emergency situation. The use of thermographic cameras will become massive and dual measurement and communication purposes could be considered for applications such as sensor networks, using a yet unexploited wavelength range.

Three-Layer Image Representation by an Enhanced Illumination-Based Image Fusion Method

The recently developed multiscale-based fusion methods can be improved with two approaches: an advanced image decomposition scheme and an advanced fusion rule. In this paper, three-layer image decomposition, enhanced illumination fusion rule-based method is proposed. The proposed method includes three steps. First, each input image is decomposed into its corresponding smooth, texture, and edge layers using defined local extrema and low-pass filters in the spatial domain. Second, three different strategies are applied as fusion rules for the three-layer representation. To preserve the illumination closely related to tumors, the illumination is corrected by applying a higher contrast to the decomposed image details, including the texture and edge inputs, such as those found in grayscale CT and MRI images. The final fused image is created by the addition of the normalized smooth, texture, and edge image layers. The experiments demonstrate that the proposed method performs better than the existing state-of-the-art fusion methods.