Multispectral image alignment using a three channel endoscope in vivo during minimally invasive surgery

Multispectral tissue mapping: developing a concept for the optical evaluation of liver disease

Purpose: Alterations in the optical absorption behavior of liver tissue secondary to pathological processes can be evaluated by multispectral analysis, which is increasingly being explored as an imaging adjunct for use in liver surgery. Current methods are either invasive or have a limited wavelength spectrum, which restricts utility. This proof of concept study describes the development of a multispectral imaging (MSI) method called multispectral tissue mapping (MTM) that addresses these issues. Approach: The imaging system consists of a tunable excitation light source and a near-infrared camera. Following the development stage, proof of concept experiments are carried out where absorption spectra from colorectal cancer liver metastasis (CRLM), hepatocellular carcinoma (HCC), and liver steatosis specimen are acquired and compared to controls. Absorption spectra are compared to histopathology examination as the current gold standard for tissue assessment. Generalized linear mixed modeling is employed to compare absorption characteristics of individual pixels and to select wavelengths for false color image processing with the aim of visually enhancing cancer tissue. Results: Analysis of individual pixels revealed distinct absorption spectra therefore suggesting that MTM is possible. A prominent absorption peak at 1210 nm was found in lipid-rich animal tissues and steatotic liver specimen. Liver cancer tissue had a heterogeneous appearance on MSI. Subsequent statistical analysis suggests that measuring changes in absorption behavior may be a feasible method to estimate the pixel-based probability of cancer being present. In CRLM, this was observed throughout 1100 to 1700 nm, whereas in HCC it was concentrated around 1140 and 1430 nm. False color image processing visibly enhances contrast between cancer and normal liver tissues. Conclusions: The system's ability to enable no-touch MSI at 1100 to 1700 nm was demonstrated. Preliminary data suggest that MTM warrants further exploration as a potential imaging tool for the detection of liver cancer during surgery.

Snapshot multispectral endomicroscopy

Multispectral endomicroscopy provides tissue functional information in addition to structural information for accurate disease diagnosis. In this Letter, we propose a snapshot multispectral endomicroscope that employs a fiber bundle to deliver an in-body tissue spatial-spectral datastream to an external compressive spectral imager. Equipped with an end-to-end deep-learning-based reconstruction algorithm, we are able to capture tissue multispectral data in video rates and reconstruct high-resolution multispectral images with up to 24 spectral channels in near-real time.

Surgical spectral imaging

Recent technological developments have resulted in the availability of miniaturised spectral imaging sensors capable of operating in the multi- (MSI) and hyperspectral imaging (HSI) regimes. Simultaneous advances in image-processing techniques and artificial intelligence (AI), especially in machine learning and deep learning, have made these data-rich modalities highly attractive as a means of extracting biological information non-destructively. Surgery in particular is poised to benefit from this, as spectrally-resolved tissue optical properties can offer enhanced contrast as well as diagnostic and guidance information during interventions. This is particularly relevant for procedures where inherent contrast is low under standard white light visualisation. This review summarises recent work in surgical spectral imaging (SSI) techniques, taken from Pubmed, Google Scholar and arXiv searches spanning the period 2013-2019. New hardware, optimised for use in both open and minimally-invasive surgery (MIS), is described, and recent commercial activity is summarised. Computational approaches to extract spectral information from conventional colour images are reviewed, as tip-mounted cameras become more commonplace in MIS. Model-based and machine learning methods of data analysis are discussed in addition to simulation, phantom and clinical validation experiments. A wide variety of surgical pilot studies are reported but it is apparent that further work is needed to quantify the clinical value of MSI/HSI. The current trend toward data-driven analysis emphasises the importance of widely-available, standardised spectral imaging datasets, which will aid understanding of variability across organs and patients, and drive clinical translation.

Estimation of tissue oxygen saturation from RGB images and sparse hyperspectral signals based on conditional generative adversarial network

PURPOSE

Intra-operative measurement of tissue oxygen saturation ([Formula: see text]) is important in detection of ischaemia, monitoring perfusion and identifying disease. Hyperspectral imaging (HSI) measures the optical reflectance spectrum of the tissue and uses this information to quantify its composition, including [Formula: see text]. However, real-time monitoring is difficult due to capture rate and data processing time.

METHODS

An endoscopic system based on a multi-fibre probe was previously developed to sparsely capture HSI data (sHSI). These were combined with RGB images, via a deep neural network, to generate high-resolution hypercubes and calculate [Formula: see text]. To improve accuracy and processing speed, we propose a dual-input conditional generative adversarial network, Dual2StO2, to directly estimate [Formula: see text] by fusing features from both RGB and sHSI.

RESULTS

Validation experiments were carried out on in vivo porcine bowel data, where the ground truth [Formula: see text] was generated from the HSI camera. Performance was also compared to our previous super-spectral-resolution network, SSRNet in terms of mean [Formula: see text] prediction accuracy and structural similarity metrics. Dual2StO2 was also tested using simulated probe data with varying fibre number.

CONCLUSIONS

[Formula: see text] estimation by Dual2StO2 is visually closer to ground truth in general structure and achieves higher prediction accuracy and faster processing speed than SSRNet. Simulations showed that results improved when a greater number of fibres are used in the probe. Future work will include refinement of the network architecture, hardware optimization based on simulation results, and evaluation of the technique in clinical applications beyond [Formula: see text] estimation.

Reconstructing a 3D heart surface with stereo-endoscope by learning eigen-shapes

An efficient approach to dynamically reconstruct a region of interest (ROI) on a beating heart from stereo-endoscopic video is developed. A ROI is first pre-reconstructed with a decoupled high-rank thin plate spline model. Eigen-shapes are learned from the pre-reconstructed data by using principal component analysis (PCA) to build a low-rank statistical deformable model for reconstructing subsequent frames. The linear transferability of PCA is proved, which allows fast eigen-shape learning. A general dynamic reconstruction framework is developed that formulates ROI reconstruction as an optimization problem of model parameters, and an efficient second-order minimization algorithm is derived to iteratively solve it. The performance of the proposed method is finally validated on stereo-endoscopic videos recorded by da Vinci robots.

Emerging optical methods for endoscopic surveillance of Barrett's oesophagus

Barrett's oesophagus is an acquired metaplastic condition that predisposes patients to the development of oesophageal adenocarcinoma, prompting the use of surveillance regimes to detect early malignancy for endoscopic therapy with curative intent. The currently accepted surveillance regime uses white light endoscopy together with random biopsies, but has poor sensitivity and discards information from numerous light-tissue interactions that could be exploited to probe structural, functional, and molecular changes in the tissue. Advanced optical methods are now emerging that are highly sensitive to these changes and hold potential to improve surveillance of Barrett's oesophagus if they can be applied endoscopically. The next decade will see some of these exciting new methods applied to surveillance of Barrett's oesophagus in new device architectures for the first time, potentially leading to a long-awaited improvement in the standard of care.

Use of biomedical photonics in gynecological surgery: a uterine transplantation model

Aim:

Uterine transplantation (UTx) has been proposed as a treatment for permanent absolute uterine factor infertility. The study aims were to compare pulse oximetry and multispectral imaging (MSI), for intraoperative tracking of uterine oxygen saturation in animal UTx models (rabbit and sheep).

Results/methodology:

Imaging results confirmed the re-establishment of adequate perfusion in the transplanted organ after surgery. Comparison of oxygen saturation values between the pre-UTx donor and post-UTx recipient, and pre-UTx and post-UTx recipient reveals a statistically significant decrease in saturation levels post-UTx.

Conclusion:

The use of MSI is the first case in gynecology and has demonstrated promise of possible future human use. MSI technique has advantages over pulse oximetry – it provides spatial information in a real-time, noncontact manner.

Transplanting wombs into women has been offered as a potential solution to treat women who have had them removed previously or who were born without one. Our aim here was to compare two techniques: pulse oximetry and multispectral imaging, for checking oxygen levels in the newly transplanted womb. The experiments were performed in two types of animals, rabbit and sheep. The use of multispectral imaging is the first case in gynecology and has demonstrated promise of possible future human use. It has advantages over pulse oximetry – it provides valuable information in a real-time, non-contact manner.

Bayesian Estimation of Intrinsic Tissue Oxygenation and Perfusion From RGB Images

Multispectral imaging (MSI) can potentially assist the intra-operative assessment of tissue structure, function and viability, by providing information about oxygenation. In this paper, we present a novel technique for recovering intrinsic MSI measurements from endoscopic RGB images without custom hardware adaptations. The advantage of this approach is that it requires no modification to existing surgical and diagnostic endoscopic imaging systems. Our method uses a radiometric color calibration of the endoscopic camera's sensor in conjunction with a Bayesian framework to recover a per-pixel measurement of the total blood volume (THb) and oxygen saturation (SO₂) in the observed tissue. The sensor's pixel measurements are modeled as weighted sums over a mixture of Poisson distributions and we optimize the variables SO₂ and THb to maximize the likelihood of the observations. To validate our technique, we use synthetic images generated from Monte Carlo physics simulation of light transport through soft tissue containing sub-surface blood vessels. We also validate our method on in vivo data by comparing it to a MSI dataset acquired with a hardware system that sequentially images multiple spectral bands without overlap. Our results are promising and show that we are able to provide surgeons with additional relevant information by processing endoscopic images with our modeling and inference framework.

Multispectral imaging of organ viability during uterine transplantation surgery in rabbits and sheep

Uterine transplantation surgery (UTx) has been proposed as a treatment for permanent absolute uterine factor infertility (AUFI) in the case of the congenital absence or surgical removal of the uterus. Successful surgical attachment of the organ and its associated vasculature is essential for the organ’s reperfusion and long-term viability. Spectral imaging techniques have demonstrated the potential for the measurement of hemodynamics in medical applications. These involve the measurement of reflectance spectra by acquiring images of the tissue in different wavebands. Measures of tissue constituents at each pixel can then be extracted from these spectra through modeling of the light–tissue interaction. A multispectral imaging (MSI) laparoscope was used in sheep and rabbit UTx models to study short- and long-term changes in oxygen saturation following surgery. The whole organ was imaged in the donor and recipient animals in parallel with point measurements from a pulse oximeter. Imaging results confirmed the re-establishment of adequate perfusion in the transplanted organ after surgery. Cornual oxygenation trends measured with MSI are consistent with pulse oximeter readings, showing decreased StO2 immediately after anastomosis of the blood vessels. Long-term results show recovery of StO2 to preoperative levels.

Multispectral imaging of organ viability during uterine transplantation surgery in rabbits and sheep

Uterine transplantation surgery (UTx) has been proposed as a treatment for permanent absolute uterine factor infertility (AUFI) in the case of the congenital absence or surgical removal of the uterus. Successful surgical attachment of the organ and its associated vasculature is essential for the organ’s reperfusion and long-term viability. Spectral imaging techniques have demonstrated the potential for the measurement of hemodynamics in medical applications. These involve the measurement of reflectance spectra by acquiring images of the tissue in different wavebands. Measures of tissue constituents at each pixel can then be extracted from these spectra through modeling of the light–tissue interaction. A multispectral imaging (MSI) laparoscope was used in sheep and rabbit UTx models to study short- and long-term changes in oxygen saturation following surgery. The whole organ was imaged in the donor and recipient animals in parallel with point measurements from a pulse oximeter. Imaging results confirmed the re-establishment of adequate perfusion in the transplanted organ after surgery. Cornual oxygenation trends measured with MSI are consistent with pulse oximeter readings, showing decreased StO2 immediately after anastomosis of the blood vessels. Long-term results show recovery of StO2 to preoperative levels.

Image enhancement based on in vivo hyperspectral gastroscopic images: a case study

Hyperspectral imaging (HSI) has been recognized as a powerful tool for noninvasive disease detection in the gastrointestinal field. However, most of the studies on HSI in this field have involved ex vivo biopsies or resected tissues. We proposed an image enhancement method based on in vivo hyperspectral gastroscopic images. First, we developed a flexible gastroscopy system capable of obtaining in vivo hyperspectral images of different types of stomach disease mucosa. Then, depending on a specific object, an appropriate band selection algorithm based on dependence of information was employed to determine a subset of spectral bands that would yield useful spatial information. Finally, these bands were assigned to be the color components of an enhanced image of the object. A gastric ulcer case study demonstrated that our method yields higher color tone contrast, which enhanced the displays of the gastric ulcer regions, and that it will be valuable in clinical applications.

Hyperspectral interventional imaging for enhanced tissue visualization and discrimination combining band selection methods

PURPOSE

Hyperspectral imaging is an emerging technology recently introduced in medical applications inasmuch as it provides a powerful tool for noninvasive tissue characterization. In this context, a new system was designed to be easily integrated in the operating room in order to detect anatomical tissues hardly noticed by the surgeon's naked eye.

METHOD

Our LCTF-based spectral imaging system is operative over visible, near- and middle-infrared spectral ranges (400-1700 nm). It is dedicated to enhance critical biological tissues such as the ureter and the facial nerve. We aim to find the best three relevant bands to create a RGB image to display during the intervention with maximal contrast between the target tissue and its surroundings. A comparative study is carried out between band selection methods and band transformation methods. Combined band selection methods are proposed. All methods are compared using different evaluation criteria.

RESULTS

Experimental results show that the proposed combined band selection methods provide the best performance with rich information, high tissue separability and short computational time. These methods yield a significant discrimination between biological tissues.

CONCLUSION

We developed a hyperspectral imaging system in order to enhance some biological tissue visualization. The proposed methods provided an acceptable trade-off between the evaluation criteria especially in SWIR spectral band that outperforms the naked eye's capacities.

Uniqueness in multispectral constant-wave epi-illumination imaging

Multispectral tissue imaging based on optical cameras and continuous-wave tissue illumination is commonly used in medicine and biology. Surprisingly, there is a characteristic absence of a critical look at the quantities that can be uniquely characterized from optically diffuse matter by multispectral imaging. Here, we investigate the fundamental question of uniqueness in epi-illumination measurements from turbid media obtained at multiple wavelengths. By utilizing an analytical model, tissue-mimicking phantoms, and an in vivo imaging experiment we show that independent of the bands employed, spectral measurements cannot uniquely retrieve absorption and scattering coefficients. We also establish that it is, nevertheless, possible to uniquely quantify oxygen saturation and the Mie scattering power-a previously undocumented uniqueness condition.

A high definition Mueller polarimetric endoscope for tissue characterisation

The contrast mechanism of medical endoscopy is mainly based on metrics of optical intensity and wavelength. As another fundamental property of light, polarization can not only reveal tissue scattering and absorption information from a different perspective, but can also provide insight into directional tissue birefringence properties to monitor pathological changes in collagen and elastin. Here we demonstrate a low cost wide field high definition Mueller polarimetric endoscope with minimal alterations to a rigid endoscope. We show that this novel endoscopic imaging modality is able to provide a number of image contrast mechanisms besides traditional unpolarized radiation intensity, including linear depolarization, circular depolarization, cross-polarization, directional birefringence and dichroism. This enhances tissue features of interest, and additionally reveals tissue micro-structure and composition, which is of central importance for tissue diagnosis and image guidance for surgery. The potential applications of the Mueller polarimetric endoscope include wide field early epithelial cancer diagnosis, surgical margin detection and energy-based tissue fusion monitoring, and could further benefit a wide range of endoscopic investigations through intra-operative guidance.