Video: Clinical evaluation of a laparoscopic hyperspectral imaging system

Surgical optomics: hyperspectral imaging and deep learning towards precision intraoperative automatic tissue recognition-results from the EX-MACHYNA trial

BACKGROUND

Hyperspectral imaging (HSI), combined with machine learning, can help to identify characteristic tissue signatures enabling automatic tissue recognition during surgery. This study aims to develop the first HSI-based automatic abdominal tissue recognition with human data in a prospective bi-center setting.

METHODS

Data were collected from patients undergoing elective open abdominal surgery at two international tertiary referral hospitals from September 2020 to June 2021. HS images were captured at various time points throughout the surgical procedure. Resulting RGB images were annotated with 13 distinct organ labels. Convolutional Neural Networks (CNNs) were employed for the analysis, with both external and internal validation settings utilized.

RESULTS

A total of 169 patients were included, 73 (43.2%) from Strasbourg and 96 (56.8%) from Verona. The internal validation within centers combined patients from both centers into a single cohort, randomly allocated to the training (127 patients, 75.1%, 585 images) and test sets (42 patients, 24.9%, 181 images). This validation setting showed the best performance. The highest true positive rate was achieved for the skin (100%) and the liver (97%). Misclassifications included tissues with a similar embryological origin (omentum and mesentery: 32%) or with overlaying boundaries (liver and hepatic ligament: 22%). The median DICE score for ten tissue classes exceeded 80%.

CONCLUSION

To improve automatic surgical scene segmentation and to drive clinical translation, multicenter accurate HSI datasets are essential, but further work is needed to quantify the clinical value of HSI. HSI might be included in a new omics science, namely surgical optomics, which uses light to extract quantifiable tissue features during surgery.

Registration of hyperspectral images and mass spectrometry data for the correlation of tissue optical spectra and molecular profiles

Hyperspectral imaging (HSI) is a label-free imaging modality that is emerging for non-invasive detection of various diseases including cancers. HSI provides high-resolution spatial images where each pixel has a spectral curve with numerous wavelength bands from the visible to infrared ranges. The rich spatial and spectral information can be used to discriminate various types of tissues and pathophysiological conditions. However, it can be difficult to explain spectral data with respect to the underline cellular and molecular mechanism. In this study, we developed an approach that registers hyperspectral images and mass spectrometry (MS) data where MS provides tissue molecular profiles. Human prostate tissues that were obtained after prostatectomy were used in the experiments. The whole prostate was first sliced every six mm. A customized hyperspectral surgical microscope was used to acquire HSI data from the sliced tissue. For MS data analysis, the sliced tissue of the prostate was divided into 51 small regions and then processed separately for each region. The immediately adjacent tissue was sliced and processed histologically for H&E staining. The MS molecular profiles were correlated with the hyperspectral images in this study.

Switchable LED-based laparoscopic multispectral system for rapid high-resolution perfusion imaging

Significance

Multispectral imaging (MSI) is an approach for real-time, quantitative, and non-invasive tissue perfusion measurements. Current laparoscopic systems based on mosaic sensors or filter wheels lack high spatial resolution or acceptable frame rates.

Aim

To develop a laparoscopic system for MSI-based color video and tissue perfusion imaging during gastrointestinal surgery without compromising spatial or temporal resolution.

Approach

The system was built with 14 switchable light-emitting diodes in the visible and near-infrared spectral range, a 4K image sensor, and a 10 mm laparoscope. Illumination patterns were created for tissue oxygenation and hemoglobin content monitoring. The system was calibrated to a clinically approved laparoscopic hyperspectral system using linear regression models and evaluated in an occlusion study with 36 volunteers.

Results

The root mean squared errors between the MSI and reference system were 0.073 for hemoglobin content, 0.039 for oxygenation in deeper tissue layers, and 0.093 for superficial oxygenation. The spatial resolution at a working distance of 45 mm was 156    μ m . The effective frame rate was 20 fps.

Conclusions

High-resolution perfusion monitoring was successfully achieved. Hardware optimizations will increase the frame rate. Parameter optimizations through alternative illumination patterns, regression, or assumed tissue models are planned. Intraoperative measurements must confirm the suitability during surgery.

Intra-abdominal laparoscopic assessment of organs perfusion using imaging photoplethysmography

BACKGROUND

An objective evaluation of the functional state and viability of biological tissues during minimally invasive surgery remains unsolved task. Various non-contact methods for evaluating perfusion during laparoscopic surgery are discussed in the literature, but so far there have been no reports of their use in clinical settings.

METHODS AND PATIENTS

Imaging photoplethysmography (iPPG) is a new method for quantitative assessment of perfusion distribution along the tissue. This is the first study in which we demonstrate successful use of iPPG to assess perfusion of organs during laparoscopic surgery in an operation theater. We used a standard rigid laparoscope connected to a standard digital monochrome camera, and abdominal organs were illuminated by green light. A distinctive feature is the synchronous recording of video frames and electrocardiogram with subsequent correlation data processing. During the laparoscopically assisted surgeries in nine cancer patients, the gradient of perfusion of the affected organs was evaluated. In particular, measurements were carried out before preparing a part of the intestine or stomach for resection, after anastomosis, or during physiological tests.

RESULTS

The spatial distribution of perfusion and its changes over time were successfully measured in all surgical cases. In particular, perfusion gradient of an intestine before resection was visualized and quantified by our iPPG laparoscope in all respective cases. It was also demonstrated that systemic administration of norepinephrine leads to a sharper gradient between well and poorly perfused areas of the colon. In four surgical cases, we have shown capability of the laparoscopic iPPG system for intra-abdominal assessment of perfusion in the anastomosed organs. Moreover, good repeatability of continuous long-term measurements of tissue perfusion inside the abdominal cavity was experimentally demonstrated.

CONCLUSION

Our study carried out in real clinical settings has shown that iPPG laparoscope is feasible for intra-abdominal visualization and quantitative assessment of perfusion distribution.

Alternative intraoperative optical imaging modalities for fluorescence angiography in gastrointestinal surgery: spectral imaging and imaging photoplethysmography

INTRODUCTION

Intraoperative near-infrared fluorescence angiography with indocyanine green (ICG-FA) is a well-established modality in gastrointestinal surgery. Its main drawback is the application of a fluorescent agent with possible side effects for patients. The goal of this review paper is the presentation of alternative, non-invasive optical imaging methods and their comparison with ICG-FA.

MATERIAL AND METHODS

The principles of ICG-FA, spectral imaging, imaging photoplethysmography (iPPG), and their applications in gastrointestinal surgery are described based on selected published works.

RESULTS

The main applications of the three modalities are the evaluation of tissue perfusion, the identification of risk structures, and tissue segmentation or classification. While the ICG-FA images are mainly evaluated visually, leading to subjective interpretations, quantitative physiological parameters and tissue segmentation are provided in spectral imaging and iPPG. The combination of ICG-FA and spectral imaging is a promising method.

CONCLUSIONS

Non-invasive spectral imaging and iPPG have shown promising results in gastrointestinal surgery. They can overcome the main drawbacks of ICG-FA, i.e. the use of contrast agents, the lack of quantitative analysis, repeatability, and a difficult standardization of the acquisition. Further technical improvements and clinical evaluations are necessary to establish them in daily clinical routine.

Size and depth of residual tumor after neoadjuvant chemoradiotherapy in rectal cancer - implications for the development of new imaging modalities for response assessment

With the shift towards organ preserving treatment strategies in rectal cancer it has become increasingly important to accurately discriminate between a complete and good clinical response after neoadjuvant chemoradiotherapy (CRT). Standard of care imaging techniques such as CT and MRI are well equipped for initial staging of rectal tumors, but discrimination between a good clinical and complete response remains difficult due to their limited ability to detect small residual vital tumor fragments. To identify new promising imaging techniques that could fill this gap, it is crucial to know the size and invasion depth of residual vital tumor tissue since this determines the requirements with regard to the resolution and imaging depth of potential new optical imaging techniques. We analyzed 198 pathology slides from 30 rectal cancer patients with a Mandard tumor regression grade 2 or 3 after CRT that underwent surgery. For each patient we determined response pattern, size of the largest vital tumor fragment or bulk and the shortest distance from the vital tumor to the luminal surface. The response pattern was shrinkage in 14 patients and fragmentation in 16 patients. For both groups combined, the largest vital tumor fragment per patient was smaller than 1mm for 38% of patients, below 0.2mm for 12% of patients and for one patient as small as 0.06mm. For 29% of patients the vital tumor remnant was present within the first 0.01mm from the luminal surface and for 87% within 0.5mm. Our results explain why it is difficult to differentiate between a good clinical and complete response in rectal cancer patients using endoscopy and MRI, since in many patients submillimeter tumor fragments remain below the luminal surface. To detect residual vital tumor tissue in all patients included in this study a technique with a spatial resolution of 0.06mm and an imaging depth of 8.9mm would have been required. Optical imaging techniques offer the possibility of detecting majority of these cases due to the potential of both high-resolution imaging and enhanced contrast between tissue types. These techniques could thus serve as a complimentary tool to conventional methods for rectal cancer response assessment.

Optical Coherence Tomography Angiography of the Intestine: How to Prevent Motion Artifacts in Open and Laparoscopic Surgery?

(1) Introduction. The problem that limits the intraoperative use of OCTA for the intestinal circulation diagnostics is the low informative value of OCTA images containing too many motion artifacts. The aim of this study is to evaluate the efficiency and safety of the developed unit for the prevention of the appearance of motion artifacts in the OCTA images of the intestine in both open and laparoscopic surgery in the experiment; (2) Methods. A high-speed spectral-domain multimodal optical coherence tomograph (IAP RAS, Russia) operating at a wavelength of 1310 nm with a spectral width of 100 μm and a power of 2 mW was used. The developed unit was tested in two groups of experimental animals—on minipigs (group I, n = 10, open abdomen) and on rabbits (group II, n = 10, laparoscopy). Acute mesenteric ischemia was modeled and then 1 h later the small intestine underwent OCTA evaluation. A total of 400 OCTA images of the intact and ischemic small intestine were obtained and analyzed. The quality of the obtained OCTA images was evaluated based on the score proposed in 2020 by the group of Magnin M. (3) Results. Without stabilization, OCTA images of the intestine tissues were informative only in 32–44% of cases in open surgery and in 14–22% of cases in laparoscopic surgery. A vacuum bowel stabilizer with a pressure deficit of 22–25 mm Hg significantly reduced the number of motion artifacts. As a result, the proportion of informative OCTA images in open surgery increased up to 86.5% (Χ2 = 200.2, p = 0.001), and in laparoscopy up to 60% (Χ2 = 148.3, p = 0.001). (4) Conclusions. The used vacuum tissue stabilizer enabled a significant increase in the proportion of informative OCTA images by significantly reducing the motion artifacts.

A High-Speed Hyperspectral Laparoscopic Imaging System

Minimally invasive surgery (MIS) has expanded broadly in the field of abdominal and pelvic surgery. Laparoscopic and robotic surgery has improved surgeon ergonomics, instrument precision, operative time, and postoperative recovery across various abdominal procedures. The goal of this study is to establish the feasibility of implementing high-speed hyperspectral imaging into a standard laparoscopic setup and exploring its benefit to common intracorporeal procedures. A hyperspectral laparoscopic imaging system was constructed using a customized hyperspectral camera alongside a standard rigid laparoscope and was validated for both spectral and spatial accuracy. Demosaicing methods were investigated for improved full-resolution visualization. Hyperspectral cameras with different spectral ranges were considered and compared with one another alongside two different light sources to determine the most effective configuration. Finally, different porcine tissues were imaged ex-vivo to test the capabilities of the system and spectral footprints of the various tissues were extracted. The tissue was also imaged in a phantom to simulate the system's use in MIS. The results demonstrated a hyperspectral laparoscopic imaging system that could provide quantitative, diagnostic information while not disrupting normal workflow nor adding excessive weight to the laparoscopic setup. The high-speed hyperspectral laparoscopic imaging system can have immediate applications in image-guided surgery.

In vivo evaluation of a hyperspectral imaging system for minimally invasive surgery (HSI-MIS)

BACKGROUND

Hyperspectral Imaging (HSI) is a reliable and safe imaging method for taking intraoperative perfusion measurements. This is the first study translating intraoperative HSI to an in vivo laparoscopic setting using a CE-certified HSI-system for minimally invasive surgery (HSI-MIS). We aim to compare it to an established HSI-system for open surgery (HSI-Open).

METHODS

Intraoperative HSI was done using the HSI-MIS and HSI-Open at the Region of Interest (ROI). 19 patients undergoing gastrointestinal resections were analyzed in this study. The HSI-MIS-acquired images were aligned with those from the HSI-Open, and spectra and parameter images were compared pixel-wise. We calculated the Mean Absolute Error (MAE) for Tissue Oxygen Saturation (StO₂), Near-Infrared Perfusion Index (NIR-PI), Tissue Water Index (TWI), and Organ Hemoglobin Index (OHI), as well as the Root Mean Squared Error (RMSE) over the whole spectrum. Our analysis of parameters was optimized using partial least squares (PLS) regression. Two experienced surgeons carried out an additional color-change analysis, comparing the ROI images and deciding whether they provided the same (acceptable) or different visual information (rejected).

RESULTS

HSI and subsequent image registration was possible in 19 patients. MAE results for the original calculation were StO_{2 orig.} 17.2% (± 7.7%)_, NIR-PI_orig. 16.0 (± 9.5), TWI_orig. 18.1 (± 7.9), OHI_orig. 14.4 (± 4.5). For the PLS calculation, they were StO_{2 PLS} 12.6% (± 5.2%), NIR-PI_PLS 10.3 (± 6.0), TWI_PLS 10.6 (± 5.1), and OHI_PLS 11.6 (± 3.0). The RMSE between both systems was 0.14 (± 0.06). In the color-change analysis; both surgeons accepted more images generated using the PLS method.

CONCLUSION

Intraoperative HSI-MIS is a new technology and holds great potential for future applications in surgery. Parameter deviations are attributable to technical differences and can be reduced by applying improved calculation methods. This study is an important step toward the clinical implementation of HSI for minimally invasive surgery.

Hyperspectral Imaging-A Novel Tool to Assess Tissue Perfusion and Oxygenation in Esophageal Anastomoses

Anastomotic stricture and leakage are common complications after repair of esophageal atresia (EA). A compromised perfusion of the anastomosis is a contributing factor. Hyperspectral imaging (HSI) is an ultrashort noninvasive method to measure tissue perfusion. We present two cases of with tracheoesophageal fistula (TEF)/EA repair, in whom we applied HSI: the first patient was a newborn with EA type C who underwent open TEF repair. The second one had an EA type A and cervical esophagostomy, in whom we performed gastric transposition. In both patients, HSI confirmed a good tissue perfusion of the later anastomosis. The postoperative course was uneventful and both patients are on full enteral feeds. We conclude that HSI is a safe and noninvasive tool that allows near real-time assessment of tissue perfusion and can contribute to the identification of the optimal anastomotic region during pediatric esophageal surgery.

Imaging perfusion changes in oncological clinical applications by hyperspectral imaging: a literature review

BACKGROUND

Hyperspectral imaging (HSI) is a promising imaging modality that uses visible light to obtain information about blood flow. It has the distinct advantage of being noncontact, nonionizing, and noninvasive without the need for a contrast agent. Among the many applications of HSI in the medical field are the detection of various types of tumors and the evaluation of their blood flow, as well as the healing processes of grafts and wounds. Since tumor perfusion is one of the critical factors in oncology, we assessed the value of HSI in quantifying perfusion changes during interventions in clinical oncology through a systematic review of the literature.

MATERIALS AND METHODS

The PubMed and Web of Science electronic databases were searched using the terms "hyperspectral imaging perfusion cancer" and "hyperspectral imaging resection cancer". The inclusion criterion was the use of HSI in clinical oncology, meaning that all animal, phantom, ex vivo, experimental, research and development, and purely methodological studies were excluded.

RESULTS

Twenty articles met the inclusion criteria. The anatomic locations of the neoplasms in the selected articles were as follows: kidneys (1 article), breasts (2 articles), eye (1 article), brain (4 articles), entire gastrointestinal (GI) tract (1 article), upper GI tract (5 articles), and lower GI tract (6 articles).

CONCLUSIONS

HSI is a potentially attractive imaging modality for clinical application in oncology, with assessment of mastectomy skin flap perfusion after reconstructive breast surgery and anastomotic perfusion during reconstruction of gastrointenstinal conduit as the most promising at present.