Laparoscopic system for simultaneous high-resolution video and rapid hyperspectral imaging in the visible and near-infrared spectral range

The SPECTRAL Perfusion Arm Clamping dAtaset (SPECTRALPACA) for video-rate functional imaging of the skin

Spectral imaging has the potential to become a key technique in interventional medicine as it unveils much richer optical information compared to conventional RBG (red, green, and blue)-based imaging. Thus allowing for high-resolution functional tissue analysis in real time. Its higher information density particularly shows promise for the development of powerful perfusion monitoring methods for clinical use. However, even though in vivo validation of such methods is crucial for their clinical translation, the biomedical field suffers from a lack of publicly available datasets for this purpose. Closing this gap, we generated the SPECTRAL Perfusion Arm Clamping dAtaset (SPECTRALPACA). It comprises ten spectral videos (∼20 Hz, approx. 20,000 frames each) systematically recorded of the hands of ten healthy human participants in different functional states. We paired each spectral video with concisely tracked regions of interest, and corresponding diffuse reflectance measurements recorded with a spectrometer. Providing the first openly accessible in human spectral video dataset for perfusion monitoring, our work facilitates the development and validation of new functional imaging methods.

Analysis of cellular response to drugs with a microfluidic single-cell platform based on hyperspectral imaging

BACKGROUND

Cellular response to pharmacological action of drugs is significant for drug development. Traditional detection method for cellular response to drugs normally rely on cell proliferation assay and metabolomics examination. In principle, these analytical methods often required cell labeling, invasion analysis, and hours of co-culture with drugs, which are relatively complex and time-consuming. Moreover, these methods can only indicate the drug effectiveness on cell colony rather than single cells. Thus, to meet the requirements of personal precision medicine, the development of drug response analysis on the high resolution of single cell is demanded.

RESULTS

To provide precise result for drug response on single-cell level, a microfluidic platform coupled with the label-free hyperspectral imaging was developed. With the help of horizontal single-cell trapping sieves, hundreds of single cells were trapped independently in microfluidic channels for the purposes of real-time drug delivery and single-cell hyperspectral image recording. To significantly identify the cellular hyperspectral change after drug stimulation, the differenced single-cell spectrum was proposed. Compared with the deep learning classification method based on hyperspectral images, an optimal performance can be achieved by the classification strategy based on differenced spectra. And the cellular response to different reagents, for example, K+, Epidermal Growth Factor (EGF), and Gefitinib at different concentrations can be accurately characterized by the differenced single-cell spectra analysis.

SIGNIFICANCE AND NOVELTY

The high-throughput, rapid analysis of cellular response to drugs at the single-cell level can be accurately performed by our platform. After systematically analyzing the materials and the structures of the single-cell microfluidic chip, the optimal single-cell trapping method was proposed to contribute to the further application of hyperspectral imaging on microfluidic single-cell analysis. And the hyperspectral characterization of single-cell with cancer drug stimulation proved the application potential of our method in personal cancer medication.

Intraoperative Laparoscopic Hyperspectral Imaging during Esophagectomy-A Pilot Study Evaluating Esophagogastric Perfusion at the Anastomotic Sites

Hyperspectral imaging (HSI) is a non-invasive and contactless technique that enables the real-time acquisition of comprehensive information on tissue within the surgical field. In this pilot study, we investigated whether a new HSI system for minimally-invasive surgery, TIVITA® Mini (HSI-MIS), provides reliable insights into tissue perfusion of the proximal and distal esophagogastric anastomotic sites during 21 laparoscopic/thoracoscopic or robotic Ivor Lewis esophagectomies of patients with cancer to minimize the risk of dreaded anastomotic insufficiency. In this pioneering investigation, physiological tissue parameters were derived from HSI measurements of the proximal site of the anastomosis (esophageal stump) and the distal site of the anastomosis (tip of the gastric conduit) during the thoracic phase of the procedure. Tissue oxygenation (StO2), Near Infrared Perfusion Index (NIR-PI), and Tissue Water Index (TWI) showed similar median values at both anastomotic sites. Significant differences were observed only for NIR-PI (median: 76.5 vs. 63.9; p = 0.012) at the distal site (gastric conduit) compared to our previous study using an HSI system for open surgery. For all 21 patients, reliable and informative measurements were attainable, confirming the feasibility of HSI-MIS to assess anastomotic viability. Further studies on the added benefit of this new technique aiming to reduce anastomotic insufficiency are warranted.

A dual-camera hyperspectral laparoscopic imaging system

Minimally invasive surgery (MIS) has expanded broadly in the field of abdominal and pelvic surgery. However, there are still prevalent issues surrounding intracorporeal surgery, such as iatrogenic injury, anastomotic leakage, or the presence of positive tumor margins after resection. Current approaches to address these issues and advance laparoscopic imaging techniques often involve fluorescence imaging agents, such as indocyanine green (ICG), to improve visualization, but these have drawbacks. Hyperspectral imaging (HSI) is an emerging optical imaging modality that takes advantage of spectral characteristics of different tissues. Various applications include tissue classification and digital pathology. In this study, we developed a dual-camera system for high-speed hyperspectral imaging. This includes the development of a custom application interface and corresponding hardware setup. Characterization of the system was performed, including spectral accuracy and spatial resolution, showing little sacrifice in speed for the approximate doubling of the covered spectral range, with our system acquiring 29 spectral images from 460-850 nm. Reference color tiles with various reflectance profiles were imaged and a RMSE of 3.56 ± 1.36% was achieved. Sub-millimeter resolution was shown at 7 cm working distance for both hyperspectral cameras. Finally, we image ex vivo tissues, including porcine stomach, liver, intestine, and kidney with our system and use a high-resolution, radiometrically calibrated spectrometer for comparison and evaluation of spectral fidelity. The dual-camera hyperspectral laparoscopic imaging system can have immediate applications in various surgeries.

Switchable LED-based laparoscopic multispectral system for rapid high-resolution perfusion imaging

Significance

Multispectral imaging (MSI) is an approach for real-time, quantitative, and non-invasive tissue perfusion measurements. Current laparoscopic systems based on mosaic sensors or filter wheels lack high spatial resolution or acceptable frame rates.

Aim

To develop a laparoscopic system for MSI-based color video and tissue perfusion imaging during gastrointestinal surgery without compromising spatial or temporal resolution.

Approach

The system was built with 14 switchable light-emitting diodes in the visible and near-infrared spectral range, a 4K image sensor, and a 10 mm laparoscope. Illumination patterns were created for tissue oxygenation and hemoglobin content monitoring. The system was calibrated to a clinically approved laparoscopic hyperspectral system using linear regression models and evaluated in an occlusion study with 36 volunteers.

Results

The root mean squared errors between the MSI and reference system were 0.073 for hemoglobin content, 0.039 for oxygenation in deeper tissue layers, and 0.093 for superficial oxygenation. The spatial resolution at a working distance of 45 mm was 156    μ m . The effective frame rate was 20 fps.

Conclusions

High-resolution perfusion monitoring was successfully achieved. Hardware optimizations will increase the frame rate. Parameter optimizations through alternative illumination patterns, regression, or assumed tissue models are planned. Intraoperative measurements must confirm the suitability during surgery.

Blood cancer diagnosis using hyperspectral imaging combined with the forward searching method and machine learning

Hyperspectral imaging (HSI), a widely used biosensing technique, has been applied to tumor detection. Rapid, accurate, and low-cost detection of blood cancer using hyperspectral technology remains a challenge. We developed a new method to discriminate blood cancer using hyperspectral imaging (HSI) and the forward searching method (FSM). Four commonly used classification models are applied for four types of blood cancer spectra recognition. The support vector machine (SVM) model with the highest recognition accuracy (94.5%) combined with HSI achieves high-precision tumor identification. For higher recognition accuracy and lower hardware barriers, based on the selection probabilities of spectral lines calculated by a multi-objective atomic orbital search method, the FSM is proposed for HSI feature selection. With the proposed method, the wavelength band range of the spectrum is reduced by at least 50%. Compared with the traditional dimensionality reduction methods, the FSM can obtain a higher accuracy rate with lower hardware requirements. These results show that our proposed method can achieve non-invasive rapid screening of blood cancers with lower hardware requirements. Therefore, HSI assisted with FSM and SVM hybrid models can be a powerful and promising tool for blood cancer detection.

Compact and ultracompact spectral imagers: technology and applications in biomedical imaging

Significance

Spectral imaging, which includes hyperspectral and multispectral imaging, can provide images in numerous wavelength bands within and beyond the visible light spectrum. Emerging technologies that enable compact, portable spectral imaging cameras can facilitate new applications in biomedical imaging.

Aim

With this review paper, researchers will (1) understand the technological trends of upcoming spectral cameras, (2) understand new specific applications that portable spectral imaging unlocked, and (3) evaluate proper spectral imaging systems for their specific applications.

Approach

We performed a comprehensive literature review in three databases (Scopus, PubMed, and Web of Science). We included only fully realized systems with definable dimensions. To best accommodate many different definitions of "compact," we included a table of dimensions and weights for systems that met our definition.

Results

There is a wide variety of contributions from industry, academic, and hobbyist spaces. A variety of new engineering approaches, such as Fabry-Perot interferometers, spectrally resolved detector array (mosaic array), microelectro-mechanical systems, 3D printing, light-emitting diodes, and smartphones, were used in the construction of compact spectral imaging cameras. In bioimaging applications, these compact devices were used for in vivo and ex vivo diagnosis and surgical settings.

Conclusions

Compact and ultracompact spectral imagers are the future of spectral imaging systems. Researchers in the bioimaging fields are building systems that are low-cost, fast in acquisition time, and mobile enough to be handheld.

Spectral imaging enables contrast agent-free real-time ischemia monitoring in laparoscopic surgery

Laparoscopic surgery has evolved as a key technique for cancer diagnosis and therapy. While characterization of the tissue perfusion is crucial in various procedures, such as partial nephrectomy, doing so by means of visual inspection remains highly challenging. We developed a laparoscopic real-time multispectral imaging system featuring a compact and lightweight multispectral camera and the possibility to complement the conventional surgical view of the patient with functional information at a video rate of 25 Hz. To enable contrast agent-free ischemia monitoring during laparoscopic partial nephrectomy, we phrase the problem of ischemia detection as an out-of-distribution detection problem that does not rely on data from any other patient and uses an ensemble of invertible neural networks at its core. An in-human trial demonstrates the feasibility of our approach and highlights the potential of spectral imaging combined with advanced deep learning-based analysis tools for fast, efficient, reliable, and safe functional laparoscopic imaging.

A High-Speed Hyperspectral Laparoscopic Imaging System

Minimally invasive surgery (MIS) has expanded broadly in the field of abdominal and pelvic surgery. Laparoscopic and robotic surgery has improved surgeon ergonomics, instrument precision, operative time, and postoperative recovery across various abdominal procedures. The goal of this study is to establish the feasibility of implementing high-speed hyperspectral imaging into a standard laparoscopic setup and exploring its benefit to common intracorporeal procedures. A hyperspectral laparoscopic imaging system was constructed using a customized hyperspectral camera alongside a standard rigid laparoscope and was validated for both spectral and spatial accuracy. Demosaicing methods were investigated for improved full-resolution visualization. Hyperspectral cameras with different spectral ranges were considered and compared with one another alongside two different light sources to determine the most effective configuration. Finally, different porcine tissues were imaged ex-vivo to test the capabilities of the system and spectral footprints of the various tissues were extracted. The tissue was also imaged in a phantom to simulate the system's use in MIS. The results demonstrated a hyperspectral laparoscopic imaging system that could provide quantitative, diagnostic information while not disrupting normal workflow nor adding excessive weight to the laparoscopic setup. The high-speed hyperspectral laparoscopic imaging system can have immediate applications in image-guided surgery.

In vivo evaluation of a hyperspectral imaging system for minimally invasive surgery (HSI-MIS)

BACKGROUND

Hyperspectral Imaging (HSI) is a reliable and safe imaging method for taking intraoperative perfusion measurements. This is the first study translating intraoperative HSI to an in vivo laparoscopic setting using a CE-certified HSI-system for minimally invasive surgery (HSI-MIS). We aim to compare it to an established HSI-system for open surgery (HSI-Open).

METHODS

Intraoperative HSI was done using the HSI-MIS and HSI-Open at the Region of Interest (ROI). 19 patients undergoing gastrointestinal resections were analyzed in this study. The HSI-MIS-acquired images were aligned with those from the HSI-Open, and spectra and parameter images were compared pixel-wise. We calculated the Mean Absolute Error (MAE) for Tissue Oxygen Saturation (StO₂), Near-Infrared Perfusion Index (NIR-PI), Tissue Water Index (TWI), and Organ Hemoglobin Index (OHI), as well as the Root Mean Squared Error (RMSE) over the whole spectrum. Our analysis of parameters was optimized using partial least squares (PLS) regression. Two experienced surgeons carried out an additional color-change analysis, comparing the ROI images and deciding whether they provided the same (acceptable) or different visual information (rejected).

RESULTS

HSI and subsequent image registration was possible in 19 patients. MAE results for the original calculation were StO_{2 orig.} 17.2% (± 7.7%)_, NIR-PI_orig. 16.0 (± 9.5), TWI_orig. 18.1 (± 7.9), OHI_orig. 14.4 (± 4.5). For the PLS calculation, they were StO_{2 PLS} 12.6% (± 5.2%), NIR-PI_PLS 10.3 (± 6.0), TWI_PLS 10.6 (± 5.1), and OHI_PLS 11.6 (± 3.0). The RMSE between both systems was 0.14 (± 0.06). In the color-change analysis; both surgeons accepted more images generated using the PLS method.

CONCLUSION

Intraoperative HSI-MIS is a new technology and holds great potential for future applications in surgery. Parameter deviations are attributable to technical differences and can be reduced by applying improved calculation methods. This study is an important step toward the clinical implementation of HSI for minimally invasive surgery.

Hyperspectral Imaging-A Novel Tool to Assess Tissue Perfusion and Oxygenation in Esophageal Anastomoses

Anastomotic stricture and leakage are common complications after repair of esophageal atresia (EA). A compromised perfusion of the anastomosis is a contributing factor. Hyperspectral imaging (HSI) is an ultrashort noninvasive method to measure tissue perfusion. We present two cases of with tracheoesophageal fistula (TEF)/EA repair, in whom we applied HSI: the first patient was a newborn with EA type C who underwent open TEF repair. The second one had an EA type A and cervical esophagostomy, in whom we performed gastric transposition. In both patients, HSI confirmed a good tissue perfusion of the later anastomosis. The postoperative course was uneventful and both patients are on full enteral feeds. We conclude that HSI is a safe and noninvasive tool that allows near real-time assessment of tissue perfusion and can contribute to the identification of the optimal anastomotic region during pediatric esophageal surgery.

Deep Learning in Medical Hyperspectral Images: A Review

With the continuous progress of development, deep learning has made good progress in the analysis and recognition of images, which has also triggered some researchers to explore the area of combining deep learning with hyperspectral medical images and achieve some progress. This paper introduces the principles and techniques of hyperspectral imaging systems, summarizes the common medical hyperspectral imaging systems, and summarizes the progress of some emerging spectral imaging systems through analyzing the literature. In particular, this article introduces the more frequently used medical hyperspectral images and the pre-processing techniques of the spectra, and in other sections, it discusses the main developments of medical hyperspectral combined with deep learning for disease diagnosis. On the basis of the previous review, tne limited factors in the study on the application of deep learning to hyperspectral medical images are outlined, promising research directions are summarized, and the future research prospects are provided for subsequent scholars.

Intraoperative bowel perfusion quantification with hyperspectral imaging: a guidance tool for precision colorectal surgery

BACKGROUND

Poor anastomotic perfusion can cause anastomotic leaks (AL). Hyperspectral imaging (HSI), previously validated experimentally, provides accurate, real-time, contrast-free intestinal perfusion quantification. Clinical experience with HSI is limited. In this study, HSI was used to evaluate bowel perfusion intraoperatively.

METHODS

Fifty-two patients undergoing elective colorectal surgeries for neoplasia (n = 40) or diverticular disease (n = 12), were enrolled. Intestinal perfusion was assessed with HSI (TIVITA®, Diaspective Vision, Am Salzhaff, Germany). This device generates a perfusion heat map reflecting the tissue oxygen saturation (StO2) amount. Prior to anastomose creation, the clinical transection line (CTL) was highlighted on the proximal bowel and imaged with HSI. Upon StO2 heat map evaluation, the hyperspectral transection line (HTL) was identified. In case of CTL/HTL discrepancy > 5 mm, the bowel was always resected at the HTL. HSI outcomes were compared to the clinical ones.

RESULTS

AL occurred in one patient who underwent neoadjuvant radiochemotherapy and ultralow anterior resection for rectal cancer. HSI assessment was feasible in all patients, and StO2-values were significantly higher at proximal segments than distal ones. Twenty-six patients showed CTL/HTL discrepancy, and these patients had a lower mean StO2 (54.55 ± 21.30%) than patients without discrepancy (65.10 ± 21.30%, p = 0.000). Patients undergoing neoadjuvant radiochemotherapy showed a lower StO2 (51.41 ± 23.41%) than non-neoadjuvated patients (60.51 ± 24.98%, p = 0.010).

CONCLUSION

HSI is useful in detecting intraoperatively marginally perfused segments, for which the clinical appreciation is unreliable. Intestinal vascular supply is lower in patients undergoing neoadjuvant radiochemotherapy, and this novel finding together with the clinical impact of HSI perfusion quantification deserves further investigation in larger trials.

Comparison of image registration methods for combining laparoscopic video and spectral image data

Laparoscopic procedures can be assisted by intraoperative modalities, such as quantitative perfusion imaging based on fluorescence or hyperspectral data. If these modalities are not available at video frame rate, fast image registration is needed for the visualization in augmented reality. Three feature-based algorithms and one pre-trained deep homography neural network (DH-NN) were tested for single and multi-homography estimation. Fine-tuning was used to bridge the domain gap of the DH-NN for non-rigid registration of laparoscopic images. The methods were validated on two datasets: an open-source record of 750 manually annotated laparoscopic images, presented in this work, and in-vivo data from a novel laparoscopic hyperspectral imaging system. All feature-based single homography methods outperformed the fine-tuned DH-NN in terms of reprojection error, Structural Similarity Index Measure, and processing time. The feature detector and descriptor ORB1000 enabled video-rate registration of laparoscopic images on standard hardware with submillimeter accuracy.

[Artificial intelligence and hyperspectral imaging for image-guided assistance in minimally invasive surgery]

BACKGROUND

Intraoperative imaging assists surgeons during minimally invasive procedures. Hyperspectral imaging (HSI) is a noninvasive and noncontact optical technique with great diagnostic potential in medicine. The combination with artificial intelligence (AI) approaches to analyze HSI data is called intelligent HSI in this article.

OBJECTIVE

What are the medical applications and advantages of intelligent HSI for minimally invasive visceral surgery?

MATERIAL AND METHODS

Within various clinical studies HSI data from multiple in vivo tissue types and oncological resections were acquired using an HSI camera system. Different AI algorithms were evaluated for detection and discrimination of organs, risk structures and tumors.

RESULTS

In an experimental animal study 20 different organs could be differentiated with high precision (> 95%) using AI. In vivo, the parathyroid glands could be discriminated from surrounding tissue with an F1 score of 47% and sensitivity of 75%, and the bile duct with an F1 score of 79% and sensitivity of 90%. Furthermore, ex vivo tumor tissue could be successfully detected with an area under the receiver operating characteristic (ROC) curve (AUC) larger than 0.91.

DISCUSSION

This study demonstrates that intelligent HSI can automatically and accurately detect different tissue types. Despite great progress in the last decade intelligent HSI still has limitations. Thus, accurate AI algorithms that are easier to understand for the user and an extensive standardized and continuously growing database are needed. Further clinical studies should support the various medical applications and lead to the adoption of intelligent HSI in the clinical routine practice.

Hyperspectral Evaluation of the Human Liver During Major Resection

Objective

This study investigates the effects of PVE and vascular inflow control (VIC) on liver microperfusion and tissue oxygenation using hyperspectral imaging (HSI) technology.

Background

Mechanisms triggering future liver remnant (FLR) augmentation introduced by PVE have not been sufficiently studied in humans. Particularly, the arterial buffer response (ABR) of the liver might play a vital role.

Methods

Hyperspectral datacubes (TIVITA) acquired during 58 major liver resections were qualitatively and quantitatively analyzed for tissue oxygenation (StO2%), near-infrared (NIR) perfusion, organ-hemoglobin indices (OHI), and tissue-water indices (TWI). The primary study endpoint was measurement of hyperspectral differences in liver parenchyma subject to PVE and VIC before resection.

Results

HSI revealed parenchyma specific differences in StO2% with regard to the underlying disease (P < 0.001). Preoperative PVE (n = 23, 40%) lead to arterial hyperoxygenation and hyperperfusion of corresponding liver segments (StO2: 77.23% ± 11.93%, NIR: 0.46 ± 0.20[I]) when compared with the FLR (StO2: 66.13% ± 9.96%, NIR: 0.23 ± 0.12[I]; P < 0.001). In a case of insufficient PVE and the absence of FLR augmentation hyperspectral StO2 and NIR differences were absent. The hyperspectral assessment demonstrated increased liver tissue-oxygenation and perfusion in PVE-segments (n = 23 cases) and decreased total VIC in nonembolized FLR hemilivers (n = 35 cases; P < 0.001). Intraoperative HSI analysis of tumor tissue revealed marked tumor specific differences in StO2, NIR, OHI, and TWI (P < 0.001).

Conclusions

HSI allows intraoperative quantitative and qualitative assessment of microperfusion and StO2% of liver tissue. PVE lead to ABR-triggered tissue hyperoxygenation and cross-talk FLR augmentation. HSI furthermore facilitates intraoperative tumor tissue identification and enables image-guided liver surgery following VIC.

Video: Clinical evaluation of a laparoscopic hyperspectral imaging system

Background

Hyperspectral imaging (HSI) during surgical procedures is a new method for perfusion quantification and tissue discrimination. Its use has been limited to open surgery due to large camera sizes, missing color video, or long acquisition times. A hand-held, laparoscopic hyperspectral camera has been developed now to overcome those disadvantages and evaluated clinically for the first time.

Methods

In a clinical evaluation study, gastrointestinal resectates of ten cancer patients were investigated using the laparoscopic hyperspectral camera. Reference data from corresponding anatomical regions were acquired with a clinically approved HSI system. An image registration process was executed that allowed for pixel-wise comparisons of spectral data and parameter images (StO₂: oxygen saturation of tissue, NIR PI: near-infrared perfusion index, OHI: organ hemoglobin index, TWI: tissue water index) provided by both camera systems. The mean absolute error (MAE) and root mean square error (RMSE) served for the quantitative evaluations. Spearman’s rank correlation between factors related to the study design like the time of spectral white balancing and MAE, respectively RMSE, was calculated.

Results

The obtained mean MAEs between the TIVITA® Tissue and the laparoscopic hyperspectral system resulted in StO₂: 11% ± 7%, NIR PI: 14±3, OHI: 14± 5, and TWI: 10 ± 2. The mean RMSE between both systems was 0.1±0.03 from 500 to 750 nm and 0.15 ±0.06 from 750 to 1000 nm. Spearman’s rank correlation coefficients showed no significant correlation between MAE or RMSE and influencing factors related to the study design.

Conclusion

Qualitatively, parameter images of the laparoscopic system corresponded to those of the system for open surgery. Quantitative deviations were attributed to technical differences rather than the study design. Limitations of the presented study are addressed in current large-scale in vivo trials.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00464-022-09282-y.

Combined indocyanine green and quantitative perfusion assessment with hyperspectral imaging during colorectal resections

Anastomotic insufficiencies still represent one of the most severe complications in colorectal surgery. Since tissue perfusion highly affects anastomotic healing, its objective assessment is an unmet clinical need. Indocyanine green-based fluorescence angiography (ICG-FA) and hyperspectral imaging (HSI) have received great interest in recent years but surgeons have to decide between both techniques. For the first time, two data processing pipelines capable of reconstructing an ICG-FA correlating signal from hyperspectral data were developed. Results were technically evaluated and compared to ground truth data obtained during colorectal resections. In 87% of 46 data sets, the reconstructed images resembled the ground truth data. The combined applicability of ICG-FA and HSI within one imaging system might provide supportive and complementary information about tissue vascularization, shorten surgery time, and reduce perioperative mortality.

Colorectal cancer: Artificial intelligence and its role in surgical decision making

Despite several advances in the oncological management of colorectal cancer (CRC), there still remains a lacuna in the treatment strategy, which differs from center to center and on the philosophy of the treating clinician that is not without bias. Personalized treatment is essential for the treatment of CRC to achieve better long-term outcomes and to reduce morbidity. Surgery has an important role to play in the treatment. Surgical treatment of CRC is decided based on clinical parameters and investigations and hence likely to have judgmental errors. Artificial intelligence has been reported to be useful in the surveillance, diagnosis, treatment, and follow-up with accuracy in several malignancies. However, it is still evolving and yet to be established in surgical decision making in CRC. It is not only useful preoperatively but also intraoperatively. Artificial intelligence helps to rectify the human surgical decision when clinical data and radiological and laboratory parameters are fed into the computer and may guide correct surgical treatment.

Border Line Definition Using Hyperspectral Imaging in Colorectal Resections

Simple Summary

Good oxygenation of both bowel ends is an important prerequisite to promote anastomotic healing after colorectal resections. Bowel oxygenation is usually assessed clinically. Hyperspectral imaging is a contactless and contrast-free tool that allows quantifying tissue oxygen intraoperatively. In this study, the results of 105 colorectal resections with hyperspectral imaging are reported.

Abstract

Background: A perfusion deficit is a well-defined and intraoperatively influenceable cause of anastomotic leak (AL). Current intraoperative perfusion assessment methods do not provide objective and quantitative results. In this study, the ability of hyperspectral imaging (HSI) to quantify tissue oxygenation intraoperatively was assessed. Methods: 115 patients undergoing colorectal resections were included in the final analysis. Before anastomotic formation, the bowel was extracted and the resection line was outlined and imaged using a compact HSI camera, in order to provide instantaneously quantitative perfusion assessment. Results: In 105 patients, a clear demarcation line was visible with HSI one minute after marginal artery transection, reaching a plateau after 3 min. In 58 (55.2%) patients, the clinically determined transection line matched with HSI. In 23 (21.9%) patients, the clinically established resection margin was entirely within the less perfused area. In 24 patients (22.8%), the HSI transection line had an irregular course and crossed the clinically established resection line. In four cases, HSI disclosed a clinically undetected lesion of the marginal artery. Conclusions: Intraoperative HSI is safe, well reproducible, and does not disrupt the surgical workflow. It also quantifies bowel surface perfusion. HSI might become an intraoperative guidance tool, potentially preventing postoperative complications.

New Intraoperative Imaging Tools and Image-Guided Surgery in Gastric Cancer Surgery

Innovations and new advancements in intraoperative real-time imaging have gained significant importance in the field of gastric cancer surgery in the recent past. Currently, the most promising procedures include indocyanine green fluorescence imaging (ICG-FI) and hyperspectral imaging or multispectral imaging (HSI, MSI). ICG-FI is utilized in a broad range of clinical applications, e.g., assessment of perfusion or lymphatic drainage, and additional implementations are currently investigated. HSI is still in the experimental phase and its value and clinical relevance require further evaluation, but initial studies have shown a successful application in perfusion assessment, and prospects concerning non-invasive tissue and tumor classification are promising. The application of machine learning and artificial intelligence technologies might enable an automatic evaluation of the acquired image data in the future. Both methods facilitate the accurate visualization of tissue characteristics that are initially indistinguishable for the human eye. By aiding surgeons in optimizing the surgical procedure, image-guided surgery can contribute to the oncologic safety and reduction of complications in gastric cancer surgery and recent advances hold promise for the application of HSI in intraoperative tissue diagnostics.

Intraoperative hyperspectral imaging (HSI) as a new diagnostic tool for the detection of cartilage degeneration

To investigate, whether hyperspectral imaging (HSI) is able to reliably differentiate between healthy and damaged cartilage tissue. A prospective diagnostic study was performed including 21 patients undergoing open knee surgery. HSI data were acquired during surgery, and the joint surface's cartilage was assessed according to the ICRS cartilage injury score. The HSI system records light spectra from 500 to 1000 nm and generates several parameters including tissue water index (TWI) and the absorbance at 960 nm and 540 nm. Receiver operating characteristic curves were calculated to assess test parameters for threshold values of HSI. Areas with a cartilage defect ICRS grade ≥ 3 showed a significantly lower TWI (p = 0.026) and higher values for 540 nm (p < 0.001). No difference was seen for 960 nm (p = 0.244). For a threshold of 540 nm > 0.74, a cartilage defect ICRS grade ≥ 3 could be detected with a sensitivity of 0.81 and a specificity of 0.81. TWI was not suitable for cartilage defect detection. HSI can provide reliable parameters to differentiate healthy and damaged cartilage. Our data clearly suggest that the difference in absorbance at 540 nm would be the best parameter to achieve accurate identification of damaged cartilage.

Novel Intraoperative Imaging of Gastric Tube Perfusion during Oncologic Esophagectomy—A Pilot Study Comparing Hyperspectral Imaging (HSI) and Fluorescence Imaging (FI) with Indocyanine Green (ICG)

Background: Novel intraoperative imaging techniques, namely, hyperspectral (HSI) and fluorescence imaging (FI), are promising with respect to reducing severe postoperative complications, thus increasing patient safety. Both tools have already been used to evaluate perfusion of the gastric conduit after esophagectomy and before anastomosis. To our knowledge, this is the first study evaluating both modalities simultaneously during esophagectomy. Methods: In our pilot study, 13 patients, who underwent Ivor Lewis esophagectomy and gastric conduit reconstruction, were analyzed prospectively. HSI and FI were recorded before establishing the anastomosis in order to determine its optimum position. Results: No anastomotic leak occurred during this pilot study. In five patients, the imaging methods resulted in a more peripheral adaptation of the anastomosis. There were no significant differences between the two imaging tools, and no adverse events due to the imaging methods or indocyanine green (ICG) injection occurred. Conclusions: Simultaneous intraoperative application of both modalities was feasible and not time consuming. They are complementary with regard to the ideal anastomotic position and may contribute to better surgical outcomes. The impact of their simultaneous application will be proven in consecutive prospective trials with a large patient cohort.

Intraoperative Guidance Using Hyperspectral Imaging: A Review for Surgeons

Hyperspectral imaging (HSI) is a novel optical imaging modality, which has recently found diverse applications in the medical field. HSI is a hybrid imaging modality, combining a digital photographic camera with a spectrographic unit, and it allows for a contactless and non-destructive biochemical analysis of living tissue. HSI provides quantitative and qualitative information of the tissue composition at molecular level in a contrast-free manner, hence making it possible to objectively discriminate between different tissue types and between healthy and pathological tissue. Over the last two decades, HSI has been increasingly used in the medical field, and only recently it has found an application in the operating room. In the last few years, several research groups have used this imaging modality as an intraoperative guidance tool within different surgical disciplines. Despite its great potential, HSI still remains far from being routinely used in the daily surgical practice, since it is still largely unknown to most of the surgical community. The aim of this study is to provide clinical surgeons with an overview of the capabilities, current limitations, and future directions of HSI for intraoperative guidance.

Automatic Recognition of Colon and Esophagogastric Cancer with Machine Learning and Hyperspectral Imaging

There are approximately 1.8 million diagnoses of colorectal cancer, 1 million diagnoses of stomach cancer, and 0.6 million diagnoses of esophageal cancer each year globally. An automatic computer-assisted diagnostic (CAD) tool to rapidly detect colorectal and esophagogastric cancer tissue in optical images would be hugely valuable to a surgeon during an intervention. Based on a colon dataset with 12 patients and an esophagogastric dataset of 10 patients, several state-of-the-art machine learning methods have been trained to detect cancer tissue using hyperspectral imaging (HSI), including Support Vector Machines (SVM) with radial basis function kernels, Multi-Layer Perceptrons (MLP) and 3D Convolutional Neural Networks (3DCNN). A leave-one-patient-out cross-validation (LOPOCV) with and without combining these sets was performed. The ROC-AUC score of the 3DCNN was slightly higher than the MLP and SVM with a difference of 0.04 AUC. The best performance was achieved with the 3DCNN for colon cancer and esophagogastric cancer detection with a high ROC-AUC of 0.93. The 3DCNN also achieved the best DICE scores of 0.49 and 0.41 on the colon and esophagogastric datasets, respectively. These scores were significantly improved using a patient-specific decision threshold to 0.58 and 0.51, respectively. This indicates that, in practical use, an HSI-based CAD system using an interactive decision threshold is likely to be valuable. Experiments were also performed to measure the benefits of combining the colorectal and esophagogastric datasets (22 patients), and this yielded significantly better results with the MLP and SVM models.

Deep Learning Analysis of In Vivo Hyperspectral Images for Automated Intraoperative Nerve Detection

Nerves are critical structures that may be difficult to recognize during surgery. Inadvertent nerve injuries can have catastrophic consequences for the patient and lead to life-long pain and a reduced quality of life. Hyperspectral imaging (HSI) is a non-invasive technique combining photography with spectroscopy, allowing non-invasive intraoperative biological tissue property quantification. We show, for the first time, that HSI combined with deep learning allows nerves and other tissue types to be automatically recognized in in vivo hyperspectral images. An animal model was used, and eight anesthetized pigs underwent neck midline incisions, exposing several structures (nerve, artery, vein, muscle, fat, skin). State-of-the-art machine learning models were trained to recognize these tissue types in HSI data. The best model was a convolutional neural network (CNN), achieving an overall average sensitivity of 0.91 and a specificity of 1.0, validated with leave-one-patient-out cross-validation. For the nerve, the CNN achieved an average sensitivity of 0.76 and a specificity of 0.99. In conclusion, HSI combined with a CNN model is suitable for in vivo nerve recognition.

Intraoperative hyperspectral label-free imaging: from system design to first-in-patient translation

Despite advances in intraoperative surgical imaging, reliable discrimination of critical tissue during surgery remains challenging. As a result, decisions with potentially life-changing consequences for patients are still based on the surgeon's subjective visual assessment. Hyperspectral imaging (HSI) provides a promising solution for objective intraoperative tissue characterisation, with the advantages of being non-contact, non-ionising and non-invasive. However, while its potential to aid surgical decision-making has been investigated for a range of applications, to date no real-time intraoperative HSI (iHSI) system has been presented that follows critical design considerations to ensure a satisfactory integration into the surgical workflow. By establishing functional and technical requirements of an intraoperative system for surgery, we present an iHSI system design that allows for real-time wide-field HSI and responsive surgical guidance in a highly constrained operating theatre. Two systems exploiting state-of-the-art industrial HSI cameras, respectively using linescan and snapshot imaging technology, were designed and investigated by performing assessments against established design criteria and ex vivo tissue experiments. Finally, we report the use of our real-time iHSI system in a clinical feasibility case study as part of a spinal fusion surgery. Our results demonstrate seamless integration into existing surgical workflows.

[New intraoperative fluorescence-based and spectroscopic imaging techniques in visceral medicine - precision surgery in the "high tech"-operating room]

INTRODUCTION

Fluorescence angiography (FA) with indocyanine green (ICG) and hyperspectral imaging (HSI) are novel intraoperative visualization techniques in abdominal, vascular and transplant surgery. With the purpose of precision surgery, and in order to increase patient's safety, these new tools aim at reducing postoperative morbidity and mortality. This review discusses and highlights recent developments and the future potential of real-time imaging modalities.

METHODS

The underlying mechanisms of the novel imaging methods and their clinical impact are displayed in the context of avoiding anastomotic leaks, the most momentous complications in gastrointestinal surgery after oncologic resections.

RESULTS

While FA is associated with the admission of a fluorescence agent, HSI is contact-free and non-invasive. Both methods are able to record physiological tissue properties in real-time. Additionally, FA also measures dynamic phenomena. The techniques take a few seconds only and do not hamper the operative workflow considerably. With regard to a potential change of the surgical strategy, FA and HSI have an equal significance. Our own advancements reflect, in particular, the topics of data visualization and automated data analyses together with the implementation of artificial intelligence (AI) and minimalization of the current devices to install them into endoscopes, minimal-invasive and robot-guided surgery.

CONCLUSION

There are a limited number of studies in the field of intraoperative imaging techniques. Whether precision surgery in the "high-tech" OR together with FA, HSI and robotics will result in more secure operative procedures to minimize the postoperative morbidity and mortality will have to be evaluated in future multicenter trials.

Over 1000 nm Near-Infrared Multispectral Imaging System for Laparoscopic In Vivo Imaging

In this study, a laparoscopic imaging device and a light source able to select wavelengths by bandpass filters were developed to perform multispectral imaging (MSI) using over 1000 nm near-infrared (OTN-NIR) on regions under a laparoscope. Subsequently, MSI (wavelengths: 1000–1400 nm) was performed using the built device on nine live mice before and after tumor implantation. The normal and tumor pixels captured within the mice were used as teaching data sets, and the tumor-implanted mice data were classified using a neural network applied following a leave-one-out cross-validation procedure. The system provided a specificity of 89.5%, a sensitivity of 53.5%, and an accuracy of 87.8% for subcutaneous tumor discrimination. Aggregated true-positive (TP) pixels were confirmed in all tumor-implanted mice, which indicated that the laparoscopic OTN-NIR MSI could potentially be applied in vivo for classifying target lesions such as cancer in deep tissues.

[Status of minimally invasive gastrectomy : Current advancements: robotic surgery and intraoperative imaging for gastric cancer]

The surgical treatment of gastric cancer has arrived at a turning point towards the routine application of minimally invasive techniques. After the first results of prospective randomized trials from Asia confirmed the surgical and oncological safety, the latest results of international trials provided evidence for minimally invasive gastrectomy of advanced gastric cancer in a multimodal setting. A new addition in the field of minimally invasive procedures is robotic-assisted surgical techniques, which have already been implemented for these indications in many centers in Germany. The technical advantages that are applicable in the robotics setting in comparison to laparoscopy lead to a rapid dissemination of the procedure but still need to be evaluated in controlled trials. Further developments for the surgical treatment of gastric cancer are found in the field of intraoperative imaging procedures. In this field various technologies are available, such as fluorescence imaging using a near-infrared camera, which requires the use of a fluorescent agent or the hyperspectral camera system, which does not require the application of a fluorophore and merges pictures from visible and non-visible wavelengths to a functional image. It is to be expected that in the future various technological advancements can make a valuable contribution to the surgical treatment of gastric cancer in the clinical routine, especially if they support and facilitate the use of minimally invasive surgical techniques.

Feedforward Artificial Neural Network-Based Colorectal Cancer Detection Using Hyperspectral Imaging: A Step towards Automatic Optical Biopsy

Simple Summary

Detection of colorectal carcinoma is performed visually by investigators and is confirmed pathologically. With hyperspectral imaging, an expanded spectral range of optical information is now available for analysis. The acquired recordings were analyzed with a neural network, and it was possible to differentiate tumor from healthy mucosa in colorectal carcinoma by automatic classification with high reliability. Classification and visualization were performed based on a four-layer perceptron neural network. Based on a neural network, the classification of CA or AD resulted in a sensitivity of 86% and a specificity of 95%, by means of leave-one-patient-out cross-validation. Additionally, significant differences in terms of perfusion parameters (e.g., oxygen saturation) related to tumor staging and neoadjuvant therapy were observed. This is a step towards optical biopsy.

Abstract

Currently, colorectal cancer (CRC) is mainly identified via a visual assessment during colonoscopy, increasingly used artificial intelligence algorithms, or surgery. Subsequently, CRC is confirmed through a histopathological examination by a pathologist. Hyperspectral imaging (HSI), a non-invasive optical imaging technology, has shown promising results in the medical field. In the current study, we combined HSI with several artificial intelligence algorithms to discriminate CRC. Between July 2019 and May 2020, 54 consecutive patients undergoing colorectal resections for CRC were included. The tumor was imaged from the mucosal side with a hyperspectral camera. The image annotations were classified into three groups (cancer, CA; adenomatous margin around the central tumor, AD; and healthy mucosa, HM). Classification and visualization were performed based on a four-layer perceptron neural network. Based on a neural network, the classification of CA or AD resulted in a sensitivity of 86% and a specificity of 95%, by means of leave-one-patient-out cross-validation. Additionally, significant differences in terms of perfusion parameters (e.g., oxygen saturation) related to tumor staging and neoadjuvant therapy were observed. Hyperspectral imaging combined with automatic classification can be used to differentiate between CRC and healthy mucosa. Additionally, the biological changes induced by chemotherapy to the tissue are detectable with HSI.