In vivo use of hyperspectral imaging to develop a noncontact endoscopic diagnosis support system for malignant colorectal tumors

Handheld hyperspectral imaging as a tool for the post-mortem interval estimation of human skeletal remains

In forensic medicine, estimating human skeletal remains' post-mortem interval (PMI) can be challenging. Following death, bones undergo a series of chemical and physical transformations due to their interactions with the surrounding environment. Post-mortem changes have been assessed using various methods, but estimating the PMI of skeletal remains could still be improved. We propose a new methodology with handheld hyperspectral imaging (HSI) system based on the first results from 104 human skeletal remains with PMIs ranging between 1 day and 2000 years. To differentiate between forensic and archaeological bone material, the Convolutional Neural Network analyzed 65.000 distinct diagnostic spectra: the classification accuracy was 0.58, 0.62, 0.73, 0.81, and 0.98 for PMIs of 0 week-2 weeks, 2 weeks-6 months, 6 months-1 year, 1 year-10 years, and >100 years, respectively. In conclusion, HSI can be used in forensic medicine to distinguish bone materials >100 years old from those <10 years old with an accuracy of 98%. The model has adequate predictive performance, and handheld HSI could serve as a novel approach to objectively and accurately determine the PMI of human skeletal remains.

Endoscopic Hyperspectral Imaging System to Discriminate Tissue Characteristics in Tissue Phantom and Orthotopic Mouse Pancreatic Tumor Model

In this study, we developed an endoscopic hyperspectral imaging (eHSI) system and evaluated its performance in analyzing tissues within tissue phantoms and orthotopic mouse pancreatic tumor models. Our custom-built eHSI system incorporated a liquid crystal tunable filter. To assess its tissue discrimination capabilities, we acquired images of tissue phantoms, distinguishing between fat and muscle regions. The system underwent supervised training using labeled samples, and this classification model was then applied to other tissue phantom images for evaluation. In the tissue phantom experiment, the eHSI effectively differentiated muscle from fat and background tissues. The precision scores regarding fat tissue classification were 98.3% for the support vector machine, 97.7% for the neural network, and 96.0% with a light gradient-boosting machine algorithm, respectively. Furthermore, we applied the eHSI system to identify tumors within an orthotopic mouse pancreatic tumor model. The F-score of each pancreatic tumor-bearing model reached 73.1% for the KPC tumor model and 63.1% for the Pan02 tumor models. The refined imaging conditions and optimization of the fine-tuning of classification algorithms enhance the versatility and diagnostic efficacy of eHSI in biomedical applications.

Optimization of a real-time LED based spectral imaging system

We are designing a real-time spectral imaging system using micro-LEDs and a high-speed micro-camera for potential endoscopic applications. Currently, gastrointestinal (GI) endoscopic imaging has largely been limited to white light imaging (WLI), while other endoscopic approaches have seen advancements in imaging techniques including fluorescence imaging, narrow-band imaging, and stereoscopic visualization. To further advance GI endoscopic imaging, we are working towards a high-speed spectral imaging system that can be implemented with a chip-on-tip design for a flexible endoscope. Hyperspectral imaging has potential applications in a variety of imaging procedures with its ability to discern spectral footprints of the imaging field in a series of two-dimensional images at different wavelengths. For investigating the feasibility of a real-time LED-based hyperspectral imaging system for endoscopic applications we designed and developed a large-scale prototype using through-hole LEDs, which is further analyzed as we design our future system. For high quality imaging, the LED array must be designed with the specific illumination patterns and intensities of each LED considered. We present our work in optimizing our current LED array through optical simulations performed in silico. Damped least squares and orthogonal descent algorithms are implemented to maximize irradiance power and improve illumination homogeneity at several working distances by adjusting radial distances of each LED from the camera. With our prototype LED-based hyperspectral imaging system, the simulation and optimization approach achieved an average increase of 8.36 ± 8.79% in irradiance and a 4.3% decrease in standard deviation at multiple working distances and field-of-views for each LED, as compared to the original design, leading to improved image quality and maintained acquisition speeds. This work highlights the value of in silico optical simulation and provides a framework for improved optical system design and will inform design decisions in future works.

Development of a custom optical imaging system for non-invasive monitoring and delineation of lower limb varicose veins using hyperspectral imaging and quantitative phase analysis

BACKGROUND

Varicose veins (VV) are a prevalent chronic venous disorder, particularly affecting women of childbearing age. This condition is associated with significant complications, including pain, discomfort, leg cramps, ulceration, reduced quality of life, absenteeism, and even mortality. This study aims to develop a custom non-invasive, non-contact optical imaging system combined with magnitude and phase image calculation to monitor and visualize varicose veins and their tributaries using hyperspectral imaging and quantitative phase analysis with a k-means clustering algorithm.

RESULTS

Ten volunteers participated in the optical imaging system study. They were exposed to a polychromatic light source spanning the wavelength range of 400 nm-950 nm. The diffuse reflection spectra for varicose veins exhibited a peak at 530 nm, while leg veins showed a peak at 780 nm. Hyperspectral images obtained at these specific wavelengths were normalized in order to homogenize the spectral signatures of each pixel (converting the hyperspectral image to 8 bit RGB image) and filtered using a moving average filter. Subsequently, the varicose veins and leg veins were delineated and detected using quantitative phase analysis and a k-means clustering algorithm.

CONCLUSION

In conclusion, the custom optical imaging system, utilizing hyperspectral imaging and the associated clustering algorithm, provides detailed information regarding the spatial distribution of varicose veins. This information can assist vascular physicians in facilitating easier diagnosis and treatment planning.

Intra-abdominal laparoscopic assessment of organs perfusion using imaging photoplethysmography

BACKGROUND

An objective evaluation of the functional state and viability of biological tissues during minimally invasive surgery remains unsolved task. Various non-contact methods for evaluating perfusion during laparoscopic surgery are discussed in the literature, but so far there have been no reports of their use in clinical settings.

METHODS AND PATIENTS

Imaging photoplethysmography (iPPG) is a new method for quantitative assessment of perfusion distribution along the tissue. This is the first study in which we demonstrate successful use of iPPG to assess perfusion of organs during laparoscopic surgery in an operation theater. We used a standard rigid laparoscope connected to a standard digital monochrome camera, and abdominal organs were illuminated by green light. A distinctive feature is the synchronous recording of video frames and electrocardiogram with subsequent correlation data processing. During the laparoscopically assisted surgeries in nine cancer patients, the gradient of perfusion of the affected organs was evaluated. In particular, measurements were carried out before preparing a part of the intestine or stomach for resection, after anastomosis, or during physiological tests.

RESULTS

The spatial distribution of perfusion and its changes over time were successfully measured in all surgical cases. In particular, perfusion gradient of an intestine before resection was visualized and quantified by our iPPG laparoscope in all respective cases. It was also demonstrated that systemic administration of norepinephrine leads to a sharper gradient between well and poorly perfused areas of the colon. In four surgical cases, we have shown capability of the laparoscopic iPPG system for intra-abdominal assessment of perfusion in the anastomosed organs. Moreover, good repeatability of continuous long-term measurements of tissue perfusion inside the abdominal cavity was experimentally demonstrated.

CONCLUSION

Our study carried out in real clinical settings has shown that iPPG laparoscope is feasible for intra-abdominal visualization and quantitative assessment of perfusion distribution.

Near-Infrared II Hyperspectral Imaging Improves the Accuracy of Pathological Sampling of Multiple Cancer Types

Pathological histology is the "gold standard" for clinical diagnosis of cancer. Incomplete or excessive sampling of the formalin-fixed excised cancer specimen will result in inaccurate histologic assessment or excessive workload. Conventionally, pathologists perform specimen sampling relying on naked-eye observation, which is subjective and limited by human perception. Precise identification of cancer tissue, size, and margin is challenging, especially for lesions with inconspicuous tumors. To overcome the limits of human eye perception (visible: 400-700 nm) and improve the sampling efficiency, in this study, we propose using a second near-infrared window (NIR-II: 900-1700 nm) hyperspectral imaging (HSI) system to assist specimen sampling on the strength of the verified deep anatomical penetration and low scattering characteristics of the NIR-II optical window. We used selected NIR-II HSI narrow bands to synthesize color images for human eye observation and also applied a machine learning-based algorithm on the complete NIR-II HSI data for automatic tissue classification to assist pathologists in specimen sampling. A total of 92 tumor samples were collected, including 7 types. Sixty-two (62/92) samples were used as the validation set. Five experienced pathologists marked the contour of the cancer tissue on conventional color images by using different methods, and compared it with the "gold standard," showing that NIR-II HSI-assisted methods had significant improvements in determining cancer tissue compared with conventional methods (conventional color image with or without X-ray). The proposed system can be easily integrated into the current workflow, with high imaging efficiency and no ionizing radiation. It may also find applications in intraoperative detection of residual lesions and identification of different tissues.

The Application of Machine Perfusion as an Enhanced ex vivo Model for Optical Imaging

Optical imaging techniques such as spectral imaging show promise for the assessment of tissue health during surgery; however, the validation and translation of such techniques into clinical practise is limited by the lack of representative tissue models. In this paper, we demonstrate the application of an organ perfusion machine as an ex vivo tissue model for optical imaging. Three porcine livers are perfused at stepped blood oxygen saturations. Over the duration of each perfusion, spectral data of the tissue are captured via diffuse optical spectroscopy and multispectral imaging. These data are synchronised with blood oxygen saturation measurements recorded by the perfusion machine. Shifts in the optical properties of the tissue are demonstrated over the duration of the each perfusion, as the tissue becomes reperfused and as the oxygen saturation is varied.

Synthetic hyperspectral array video database with applications to cross-spectral reconstruction and hyperspectral video coding

In this paper, a synthetic hyperspectral video database is introduced. Since it is impossible to record ground-truth hyperspectral videos, this database offers the possibility to leverage the evaluation of algorithms in diverse applications. For all scenes, depth maps are provided as well to yield the position of a pixel in all spatial dimensions as well as the reflectance in spectral dimension. Two novel algorithms for two different applications are proposed to prove the diversity of applications that can be addressed by this novel database. First, a cross-spectral image reconstruction algorithm is extended to exploit the temporal correlation between two consecutive frames. The evaluation using this hyperspectral database shows an increase in peak signal-to-noise ratio (PSNR) of up to 5.6 dB dependent on the scene. Second, a hyperspectral video coder is introduced, which extends an existing hyperspectral image coder by exploiting temporal correlation. The evaluation shows rate savings of up to 10% depending on the scene.

Estimating quantitative physiological and morphological tissue parameters of murine tumor models using hyperspectral imaging and optical profilometry

Understanding tumors and their microenvironment are essential for successful and accurate disease diagnosis. Tissue physiology and morphology are altered in tumors compared to healthy tissues, and there is a need to monitor tumors and their surrounding tissues, including blood vessels, non-invasively. This preliminary study utilizes a multimodal optical imaging system combining hyperspectral imaging (HSI) and three-dimensional (3D) optical profilometry (OP) to capture hyperspectral images and surface shapes of subcutaneously grown murine tumor models. Hyperspectral images are corrected with 3D OP data and analyzed using the inverse-adding doubling (IAD) method to extract tissue properties such as melanin volume fraction and oxygenation. Blood vessels are segmented using the B-COSFIRE algorithm from oxygenation maps. From 3D OP data, tumor volumes are calculated and compared to manual measurements using a vernier caliper. Results show that tumors can be distinguished from healthy tissue based on most extracted tissue parameters ( p < 0.05 ). Furthermore, blood oxygenation is 50% higher within the blood vessels than in the surrounding tissue, and tumor volumes calculated using 3D OP agree within 26% with manual measurements using a vernier caliper. Results suggest that combining HSI and OP could provide relevant quantitative information about tumors and improve the disease diagnosis.

Double spiral chip-embedded micro-trapezoid filters (SMT filters) for the sensitive isolation of CTCs of prostate cancer by spectral detection

Circulating tumor cells (CTCs) are cancer cells that are released from the original tumor and circulate in the blood vessels, carrying greatly similar constituents as the original tumor. Therefore, CTCs have a significant value in cancer prognosis, early diagnosis, and anti-cancer therapy. However, their rarity and heterogeneity make the isolation of CTCs an arduous task. In the present research, we propose a double spiral chip-embedded micro-trapezoid filter (SMT filter) for the sensitive isolation of the CTCs of prostate cancer by spectral detection. SMT filters were elongated to effectively capture CTCs and this distinctive design was conducive to their isolation and enrichment. The SMT filters were verified with tumor cells and artificial patient blood with a capture efficiency as high as 94% at a flow rate of 1.5 mL h^-1. As a further validation, the SMT filters were validated in isolating CTCs from 10 prostate cancers and other cancers in 4 mL blood samples. Also, the CTCs tested positive for each patient blood sample, ranging from 83-114 CTCs. Significantly, we advanced hyperspectral imaging to detect the characteristic spectrum of CTCs both captured in situ on SMT filters and enriched after isolation. The CTCs could be positively identified by hyperspectral imaging with complete integrity of the cell morphology and an improved characteristic spectrum. This represents a breakthrough in the conventional surface-enhanced Raman scattering (SERS) spectroscopy of nanoparticles. Also, the characteristic spectrum of the CTCs would be highly beneficial for distinguishing the cancer type and accurate for enumerating tumor cells with varied intensities. Furthermore, a novel integrated flower-shaped microfilter was presented with all these aforementioned merits. The success of both the SMT filters and characteristic spectral detection indicated their feasibility for further clinical analysis, the evaluation of cancer therapy, and for potential application.

Insights into Biochemical Sources and Diffuse Reflectance Spectral Features for Colorectal Cancer Detection and Localization

Colorectal cancer (CRC) is the third most common and second most deadly type of cancer worldwide. Early detection not only reduces mortality but also improves patient prognosis by allowing the use of minimally invasive techniques to remove cancer while avoiding major surgery. Expanding the use of microsurgical techniques requires accurate diagnosis and delineation of the tumor margins in order to allow complete excision of cancer. We have used diffuse reflectance spectroscopy (DRS) to identify the main optical CRC biomarkers and to optimize parameters for the integration of such technologies into medical devices. A total number of 2889 diffuse reflectance spectra were collected in ex vivo specimens from 47 patients. Short source-detector distance (SDD) and long-SDD fiber-optic probes were employed to measure tissue layers from 0.5 to 1 mm and from 0.5 to 1.9 mm deep, respectively. The most important biomolecules contributing to differentiating DRS between tissue types were oxy- and deoxy-hemoglobin (Hb and HbO₂), followed by water and lipid. Accurate tissue classification and potential DRS device miniaturization using Hb, HbO₂, lipid and water data were achieved particularly well within the wavelength ranges 350-590 nm and 600-1230 nm for the short-SDD probe, and 380-400 nm, 420-610 nm, and 650-950 nm for the long-SDD probe.

Video: Clinical evaluation of a laparoscopic hyperspectral imaging system

Background

Hyperspectral imaging (HSI) during surgical procedures is a new method for perfusion quantification and tissue discrimination. Its use has been limited to open surgery due to large camera sizes, missing color video, or long acquisition times. A hand-held, laparoscopic hyperspectral camera has been developed now to overcome those disadvantages and evaluated clinically for the first time.

Methods

In a clinical evaluation study, gastrointestinal resectates of ten cancer patients were investigated using the laparoscopic hyperspectral camera. Reference data from corresponding anatomical regions were acquired with a clinically approved HSI system. An image registration process was executed that allowed for pixel-wise comparisons of spectral data and parameter images (StO₂: oxygen saturation of tissue, NIR PI: near-infrared perfusion index, OHI: organ hemoglobin index, TWI: tissue water index) provided by both camera systems. The mean absolute error (MAE) and root mean square error (RMSE) served for the quantitative evaluations. Spearman’s rank correlation between factors related to the study design like the time of spectral white balancing and MAE, respectively RMSE, was calculated.

Results

The obtained mean MAEs between the TIVITA® Tissue and the laparoscopic hyperspectral system resulted in StO₂: 11% ± 7%, NIR PI: 14±3, OHI: 14± 5, and TWI: 10 ± 2. The mean RMSE between both systems was 0.1±0.03 from 500 to 750 nm and 0.15 ±0.06 from 750 to 1000 nm. Spearman’s rank correlation coefficients showed no significant correlation between MAE or RMSE and influencing factors related to the study design.

Conclusion

Qualitatively, parameter images of the laparoscopic system corresponded to those of the system for open surgery. Quantitative deviations were attributed to technical differences rather than the study design. Limitations of the presented study are addressed in current large-scale in vivo trials.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00464-022-09282-y.

Optimized spectral filter design enables more accurate estimation of oxygen saturation in spectral imaging

Oxygen saturation (SO₂) in tissue is a crucially important physiological parameter with ubiquitous clinical utility in diagnosis, treatment, and monitoring, as well as widespread use as an invaluable preclinical research tool. Multispectral imaging can be used to visualize SO₂ non-invasively, non-destructively and without contact in real-time using narrow spectral filter sets, but typically, these spectral filter sets are poorly suited to a specific clinical task, application, or tissue type. In this work, we demonstrate the merit of optimizing spectral filter sets for more accurate estimation of SO₂. Using tissue modelling and simulated multispectral imaging, we demonstrate filter optimization reduces the root-mean-square-error (RMSE) in estimating SO₂ by up to 37% compared with evenly spaced filters. Moreover, we demonstrate up to a 79% decrease in RMSE for optimized filter sets compared with filter sets chosen to minimize mutual information. Wider adoption of this approach will result in more effective multispectral imaging systems that can address specific clinical needs and consequently, more widespread adoption of multispectral imaging technologies in disease diagnosis and treatment.

Optical biopsy mapping on endoscopic image mosaics with a marker-free probe

Gastric cancer is the second leading cause of cancer-related deaths worldwide. Early diagnosis significantly increases the chances of survival; therefore, improved assisted exploration and screening techniques are necessary. Previously, we made use of an augmented multi-spectral endoscope by inserting an optical probe into the instrumentation channel. However, the limited field of view and the lack of markings left by optical biopsies on the tissue complicate the navigation and revisit of the suspect areas probed in-vivo. In this contribution two innovative tools are introduced to significantly increase the traceability and monitoring of patients in clinical practice: (i) video mosaicing to build a more comprehensive and panoramic view of large gastric areas; (ii) optical biopsy targeting and registration with the endoscopic images. The proposed optical flow-based mosaicing technique selects images that minimize texture discontinuities and is robust despite the lack of texture and illumination variations. The optical biopsy targeting is based on automatic tracking of a free-marker probe in the endoscopic view using deep learning to dynamically estimate its pose during exploration. The accuracy of pose estimation is sufficient to ensure a precise overlapping of the standard white-light color image and the hyperspectral probe image, assuming that the small target area of the organ is almost flat. This allows the mapping of all spatio-temporally tracked biopsy sites onto the panoramic mosaic. Experimental validations are carried out from videos acquired on patients in hospital. The proposed technique is purely software-based and therefore easily integrable into clinical practice. It is also generic and compatible to any imaging modality that connects to a fiberscope.

Band selection for oxygenation estimation with multispectral/hyperspectral imaging

Multispectral imaging provides valuable information on tissue composition such as hemoglobin oxygen saturation. However, the real-time application of this technique in interventional medicine can be challenging due to the long acquisition times needed for large amounts of hyperspectral data with hundreds of bands. While this challenge can partially be addressed by choosing a discriminative subset of bands, the band selection methods proposed to date are mainly restricted by the availability of often hard to obtain reference measurements. We address this bottleneck with a new approach to band selection that leverages highly accurate Monte Carlo (MC) simulations. We hypothesize that a so chosen small subset of bands can reproduce or even improve upon the results of a quasi continuous spectral measurement. We further investigate whether novel domain adaptation techniques can address the inevitable domain shift stemming from the use of simulations. Initial results based on in silico and in vivo experiments suggest that 10-20 bands are sufficient to closely reproduce results from spectral measurements with 101 bands in the 500-700 nm range. The investigated domain adaptation technique, which only requires unlabeled in vivo measurements, yielded better results than the pure in silico band selection method. Overall, our method could guide development of fast multispectral imaging systems suited for interventional use without relying on complex hardware setups or manually labeled data.

LED-based Hyperspectral Endoscopic Imaging

Hyperspectral endoscopy can offer multiple advantages as compared to conventional endoscopy. Our goal is to design and develop a real-time hyperspectral endoscopic imaging system for the diagnosis of gastrointestinal (GI) tract cancers using a micro-LED array as an in-situ illumination source. The wavelengths of the system range from ultraviolet to visible and near infrared. To evaluate the use of the LED array for hyperspectral imaging, we designed a prototype system and conducted ex vivo experiments using normal and cancerous tissues of mice, chicken, and sheep. We compared the results of our LED-based approach with our reference hyperspectral camera system. The results confirm the similarity between the LED-based hyperspectral imaging system and the reference HSI camera. Our LED-based hyperspectral imaging system can be used not only as an endoscope but also as a laparoscopic or handheld devices for cancer detection and surgery.

Intraoperative colon perfusion assessment using multispectral imaging

In colorectal surgery an anastomosis performed using poorly-perfused, ischaemic bowel segments may result in a leak and consequent morbidity. Traditional measures of perfusion assessment rely on clinical judgement and are mainly subjective, based on tissue appearance, leading to variability between clinicians. This paper describes a multispectral imaging (MSI) laparoscope that can derive quantitative measures of tissue oxygen saturation (SO₂ ). The system uses a xenon surgical light source and fast filter wheel camera to capture eight narrow waveband images across the visible range in approximately 0.3 s. Spectral validation measurements were performed by imaging standardised colour tiles and comparing reflectance with ground truth spectrometer data. Tissue spectra were decomposed into individual contributions from haemoglobin, adipose tissue and scattering, using a previously-developed regression approach. Initial clinical results from seven patients undergoing colorectal surgery are presented and used to characterise measurement stability and reproducibility in vivo. Strategies to improve signal-to-noise ratio and correct for motion are described. Images of healthy bowel tissue (in vivo) indicate that baseline SO₂ is approximately 75 ± 6%. The SO₂ profile along a bowel segment following ligation of the inferior mesenteric artery (IMA) shows a decrease from the proximal to distal end. In the clinical cases shown, imaging results concurred with clinical judgements of the location of well-perfused tissue. Adipose tissue, visibly yellow in the RGB images, is shown to surround the mesentery and cover some of the serosa. SO₂ in this tissue is consistently high, with mean value of 90%. These results show that MSI is a potential intraoperative guidance tool for assessment of perfusion. Mapping of SO₂ in the colon could be used by surgeons to guide choice of transection points and ensure that well-perfused tissue is used to form an anastomosis. The observation of high mesenteric SO₂ agrees with work in the literature and warrants further exploration. Larger studies incorporating with a wider cohort of clinicians will help to provide retrospective evidence of how this imaging technique may be able to reduce inter-operator variability.

Intraoperative Guidance Using Hyperspectral Imaging: A Review for Surgeons

Hyperspectral imaging (HSI) is a novel optical imaging modality, which has recently found diverse applications in the medical field. HSI is a hybrid imaging modality, combining a digital photographic camera with a spectrographic unit, and it allows for a contactless and non-destructive biochemical analysis of living tissue. HSI provides quantitative and qualitative information of the tissue composition at molecular level in a contrast-free manner, hence making it possible to objectively discriminate between different tissue types and between healthy and pathological tissue. Over the last two decades, HSI has been increasingly used in the medical field, and only recently it has found an application in the operating room. In the last few years, several research groups have used this imaging modality as an intraoperative guidance tool within different surgical disciplines. Despite its great potential, HSI still remains far from being routinely used in the daily surgical practice, since it is still largely unknown to most of the surgical community. The aim of this study is to provide clinical surgeons with an overview of the capabilities, current limitations, and future directions of HSI for intraoperative guidance.

Artificial Intelligence in Brain Tumour Surgery-An Emerging Paradigm

Artificial intelligence (AI) platforms have the potential to cause a paradigm shift in brain tumour surgery. Brain tumour surgery augmented with AI can result in safer and more effective treatment. In this review article, we explore the current and future role of AI in patients undergoing brain tumour surgery, including aiding diagnosis, optimising the surgical plan, providing support during the operation, and better predicting the prognosis. Finally, we discuss barriers to the successful clinical implementation, the ethical concerns, and we provide our perspective on how the field could be advanced.

First-in-human pilot study of snapshot multispectral endoscopy for early detection of Barrett's-related neoplasia

SIGNIFICANCE

The early detection of dysplasia in patients with Barrett's esophagus could improve outcomes by enabling curative intervention; however, dysplasia is often inconspicuous using conventional white-light endoscopy.

AIM

We sought to determine whether multispectral imaging (MSI) could be applied in endoscopy to improve detection of dysplasia in the upper gastrointestinal (GI) tract.

APPROACH

We used a commercial fiberscope to relay imaging data from within the upper GI tract to a snapshot MSI camera capable of collecting data from nine spectral bands. The system was deployed in a pilot clinical study of 20 patients (ClinicalTrials.gov NCT03388047) to capture 727 in vivo image cubes matched with gold-standard diagnosis from histopathology. We compared the performance of seven learning-based methods for data classification, including linear discriminant analysis, k-nearest neighbor classification, and a neural network.

RESULTS

Validation of our approach using a Macbeth color chart achieved an image-based classification accuracy of 96.5%. Although our patient cohort showed significant intra- and interpatient variance, we were able to resolve disease-specific contributions to the recorded MSI data. In classification, a combined principal component analysis and k-nearest-neighbor approach performed best, achieving accuracies of 95.8%, 90.7%, and 76.1%, respectively, for squamous, non-dysplastic Barrett's esophagus and neoplasia based on majority decisions per-image.

CONCLUSIONS

MSI shows promise for disease classification in Barrett's esophagus and merits further investigation as a tool in high-definition "chip-on-tip" endoscopes.

Automatic Recognition of Colon and Esophagogastric Cancer with Machine Learning and Hyperspectral Imaging

There are approximately 1.8 million diagnoses of colorectal cancer, 1 million diagnoses of stomach cancer, and 0.6 million diagnoses of esophageal cancer each year globally. An automatic computer-assisted diagnostic (CAD) tool to rapidly detect colorectal and esophagogastric cancer tissue in optical images would be hugely valuable to a surgeon during an intervention. Based on a colon dataset with 12 patients and an esophagogastric dataset of 10 patients, several state-of-the-art machine learning methods have been trained to detect cancer tissue using hyperspectral imaging (HSI), including Support Vector Machines (SVM) with radial basis function kernels, Multi-Layer Perceptrons (MLP) and 3D Convolutional Neural Networks (3DCNN). A leave-one-patient-out cross-validation (LOPOCV) with and without combining these sets was performed. The ROC-AUC score of the 3DCNN was slightly higher than the MLP and SVM with a difference of 0.04 AUC. The best performance was achieved with the 3DCNN for colon cancer and esophagogastric cancer detection with a high ROC-AUC of 0.93. The 3DCNN also achieved the best DICE scores of 0.49 and 0.41 on the colon and esophagogastric datasets, respectively. These scores were significantly improved using a patient-specific decision threshold to 0.58 and 0.51, respectively. This indicates that, in practical use, an HSI-based CAD system using an interactive decision threshold is likely to be valuable. Experiments were also performed to measure the benefits of combining the colorectal and esophagogastric datasets (22 patients), and this yielded significantly better results with the MLP and SVM models.

First experience in clinical application of hyperspectral endoscopy for evaluation of colonic polyps

Early detection and resection of adenomatous polyps prevents their progression to colorectal cancer (CRC), significantly improving patient outcomes. Polyps are typically identified and removed during white-light colonoscopy. Unfortunately, the rate of interval cancers that arise between CRC screening events remains high, linked to poor visualization of polyps during screening and incomplete polyp removal. Here, we sought to evaluate the potential of a hyperspectral endoscope (HySE) to enhance polyp discrimination for detection and resection. We designed, built and tested a new compact HySE in a proof-of-concept clinical study. We successfully collected spectra from three tissue types in seven patients undergoing routine colonoscopy screening. The acquired spectral data from normal tissue and polyps, both pre- and post- resection, were subjected to quantitative analysis using spectral angle mapping and machine learning, which discriminated the data by tissue type, meriting further investigation of HySE as a clinical tool.

Deep Learning Analysis of In Vivo Hyperspectral Images for Automated Intraoperative Nerve Detection

Nerves are critical structures that may be difficult to recognize during surgery. Inadvertent nerve injuries can have catastrophic consequences for the patient and lead to life-long pain and a reduced quality of life. Hyperspectral imaging (HSI) is a non-invasive technique combining photography with spectroscopy, allowing non-invasive intraoperative biological tissue property quantification. We show, for the first time, that HSI combined with deep learning allows nerves and other tissue types to be automatically recognized in in vivo hyperspectral images. An animal model was used, and eight anesthetized pigs underwent neck midline incisions, exposing several structures (nerve, artery, vein, muscle, fat, skin). State-of-the-art machine learning models were trained to recognize these tissue types in HSI data. The best model was a convolutional neural network (CNN), achieving an overall average sensitivity of 0.91 and a specificity of 1.0, validated with leave-one-patient-out cross-validation. For the nerve, the CNN achieved an average sensitivity of 0.76 and a specificity of 0.99. In conclusion, HSI combined with a CNN model is suitable for in vivo nerve recognition.

Hyperspectral autofluorescence characterization of drusen and sub-RPE deposits in age-related macular degeneration

Background:

Soft drusen and basal linear deposit (BLinD) are two forms of the same extracellular lipid rich material that together make up an Oil Spill on Bruch’s membrane (BrM). Drusen are focal and can be recognized clinically. In contrast BLinD is thin and diffusely distributed, and invisible clinically, even on highest resolution OCT, but has been detected on en face hyperspectral autofluorescence (AF) imaging ex vivo. We sought to optimize histologic hyperspectral AF imaging and image analysis for recognition of drusen and sub-RPE deposits (including BLinD and basal laminar deposit), for potential clinical application.

Methods:

Twenty locations specifically with drusen and 12 additional locations specifically from fovea, perifovea and mid-periphery from RPE/BrM flatmounts from 4 AMD donors underwent hyperspectral AF imaging with 4 excitation wavelengths (λ_ex 436, 450, 480 and 505 nm), and the resulting image cubes were simultaneously decomposed with our published non-negative matrix factorization (NMF). Rank 4 recovery of 4 emission spectra was chosen for each excitation wavelength.

Results:

A composite emission spectrum, sensitive and specific for drusen and presumed sub-RPE deposits (the SDr spectrum) was recovered with peak at 510–520 nm in all tissues with drusen, with greatest amplitudes at excitations λ_ex 436, 450 and 480 nm. The RPE spectra of combined sources Lipofuscin (LF)/Melanolipofuscin (MLF) were of comparable amplitude and consistently recapitulated the spectra S1, S2 and S3 previously reported from all tissues: tissues with drusen, foveal and extra-foveal locations.

Conclusions:

A clinical hyperspectral AF camera, with properly chosen excitation wavelengths in the blue range and a hyperspectral AF detector, should be capable of detecting and quantifying drusen and sub-RPE deposits, the earliest known lesions of AMD, before any other currently available imaging modality.

Pose estimation of a markerless fiber bundle for endoscopic optical biopsy

Purpose: We present a markerless vision-based method for on-the-fly three-dimensional (3D) pose estimation of a fiberscope instrument to target pathologic areas in the endoscopic view during exploration. Approach: A 2.5-mm-diameter fiberscope is inserted through the endoscope's operating channel and connected to an additional camera to perform complementary observation of a targeted area such as a multimodal magnifier. The 3D pose of the fiberscope is estimated frame-by-frame by maximizing the similarity between its silhouette (automatically detected in the endoscopic view using a deep learning neural network) and a cylindrical shape bound to a kinematic model reduced to three degrees-of-freedom. An alignment of the cylinder axis, based on Plücker coordinates from the straight edges detected in the image, makes convergence faster and more reliable. Results: The performance on simulations has been validated with a virtual trajectory mimicking endoscopic exploration and on real images of a chessboard pattern acquired with different endoscopic configurations. The experiments demonstrated a good accuracy and robustness of the proposed algorithm with errors of 0.33 ± 0.68    mm in distance position and 0.32 ± 0.11   deg in axis orientation for the 3D pose estimation, which reveals its superiority over previous approaches. This allows multimodal image registration with sufficient accuracy of < 3    pixels . Conclusion: Our pose estimation pipeline was executed on simulations and patterns; the results demonstrate the robustness of our method and the potential of fiber-optical instrument image-based tracking for pose estimation and multimodal registration. It can be fully implemented in software and therefore easily integrated into a routine clinical environment.

Feedforward Artificial Neural Network-Based Colorectal Cancer Detection Using Hyperspectral Imaging: A Step towards Automatic Optical Biopsy

Simple Summary

Detection of colorectal carcinoma is performed visually by investigators and is confirmed pathologically. With hyperspectral imaging, an expanded spectral range of optical information is now available for analysis. The acquired recordings were analyzed with a neural network, and it was possible to differentiate tumor from healthy mucosa in colorectal carcinoma by automatic classification with high reliability. Classification and visualization were performed based on a four-layer perceptron neural network. Based on a neural network, the classification of CA or AD resulted in a sensitivity of 86% and a specificity of 95%, by means of leave-one-patient-out cross-validation. Additionally, significant differences in terms of perfusion parameters (e.g., oxygen saturation) related to tumor staging and neoadjuvant therapy were observed. This is a step towards optical biopsy.

Abstract

Currently, colorectal cancer (CRC) is mainly identified via a visual assessment during colonoscopy, increasingly used artificial intelligence algorithms, or surgery. Subsequently, CRC is confirmed through a histopathological examination by a pathologist. Hyperspectral imaging (HSI), a non-invasive optical imaging technology, has shown promising results in the medical field. In the current study, we combined HSI with several artificial intelligence algorithms to discriminate CRC. Between July 2019 and May 2020, 54 consecutive patients undergoing colorectal resections for CRC were included. The tumor was imaged from the mucosal side with a hyperspectral camera. The image annotations were classified into three groups (cancer, CA; adenomatous margin around the central tumor, AD; and healthy mucosa, HM). Classification and visualization were performed based on a four-layer perceptron neural network. Based on a neural network, the classification of CA or AD resulted in a sensitivity of 86% and a specificity of 95%, by means of leave-one-patient-out cross-validation. Additionally, significant differences in terms of perfusion parameters (e.g., oxygen saturation) related to tumor staging and neoadjuvant therapy were observed. Hyperspectral imaging combined with automatic classification can be used to differentiate between CRC and healthy mucosa. Additionally, the biological changes induced by chemotherapy to the tissue are detectable with HSI.

Forward-Looking Multimodal Endoscopic System Based on Optical Multispectral and High-Frequency Ultrasound Imaging Techniques for Tumor Detection

We developed a forward-looking (FL) multimodal endoscopic system that offers color, spectral classified, high-frequency ultrasound (HFUS) B-mode, and integrated backscattering coefficient (IBC) images for tumor detection in situ. Examination of tumor distributions from the surface of the colon to deeper inside is essential for determining a treatment plan of cancer. For example, the submucosal invasion depth of tumors in addition to the tumor distributions on the colon surface is used as an indicator of whether the endoscopic dissection would be operated. Thus, we devised the FL multimodal endoscopic system to offer information on the tumor distribution from the surface to deep tissue with high accuracy. This system was evaluated with bilayer gelatin phantoms which have different properties at each layer of the phantom in a lateral direction. After evaluating the system with phantoms, it was employed to characterize forty human colon tissues excised from cancer patients. The proposed system could allow us to obtain highly resolved chemical, anatomical, and macro-molecular information on excised colon tissues including tumors, thus enhancing the detection of tumor distributions from the surface to deep tissue. These results suggest that the FL multimodal endoscopic system could be an innovative screening instrument for quantitative tumor characterization.

Evaluation of wavelength ranges and tissue depth probed by diffuse reflectance spectroscopy for colorectal cancer detection

Colorectal cancer (CRC) is the third most common type of cancer worldwide and the second most deadly. Recent research efforts have focused on developing non-invasive techniques for CRC detection. In this study, we evaluated the diagnostic capabilities of diffuse reflectance spectroscopy (DRS) for CRC detection by building 6 classification models based on support vector machines (SVMs). Our dataset consists of 2889 diffuse reflectance spectra collected from freshly excised ex vivo tissues of 47 patients over wavelengths ranging from 350 and 1919 nm with source-detector distances of 630-µm and 2500-µm to probe different depths. Quadratic SVMs were used and performance was evaluated using twofold cross-validation on 10 iterations of randomized training and test sets. We achieved (93.5 ± 2.4)% sensitivity, (94.0 ± 1.7)% specificity AUC by probing the superficial colorectal tissue and (96.1 ± 1.8)% sensitivity, (95.7 ± 0.6)% specificity AUC by sampling deeper tissue layers. To the best of our knowledge, this is the first DRS study to investigate the potential of probing deeper tissue layers using larger SDD probes for CRC detection in the luminal wall. The data analysis showed that using a broader spectrum and longer near-infrared wavelengths can improve the diagnostic accuracy of CRC as well as probing deeper tissue layers.

Novel endoscopic optical diagnostic technologies in medical trial research: recent advancements and future prospects

Novel endoscopic biophotonic diagnostic technologies have the potential to non-invasively detect the interior of a hollow organ or cavity of the human body with subcellular resolution or to obtain biochemical information about tissue in real time. With the capability to visualize or analyze the diagnostic target in vivo, these techniques gradually developed as potential candidates to challenge histopathology which remains the gold standard for diagnosis. Consequently, many innovative endoscopic diagnostic techniques have succeeded in detection, characterization, and confirmation: the three critical steps for routine endoscopic diagnosis. In this review, we mainly summarize researches on emerging endoscopic optical diagnostic techniques, with emphasis on recent advances. We also introduce the fundamental principles and the development of those techniques and compare their characteristics. Especially, we shed light on the merit of novel endoscopic imaging technologies in medical research. For example, hyperspectral imaging and Raman spectroscopy provide direct molecular information, while optical coherence tomography and multi-photo endomicroscopy offer a more extensive detection range and excellent spatial-temporal resolution. Furthermore, we summarize the unexplored application fields of these endoscopic optical techniques in major hospital departments for biomedical researchers. Finally, we provide a brief overview of the future perspectives, as well as bottlenecks of those endoscopic optical diagnostic technologies. We believe all these efforts will enrich the diagnostic toolbox for endoscopists, enhance diagnostic efficiency, and reduce the rate of missed diagnosis and misdiagnosis.

Excitation-scanning hyperspectral video endoscopy: enhancing the light at the end of the tunnel

Colorectal cancer is the 3^rd leading cancer for incidence and mortality rates. Positive treatment outcomes have been associated with early detection; however, early stage lesions have limited contrast to surrounding mucosa. A potential technology to enhance early stagise detection is hyperspectral imaging (HSI). While HSI technologies have been previously utilized to detect colorectal cancer ex vivo or post-operation, they have been difficult to employ in real-time endoscopy scenarios. Here, we describe an LED-based multifurcated light guide and spectral light source that can provide illumination for spectral imaging at frame rates necessary for video-rate endoscopy. We also present an updated light source optical ray-tracing model that resulted in further optimization and provided a ∼10X light transmission increase compared to the initial prototype. Future work will iterate simulation and benchtop testing of the hyperspectral endoscopic system to achieve the goal of video-rate spectral endoscopy.

Validation of two techniques for intraoperative hyperspectral human tissue determination

Purpose: Hyperspectral imaging (HSI) is a non-contact optical imaging technique with the potential to serve as an intraoperative computer-aided diagnostic tool. Our work analyzes the optical properties of visible structures in the surgical field for automatic tissue categorization. Approach: Building an HSI-based computer-aided tissue analysis system requires accurate ground truth and validation of optical soft tissue properties as these show large variability. We introduce and validate two different hyperspectral intraoperative imaging setups and their use for the analysis of optical tissue properties. First, we present an improved multispectral filter-wheel setup integrated into a fully digital microscope. Second, we present a novel setup of two hyperspectral snapshot cameras for intraoperative usage. Both setups are operating in the spectral range of 400 up to 975 nm. They are calibrated and validated using the same database and calibration set. Results: For validation, a color chart with 18 well-defined color spectra in the visual range is analyzed. Thus the results acquired with both settings become transferable and comparable to each other as well as between different interventions. On patient data of two different otorhinolaryngology procedures, we analyze the optical behaviors of different soft tissues and show a visualization of such different spectral information. Conclusion: The introduced calibration pipeline for different HSI setups allows comparison between all acquired spectral information. Clinical in vivo data underline the potential of HSI as an intraoperative diagnostic tool and the clinical usability of both introduced setups. Thereby, we demonstrate their feasibility for the in vivo analysis and categorization of different human soft tissues.

Validation of two techniques for intraoperative hyperspectral human tissue determination

Purpose: Hyperspectral imaging (HSI) is a non-contact optical imaging technique with the potential to serve as an intraoperative computer-aided diagnostic tool. Our work analyzes the optical properties of visible structures in the surgical field for automatic tissue categorization. Approach: Building an HSI-based computer-aided tissue analysis system requires accurate ground truth and validation of optical soft tissue properties as these show large variability. We introduce and validate two different hyperspectral intraoperative imaging setups and their use for the analysis of optical tissue properties. First, we present an improved multispectral filter-wheel setup integrated into a fully digital microscope. Second, we present a novel setup of two hyperspectral snapshot cameras for intraoperative usage. Both setups are operating in the spectral range of 400 up to 975 nm. They are calibrated and validated using the same database and calibration set. Results: For validation, a color chart with 18 well-defined color spectra in the visual range is analyzed. Thus the results acquired with both settings become transferable and comparable to each other as well as between different interventions. On patient data of two different otorhinolaryngology procedures, we analyze the optical behaviors of different soft tissues and show a visualization of such different spectral information. Conclusion: The introduced calibration pipeline for different HSI setups allows comparison between all acquired spectral information. Clinical in vivo data underline the potential of HSI as an intraoperative diagnostic tool and the clinical usability of both introduced setups. Thereby, we demonstrate their feasibility for the in vivo analysis and categorization of different human soft tissues.

Laparoscopic system for simultaneous high-resolution video and rapid hyperspectral imaging in the visible and near-infrared spectral range

SIGNIFICANCE

Hyperspectral imaging (HSI) can support intraoperative perfusion assessment, the identification of tissue structures, and the detection of cancerous lesions. The practical use of HSI for minimal-invasive surgery is currently limited, for example, due to long acquisition times, missing video, or large set-ups.

AIM

An HSI laparoscope is described and evaluated to address the requirements for clinical use and high-resolution spectral imaging.

APPROACH

Reflectance measurements with reference objects and resected human tissue from 500 to 1000 nm are performed to show the consistency with an approved medical HSI device for open surgery. Varying object distances are investigated, and the signal-to-noise ratio (SNR) is determined for different light sources.

RESULTS

The handheld design enables real-time processing and visualization of HSI data during acquisition within 4.6 s. A color video is provided simultaneously and can be augmented with spectral information from push-broom imaging. The reflectance data from the HSI system for open surgery at 50 cm and the HSI laparoscope are consistent for object distances up to 10 cm. A standard rigid laparoscope in combination with a customized LED light source resulted in a mean SNR of 30 to 43 dB (500 to 950 nm).

CONCLUSIONS

Compact and rapid HSI with a high spatial- and spectral-resolution is feasible in clinical practice. Our work may support future studies on minimally invasive HSI to reduce intra- and postoperative complications.

Hyperspectral imaging for diagnosis and detection of ex-vivo breast cancer

BACKGROUND AND PURPOSE

Breast cancer is one of the most widely recognized tumors. .Diagnosis made in the early stage of disease may imporve outcomes. The discovery of malignant growth utilizing noninvasive light intrusive methods in lieu of conventional excisional biopsy may assist in achieving this goal.

MATERIALS AND METHODS

The change of the optical properties of ex-vivo breast tissues provides different responses to light transmission, absorption, and particularly the reflection over the spectrum range. We offer the use of Hyperspectral imaging (HSI) with advanced image processing and pattern recognition in order to analyze HSI data for breast cancer detection. The spectral signatures were mined and evaluated in both malignant and normal tissue. K-mean clustering was designed for classifying hyperspectral data in order to evaluate and detection of cancer tissue. This method was used to detect ex-vivo breast cancer. Spatial spectral images were created to high spot the differences in the reflectance properties of malignant versus normal tissue.

RESULTS

Trials showed that the superficial spectral reflection images within 500 nm wavelength showed high variance (214.65) between cancerous and normal breast tissues. On the other hand, image within 620 nm wavelength showed low variance (0.0020).However, the superimposed of spectral region 420-620 nm was proposed as the optimum bandwidth. Finally, the proposed HS imaging system was capable to discriminate the tumor region from normal tissue of the ex-vivo breast sample with sensitivity and a specificity of 95 % and 96 %.

CONCLUSIONS

High sensitivity and specificity were achieved, which proposes potential for HSI as an edge evaluation method to enhance the surgical outcome compared to the presently available techniques in the clinics.

Surgical spectral imaging

Recent technological developments have resulted in the availability of miniaturised spectral imaging sensors capable of operating in the multi- (MSI) and hyperspectral imaging (HSI) regimes. Simultaneous advances in image-processing techniques and artificial intelligence (AI), especially in machine learning and deep learning, have made these data-rich modalities highly attractive as a means of extracting biological information non-destructively. Surgery in particular is poised to benefit from this, as spectrally-resolved tissue optical properties can offer enhanced contrast as well as diagnostic and guidance information during interventions. This is particularly relevant for procedures where inherent contrast is low under standard white light visualisation. This review summarises recent work in surgical spectral imaging (SSI) techniques, taken from Pubmed, Google Scholar and arXiv searches spanning the period 2013-2019. New hardware, optimised for use in both open and minimally-invasive surgery (MIS), is described, and recent commercial activity is summarised. Computational approaches to extract spectral information from conventional colour images are reviewed, as tip-mounted cameras become more commonplace in MIS. Model-based and machine learning methods of data analysis are discussed in addition to simulation, phantom and clinical validation experiments. A wide variety of surgical pilot studies are reported but it is apparent that further work is needed to quantify the clinical value of MSI/HSI. The current trend toward data-driven analysis emphasises the importance of widely-available, standardised spectral imaging datasets, which will aid understanding of variability across organs and patients, and drive clinical translation.

Gastric cancer diagnosis using hyperspectral imaging with principal component analysis and spectral angle mapper

SIGNIFICANCE

Hyperspectral imaging (HSI) is an emerging optical technique that has a double function of spectroscopy and imaging.

AIM

Near-infrared hyperspectral imaging (NIR-HSI) (900 to 1700 nm) with the help of chemometrics was investigated for gastric cancer diagnosis.

APPROACH

Mean spectra and standard deviation of normal and cancerous pixels were extracted. Principal component analysis (PCA) was used to compress the dimension of hypercube data and select the optimal wavelengths. Moreover, spectral angle mapper (SAM) was utilized as chemometrics to discriminate gastric cancer from normal.

RESULTS

Major spectral difference of cancerous and normal gastric tissue was observed around 975, 1215, and 1450 nm by comparison. A total of six wavelengths (i.e., 975, 1075, 1215, 1275, 1390, and 1450 nm) were then selected as optimal wavelengths by PCA. The accuracy using SAM is up to 90% according to hematoxylin-eosin results.

CONCLUSIONS

These results suggest that NIR-HSI has the potential as a cutting-edge optical diagnostic technique for gastric cancer diagnosis with suitable chemometrics.

Use of Fluorescent Dyes in Endoscopy and Diagnostic Investigation

Background

The advancement of innovative endoscopic technology in terms of improving the visualization of the mucosa has been of significant benefit.

Summary

Advancements in image resolution, software processing, and optical filter technology have resulted in several techniques complemental to traditional white light endoscopy. These new techniques provide a real-time optical diagnosis as well as virtual histology of detected lesions. Optical molecular imaging permits a functional assessment within cells.

Key Message

Optical molecular imaging provides an understanding of cellular processes and permits validation of the specificity of fluorescent tracers and the possibility of quantifying the signal.

Factors influencing the accuracy for tissue classification in multi spectral in-vivo endoscopy for the upper gastro-internal tract

Hyper spectral imaging is a possible way for disease detection. However, for carcinoma detection most of the results are ex-vivo. However, in-vivo results of endoscopic studies still show fairly low accuracies in contrast to the good results of many ex-vivo studies. To overcome this problem and to provide a reasonable explanation, Monte-Carlo simulations of photon trajectories are proposed as a tool to generate multi spectral images including inter patient variations to simulate 40 patients. Furthermore, these simulations have the huge advantage that the position of the carcinoma is known. Due to this, the effect of mislabelled data can be studied. As shown in this study, a percentage of 30–35% of mislabelled data might lead to significant decrease of the accuracy from around 90% to around 70–75%. Therefore, the main focus of hyper spectral imaging has to be the exact characterization of the training data in the future.

O’Shanahan A, Carrera D, Bisshopp S, Sosa C, Marquez M, Camacho R, Plaza MDLL, Morera J, M. Callico G. Most Relevant Spectral Bands Identification for Brain Cancer Detection Using Hyperspectral Imaging

Hyperspectral imaging (HSI) is a non-ionizing and non-contact imaging technique capable of obtaining more information than conventional RGB (red green blue) imaging. In the medical field, HSI has commonly been investigated due to its great potential for diagnostic and surgical guidance purposes. However, the large amount of information provided by HSI normally contains redundant or non-relevant information, and it is extremely important to identify the most relevant wavelengths for a certain application in order to improve the accuracy of the predictions and reduce the execution time of the classification algorithm. Additionally, some wavelengths can contain noise and removing such bands can improve the classification stage. The work presented in this paper aims to identify such relevant spectral ranges in the visual-and-near-infrared (VNIR) region for an accurate detection of brain cancer using in vivo hyperspectral images. A methodology based on optimization algorithms has been proposed for this task, identifying the relevant wavelengths to achieve the best accuracy in the classification results obtained by a supervised classifier (support vector machines), and employing the lowest possible number of spectral bands. The results demonstrate that the proposed methodology based on the genetic algorithm optimization slightly improves the accuracy of the tumor identification in ~5%, using only 48 bands, with respect to the reference results obtained with 128 bands, offering the possibility of developing customized acquisition sensors that could provide real-time HS imaging. The most relevant spectral ranges found comprise between 440.5-465.96 nm, 498.71-509.62 nm, 556.91-575.1 nm, 593.29-615.12 nm, 636.94-666.05 nm, 698.79-731.53 nm and 884.32-902.51 nm.

Optimizing algorithm development for tissue classification in colorectal cancer based on diffuse reflectance spectra

Diffuse reflectance spectroscopy can be used in colorectal cancer surgery for tissue classification. The main challenge in the classification task is to separate healthy colorectal wall from tumor tissue. In this study, four normalization techniques, four feature extraction methods and five classifiers are applied to nine datasets, to obtain the optimal method to separate spectra measured on healthy colorectal wall from spectra measured on tumor tissue. All results are compared to the use of the entire non-normalized spectra. It is found that the most optimal classification approach is to apply a feature extraction method on non-normalized spectra combined with support vector machine or neural network classifier.

Broadband hyperspectral imaging for breast tumor detection using spectral and spatial information

Complete tumor removal during breast-conserving surgery remains challenging due to the lack of optimal intraoperative margin assessment techniques. Here, we use hyperspectral imaging for tumor detection in fresh breast tissue. We evaluated different wavelength ranges and two classification algorithms; a pixel-wise classification algorithm and a convolutional neural network that combines spectral and spatial information. The highest classification performance was obtained using the full wavelength range (450-1650 nm). Adding spatial information mainly improved the differentiation of tissue classes within the malignant and healthy classes. High sensitivity and specificity were accomplished, which offers potential for hyperspectral imaging as a margin assessment technique to improve surgical outcome.

In-Vivo and Ex-Vivo Tissue Analysis through Hyperspectral Imaging Techniques: Revealing the Invisible Features of Cancer

In contrast to conventional optical imaging modalities, hyperspectral imaging (HSI) is able to capture much more information from a certain scene, both within and beyond the visual spectral range (from 400 to 700 nm). This imaging modality is based on the principle that each material provides different responses to light reflection, absorption, and scattering across the electromagnetic spectrum. Due to these properties, it is possible to differentiate and identify the different materials/substances presented in a certain scene by their spectral signature. Over the last two decades, HSI has demonstrated potential to become a powerful tool to study and identify several diseases in the medical field, being a non-contact, non-ionizing, and a label-free imaging modality. In this review, the use of HSI as an imaging tool for the analysis and detection of cancer is presented. The basic concepts related to this technology are detailed. The most relevant, state-of-the-art studies that can be found in the literature using HSI for cancer analysis are presented and summarized, both in-vivo and ex-vivo. Lastly, we discuss the current limitations of this technology in the field of cancer detection, together with some insights into possible future steps in the improvement of this technology.

Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks

PURPOSE

Optical imaging is evolving as a key technique for advanced sensing in the operating room. Recent research has shown that machine learning algorithms can be used to address the inverse problem of converting pixel-wise multispectral reflectance measurements to underlying tissue parameters, such as oxygenation. Assessment of the specific hardware used in conjunction with such algorithms, however, has not properly addressed the possibility that the problem may be ill-posed.

METHODS

We present a novel approach to the assessment of optical imaging modalities, which is sensitive to the different types of uncertainties that may occur when inferring tissue parameters. Based on the concept of invertible neural networks, our framework goes beyond point estimates and maps each multispectral measurement to a full posterior probability distribution which is capable of representing ambiguity in the solution via multiple modes. Performance metrics for a hardware setup can then be computed from the characteristics of the posteriors.

RESULTS

Application of the assessment framework to the specific use case of camera selection for physiological parameter estimation yields the following insights: (1) estimation of tissue oxygenation from multispectral images is a well-posed problem, while (2) blood volume fraction may not be recovered without ambiguity. (3) In general, ambiguity may be reduced by increasing the number of spectral bands in the camera.

CONCLUSION

Our method could help to optimize optical camera design in an application-specific manner.

Use of Hyperspectral/Multispectral Imaging in Gastroenterology. Shedding Some⁻Different⁻Light into the Dark

Hyperspectral/Multispectral imaging (HSI/MSI) technologies are able to sample from tens to hundreds of spectral channels within the electromagnetic spectrum, exceeding the capabilities of human vision. These spectral techniques are based on the principle that every material has a different response (reflection and absorption) to different wavelengths. Thereby, this technology facilitates the discrimination between different materials. HSI has demonstrated good discrimination capabilities for materials in fields, for instance, remote sensing, pollution monitoring, field surveillance, food quality, agriculture, astronomy, geological mapping, and currently, also in medicine. HSI technology allows tissue observation beyond the limitations of the human eye. Moreover, many researchers are using HSI as a new diagnosis tool to analyze optical properties of tissue. Recently, HSI has shown good performance in identifying human diseases in a non-invasive manner. In this paper, we show the potential use of these technologies in the medical domain, with emphasis in the current advances in gastroenterology. The main aim of this review is to provide an overview of contemporary concepts regarding HSI technology together with state-of-art systems and applications in gastroenterology. Finally, we discuss the current limitations and upcoming trends of HSI in gastroenterology.

Hyperspectral imaging for tissue classification, a way toward smart laparoscopic colorectal surgery

In the last decades, laparoscopic surgery has become the gold standard in patients with colorectal cancer. To overcome the drawback of reduced tactile feedback, real-time tissue classification could be of great benefit. In this ex vivo study, hyperspectral imaging (HSI) was used to distinguish tumor tissue from healthy surrounding tissue. A sample of fat, healthy colorectal wall, and tumor tissue was collected per patient and imaged using two hyperspectral cameras, covering the wavelength range from 400 to 1700 nm. The data were randomly divided into a training (75%) and test (25%) set. After feature reduction, a quadratic classifier and support vector machine were used to distinguish the three tissue types. Tissue samples of 32 patients were imaged using both hyperspectral cameras. The accuracy to distinguish the three tissue types using both hyperspectral cameras was 0.88 (STD  =  0.13) on the test dataset. When the accuracy was determined per patient, a mean accuracy of 0.93 (STD  =  0.12) was obtained on the test dataset. This study shows the potential of using HSI in colorectal cancer surgery for fast tissue classification, which could improve clinical outcome. Future research should be focused on imaging entire colon/rectum specimen and the translation of the technique to an intraoperative setting.

Quantitative evaluation of comb-structure correction methods for multispectral fibrescopic imaging

Removing the comb artifact introduced by imaging fibre bundles, or 'fibrescopes', for example in medical endoscopy, is essential to provide high quality images to the observer. Multispectral imaging (MSI) is an emerging method that combines morphological (spatial) and chemical (spectral) information in a single data 'cube'. When a fibrescope is coupled to a spectrally resolved detector array (SRDA) to perform MSI, comb removal is complicated by the demosaicking step required to reconstruct the multispectral data cube. To understand the potential for using SRDAs as multispectral imaging sensors in medical endoscopy, we assessed five comb correction methods with respect to five performance metrics relevant to biomedical imaging applications: processing time, resolution, smoothness, signal and the accuracy of spectral reconstruction. By assigning weights to each metric, which are determined by the particular imaging application, our results can be used to select the correction method to achieve best overall performance. In most cases, interpolation gave the best compromise between the different performance metrics when imaging using an SRDA.

Bimodal reflectance and fluorescence multispectral endoscopy based on spectrally resolving detector arrays

Emerging clinical interest in combining standard white light endoscopy with targeted near-infrared (NIR) fluorescent contrast agents for improved early cancer detection has created demand for multimodal imaging endoscopes. We used two spectrally resolving detector arrays (SRDAs) to realize a bimodal endoscope capable of simultaneous reflectance-based imaging in the visible spectral region and multiplexed fluorescence-based imaging in the NIR. The visible SRDA was composed of 16 spectral bands, with peak wavelengths in the range of 463 to 648 nm and full-width at half-maximum (FWHM) between 9 and 26 nm. The NIR SRDA was composed of 25 spectral bands, with peak wavelengths in the range 659 to 891 nm and FWHM 7 to 15 nm. The spectral endoscope design was based on a "babyscope" model using a commercially available imaging fiber bundle. We developed a spectral transmission model to select optical components and provide reference endmembers for linear spectral unmixing of the recorded image data. The technical characterization of the spectral endoscope is presented, including evaluation of the angular field-of-view, barrel distortion, spatial resolution and spectral fidelity, which showed encouraging performance. An agarose phantom containing oxygenated and deoxygenated blood with three fluorescent dyes was then imaged. After spectral unmixing, the different chemical components of the phantom could be successfully identified via majority decision with high signal-to-background ratio (>3). Imaging performance was further assessed in an ex vivo porcine esophagus model. Our preliminary imaging results demonstrate the capability to simultaneously resolve multiple biological components using a compact spectral endoscopy system.

Spatio-spectral classification of hyperspectral images for brain cancer detection during surgical operations

Surgery for brain cancer is a major problem in neurosurgery. The diffuse infiltration into the surrounding normal brain by these tumors makes their accurate identification by the naked eye difficult. Since surgery is the common treatment for brain cancer, an accurate radical resection of the tumor leads to improved survival rates for patients. However, the identification of the tumor boundaries during surgery is challenging. Hyperspectral imaging is a non-contact, non-ionizing and non-invasive technique suitable for medical diagnosis. This study presents the development of a novel classification method taking into account the spatial and spectral characteristics of the hyperspectral images to help neurosurgeons to accurately determine the tumor boundaries in surgical-time during the resection, avoiding excessive excision of normal tissue or unintentionally leaving residual tumor. The algorithm proposed in this study to approach an efficient solution consists of a hybrid framework that combines both supervised and unsupervised machine learning methods. Firstly, a supervised pixel-wise classification using a Support Vector Machine classifier is performed. The generated classification map is spatially homogenized using a one-band representation of the HS cube, employing the Fixed Reference t-Stochastic Neighbors Embedding dimensional reduction algorithm, and performing a K-Nearest Neighbors filtering. The information generated by the supervised stage is combined with a segmentation map obtained via unsupervised clustering employing a Hierarchical K-Means algorithm. The fusion is performed using a majority voting approach that associates each cluster with a certain class. To evaluate the proposed approach, five hyperspectral images of surface of the brain affected by glioblastoma tumor in vivo from five different patients have been used. The final classification maps obtained have been analyzed and validated by specialists. These preliminary results are promising, obtaining an accurate delineation of the tumor area.

An Intraoperative Visualization System Using Hyperspectral Imaging to Aid in Brain Tumor Delineation

Hyperspectral imaging (HSI) allows for the acquisition of large numbers of spectral bands throughout the electromagnetic spectrum (within and beyond the visual range) with respect to the surface of scenes captured by sensors. Using this information and a set of complex classification algorithms, it is possible to determine which material or substance is located in each pixel. The work presented in this paper aims to exploit the characteristics of HSI to develop a demonstrator capable of delineating tumor tissue from brain tissue during neurosurgical operations. Improved delineation of tumor boundaries is expected to improve the results of surgery. The developed demonstrator is composed of two hyperspectral cameras covering a spectral range of 400–1700 nm. Furthermore, a hardware accelerator connected to a control unit is used to speed up the hyperspectral brain cancer detection algorithm to achieve processing during the time of surgery. A labeled dataset comprised of more than 300,000 spectral signatures is used as the training dataset for the supervised stage of the classification algorithm. In this preliminary study, thematic maps obtained from a validation database of seven hyperspectral images of in vivo brain tissue captured and processed during neurosurgical operations demonstrate that the system is able to discriminate between normal and tumor tissue in the brain. The results can be provided during the surgical procedure (~1 min), making it a practical system for neurosurgeons to use in the near future to improve excision and potentially improve patient outcomes.

Hyperspectral imaging acousto-optic system with spatial filtering for optical phase visualization

We report a method for phase visualization in the images of transparent specimens using analog image processing in incoherent light. The experimental technique is based on adaptive bandpass spatial filtering with an amplitude mask matched with an acousto-optic tunable filter in a telecentric optical system. We demonstrate the processing of microscopic images of unstained and stained histological sections of human thyroid tumor with improved contrast.