Framework for hyperspectral image processing and quantification for cancer detection during animal tumor surgery

Hyperspectral imaging combined with blood oxygen saturation for in vivo analysis of small intestinal necrosis tissue

Acute mesenteric ischemia (AMI) is a clinically significant vascular and gastrointestinal condition, which is closely related to the blood supply of the small intestine. Unfortunately, it is still challenging to properly discriminate small intestinal tissues with different degrees of ischemia. In this study, hyperspectral imaging (HSI) was used to construct pseudo-color images of oxygen saturation about small intestinal tissues and to discriminate different degrees of ischemia. First, several small intestine tissue models of New Zealand white rabbits were prepared and collected their hyperspectral data. Then, a set of isosbestic points were used to linearly transform the measurement data twice to match the reference spectra of oxyhemoglobin and deoxyhemoglobin, respectively. The oxygen saturation was measured at the characteristic peak band of oxyhemoglobin (560 nm). Ultimately, using the oxygenated hemoglobin reflectance spectrum as the benchmark, we obtained the relative amount of median oxygen saturation in normal tissues was 70.0 %, the IQR was 10.1 %, the relative amount of median oxygen saturation in ischemic tissues was 49.6 %, and the IQR was 14.6 %. The results demonstrate that HSI combined with the oxygen saturation computation method can efficiently differentiate between normal and ischemic regions of the small intestinal tissues. This technique provides a powerful support for internist to discriminate small bowel tissues with different degrees of ischemia, and also provides a new way of thinking for the diagnosis of AMI.

Systematic meta-analysis of computer-aided detection to detect early esophageal cancer using hyperspectral imaging

One of the leading causes of cancer deaths is esophageal cancer (EC) because identifying it in early stage is challenging. Computer-aided diagnosis (CAD) could detect the early stages of EC have been developed in recent years. Therefore, in this study, complete meta-analysis of selected studies that only uses hyperspectral imaging to detect EC is evaluated in terms of their diagnostic test accuracy (DTA). Eight studies are chosen based on the Quadas-2 tool results for systematic DTA analysis, and each of the methods developed in these studies is classified based on the nationality of the data, artificial intelligence, the type of image, the type of cancer detected, and the year of publishing. Deeks' funnel plot, forest plot, and accuracy charts were made. The methods studied in these articles show the automatic diagnosis of EC has a high accuracy, but external validation, which is a prerequisite for real-time clinical applications, is lacking.

Deep Learning in Medical Hyperspectral Images: A Review

With the continuous progress of development, deep learning has made good progress in the analysis and recognition of images, which has also triggered some researchers to explore the area of combining deep learning with hyperspectral medical images and achieve some progress. This paper introduces the principles and techniques of hyperspectral imaging systems, summarizes the common medical hyperspectral imaging systems, and summarizes the progress of some emerging spectral imaging systems through analyzing the literature. In particular, this article introduces the more frequently used medical hyperspectral images and the pre-processing techniques of the spectra, and in other sections, it discusses the main developments of medical hyperspectral combined with deep learning for disease diagnosis. On the basis of the previous review, tne limited factors in the study on the application of deep learning to hyperspectral medical images are outlined, promising research directions are summarized, and the future research prospects are provided for subsequent scholars.

Comparison of preprocessing techniques to reduce nontissue-related variations in hyperspectral reflectance imaging

SIGNIFICANCE

Hyperspectral reflectance imaging can be used in medicine to identify tissue types, such as tumor tissue. Tissue classification algorithms are developed based on, e.g., machine learning or principle component analysis. For the development of these algorithms, data are generally preprocessed to remove variability in data not related to the tissue itself since this will improve the performance of the classification algorithm. In hyperspectral imaging, the measured spectra are also influenced by reflections from the surface (glare) and height variations within and between tissue samples.

AIM

To compare the ability of different preprocessing algorithms to decrease variations in spectra induced by glare and height differences while maintaining contrast based on differences in optical properties between tissue types.

APPROACH

We compare eight preprocessing algorithms commonly used in medical hyperspectral imaging: standard normal variate, multiplicative scatter correction, min-max normalization, mean centering, area under the curve normalization, single wavelength normalization, first derivative, and second derivative. We investigate conservation of contrast stemming from differences in: blood volume fraction, presence of different absorbers, scatter amplitude, and scatter slope-while correcting for glare and height variations. We use a similarity metric, the overlap coefficient, to quantify contrast between spectra. We also investigate the algorithms for clinical datasets from the colon and breast.

CONCLUSIONS

Preprocessing reduces the overlap due to glare and distance variations. In general, the algorithms standard normal variate, min-max, area under the curve, and single wavelength normalization are the most suitable to preprocess data used to develop a classification algorithm for tissue classification. The type of contrast between tissue types determines which of these four algorithms is most suitable.

Screening chronic myeloid leukemia neutrophils using a novel 3-Dimensional Spectral Gradient Mapping algorithm on hyperspectral images

Background and objective Early diagnosis of chronic myeloid leukemia (CML) is important for effective treatment. The high spectral and spatial resolution of hyperspectral cellular or tissue images coupled with image analysis algorithms may provide avenues to detect and diagnose diseases early. Many algorithms have been used to analyze medical hyperspectral image data, each having their own strengths and short-comings. We present a novel 3-Dimensional Spectral Gradient Mapping (3-D SGM) method to analyze hyperspectral image cubes of CML versus healthy blood smears. Methods In the present study, we analyzed 13 hyperspectral image cubes of CML and healthy neutrophils. The 3-D SGM algorithm was compared to the conventional Windowed Spectral Angle Mapping (Windowed SAM) method. The 3-D SGM exploited the spectral information of the image cube together with the inter-band and inter-pixel data by extracting the 3-D gradient vector from each pixel. The Windowed SAM determined the similarity between the averaged window of a 2×2 training pixel group and the test pixel, in the multidimensional spectral angle. Results The specificity measure of 3-D SGM (97.7%) was superior to Windowed SAM (72.7%) at ruling out the presence of the disease, making it potentially ideal for screening patients. The positive likelihood ratio value of 3-D SGM (16.70) was superior in diagnosing the presence of the disease (i.e., positive test for CML) versus Windowed SAM (2.26). An accuracy value of 84.2% was achieved with 3-D SGM versus only 70.2% for Windowed SAM. Conclusion The new method is efficient and robust for analyzing hyperspectral images of CML versus healthy neutrophils. It has the potential to be developed into an inexpensive, minimally invasive method for screening CML, and could directly facilitate early diagnosis and treatment of the disease.

Band selection for oxygenation estimation with multispectral/hyperspectral imaging

Multispectral imaging provides valuable information on tissue composition such as hemoglobin oxygen saturation. However, the real-time application of this technique in interventional medicine can be challenging due to the long acquisition times needed for large amounts of hyperspectral data with hundreds of bands. While this challenge can partially be addressed by choosing a discriminative subset of bands, the band selection methods proposed to date are mainly restricted by the availability of often hard to obtain reference measurements. We address this bottleneck with a new approach to band selection that leverages highly accurate Monte Carlo (MC) simulations. We hypothesize that a so chosen small subset of bands can reproduce or even improve upon the results of a quasi continuous spectral measurement. We further investigate whether novel domain adaptation techniques can address the inevitable domain shift stemming from the use of simulations. Initial results based on in silico and in vivo experiments suggest that 10-20 bands are sufficient to closely reproduce results from spectral measurements with 101 bands in the 500-700 nm range. The investigated domain adaptation technique, which only requires unlabeled in vivo measurements, yielded better results than the pure in silico band selection method. Overall, our method could guide development of fast multispectral imaging systems suited for interventional use without relying on complex hardware setups or manually labeled data.

Comprehensive review of surgical microscopes: technology development and medical applications

SIGNIFICANCE

Surgical microscopes provide adjustable magnification, bright illumination, and clear visualization of the surgical field and have been increasingly used in operating rooms. State-of-the-art surgical microscopes are integrated with various imaging modalities, such as optical coherence tomography (OCT), fluorescence imaging, and augmented reality (AR) for image-guided surgery.

AIM

This comprehensive review is based on the literature of over 500 papers that cover the technology development and applications of surgical microscopy over the past century. The aim of this review is threefold: (i) providing a comprehensive technical overview of surgical microscopes, (ii) providing critical references for microscope selection and system development, and (iii) providing an overview of various medical applications.

APPROACH

More than 500 references were collected and reviewed. A timeline of important milestones during the evolution of surgical microscope is provided in this study. An in-depth technical overview of the optical system, mechanical system, illumination, visualization, and integration with advanced imaging modalities is provided. Various medical applications of surgical microscopes in neurosurgery and spine surgery, ophthalmic surgery, ear-nose-throat (ENT) surgery, endodontics, and plastic and reconstructive surgery are described.

RESULTS

Surgical microscopy has been significantly advanced in the technical aspects of high-end optics, bright and shadow-free illumination, stable and flexible mechanical design, and versatile visualization. New imaging modalities, such as hyperspectral imaging, OCT, fluorescence imaging, photoacoustic microscopy, and laser speckle contrast imaging, are being integrated with surgical microscopes. Advanced visualization and AR are being added to surgical microscopes as new features that are changing clinical practices in the operating room.

CONCLUSIONS

The combination of new imaging technologies and surgical microscopy will enable surgeons to perform challenging procedures and improve surgical outcomes. With advanced visualization and improved ergonomics, the surgical microscope has become a powerful tool in neurosurgery, spinal, ENT, ophthalmic, plastic and reconstructive surgeries.

Surgical spectral imaging

Recent technological developments have resulted in the availability of miniaturised spectral imaging sensors capable of operating in the multi- (MSI) and hyperspectral imaging (HSI) regimes. Simultaneous advances in image-processing techniques and artificial intelligence (AI), especially in machine learning and deep learning, have made these data-rich modalities highly attractive as a means of extracting biological information non-destructively. Surgery in particular is poised to benefit from this, as spectrally-resolved tissue optical properties can offer enhanced contrast as well as diagnostic and guidance information during interventions. This is particularly relevant for procedures where inherent contrast is low under standard white light visualisation. This review summarises recent work in surgical spectral imaging (SSI) techniques, taken from Pubmed, Google Scholar and arXiv searches spanning the period 2013-2019. New hardware, optimised for use in both open and minimally-invasive surgery (MIS), is described, and recent commercial activity is summarised. Computational approaches to extract spectral information from conventional colour images are reviewed, as tip-mounted cameras become more commonplace in MIS. Model-based and machine learning methods of data analysis are discussed in addition to simulation, phantom and clinical validation experiments. A wide variety of surgical pilot studies are reported but it is apparent that further work is needed to quantify the clinical value of MSI/HSI. The current trend toward data-driven analysis emphasises the importance of widely-available, standardised spectral imaging datasets, which will aid understanding of variability across organs and patients, and drive clinical translation.

A Review on Advances in Intra-operative Imaging for Surgery and Therapy: Imagining the Operating Room of the Future

With the advent of Minimally Invasive Surgery (MIS), intra-operative imaging has become crucial for surgery and therapy guidance, allowing to partially compensate for the lack of information typical of MIS. This paper reviews the advancements in both classical (i.e. ultrasounds, X-ray, optical coherence tomography and magnetic resonance imaging) and more recent (i.e. multispectral, photoacoustic and Raman imaging) intra-operative imaging modalities. Each imaging modality was analyzed, focusing on benefits and disadvantages in terms of compatibility with the operating room, costs, acquisition time and image characteristics. Tables are included to summarize this information. New generation of hybrid surgical room and algorithms for real time/in room image processing were also investigated. Each imaging modality has its own (site- and procedure-specific) peculiarities in terms of spatial and temporal resolution, field of view and contrasted tissues. Besides the benefits that each technique offers for guidance, considerations about operators and patient risk, costs, and extra time required for surgical procedures have to be considered. The current trend is to equip surgical rooms with multimodal imaging systems, so as to integrate multiple information for real-time data extraction and computer-assisted processing. The future of surgery is to enhance surgeons eye to minimize intra- and after-surgery adverse events and provide surgeons with all possible support to objectify and optimize the care-delivery process.

Hyperspectral Imaging for Skin Feature Detection: Advances in Markerless Tracking for Spine Surgery

In spinal surgery, surgical navigation is an essential tool for safe intervention, including the placement of pedicle screws without injury to nerves and blood vessels. Commercially available systems typically rely on the tracking of a dynamic reference frame attached to the spine of the patient. However, the reference frame can be dislodged or obscured during the surgical procedure, resulting in loss of navigation. Hyperspectral imaging (HSI) captures a large number of spectral information bands across the electromagnetic spectrum, providing image information unseen by the human eye. We aim to exploit HSI to detect skin features in a novel methodology to track patient position in navigated spinal surgery. In our approach, we adopt two local feature detection methods, namely a conventional handcrafted local feature and a deep learning-based feature detection method, which are compared to estimate the feature displacement between different frames due to motion. To demonstrate the ability of the system in tracking skin features, we acquire hyperspectral images of the skin of 17 healthy volunteers. Deep-learned skin features are detected and localized with an average error of only 0.25 mm, outperforming the handcrafted local features with respect to the ground truth based on the use of optical markers.

In Vivo Cancer Detection in Animal Model Using Hyperspectral Image Classification with Wavelet Feature Extraction

Hyperspectral imaging (HSI) is a promising optical imaging technique for cancer detection. However, quantitative methods need to be developed in order to utilize the rich spectral information and subtle spectral variation in such images. In this study, we explore the feasibility of using wavelet-based features from in vivo hyperspectral images for head and neck cancer detection. Hyperspectral reflectance data were collected from 12 mice bearing head and neck cancer. Catenation of 5-level wavelet decomposition outputs of hyperspectral images was used as a feature for tumor discrimination. A support vector machine (SVM) was utilized as the classifier. Seven types of mother wavelets were tested to select the one with the best performance. Classifications with raw reflectance spectra, 1-level wavelet decomposition output, and 2-level wavelet decomposition output, as well as the proposed feature were carried out for comparison. Our results show that the proposed wavelet-based feature yields better classification accuracy, and that using different type and order of mother wavelet achieves different classification results. The wavelet-based classification method provides a new approach for HSI detection of head and neck cancer in the animal model.

Hyperspectral Imaging System with Rotation Platform for Investigation of Jujube Skin Defects

A novel object rotation hyperspectral imaging system with the wavelength range of 468–950 nm for investigating round-shaped fruits was developed. This system was used to obtain the reflection spectra of jujubes for the application of surface defect detection. Compared to the traditional linear scan system, which can scan about 49% of jujube surface in one scan pass, this novel object rotation scan system can scan 95% of jujube surface in one scan pass. Six types of jujube skin condition, including rusty spots, decay, white fungus, black fungus, cracks, and glare, were classified by using hyperspectral data. Support vector machine (SVM) and artificial neural network (ANN) models were used to differentiate the six jujube skin conditions. Classification effectiveness of models was evaluated based on confusion matrices. The percentage of classification accuracy of SVM and ANN models were 97.3% and 97.4%, respectively. The object rotation scan method developed for this study could be used for other round-shaped fruits and integrated into online hyperspectral investigation systems.

Real-time diagnosis and visualization of tumor margins in excised breast specimens using fluorescence lifetime imaging and machine learning

Tumor-free surgical margins are critical in breast-conserving surgery. In up to 38% of the cases, however, patients undergo a second surgery since malignant cells are found at the margins of the excised resection specimen. Thus, advanced imaging tools are needed to ensure clear margins at the time of surgery. The objective of this study was to evaluate a random forest classifier that makes use of parameters derived from point-scanning label-free fluorescence lifetime imaging (FLIm) measurements of breast specimens as a means to diagnose tumor at the resection margins and to enable an intuitive visualization of a probabilistic classifier on tissue specimen. FLIm data from fresh lumpectomy and mastectomy specimens from 18 patients were used in this study. The supervised training was based on a previously developed registration technique between autofluorescence imaging data and cross-sectional histology slides. A pathologist's histology annotations provide the ground truth to distinguish between adipose, fibrous, and tumor tissue. Current results demonstrate the ability of this approach to classify the tumor with 89% sensitivity and 93% specificity and to rapidly (∼ 20 frames per second) overlay the probabilistic classifier overlaid on excised breast specimens using an intuitive color scheme. Furthermore, we show an iterative imaging refinement that allows surgeons to switch between rapid scans with a customized, low spatial resolution to quickly cover the specimen and slower scans with enhanced resolution (400 μm per point measurement) in suspicious regions where more details are required. In summary, this technique provides high diagnostic prediction accuracy, rapid acquisition, adaptive resolution, nondestructive probing, and facile interpretation of images, thus holding potential for clinical breast imaging based on label-free FLIm.

Applications of hyperspectral imaging in the detection and diagnosis of solid tumors

Hyperspectral imaging (HSI) is an emerging new technology in solid tumor diagnosis and detection. It incorporates traditional imaging and spectroscopy together to obtain both spatial and spectral information from tissues simultaneously in a non-invasive manner. This imaging modality is based on the principle that different tissues inherit different spectral reflectance responses that present as unique spectral fingerprints. HSI captures those composition-specific fingerprints to identify cancerous and normal tissues. It becomes a promising tool for performing tumor diagnosis and detection from the label-free histopathological examination to real-time intraoperative assistance. This review introduces the basic principles of HSI and summarizes its methodology and recent advances in solid tumor detection. In particular, the advantages of HSI applied to solid tumors are highlighted to show its potential for clinical use.

Adaptive deep learning for head and neck cancer detection using hyperspectral imaging

It can be challenging to detect tumor margins during surgery for complete resection. The purpose of this work is to develop a novel learning method that learns the difference between the tumor and benign tissue adaptively for cancer detection on hyperspectral images in an animal model. Specifically, an auto-encoder network is trained based on the wavelength bands on hyperspectral images to extract the deep information to create a pixel-wise prediction of cancerous and benign pixel. According to the output hypothesis of each pixel, the misclassified pixels would be reclassified in the right prediction direction based on their adaptive weights. The auto-encoder network is again trained based on these updated pixels. The learner can adaptively improve the ability to identify the cancer and benign tissue by focusing on the misclassified pixels, and thus can improve the detection performance. The adaptive deep learning method highlighting the tumor region proved to be accurate in detecting the tumor boundary on hyperspectral images and achieved a sensitivity of 92.32% and a specificity of 91.31% in our animal experiments. This adaptive learning method on hyperspectral imaging has the potential to provide a noninvasive tool for tumor detection, especially, for the tumor whose margin is indistinct and irregular.

Hyperspectral imaging for head and neck cancer detection: specular glare and variance of the tumor margin in surgical specimens

Head and neck squamous cell carcinoma (SCC) is primarily managed by surgical cancer resection. Recurrence rates after surgery can be as high as 55%, if residual cancer is present. Hyperspectral imaging (HSI) is evaluated for detection of SCC in ex-vivo surgical specimens. Several machine learning methods are investigated, including convolutional neural networks (CNNs) and a spectral-spatial classification framework based on support vector machines. Quantitative results demonstrate that additional data preprocessing and unsupervised segmentation can improve CNN results to achieve optimal performance. The methods are trained in two paradigms, with and without specular glare. Classifying regions that include specular glare degrade the overall results, but the combination of the CNN probability maps and unsupervised segmentation using a majority voting method produces an area under the curve value of 0.81 [0.80, 0.83]. As the wavelengths of light used in HSI can penetrate different depths into biological tissue, cancer margins may change with depth and create uncertainty in the ground truth. Through serial histological sectioning, the variance in the cancer margin with depth is investigated and paired with qualitative classification heat maps using the methods proposed for the testing group of SCC patients. The results determined that the validity of the top section alone as the ground truth may be limited to 1 to 2 mm. The study of specular glare and margin variation provided better understanding of the potential of HSI for the use in the operating room.

Method for coregistration of optical measurements of breast tissue with histopathology: the importance of accounting for tissue deformations

For the validation of optical diagnostic technologies, experimental results need to be benchmarked against the gold standard. Currently, the gold standard for tissue characterization is assessment of hematoxylin and eosin (H&E)-stained sections by a pathologist. When processing tissue into H&E sections, the shape of the tissue deforms with respect to the initial shape when it was optically measured. We demonstrate the importance of accounting for these tissue deformations when correlating optical measurement with routinely acquired histopathology. We propose a method to register the tissue in the H&E sections to the optical measurements, which corrects for these tissue deformations. We compare the registered H&E sections to H&E sections that were registered with an algorithm that does not account for tissue deformations by evaluating both the shape and the composition of the tissue and using microcomputer tomography data as an independent measure. The proposed method, which did account for tissue deformations, was more accurate than the method that did not account for tissue deformations. These results emphasize the need for a registration method that accounts for tissue deformations, such as the method presented in this study, which can aid in validating optical techniques for clinical use.

In-Vivo and Ex-Vivo Tissue Analysis through Hyperspectral Imaging Techniques: Revealing the Invisible Features of Cancer

In contrast to conventional optical imaging modalities, hyperspectral imaging (HSI) is able to capture much more information from a certain scene, both within and beyond the visual spectral range (from 400 to 700 nm). This imaging modality is based on the principle that each material provides different responses to light reflection, absorption, and scattering across the electromagnetic spectrum. Due to these properties, it is possible to differentiate and identify the different materials/substances presented in a certain scene by their spectral signature. Over the last two decades, HSI has demonstrated potential to become a powerful tool to study and identify several diseases in the medical field, being a non-contact, non-ionizing, and a label-free imaging modality. In this review, the use of HSI as an imaging tool for the analysis and detection of cancer is presented. The basic concepts related to this technology are detailed. The most relevant, state-of-the-art studies that can be found in the literature using HSI for cancer analysis are presented and summarized, both in-vivo and ex-vivo. Lastly, we discuss the current limitations of this technology in the field of cancer detection, together with some insights into possible future steps in the improvement of this technology.

Surgical Aid Visualization System for Glioblastoma Tumor Identification based on Deep Learning and In-Vivo Hyperspectral Images of Human Patients

Brain cancer surgery has the goal of performing an accurate resection of the tumor and preserving as much as possible the quality of life of the patient. There is a clinical need to develop non-invasive techniques that can provide reliable assistance for tumor resection in real-time during surgical procedures. Hyperspectral imaging (HSI) arises as a new, noninvasive and non-ionizing technique that can assist neurosurgeons during this difficult task. In this paper, we explore the use of deep learning (DL) techniques for processing hyperspectral (HS) images of in-vivo human brain tissue. We developed a surgical aid visualization system capable of offering guidance to the operating surgeon to achieve a successful and accurate tumor resection. The employed HS database is composed of 26 in-vivo hypercubes from 16 different human patients, among which 258,810 labelled pixels were used for evaluation. The proposed DL methods achieve an overall accuracy of 95% and 85% for binary and multiclass classifications, respectively. The proposed visualization system is able to generate a classification map that is formed by the combination of the DL map and an unsupervised clustering via a majority voting algorithm. This map can be adjusted by the operating surgeon to find the suitable configuration for the current situation during the surgical procedure.

Optical biopsy of head and neck cancer using hyperspectral imaging and convolutional neural networks

For patients undergoing surgical cancer resection of squamous cell carcinoma (SCCa), cancer-free surgical margins are essential for good prognosis. We developed a method to use hyperspectral imaging (HSI), a noncontact optical imaging modality, and convolutional neural networks (CNNs) to perform an optical biopsy of ex-vivo, surgical gross-tissue specimens, collected from 21 patients undergoing surgical cancer resection. Using a cross-validation paradigm with data from different patients, the CNN can distinguish SCCa from normal aerodigestive tract tissues with an area under the receiver operator curve (AUC) of 0.82. Additionally, normal tissue from the upper aerodigestive tract can be subclassified into squamous epithelium, muscle, and gland with an average AUC of 0.94. After separately training on thyroid tissue, the CNN can differentiate between thyroid carcinoma and normal thyroid with an AUC of 0.95, 92% accuracy, 92% sensitivity, and 92% specificity. Moreover, the CNN can discriminate medullary thyroid carcinoma from benign multinodular goiter (MNG) with an AUC of 0.93. Classical-type papillary thyroid carcinoma is differentiated from MNG with an AUC of 0.91. Our preliminary results demonstrate that an HSI-based optical biopsy method using CNNs can provide multicategory diagnostic information for normal and cancerous head-and-neck tissue, and more patient data are needed to fully investigate the potential and reliability of the proposed technique.

Deep Learning-Based Framework for In Vivo Identification of Glioblastoma Tumor using Hyperspectral Images of Human Brain

The main goal of brain cancer surgery is to perform an accurate resection of the tumor, preserving as much normal brain tissue as possible for the patient. The development of a non-contact and label-free method to provide reliable support for tumor resection in real-time during neurosurgical procedures is a current clinical need. Hyperspectral imaging is a non-contact, non-ionizing, and label-free imaging modality that can assist surgeons during this challenging task without using any contrast agent. In this work, we present a deep learning-based framework for processing hyperspectral images of in vivo human brain tissue. The proposed framework was evaluated by our human image database, which includes 26 in vivo hyperspectral cubes from 16 different patients, among which 258,810 pixels were labeled. The proposed framework is able to generate a thematic map where the parenchymal area of the brain is delineated and the location of the tumor is identified, providing guidance to the operating surgeon for a successful and precise tumor resection. The deep learning pipeline achieves an overall accuracy of 80% for multiclass classification, improving the results obtained with traditional support vector machine (SVM)-based approaches. In addition, an aid visualization system is presented, where the final thematic map can be adjusted by the operating surgeon to find the optimal classification threshold for the current situation during the surgical procedure.

Cancer Detection Using Hyperspectral Imaging and Evaluation of the Superficial Tumor Margin Variance with Depth

Head and neck squamous cell carcinoma (SCCa) is primarily managed by surgical resection. Recurrence rates after surgery can be as high as 55% if residual cancer is present. In this study, hyperspectral imaging (HSI) is evaluated for detection of SCCa in ex-vivo surgical specimens. Several methods are investigated, including convolutional neural networks (CNNs) and a spectral-spatial variant of support vector machines. Quantitative results demonstrate that additional processing and unsupervised filtering can improve CNN results to achieve optimal performance. Classifying regions that include specular glare, the average AUC is increased from 0.73 [0.71, 0.75 (95% confidence interval)] to 0.81 [0.80, 0.83] through an unsupervised filtering and majority voting method described. The wavelengths of light used in HSI can penetrate different depths into biological tissue, while the cancer margin may change with depth and create uncertainty in the ground-truth. Through serial histological sectioning, the variance in cancer-margin with depth is also investigated and paired with qualitative classification heat maps using the methods proposed for the testing group SCC patients.

Use of Hyperspectral/Multispectral Imaging in Gastroenterology. Shedding Some⁻Different⁻Light into the Dark

Hyperspectral/Multispectral imaging (HSI/MSI) technologies are able to sample from tens to hundreds of spectral channels within the electromagnetic spectrum, exceeding the capabilities of human vision. These spectral techniques are based on the principle that every material has a different response (reflection and absorption) to different wavelengths. Thereby, this technology facilitates the discrimination between different materials. HSI has demonstrated good discrimination capabilities for materials in fields, for instance, remote sensing, pollution monitoring, field surveillance, food quality, agriculture, astronomy, geological mapping, and currently, also in medicine. HSI technology allows tissue observation beyond the limitations of the human eye. Moreover, many researchers are using HSI as a new diagnosis tool to analyze optical properties of tissue. Recently, HSI has shown good performance in identifying human diseases in a non-invasive manner. In this paper, we show the potential use of these technologies in the medical domain, with emphasis in the current advances in gastroenterology. The main aim of this review is to provide an overview of contemporary concepts regarding HSI technology together with state-of-art systems and applications in gastroenterology. Finally, we discuss the current limitations and upcoming trends of HSI in gastroenterology.

Diagnosis of nasopharyngeal carcinoma from serum samples using hyperspectral imaging combined with a chemometric method

Diagnosing nasopharyngeal carcinoma (NPC) is a significant challenge because of the highly complex process. We proposed an approach to diagnose NPC serum using a combination of hyperspectral imaging and weight-based principal component analysis. Samples were prepared by pressing boric acid into pellets for use as the sera substrate. The sera, collected from 100 healthy volunteers and 60 NPC patients, was dripped onto the surface of the substrate for hyperspectral imaging. The characteristic spectral bands were selected based on the variable weight obtained from a support vector machine (SVM) model, using principal component analysis (PCA) to reduce the dimension in the extracted bands. Obtained results show that the accuracy rate, sensitivity, and specificity between the NPC sera and the sera of the healthy controls reached extremely high levels of 99.15%, 98.79%, and 99.36%, respectively. For the model's consistency evaluation, we found that the Kappa and area under the curve (AUC) of the receiver operating characteristic (ROC) curve were 0.99 and 0.98, respectively. These results suggest that the developed approach could serve as a noninvasive diagnostic and screening tool for highly accurate and consistent detection of NPC. Hence, a combination of hyperspectral imaging (HSI) and a weighted principal component analysis (WPCA)-SVM model represents a powerful and promising tool for NPC diagnosis.

Label-free hyperspectral imaging and quantification methods for surgical margin assessment of tissue specimens of cancer patients

Hyperspectral imaging (HSI) is a relatively new modality in medicine and can have many potential applications. In this study, we developed label-free hyperspectral imaging for tumor margin assessment. HSI data, hypercube (x,y,λ), consists of a series of images of the same field of view that are acquired at different wavelengths. Every pixel in the hypercube has an optical spectrum. We collected surgical tissue specimens from 16 human subjects who underwent head and neck (H&N) cancer surgery. We acquired both HSI, autofluorescence images, and fluorescence images with 2-NBDG and proflavine from the specimens. Digitized histologic slides were examined by an H&N pathologist. We developed image preprocessing and classification methods for HSI data and differentiate cancer from benign tissue. The hyperspectral imaging and classification method was able to distinguish between cancer and normal tissue from oral cavity with an average accuracy of 90±8%, sensitivity of 89±9%, and specificity of 91±6%. This study suggests that label-free hyperspectral imaging has great potential for surgical margin assessment in tissue specimens of H&N cancer patients. Further development of the imaging technology and quantification methods is warranted for its application in image-guided surgery.

Spatio-spectral classification of hyperspectral images for brain cancer detection during surgical operations

Surgery for brain cancer is a major problem in neurosurgery. The diffuse infiltration into the surrounding normal brain by these tumors makes their accurate identification by the naked eye difficult. Since surgery is the common treatment for brain cancer, an accurate radical resection of the tumor leads to improved survival rates for patients. However, the identification of the tumor boundaries during surgery is challenging. Hyperspectral imaging is a non-contact, non-ionizing and non-invasive technique suitable for medical diagnosis. This study presents the development of a novel classification method taking into account the spatial and spectral characteristics of the hyperspectral images to help neurosurgeons to accurately determine the tumor boundaries in surgical-time during the resection, avoiding excessive excision of normal tissue or unintentionally leaving residual tumor. The algorithm proposed in this study to approach an efficient solution consists of a hybrid framework that combines both supervised and unsupervised machine learning methods. Firstly, a supervised pixel-wise classification using a Support Vector Machine classifier is performed. The generated classification map is spatially homogenized using a one-band representation of the HS cube, employing the Fixed Reference t-Stochastic Neighbors Embedding dimensional reduction algorithm, and performing a K-Nearest Neighbors filtering. The information generated by the supervised stage is combined with a segmentation map obtained via unsupervised clustering employing a Hierarchical K-Means algorithm. The fusion is performed using a majority voting approach that associates each cluster with a certain class. To evaluate the proposed approach, five hyperspectral images of surface of the brain affected by glioblastoma tumor in vivo from five different patients have been used. The final classification maps obtained have been analyzed and validated by specialists. These preliminary results are promising, obtaining an accurate delineation of the tumor area.

Detection and delineation of squamous neoplasia with hyperspectral imaging in a mouse model of tongue carcinogenesis

Hyperspectral imaging (HSI) holds the potential for the noninvasive detection of cancers. Oral cancers are often diagnosed at a late stage when treatment is less effective and the mortality and morbidity rates are high. Early detection of oral cancer is, therefore, crucial in order to improve the clinical outcomes. To investigate the potential of HSI as a noninvasive diagnostic tool, an animal study was designed to acquire hyperspectral images of in vivo and ex vivo mouse tongues from a chemically induced tongue carcinogenesis model. A variety of machine-learning algorithms, including discriminant analysis, ensemble learning, and support vector machines, were evaluated for tongue neoplasia detection using HSI and were validated by the reconstructed pathological gold-standard maps. The diagnostic performance of HSI, autofluorescence imaging, and fluorescence imaging were compared in this study. Color-coded prediction maps were generated to display the predicted location and distribution of premalignant and malignant lesions. This study suggests that hyperspectral imaging combined with machine-learning techniques can provide a noninvasive tool for the quantitative detection and delineation of squamous neoplasia.

Tumor Margin Classification of Head and Neck Cancer Using Hyperspectral Imaging and Convolutional Neural Networks

One of the largest factors affecting disease recurrence after surgical cancer resection is negative surgical margins. Hyperspectral imaging (HSI) is an optical imaging technique with potential to serve as a computer aided diagnostic tool for identifying cancer in gross ex-vivo specimens. We developed a tissue classifier using three distinct convolutional neural network (CNN) architectures on HSI data to investigate the ability to classify the cancer margins from ex-vivo human surgical specimens, collected from 20 patients undergoing surgical cancer resection as a preliminary validation group. A new approach for generating the HSI ground truth using a registered histological cancer margin is applied in order to create a validation dataset. The CNN-based method classifies the tumor-normal margin of squamous cell carcinoma (SCCa) versus normal oral tissue with an area under the curve (AUC) of 0.86 for inter-patient validation, performing with 81% accuracy, 84% sensitivity, and 77% specificity. Thyroid carcinoma cancer-normal margins are classified with an AUC of 0.94 for inter-patient validation, performing with 90% accuracy, 91% sensitivity, and 88% specificity. Our preliminary results on a limited patient dataset demonstrate the predictive ability of HSI-based cancer margin detection, which warrants further investigation with more patient data and additional processing techniques to optimize the proposed deep learning method.

Optical Biopsy of Head and Neck Cancer Using Hyperspectral Imaging and Convolutional Neural Networks

Successful outcomes of surgical cancer resection necessitate negative, cancer-free surgical margins. Currently, tissue samples are sent to pathology for diagnostic confirmation. Hyperspectral imaging (HSI) is an emerging, non-contact optical imaging technique. A reliable optical method could serve to diagnose and biopsy specimens in real-time. Using convolutional neural networks (CNNs) as a tissue classifier, we developed a method to use HSI to perform an optical biopsy of ex-vivo surgical specimens, collected from 21 patients undergoing surgical cancer resection. Training and testing on samples from different patients, the CNN can distinguish squamous cell carcinoma (SCCa) from normal aerodigestive tract tissues with an area under the curve (AUC) of 0.82, 81% accuracy, 81% sensitivity, and 80% specificity. Additionally, normal oral tissues can be sub-classified into epithelium, muscle, and glandular mucosa using a decision tree method, with an average AUC of 0.94, 90% accuracy, 93% sensitivity, and 89% specificity. After separately training on thyroid tissue, the CNN differentiates between thyroid carcinoma and normal thyroid with an AUC of 0.95, 92% accuracy, 92% sensitivity, and 92% specificity. Moreover, the CNN can discriminate medullary thyroid carcinoma from benign multi-nodular goiter (MNG) with an AUC of 0.93, 87% accuracy, 88% sensitivity, and 85% specificity. Classical-type papillary thyroid carcinoma is differentiated from benign MNG with an AUC of 0.91, 86% accuracy, 86% sensitivity, and 86% specificity. Our preliminary results demonstrate that an HSI-based optical biopsy method using CNNs can provide multi-category diagnostic information for normal head-and-neck tissue, SCCa, and thyroid carcinomas. More patient data are needed in order to fully investigate the proposed technique to establish reliability and generalizability of the work.

Label-free reflectance hyperspectral imaging for tumor margin assessment: a pilot study on surgical specimens of cancer patients

A label-free, hyperspectral imaging (HSI) approach has been proposed for tumor margin assessment. HSI data, i.e., hypercube (x,y,λ), consist of a series of high-resolution images of the same field of view that are acquired at different wavelengths. Every pixel on an HSI image has an optical spectrum. In this pilot clinical study, a pipeline of a machine-learning-based quantification method for HSI data was implemented and evaluated in patient specimens. Spectral features from HSI data were used for the classification of cancer and normal tissue. Surgical tissue specimens were collected from 16 human patients who underwent head and neck (H&N) cancer surgery. HSI, autofluorescence images, and fluorescence images with 2-deoxy-2-[(7-nitro-2,1,3-benzoxadiazol-4-yl)amino]-D-glucose (2-NBDG) and proflavine were acquired from each specimen. Digitized histologic slides were examined by an H&N pathologist. The HSI and classification method were able to distinguish between cancer and normal tissue from the oral cavity with an average accuracy of 90%±8%, sensitivity of 89%±9%, and specificity of 91%±6%. For tissue specimens from the thyroid, the method achieved an average accuracy of 94%±6%, sensitivity of 94%±6%, and specificity of 95%±6%. HSI outperformed autofluorescence imaging or fluorescence imaging with vital dye (2-NBDG or proflavine). This study demonstrated the feasibility of label-free, HSI for tumor margin assessment in surgical tissue specimens of H&N cancer patients. Further development of the HSI technology is warranted for its application in image-guided surgery.

Detection of Head and Neck Cancer in Surgical Specimens Using Quantitative Hyperspectral Imaging

Purpose: This study intends to investigate the feasibility of using hyperspectral imaging (HSI) to detect and delineate cancers in fresh, surgical specimens of patients with head and neck cancers.Experimental Design: A clinical study was conducted in order to collect and image fresh, surgical specimens from patients (N = 36) with head and neck cancers undergoing surgical resection. A set of machine-learning tools were developed to quantify hyperspectral images of the resected tissue in order to detect and delineate cancerous regions which were validated by histopathologic diagnosis. More than two million reflectance spectral signatures were obtained by HSI and analyzed using machine-learning methods. The detection results of HSI were compared with autofluorescence imaging and fluorescence imaging of two vital-dyes of the same specimens.Results: Quantitative HSI differentiated cancerous tissue from normal tissue in ex vivo surgical specimens with a sensitivity and specificity of 91% and 91%, respectively, and which was more accurate than autofluorescence imaging (P < 0.05) or fluorescence imaging of 2-NBDG (P < 0.05) and proflavine (P < 0.05). The proposed quantification tools also generated cancer probability maps with the tumor border demarcated and which could provide real-time guidance for surgeons regarding optimal tumor resection.Conclusions: This study highlights the feasibility of using quantitative HSI as a diagnostic tool to delineate the cancer boundaries in surgical specimens, and which could be translated into the clinic application with the hope of improving clinical outcomes in the future. Clin Cancer Res; 23(18); 5426-36. ©2017 AACR.

Deep convolutional neural networks for classifying head and neck cancer using hyperspectral imaging

Surgical cancer resection requires an accurate and timely diagnosis of the cancer margins in order to achieve successful patient remission. Hyperspectral imaging (HSI) has emerged as a useful, noncontact technique for acquiring spectral and optical properties of tissue. A convolutional neural network (CNN) classifier is developed to classify excised, squamous-cell carcinoma, thyroid cancer, and normal head and neck tissue samples using HSI. The CNN classification was validated by the manual annotation of a pathologist specialized in head and neck cancer. The preliminary results of 50 patients indicate the potential of HSI and deep learning for automatic tissue-labeling of surgical specimens of head and neck patients.

Tissue classification for laparoscopic image understanding based on multispectral texture analysis

Intraoperative tissue classification is one of the prerequisites for providing context-aware visualization in computer-assisted minimally invasive surgeries. As many anatomical structures are difficult to differentiate in conventional RGB medical images, we propose a classification method based on multispectral image patches. In a comprehensive ex vivo study through statistical analysis, we show that (1) multispectral imaging data are superior to RGB data for organ tissue classification when used in conjunction with widely applied feature descriptors and (2) combining the tissue texture with the reflectance spectrum improves the classification performance. The classifier reaches an accuracy of 98.4% on our dataset. Multispectral tissue analysis could thus evolve as a key enabling technique in computer-assisted laparoscopy.

Tumor margin assessment of surgical tissue specimen of cancer patients using label-free hyperspectral imaging

We are developing label-free hyperspectral imaging (HSI) for tumor margin assessment. HSI data, hypercube (x,y,λ), consists of a series of high-resolution images of the same field of view that are acquired at different wavelengths. Every pixel on the HSI image has an optical spectrum. We developed preprocessing and classification methods for HSI data. We used spectral features from HSI data for the classification of cancer and benign tissue. We collected surgical tissue specimens from 16 human patients who underwent head and neck (H&N) cancer surgery. We acquired both HSI, autofluorescence images, and fluorescence images with 2-NBDG and proflavine from the specimens. Digitized histologic slides were examined by an H&N pathologist. The hyperspectral imaging and classification method was able to distinguish between cancer and normal tissue from oral cavity with an average accuracy of 90±8%, sensitivity of 89±9%, and specificity of 91±6%. For tissue specimens from the thyroid, the method achieved an average accuracy of 94±6%, sensitivity of 94±6%, and specificity of 95±6%. Hyperspectral imaging outperformed autofluorescence imaging or fluorescence imaging with vital dye (2-NBDG or proflavine). This study suggests that label-free hyperspectral imaging has great potential for tumor margin assessment in surgical tissue specimens of H&N cancer patients. Further development of the hyperspectral imaging technology is warranted for its application in image-guided surgery.

Hyperspectral Imaging of Neoplastic Progression in a Mouse Model of Oral Carcinogenesis

Hyperspectral imaging (HSI) is an emerging modality for medical applications and holds great potential for noninvasive early detection of cancer. It has been reported that early cancer detection can improve the survival and quality of life of head and neck cancer patients. In this paper, we explored the possibility of differentiating between premalignant lesions and healthy tongue tissue using hyperspectral imaging in a chemical induced oral cancer animal model. We proposed a novel classification algorithm for cancer detection using hyperspectral images. The method detected the dysplastic tissue with an average area under the curve (AUC) of 0.89. The hyperspectral imaging and classification technique may provide a new tool for oral cancer detection.

Quantitative Diagnosis of Tongue Cancer from Histological Images in an Animal Model

We developed a chemically-induced oral cancer animal model and a computer aided method for tongue cancer diagnosis. The animal model allows us to monitor the progress of the lesions over time. Tongue tissue dissected from mice was sent for histological processing. Representative areas of hematoxylin and eosin (H&E) stained tissue from tongue sections were captured for classifying tumor and non-tumor tissue. The image set used in this paper consisted of 214 color images (114 tumor and 100 normal tissue samples). A total of 738 color, texture, morphometry and topology features were extracted from the histological images. The combination of image features from epithelium tissue and its constituent nuclei and cytoplasm has been demonstrated to improve the classification results. With ten iteration nested cross validation, the method achieved an average sensitivity of 96.5% and a specificity of 99% for tongue cancer detection. The next step of this research is to apply this approach to human tissue for computer aided diagnosis of tongue cancer.

Both idebenone and idebenol are localized near the lipid-water interface of the membrane and increase its fluidity

Idebenone is a synthetic analog of coenzyme Q; both share a quinone moiety but idebenone has a shorter lipophilic tail ending with a hydroxyl group. Differential scanning calorimetry experiments showed that both idebenone and idebenol widened and shifted the phase transition of 1,2-dipalmitoylphosphatidylcholine (DPPC) to a lower temperature and a phase separation with different concentrations of these molecules was observed. Also small angle X-ray diffraction and wide angle X-ray diffraction revealed that both, idebenone and idebenol, induced laterally separated phases in fluid membranes when included in DPPC membranes. Electronic profiles showed that both forms, idebenone and idebenol, reduced the thickness of the fluid membrane. (2)H NMR measurements showed that the order of the membrane decreased at all temperatures in the presence of idebenone or idebenol, the greatest disorder being observed in the segments of the acyl chains close to the lipid-water interface. (1)H NOESY MAS NMR spectra were obtained using 1-palmitoyl-2-oleoyl-phosphatidylcholine membranes and results pointed to a similar location in the membrane for both forms, with the benzoquinone or benzoquinol rings and their terminal hydroxyl group of the hydrophobic chain located near the lipid/water interface of the phospholipid bilayer and the terminal hydroxyl group of the hydrophobic chain of both compounds located at the lipid/water interface. Taken together, all these different locations might explain the different physiological behavior shown by the idebenone/idebenol compared with the ubiquinone-10/ubiquinol-10 pair in which both compounds are differently localized in the membrane.

Superpixel-based spectral classification for the detection of head and neck cancer with hyperspectral imaging

Hyperspectral imaging (HSI) is an emerging imaging modality for medical applications. HSI acquires two dimensional images at various wavelengths. The combination of both spectral and spatial information provides quantitative information for cancer detection and diagnosis. This paper proposes using superpixels, principal component analysis (PCA), and support vector machine (SVM) to distinguish regions of tumor from healthy tissue. The classification method uses 2 principal components decomposed from hyperspectral images and obtains an average sensitivity of 93% and an average specificity of 85% for 11 mice. The hyperspectral imaging technology and classification method can have various applications in cancer research and management.