Image enhancement based on in vivo hyperspectral gastroscopic images: a case study

Unsupervised band selection of medical hyperspectral images guided by data gravitation and weak correlation

BACKGROUND AND OBJECTIVE

Medical hyperspectral images (MHSIs) are used for a contact-free examination of patients without harmful radiation. However, high-dimensionality images contain large amounts of data that are sparsely distributed in a high-dimensional space, which leads to the "curse of dimensionality" (called Hughes' phenomenon) and increases the complexity and cost of data processing and storage. Hence, there is a need for spectral dimensionality reduction before the clinical application of MHSIs. Some dimensionality-reducing strategies have been proposed; however, they distort the data within MHSIs.

METHODS

To compress dimensionality without destroying the original data structure, we propose a method that involves data gravitation and weak correlation-based ranking (DGWCR) for removing bands of noise from MHSIs while clustering signal-containing bands. Band clustering is done by using the connection centre evolution (CCE) algorithm and selecting the most representative bands in each cluster based on the composite force. The bands within the clusters are ranked using the new entropy-containing matrix, and a global ranking of bands is obtained by applying an S-shaped strategy. The source code is available at https://www.github.com/zhangchenglong1116/DGWCR.

RESULTS

Upon feeding the reduced-dimensional images into various classifiers, the experimental results demonstrated that the small number of bands selected by the proposed DGWCR consistently achieved higher classification accuracy than the original data. Unlike other reference methods (e.g. the latest deep-learning-based strategies), DGWCR chooses the spectral bands with the least redundancy and greatest discrimination.

CONCLUSION

In this study, we present a method for efficient band selection for MHSIs that alleviates the "curse of dimensionality". Experiments were validated with three MHSIs in the human brain, and they outperformed several other band selection methods, demonstrating the clinical potential of DGWCR.

Band selection for oxygenation estimation with multispectral/hyperspectral imaging

Multispectral imaging provides valuable information on tissue composition such as hemoglobin oxygen saturation. However, the real-time application of this technique in interventional medicine can be challenging due to the long acquisition times needed for large amounts of hyperspectral data with hundreds of bands. While this challenge can partially be addressed by choosing a discriminative subset of bands, the band selection methods proposed to date are mainly restricted by the availability of often hard to obtain reference measurements. We address this bottleneck with a new approach to band selection that leverages highly accurate Monte Carlo (MC) simulations. We hypothesize that a so chosen small subset of bands can reproduce or even improve upon the results of a quasi continuous spectral measurement. We further investigate whether novel domain adaptation techniques can address the inevitable domain shift stemming from the use of simulations. Initial results based on in silico and in vivo experiments suggest that 10-20 bands are sufficient to closely reproduce results from spectral measurements with 101 bands in the 500-700 nm range. The investigated domain adaptation technique, which only requires unlabeled in vivo measurements, yielded better results than the pure in silico band selection method. Overall, our method could guide development of fast multispectral imaging systems suited for interventional use without relying on complex hardware setups or manually labeled data.

First-in-human pilot study of snapshot multispectral endoscopy for early detection of Barrett's-related neoplasia

SIGNIFICANCE

The early detection of dysplasia in patients with Barrett's esophagus could improve outcomes by enabling curative intervention; however, dysplasia is often inconspicuous using conventional white-light endoscopy.

AIM

We sought to determine whether multispectral imaging (MSI) could be applied in endoscopy to improve detection of dysplasia in the upper gastrointestinal (GI) tract.

APPROACH

We used a commercial fiberscope to relay imaging data from within the upper GI tract to a snapshot MSI camera capable of collecting data from nine spectral bands. The system was deployed in a pilot clinical study of 20 patients (ClinicalTrials.gov NCT03388047) to capture 727 in vivo image cubes matched with gold-standard diagnosis from histopathology. We compared the performance of seven learning-based methods for data classification, including linear discriminant analysis, k-nearest neighbor classification, and a neural network.

RESULTS

Validation of our approach using a Macbeth color chart achieved an image-based classification accuracy of 96.5%. Although our patient cohort showed significant intra- and interpatient variance, we were able to resolve disease-specific contributions to the recorded MSI data. In classification, a combined principal component analysis and k-nearest-neighbor approach performed best, achieving accuracies of 95.8%, 90.7%, and 76.1%, respectively, for squamous, non-dysplastic Barrett's esophagus and neoplasia based on majority decisions per-image.

CONCLUSIONS

MSI shows promise for disease classification in Barrett's esophagus and merits further investigation as a tool in high-definition "chip-on-tip" endoscopes.

Reflectance spectra analysis for mucous assessment

This review report represents an overview of research and development on medical hyperspectral imaging technology and its applications. Spectral imaging technology is attracting attention as a new imaging modality for medical applications, especially in disease diagnosis and image-guided surgery. Considering the recent advances in imaging, this technology provides an opportunity for two-dimensional mapping of oxygen saturation (SatO₂) of blood with high accuracy, spatial spectral imaging, and its analysis and provides detection and diagnostic information about the tissue physiology and morphology. Multispectral imaging also provides information about tissue oxygenation, perfusion, and potential function during surgery. Analytical algorithm has been examined, and indication of accurate map of relative hemoglobin concentration and SatO₂ can be indicated with preferable resolution and frame rate. This technology is expected to provide promising biomedical information in practical use. Several studies suggested that blood flow and SatO₂ are associated with gastrointestinal disorders, particularly malignant tumor conditions. The use and analysis of spectroscopic images are expected to potentially play a role in the detection and diagnosis of these diseases.

Novel endoscopic optical diagnostic technologies in medical trial research: recent advancements and future prospects

Novel endoscopic biophotonic diagnostic technologies have the potential to non-invasively detect the interior of a hollow organ or cavity of the human body with subcellular resolution or to obtain biochemical information about tissue in real time. With the capability to visualize or analyze the diagnostic target in vivo, these techniques gradually developed as potential candidates to challenge histopathology which remains the gold standard for diagnosis. Consequently, many innovative endoscopic diagnostic techniques have succeeded in detection, characterization, and confirmation: the three critical steps for routine endoscopic diagnosis. In this review, we mainly summarize researches on emerging endoscopic optical diagnostic techniques, with emphasis on recent advances. We also introduce the fundamental principles and the development of those techniques and compare their characteristics. Especially, we shed light on the merit of novel endoscopic imaging technologies in medical research. For example, hyperspectral imaging and Raman spectroscopy provide direct molecular information, while optical coherence tomography and multi-photo endomicroscopy offer a more extensive detection range and excellent spatial-temporal resolution. Furthermore, we summarize the unexplored application fields of these endoscopic optical techniques in major hospital departments for biomedical researchers. Finally, we provide a brief overview of the future perspectives, as well as bottlenecks of those endoscopic optical diagnostic technologies. We believe all these efforts will enrich the diagnostic toolbox for endoscopists, enhance diagnostic efficiency, and reduce the rate of missed diagnosis and misdiagnosis.

Surgical spectral imaging

Recent technological developments have resulted in the availability of miniaturised spectral imaging sensors capable of operating in the multi- (MSI) and hyperspectral imaging (HSI) regimes. Simultaneous advances in image-processing techniques and artificial intelligence (AI), especially in machine learning and deep learning, have made these data-rich modalities highly attractive as a means of extracting biological information non-destructively. Surgery in particular is poised to benefit from this, as spectrally-resolved tissue optical properties can offer enhanced contrast as well as diagnostic and guidance information during interventions. This is particularly relevant for procedures where inherent contrast is low under standard white light visualisation. This review summarises recent work in surgical spectral imaging (SSI) techniques, taken from Pubmed, Google Scholar and arXiv searches spanning the period 2013-2019. New hardware, optimised for use in both open and minimally-invasive surgery (MIS), is described, and recent commercial activity is summarised. Computational approaches to extract spectral information from conventional colour images are reviewed, as tip-mounted cameras become more commonplace in MIS. Model-based and machine learning methods of data analysis are discussed in addition to simulation, phantom and clinical validation experiments. A wide variety of surgical pilot studies are reported but it is apparent that further work is needed to quantify the clinical value of MSI/HSI. The current trend toward data-driven analysis emphasises the importance of widely-available, standardised spectral imaging datasets, which will aid understanding of variability across organs and patients, and drive clinical translation.

Pre-Cancerous Stomach Lesion Detections with Multispectral-Augmented Endoscopic Prototype

In this paper, we are interested in the in vivo detection of pre-cancerous stomach lesions. Pre-cancerous lesions are unfortunately rarely explored in research papers as most of them are focused on cancer detection or conducted ex-vivo. For this purpose, a novel prototype is introduced. It consists of a standard endoscope with multispectral cameras, an optical setup, a fiberscope, and an external light source. Reflectance spectra are acquired in vivo on 16 patients with a healthy stomach, chronic gastritis, or intestinal metaplasia. A specific pipeline has been designed for the classification of spectra between healthy mucosa and different pathologies. The pipeline includes a wavelength clustering algorithm, spectral features computation, and the training of a classifier in a “leave one patient out” manner. Good classification results, around 80%, have been obtained, and two attractive wavelength ranges were found in the red and near-infrared ranges: [745, 755 nm] and [780, 840 nm]. The new prototype and the associated results give good arguments in favor of future common use in operating rooms, during upper gastrointestinal exploration of the stomach for the detection of stomach diseases.

Context-free hyperspectral image enhancement for wide-field optical biomarker visualization

Many well-known algorithms for the color enhancement of hyperspectral measurements in biomedical imaging are based on statistical assumptions that vary greatly with respect to the proportions of different pixels that appear in a given image, and thus may thwart their application in a surgical environment. This article attempts to explain why this occurs with SVD-based enhancement methods, and proposes the separation of spectral enhancement from analysis. The resulting method, termed affinity-based color enhancement, or ACE for short, achieves multi- and hyperspectral image coloring and contrast based on current spectral affinity metrics that can physically relate spectral data to a particular biomarker. This produces tunable, real-time results which are analogous to the current state-of-the-art algorithms, without suffering any of their inherent context-dependent limitations. Two applications of this method are shown as application examples: vein contrast enhancement and high-precision chromophore concentration estimation.

Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks

PURPOSE

Optical imaging is evolving as a key technique for advanced sensing in the operating room. Recent research has shown that machine learning algorithms can be used to address the inverse problem of converting pixel-wise multispectral reflectance measurements to underlying tissue parameters, such as oxygenation. Assessment of the specific hardware used in conjunction with such algorithms, however, has not properly addressed the possibility that the problem may be ill-posed.

METHODS

We present a novel approach to the assessment of optical imaging modalities, which is sensitive to the different types of uncertainties that may occur when inferring tissue parameters. Based on the concept of invertible neural networks, our framework goes beyond point estimates and maps each multispectral measurement to a full posterior probability distribution which is capable of representing ambiguity in the solution via multiple modes. Performance metrics for a hardware setup can then be computed from the characteristics of the posteriors.

RESULTS

Application of the assessment framework to the specific use case of camera selection for physiological parameter estimation yields the following insights: (1) estimation of tissue oxygenation from multispectral images is a well-posed problem, while (2) blood volume fraction may not be recovered without ambiguity. (3) In general, ambiguity may be reduced by increasing the number of spectral bands in the camera.

CONCLUSION

Our method could help to optimize optical camera design in an application-specific manner.

Use of Hyperspectral/Multispectral Imaging in Gastroenterology. Shedding Some⁻Different⁻Light into the Dark

Hyperspectral/Multispectral imaging (HSI/MSI) technologies are able to sample from tens to hundreds of spectral channels within the electromagnetic spectrum, exceeding the capabilities of human vision. These spectral techniques are based on the principle that every material has a different response (reflection and absorption) to different wavelengths. Thereby, this technology facilitates the discrimination between different materials. HSI has demonstrated good discrimination capabilities for materials in fields, for instance, remote sensing, pollution monitoring, field surveillance, food quality, agriculture, astronomy, geological mapping, and currently, also in medicine. HSI technology allows tissue observation beyond the limitations of the human eye. Moreover, many researchers are using HSI as a new diagnosis tool to analyze optical properties of tissue. Recently, HSI has shown good performance in identifying human diseases in a non-invasive manner. In this paper, we show the potential use of these technologies in the medical domain, with emphasis in the current advances in gastroenterology. The main aim of this review is to provide an overview of contemporary concepts regarding HSI technology together with state-of-art systems and applications in gastroenterology. Finally, we discuss the current limitations and upcoming trends of HSI in gastroenterology.

Quantitative evaluation of comb-structure correction methods for multispectral fibrescopic imaging

Removing the comb artifact introduced by imaging fibre bundles, or 'fibrescopes', for example in medical endoscopy, is essential to provide high quality images to the observer. Multispectral imaging (MSI) is an emerging method that combines morphological (spatial) and chemical (spectral) information in a single data 'cube'. When a fibrescope is coupled to a spectrally resolved detector array (SRDA) to perform MSI, comb removal is complicated by the demosaicking step required to reconstruct the multispectral data cube. To understand the potential for using SRDAs as multispectral imaging sensors in medical endoscopy, we assessed five comb correction methods with respect to five performance metrics relevant to biomedical imaging applications: processing time, resolution, smoothness, signal and the accuracy of spectral reconstruction. By assigning weights to each metric, which are determined by the particular imaging application, our results can be used to select the correction method to achieve best overall performance. In most cases, interpolation gave the best compromise between the different performance metrics when imaging using an SRDA.

Bimodal reflectance and fluorescence multispectral endoscopy based on spectrally resolving detector arrays

Emerging clinical interest in combining standard white light endoscopy with targeted near-infrared (NIR) fluorescent contrast agents for improved early cancer detection has created demand for multimodal imaging endoscopes. We used two spectrally resolving detector arrays (SRDAs) to realize a bimodal endoscope capable of simultaneous reflectance-based imaging in the visible spectral region and multiplexed fluorescence-based imaging in the NIR. The visible SRDA was composed of 16 spectral bands, with peak wavelengths in the range of 463 to 648 nm and full-width at half-maximum (FWHM) between 9 and 26 nm. The NIR SRDA was composed of 25 spectral bands, with peak wavelengths in the range 659 to 891 nm and FWHM 7 to 15 nm. The spectral endoscope design was based on a "babyscope" model using a commercially available imaging fiber bundle. We developed a spectral transmission model to select optical components and provide reference endmembers for linear spectral unmixing of the recorded image data. The technical characterization of the spectral endoscope is presented, including evaluation of the angular field-of-view, barrel distortion, spatial resolution and spectral fidelity, which showed encouraging performance. An agarose phantom containing oxygenated and deoxygenated blood with three fluorescent dyes was then imaged. After spectral unmixing, the different chemical components of the phantom could be successfully identified via majority decision with high signal-to-background ratio (>3). Imaging performance was further assessed in an ex vivo porcine esophagus model. Our preliminary imaging results demonstrate the capability to simultaneously resolve multiple biological components using a compact spectral endoscopy system.