Tissue classification for laparoscopic image understanding based on multispectral texture analysis

Compact and ultracompact spectral imagers: technology and applications in biomedical imaging

Significance

Spectral imaging, which includes hyperspectral and multispectral imaging, can provide images in numerous wavelength bands within and beyond the visible light spectrum. Emerging technologies that enable compact, portable spectral imaging cameras can facilitate new applications in biomedical imaging.

Aim

With this review paper, researchers will (1) understand the technological trends of upcoming spectral cameras, (2) understand new specific applications that portable spectral imaging unlocked, and (3) evaluate proper spectral imaging systems for their specific applications.

Approach

We performed a comprehensive literature review in three databases (Scopus, PubMed, and Web of Science). We included only fully realized systems with definable dimensions. To best accommodate many different definitions of "compact," we included a table of dimensions and weights for systems that met our definition.

Results

There is a wide variety of contributions from industry, academic, and hobbyist spaces. A variety of new engineering approaches, such as Fabry-Perot interferometers, spectrally resolved detector array (mosaic array), microelectro-mechanical systems, 3D printing, light-emitting diodes, and smartphones, were used in the construction of compact spectral imaging cameras. In bioimaging applications, these compact devices were used for in vivo and ex vivo diagnosis and surgical settings.

Conclusions

Compact and ultracompact spectral imagers are the future of spectral imaging systems. Researchers in the bioimaging fields are building systems that are low-cost, fast in acquisition time, and mobile enough to be handheld.

Computer-Aided Diagnosis by Tissue Image Analysis as an Optical Biopsy in Hysteroscopy

This review of our experience in computer-assisted tissue image analysis (CATIA) research shows that significant information can be extracted and used to diagnose and distinguish normal from abnormal endometrium. CATIA enabled the evaluation and differentiation between the benign and malignant endometrium during diagnostic hysteroscopy. The efficacy of texture analysis in the endometrium image during hysteroscopy was examined in 40 women, where 209 normal and 209 abnormal regions of interest (ROIs) were extracted. There was a significant difference between normal and abnormal endometrium for the statistical features (SF) features mean, variance, median, energy and entropy; for the spatial grey-level difference matrix (SGLDM) features contrast, correlation, variance, homogeneity and entropy; and for the gray-level difference statistics (GLDS) features homogeneity, contrast, energy, entropy and mean. We further evaluated 52 hysteroscopic images of 258 normal and 258 abnormal endometrium ROIs, and tissue diagnosis was verified by histopathology after biopsy. The YCrCb color system with SF, SGLDM and GLDS color texture features based on support vector machine (SVM) modeling correctly classified 81% of the cases with a sensitivity and a specificity of 78% and 81%, respectively, for normal and hyperplastic endometrium. New technical and computational advances may improve optical biopsy accuracy and assist in the precision of lesion excision during hysteroscopy. The exchange of knowledge, collaboration, identification of tasks and CATIA method selection strategy will further improve computer-aided diagnosis implementation in the daily practice of hysteroscopy.

Spectral organ fingerprints for machine learning-based intraoperative tissue classification with hyperspectral imaging in a porcine model

Visual discrimination of tissue during surgery can be challenging since different tissues appear similar to the human eye. Hyperspectral imaging (HSI) removes this limitation by associating each pixel with high-dimensional spectral information. While previous work has shown its general potential to discriminate tissue, clinical translation has been limited due to the method's current lack of robustness and generalizability. Specifically, the scientific community is lacking a comprehensive spectral tissue atlas, and it is unknown whether variability in spectral reflectance is primarily explained by tissue type rather than the recorded individual or specific acquisition conditions. The contribution of this work is threefold: (1) Based on an annotated medical HSI data set (9059 images from 46 pigs), we present a tissue atlas featuring spectral fingerprints of 20 different porcine organs and tissue types. (2) Using the principle of mixed model analysis, we show that the greatest source of variability related to HSI images is the organ under observation. (3) We show that HSI-based fully-automatic tissue differentiation of 20 organ classes with deep neural networks is possible with high accuracy (> 95%). We conclude from our study that automatic tissue discrimination based on HSI data is feasible and could thus aid in intraoperative decisionmaking and pave the way for context-aware computer-assisted surgery systems and autonomous robotics.

Is Computer-Assisted Tissue Image Analysis the Future in Minimally Invasive Surgery? A Review on the Current Status of Its Applications

Purpose: Computer-assisted tissue image analysis (CATIA) enables an optical biopsy of human tissue during minimally invasive surgery and endoscopy. Thus far, it has been implemented in gastrointestinal, endometrial, and dermatologic examinations that use computational analysis and image texture feature systems. We review and evaluate the impact of in vivo optical biopsies performed by tissue image analysis on the surgeon’s diagnostic ability and sampling precision and investigate how operation complications could be minimized. Methods: We performed a literature search in PubMed, IEEE, Xplore, Elsevier, and Google Scholar, which yielded 28 relevant articles. Our literature review summarizes the available data on CATIA of human tissues and explores the possibilities of computer-assisted early disease diagnoses, including cancer. Results: Hysteroscopic image texture analysis of the endometrium successfully distinguished benign from malignant conditions up to 91% of the time. In dermatologic studies, the accuracy of distinguishing nevi melanoma from benign disease fluctuated from 73% to 81%. Skin biopsies of basal cell carcinoma and melanoma exhibited an accuracy of 92.4%, sensitivity of 99.1%, and specificity of 93.3% and distinguished nonmelanoma and normal lesions from benign precancerous lesions with 91.9% and 82.8% accuracy, respectively. Gastrointestinal and endometrial examinations are still at the experimental phase. Conclusions: CATIA is a promising application for distinguishing normal from abnormal tissues during endoscopic procedures and minimally invasive surgeries. However, the efficacy of computer-assisted diagnostics in distinguishing benign from malignant states is still not well documented. Prospective and randomized studies are needed before CATIA is implemented in clinical practice.

A Novel Hyperspectral Imaging System for Intraoperative Prediction of Cerebral Hyperperfusion Syndrome after Superficial Temporal Artery-Middle Cerebral Artery Anastomosis in Patients with Moyamoya Disease

BACKGROUND

Postoperative cerebral hyperperfusion syndrome (CHS) may occur after superficial temporal artery (STA)-middle cerebral artery (MCA) bypass for moyamoya disease (MMD). Predicting postoperative CHS is challenging; however, we previously reported the feasibility of using a hyperspectral camera (HSC) for monitoring intraoperative changes in brain surface hemodynamics during STA-MCA bypass.

OBJECTIVE

To investigate the utility of HSC to predict postoperative CHS during STA-MCA bypass for patients with MMD.

METHODS

Hyperspectral images of the cerebral cortex of 29 patients with MMD who underwent STA-MCA bypass were acquired by using an HSC before and after anastomosis. We then analyzed the changes in oxygen saturation after anastomosis and assessed its correlation with CHS.

RESULTS

Five patients experienced transient neurological deterioration several days after surgery. 123I-N-Isopropyl-iodoamphetamine single-photon emission computed tomography scan results revealed an intense, focal increase in cerebral blood flow at the site of anastomosis without any cerebral infarction. Patients with CHS showed significantly increased oxygen saturation (SO2) in the cerebral cortex after anastomosis relative to those without CHS (33 ± 28 vs. 8 ± 14%, p < 0.0001). Receiver operating characteristic analysis results show that postoperative CHS likely occurs when the increase rate of cortical SO2 value is >15% (sensitivity, 85.0%; specificity, 81.3%; area under curve, 0.871).

CONCLUSIONS

This study indicates that hyperspectral imaging of the cerebral cortex may be used to predict postoperative CHS in patients with MMD undergoing STA-MCA bypass.

Laparoscopic system for simultaneous high-resolution video and rapid hyperspectral imaging in the visible and near-infrared spectral range

SIGNIFICANCE

Hyperspectral imaging (HSI) can support intraoperative perfusion assessment, the identification of tissue structures, and the detection of cancerous lesions. The practical use of HSI for minimal-invasive surgery is currently limited, for example, due to long acquisition times, missing video, or large set-ups.

AIM

An HSI laparoscope is described and evaluated to address the requirements for clinical use and high-resolution spectral imaging.

APPROACH

Reflectance measurements with reference objects and resected human tissue from 500 to 1000 nm are performed to show the consistency with an approved medical HSI device for open surgery. Varying object distances are investigated, and the signal-to-noise ratio (SNR) is determined for different light sources.

RESULTS

The handheld design enables real-time processing and visualization of HSI data during acquisition within 4.6 s. A color video is provided simultaneously and can be augmented with spectral information from push-broom imaging. The reflectance data from the HSI system for open surgery at 50 cm and the HSI laparoscope are consistent for object distances up to 10 cm. A standard rigid laparoscope in combination with a customized LED light source resulted in a mean SNR of 30 to 43 dB (500 to 950 nm).

CONCLUSIONS

Compact and rapid HSI with a high spatial- and spectral-resolution is feasible in clinical practice. Our work may support future studies on minimally invasive HSI to reduce intra- and postoperative complications.

Surgical spectral imaging

Recent technological developments have resulted in the availability of miniaturised spectral imaging sensors capable of operating in the multi- (MSI) and hyperspectral imaging (HSI) regimes. Simultaneous advances in image-processing techniques and artificial intelligence (AI), especially in machine learning and deep learning, have made these data-rich modalities highly attractive as a means of extracting biological information non-destructively. Surgery in particular is poised to benefit from this, as spectrally-resolved tissue optical properties can offer enhanced contrast as well as diagnostic and guidance information during interventions. This is particularly relevant for procedures where inherent contrast is low under standard white light visualisation. This review summarises recent work in surgical spectral imaging (SSI) techniques, taken from Pubmed, Google Scholar and arXiv searches spanning the period 2013-2019. New hardware, optimised for use in both open and minimally-invasive surgery (MIS), is described, and recent commercial activity is summarised. Computational approaches to extract spectral information from conventional colour images are reviewed, as tip-mounted cameras become more commonplace in MIS. Model-based and machine learning methods of data analysis are discussed in addition to simulation, phantom and clinical validation experiments. A wide variety of surgical pilot studies are reported but it is apparent that further work is needed to quantify the clinical value of MSI/HSI. The current trend toward data-driven analysis emphasises the importance of widely-available, standardised spectral imaging datasets, which will aid understanding of variability across organs and patients, and drive clinical translation.

Confident texture-based laryngeal tissue classification for early stage diagnosis support

Early stage diagnosis of laryngeal squamous cell carcinoma (SCC) is of primary importance for lowering patient mortality or after treatment morbidity. Despite the challenges in diagnosis reported in the clinical literature, few efforts have been invested in computer-assisted diagnosis. The objective of this paper is to investigate the use of texture-based machine-learning algorithms for early stage cancerous laryngeal tissue classification. To estimate the classification reliability, a measure of confidence is also exploited. From the endoscopic videos of 33 patients affected by SCC, a well-balanced dataset of 1320 patches, relative to four laryngeal tissue classes, was extracted. With the best performing feature, the achieved median classification recall was 93% [interquartile range [Formula: see text]]. When excluding low-confidence patches, the achieved median recall was increased to 98% ([Formula: see text]), proving the high reliability of the proposed approach. This research represents an important advancement in the state-of-the-art computer-assisted laryngeal diagnosis, and the results are a promising step toward a helpful endoscope-integrated processing system to support early stage diagnosis.

Tissue classification for laparoscopic image understanding based on multispectral texture analysis

Intraoperative tissue classification is one of the prerequisites for providing context-aware visualization in computer-assisted minimally invasive surgeries. As many anatomical structures are difficult to differentiate in conventional RGB medical images, we propose a classification method based on multispectral image patches. In a comprehensive ex vivo study through statistical analysis, we show that (1) multispectral imaging data are superior to RGB data for organ tissue classification when used in conjunction with widely applied feature descriptors and (2) combining the tissue texture with the reflectance spectrum improves the classification performance. The classifier reaches an accuracy of 98.4% on our dataset. Multispectral tissue analysis could thus evolve as a key enabling technique in computer-assisted laparoscopy.