Real-time video mosaicing with a high-resolution microendoscope

Optical fiber bundle differential compressive imaging

We present a differential compressive imaging method for an optical fiber bundle (OFB), which provides a solution for an ultrathin bend-resistant endoscope with high resolution. This method uses an OFB and a diffuser to generate speckle illumination patterns. Differential operation is additionally applied to the speckle patterns to produce sensing matrices, by which the correlation between the matrices is greatly reduced from 0.875 to 0.0275, which ensures the high quality of image reconstruction. Pixilation artifacts from the fiber core arrangement are also effectively eliminated with this configuration. We demonstrate high-resolution reconstruction of images of 132 × 132 pixels with a compression rate of 12% using 77 fiber cores, the total diameter of which is only about 91 µm. An experimental verification proves that this method is tolerant to a limited degree of fiber bending, which provides a potential approach for robust high-resolution fiber endoscopy.

Real-timing processing of fiber bundle endomicroscopy images in Python using PyFibreBundle

Fiber imaging bundles allow the transfer of optical images from place-to-place along narrow and flexible conduits. Traditionally used extensively in medical endoscopy, bundles are now finding new applications in endoscopic microscopy and other emerging techniques. PyFibreBundle is an open-source Python package for fast processing of images acquired through imaging bundles. This includes detection and removal of the fiber core pattern by filtering or interpolation, and application of background and flat-field corrections. It also allows images to be stitched together to create mosaics and resolution to be improved by combining multiple shifted images. This paper describes the technical implementation of PyFibreBundle and provides example results from three endomicroscopy imaging systems: color transmission, monochrome transmission, and confocal fluorescence. This allows various processing options to be compared quantitatively and qualitatively, and benchmarking demonstrates that PyFibreBundle can achieve state-of-the-art performance in an open-source package. The paper demonstrates core removal by interpolation and mosaicing at over 100 fps, real-time multi-frame resolution enhancement and the first demonstration of real-time endomicroscopy image processing, including core removal, on a Raspberry Pi single board computer. This demonstrates that PyFibreBundle is potentially a valuable tool for the development of low-cost, high-performance fiber bundle imaging systems.

Block-based compressed sensing for fast optic fiber bundle imaging with high spatial resolution

The resolution of traditional fiber bundle imaging is usually limited by the density and the diameter of the fiber cores. To improve the resolution, compression sensing was introduced to resolve multiple pixels from a single fiber core, but current methods have the drawbacks of excessive sampling and long reconstruction time. In this paper, we present, what we believe to be, a novel block-based compressed sensing scheme for fast realization of high-resolution optic fiber bundle imaging. In this method, the target image is segmented into multiple small blocks, each of which covers the projection area of one fiber core. All block images are independently and simultaneously sampled and the intensities are recorded by a two-dimensional detector after they are collected and transmitted through corresponding fiber cores. Because the size of sampling patterns and the sampling numbers are greatly reduced, the reconstruction complexity and reconstruction time are also decreased. According to the simulation analysis, our method is 23 times faster than the current compressed sensing optical fiber imaging for reconstructing a fiber image of 128 × 128 pixels, while the sampling number is only 0.39%. Experiment results demonstrate that the method is also effective for reconstructing large target images and the number of sampling does not increase with the size of the image. Our finding may provide a new idea for high-resolution real-time imaging of fiber bundle endoscope.

Depixelation and image restoration with meta-learning in fiber-bundle-based endomicroscopy

In order to efficiently remove honeycomb artifacts and restore images in fiber-bundle-based endomicroscopy, we develop a meta-learning algorithm in this work. Two sub-networks are used to extract different levels of features. Meta-training is employed to train the network with small amount of simulated training data, enabling the optimal model to generalize to new tasks not seen in the training set. Numerical results on both USAF target and endomicroscopy images of living mice tissues demonstrate that the algorithm can restore high contrast image without pixilated noise using shorter time. Additionally, no prior information on the shape of the underlying tissues and the distribution of fiber bundles is required, making the method applicable to a variety of fiber-bundle-based endomicroscopy imaging conditions.

Depixelation of coherent fiber bundle imaging by fiber-core-targeted scanning

A novel fast proximal scanning method, to the best of our knowledge, termed fiber-core-targeted scanning (FCTS), is proposed for illuminating individual fiber cores sequentially to remove the pixelation effect in fiber bundle (FB) imaging. FCTS is based on a galvanometer scanning system. Through a dynamic control of the scan trajectory and speed using the prior knowledge of fiber core positions, FCTS experimentally verifies a precise sequential delivery of laser pulses into fiber cores at a maximal speed of 45,000 cores per second. By applying FCTS on a FB-based photoacoustic forward-imaging probe, the results demonstrate that FCTS eliminates the pixelation effect and improves the imaging quality.

High frame rate video mosaicking microendoscope to image large regions of intact tissue with subcellular resolution

High-resolution microendoscopy (HRME) is a low-cost strategy to acquire images of intact tissue with subcellular resolution at frame rates ranging from 11 to 18 fps. Current HRME imaging strategies are limited by the small microendoscope field of view (∼0.5 mm2); multiple images must be acquired and reliably registered to assess large regions of clinical interest. Image mosaics have been assembled from co-registered frames of video acquired as a microendoscope is slowly moved across the tissue surface, but the slow frame rate of previous HRME systems made this approach impractical for acquiring quality mosaicked images from large regions of interest. Here, we present a novel video mosaicking microendoscope incorporating a high frame rate CMOS sensor and optical probe holder to enable high-speed, high quality interrogation of large tissue regions of interest. Microendoscopy videos acquired at >90 fps are assembled into an image mosaic. We assessed registration accuracy and image sharpness across the mosaic for images acquired with a handheld probe over a range of translational speeds. This high frame rate video mosaicking microendoscope enables in vivo probe translation at >15 millimeters per second while preserving high image quality and accurate mosaicking, increasing the size of the region of interest that can be interrogated at high resolution from 0.5 mm2 to >30 mm2. Real-time deployment of this high-frame rate system is demonstrated in vivo and source code made publicly available.

Laparoscopic surgical device with modular end tools for real-time endomicroscopy and therapy

In recent years, laparoscopic surgery attracts considerable attention for its distinct advantages owing to minimized invasiveness. However, traditional surgical instruments have a single function and lack the ability to provide high-resolution images for real-time tissue assessment. Thus, it remains a challenge to perform complex procedures with the traditional surgical instruments. In this paper, a novel laparoscopic surgical device for real-time endomicroscopy and therapy is proposed. The device is integrated with a visualization unit to provide endoscopic images in a constrained cavity for real-time observation. Moreover, modular end tools can be assembled and switched with the multi-DOF manipulator. The experimental results demonstrate that the surface-scanning tool can perform smooth scanning and obtain a large area mosaic of 6.99 mm² in 40.2 s for tissue assessment. The spring-loaded forceps can generate an output force of 1.47 N to manipulate tissue. These experimental results demonstrate the potential value of the device for intraoperative endomicroscopy-guided surgery and robot-assisted surgery.

TEM image restoration from fast image streams

Microscopy imaging experiments generate vast amounts of data, and there is a high demand for smart acquisition and analysis methods. This is especially true for transmission electron microscopy (TEM) where terabytes of data are produced if imaging a full sample at high resolution, and analysis can take several hours. One way to tackle this issue is to collect a continuous stream of low resolution images whilst moving the sample under the microscope, and thereafter use this data to find the parts of the sample deemed most valuable for high-resolution imaging. However, such image streams are degraded by both motion blur and noise. Building on deep learning based approaches developed for deblurring videos of natural scenes we explore the opportunities and limitations of deblurring and denoising images captured from a fast image stream collected by a TEM microscope. We start from existing neural network architectures and make adjustments of convolution blocks and loss functions to better fit TEM data. We present deblurring results on two real datasets of images of kidney tissue and a calibration grid. Both datasets consist of low quality images from a fast image stream captured by moving the sample under the microscope, and the corresponding high quality images of the same region, captured after stopping the movement at each position to let all motion settle. We also explore the generalizability and overfitting on real and synthetically generated data. The quality of the restored images, evaluated both quantitatively and visually, show that using deep learning for image restoration of TEM live image streams has great potential but also comes with some limitations.

Robust Mosaicing of Endomicroscopic Videos via Context-Weighted Correlation Ratio

Probe-based confocal laser endomicroscopy (pCLE) is a promising imaging tool that provides in situ and in vivo optical imaging to perform real-time pathological assessments. However, due to limited field of view, it is difficult for clinicians to get a full understanding of the scanned tissues. In this paper, we develop a novel mosaicing framework to assemble all frame sequences into a full view image. First, a hybrid rigid registration that combines feature matching and template matching is presented to achieve a global alignment of all frames. Then, the parametric free-form deformation (FFD) model with a multiresolution architecture is implemented to accommodate non-rigid tissue distortions. More importantly, we devise a robust similarity metric called context-weighted correlation ratio (CWCR) to promote registration accuracy, where spatial and geometric contexts are incorporated into the estimation of functional intensity dependence. Experiments on both robotic setup and manual manipulation have demonstrated that the proposed scheme significantly precedes some state-of-the-art mosaicing schemes in the presence of intensity fluctuations, insufficient overlap and tissue distortions. Moreover, the comparisons of the proposed CWCR metric and two other metrics have validated the effectiveness of the context-weighted strategy in quantifying the differences between two frames. Benefiting from more rational and delicate mosaics, the proposed scheme is more suitable to instruct diagnosis and treatment during optical biopsies.

MOSAICING OF DYNAMIC MESENTERY VIDEO WITH GRADIENT BLENDING

In biomedical imaging using video microscopy, understanding large tissue structures at cellular and finer resolution poses many image acquisition challenges including limited field-of-view and tissue dynamics during imaging. Automated mosaicing or stitching of live tissue video microscopy enables the visualization and analysis of subtle morphological structures and large scale vessel network architecture in tissues like the mesentery. But mosacing can be challenging if there are deformable, motion-blurred, textureless, feature-poor frames. Feature-based methods perform poorly in such cases for the lack of distinctive keypoints. Standard single block correlation matching strategies might not provide robust registration due to deformable content. In addition, the panorama suffers if there is motion blur present in a sequence. To handle these challenges, we propose a novel algorithm, Deformable Normalized Cross Correlation (DNCC) image matching with RANSAC to establish robust registration. Besides, to produce seamless panorama from motion-blurred frames we present gradient blending method based on image edge information. The DNCC algorithm is applied on Frog Mesentery sequences. Our result is compared with PSS/AutoStitch [1, 2] to establish the efficiency and robustness of the proposed DNCC method.

Real-time video mosaicking to guide handheld in vivo microscopy

Handheld and endoscopic optical-sectioning microscopes are being developed for noninvasive screening and intraoperative consultation. Imaging a large extent of tissue is often desired, but miniature in vivo microscopes tend to suffer from limited fields of view. To extend the imaging field during clinical use, we have developed a real-time video mosaicking method, which allows users to efficiently survey larger areas of tissue. Here, we modified a previous post-processing mosaicking method so that real-time mosaicking is possible at >30 frames/second when using a device that outputs images that are 400 × 400 pixels in size. Unlike other real-time mosaicking methods, our strategy can accommodate image rotations and deformations that often occur during clinical use of a handheld microscope. We perform a feasibility study to demonstrate that the use of real-time mosaicking is necessary to enable efficient sampling of a desired imaging field when using a handheld dual-axis confocal microscope.

Image computing for fibre-bundle endomicroscopy: A review

Endomicroscopy is an emerging imaging modality, that facilitates the acquisition of in vivo, in situ optical biopsies, assisting diagnostic and potentially therapeutic interventions. While there is a diverse and constantly expanding range of commercial and experimental optical biopsy platforms available, fibre-bundle endomicroscopy is currently the most widely used platform and is approved for clinical use in a range of clinical indications. Miniaturised, flexible fibre-bundles, guided through the working channel of endoscopes, needles and catheters, enable high-resolution imaging across a variety of organ systems. Yet, the nature of image acquisition though a fibre-bundle gives rise to several inherent characteristics and limitations necessitating novel and effective image pre- and post-processing algorithms, ranging from image formation, enhancement and mosaicing to pathology detection and quantification. This paper introduces the underlying technology and most prevalent clinical applications of fibre-bundle endomicroscopy, and provides a comprehensive, up-to-date, review of relevant image reconstruction, analysis and understanding/inference methodologies. Furthermore, current limitations as well as future challenges and opportunities in fibre-bundle endomicroscopy computing are identified and discussed.

Low-cost, ultracompact handheld optical coherence tomography probe for in vivo oral maxillofacial tissue imaging

SIGNIFICANCE

Optical coherence tomography (OCT) has proven useful for detecting various oral maxillofacial abnormalities. To apply it to clinical applications including biopsy guidance and routine screening, a handheld imaging probe is indispensable. OCT probes reported for oral maxillofacial imaging were either based on a bulky galvanometric mirror pair (not compact or long enough) or a distal-end microelectromechanical systems (MEMS) scanner (raised safety concerns), or adapted from fiber-optic catheters (ill-suited for oral cavity geometry).

AIM

To develop a handheld probe featuring great compactness and excellent maneuverability for oral maxillofacial tissue imaging.

APPROACH

A dual-axis MEMS scanner was deployed at the proximal end of the probe and the scanned beam was relayed to the distal end through a 4f configuration. Such design provides both a perfect dual-axis telecentric scan and excellent compactness.

RESULTS

A handheld probe with a rigid part 70 mm in length and 7 mm in diameter and weighing 25 g in total was demonstrated through both ex vivo and in vivo experiments, including structural visualization of various oral maxillofacial tissues and monitoring the recovery process of an oral mucosa canker sore.

CONCLUSIONS

The proposed probe exhibits excellent maneuverability and imaging performance showing great potential in clinical applications.

Software-Based Phase Control, Video-Rate Imaging, and Real-Time Mosaicing With a Lissajous-Scanned Confocal Microscope

We present software-based methods for automatic phase control and for mosaicing high-speed, Lissajous-scanned images. To achieve imaging speeds fast enough for mosaicing, we first increase the image update rate tenfold from 3 to 30 Hz, then vertically interpolate each sparse image in real-time to eliminate fixed pattern noise. We validate our methods by imaging fluorescent beads and automatically maintaining phase control over the course of one hour. We then image fixed mouse brain tissues at varying update rates and compare the resulting mosaics. Using reconstructed image data as feedback for phase control eliminates the need for phase sensors and feedback controllers, enabling long-term imaging experiments without additional hardware. Mosaicing subsampled images results in video-rate imaging speeds, nearly fully recovered spatial resolution, and millimeter-scale fields of view.

Automatic motion compensation for structured illumination endomicroscopy using a flexible fiber bundle

SIGNIFICANCE

Confocal laser scanning enables optical sectioning in clinical fiber bundle endomicroscopes, but lower-cost, simplified endomicroscopes use widefield incoherent illumination instead. Optical sectioning can be introduced in these simple systems using structured illumination microscopy (SIM), a multiframe digital subtraction process. However, SIM results in artifacts when the probe is in motion, making the technique difficult to use in vivo and preventing the use of mosaicking to synthesize a larger effective field of view (FOV).

AIM

We report and validate an automatic motion compensation technique to overcome motion artifacts and allow generation of mosaics in SIM endomicroscopy.

APPROACH

Motion compensation is achieved using image registration and real-time pattern orientation correction via a digital micromirror device. We quantify the similarity of moving probe reconstructions to those acquired with a stationary probe using the relative mean of the absolute differences (MAD). We further demonstrate mosaicking with a moving probe in mechanical and freehand operation.

RESULTS

Reconstructed SIM images show an improvement in the MAD from 0.85 to 0.13 for lens paper and from 0.27 to 0.12 for bovine tissue. Mosaics also show vastly reduced artifacts.

CONCLUSION

The reduction in motion artifacts in individual SIM reconstructions leads to mosaics that more faithfully represent the morphology of tissue, giving clinicians a larger effective FOV than the probe itself can provide.

Depixelation and enhancement of fiber bundle images by bundle rotation

Fiber bundles have become widely adopted for use in endoscopy, live-organism imaging, and other imaging applications. An inherent consequence of imaging with these bundles is the introduction of a honeycomb-like artifact that arises from the inter-fiber spacing, which obscures features of objects in the image. This artifact subsequently limits applicability and can make interpretation of the image-based data difficult. This work presents a method to reduce this artifact by on-axis rotation of the fiber bundle. Fiber bundle images were first low-pass and median filtered to improve image quality. Consecutive filtered images with rotated samples were then co-registered and averaged to generate a final, reconstructed image. The results demonstrate removal of the artifacts, in addition to increased signal contrast and signal-to-noise ratio. This approach combines digital filtering and spatial resampling to reconstruct higher-quality images, enhancing the utility of images acquired using fiber bundles.

Multichannel correlation improves the noise tolerance of real-time hyperspectral microimage mosaicking

Live-subject microscopies, including microendoscopy and other related technologies, offer promise for basic biology research as well as the optical biopsy of disease in the clinic. However, cellular resolution generally comes with the trade-off of a microscopic field-of-view. Microimage mosaicking enables stitching many small scenes together to aid visualization, quantitative interpretation, and mapping of microscale features, for example, to guide surgical intervention. The development of hyperspectral and multispectral systems for biomedical applications provides motivation for adapting mosaicking algorithms to process a number of simultaneous spectral channels. We present an algorithm that mosaics multichannel video by correlating channels of consecutive frames as a basis for efficiently calculating image alignments. We characterize the noise tolerance of the algorithm by using simulated video with known ground-truth alignments to quantify mosaicking accuracy and speed, showing that multiplexed molecular imaging enhances mosaic accuracy by leveraging observations of distinct molecular constituents to inform frame alignment. A simple mathematical model is introduced to characterize the noise suppression provided by a given group of spectral channels, thus predicting the performance of selected subsets of data channels in order to balance mosaic computation accuracy and speed. The characteristic noise tolerance of a given number of channels is shown to improve through selection of an optimal subset of channels that maximizes this model. We also demonstrate that the multichannel algorithm produces higher quality mosaics than the analogous single-channel methods in an empirical test case. To compensate for the increased data rate of hyperspectral video compared to single-channel systems, we employ parallel processing via GPUs to alleviate computational bottlenecks and to achieve real-time mosaicking even for video-rate multichannel systems anticipated in the future. This implementation paves the way for real-time multichannel mosaicking to accompany next-generation hyperspectral and multispectral video microscopy.

Toward development of a large field-of-view cancer screening patch (CASP) to detect cervical intraepithelial neoplasia

Cervical cancers are primarily diagnosed via colposcopy, in which the tissue is visually assessed by a clinician for abnormalities, followed by directed biopsies and histologic analysis of excised tissue. Optical biopsy technologies offer a less invasive method of imaging such that subcellular features can be resolved without removing tissue. These techniques, however, are limited in field-of-view by the distal end of the probe. We present a prototype that incorporates a rigid, machinable waveguide that is in direct contact with a fluorescently-labeled sample paired with a scanning fluorescent microscope. The system is capable of imaging large areas of tissue without the need to re-position the tissue-probe interface. A mosaicing algorithm was developed to quantify scanning shifts and stitch neighboring frames together to increase the field-of-view. Our prototype can yield a maximum axial resolution of <5 µm for individual frames and can produce mosaiced images with a field-of-view greater than 15 mm x 15 mm without sacrificing resolution. We validated the system with a 1951 USAF resolution target, fluorescent in vitro standards, and a patient study where ex vivo conization samples of squamous cervical epithelium were imaged. The results of the patient study indicate that architectural features of subcellular components could be detected and differentiated between normal tissue and precancerous lesions.

Feasibility of a Video-Mosaicking Approach to Extend the Field-of-View For Reflectance Confocal Microscopy in the Oral Cavity In Vivo

BACKGROUND

Reflectance confocal microscopy (RCM) is a developing approach for noninvasive detection of oral lesions with label-free contrast and cellular-level resolution. For access into the oral cavity, confocal microscopes are being configured with small-diameter telescopic probes and small objective lenses. However, a small probe and objective lens allows for a rather small field-of-view relative to the large areas of tissue that must be examined for diagnosis. To extend the field-of-view for intraoral RCM imaging, we are investigating a video-mosaicking approach.

METHODS

A relay telescope and objective lens were adapted to an existing confocal microscope for access into the oral cavity. Imaging was performed using metal three-dimensional-printed objective lens front-end caps with coverslip windows to contact and stabilize the tissue and set depth. Four healthy volunteers (normal oral mucosa), one patient (with an amalgam tattoo) in a clinical setting, and 20 anesthetized patients (with oral squamous cell carcinoma [OSCC]) in a surgical setting were imaged. Instead of the usual still RCM images, videos were recorded and then processed into video-mosaics. Thirty video-mosaics were read and qualitatively assessed by an expert reader of RCM images of the oral mucosa.

RESULTS

Whereas the objective lens' native field-of-view is 0.75 mm × 0.75 mm, the video-mosaics display larger areas, ranging from 2 mm × 2 mm to 4 mm × 2 mm, with resolution, morphologic detail, and image quality that is preserved relative to that observed in the original videos (individual images). Video-mosaics in healthy volunteers' and the patients' images showed cellular morphologic patterns in the lower epithelium and at the epithelial junction, and connective tissue along with capillary loops and blood flow in the deeper lamina propria. In OSCC, tumor nests could be observed along with normal looking mucosa in margin areas.

CONCLUSIONS

Video-mosaicking is a reasonably quick and efficient approach for extending the field-of-view of RCM imaging, which can, to some extent, overcome the inherent limitation of an intraoral probe's small field-of-view. Reading video-mosaics can mimic the procedure for examining pathology: initial visualization of the spatial cellular and morphologic patterns of the tumor and the spread of tumor margins over larger areas of the lesion, followed by digitally zooming (magnifying) for closer inspection of suspicious areas. However, faster processing of videos into video-mosaics will be necessary, to allow examination of video-mosaics in real-time at the bedside. Lasers Surg. Med. 51:439-451, 2019. © 2019 Wiley Periodicals, Inc.

En-face optical coherence tomography/fluorescence endomicroscopy for minimally invasive imaging using a robotic scanner

We report a compact rigid instrument capable of delivering en-face optical coherence tomography (OCT) images alongside (epi)-fluorescence endomicroscopy (FEM) images by means of a robotic scanning device. Two working imaging channels are included: one for a one-dimensional scanning, forward-viewing OCT probe and another for a fiber bundle used for the FEM system. The robotic scanning system provides the second axis of scanning for the OCT channel while allowing the field of view (FoV) of the FEM channel to be increased by mosaicking. The OCT channel has resolutions of 25  /  60  μm (axial/lateral) and can provide en-face images with an FoV of 1.6  ×  2.7  mm2. The FEM channel has a lateral resolution of better than 8  μm and can generate an FoV of 0.53  ×  3.25  mm2 through mosaicking. The reproducibility of the scanning was determined using phantoms to be better than the lateral resolution of the OCT channel. Combined OCT and FEM imaging were validated with ex-vivo ovine and porcine tissues, with the instrument mounted on an arm to ensure constant contact of the probe with the tissue. The OCT imaging system alone was validated for in-vivo human dermal imaging with the handheld instrument. In both cases, the instrument was capable of resolving fine features such as the sweat glands in human dermal tissue and the alveoli in porcine lung tissue.

Neurosurgical Flexible Probe Microscopy with Enhanced Architectural and Cytological Detail

BACKGROUND

Microscopic delineation and clearance of tumor cells at neurosurgical excision margins potentially reduce tumor recurrence and increase patient survival. Probe-based in vivo fluorescence microscopy technologies are promising for neurosurgical in vivo microscopy.

OBJECTIVE

We sought to demonstrate a flexible fiberoptic epifluorescence microscope capable of enhanced architectural and cytological imaging for in vivo microscopy during neurosurgical procedures.

METHODS

Eighteen specimens were procured from neurosurgical procedures. These specimens were stained with acridine orange and imaged with a 3-dimensional (3D)-printed epifluorescent microscope that incorporates a flexible fiberoptic probe. Still images and video sequence frames were processed using frame alignment, signal projection, and pseudo-coloring, resulting in resolution enhancement and an increased field of view.

RESULTS

Images produced displayed good nuclear contrast and architectural detail. Grade 1 meningiomas demonstrated 3D chords and whorls. Low-grade meningothelial nuclei showed streaming and displayed regularity in size, shape, and distribution. Oligodendrogliomas showed regular round nuclei and a variably staining background. Glioblastomas showed high degrees of nuclear pleomorphism and disarray. Mitoses, vascular proliferation, and necrosis were evident.

CONCLUSIONS

We demonstrate the utility of a 3D-printed, flexible probe microscope for high-resolution microscopic imaging with increased architectural detail. Enhanced in vivo imaging using this device may improve our ability to detect and decrease microscopic tumor burden at excision margins during neurosurgical procedures.

A mobile-phone based high-resolution microendoscope to image cervical precancer

Nearly 90% of cervical cancer cases and deaths occur in low- and middle-income countries that lack comprehensive national HPV immunization and cervical cancer screening programs. In these settings, it is difficult to implement screening programs due to a lack of infrastructure and shortage of trained personnel. Screening programs based on visual inspection with acetic acid (VIA) have been successfully implemented in some low-resource settings. However, VIA has poor specificity and up to 90% of patients receiving treatment based on a positive VIA exam are over-treated. A number of studies have suggested that high-resolution cervical imaging to visualize nuclear morphology in vivo can improve specificity by better distinguishing precancerous and benign lesions. To enable high-resolution imaging in low-resource settings, we developed a portable, low-cost, high-resolution microendoscope that uses a mobile phone to detect and display images of cervical epithelium in vivo with subcellular resolution. The device was fabricated for less than $2,000 using commercially available optical components including filters, an LED and triplet lenses assembled in a 3D-printed opto-mechanical mount. We show that the mobile high-resolution microendoscope achieves similar resolution and signal-to-background ratio as previously reported high-resolution microendoscope systems using traditional cameras and computers to detect and display images. Finally, we demonstrate the ability of the mobile high-resolution microendoscope to image normal and precancerous squamous epithelium of the cervix in vivo in a gynecological referral clinic in Barretos, Brazil.

Quantitative analysis of in vivo high-resolution microendoscopic images for the detection of neoplastic colorectal polyps

Colonoscopy is routinely performed for colorectal cancer screening but lacks the capability to accurately characterize precursor lesions and early cancers. High-resolution microendoscopy (HRME) is a low-cost imaging tool to visualize colorectal polyps with subcellular resolution. We present a computer-aided algorithm to evaluate HRME images of colorectal polyps and classify neoplastic from benign lesions. Using histopathology as the gold standard, clinically relevant features based on luminal morphology and texture are quantified to build the classification algorithm. We demonstrate that adenomatous polyps can be identified with a sensitivity and specificity of 100% and 80% using a two-feature linear discriminant model in a pilot test set. The classification algorithm presented here offers an objective framework to detect adenomatous lesions in the colon with high accuracy and can potentially improve real-time assessment of colorectal polyps.

Economical and compact briefcase spectral-domain optical coherence tomography system for primary care and point-of-care applications

Development of low-cost and portable optical coherence tomography (OCT) systems is of global interest in the OCT research community. Such systems enable utility broadly throughout a clinical facility, or in remote areas that often lack clinical infrastructure. We report the development and validation of a low-cost, portable briefcase spectral-domain optical coherence tomography (SD-OCT) system for point-of-care diagnostics in primary care centers and/or in remote settings. The self-contained briefcase OCT contains all associated optical hardware, including light source, spectrometer, hand-held probe, and a laptop. Additionally, this system utilizes unique real-time mosaicking of surface video images that are synchronized with rapid A-scan acquisition to eliminate the need for lateral scanning hardware, and enable the construction of cross-sectional B-mode images over extended lateral distances. The entire briefcase system weighs 9 kg and costs ∼USD$8000 using off-the-shelf components. System performance was validated by acquiring images of in vivo human skin on the fingertip, palm, and nail fold. The efficiency, portability, and low-cost enable accessibility and utility in primary care centers and low-resource settings.

In-vivo optical imaging in head and neck oncology: basic principles, clinical applications and future directions

Head and neck cancers become a severe threat to human's health nowadays and represent the sixth most common cancer worldwide. Surgery remains the first-line choice for head and neck cancer patients. Limited resectable tissue mass and complicated anatomy structures in the head and neck region put the surgeons in a dilemma between the extensive resection and a better quality of life for the patients. Early diagnosis and treatment of the pre-malignancies, as well as real-time in vivo detection of surgical margins during en bloc resection, could be leveraged to minimize the resection of normal tissues. With the understanding of the head and neck oncology, recent advances in optical hardware and reagents have provided unique opportunities for real-time pre-malignancies and cancer imaging in the clinic or operating room. Optical imaging in the head and neck has been reported using autofluorescence imaging, targeted fluorescence imaging, high-resolution microendoscopy, narrow band imaging and the Raman spectroscopy. In this study, we reviewed the basic theories and clinical applications of optical imaging for the diagnosis and treatment in the field of head and neck oncology with the goal of identifying limitations and facilitating future advancements in the field.

High-resolution microendoscopy: a point-of-care diagnostic for cervical dysplasia in low-resource settings

Cervical cancer is the third leading cause of cancer-related death among women in low-to-middle income countries. Pap testing and pathological services are difficult to implement under these settings. Alternative techniques for the diagnosis of cervical precancer in these settings are needed to reduce the burden of the disease. The objective of this study was to evaluate the diagnostic accuracy of a low-cost, high-resolution microendoscope imaging system in identifying precancerous lesions of the cervix in vivo. A retrospective study of 59 patients undergoing colposcopy for an abnormal Pap test was performed at Hospital de Câncer de Barretos in Brazil. All patients underwent colposcopy as per standard of care, and acetowhite lesions were recorded. High-resolution microendoscopy (HRME) images were obtained from one colposcopically normal region and from all lesions observed on colposcopy. Biopsies of abnormal areas were obtained and reviewed by three independent, blinded pathologists and compared with HRME findings. The mean nuclear area and the median nuclear eccentricity were calculated from HRME images acquired from each site. A diagnostic algorithm to distinguish histopathologically diagnosed cervical intraepithelial neoplasias of grade 2 or more severe lesions (high grade) from less severe lesions (low grade) was developed using these parameters. A test of trend was used to analyze the relationship between HRME positivity and severity of histopathogical diagnosis. Fisher's exact test was used to analyze differences in HRME positivity between high-grade and low-grade lesions. Evaluable images were obtained from 108 of 143 discrete sites. Of these, 71 sites were colposcopically normal or low grade according to histopathology and 37 were diagnosed as high grade on the basis of histopathology. Using the mean nuclear area and the median nuclear eccentricity, HRME images from 59 colposcopically abnormal sites were classified as high grade or low grade with 92% sensitivity and 77% specificity compared with histopathological findings. Increasing HRME positivity showed a significant trend with increasing severity of diagnosis (Ptrend<0.001). We found a strong association (P<0.001) between HRME positivity and a histopathological diagnosis of cervical intraepithelial neoplasia of grade 2 or higher. HRME demonstrated an accurate in-situ diagnosis of high-grade dysplasia. In low-resource settings in which colposcopy and histopathology services are severely limited or unavailable, HRME may provide a low-cost, accurate method for diagnosis of cervical precancer without the need for biopsy, allowing for a single 'screen-and-treat' approach.

Robotic Scanning Device for Intraoperative Thyroid Gland Endomicroscopy

Probe-based confocal laser endomicroscopy has a high potential to be a promising tool that can provide intraoperative high-resolution in vivo morphological imaging at cellular and subcellular levels for thyroidectomy, and allow real-time assessment of tumor margins. However, the typical images acquired with this technique cover a very small area limited by the field of view of the probe, accompanied by tissue deformation and inconsistent probe-tissue contact when operated manually. In this paper, a novel compact robotic device for large area scanning has been developed. The device can scan a large surface in a spiral trajectory by rotating the tip frame along the spiral groove of the base frame. The fiber Bragg grating sensor with a passive linear structure is used to detect and maintain a stable probe-tissue contact force during scanning. An active linear actuation is also integrated for adjusting the probe-tissue contact level prior to each scan. Results demonstrate that the scanning device ensures a suitable probe-tissue contact force and compensates for simulated hand tremor. Mosaicing results of lens tissue paper and porcine belly tissue with both bench and hand-held experiments show the effectiveness and usability of the device, demonstrating the potential clinical value of the system.

Automated video-mosaicking approach for confocal microscopic imaging in vivo: an approach to address challenges in imaging living tissue and extend field of view

We describe a computer vision-based mosaicking method for in vivo videos of reflectance confocal microscopy (RCM). RCM is a microscopic imaging technique, which enables the users to rapidly examine tissue in vivo. Providing resolution at cellular-level morphology, RCM imaging combined with mosaicking has shown to be highly sensitive and specific for non-invasively guiding skin cancer diagnosis. However, current RCM mosaicking techniques with existing microscopes have been limited to two-dimensional sequences of individual still images, acquired in a highly controlled manner, and along a specific predefined raster path, covering a limited area. The recent advent of smaller handheld microscopes is enabling acquisition of videos, acquired in a relatively uncontrolled manner and along an ad-hoc arbitrarily free-form, non-rastered path. Mosaicking of video-images (video-mosaicking) is necessary to display large areas of tissue. Our video-mosaicking methods addresses this need. The method can handle unique challenges encountered during video capture such as motion blur artifacts due to rapid motion of the microscope over the imaged area, warping in frames due to changes in contact angle and varying resolution with depth. We present test examples of video-mosaics of melanoma and non-melanoma skin cancers, to demonstrate potential clinical utility.

Handheld tunable focus confocal microscope utilizing a double-clad fiber coupler for in vivo imaging of oral epithelium

A reflectance confocal endomicroscope with double-clad fiber coupler and electrically tunable focus lens is applied to imaging of the oral mucosa. The instrument is designed to be lightweight and robust for clinical use. The tunable lens allows axial scanning through > 250 ?? ? m in the epithelium when the probe tip is placed in contact with tissue. Images are acquired at 6.6 frames per second with a field of view diameter up to 850 ?? ? m . In vivo imaging of a wide range of normal sites in the oral cavity demonstrates the accessibility of the handheld probe. In vivo imaging of clinical lesions diagnosed as inflammation and dysplasia illustrates the ability of reflectance confocal endomicroscopy to image cellular changes associated with pathology.

Endomicroscopy for Computer and Robot Assisted Intervention

Endomicroscopy is a new technique that allows human tissue to be characterized in vivo and in situ, circumventing the need for conventional biopsy and histology. Despite increased application and growing research interests in this area, the clinical application of endomicroscopy, however, is limited by difficulties in ergonomic control, consistent probe-tissue contact, large area surveillance, and retargeting. Recently, advances in high-speed imaging, mosaicing, and robotics have aimed to address these difficulties. The development of robot-assisted devices in particular has shown great promises in extending the clinical potential of endomicroscopy. Issues related to miniaturization, adaptation to tissue deformation, control stability, force and position compensation, cost, and sterility are being pursued by both research and commercial communities. In this review, recent clinical and technical developments in different aspects of computer and robotic assisted endomicroscopy interventions including instrumentation, multiscale integration, and high-speed imaging techniques are presented. We further address emerging trends and new research opportunities toward more widespread clinical acceptance of robotically assisted endomicroscopy technologies.

Linked color imaging application for improving the endoscopic diagnosis accuracy: a pilot study

Endoscopy has been widely used in diagnosing gastrointestinal mucosal lesions. However, there are still lack of objective endoscopic criteria. Linked color imaging (LCI) is newly developed endoscopic technique which enhances color contrast. Thus, we investigated the clinical application of LCI and further analyzed pixel brightness for RGB color model. All the lesions were observed by white light endoscopy (WLE), LCI and blue laser imaging (BLI). Matlab software was used to calculate pixel brightness for red (R), green (G) and blue color (B). Of the endoscopic images for lesions, LCI had significantly higher R compared with BLI but higher G compared with WLE (all P < 0.05). R/(G + B) was significantly different among 3 techniques and qualified as a composite LCI marker. Our correlation analysis of endoscopic diagnosis with pathology revealed that LCI was quite consistent with pathological diagnosis (P = 0.000) and the color could predict certain kinds of lesions. ROC curve demonstrated at the cutoff of R/(G+B) = 0.646, the area under curve was 0.646, and the sensitivity and specificity was 0.514 and 0.773. Taken together, LCI could improve efficiency and accuracy of diagnosing gastrointestinal mucosal lesions and benefit target biopsy. R/(G + B) based on pixel brightness may be introduced as a objective criterion for evaluating endoscopic images.

Differential structured illumination microendoscopy for in vivo imaging of molecular contrast agents

Fiber optic microendoscopy has shown promise for visualization of molecular contrast agents used to study disease in vivo. However, fiber optic microendoscopes have limited optical sectioning capability, and image contrast is limited by out-of-focus light generated in highly scattering tissue. Optical sectioning techniques have been used in microendoscopes to remove out-of-focus light but reduce imaging speed or rely on bulky optical elements that prevent in vivo imaging. Here, we present differential structured illumination microendoscopy (DSIMe), a fiber optic system that can perform structured illumination in real time for optical sectioning without any opto-mechanical components attached to the distal tip of the fiber bundle. We demonstrate the use of DSIMe during in vivo fluorescence imaging in patients undergoing surgery for cervical adenocarcinoma in situ. Images acquired using DSIMe show greater contrast than standard microendoscopy, improving the ability to detect cellular atypia associated with neoplasia.

Quantitative analysis of ex vivo colorectal epithelium using an automated feature extraction algorithm for microendoscopy image data

Qualitative screening for colorectal polyps via fiber bundle microendoscopy imaging has shown promising results, with studies reporting high rates of sensitivity and specificity, as well as low interobserver variability with trained clinicians. A quantitative image quality control and image feature extraction algorithm (QFEA) was designed to lessen the burden of training and provide objective data for improved clinical efficacy of this method. After a quantitative image quality control step, QFEA extracts field-of-view area, crypt area, crypt circularity, and crypt number per image. To develop and validate this QFEA, a training set of microendoscopy images was collected from freshly resected porcine colon epithelium. The algorithm was then further validated on ex vivo image data collected from eight human subjects, selected from clinically normal appearing regions distant from grossly visible tumor in surgically resected colorectal tissue. QFEA has proven flexible in application to both mosaics and individual images, and its automated crypt detection sensitivity ranges from 71 to 94% despite intensity and contrast variation within the field of view. It also demonstrates the ability to detect and quantify differences in grossly normal regions among different subjects, suggesting the potential efficacy of this approach in detecting occult regions of dysplasia.

Quantitative analysis of high-resolution microendoscopic images for diagnosis of neoplasia in patients with Barrett's esophagus

BACKGROUND AND AIMS

Previous studies show that microendoscopic images can be interpreted visually to identify the presence of neoplasia in patients with Barrett's esophagus (BE), but this approach is subjective and requires clinical expertise. This study describes an approach for quantitative image analysis of microendoscopic images to identify neoplastic lesions in patients with BE.

METHODS

Images were acquired from 230 sites from 58 patients by using a fiberoptic high-resolution microendoscope during standard endoscopic procedures. Images were analyzed by a fully automated image processing algorithm, which automatically selected a region of interest and calculated quantitative image features. Image features were used to develop an algorithm to identify the presence of neoplasia; results were compared with a histopathology diagnosis.

RESULTS

A sequential classification algorithm that used image features related to glandular and cellular morphology resulted in a sensitivity of 84% and a specificity of 85%. Applying the algorithm to an independent validation set resulted in a sensitivity of 88% and a specificity of 85%.

CONCLUSIONS

This pilot study demonstrates that automated analysis of microendoscopic images can provide an objective, quantitative framework to assist clinicians in evaluating esophageal lesions from patients with BE. (

CLINICAL TRIAL REGISTRATION NUMBER

NCT01384227 and NCT02018367.).

High-resolution microendoscopy in differentiating neoplastic from non-neoplastic colorectal polyps

Colorectal cancer is one of the leading causes of death worldwide. The progression from adenoma to cancer is a well known phenomenon. Current clinical practice favors colonoscopy as the preferred modality for colorectal cancer screening. Many novel endoscopic technologies are emerging for the purposes of performing "optical biopsy" to allow real-time histologic diagnosis of polyps. High resolution microendoscopy is a low-cost endoscopic technology that has demonstrated high sensitivity and specificity in differentiating neoplastic and non-neoplastic polyps. With the ability to make real-time conclusions based on the endoscopic appearance of polyps, it is becoming increasingly possible to decrease the rate of unnecessary polypectomies and utilize a "resect and discard" strategy to decrease costs of pathology evaluation. Future directions for this technology include surveillance of premalignant conditions such as inflammatory bowel disease. Moreover, the low cost and relative ease of use of this technology lends itself to widespread applicability.

Feasibility of transoral robotic-assisted high-resolution microendoscopic imaging of oropharyngeal squamous cell carcinoma

BACKGROUND

Transoral robotic-assisted oncologic surgery of the head and neck offers promising functional results. Nonetheless, the efficacy of oncologic surgery remains critically dependent on obtaining negative margins. We aimed to integrate a miniaturized high-resolution fiber-optic microendoscope (HRME), which provides real-time histological assessment, with the da Vinci robotic system (Intuitive Surgical, Sunnyvale, CA).

METHODS

Three patients undergoing transoral robotic surgery (TORS) were prospectively enrolled in this study. Optical imaging of the oropharynx was performed intraoperatively with the robotic-assisted HRME.

RESULTS

All patients underwent the procedure successfully with no complications. The HRME was successfully integrated with the da Vinci robotic system. Several sites of the oropharynx and associated malignancy were imaged, which correlated with the standard histopathological analysis.

CONCLUSION

Transoral robotic-assisted HRME imaging of the oropharynx is a safe and technically feasible approach, providing a real-time histological assessment and may serve as a valuable aid in oncologic surgery.

Stitching and Surface Reconstruction From Endoscopic Image Sequences: A Review of Applications and Methods

Endoscopic procedures form part of routine clinical practice for minimally invasive examinations and interventions. While they are beneficial for the patient, reducing surgical trauma and making convalescence times shorter, they make orientation and manipulation more challenging for the physician, due to the limited field of view through the endoscope. However, this drawback can be reduced by means of medical image processing and computer vision, using image stitching and surface reconstruction methods to expand the field of view. This paper provides a comprehensive overview of the current state of the art in endoscopic image stitching and surface reconstruction. The literature in the relevant fields of application and algorithmic approaches is surveyed. The technological maturity of the methods and current challenges and trends are analyzed.

Applications and advancements in the use of high-resolution microendoscopy for detection of gastrointestinal neoplasia

The high-resolution microendoscope (HRME) is a novel imaging modality that allows real-time epithelial imaging at subcellular resolution. Used in concert with any standard endoscope, this portable, low cost, "optical biopsy" technology has the ability to provide images of cellular morphology during a procedure. This technology has been the subject of a number of studies investigating its use in screening and surveillance of a range of gastrointestinal neoplasias, including esophageal adenocarcinoma, esophageal squamous cell cancer, colorectal neoplasia, and anal neoplasia. These studies have shown that HRME is a modality that consistently provides high specificity, negative predictive value, and accuracy across different diseases. In addition, they have illustrated that HRME users can be relatively easily trained in a short period of time, and that users have demonstrated solid inter-rater reliability. These features make HRME a potential complement to high-definition white-light imaging, narrow-band imaging, and other red flag technologies in facilitating real-time clinical diagnosis, endoscopic therapy, and margin determination. Further clinical validation is needed to determine whether this translates to reduced procedure times, pathology costs, and follow-up procedures. Finally, the HRME has a relatively simple design compared with other similar technologies, making it portable, simple to maintain, and low cost. This may allow the HRME device to function in both advanced care settings as well as in places with less resources and specialized support systems. As a whole, the HRME device has shown good performance along with low cost and portable construction, and its application in different conditions and settings has been promising.

Color reflectance fiber bundle endomicroscopy without back-reflections

Coherent fiber imaging bundles can be used as passive probes for reflectance-mode endomicroscopy providing that the back-reflections from the fiber ends are efficiently rejected. We describe an approach specific to widefield endomicroscopy in which light is injected into a leached fiber bundle near the distal end, thereby avoiding reflections from the proximal face. We use this method to demonstrate the color widefield reflectance endomicroscopy of ex vivo animal tissue.

Automated image mosaics by non-automated light microscopes: the MicroMos software tool

Light widefield microscopes and digital imaging are the basis for most of the analyses performed in every biological laboratory. In particular, the microscope's user is typically interested in acquiring high-detailed images for analysing observed cells and tissues, meanwhile being representative of a wide area to have reliable statistics. The microscopist has to choose between higher magnification factor and extension of the observed area, due to the finite size of the camera's field of view. To overcome the need of arrangement, mosaicing techniques have been developed in the past decades for increasing the camera's field of view by stitching together more images. Nevertheless, these approaches typically work in batch mode and rely on motorized microscopes. Or alternatively, the methods are conceived just to provide visually pleasant mosaics not suitable for quantitative analyses. This work presents a tool for building mosaics of images acquired with nonautomated light microscopes. The method proposed is based on visual information only and the mosaics are built by incrementally stitching couples of images, making the approach available also for online applications. Seams in the stitching regions as well as tonal inhomogeneities are corrected by compensating the vignetting effect. In the experiments performed, we tested different registration approaches, confirming that the translation model is not always the best, despite the fact that the motion of the sample holder of the microscope is apparently translational and typically considered as such. The method's implementation is freely distributed as an open source tool called MicroMos. Its usability makes building mosaics of microscope images at subpixel accuracy easier. Furthermore, optional parameters for building mosaics according to different strategies make MicroMos an easy and reliable tool to compare different registration approaches, warping models and tonal corrections.

Optical molecular imaging in the gastrointestinal tract

Recent developments in optical molecular imaging allow for real-time identification of morphologic and biochemical changes in tissue associated with gastrointestinal neoplasia. This review summarizes widefield and high-resolution imaging modalities in preclinical and clinical evaluation for the detection of colorectal cancer and esophageal cancer. Widefield techniques discussed include high-definition white light endoscopy, narrow band imaging, autofluoresence imaging, and chromoendoscopy; high-resolution techniques discussed include probe-based confocal laser endomicroscopy, high-resolution microendoscopy, and optical coherence tomography. New approaches to enhance image contrast using vital dyes and molecular-specific targeted contrast agents are evaluated.

Confocal microscopy with strip mosaicing for rapid imaging over large areas of excised tissue

Confocal mosaicing microscopy is a developing technology platform for imaging tumor margins directly in freshly excised tissue, without the processing required for conventional pathology. Previously, mosaicing on 12-×-12  mm² of excised skin tissue from Mohs surgery and detection of basal cell carcinoma margins was demonstrated in 9 min. Last year, we reported the feasibility of a faster approach called "strip mosaicing," which was demonstrated on a 10-×-10  mm² of tissue in 3 min. Here we describe further advances in instrumentation, software, and speed. A mechanism was also developed to flatten tissue in order to enable consistent and repeatable acquisition of images over large areas. We demonstrate mosaicing on 10-×-10  mm² of skin tissue with 1-μm lateral resolution in 90 s. A 2.5-×-3.5  cm² piece of breast tissue was scanned with 0.8-μm lateral resolution in 13 min. Rapid mosaicing of confocal images on large areas of fresh tissue potentially offers a means to perform pathology at the bedside. Imaging of tumor margins with strip mosaicing confocal microscopy may serve as an adjunct to conventional (frozen or fixed) pathology for guiding surgery.