In vivo micro-image mosaicing

Robust endoscopic image mosaicking via fusion of multimodal estimation

We propose an endoscopic image mosaicking algorithm that is robust to light conditioning changes, specular reflections, and feature-less scenes. These conditions are especially common in minimally invasive surgery where the light source moves with the camera to dynamically illuminate close range scenes. This makes it difficult for a single image registration method to robustly track camera motion and then generate consistent mosaics of the expanded surgical scene across different and heterogeneous environments. Instead of relying on one specialised feature extractor or image registration method, we propose to fuse different image registration algorithms according to their uncertainties, formulating the problem as affine pose graph optimisation. This allows to combine landmarks, dense intensity registration, and learning-based approaches in a single framework. To demonstrate our application we consider deep learning-based optical flow, hand-crafted features, and intensity-based registration, however, the framework is general and could take as input other sources of motion estimation, including other sensor modalities. We validate the performance of our approach on three datasets with very different characteristics to highlighting its generalisability, demonstrating the advantages of our proposed fusion framework. While each individual registration algorithm eventually fails drastically on certain surgical scenes, the fusion approach flexibly determines which algorithms to use and in which proportion to more robustly obtain consistent mosaics.

Intensity-based nonrigid endomicroscopic image mosaicking incorporating texture relevance for compensation of tissue deformation

Image mosaicking has emerged as a universal technique to broaden the field-of-view of the probe-based confocal laser endomicroscopy (pCLE) imaging system. However, due to the influence of probe-tissue contact forces and optical components on imaging quality, existing mosaicking methods remain insufficient to deal with practical challenges. In this paper, we present the texture encoded sum of conditional variance (TESCV) as a novel similarity metric, and effectively incorporate it into a sequential mosaicking scheme to simultaneously correct rigid probe shift and nonrigid tissue deformation. TESCV combines both intensity dependency and texture relevance to quantify the differences between pCLE image frames, where a discriminative binary descriptor named fully cross-detected local derivative pattern (FCLDP) is designed to extract more detailed structural textures. Furthermore, we also analytically derive the closed-form gradient of TESCV with respect to the transformation variables. Experiments on the circular dataset highlighted the advantage of the TESCV metric in improving mosaicking performance compared with the other four recently published metrics. The comparison with the other four state-of-the-art mosaicking methods on the spiral and manual datasets indicated that the proposed TESCV-based method not only worked stably at different contact forces, but was also suitable for both low- and high-resolution imaging systems. With more accurate and delicate mosaics, the proposed method holds promises to meet clinical demands for intraoperative optical biopsy.

Intensity-Mosaic: automatic panorama mosaicking of disordered images with insufficient features

Purpose: Handling low-quality and few-feature medical images is a challenging task in automatic panorama mosaicking. Current mosaicking methods for disordered input images are based on feature point matching, whereas in this case intensity-based registration achieves better performance than feature-point registration methods. We propose a mosaicking method that enables the use of mutual information (MI) registration for mosaicking randomly ordered input images with insufficient features. Approach: Dimensionality reduction is used to map disordered input images into a low dimensional space. Based on the low dimensional representation, the image global correspondence can be recognized efficiently. For adjacent image pairs, we optimize the MI metric for registration. The panorama is then created after image blending. We demonstrate our method on relatively lower-cost handheld devices that acquire images from the retina in vivo, kidney ex vivo, and bladder phantom, all of which contain sparse features. Results: Our method is compared with three baselines: AutoStitch, "dimension reduction + SIFT," and "MI-Only." Our method compared to the first two feature-point based methods exhibits 1.25 (ex vivo microscope dataset) to two times (in vivo retina dataset) rate of mosaic completion, and MI-Only has the lowest complete rate among three datasets. When comparing the subsequent complete mosaics, our target registration errors can be 2.2 and 3.8 times reduced when using the microscopy and bladder phantom datasets. Conclusions: Using dimensional reduction increases the success rate of detecting adjacent images, which makes MI-based registration feasible and narrows the search range of MI optimization. To the best of our knowledge, this is the first mosaicking method that allows automatic stitching of disordered images with intensity-based alignment, which provides more robust and accurate results when there are insufficient features for classic mosaicking methods.

Real-Time Nonrigid Mosaicking of Laparoscopy Images

The ability to extend the field of view of laparoscopy images can help the surgeons to obtain a better understanding of the anatomical context. However, due to tissue deformation, complex camera motion and significant three-dimensional (3D) anatomical surface, image pixels may have non-rigid deformation and traditional mosaicking methods cannot work robustly for laparoscopy images in real-time. To solve this problem, a novel two-dimensional (2D) non-rigid simultaneous localization and mapping (SLAM) system is proposed in this paper, which is able to compensate for the deformation of pixels and perform image mosaicking in real-time. The key algorithm of this 2D non-rigid SLAM system is the expectation maximization and dual quaternion (EMDQ) algorithm, which can generate smooth and dense deformation field from sparse and noisy image feature matches in real-time. An uncertainty-based loop closing method has been proposed to reduce the accumulative errors. To achieve real-time performance, both CPU and GPU parallel computation technologies are used for dense mosaicking of all pixels. Experimental results on in vivo and synthetic data demonstrate the feasibility and accuracy of our non-rigid mosaicking method.

Robust Mosaicing of Endomicroscopic Videos via Context-Weighted Correlation Ratio

Probe-based confocal laser endomicroscopy (pCLE) is a promising imaging tool that provides in situ and in vivo optical imaging to perform real-time pathological assessments. However, due to limited field of view, it is difficult for clinicians to get a full understanding of the scanned tissues. In this paper, we develop a novel mosaicing framework to assemble all frame sequences into a full view image. First, a hybrid rigid registration that combines feature matching and template matching is presented to achieve a global alignment of all frames. Then, the parametric free-form deformation (FFD) model with a multiresolution architecture is implemented to accommodate non-rigid tissue distortions. More importantly, we devise a robust similarity metric called context-weighted correlation ratio (CWCR) to promote registration accuracy, where spatial and geometric contexts are incorporated into the estimation of functional intensity dependence. Experiments on both robotic setup and manual manipulation have demonstrated that the proposed scheme significantly precedes some state-of-the-art mosaicing schemes in the presence of intensity fluctuations, insufficient overlap and tissue distortions. Moreover, the comparisons of the proposed CWCR metric and two other metrics have validated the effectiveness of the context-weighted strategy in quantifying the differences between two frames. Benefiting from more rational and delicate mosaics, the proposed scheme is more suitable to instruct diagnosis and treatment during optical biopsies.

MOSAICING OF DYNAMIC MESENTERY VIDEO WITH GRADIENT BLENDING

In biomedical imaging using video microscopy, understanding large tissue structures at cellular and finer resolution poses many image acquisition challenges including limited field-of-view and tissue dynamics during imaging. Automated mosaicing or stitching of live tissue video microscopy enables the visualization and analysis of subtle morphological structures and large scale vessel network architecture in tissues like the mesentery. But mosacing can be challenging if there are deformable, motion-blurred, textureless, feature-poor frames. Feature-based methods perform poorly in such cases for the lack of distinctive keypoints. Standard single block correlation matching strategies might not provide robust registration due to deformable content. In addition, the panorama suffers if there is motion blur present in a sequence. To handle these challenges, we propose a novel algorithm, Deformable Normalized Cross Correlation (DNCC) image matching with RANSAC to establish robust registration. Besides, to produce seamless panorama from motion-blurred frames we present gradient blending method based on image edge information. The DNCC algorithm is applied on Frog Mesentery sequences. Our result is compared with PSS/AutoStitch [1, 2] to establish the efficiency and robustness of the proposed DNCC method.

Real-time video mosaicking to guide handheld in vivo microscopy

Handheld and endoscopic optical-sectioning microscopes are being developed for noninvasive screening and intraoperative consultation. Imaging a large extent of tissue is often desired, but miniature in vivo microscopes tend to suffer from limited fields of view. To extend the imaging field during clinical use, we have developed a real-time video mosaicking method, which allows users to efficiently survey larger areas of tissue. Here, we modified a previous post-processing mosaicking method so that real-time mosaicking is possible at >30 frames/second when using a device that outputs images that are 400 × 400 pixels in size. Unlike other real-time mosaicking methods, our strategy can accommodate image rotations and deformations that often occur during clinical use of a handheld microscope. We perform a feasibility study to demonstrate that the use of real-time mosaicking is necessary to enable efficient sampling of a desired imaging field when using a handheld dual-axis confocal microscope.

Image computing for fibre-bundle endomicroscopy: A review

Endomicroscopy is an emerging imaging modality, that facilitates the acquisition of in vivo, in situ optical biopsies, assisting diagnostic and potentially therapeutic interventions. While there is a diverse and constantly expanding range of commercial and experimental optical biopsy platforms available, fibre-bundle endomicroscopy is currently the most widely used platform and is approved for clinical use in a range of clinical indications. Miniaturised, flexible fibre-bundles, guided through the working channel of endoscopes, needles and catheters, enable high-resolution imaging across a variety of organ systems. Yet, the nature of image acquisition though a fibre-bundle gives rise to several inherent characteristics and limitations necessitating novel and effective image pre- and post-processing algorithms, ranging from image formation, enhancement and mosaicing to pathology detection and quantification. This paper introduces the underlying technology and most prevalent clinical applications of fibre-bundle endomicroscopy, and provides a comprehensive, up-to-date, review of relevant image reconstruction, analysis and understanding/inference methodologies. Furthermore, current limitations as well as future challenges and opportunities in fibre-bundle endomicroscopy computing are identified and discussed.

Software-Based Phase Control, Video-Rate Imaging, and Real-Time Mosaicing With a Lissajous-Scanned Confocal Microscope

We present software-based methods for automatic phase control and for mosaicing high-speed, Lissajous-scanned images. To achieve imaging speeds fast enough for mosaicing, we first increase the image update rate tenfold from 3 to 30 Hz, then vertically interpolate each sparse image in real-time to eliminate fixed pattern noise. We validate our methods by imaging fluorescent beads and automatically maintaining phase control over the course of one hour. We then image fixed mouse brain tissues at varying update rates and compare the resulting mosaics. Using reconstructed image data as feedback for phase control eliminates the need for phase sensors and feedback controllers, enabling long-term imaging experiments without additional hardware. Mosaicing subsampled images results in video-rate imaging speeds, nearly fully recovered spatial resolution, and millimeter-scale fields of view.

Automatic motion compensation for structured illumination endomicroscopy using a flexible fiber bundle

SIGNIFICANCE

Confocal laser scanning enables optical sectioning in clinical fiber bundle endomicroscopes, but lower-cost, simplified endomicroscopes use widefield incoherent illumination instead. Optical sectioning can be introduced in these simple systems using structured illumination microscopy (SIM), a multiframe digital subtraction process. However, SIM results in artifacts when the probe is in motion, making the technique difficult to use in vivo and preventing the use of mosaicking to synthesize a larger effective field of view (FOV).

AIM

We report and validate an automatic motion compensation technique to overcome motion artifacts and allow generation of mosaics in SIM endomicroscopy.

APPROACH

Motion compensation is achieved using image registration and real-time pattern orientation correction via a digital micromirror device. We quantify the similarity of moving probe reconstructions to those acquired with a stationary probe using the relative mean of the absolute differences (MAD). We further demonstrate mosaicking with a moving probe in mechanical and freehand operation.

RESULTS

Reconstructed SIM images show an improvement in the MAD from 0.85 to 0.13 for lens paper and from 0.27 to 0.12 for bovine tissue. Mosaics also show vastly reduced artifacts.

CONCLUSION

The reduction in motion artifacts in individual SIM reconstructions leads to mosaics that more faithfully represent the morphology of tissue, giving clinicians a larger effective FOV than the probe itself can provide.

Multichannel correlation improves the noise tolerance of real-time hyperspectral microimage mosaicking

Live-subject microscopies, including microendoscopy and other related technologies, offer promise for basic biology research as well as the optical biopsy of disease in the clinic. However, cellular resolution generally comes with the trade-off of a microscopic field-of-view. Microimage mosaicking enables stitching many small scenes together to aid visualization, quantitative interpretation, and mapping of microscale features, for example, to guide surgical intervention. The development of hyperspectral and multispectral systems for biomedical applications provides motivation for adapting mosaicking algorithms to process a number of simultaneous spectral channels. We present an algorithm that mosaics multichannel video by correlating channels of consecutive frames as a basis for efficiently calculating image alignments. We characterize the noise tolerance of the algorithm by using simulated video with known ground-truth alignments to quantify mosaicking accuracy and speed, showing that multiplexed molecular imaging enhances mosaic accuracy by leveraging observations of distinct molecular constituents to inform frame alignment. A simple mathematical model is introduced to characterize the noise suppression provided by a given group of spectral channels, thus predicting the performance of selected subsets of data channels in order to balance mosaic computation accuracy and speed. The characteristic noise tolerance of a given number of channels is shown to improve through selection of an optimal subset of channels that maximizes this model. We also demonstrate that the multichannel algorithm produces higher quality mosaics than the analogous single-channel methods in an empirical test case. To compensate for the increased data rate of hyperspectral video compared to single-channel systems, we employ parallel processing via GPUs to alleviate computational bottlenecks and to achieve real-time mosaicking even for video-rate multichannel systems anticipated in the future. This implementation paves the way for real-time multichannel mosaicking to accompany next-generation hyperspectral and multispectral video microscopy.

Toward development of a large field-of-view cancer screening patch (CASP) to detect cervical intraepithelial neoplasia

Cervical cancers are primarily diagnosed via colposcopy, in which the tissue is visually assessed by a clinician for abnormalities, followed by directed biopsies and histologic analysis of excised tissue. Optical biopsy technologies offer a less invasive method of imaging such that subcellular features can be resolved without removing tissue. These techniques, however, are limited in field-of-view by the distal end of the probe. We present a prototype that incorporates a rigid, machinable waveguide that is in direct contact with a fluorescently-labeled sample paired with a scanning fluorescent microscope. The system is capable of imaging large areas of tissue without the need to re-position the tissue-probe interface. A mosaicing algorithm was developed to quantify scanning shifts and stitch neighboring frames together to increase the field-of-view. Our prototype can yield a maximum axial resolution of <5 µm for individual frames and can produce mosaiced images with a field-of-view greater than 15 mm x 15 mm without sacrificing resolution. We validated the system with a 1951 USAF resolution target, fluorescent in vitro standards, and a patient study where ex vivo conization samples of squamous cervical epithelium were imaged. The results of the patient study indicate that architectural features of subcellular components could be detected and differentiated between normal tissue and precancerous lesions.

Confidence-based dynamic optimization model for biomedical image mosaicking

Biomedical image mosaicking is a trending topic. It consists of computing a single large image from multiple observations and becomes a challenging task when said observations barely overlap or are subject to illumination changes, poor resolution, blur, or either highly textured or predominantly homogeneous content. Because such challenges are common in biomedical images, classical keypoint/feature-based methods perform poorly. In this paper, we propose a new framework based on pairwise template matching coupled with a constrained, confidence-driven global optimization strategy to solve the issue of microscopic biomedical image mosaicking. First we provide experimental results that show significant improvement on a subjective level. Then we describe a new validation strategy for objective assessment, which shows improvement as well.

Toward Quantitative Neurosurgical Guidance With High-Resolution Microscopy of 5-Aminolevulinic Acid-Induced Protoporphyrin IX

Low-power fluorescence microscopy of 5-ALA-induced PpIX has emerged as a valuable intraoperative imaging technology for improving the resection of malignant gliomas. However, current fluorescence imaging tools are not highly sensitive nor quantitative, which limits their effectiveness for optimizing operative decisions near the surgical margins of gliomas, in particular non-enhancing low-grade gliomas. Intraoperative high-resolution optical-sectioning microscopy can potentially serve as a valuable complement to low-power fluorescence microscopy by providing reproducible quantification of tumor parameters at the infiltrative margins of diffuse gliomas. In this forward-looking perspective article, we provide a brief discussion of recent technical advancements, pilot clinical studies, and our vision of the future adoption of handheld optical-sectioning microscopy at the final stages of glioma surgeries to enhance the extent of resection. We list a number of challenges for clinical acceptance, as well as potential strategies to overcome such obstacles for the surgical implementation of these in vivo microscopy techniques.

Feasibility of a Video-Mosaicking Approach to Extend the Field-of-View For Reflectance Confocal Microscopy in the Oral Cavity In Vivo

BACKGROUND

Reflectance confocal microscopy (RCM) is a developing approach for noninvasive detection of oral lesions with label-free contrast and cellular-level resolution. For access into the oral cavity, confocal microscopes are being configured with small-diameter telescopic probes and small objective lenses. However, a small probe and objective lens allows for a rather small field-of-view relative to the large areas of tissue that must be examined for diagnosis. To extend the field-of-view for intraoral RCM imaging, we are investigating a video-mosaicking approach.

METHODS

A relay telescope and objective lens were adapted to an existing confocal microscope for access into the oral cavity. Imaging was performed using metal three-dimensional-printed objective lens front-end caps with coverslip windows to contact and stabilize the tissue and set depth. Four healthy volunteers (normal oral mucosa), one patient (with an amalgam tattoo) in a clinical setting, and 20 anesthetized patients (with oral squamous cell carcinoma [OSCC]) in a surgical setting were imaged. Instead of the usual still RCM images, videos were recorded and then processed into video-mosaics. Thirty video-mosaics were read and qualitatively assessed by an expert reader of RCM images of the oral mucosa.

RESULTS

Whereas the objective lens' native field-of-view is 0.75 mm × 0.75 mm, the video-mosaics display larger areas, ranging from 2 mm × 2 mm to 4 mm × 2 mm, with resolution, morphologic detail, and image quality that is preserved relative to that observed in the original videos (individual images). Video-mosaics in healthy volunteers' and the patients' images showed cellular morphologic patterns in the lower epithelium and at the epithelial junction, and connective tissue along with capillary loops and blood flow in the deeper lamina propria. In OSCC, tumor nests could be observed along with normal looking mucosa in margin areas.

CONCLUSIONS

Video-mosaicking is a reasonably quick and efficient approach for extending the field-of-view of RCM imaging, which can, to some extent, overcome the inherent limitation of an intraoral probe's small field-of-view. Reading video-mosaics can mimic the procedure for examining pathology: initial visualization of the spatial cellular and morphologic patterns of the tumor and the spread of tumor margins over larger areas of the lesion, followed by digitally zooming (magnifying) for closer inspection of suspicious areas. However, faster processing of videos into video-mosaics will be necessary, to allow examination of video-mosaics in real-time at the bedside. Lasers Surg. Med. 51:439-451, 2019. © 2019 Wiley Periodicals, Inc.

Automated video-mosaicking approach for confocal microscopic imaging in vivo: an approach to address challenges in imaging living tissue and extend field of view

We describe a computer vision-based mosaicking method for in vivo videos of reflectance confocal microscopy (RCM). RCM is a microscopic imaging technique, which enables the users to rapidly examine tissue in vivo. Providing resolution at cellular-level morphology, RCM imaging combined with mosaicking has shown to be highly sensitive and specific for non-invasively guiding skin cancer diagnosis. However, current RCM mosaicking techniques with existing microscopes have been limited to two-dimensional sequences of individual still images, acquired in a highly controlled manner, and along a specific predefined raster path, covering a limited area. The recent advent of smaller handheld microscopes is enabling acquisition of videos, acquired in a relatively uncontrolled manner and along an ad-hoc arbitrarily free-form, non-rastered path. Mosaicking of video-images (video-mosaicking) is necessary to display large areas of tissue. Our video-mosaicking methods addresses this need. The method can handle unique challenges encountered during video capture such as motion blur artifacts due to rapid motion of the microscope over the imaged area, warping in frames due to changes in contact angle and varying resolution with depth. We present test examples of video-mosaics of melanoma and non-melanoma skin cancers, to demonstrate potential clinical utility.

Retinal slit lamp video mosaicking

PURPOSE

To this day, the slit lamp remains the first tool used by an ophthalmologist to examine patient eyes. Imaging of the retina poses, however, a variety of problems, namely a shallow depth of focus, reflections from the optical system, a small field of view and non-uniform illumination. For ophthalmologists, the use of slit lamp images for documentation and analysis purposes, however, remains extremely challenging due to large image artifacts. For this reason, we propose an automatic retinal slit lamp video mosaicking, which enlarges the field of view and reduces amount of noise and reflections, thus enhancing image quality.

METHODS

Our method is composed of three parts: (i) viable content segmentation, (ii) global registration and (iii) image blending. Frame content is segmented using gradient boosting with custom pixel-wise features. Speeded-up robust features are used for finding pair-wise translations between frames with robust random sample consensus estimation and graph-based simultaneous localization and mapping for global bundle adjustment. Foreground-aware blending based on feathering merges video frames into comprehensive mosaics.

RESULTS

Foreground is segmented successfully with an area under the curve of the receiver operating characteristic curve of 0.9557. Mosaicking results and state-of-the-art methods were compared and rated by ophthalmologists showing a strong preference for a large field of view provided by our method.

CONCLUSIONS

The proposed method for global registration of retinal slit lamp images of the retina into comprehensive mosaics improves over state-of-the-art methods and is preferred qualitatively.

Application of robotics in gastrointestinal endoscopy: A review

Multiple robotic flexible endoscope platforms have been developed based on cross specialty collaboration between engineers and medical doctors. However, significant number of these platforms have been developed for the natural orifice transluminal endoscopic surgery paradigm. Increasing amount of evidence suggest the focus of development should be placed on advanced endolumenal procedures such as endoscopic submucosal dissection instead. A thorough literature analysis was performed to assess the current status of robotic flexible endoscopic platforms designed for advanced endolumenal procedures. Current efforts are mainly focused on robotic locomotion and robotic instrument control. In the future, advances in actuation and servoing technology, optical analysis, augmented reality and wireless power transmission technology will no doubt further advance the field of robotic endoscopy. Globally, health systems have become increasingly budget conscious; widespread acceptance of robotic endoscopy will depend on careful design to ensure its delivery of a cost effective service.

Miniature in vivo MEMS-based line-scanned dual-axis confocal microscope for point-of-care pathology

There is a need for miniature optical-sectioning microscopes to enable in vivo interrogation of tissues as a real-time and noninvasive alternative to gold-standard histopathology. Such devices could have a transformative impact for the early detection of cancer as well as for guiding tumor-resection procedures. Miniature confocal microscopes have been developed by various researchers and corporations to enable optical sectioning of highly scattering tissues, all of which have necessitated various trade-offs in size, speed, depth selectivity, field of view, resolution, image contrast, and sensitivity. In this study, a miniature line-scanned (LS) dual-axis confocal (DAC) microscope, with a 12-mm diameter distal tip, has been developed for clinical point-of-care pathology. The dual-axis architecture has demonstrated an advantage over the conventional single-axis confocal configuration for reducing background noise from out-of-focus and multiply scattered light. The use of line scanning enables fast frame rates (16 frames/sec is demonstrated here, but faster rates are possible), which mitigates motion artifacts of a hand-held device during clinical use. We have developed a method to actively align the illumination and collection beams in a DAC microscope through the use of a pair of rotatable alignment mirrors. Incorporation of a custom objective lens, with a small form factor for in vivo clinical use, enables our device to achieve an optical-sectioning thickness and lateral resolution of 2.0 and 1.1 microns respectively. Validation measurements with reflective targets, as well as in vivo and ex vivo images of tissues, demonstrate the clinical potential of this high-speed optical-sectioning microscopy device.

Optical flow with structure information for epithelial image mosaicing

Mosaicing of biological tissue surfaces is challenging due to the weak image textures. This contribution presents a mosaicing algorithm based on a robust and accurate variational optical flow scheme. A Riesz pyramid based multiscale approach aims at overcoming the "flattening-out" problem at coarser levels. Moreover, the structure information present in images of epithelial surfaces is incorporated into the data-term to improve the algorithm robustness. The algorithm accuracy is first assessed with simulated sequences and then used for mosaicing standard clinical endoscopic data.

Global and Local Panoramic Views for Gastroscopy: An Assisted Method of Gastroscopic Lesion Surveillance

Gastroscopy plays an important role in the diagnosis of gastric disease. In this paper, we develop an image panoramic system to assist endoscopists in improving lesion surveillance and reducing many of the tedious operations associated with gastroscopy. The constructed panoramic view has two categories: 1) the local view broadens the endoscopist's field of view in real time. Combining with the original gastroscopic video, this mosaicking view enables the endoscopist to diagnose the lesion comprehensively; 2) the global view constructs a large-area panoramic scene of the internal gastric surface, which can be used for intraoperative surgical navigation and postoperative scene review. Due to the irregular texture and inconsistent reflection of the gastric internal surface, common registration methods cannot accurately stitch this surface. Thereby, a six degree of freedom position tracking endoscope is devised to accommodate for the accumulated mosaicking error and provide efficient mosaicking results. For the global view, a dual-cube constraint model and a Bundle Adjustment algorithm are incorporated to deal with the mosaicking error caused by the irregular inflation and nonrigid deformation of the stomach. Moreover, texture blending and frame selection schemes are developed to make the mosaicking results feasible in real-clinical applications. The experimental results demonstrate that our system performs with a speed of 7.12 frames/s in a standard computer environment, and the mosaicking mean error is 0.43 mm for local panoramic view and 3.71 mm for global panoramic view.

Stitching and Surface Reconstruction From Endoscopic Image Sequences: A Review of Applications and Methods

Endoscopic procedures form part of routine clinical practice for minimally invasive examinations and interventions. While they are beneficial for the patient, reducing surgical trauma and making convalescence times shorter, they make orientation and manipulation more challenging for the physician, due to the limited field of view through the endoscope. However, this drawback can be reduced by means of medical image processing and computer vision, using image stitching and surface reconstruction methods to expand the field of view. This paper provides a comprehensive overview of the current state of the art in endoscopic image stitching and surface reconstruction. The literature in the relevant fields of application and algorithmic approaches is surveyed. The technological maturity of the methods and current challenges and trends are analyzed.

Maintenance and neuronal cell differentiation of neural stem cells C17.2 correlated to medium availability sets design criteria in microfluidic systems

BACKGROUND

Neural stem cells (NSCs) play an important role in developing potential cell-based therapeutics for neurodegenerative disease. Microfluidics has proven a powerful tool in mechanistic studies of NSC differentiation. However, NSCs are prone to differentiate when the nutrients are limited, which occurs unfavorable by fast medium consumption in miniaturized culture environment. For mechanistic studies of NSCs in microfluidics, it is vital that neuronal cell differentiation is triggered by controlled factors only. Thus, we studied the correlation between available cell medium and spontaneous neuronal cell differentiation of C17.2 NSCs in standard culture medium, and proposed the necessary microfluidic design criteria to prevent undesirable cell phenotype changes.

METHODOLOGY/PRINCIPAL FINDINGS

A series of microchannels with specific geometric parameters were designed to provide different amount of medium to the cells over time. A medium factor (MF, defined as the volume of stem cell culture medium divided by total number of cells at seeding and number of hours between medium replacement) successfully correlated the amount of medium available to each cell averaged over time to neuronal cell differentiation. MF smaller than 8.3×10(4) µm3/cell⋅hour produced significant neuronal cell differentiation marked by cell morphological change and significantly more cells with positive β-tubulin-III and MAP2 staining than the control. When MF was equal or greater than 8.3×10(4) µm3/cell⋅hour, minimal spontaneous neuronal cell differentiation happened relative to the control. MF had minimal relation with the average neurite length.

SIGNIFICANCE

MFs can be controlled easily to maintain the stem cell status of C17.2 NSCs or to induce spontaneous neuronal cell differentiation in standard stem cell culture medium. This finding is useful in designing microfluidic culture platforms for controllable NSC maintenance and differentiation. This study also offers insight about consumption rate of serum molecules involved in maintaining the stemness of NSCs.

Comparison of two algorithms for the application of real-time image mosaicking in neuroendoscopy

OBJECT

Neuroendoscopy is used more and more frequently in neurosurgical procedures and has become an important tool in the neurosurgical armamentarium. However, the main restriction of neuroendoscopy is the limited field of view. A better overview of the area of interest would increase surgical safety and decrease procedure-related morbidity rates. In the present study, the authors aimed to improve this restriction by using and comparing two algorithms to create endoscopic panoramic images, which increase the field of view during neuroendoscopic procedures.

METHODS

Different endoscopic methods with or without a stand and with linear or circular endoscope movements were performed in cadaveric ventricles. Video of the endoscopy was used to create image mosaics of the lateral ventricle with the help of the Kourogi or LogSearch (LS) algorithm. In the LS algorithm, different template sizes were used. Three observers graded the quality of the image mosaic in terms of usefulness in surgery. The fastest frame rate was 3-4 frames/second.

RESULTS

The LS algorithm with a larger template size showed significantly better results for the creation of image mosaics than the Kourogi algorithm in linear endoscopic movement with or without a stand. In circular endoscopic movements, the results seemed to be better with the LS algorithm but were not significantly different from those obtained with the Kourogi algorithm. In summary, image quality in the experimental paradigms was satisfying.

CONCLUSIONS

Results in the study showed that the creation of image mosaics is possible and reliable with the featured algorithms. Image mosaicking is an applicable device for neuroendoscopy and can increase the field of view during endoscopic procedures. Its use can increase the safety and the field of application of neuroendoscopy. However, faster frame rates will be required to create a smooth image for practical use during surgery.

Advances in imaging probes and optical microendoscopic imaging techniques for early in vivo cancer assessment

A new chapter in the history of medical diagnosis happened when the first X-ray technology was invented in the late 1800s. Since then, many non-invasive and minimally invasive imaging techniques have been invented for clinical diagnosis to research in cellular biology, drug discovery, and disease monitoring. These imaging modalities have leveraged the benefits of significant advances in computer, electronics, and information technology and, more recently, targeted molecular imaging. The development of targeted contrast agents such as fluorescent and nanoparticle probes coupled with optical imaging techniques has made it possible to selectively view specific biological events and processes in both in vivo and ex vivo systems with great sensitivity and selectivity. Thus, the combination of targeted molecular imaging probes and optical imaging techniques have become a mainstay in modern medicinal and biological research. Many promising results have demonstrated great potentials to translate to clinical applications. In this review, we describe a discussion of employing imaging probes and optical microendoscopic imaging techniques for cancer diagnosis.

Fundus image mosaicking for information augmentation in computer-assisted slit-lamp imaging

Laser photocoagulation is currently the standard treatment for sight-threatening diseases worldwide, namely diabetic retinopathy and retinal vein occlusions. The slit lamp biomicroscope is the most commonly used device for this procedure, specially for the treatment of the eye periphery. However, only a small portion of the retina can be visualized through the biomicroscope, complicating the task of localizing and identifying surgical targets, increasing treatment duration and patient discomfort. In order to assist surgeons, we propose a method for creating intraoperative retina maps for view expansion using a slit-lamp device. Based on the mosaicking method described by Richa et al, 2012, the proposed method is a combination of direct and feature-based methods, suitable for the textured nature of the human retina. In this paper, we describe three major enhancements to the original formulation. The first is a visual tracking method using local illumination compensation to cope with the challenging visualization conditions. The second is an efficient pixel selection scheme for increased computational efficiency. The third is an entropy-based mosaic update method to dynamically improve the retina map during exploration. To evaluate the performance of the proposed method, we conducted several experiments on human subjects with a computer-assisted slit-lamp prototype. We also demonstrate the practical value of the system for photo documentation, diagnosis and intraoperative navigation.

Automated image mosaics by non-automated light microscopes: the MicroMos software tool

Light widefield microscopes and digital imaging are the basis for most of the analyses performed in every biological laboratory. In particular, the microscope's user is typically interested in acquiring high-detailed images for analysing observed cells and tissues, meanwhile being representative of a wide area to have reliable statistics. The microscopist has to choose between higher magnification factor and extension of the observed area, due to the finite size of the camera's field of view. To overcome the need of arrangement, mosaicing techniques have been developed in the past decades for increasing the camera's field of view by stitching together more images. Nevertheless, these approaches typically work in batch mode and rely on motorized microscopes. Or alternatively, the methods are conceived just to provide visually pleasant mosaics not suitable for quantitative analyses. This work presents a tool for building mosaics of images acquired with nonautomated light microscopes. The method proposed is based on visual information only and the mosaics are built by incrementally stitching couples of images, making the approach available also for online applications. Seams in the stitching regions as well as tonal inhomogeneities are corrected by compensating the vignetting effect. In the experiments performed, we tested different registration approaches, confirming that the translation model is not always the best, despite the fact that the motion of the sample holder of the microscope is apparently translational and typically considered as such. The method's implementation is freely distributed as an open source tool called MicroMos. Its usability makes building mosaics of microscope images at subpixel accuracy easier. Furthermore, optional parameters for building mosaics according to different strategies make MicroMos an easy and reliable tool to compare different registration approaches, warping models and tonal corrections.

Structure propagation for image registration

Mosaicing is a commonly used technique in many medical imaging applications where subimages are stitched together in order to obtain a larger field of view. However, stitching, which involves alignment or registration in overlapping regions, is often challenging when the information shared by subimages is absent or small. While it is not possible to perform an alignment without overlap using existing techniques, imaging artifacts such as distortions towards image boundaries present further complications during registration by decreasing the reliability of available information. Without taking these into consideration, a registration approach might violate the continuity and the smoothness of structures across subimages. In this paper, we propose a novel registration approach for the stitching of subimages in such challenging scenarios. By using a perceptual grouping approach, we extend subimages beyond their boundaries by propagating available structures in order to obtain structural maps in the extended regions. These maps are then used to establish correspondences between subimages when the shared information is absent, small or unreliable. Using our approach ensures the continuity and the smoothness of structures across subimage boundaries. Furthermore, since only structures are used, the proposed method can also be used for the stitching of multi-modal images. Our approach is unique in that it also enables contactless stitching. We demonstrate the effectiveness of the proposed method by performing several experiments on synthetic and medical images. Moreover, we show how stitching is possible in the presence of a physical gap between subimages.

Confocal microscopy with strip mosaicing for rapid imaging over large areas of excised tissue

Confocal mosaicing microscopy is a developing technology platform for imaging tumor margins directly in freshly excised tissue, without the processing required for conventional pathology. Previously, mosaicing on 12-×-12  mm² of excised skin tissue from Mohs surgery and detection of basal cell carcinoma margins was demonstrated in 9 min. Last year, we reported the feasibility of a faster approach called "strip mosaicing," which was demonstrated on a 10-×-10  mm² of tissue in 3 min. Here we describe further advances in instrumentation, software, and speed. A mechanism was also developed to flatten tissue in order to enable consistent and repeatable acquisition of images over large areas. We demonstrate mosaicing on 10-×-10  mm² of skin tissue with 1-μm lateral resolution in 90 s. A 2.5-×-3.5  cm² piece of breast tissue was scanned with 0.8-μm lateral resolution in 13 min. Rapid mosaicing of confocal images on large areas of fresh tissue potentially offers a means to perform pathology at the bedside. Imaging of tumor margins with strip mosaicing confocal microscopy may serve as an adjunct to conventional (frozen or fixed) pathology for guiding surgery.

Understanding soft-tissue behavior for application to microlaparoscopic surface scan

This paper presents an approach for understanding the soft-tissue behavior in surface contact with a probe scanning the tissue. The application domain is confocal microlaparoscopy, mostly used for imaging the outer surface of the organs in the abdominal cavity. The probe is swept over the tissue to collect sequential images to obtain a large field of view with mosaicking. The problem we address is that the tissue also moves with the probe due to its softness; therefore, the resulting mosaic is not in the same shape and dimension as traversed by the probe. Our approach is inspired by the finger slip studies and adapts the idea of load-slip phenomenon that explains the movement of the soft part of the finger when dragged on a hard surface. We propose the concept of loading-distance and perform measurements on beef liver and chicken breast tissues. We propose a protocol to determine the loading-distance prior to an automated scan and introduce an approach to compensate the tissue movement in raster scans. Our implementation and experiments show that we can have an image mosaic of the tissue surface in a desired rectangular shape with this approach.

Building large mosaics of confocal edomicroscopic images using visual servoing

Probe-based confocal laser endomicroscopy provides real-time microscopic images of tissues contacted by a small probe that can be inserted in vivo through a minimally invasive access. Mosaicking consists in sweeping the probe in contact with a tissue to be imaged while collecting the video stream, and process the images to assemble them in a large mosaic. While most of the literature in this field has focused on image processing, little attention has been paid so far to the way the probe motion can be controlled. This is a crucial issue since the precision of the probe trajectory control drastically influences the quality of the final mosaic. Robotically controlled motion has the potential of providing enough precision to perform mosaicking. In this paper, we emphasize the difficulties of implementing such an approach. First, probe-tissue contacts generate deformations that prevent from properly controlling the image trajectory. Second, in the context of minimally invasive procedures targeted by our research, robotic devices are likely to exhibit limited quality of the distal probe motion control at the microscopic scale. To cope with these problems visual servoing from real-time endomicroscopic images is proposed in this paper. It is implemented on two different devices (a high-accuracy industrial robot and a prototype minimally invasive device). Experiments on different kinds of environments (printed paper and ex vivo tissues) show that the quality of the visually servoed probe motion is sufficient to build mosaics with minimal distortion in spite of disturbances.

Real-time video mosaicing with a high-resolution microendoscope

Microendoscopes allow clinicians to view subcellular features in vivo and in real-time, but their field-of-view is inherently limited by the small size of the probe's distal end. Video mosaicing has emerged as an effective technique to increase the acquired image size. Current implementations are performed post-procedure, which removes the benefits of live imaging. In this manuscript we present an algorithm for real-time video mosaicing using a low-cost high-resolution microendoscope. We present algorithm execution times and show image results obtained from in vivo tissue.

In vivo near-infrared dual-axis confocal microendoscopy in the human lower gastrointestinal tract

Near-infrared confocal microendoscopy is a promising technique for deep in vivo imaging of tissues and can generate high-resolution cross-sectional images at the micron-scale. We demonstrate the use of a dual-axis confocal (DAC) near-infrared fluorescence microendoscope with a 5.5-mm outer diameter for obtaining clinical images of human colorectal mucosa. High-speed two-dimensional en face scanning was achieved through a microelectromechanical systems (MEMS) scanner while a micromotor was used for adjusting the axial focus. In vivo images of human patients are collected at 5 frames/sec with a field of view of 362×212 μm(2) and a maximum imaging depth of 140 μm. During routine endoscopy, indocyanine green (ICG) was topically applied a nonspecific optical contrasting agent to regions of the human colon. The DAC microendoscope was then used to obtain microanatomic images of the mucosa by detecting near-infrared fluorescence from ICG. These results suggest that DAC microendoscopy may have utility for visualizing the anatomical and, perhaps, functional changes associated with colorectal pathology for the early detection of colorectal cancer.

Surgical vision

The emergence of Minimal Access Surgery (MAS) as a paradigm in modern healthcare treatment has created new challenges and opportunities for automated image understanding and computer vision. In MAS, images recovered from inside the body using specialized devices are used to visualize and operate on the surgical site but they can also be used to computationally infer in vivo 3D tissue surface shape, soft-tissue morphology, and surgical instrument motion. This information is important for facilitating in vivo biophotonic imaging modalities where the interaction between light and tissue is used to infer the structural and functional properties of the tissue. This article provides a review of the literature for computer vision and image understanding techniques applied to MAS images. The focus of this article is to elucidate a perspective on how computer vision techniques can be used to support and enhance the capabilities of biophotonic imaging modalities during surgery. Note that while MAS encompasses a variety of surgical specializations this review does not involve procedures performed in the interventional suite. The review has been carried out based on searches in the PubMed and IEEE databases using the article's keywords.