Selective image similarity measure for bronchoscope tracking based on image registration

Landmark Based Bronchoscope Localization for Needle Insertion Under Respiratory Deformation

Bronchoscopy is currently the least invasive method for definitively diagnosing lung cancer, which kills more people in the United States than any other form of cancer. Successfully diagnosing suspicious lung nodules requires accurate localization of the bronchoscope relative to a planned biopsy site in the airways. This task is challenging because the lung deforms intraoperatively due to respiratory motion, the airways lack photometric features, and the anatomy's appearance is repetitive. In this paper, we introduce a real-time camera-based method for accurately localizing a bronchoscope with respect to a planned needle insertion pose. Our approach uses deep learning and accounts for deformations and overcomes limitations of global pose estimation by estimating pose relative to anatomical landmarks. Specifically, our learned model considers airway bifurcations along the airway wall as landmarks because they are distinct geometric features that do not vary significantly with respiratory motion. We evaluate our method in a simulated dataset of lungs undergoing respiratory motion. The results show that our method generalizes across patients and localizes the bronchoscope with accuracy sufficient to access the smallest clinically-relevant nodules across all levels of respiratory deformation, even in challenging distal airways. Our method could enable physicians to perform more accurate biopsies and serve as a key building block toward accurate autonomous robotic bronchoscopy.

Visually Navigated Bronchoscopy using three cycle-Consistent generative adversarial network for depth estimation

[Background] Electromagnetically Navigated Bronchoscopy (ENB) is currently the state-of-the art diagnostic and interventional bronchoscopy. CT-to-body divergence is a critical hurdle in ENB, causing navigation error and ultimately limiting the clinical efficacy of diagnosis and treatment. In this study, Visually Navigated Bronchoscopy (VNB) is proposed to address the aforementioned issue of CT-to-body divergence. [Materials and Methods] We extended and validated an unsupervised learning method to generate a depth map directly from bronchoscopic images using a Three Cycle-Consistent Generative Adversarial Network (3cGAN) and registering the depth map to preprocedural CTs. We tested the working hypothesis that the proposed VNB can be integrated to the navigated bronchoscopic system based on 3D Slicer, and accurately register bronchoscopic images to pre-procedural CTs to navigate transbronchial biopsies. The quantitative metrics to asses the hypothesis we set was Absolute Tracking Error (ATE) of the tracking and the Target Registration Error (TRE) of the total navigation system. We validated our method on phantoms produced from the pre-procedural CTs of five patients who underwent ENB and on two ex-vivo pig lung specimens. [Results] The ATE using 3cGAN was 6.2 +/- 2.9 [mm]. The ATE of 3cGAN was statistically significantly lower than that of cGAN, particularly in the trachea and lobar bronchus (p < 0.001). The TRE of the proposed method had a range of 11.7 to 40.5 [mm]. The TRE computed by 3cGAN was statistically significantly smaller than those computed by cGAN in two of the five cases enrolled (p < 0.05). [Conclusion] VNB, using 3cGAN to generate the depth maps was technically and clinically feasible. While the accuracy of tracking by cGAN was acceptable, the TRE warrants further investigation and improvement.

Depth-based branching level estimation for bronchoscopic navigation

PURPOSE

Bronchoscopists rely on navigation systems during bronchoscopy to reduce the risk of getting lost in the complex bronchial tree-like structure and the homogeneous bronchus lumens. We propose a patient-specific branching level estimation method for bronchoscopic navigation because it is vital to identify the branches being examined in the bronchus tree during examination.

METHODS

We estimate the branching level by integrating the changes in the number of bronchial orifices and the camera motions among the frames. We extract the bronchial orifice regions from a depth image, which is generated using a cycle generative adversarial network (CycleGAN) from real bronchoscopic images. We calculate the number of orifice regions using the vertical and horizontal projection profiles of the depth images and obtain the camera-moving direction using the feature point-based camera motion estimation. The changes in the number of bronchial orifices are combined with the camera-moving direction to estimate the branching level.

RESULTS

We used three in vivo and one phantom case to train the CycleGAN model and four in vivo cases to validate the proposed method. We manually created the ground truth of the branching level. The experimental results showed that the proposed method can estimate the branching level with an average accuracy of 87.6%. The processing time per frame was about 61 ms.

CONCLUSION

Experimental results show that it is feasible to estimate the branching level using the number of bronchial orifices and camera-motion estimation from real bronchoscopic images.

A Bronchoscope Localization Method Using an Augmented Reality Co-Display of Real Bronchoscopy Images with a Virtual 3D Bronchial Tree Model

In recent years, Image-Guide Navigation Systems (IGNS) have become an important tool for various surgical operations. In the preparations for planning a surgical path, verifying the location of a lesion, etc., it is an essential tool; in operations such as bronchoscopy, which is the procedure for the inspection and retrieval of diagnostic samples for lung-related surgeries, it is even more so. The IGNS for bronchoscopy uses 2D-based images from a flexible bronchoscope to navigate through the bronchial airways in order to reach the targeted location. In this procedure, the accurate localization of the scope becomes very important, because incorrect information could potentially cause a surgeon to mistakenly direct the scope down the wrong passage. It would be a great aid for the surgeon to be able to visualize the bronchoscope images alongside the current location of the bronchoscope. For this purpose, in this paper, we propose a novel registration method to match real bronchoscopy images with virtual bronchoscope images from a 3D bronchial tree model built using computed tomography (CT) image stacks in order to obtain the current 3D position of the bronchoscope in the airways. This method is a combination of a novel position-tracking method using the current frames from the bronchoscope and the verification of the position of the real bronchoscope image against an image extracted from the 3D model using an adaptive-network-based fuzzy inference system (ANFIS)-based image matching method. Experimental results show that the proposed method performs better than the other methods used in the comparison.

Cost-function testing methodology for image-based registration of endoscopy to CT images in the head and neck

One of the largest geometric uncertainties in designing radiotherapy treatment plans for squamous cell cancers of the head and neck is contouring the gross tumour volume. We have previously described a method of projecting mucosal disease contours, visible on endoscopy, to volumetrically reconstructed planning CT datasets, using electromagnetic (EM) tracking of a flexible endoscope, enabling rigid registration between endoscopic and CT images. However, to achieve better accuracy for radiotherapy planning, we propose refining this initial registration with image-based registration methods. In this paper, several types of cost functions are evaluated based on accuracy and robustness. Three phantoms and eight clinical cases are used to test each cost function, with initial registration of endoscopy to CT provided by the pose of the flexible endoscope recovered from EM tracking. Cost function classes include: cross correlation, mutual information and gradient methods. For each test case, a ground truth virtual camera pose was first defined by manual registration of anatomical features visible in both real and virtual endoscope images. A new set of evenly spaced fiducial points and a sample contour were created and projected onto the CT image to be used in assessing image registration quality. A new set of 5000 displaced poses was generated by random sampling displacements along each translational and rotational dimension. At each pose, fiducial and contour points in the real image were again projected on the CT image. The cost function, fiducial registration error and contouring error values were then calculated. While all cost functions performed well in select cases, only the normalized gradient field function consistently had registration errors less than 2 mm, which is the accuracy needed if this application of registering mucosal disease identified on optical image to CT images is to be used in the clinical practice of radiation treatment planning. (Registration: ClinicalTrials.gov NCT02704169).

Advanced Endoscopic Navigation: Surgical Big Data, Methodology, and Applications

Interventional endoscopy (e.g., bronchoscopy, colonoscopy, laparoscopy, cystoscopy) is a widely performed procedure that involves either diagnosis of suspicious lesions or guidance for minimally invasive surgery in a variety of organs within the body cavity. Endoscopy may also be used to guide the introduction of certain items (e.g., stents) into the body. Endoscopic navigation systems seek to integrate big data with multimodal information (e.g., computed tomography, magnetic resonance images, endoscopic video sequences, ultrasound images, external trackers) relative to the patient's anatomy, control the movement of medical endoscopes and surgical tools, and guide the surgeon's actions during endoscopic interventions. Nevertheless, it remains challenging to realize the next generation of context-aware navigated endoscopy. This review presents a broad survey of various aspects of endoscopic navigation, particularly with respect to the development of endoscopic navigation techniques. First, we investigate big data with multimodal information involved in endoscopic navigation. Next, we focus on numerous methodologies used for endoscopic navigation. We then review different endoscopic procedures in clinical applications. Finally, we discuss novel techniques and promising directions for the development of endoscopic navigation.

Robust colonoscope tracking method for colon deformations utilizing coarse-to-fine correspondence findings

PURPOSE

Polyps found during CT colonography can be removed by colonoscopic polypectomy. A colonoscope navigation system that navigates a physician to polyp positions while performing the colonoscopic polypectomy is required. Colonoscope tracking methods are essential for implementing colonoscope navigation systems. Previous colonoscope tracking methods have failed when the colon deforms during colonoscope insertions. This paper proposes a colonoscope tracking method that is robust against colon deformations.

METHOD

The proposed method generates a colon centerline from a CT volume and a curved line representing the colonoscope shape (colonoscope line) by using electromagnetic sensors. We find correspondences between points on a deformed colon centerline and colonoscope line by a landmark-based coarse correspondence finding and a length-based fine correspondence finding processes. Even if the coarse correspondence finding process fails to find some correspondences, which occurs with colon deformations, the fine correspondence finding process is able to find correct correspondences by using previously recorded line lengths.

RESULT

Experimental results using a colon phantom showed that the proposed method finds the colonoscope tip position with tracking errors smaller than 50 mm in most trials. A physician who specializes in gastroenterology commented that tracking errors smaller than 50 mm are acceptable. This is because polyps are observable from the colonoscope camera when positions of the colonoscope tip and polyps are closer than 50 mm.

CONCLUSIONS

We developed a colonoscope tracking method that is robust against deformations of the colon. Because the process was designed to consider colon deformations, the proposed method can track the colonoscope tip position even if the colon deforms.

Video see-through augmented reality for oral and maxillofacial surgery

BACKGROUND

Oral and maxillofacial surgery has not been benefitting from image guidance techniques owing to the limitations in image registration.

METHODS

A real-time markerless image registration method is proposed by integrating a shape matching method into a 2D tracking framework. The image registration is performed by matching the patient's teeth model with intraoperative video to obtain its pose. The resulting pose is used to overlay relevant models from the same CT space on the camera video for augmented reality.

RESULTS

The proposed system was evaluated on mandible/maxilla phantoms, a volunteer and clinical data. Experimental results show that the target overlay error is about 1 mm, and the frame rate of registration update yields 3-5 frames per second with a 4 K camera.

CONCLUSIONS

The significance of this work lies in its simplicity in clinical setting and the seamless integration into the current medical procedure with satisfactory response time and overlay accuracy. Copyright © 2016 John Wiley & Sons, Ltd.

Reconstruction of Banknote Fragments Based on Keypoint Matching Method

Banknotes may be shredded by a scrap machine, ripped up by hand, or damaged in accidents. This study proposes an image registration method for reconstruction of multiple sheets of banknotes. The proposed method first constructs different scale spaces to identify keypoints in the underlying banknote fragments. Next, the features of those keypoints are extracted to represent their local patterns around keypoints. Then, similarity is computed to find the keypoint pairs between the fragment and the reference banknote. The banknote fragments can determine the coordinate and amend the orientation. Finally, an assembly strategy is proposed to piece multiple sheets of banknote fragments together. Experimental results show that the proposed method causes, on average, a deviation of 0.12457 ± 0.12810° for each fragment while the SIFT method deviates 1.16893 ± 2.35254° on average. The proposed method not only reconstructs the banknotes but also decreases the computing cost. Furthermore, the proposed method can estimate relatively precisely the orientation of the banknote fragments to assemble.

Robust electromagnetically guided endoscopic procedure using enhanced particle swarm optimization for multimodal information fusion

PURPOSE

Electromagnetically guided endoscopic procedure, which aims at accurately and robustly localizing the endoscope, involves multimodal sensory information during interventions. However, it still remains challenging in how to integrate these information for precise and stable endoscopic guidance. To tackle such a challenge, this paper proposes a new framework on the basis of an enhanced particle swarm optimization method to effectively fuse these information for accurate and continuous endoscope localization.

METHODS

The authors use the particle swarm optimization method, which is one of stochastic evolutionary computation algorithms, to effectively fuse the multimodal information including preoperative information (i.e., computed tomography images) as a frame of reference, endoscopic camera videos, and positional sensor measurements (i.e., electromagnetic sensor outputs). Since the evolutionary computation method usually limits its possible premature convergence and evolutionary factors, the authors introduce the current (endoscopic camera and electromagnetic sensor's) observation to boost the particle swarm optimization and also adaptively update evolutionary parameters in accordance with spatial constraints and the current observation, resulting in advantageous performance in the enhanced algorithm.

RESULTS

The experimental results demonstrate that the authors' proposed method provides a more accurate and robust endoscopic guidance framework than state-of-the-art methods. The average guidance accuracy of the authors' framework was about 3.0 mm and 5.6° while the previous methods show at least 3.9 mm and 7.0°. The average position and orientation smoothness of their method was 1.0 mm and 1.6°, which is significantly better than the other methods at least with (2.0 mm and 2.6°). Additionally, the average visual quality of the endoscopic guidance was improved to 0.29.

CONCLUSIONS

A robust electromagnetically guided endoscopy framework was proposed on the basis of an enhanced particle swarm optimization method with using the current observation information and adaptive evolutionary factors. The authors proposed framework greatly reduced the guidance errors from (4.3, 7.8) to (3.0 mm, 5.6°), compared to state-of-the-art methods.

A discriminative structural similarity measure and its application to video-volume registration for endoscope three-dimensional motion tracking

Endoscope 3-D motion tracking, which seeks to synchronize pre- and intra-operative images in endoscopic interventions, is usually performed as video-volume registration that optimizes the similarity between endoscopic video and pre-operative images. The tracking performance, in turn, depends significantly on whether a similarity measure can successfully characterize the difference between video sequences and volume rendering images driven by pre-operative images. The paper proposes a discriminative structural similarity measure, which uses the degradation of structural information and takes image correlation or structure, luminance, and contrast into consideration, to boost video-volume registration. By applying the proposed similarity measure to endoscope tracking, it was demonstrated to be more accurate and robust than several available similarity measures, e.g., local normalized cross correlation, normalized mutual information, modified mean square error, or normalized sum squared difference. Based on clinical data evaluation, the tracking error was reduced significantly from at least 14.6 mm to 4.5 mm. The processing time was accelerated more than 30 frames per second using graphics processing unit.

Optimal procedure planning and guidance system for peripheral bronchoscopy

With the development of multidetector computed-tomography (MDCT) scanners and ultrathin bronchoscopes, the use of bronchoscopy for diagnosing peripheral lung-cancer nodules is becoming a viable option. The work flow for assessing lung cancer consists of two phases: 1) 3-D MDCT analysis and 2) live bronchoscopy. Unfortunately, the yield rates for peripheral bronchoscopy have been reported to be as low as 14%, and bronchoscopy performance varies considerably between physicians. Recently, proposed image-guided systems have shown promise for assisting with peripheral bronchoscopy. Yet, MDCT-based route planning to target sites has relied on tedious error-prone techniques. In addition, route planning tends not to incorporate known anatomical, device, and procedural constraints that impact a feasible route. Finally, existing systems do not effectively integrate MDCT-derived route information into the live guidance process. We propose a system that incorporates an automatic optimal route-planning method, which integrates known route constraints. Furthermore, our system offers a natural translation of the MDCT-based route plan into the live guidance strategy via MDCT/video data fusion. An image-based study demonstrates the route-planning method's functionality. Next, we present a prospective lung-cancer patient study in which our system achieved a successful navigation rate of 91% to target sites. Furthermore, when compared to a competing commercial system, our system enabled bronchoscopy over two airways deeper into the airway-tree periphery with a sample time that was nearly 2 min shorter on average. Finally, our system's ability to almost perfectly predict the depth of a bronchoscope's navigable route in advance represents a substantial benefit of optimal route planning.

Image partitioning and illumination in image-based pose detection for teleoperated flexible endoscopes

OBJECTIVE

Colorectal cancer is one of the leading causes of cancer-related deaths in the world, although it can be effectively treated if detected early. Teleoperated flexible endoscopes are an emerging technology to ease patient apprehension about the procedure, and subsequently increase compliance. Essential to teleoperation is robust feedback reflecting the change in pose (i.e., position and orientation) of the tip of the endoscope. The goal of this study is to first describe a novel image-based tracking system for teleoperated flexible endoscopes, and subsequently determine its viability in a clinical setting. The proposed approach leverages artificial neural networks (ANNs) to learn the mapping that links the optical flow between two sequential images to the change in the pose of the camera. Secondly, the study investigates for the first time how narrow band illumination (NBI) - today available in commercial gastrointestinal endoscopes - can be applied to enhance feature extraction, and quantify the effect of NBI and white light illumination (WLI), as well as their color information, on the strength of features extracted from the endoscopic camera stream.

METHODS AND MATERIALS

In order to provide the best features for the neural networks to learn the change in pose based on the image stream, we investigated two different imaging modalities - WLI and NBI - and we applied two different spatial partitions - lumen-centered and grid-based - to create descriptors used as input to the ANNs. An experiment was performed to compare the error of these four variations, measured in root mean square error (RMSE) from ground truth given by a robotic arm, to that of a commercial state-of-the-art magnetic tracker. The viability of this technique for a clinical setting was then tested using the four ANN variations, a magnetic tracker, and a commercial colonoscope. The trial was performed by an expert endoscopist (>2000 lifetime procedures) on a colonoscopy training model with porcine blood, and the RMSE of the ANN output was calculated with respect to the magnetic tracker readings. Using the image stream obtained from the commercial endoscope, the strength of features extracted was evaluated.

RESULTS

In the first experiment, the best ANNs resulted from grid-based partitioning under WLI (2.42mm RMSE) for position, and from lumen-centered partitioning under NBI (1.69° RMSE) for rotation. By comparison, the performance of the tracker was 2.49mm RMSE in position and 0.89° RMSE in rotation. The trial with the commercial endoscope indicated that lumen-centered partitioning was the best overall, while NBI outperformed WLI in terms of illumination modality. The performance of lumen-centered partitioning with NBI was 1.03±0.8mm RMSE in positional degrees of freedom (DOF), and 1.26±0.98° RMSE in rotational DOF, while with WLI, the performance was 1.56±1.15mm RMSE in positional DOF and 2.45±1.90° RMSE in rotational DOF. Finally, the features extracted under NBI were found to be twice as strong as those extracted under WLI, but no significance in feature strengths was observed between a grayscale version of the image, and the red, blue, and green color channels.

CONCLUSIONS

This work demonstrates that both WLI and NBI, combined with feature partitioning based on the anatomy of the colon, provide valid mechanisms for endoscopic camera pose estimation via image stream. Illumination provided by WLI and NBI produce ANNs with similar performance which are comparable to that of a state-of-the-art magnetic tracker. However, NBI produces features that are stronger than WLI, which enables more robust feature tracking, and better performance of the ANN in terms of accuracy. Thus, NBI with lumen-centered partitioning resulted the best approach among the different variations tested for vision-based pose estimation. The proposed approach takes advantage of components already available in commercial gastrointestinal endoscopes to provide accurate feedback about the motion of the tip of the endoscope. This solution may serve as an enabling technology for closed-loop control of teleoperated flexible endoscopes.

Externally navigated bronchoscopy using 2-D motion sensors: dynamic phantom validation

The paper presents a new endoscope motion tracking method that is based on a novel external endoscope tracking device and our modified stochastic optimization method for boosting endoscopy navigation. We designed a novel tracking prototype where a 2-D motion sensor was introduced to directly measure the insertion-retreat linear motion and also the rotation of the endoscope. With our developed stochastic optimization method, which embeds traceable particle swarm optimization in the Condensation algorithm, a full six degrees-of-freedom endoscope pose (position and orientation) can be recovered from 2-D motion sensor measurements. Experiments were performed on a dynamic bronchial phantom with maximal simulated respiratory motion around 24.0 mm. The experimental results demonstrate that our proposed method provides a promising endoscope motion tracking approach with more effective and robust performance than several current available tracking techniques. The average tracking accuracy of the position improved from 6.5 to 3.3 mm, which further approaches the clinical requirement of 2.0 mm in practice.

Interactive CT-video registration for the continuous guidance of bronchoscopy

Bronchoscopy is a major step in lung cancer staging. To perform bronchoscopy, the physician uses a procedure plan, derived from a patient's 3D computed-tomography (CT) chest scan, to navigate the bronchoscope through the lung airways. Unfortunately, physicians vary greatly in their ability to perform bronchoscopy. As a result, image-guided bronchoscopy systems, drawing upon the concept of CT-based virtual bronchoscopy (VB), have been proposed. These systems attempt to register the bronchoscope's live position within the chest to a CT-based virtual chest space. Recent methods, which register the bronchoscopic video to CT-based endoluminal airway renderings, show promise but do not enable continuous real-time guidance. We present a CT-video registration method inspired by computer-vision innovations in the fields of image alignment and image-based rendering. In particular, motivated by the Lucas-Kanade algorithm, we propose an inverse-compositional framework built around a gradient-based optimization procedure. We next propose an implementation of the framework suitable for image-guided bronchoscopy. Laboratory tests, involving both single frames and continuous video sequences, demonstrate the robustness and accuracy of the method. Benchmark timing tests indicate that the method can run continuously at 300 frames/s, well beyond the real-time bronchoscopic video rate of 30 frames/s. This compares extremely favorably to the ≥ 1 s/frame speeds of other methods and indicates the method's potential for real-time continuous registration. A human phantom study confirms the method's efficacy for real-time guidance in a controlled setting, and, hence, points the way toward the first interactive CT-video registration approach for image-guided bronchoscopy. Along this line, we demonstrate the method's efficacy in a complete guidance system by presenting a clinical study involving lung cancer patients.

Towards automated visual flexible endoscope navigation

BACKGROUND

The design of flexible endoscopes has not changed significantly in the past 50 years. A trend is observed towards a wider application of flexible endoscopes with an increasing role in complex intraluminal therapeutic procedures. The nonintuitive and nonergonomical steering mechanism now forms a barrier in the extension of flexible endoscope applications. Automating the navigation of endoscopes could be a solution for this problem. This paper summarizes the current state of the art in image-based navigation algorithms. The objectives are to find the most promising navigation system(s) to date and to indicate fields for further research.

METHODS

A systematic literature search was performed using three general search terms in two medical-technological literature databases. Papers were included according to the inclusion criteria. A total of 135 papers were analyzed. Ultimately, 26 were included.

RESULTS

Navigation often is based on visual information, which means steering the endoscope using the images that the endoscope produces. Two main techniques are described: lumen centralization and visual odometry. Although the research results are promising, no successful, commercially available automated flexible endoscopy system exists to date.

CONCLUSIONS

Automated systems that employ conventional flexible endoscopes show the most promising prospects in terms of cost and applicability. To produce such a system, the research focus should lie on finding low-cost mechatronics and technologically robust steering algorithms. Additional functionality and increased efficiency can be obtained through software development. The first priority is to find real-time, robust steering algorithms. These algorithms need to handle bubbles, motion blur, and other image artifacts without disrupting the steering process.

Hybrid electromagnetic and image-based tracking of endoscopes with guaranteed smooth output

PURPOSE

Flexible fiber-optic bronchoscopy is a widespread medical procedure for the diagnosis and treatment of lung diseases. Navigation systems are needed to track the flexible endoscope within the bronchial tree. Electromagnetic (EM) tracking is currently the only technology used clinically for this purpose. The registration between EM tracking and patient anatomy may become inaccurate due to breathing motion, so the addition of image-based tracking has been proposed as a hybrid EM-image-based system.

METHODS

When EM tracking is used as an initialization for image registration, small changes in the initialization may lead to different local minima and noise is amplified by hybrid tracking. The tracking output is modeled as continuous and uses splines for interpolation, thus smoothness is greatly improved. The bronchoscope pose relative to computed tomography data is interpolated using Catmull-Rom splines for position and spherical linear interpolation (SLERP) for orientation.

RESULTS

The hybrid method was evaluated using ground truth poses manually selected by experts, where mean inter-expert agreement was determined as 1.26 mm. Using four dynamic phantom data sets, the accuracy was 4.91 mm, which is equivalent to previous methods. Compared to state-of-art methods, inter-frame smoothness was improved from 2.77-3.72 to 1.24 mm.

CONCLUSIONS

Hybrid image and electromagnetic endoscope guidance provides a more realistic and physically plausible solution with significantly less jitter. This quantitative result is confirmed by visual comparison of real and virtual video, where the virtual video output is much more consistent and robust, with fewer occasions of tracking loss or unexpected movement compared with previous methods.

An endotracheal intubation confirmation system based on carina image detection: a proof of concept

In this paper, a novel system for automatic confirmation of endotracheal intubation is proposed. The system comprises a miniature CMOS sensor and electric wires attached to a rigid stylet. Video signals are continuously acquired and processed by the algorithm implemented on a PC/DSP. The system is based on detection of the carina image as an anatomical landmark of correct tube positioning and it thus utilizes direct visual cues. Detection of the carina is performed based on unsupervised clustering, using a greedy-Gaussian mixture framework. The performance of the proposed system was initially evaluated using a mannequin model. A scientific prototype was assembled and used to perform repeated intubations on the model and collect a database of video signals which were processed off-line. The videos were categoried by a medical professional into carina, upper-trachea, and esophagus. An accuracy of 100% was achieved in discriminating between the carina and other anatomical structures including esophagus and upper-trachea. As an additional validation, the system was tested using a dataset of 231 video images recorded from five human subjects during intubation. The system correctly classified 120 out of 125 non-carina images (i.e. a sensitivity of 96.0%), and 100 out of 106 carina images (i.e. a specificity 94.3%). Using a 10th-order median filter, applied on the frame-based classification results, a 100% accuracy rate was obtained.

Endotracheal intubation confirmation based on video image classification using a parallel GMMs framework: a preliminary evaluation

In this paper, the problem of endotracheal intubation confirmation is addressed. Endotracheal intubation is a complex procedure which requires high skills and the use of secondary confirmation devices to ensure correct positioning of the tube. A novel confirmation approach, based on video images classification, is introduced. The approach is based on identification of specific anatomical landmarks, including esophagus, upper trachea and main bifurcation of the trachea into the two primary bronchi ("carina"), as indicators of correct or incorrect tube insertion and positioning. Classification of the images is performed using a parallel Gaussian mixture models (GMMs) framework, which is composed of several GMMs, schematically connected in parallel, where each GMM represents a different imaging angle. The performance of the proposed approach was evaluated using a dataset of cow-intubation videos and a dataset of human-intubation videos. Each one of the video images was manually (visually) classified by a medical expert into one of three categories: upper-tracheal intubation, correct (carina) intubation, and esophageal intubation. The image classification algorithm was applied off-line using a leave-one-case-out method. The results show that the system correctly classified 1517 out of 1600 (94.8%) of the cow-intubation images, and 340 out of the 358 human images (95.0%). The classification results compared favorably with a "standard" GMM approach utilizing textural based features, as well as with a state-of-the-art classification method, tested on the cow-intubation dataset.