Use of shape-from-shading to estimate three-dimensional architecture in the small intestinal lumen of celiac and control patients

Real-time virtual intraoperative CT in endoscopic sinus surgery

PURPOSE

Endoscopic sinus surgery (ESS) is typically guided under preoperative computed tomography (CT), which increasingly diverges from actual patient anatomy as the surgery progresses. Studies have reported that the revision surgery rate in ESS ranges between 28 and 47%. This paper presents a method that can update the preoperative CT in real time to improve surgical completeness in ESS.

APPROACH

The work presents and compares three novel methods that use instrument motion data and anatomical structures to predict surgical modifications in real time. The methods use learning techniques, such as nonparametric filtering and Gaussian process regression, to correlate surgical modifications with instrument tip positions, tip trajectories, and instrument shapes. Preoperative CT image sets are updated with modification predictions to serve as a virtual intraoperative CT.

RESULTS

The three methods were compared in eight ESS cadaver cases, which were performed by five surgeons and included the following representative ESS operations: maxillary antrostomy, uncinectomy, anterior and posterior ethmoidectomy, and sphenoidotomy. Experimental results showed accuracy metrics that were clinically acceptable with dice similarity coefficients > 86%, with F-score > 92% and precision > 89.91% in surgical completeness evaluation. Among the three methods, the tip trajectory-based estimator had the highest precision of 96.87%.

CONCLUSIONS

This work demonstrated that virtually modified intraoperative CT scans improved the consistency between the actual surgical scene and the reference model, and could lead to improved surgical completeness in ESS. Compared to actual intraoperative CT scans, the proposed method has no impact on existing surgical protocols, does not require extra hardware, does not expose the patient to radiation, and does not lengthen time under anesthesia.

Current Evidence on Computer-Aided Diagnosis of Celiac Disease: Systematic Review

Celiac disease (CD) is a chronic autoimmune disease that occurs in genetically predisposed individuals in whom the ingestion of gluten leads to damage of the small bowel. It is estimated to affect 1 in 100 people worldwide, but is severely underdiagnosed. Currently available guidelines require CD-specific serology and atrophic histology in duodenal biopsy samples for the diagnosis of adult CD. In pediatric CD, but in recent years in adults also, nonbioptic diagnostic strategies have become increasingly popular. In this setting, in order to increase the diagnostic rate of this pathology, endoscopy itself has been thought of as a case finding strategy by use of digital image processing techniques. Research focused on computer aided decision support used as database video capsule, endoscopy and even biopsy duodenal images. Early automated methods for diagnosis of celiac disease used feature extraction methods like spatial domain features, transform domain features, scale-invariant features and spatio-temporal features. Recent artificial intelligence (AI) techniques using deep learning (DL) methods such as convolutional neural network (CNN), support vector machines (SVM) or Bayesian inference have emerged as a breakthrough computer technology which can be used for computer aided diagnosis of celiac disease. In the current review we summarize methods used in clinical studies for classification of CD from feature extraction methods to AI techniques.

Sampling Based on Kalman Filter for Shape from Focus in the Presence of Noise

Recovering three-dimensional (3D) shape of an object from two-dimensional (2D) information is one of the major domains of computer vision applications. Shape from Focus (SFF) is a passive optical technique that reconstructs 3D shape of an object using 2D images with different focus settings. When a 2D image sequence is obtained with constant step size in SFF, mechanical vibrations, referred as jitter noise, occur in each step. Since the jitter noise changes the focus values of 2D images, it causes erroneous recovery of 3D shape. In this paper, a new filtering method for estimating optimal image positions is proposed. First, jitter noise is modeled as Gaussian or speckle function, secondly, the focus curves acquired by one of the focus measure operators are modeled as a quadratic function for application of the filter. Finally, Kalman filter as the proposed method is designed and applied for removing jitter noise. The proposed method is experimented by using image sequences of synthetic and real objects. The performance is evaluated through various metrics to show the effectiveness of the proposed method in terms of reconstruction accuracy and computational complexity. Root Mean Square Error (RMSE), correlation, Peak Signal-to-Noise Ratio (PSNR), and computational time of the proposed method are improved on average by about 48%, 11%, 15%, and 5691%, respectively, compared with conventional filtering methods.

Capsule endoscopy for patients with coeliac disease

Coeliac disease is an autoimmune mediated condition in response to gluten. A combination of innate and adaptive immune responses results in villous shortening in the small bowel (SB) that can be morphologically picked up on capsule endoscopy. It is the only imaging modality that can provide mucosal views of the entire SB, while histology is generally limited to the proximal SB. Radiological modalities are not designed to pick up changes in villous morphology. Areas covered: In this review, we provide a comprehensive analysis on the justified use of small bowel capsule endoscopy (SBCE) in the assessment of patients with coeliac disease; compare SBCE to histology, serology, and symptomatology; and provide an overview on automated quantitative analysis for the detection of coeliac disease. We also provide insight into future work on SBCE in relation to coeliac disease. Expert commentary: SBCE has opened up new avenues for the diagnosis and monitoring of patients with coeliac disease. However, larger studies with new and established coeliac disease patients and with greater emphasis on morphological features on SBCE are required to better define the role of SBCE in the setting of coeliac disease.

Use of shape-from-shading to characterize mucosal topography in celiac disease videocapsule images

AIM

To use a computerized shape-from-shading technique to characterize the topography of the small intestinal mucosa.

METHODS

Videoclips comprised of 100-200 images each were obtained from the distal duodenum in 10 celiac and 10 control patients. Images with high texture were selected from each videoclip and projected from two to three dimensions by using grayscale pixel brightness as the Z-axis spatial variable. The resulting images for celiac patients were then ordered using the Marsh score to estimate the degree of villous atrophy, and compared with control data.

RESULTS

Topographic changes in celiac patient three-dimensional constructs were often more variable as compared to controls. The mean absolute derivative in elevation was 2.34 ± 0.35 brightness units for celiacs vs 1.95 ± 0.28 for controls (P = 0.014). The standard deviation of the derivative in elevation was 4.87 ± 0.35 brightness units for celiacs vs 4.47 ± 0.36 for controls (P = 0.023). Celiac patients with Marsh IIIC villous atrophy tended to have the largest topographic changes. Plotted in two dimensions, celiac data could be separated from controls with 80% sensitivity and specificity.

CONCLUSION

Use of shape-from-shading to construct three-dimensional projections approximating the actual spatial geometry of the small intestinal substrate is useful to observe features not readily apparent in two-dimensional videocapsule images. This method represents a potentially helpful adjunct to detect areas of pathology during videocapsule analysis.

Coeliac disease and the videocapsule: what have we learned till now

Celiac disease is diagnosed in part by finding areas of pathology in the small bowel (SB) mucosa. This can often be difficult because the pathologic alterations, including atrophy of the small intestinal villi, can often be sparse and subtle. Some of the quantitative methods for detecting and measuring the presence of villous atrophy from videocapsule endoscopy (VCE) images are presented and discussed. These methods consist of static features of measurement including texture, gray level, and presence and abundance of fissures contained within each acquired image. The methods also consist of dynamic measurements including spectral analysis, and determining motion from a sequence of endoscopic images as obtained from a VCE clip. Thus far, several methods have been found useful to characterize the SB mucosa of untreated celiac disease patients versus control patients lacking villous atrophy, which have revealed significant differences in texture, frequency, and motion on analysis of VCE. In untreated celiac patients undergoing endoscopy, there tends to be greater magnitude of changes and spatial differences in textural descriptors, longer periodic components, indicating slower periodic activity, and differences in feature location, suggesting alterations in motility at areas of pathology as compared to patients without villous atrophy. Improvements in the quantitative analysis of VCE imaging in celiac patients is important to detect pathology in suspected patients, so that biopsies can be obtained from pertinent regions of the small intestinal mucosa. Improvements are also necessary so that patients with celiac disease can be monitored to evaluate the progress of mucosal healing after onset of treatment.

Recommendations to quantify villous atrophy in video capsule endoscopy images of celiac disease patients

AIM

To quantify the presence of villous atrophy in endoscopic images for improved automation.

METHODS

There are two main categories of quantitative descriptors helpful to detect villous atrophy: (1) Statistical and (2) Syntactic. Statistical descriptors measure the small intestinal substrate in endoscope-acquired images based on mathematical methods. Texture is the most commonly used statistical descriptor to quantify villous atrophy. Syntactic descriptors comprise a syntax, or set of rules, for analyzing and parsing the substrate into a set of objects with boundaries. The syntax is designed to identify and distinguish three-dimensional structures based on their shape.

RESULTS

The variance texture statistical descriptor is useful to describe the average variability in image gray level representing villous atrophy, but does not determine the range in variability and the spatial relationships between regions. Improved textural descriptors will incorporate these factors, so that areas with variability gradients and regions that are orientation dependent can be distinguished. The protrusion syntactic descriptor is useful to detect three-dimensional architectural components, but is limited to identifying objects of a certain shape. Improvement in this descriptor will require incorporating flexibility to the prototypical template, so that protrusions of any shape can be detected, measured, and distinguished.

CONCLUSION

Improved quantitative descriptors of villous atrophy are being developed, which will be useful in detecting subtle, varying patterns of villous atrophy in the small intestinal mucosa of suspected and known celiac disease patients.

Extraction and processing of videocapsule data to detect and measure the presence of villous atrophy in celiac disease patients

BACKGROUND

Videocapsule endoscopy is a relative new method to analyze the gastrointestinal tract for the presence of pathologic features. It is of relevance to detect villous atrophy in the small bowel, which is a defining symptom of celiac disease.

METHOD

In this tutorial, methods to extract and process videocapsule endoscopy data are elucidated. The algorithms, computer code, and paradigms to analyze image series are described in detail. The topics covered include extraction of data, analysis of texture, eigenanalysis, spectral analysis, three-dimensional projection, and estimation of motility. The basic paradigms to implement these processes are provided.

RESULTS

Examples of successful quantitative analysis implementations for selected untreated celiac disease patients with villous atrophy versus control patients with normal villi were illustrated. Based on the implementations, it was evident that celiac patients tended to have a rougher small intestinal texture as compared with control patients. From three-dimensional projection, celiac patients exhibited larger surface protrusions emanating from the small intestinal mucosa, which may represent clumps of atrophied villi. The periodicity of small intestinal contractions tends to be slower when villous atrophy is present, and the estimated degree of motility is reduced as compared with control image series. Basis image construction suggested that fissuring and mottling of the mucosal surface is predominant in untreated celiac patients, and mostly absent in controls.

CONCLUSIONS

Implementation of computerized methods, as described in this tutorial, will likely be useful for the automated detection and measurement of villous atrophy, and to map its extent along the small intestine of celiac patients.

Suggestions for automatic quantitation of endoscopic image analysis to improve detection of small intestinal pathology in celiac disease patients

Although many groups have attempted to develop an automated computerized method to detect pathology of the small intestinal mucosa caused by celiac disease, the efforts have thus far failed. This is due in part to the occult presence of the disease. When pathological evidence of celiac disease exists in the small bowel it is visually often patchy and subtle. Due to presence of extraneous substances such as air bubbles and opaque fluids, the use of computerized automation methods have only been partially successful in detecting the hallmarks of the disease in the small intestine-villous atrophy, fissuring, and a mottled appearance. By using a variety of computerized techniques and assigning a weight or vote to each technique, it is possible to improve the detection of abnormal regions which are indicative of celiac disease, and of treatment progress in diagnosed patients. Herein a paradigm is suggested for improving the efficacy of automated methods for measuring celiac disease manifestation in the small intestinal mucosa. The suggestions are applicable to both standard and videocapsule endoscopic imaging, since both methods could potentially benefit from computerized quantitation to improve celiac disease diagnosis.

Quantitative image analysis of celiac disease

We outline the use of quantitative techniques that are currently used for analysis of celiac disease. Image processing techniques can be useful to statistically analyze the pixular data of endoscopic images that is acquired with standard or videocapsule endoscopy. It is shown how current techniques have evolved to become more useful for gastroenterologists who seek to understand celiac disease and to screen for it in suspected patients. New directions for focus in the development of methodology for diagnosis and treatment of this disease are suggested. It is evident that there are yet broad areas where there is potential to expand the use of quantitative techniques for improved analysis in suspected or known celiac disease patients.

Survey on computer aided decision support for diagnosis of celiac disease

Celiac disease (CD) is a complex autoimmune disorder in genetically predisposed individuals of all age groups triggered by the ingestion of food containing gluten. A reliable diagnosis is of high interest in view of embarking on a strict gluten-free diet, which is the CD treatment modality of first choice. The gold standard for diagnosis of CD is currently based on a histological confirmation of serology, using biopsies performed during upper endoscopy. Computer aided decision support is an emerging option in medicine and endoscopy in particular. Such systems could potentially save costs and manpower while simultaneously increasing the safety of the procedure. Research focused on computer-assisted systems in the context of automated diagnosis of CD has started in 2008. Since then, over 40 publications on the topic have appeared. In this context, data from classical flexible endoscopy as well as wireless capsule endoscopy (WCE) and confocal laser endomicrosopy (CLE) has been used. In this survey paper, we try to give a comprehensive overview of the research focused on computer-assisted diagnosis of CD.

•

The state-of-the-art research in automated diagnosis of celiac disease is presented.

•

A systematic review of methods and techniques used in this field is given.

•

Specific issues and challenges in the field are identified and discussed.

Utility of 3-dimensional image reconstruction in the diagnosis of small-bowel masses in capsule endoscopy (with video)

BACKGROUND

In small-bowel capsule endoscopy (SBCE), differentiating masses (ie, lesions of higher probability for neoplasia) requiring more aggressive intervention from bulges (essentially, false-positive findings) is a challenging task; recently, software that enables 3-dimensional (3D) reconstruction has become available.

OBJECTIVE

To evaluate whether "coupling" 3D reconstructed video clips with the standard 2-dimensional (s2D) counterparts helps in distinguishing masses from bulges.

DESIGN

Three expert and 3 novice SBCE readers, blind to others and in a random order, reviewed the s2D video clips and subsequently the s2D clips coupled with their 3D reconstruction (2D+3D).

SETTING

Multicenter study in 3 community hospitals in Italy and a university hospital in Scotland.

PATIENTS

Thirty-two deidentified 5-minute video clips, containing mucosal bulging (19) or masses (13).

INTERVENTION

3D reconstruction of s2D SBCE video clips.

MAIN OUTCOME MEASURE

Differentiation of masses from bulges with s2D and 2D+3D video clips, estimated by the area under the receiver operating characteristic curve (AUC); interobserver agreement.

RESULTS

AUC for experts and novices for s2D video clips was .74 and .5, respectively (P = .0053). AUC for experts and novices with 2D+3D was .70 (compared with s2D: P = .245) and .57 (compared s2D: P = .049), respectively. AUC for experts and novices with 2D+3D was similar (P = .1846). The interobserver agreement was good for both experts and novices with the s2D (k = .71 and .54, respectively) and the 2D+3D video clips (k = .58 in both groups).

LIMITATIONS

Few, short video clips; fixed angle of 3D reconstruction.

CONCLUSIONS

The adjunction of a 3D reconstruction to the s2D video reading platform does not improve the performance of expert SBCE readers, although it significantly increases the performance of novices in distinguishing masses from bulging.

Stereoscopic visualization of laparoscope image using depth information from 3D model

Laparoscopic surgery is indispensable from the current surgical procedures. It uses an endoscope system of camera and light source, and surgical instruments which pass through the small incisions on the abdomen of the patients undergoing laparoscopic surgery. Conventional laparoscope (endoscope) systems produce 2D colored video images which do not provide surgeons an actual depth perception of the scene. In this work, the problem was formulated as synthesizing a stereo image of the monocular (conventional) laparoscope image by incorporating into them the depth information from a 3D CT model. Various algorithms of the computer vision including the algorithms for the feature detection, matching and tracking in the video frames, and for the reconstruction of 3D shape from shading in the 2D laparoscope image were combined for making the system. The current method was applied to the laparoscope video at the rate of up to 5 frames per second to visualize its stereo video. A correlation was investigated between the depth maps calculated with our method with those from the shape from shading algorithm. The correlation coefficients between the depth maps were within the range of 0.70-0.95 (P<0.05). A t-test was used for the statistical analysis.

Evaluation of 4 three-dimensional representation algorithms in capsule endoscopy images

AIM: To evaluate the three-dimensional (3-D) representation performance of 4 publicly available Shape-from-Shading (SfS) algorithms in small-bowel capsule endoscopy (SBCE).

METHODS: SfS techniques recover the shape of objects using the gradual variation of shading. There are 4 publicly available SfS algorithms. To the best of our knowledge, no comparative study with images obtained during clinical SBCE has been performed to date. Three experienced reviewers were asked to evaluate 54 two-dimensional (2-D) images (categories: protrusion/inflammation/vascular) transformed to 3-D by the aforementioned SfS 3-D algorithms. The best algorithm was selected and inter-rater agreement was calculated.

RESULTS: Four publicly available SfS algorithms were compared. Tsai’s SfS algorithm outperformed the rest (selected as best performing in 45/54 SBCE images), followed by Ciuti’s algorithm (best performing in 7/54 images) and Torreão’s (in 1/54 images). In 26/54 images; Tsai’s algorithm was unanimously selected as the best performing 3-D representation SfS software. Tsai’s 3-D algorithm superiority was independent of lesion category (protrusion/inflammatory/vascular; P = 0.678) and/or CE system used to obtain the 2-D images (MiroCam^®/PillCam^®; P = 0.558). Lastly, the inter-observer agreement was good (kappa = 0.55).

CONCLUSION: 3-D representation software offers a plausible alternative for 3-D representation of conventional capsule endoscopy images (until optics technology matures enough to allow hardware enabled-“real” 3-D reconstruction of the gastrointestinal tract).