Specular Reflection Removal for 3D Reconstruction of Tissues using Endoscopy Videos

Endoscopy is widely employed for diagnostic examination of the interior of organs and body cavities and numerous surgical interventions. Still, the inability to correlate individual 2D images with 3D organ morphology limits its applications, especially in intra-operative planning and navigation, disease physiology, cancer surveillance, etc. As a result, most endoscopy videos, which carry enormous data potential, are used only for real-time guidance and are discarded after collection. We present a complete method for the 3D reconstruction of inner organs that suggests image extraction techniques from endoscopic videos and a novel image pre-processing technique to reconstruct and visualize a 3D model of organs from an endoscopic video. We use advanced computer vision methods and do not require any modifications to the clinical-grade endoscopy hardware. We have also formalized an image acquisition protocol through experimentation with a calibrated test bed. We validate the accuracy and robustness of our reconstruction using a test bed with known ground truth. Our method can significantly contribute to endoscopy-based diagnostic and surgical procedures using comprehensive tissue and tumor 3D visualization.

A Two-Phase Approach using Mask R-CNN and 3D U-Net for High-Accuracy Automatic Segmentation of Pancreas in CT Imaging

BACKGROUND AND OBJECTIVE

The size, shape, and position of the pancreas are affected by the patient characteristics such as age, sex, adiposity. Owing to more complex anatomical structures (size, shape, and position) of the pancreas, specialists have some difficulties in the analysis of pancreatic diseases (diabetes, pancreatic cancer, pancreatitis). Therefore, the treatment of the disease requires enormous time and depends on the experience of specialists. In order to decrease the rate of pancreatic disease deaths and to assist the specialist in the analysis of pancreatic diseases, automatic pancreas segmentation techniques have been actively developed in the research article for many years.

METHODS

Although the rapid growth of deep learning and proving satisfactory performance in many medical image processing and computer vision applications, the maximum Dice Similarity Coefficients (DSC) value of these techniques related to automatic pancreas segmentation is only around 85% due to complex structure of the pancreas and other factors. Contrary to previous techniques which are required significantly higher computational power and memory, this paper suggests a novel two-phase approach for high-accuracy automatic pancreas segmentation in computed tomography (CT) imaging. The proposed approach consists of two phases; (1) Pancreas Localization, where the rough pancreas position is detected on the 2D CT slice by adopting Mask R-CNN model, (2) Pancreas Segmentation, where the segmented pancreas region is produced by refining the candidate pancreas region with 3D U-Net on the 2D sub-CT slices generated in the first phase. Both qualitative and quantitative assessments of the proposed approach are performed on the NIH data set.

RESULTS

In order to evaluate the achievement of the recommended approach, a total of 16 different automatic pancreas segmentation techniques reported in the literature are compared by utilizing performance assessment procedures which are Dice Similarity Coefficient (DSC), Jaccard Index (JI), Precision (PRE), Recall (REC), Pixel Accuracy (ACC), Specificity (SPE), Receiver Operating Characteristics (ROC) and Area under ROC curve (AUC). The average values of DSC, JI, REC and ACC are computed as 86.15%, 75.93%, 86.27%, 86.27% and 99.95% respectively, which are the best values among well-established studies for automatic pancreas segmentation.

CONCLUSION

It is demonstrated with qualitative and quantitative results that our suggested two-phase approach creates more favorable results than other existing approaches.

A task and performance analysis of endoscopic submucosal dissection (ESD) surgery

BACKGROUND

ESD is an endoscopic technique for en bloc resection of gastrointestinal lesions. ESD is a widely-used in Japan and throughout Asia, but not as prevalent in Europe or the US. The procedure is technically challenging and has higher adverse events (bleeding, perforation) compared to endoscopic mucosal resection. Inadequate training platforms and lack of established training curricula have restricted its wide acceptance in the US. Thus, we aim to develop a Virtual Endoluminal Surgery Simulator (VESS) for objective ESD training and assessment. In this work, we performed task and performance analysis of ESD surgeries.

METHODS

We performed a detailed colorectal ESD task analysis and identified the critical ESD steps for lesion identification, marking, injection, circumferential cutting, dissection, intraprocedural complication management, and post-procedure examination. We constructed a hierarchical task tree that elaborates the order of tasks in these steps. Furthermore, we developed quantitative ESD performance metrics. We measured task times and scores of 16 ESD surgeries performed by four different endoscopic surgeons.

RESULTS

The average time of the marking, injection, and circumferential cutting phases are 203.4 (σ: 205.46), 83.5 (σ: 49.92), 908.4 s. (σ: 584.53), respectively. Cutting the submucosal layer takes most of the time of overall ESD procedure time with an average of 1394.7 s (σ: 908.43). We also performed correlation analysis (Pearson's test) among the performance scores of the tasks. There is a moderate positive correlation (R = 0.528, p = 0.0355) between marking scores and total scores, a strong positive correlation (R = 0.7879, p = 0.0003) between circumferential cutting and submucosal dissection and total scores. Similarly, we noted a strong positive correlation (R = 0.7095, p = 0.0021) between circumferential cutting and submucosal dissection and marking scores.

CONCLUSIONS

We elaborated ESD tasks and developed quantitative performance metrics used in analysis of actual surgery performance. These ESD metrics will be used in future validation studies of our VESS simulator.

Generative Anatomy Modeling Language (GAML)

BACKGROUND

This paper presents the Generative Anatomy Modeling Language (GAML) for generating variation of 3D virtual human anatomy in real-time. This framework provides a set of operators for modification of a reference base 3D anatomy. The perturbation of the 3D models is satisfied with nonlinear geometry constraints to create an authentic human anatomy.

METHODS

GAML was used to create 3D difficult anatomical scenarios for virtual simulation of airway management techniques such as Endotracheal Intubation (ETI) and Cricothyroidotomy (CCT). Difficult scenarios for each technique were defined and the model variations procedurally created with GAML.

CONCLUSION

This study presents details of the GAML design, set of operators, types of constraints. Cases of CCT and ETI difficulty were generated and confirmed by expert surgeons. Execution performance pertaining to an increasing complexity of constraints using nonlinear programming was in real-time execution.

Determination of ECoG information flow activity based on Granger causality and Hilbert transformation

Analysis of directional information flow patterns among different regions of the brain is important for investigating the relation between ECoG (electrocorticographic) and mental activity. The objective is to study and evaluate the information flow activity at different frequencies in the primary motor cortex. We employed Granger causality for capturing the future state of the propagation path and direction between recording electrode sites on the cerebral cortex. A grid covered the right motor cortex completely due to its size (approx. 8 cm×8 cm) but grid area extends to the surrounding cortex areas. During the experiment, a subject was asked to imagine performing two activities: movement of the left small finger and/or movement of the tongue. The time series of the electrical brain activity was recorded during these trials using an 8×8 (0.016-300 Hz band with) ECoG platinum electrode grid, which was placed on the contralateral (right) motor cortex. For detection of information flow activity and communication frequencies among the electrodes, we have proposed a method based on following steps: (i) calculation of analytical time series such as amplitude and phase difference acquired from Hilbert transformation, (ii) selection of frequency having highest interdependence for the electrode pairs for the concerned time series over a sliding window in which we assumed time series were stationary, (iii) calculation of Granger causality values for each pair with selected frequency. The information flow (causal influence) activity and communication frequencies between the electrodes in grid were determined and shown successfully. It is supposed that information flow activity and communication frequencies between the electrodes in the grid are approximately the same for the same pattern. The successful employment of Granger causality and Hilbert transformation for the detection of the propagation path and direction of each component of ECoG among different sub-cortex areas were capable of determining the information flow (causal influence) activity and communication frequencies between the populations of neurons successfully.

Content-based management service for medical videos

Development of health information technology has had a dramatic impact to improve the efficiency and quality of medical care. Developing interoperable health information systems for healthcare providers has the potential to improve the quality and equitability of patient-centered healthcare. In this article, we describe an automated content-based medical video analysis and management service that provides convenience and ease in accessing the relevant medical video content without sequential scanning. The system facilitates effective temporal video segmentation and content-based visual information retrieval that enable a more reliable understanding of medical video content. The system is implemented as a Web- and mobile-based service and has the potential to offer a knowledge-sharing platform for the purpose of efficient medical video content access.

Text-to-audiovisual speech synthesizer for children with learning disabilities

Learning disabilities affect the ability of children to learn, despite their having normal intelligence. Assistive tools can highly increase functional capabilities of children with learning disorders such as writing, reading, or listening. In this article, we describe a text-to-audiovisual synthesizer that can serve as an assistive tool for such children. The system automatically converts an input text to audiovisual speech, providing synchronization of the head, eye, and lip movements of the three-dimensional face model with appropriate facial expressions and word flow of the text. The proposed system can enhance speech perception and help children having learning deficits to improve their chances of success.

Hybrid Matching Algorithm for Personal Names

Companies acquire personal information from phone, World Wide Web, or email in order to sell or send an advertisement about their product. However, when this information is acquired, moved, copied, or edited, the data may lose its quality. Often, the use of data administrators or a tool that has limited capabilities to correct the mistyped information can cause many problems. Moreover, most of the correction techniques are particularly implemented for the words used in daily conversations. Since personal names have different characteristics compared to general text, a hybrid matching algorithm (PNRS) which employs phonetic encoding, string matching and statistical facts to provide a possible candidate for misspelled names is developed. At the end, the efficiency of the proposed algorithm is compared with other well known spelling correction techniques.

Mixed reality simulation of rasping procedure in artificial cervical disc replacement (ACDR) surgery

BACKGROUND

Until quite recently spinal disorder problems in the U.S. have been operated by fusing cervical vertebrae instead of replacement of the cervical disc with an artificial disc. Cervical disc replacement is a recently approved procedure in the U.S. It is one of the most challenging surgical procedures in the medical field due to the deficiencies in available diagnostic tools and insufficient number of surgical practices For physicians and surgical instrument developers, it is critical to understand how to successfully deploy the new artificial disc replacement systems. Without proper understanding and practice of the deployment procedure, it is possible to injure the vertebral body. Mixed reality (MR) and virtual reality (VR) surgical simulators are becoming an indispensable part of physicians' training, since they offer a risk free training environment. In this study, MR simulation framework and intricacies involved in the development of a MR simulator for the rasping procedure in artificial cervical disc replacement (ACDR) surgery are investigated. The major components that make up the MR surgical simulator with motion tracking system are addressed.

FINDINGS

A mixed reality surgical simulator that targets rasping procedure in the artificial cervical disc replacement surgery with a VICON motion tracking system was developed. There were several challenges in the development of MR surgical simulator. First, the assembly of different hardware components for surgical simulation development that involves knowledge and application of interdisciplinary fields such as signal processing, computer vision and graphics, along with the design and placements of sensors etc . Second challenge was the creation of a physically correct model of the rasping procedure in order to attain critical forces. This challenge was handled with finite element modeling. The third challenge was minimization of error in mapping movements of an actor in real model to a virtual model in a process called registration. This issue was overcome by a two-way (virtual object to real domain and real domain to virtual object) semi-automatic registration method.

CONCLUSIONS

The applicability of the VICON MR setting for the ACDR surgical simulator is demonstrated. The main stream problems encountered in MR surgical simulator development are addressed. First, an effective environment for MR surgical development is constructed. Second, the strain and the stress intensities and critical forces are simulated under the various rasp instrument loadings with impacts that are applied on intervertebral surfaces of the anterior vertebrae throughout the rasping procedure. Third, two approaches are introduced to solve the registration problem in MR setting. Results show that our system creates an effective environment for surgical simulation development and solves tedious and time-consuming registration problems caused by misalignments. Further, the MR ACDR surgery simulator was tested by 5 different physicians who found that the MR simulator is effective enough to teach the anatomical details of cervical discs and to grasp the basics of the ACDR surgery and rasping procedure.