Development and assessment of a Microsoft Kinect based system for imaging the breast in three dimensions

Comparison of Nasal Analysis by Photographs (2D) against Low-cost Surface Laser Imaging (3D) and against Computed Axial Tomography Imaging

Introduction In aesthetic surgery, we have a few evaluation tools that numerically and objectively measure the changes we make in patients. This article aimed to evaluate the nasal systematic analysis and compare findings between the three systems of nasal evaluation: photographs 2D, 3D surface imaging with the Kinect system, and 3D CT scan imaging.

Methods We designed a longitudinal and descriptive prospective study with simple non-blind randomization. To compare the systematic nasal analysis between the three methods. If the findings are similar, all three methods would be useful in independent clinical scenarios.

Results A total of 42 observations were included finding a minimum age of 21 with a mean of 28 years old. Also, 64% were female, 93% had adequate facial proportions, and 50% were Fitzpatrick III. For outcome statistics, we found differential nasal deviation between 3D images with a mean of 6.53 mm. While when comparing the nasal dorsum length, we found a statistical significance of p = 0.051. When comparing the nasal dorsum length index, we found no significant difference p = 0.32. Also, we did not find statistical significance when comparing the nasofrontal angle and tip rotation angle p = 1 for both.

Conclusion We found that the population we serve has characteristics of Hispanic mestizo nose. The three methods seem to evaluate systematic nasal analysis in a very similar way, and any of them can be used depending on the scenario and the needs of plastic surgeons.

Structured-light surface scanning system to evaluate breast morphology in standing and supine positions

Breast shapes are affected by gravitational loads and deformities. Measurements obtained in the standing position may not correlate well with measurements in the supine position, which is more representative of patient position during breast surgery. A dual color 3D surface imaging system capable of scanning patients in both supine and standing positions was developed to evaluate the effect of changes in body posture on breast morphology. The system was evaluated with  breast phantoms to assess accuracy, then tested on ten subjects in three body postures to assess its effectiveness as a clinical tool. The accuracy of the system was within 0.4 mm on average across the model. For the human study, there was no effect of body posture on breast volumes (p value > 0.05), but we observed an effect of completeness of breast scans on body posture (p value  < 0.05). Post-hoc tests showed that the supine position and the standing position with hands at the waist differed significantly (p value  < 0.05). This study shows that the system can quantitatively evaluate the effect of subject postures, and thereby has the potential to be used to investigate peri-operative changes in breast morphology.

Relationship between lip motion detected with a compact 3D camera and swallowing dynamics during bolus flow swallowing in Japanese elderly men

Clinical application of a swallowing function evaluation system that is minimally invasive and enables an objective evaluation is necessary. We constructed a system that can synchronise and analyse lip motion with a three-dimensional (3D) camera and swallowing dynamics with videofluoroscopy (VF) and clarified the relationship between lip movement and swallowing dynamics. A compact 3D camera was adapted to Microsoft XBox One Kinect Sensor^® . We examined Kinect's accuracy and repeatability and analysed the highest measurement accuracy and repeatability of the distance between anguli oris muscles. The constructed system simultaneously measured, synchronised and analysed lip motion by Kinect and swallowing dynamics by VF. Fourteen elderly men without dysphagia were included. Barium turbid solution (5, 10, 15 and 20 mL) was used for swallowing. Measurement parameters were the 3D distance between angulus oris displacement (TDDD), swallowing quantity (SQ), oral transit time (OTT), stage transit duration (STD), pharyngeal transit time (PTT) and total swallowing duration (TSD). Statistical analyses were performed. The measurement accuracy and reproducibility were optimum within a 10° horizontal imaging angle at a 120-cm measurement distance. SQ, TDDD, OTT, STD, PTT and TSD showed significant differences, and correlation was found between TDDD and OTT. SQ affected TDDD, OTT, STD, PTT and TSD; OTT could be predicted from TDDD. Improvement in the system and analysis method was considered to enable prediction of swallowing dynamics from body surface movement of the head and neck, including realisation of measurement conditions with higher precision and reproducibility, and from lip motion.

Contactless recording of sleep apnea and periodic leg movements by nocturnal 3-D-video and subsequent visual perceptive computing

Contactless measurements during the night by a 3-D-camera are less time-consuming in comparison to polysomnography because they do not require sophisticated wiring. However, it is not clear what might be the diagnostic benefit and accuracy of this technology. We investigated 59 persons simultaneously by polysomnography and 3-D-camera and visual perceptive computing (19 patients with restless legs syndrome (RLS), 21 patients with obstructive sleep apnea (OSA), and 19 healthy volunteers). There was a significant correlation between the apnea hypopnea index (AHI) measured by polysomnography and respiratory events measured with the 3-D-camera in OSA patients (r = 0.823; p < 0.001). The receiver operating characteristic curve yielded a sensitivity of 90% for OSA with a specificity of 71.4%. In RLS patients 72.8% of leg movements confirmed by polysomnography could be detected by 3-D-video and a significant moderate correlation was found between PLM measured by polysomnography and by the 3-D-camera (RLS: r = 0.654; p = 0.004). In total, 95.4% of the sleep epochs were correctly classified by the machine learning approach, but only 32.5% of awake epochs. Further studies should investigate, if this technique might be an alternative to home sleep testing in persons with an increased pre-test probability for OSA.

Comparison of depth cameras for three-dimensional reconstruction in medicine

KinectFusion is a typical three-dimensional reconstruction technique which enables generation of individual three-dimensional human models from consumer depth cameras for understanding body shapes. The aim of this study was to compare three-dimensional reconstruction results obtained using KinectFusion from data collected with two different types of depth camera (time-of-flight and stereoscopic cameras) and compare these results with those of a commercial three-dimensional scanning system to determine which type of depth camera gives improved reconstruction. Torso mannequins and machined aluminium cylinders were used as the test objects for this study. Two depth cameras, Microsoft Kinect V2 and Intel Realsense D435, were selected as the representatives of time-of-flight and stereoscopic cameras, respectively, to capture scan data for the reconstruction of three-dimensional point clouds by KinectFusion techniques. The results showed that both time-of-flight and stereoscopic cameras, using the developed rotating camera rig, provided repeatable body scanning data with minimal operator-induced error. However, the time-of-flight camera generated more accurate three-dimensional point clouds than the stereoscopic sensor. Thus, this suggests that applications requiring the generation of accurate three-dimensional human models by KinectFusion techniques should consider using a time-of-flight camera, such as the Microsoft Kinect V2, as the image capturing sensor.

Use of Commercial Off-The-Shelf Devices for the Detection of Manual Gestures in Surgery: Systematic Literature Review

Background

The increasingly pervasive presence of technology in the operating room raises the need to study the interaction between the surgeon and computer system. A new generation of tools known as commercial off-the-shelf (COTS) devices enabling touchless gesture–based human-computer interaction is currently being explored as a solution in surgical environments.

Objective

The aim of this systematic literature review was to provide an account of the state of the art of COTS devices in the detection of manual gestures in surgery and to identify their use as a simulation tool for motor skills teaching in minimally invasive surgery (MIS).

Methods

For this systematic literature review, a search was conducted in PubMed, Excerpta Medica dataBASE, ScienceDirect, Espacenet, OpenGrey, and the Institute of Electrical and Electronics Engineers databases. Articles published between January 2000 and December 2017 on the use of COTS devices for gesture detection in surgical environments and in simulation for surgical skills learning in MIS were evaluated and selected.

Results

A total of 3180 studies were identified, 86 of which met the search selection criteria. Microsoft Kinect (Microsoft Corp) and the Leap Motion Controller (Leap Motion Inc) were the most widely used COTS devices. The most common intervention was image manipulation in surgical and interventional radiology environments, followed by interaction with virtual reality environments for educational or interventional purposes. The possibility of using this technology to develop portable low-cost simulators for skills learning in MIS was also examined. As most of the articles identified in this systematic review were proof-of-concept or prototype user testing and feasibility testing studies, we concluded that the field was still in the exploratory phase in areas requiring touchless manipulation within environments and settings that must adhere to asepsis and antisepsis protocols, such as angiography suites and operating rooms.

Conclusions

COTS devices applied to hand and instrument gesture–based interfaces in the field of simulation for skills learning and training in MIS could open up a promising field to achieve ubiquitous training and presurgical warm up.

Nonrigid reconstruction of 3D breast surfaces with a low-cost RGBD camera for surgical planning and aesthetic evaluation

•

A nonrigid 3D breast surface reconstruction pipeline running on a standard PC taking a noisy RGBD input video from a Kinect-style camera is proposed.

•

Pairwise nonrigid ICP is extended to the multi-view case incorporating soft mobility constraints in areas of non-overlap.

•

Shortest distance correspondences as a new technique for data association are shown to lead to consistently better alignment.

•

The method is able to reconstruct clinical-quality surface models in spite of varying degrees of postural sway during data capture.

•

Landmark and volumetric quantitative validation in metric units demonstrate improved reconstruction quality on par with the gold standard and superior to a competing method.

Accounting for 26% of all new cancer cases worldwide, breast cancer remains the most common form of cancer in women. Although early breast cancer has a favourable long-term prognosis, roughly a third of patients suffer from a suboptimal aesthetic outcome despite breast conserving cancer treatment. Clinical-quality 3D modelling of the breast surface therefore assumes an increasingly important role in advancing treatment planning, prediction and evaluation of breast cosmesis. Yet, existing 3D torso scanners are expensive and either infrastructure-heavy or subject to motion artefacts. In this paper we employ a single consumer-grade RGBD camera with an ICP-based registration approach to jointly align all points from a sequence of depth images non-rigidly. Subtle body deformation due to postural sway and respiration is successfully mitigated leading to a higher geometric accuracy through regularised locally affine transformations. We present results from 6 clinical cases where our method compares well with the gold standard and outperforms a previous approach. We show that our method produces better reconstructions qualitatively by visual assessment and quantitatively by consistently obtaining lower landmark error scores and yielding more accurate breast volume estimates.

The potential role of three-dimensional surface imaging as a tool to evaluate aesthetic outcome after Breast Conserving Therapy (BCT)

PURPOSE

To establish whether objective measurements of symmetry of volume and shape using three-dimensional surface imaging (3D-SI) can be used as surrogate markers of aesthetic outcome in patients who have undergone breast conserving therapy (BCT).

METHODS

Women who had undergone unilateral BCT in the preceding 1-6 years were invited to participate. Participants completed a satisfaction questionnaire (BREAST-Q) and underwent 3D-SI. Volume and surface symmetry were measured on the images. Assessment of aesthetic outcome was undertaken by a panel of clinicians. The Kruskal-Wallis test was used to assess the relationship between volume and shape symmetry measurements with the panel score. Spearman's rho correlations were used to assess the relationship between the measurements and patient satisfaction.

RESULTS

200 women participated. Median volume symmetry was 87% (IQR 78-93) and shape symmetry was 5.9 mm (IQR 4.2-8.0). The participants were grouped according to panel assessment of aesthetic outcome (poor, fair, good, excellent) and the median volume and shape symmetry was calculated for each group. Volume symmetry significantly differed between the groups. Post hoc pairwise comparisons demonstrated that these differences existed between panel scores of fair versus good and good versus excellent. Median shape symmetry also differed according to patient panel groups with four significant pairwise comparisons between poor versus good, poor versus excellent, fair versus good and fair versus excellent. There was a significant but weak correlation of both volume symmetry and surface asymmetry with BREAST-Q scores (correlation coefficients 0.187 and -0.229, respectively).

CONCLUSION

Breast volume and shape symmetry are both associated with panel assessment scores and patient satisfaction. The objective volume and shape symmetry measures were strongly associated with panel assessment scores, such that a 3D-SI tool could replace panel assessment as a faster and more objective method of evaluating aesthetic outcomes.

3D measurements in conventional X-ray imaging with RGB-D sensors

A method for deriving 3D internal information in conventional X-ray settings is presented. It is based on the combination of a pair of radiographs from a patient and it avoids the use of X-ray-opaque fiducials and external reference structures. To achieve this goal, we augment an ordinary X-ray device with a consumer RGB-D camera. The patient' s rotation around the craniocaudal axis is tracked relative to this camera thanks to the depth information provided and the application of a modern surface-mapping algorithm. The measured spatial information is then translated to the reference frame of the X-ray imaging system. By using the intrinsic parameters of the diagnostic equipment, epipolar geometry, and X-ray images of the patient at different angles, 3D internal positions can be obtained. Both the RGB-D and X-ray instruments are first geometrically calibrated to find their joint spatial transformation. The proposed method is applied to three rotating phantoms. The first two consist of an anthropomorphic head and a torso, which are filled with spherical lead bearings at precise locations. The third one is made of simple foam and has metal needles of several known lengths embedded in it. The results show that it is possible to resolve anatomical positions and lengths with a millimetric level of precision. With the proposed approach, internal 3D reconstructed coordinates and distances can be provided to the physician. It also contributes to reducing the invasiveness of ordinary X-ray environments and can replace other types of clinical explorations that are mainly aimed at measuring or geometrically relating elements that are present inside the patient's body.

Evaluation of Kinect 3D Sensor for Healthcare Imaging

Microsoft Kinect is a three-dimensional (3D) sensor originally designed for gaming that has received growing interest as a cost-effective and safe device for healthcare imaging. Recent applications of Kinect in health monitoring, screening, rehabilitation, assistance systems, and intervention support are reviewed here. The suitability of available technologies for healthcare imaging applications is assessed. The performance of Kinect I, based on structured light technology, is compared with that of the more recent Kinect II, which uses time-of-flight measurement, under conditions relevant to healthcare applications. The accuracy, precision, and resolution of 3D images generated with Kinect I and Kinect II are evaluated using flat cardboard models representing different skin colors (pale, medium, and dark) at distances ranging from 0.5 to 1.2 m and measurement angles of up to 75°. Both sensors demonstrated high accuracy (majority of measurements <2 mm) and precision (mean point to plane error <2 mm) at an average resolution of at least 390 points per cm². Kinect I is capable of imaging at shorter measurement distances, but Kinect II enables structures angled at over 60° to be evaluated. Kinect II showed significantly higher precision and Kinect I showed significantly higher resolution (both p < 0.001). The choice of object color can influence measurement range and precision. Although Kinect is not a medical imaging device, both sensor generations show performance adequate for a range of healthcare imaging applications. Kinect I is more appropriate for short-range imaging and Kinect II is more appropriate for imaging highly curved surfaces such as the face or breast.

Markerless three-dimensional tracking of masticatory movement

Conventional methods for measuring mandibular movement are expensive and require headgear and a marker attached to the mandibular incisors. These make assessment of normal chewing difficult. The aim of the present study was to test the validity of a markerless three-dimensional system for tracking masticatory movement by comparing it with a conventional method using an incisal marker. The study investigated 100 chewing cycles in 10 participants. The jaw tracking system consisted of a camera capable of recording depth and red, green, and blue data simultaneously, a laptop computer, and data analysis software. Depth data for each participant's face, tracked in real time, produced a computed 3D mask. The most prominent point of the soft tissue under the lip was defined as the chin point. A dental clasp cemented to the labial surface of the mandibular incisors was defined as the incisal point. The movement of these two measuring points was simultaneously recorded during mastication of chewing gum for 20s. To conduct the same analysis on each cycle from the two measuring points, all cycles were normalized by dividing by the corresponding vertical displacement because of their size variation. The findings showed excellent intramethod correlation for normalized horizontal displacement at every level (>0.9; except for 2 out of 19 levels; 0.896 and 0.898), and a lack of proportional bias. These findings suggest a correlation between the chewing cycles from two measuring points, the incisor and the chin, further suggesting the feasibility of a markerless system for tracking masticatory movement.

The Kinect Recording System for objective three- and four-dimensional breast assessment with image overlays

INTRODUCTION

We investigated the application of the validated portable Kinect camera for three- and four-dimensional breast assessment in female life models.

METHOD

Breast images from six life models were captured using the Kinect camera. Capture was conducted with taking three different arm positions while standing upright: with the arms straight down, straight up to the side at 90° and straight all the way up. Images of the volunteers were superimposed on each other. Digital linear distances between sternal notch and nipple-areola complexes were obtained and compared. The views of plastic and breast surgeons to arm positions were questioned. An example for clinical application was provided.

RESULTS

Successful capture of images of the female life breast models was achieved. Digital breast measurements at the three different arm positions revealed considerable variation in linear distances measured on the images obtained with the Kinect camera. The dynamic of breast movements could be demonstrated by image overlay and the first ever four-dimensional breast assessment was demonstrated. Fourteen plastic and breast surgeons were found to have nine different opinions regarding their favoured arm positions for breast capture. Even though precision of image sharpness still needs improvement, the images were satisfactory for clinical patient use. The Kinect data were shown to be applicable to surgery planning by designing a planar flap from the 3D mesh.

CONCLUSION

The portable and low-cost Kinect camera proved to be easy to use for the first application in life models for three- and four-dimensional breast assessment.

Assessment of a Microsoft Kinect-based 3D scanning system for taking body segment girth measurements: a comparison to ISAK and ISO standards

Use of anthropometric data to infer sporting performance is increasing in popularity, particularly within elite sport programmes. Measurement typically follows standards set by the International Society for the Advancement of Kinanthropometry (ISAK). However, such techniques are time consuming, which reduces their practicality. Schranz et al. recently suggested 3D body scanners could replace current measurement techniques; however, current systems are costly. Recent interest in natural user interaction has led to a range of low-cost depth cameras capable of producing 3D body scans, from which anthropometrics can be calculated. A scanning system comprising 4 depth cameras was used to scan 4 cylinders, representative of the body segments. Girth measurements were calculated from the 3D scans and compared to gold standard measurements. Requirements of a Level 1 ISAK practitioner were met in all 4 cylinders, and ISO standards for scan-derived girth measurements were met in the 2 larger cylinders only. A fixed measurement bias was identified that could be corrected with a simple offset factor. Further work is required to determine comparable performance across a wider range of measurements performed upon living participants. Nevertheless, findings of the study suggest such a system offers many advantages over current techniques, having a range of potential applications.