Laryngeal Image Processing of Vocal Folds Motion

Deep Learning-Based Detection of Glottis Segmentation Failures

Medical image segmentation is crucial for clinical applications, but challenges persist due to noise and variability. In particular, accurate glottis segmentation from high-speed videos is vital for voice research and diagnostics. Manual searching for failed segmentations is labor-intensive, prompting interest in automated methods. This paper proposes the first deep learning approach for detecting faulty glottis segmentations. For this purpose, faulty segmentations are generated by applying both a poorly performing neural network and perturbation procedures to three public datasets. Heavy data augmentations are added to the input until the neural network's performance decreases to the desired mean intersection over union (IoU). Likewise, the perturbation procedure involves a series of image transformations to the original ground truth segmentations in a randomized manner. These data are then used to train a ResNet18 neural network with custom loss functions to predict the IoU scores of faulty segmentations. This value is then thresholded with a fixed IoU of 0.6 for classification, thereby achieving 88.27% classification accuracy with 91.54% specificity. Experimental results demonstrate the effectiveness of the presented approach. Contributions include: (i) a knowledge-driven perturbation procedure, (ii) a deep learning framework for scoring and detecting faulty glottis segmentations, and (iii) an evaluation of custom loss functions.

A Generative Method for a Laryngeal Biosignal

The Glottal Area Waveform (GAW) is an important component in quantitative clinical voice assessment, providing valuable insights into vocal fold function. In this study, we introduce a novel method employing Variational Autoencoders (VAEs) to generate synthetic GAWs. Our approach enables the creation of synthetic GAWs that closely replicate real-world data, offering a versatile tool for researchers and clinicians. We elucidate the process of manipulating the VAE latent space using the Glottal Opening Vector (GlOVe). The GlOVe allows precise control over the synthetic closure and opening of the vocal folds. By utilizing the GlOVe, we generate synthetic laryngeal biosignals. These biosignals accurately reflect vocal fold behavior, allowing for the emulation of realistic glottal opening changes. This manipulation extends to the introduction of arbitrary oscillations in the vocal folds, closely resembling real vocal fold oscillations. The range of factor coefficient values enables the generation of diverse biosignals with varying frequencies and amplitudes. Our results demonstrate that this approach yields highly accurate laryngeal biosignals, with the Normalized Mean Absolute Error values for various frequencies ranging from 9.6 ⋅ 10-3 to 1.20 ⋅ 10-2 for different experimented frequencies, alongside a remarkable training effectiveness, reflected in reductions of up to approximately 89.52% in key loss components. This proposed method may have implications for downstream speech synthesis and phonetics research, offering the potential for advanced and natural-sounding speech technologies.

High-Speed Videoendoscopy Enhances the Objective Assessment of Glottic Organic Lesions: A Case-Control Study with Multivariable Data-Mining Model Development

The aim of the study was to utilize a quantitative assessment of the vibratory characteristics of vocal folds in diagnosing benign and malignant lesions of the glottis using high-speed videolaryngoscopy (HSV).

METHODS

Case-control study including 100 patients with unilateral vocal fold lesions in comparison to 38 normophonic subjects. Quantitative assessment with the determination of vocal fold oscillation parameters was performed based on HSV kymography. Machine-learning predictive models were developed and validated.

RESULTS

All calculated parameters differed significantly between healthy subjects and patients with organic lesions. The first predictive model distinguishing any organic lesion patients from healthy subjects reached an area under the curve (AUC) equal to 0.983 and presented with 89.3% accuracy, 97.0% sensitivity, and 71.4% specificity on the testing set. The second model identifying malignancy among organic lesions reached an AUC equal to 0.85 and presented with 80.6% accuracy, 100% sensitivity, and 71.1% specificity on the training set. Important predictive factors for the models were frequency perturbation measures.

CONCLUSIONS

The standard protocol for distinguishing between benign and malignant lesions continues to be clinical evaluation by an experienced ENT specialist and confirmed by histopathological examination. Our findings did suggest that advanced machine learning models, which consider the complex interactions present in HSV data, could potentially indicate a heightened risk of malignancy. Therefore, this technology could prove pivotal in aiding in early cancer detection, thereby emphasizing the need for further investigation and validation.

Classifying Vocal Folds Fixation from Endoscopic Videos with Machine Learning

Vocal folds motility evaluation is paramount in both the assessment of functional deficits and in the accurate staging of neoplastic disease of the glottis. Diagnostic endoscopy, and in particular videoendoscopy, is nowadays the method through which the motility is estimated. The clinical diagnosis, however, relies on the examination of the videoendoscopic frames, which is a subjective and professional-dependent task. Hence, a more rigorous, objective, reliable, and repeatable method is needed. To support clinicians, this paper proposes a machine learning (ML) approach for vocal cords motility classification. From the endoscopic videos of 186 patients with both vocal cords preserved motility and fixation, a dataset of 558 images relative to the two classes was extracted. Successively, a number of features was retrieved from the images and used to train and test four well-grounded ML classifiers. From test results, the best performance was achieved using XGBoost, with precision = 0.82, recall = 0.82, F1 score = 0.82, and accuracy = 0.82. After comparing the most relevant ML models, we believe that this approach could provide precise and reliable support to clinical evaluation.Clinical Relevance- This research represents an important advancement in the state-of-the-art of computer-assisted otolaryngology, to develop an effective tool for motility assessment in the clinical practice.

Optical Flow Glottovibrogram for the examination of vocal fold pathology

Laryngeal high-speed video endoscopy is performed to examine the cycles of vocal fold vibrations in detail and to diagnose voice abnormalities. One of the recent image processing techniques for visualizing vocal fold vibration is optical flow-based playbacks, which include optical flow kymograms (OFKG) for local dynamics, optical flow glottovibrogram (OFGVG) and glottal optical flow waveforms (GOFW) for global dynamics. In recent times, various optical flow computing algorithms have been developed. In this paper, we used four well-known optical flow algorithms Horn Schunk, Lucas Kanade, Gunnar Farneback, and TVL1 to construct the optical flow playbacks. The proposed playback reliability is examined by comparing them to traditional representations such as Phonovibrogram (PVG). Since PVG and OFGVG are interconnected, a comparison study was carried out to better comprehend their interaction.Clinical Relevance- Both OFGVG and PVG add to the precision of interpreting pathological conditions by offering complementary information to the conventional spatiotemporal representations.

GlottisNetV2: Temporal Glottal Midline Detection Using Deep Convolutional Neural Networks

High-speed videoendoscopy is a major tool for quantitative laryngology. Glottis segmentation and glottal midline detection are crucial for computing vocal fold-specific, quantitative parameters. However, fully automated solutions show limited clinical applicability. Especially unbiased glottal midline detection remains a challenging problem. We developed a multitask deep neural network for glottis segmentation and glottal midline detection. We used techniques from pose estimation to estimate the anterior and posterior points in endoscopy images. Neural networks were set up in TensorFlow/Keras and trained and evaluated with the BAGLS dataset. We found that a dual decoder deep neural network termed GlottisNetV2 outperforms the previously proposed GlottisNet in terms of MAPE on the test dataset (1.85% to 6.3%) while converging faster. Using various hyperparameter tunings, we allow fast and directed training. Using temporal variant data on an additional data set designed for this task, we can improve the median prediction accuracy from 2.1% to 1.76% when using 12 consecutive frames and additional temporal filtering. We found that temporal glottal midline detection using a dual decoder architecture together with keypoint estimation allows accurate midline prediction. We show that our proposed architecture allows stable and reliable glottal midline predictions ready for clinical use and analysis of symmetry measures.

Computational Analysis of the Droplet-Stimulated Laryngeal Adductor Reflex in High-Speed Sequences

OBJECTIVES/HYPOTHESIS

The laryngeal adductor reflex (LAR) is an important protective mechanism of the airways. Its physiology is still not completely understood. The available methods for LAR evaluation offer limited reproducibility and/or rely on subjective interpretation. A new approach, termed Microdroplet Impulse Testing of the LAR (MIT-LAR), was recently introduced. Here, the LAR is elicited by a droplet and a laryngoscopic high-speed recording is acquired simultaneously. In the present work, image-processing algorithms for autonomous MIT-LAR sequence analysis were developed. This allowed the automated approximation of kinematic LAR parameters in humans.

STUDY DESIGN

Development and testing of computational methods.

METHODS

Computational image processing enabled the autonomous estimation of the glottal area, the glottal angle, and the vocal fold edge distance in MIT-LAR sequences. A suitable analytical representation of these glottal parameters allowed the extraction of seven relevant LAR parameters. The obtained values were compared to the literature.

RESULTS

A generalized logistic function showed the highest average goodness of fit among four different analytical approaches for each of the glottal parameters. Autonomous sequence analysis yielded bilateral LAR response latencies of (229 ± 116) ms and (182 ± 60) ms for cases of complete and incomplete glottal closure, respectively. The initial/average/maximum angular vocal fold adduction velocity was estimated at (157 ± 115) °s^-1 /(891 ± 516) °s^-1 /(929 ±  583) °s^-1 and (88 ± 53) °s^-1 /(421 ± 221) °s^-1 /(520 ± 238) °s^-1 for complete and incomplete glottal closure, respectively.

CONCLUSION

The automated extraction of LAR parameters from laryngoscopic high-speed sequences can potentially increase the objectiveness of optical LAR characterization and reduce the associated workload. The proposed methods may thus be helpful for future research on this vital reflex.

LEVEL OF EVIDENCE

NA Laryngoscope, 132:2412-2419, 2022.

Assessment of Vocal Fold Stiffness by Means of High-Speed Videolaryngoscopy with Laryngotopography in Prediction of Early Glottic Malignancy: Preliminary Report

Simple Summary

The method described in our manuscript can help to objectively assess the vibration of each vocal fold using larygotopographic analysis of high-speed videoendoscopy (HSV) recordings. We have developed image processing and analysis procedures to detect vocal fold regions in HSV films and quantitatively analyze their shape and kinematics. We proposed the term Stiffness Asymmetry Index which can provide valuable information on the texture and kinematic properties of individual vocal fold tissues, which can be important in the diagnosis of early glottis cancer. Our study showed that a low value of SAI indicated large, non-vibrating vocal fold areas, characteristic of infiltrative lesions such as invasive carcinoma. This important clinical information can help to assess the depth of vocal fold invasion before direct histologic examination and discriminate benign from malignant lesions.

Abstract

One of the most important challenges in laryngological practice is the early diagnosis of laryngeal cancer. Detection of non-vibrating areas affected by neoplastic lesions of the vocal folds can be crucial in the recognition of early cancerogenous infiltration. Glottal pathologies associated with abnormal vibration patterns of the vocal folds can be detected and quantified using High-speed Videolaryngoscopy (HSV), also in subjects with severe voice disorders, and analyzed with the aid of computer image processing procedures. We present a method that enables the assessment of vocal fold pathologies with the use of HSV. The calculated laryngotopographic (LTG) maps of the vocal folds based on HSV allowed for a detailed characterization of vibration patterns and abnormalities in different regions of the vocal folds. We verified our methods with HSV recordings from 31 subjects with a normophonic voice and benign and malignant vocal fold lesions. We proposed the novel Stiffness Asymmetry Index (SAI) to differentiate between early glottis cancer (SAI = 0.65 ± 0.18) and benign vocal fold masses (SAI = 0.16 ± 0.13). Our results showed that these glottal pathologies might be noninvasively distinguished prior to histopathological examination. However, this needs to be confirmed by further research on larger groups of benign and malignant laryngeal lesions.

Long-term performance assessment of fully automatic biomedical glottis segmentation at the point of care

Deep Learning has a large impact on medical image analysis and lately has been adopted for clinical use at the point of care. However, there is only a small number of reports of long-term studies that show the performance of deep neural networks (DNNs) in such an environment. In this study, we measured the long-term performance of a clinically optimized DNN for laryngeal glottis segmentation. We have collected the video footage for two years from an AI-powered laryngeal high-speed videoendoscopy imaging system and found that the footage image quality is stable across time. Next, we determined the DNN segmentation performance on lossy and lossless compressed data revealing that only 9% of recordings contain segmentation artifacts. We found that lossy and lossless compression is on par for glottis segmentation, however, lossless compression provides significantly superior image quality. Lastly, we employed continual learning strategies to continuously incorporate new data into the DNN to remove the aforementioned segmentation artifacts. With modest manual intervention, we were able to largely alleviate these segmentation artifacts by up to 81%. We believe that our suggested deep learning-enhanced laryngeal imaging platform consistently provides clinically sound results, and together with our proposed continual learning scheme will have a long-lasting impact on the future of laryngeal imaging.

A single latent channel is sufficient for biomedical glottis segmentation

Glottis segmentation is a crucial step to quantify endoscopic footage in laryngeal high-speed videoendoscopy. Recent advances in deep neural networks for glottis segmentation allow for a fully automatic workflow. However, exact knowledge of integral parts of these deep segmentation networks remains unknown, and understanding the inner workings is crucial for acceptance in clinical practice. Here, we show that a single latent channel as a bottleneck layer is sufficient for glottal area segmentation using systematic ablations. We further demonstrate that the latent space is an abstraction of the glottal area segmentation relying on three spatially defined pixel subtypes allowing for a transparent interpretation. We further provide evidence that the latent space is highly correlated with the glottal area waveform, can be encoded with four bits, and decoded using lean decoders while maintaining a high reconstruction accuracy. Our findings suggest that glottis segmentation is a task that can be highly optimized to gain very efficient and explainable deep neural networks, important for application in the clinic. In the future, we believe that online deep learning-assisted monitoring is a game-changer in laryngeal examinations.

Segmentation of Glottal Images from High-Speed Videoendoscopy Optimized by Synchronous Acoustic Recordings

Laryngeal high-speed videoendoscopy (LHSV) is an imaging technique offering novel visualization quality of the vibratory activity of the vocal folds. However, in most image analysis methods, the interaction of the medical personnel and access to ground truth annotations are required to achieve accurate detection of vocal folds edges. In our fully automatic method, we combine video and acoustic data that are synchronously recorded during the laryngeal endoscopy. We show that the image segmentation algorithm of the glottal area can be optimized by matching the Fourier spectra of the pre-processed video and the spectra of the acoustic recording during the phonation of sustained vowel /i:/. We verify our method on a set of LHSV recordings taken from subjects with normophonic voice and patients with voice disorders due to glottal insufficiency. We show that the computed geometric indices of the glottal area make it possible to discriminate between normal and pathologic voices. The median of the Open Quotient and Minimal Relative Glottal Area values for healthy subjects were 0.69 and 0.06, respectively, while for dysphonic subjects were 1 and 0.35, respectively. We also validate these results using independent phoniatrician experts.

OpenHSV: an open platform for laryngeal high-speed videoendoscopy

High-speed videoendoscopy is an important tool to study laryngeal dynamics, to quantify vocal fold oscillations, to diagnose voice impairments at laryngeal level and to monitor treatment progress. However, there is a significant lack of an open source, expandable research tool that features latest hardware and data analysis. In this work, we propose an open research platform termed OpenHSV that is based on state-of-the-art, commercially available equipment and features a fully automatic data analysis pipeline. A publicly available, user-friendly graphical user interface implemented in Python is used to interface the hardware. Video and audio data are recorded in synchrony and are subsequently fully automatically analyzed. Video segmentation of the glottal area is performed using efficient deep neural networks to derive glottal area waveform and glottal midline. Established quantitative, clinically relevant video and audio parameters were implemented and computed. In a preliminary clinical study, we recorded video and audio data from 28 healthy subjects. Analyzing these data in terms of image quality and derived quantitative parameters, we show the applicability, performance and usefulness of OpenHSV. Therefore, OpenHSV provides a valid, standardized access to high-speed videoendoscopy data acquisition and analysis for voice scientists, highlighting its use as a valuable research tool in understanding voice physiology. We envision that OpenHSV serves as basis for the next generation of clinical HSV systems.

A Deep Learning Enhanced Novel Software Tool for Laryngeal Dynamics Analysis

Purpose High-speed videoendoscopy (HSV) is an emerging, but barely used, endoscopy technique in the clinic to assess and diagnose voice disorders because of the lack of dedicated software to analyze the data. HSV allows to quantify the vocal fold oscillations by segmenting the glottal area. This challenging task has been tackled by various studies; however, the proposed approaches are mostly limited and not suitable for daily clinical routine. Method We developed a user-friendly software in C# that allows the editing, motion correction, segmentation, and quantitative analysis of HSV data. We further provide pretrained deep neural networks for fully automatic glottis segmentation. Results We freely provide our software Glottis Analysis Tools (GAT). Using GAT, we provide a general threshold-based region growing platform that enables the user to analyze data from various sources, such as in vivo recordings, ex vivo recordings, and high-speed footage of artificial vocal folds. Additionally, especially for in vivo recordings, we provide three robust neural networks at various speed and quality settings to allow a fully automatic glottis segmentation needed for application by untrained personnel. GAT further evaluates video and audio data in parallel and is able to extract various features from the video data, among others the glottal area waveform, that is, the changing glottal area over time. In total, GAT provides 79 unique quantitative analysis parameters for video- and audio-based signals. Many of these parameters have already been shown to reflect voice disorders, highlighting the clinical importance and usefulness of the GAT software. Conclusion GAT is a unique tool to process HSV and audio data to determine quantitative, clinically relevant parameters for research, diagnosis, and treatment of laryngeal disorders. Supplemental Material https://doi.org/10.23641/asha.14575533.

Description of the Features and Vibratory Behaviors of the Nyquist Plot Analyzed From Laryngeal High-Speed Videoendoscopy Images

Facilitative playback-based subjective measures offer a more reliable evaluation of the vocal fold vibration than those derived from direct inspection of video playback. One of the measures is a Nyquist plot, which presents the analyzed cycle-to-cycle vibratory information in a graphical form. While the potential is evident, the information of the features of the Nyquist plot, which the evaluation is based on, is still incomplete. The current identified features and their vibratory behaviors may be inadequate to guarantee accurate interpretation of the findings. The present study aims to address this issue by examining the features of the Nyquist plot and their vibratory behaviors. A total of 56 young normophonic speakers, that is, 20 males and 36 females were recruited as the participants. Each of them underwent laryngeal high-speed videoendoscopy to record the images of the vocal fold vibration, which were then analyzed to generate the Nyquist plots. The features were identified by inspecting the properties of the plot points forming the Nyquist plots. For each identified feature, its vibratory behaviors were examined. The results revealed four features: rim contour depicting the longitudinal phase difference; left edge shape signifying the glottal configuration, phase closure, and closed phase duration; rim width and rim pattern visualizing the regularity of glottal areas and the regularity of the intracycle variations, respectively. The findings present a more complete reference of the features and their vibratory behaviors that is pertinent for the Nyquist plot interpretation.