Reconstruction of normal sounding speech for laryngectomy patients through a modified CELP codec

LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading

The aim of this work is to investigate the impact of crossmodal self-supervised pre-training for speech reconstruction (video-to-audio) by leveraging the natural co-occurrence of audio and visual streams in videos. We propose LipSound2 that consists of an encoder-decoder architecture and location-aware attention mechanism to map face image sequences to mel-scale spectrograms directly without requiring any human annotations. The proposed LipSound2 model is first pre-trained on  ∼ 2400 -h multilingual (e.g., English and German) audio-visual data (VoxCeleb2). To verify the generalizability of the proposed method, we then fine-tune the pre-trained model on domain-specific datasets (GRID and TCD-TIMIT) for English speech reconstruction and achieve a significant improvement on speech quality and intelligibility compared to previous approaches in speaker-dependent and speaker-independent settings. In addition to English, we conduct Chinese speech reconstruction on the Chinese Mandarin Lip Reading (CMLR) dataset to verify the impact on transferability. Finally, we train the cascaded lip reading (video-to-text) system by fine-tuning the generated audios on a pre-trained speech recognition system and achieve the state-of-the-art performance on both English and Chinese benchmark datasets.

[Intraoral voice recording-towards a new smartphone-based method for vocal rehabilitation. German version]

After laryngectomy, a new voice is needed. We present the first steps in the development of a smartphone-based method. A microphone is placed in the mouth to record the pseudo-whispering voice of laryngectomized patients. This recording is analyzed by voice recognition software followed by voice synthesis. Eventually, this will be performed on a smartphone. We placed a microphone at 10 different places inside and outside the mouth (two in front of the mouth (at 2 and 20 cm), five on the palate and three on the lower jaw) and made voice recordings in eight healthy men. These recordings were analyzed by voice recognition software. The text generated by the software was compared with the original text. Over all positions, the correct detection of words recorded in the mouth was 19.3% vs. 75.2% (p = 0.01) outside the mouth. In the mouth, recording taken on the maxilla (22.8%) was much better than on the mandible (13.5%) (p = 0.01). The optimum position for a microphone on the maxilla was at the highest point of the palate with 31.9% correct word identification there (p = 0.028). Further investigations have to be undertaken with forthcoming development of smartphone processing power and with development of a smartphone-based voice recognition application.

Intraoral voice recording-towards a new smartphone-based method for vocal rehabilitation

After laryngectomy, a new voice is needed. We present the first steps in the development of a smartphone-based method. A microphone is placed in the mouth to record the pseudo-whispering voice of laryngectomized patients. This recording is analyzed by voice recognition software followed by voice synthesis. Eventually, this will be performed on a smartphone. We placed a microphone at 10 different places inside and outside the mouth (two in front of the mouth (at 2 and 20 cm), five on the palate and three on the lower jaw) and made voice recordings in eight healthy men. These recordings were analyzed by voice recognition software. The text generated by the software was compared with the original text. Over all positions, the correct detection of words recorded in the mouth was 19.3% vs. 75.2% (p = 0.01) outside the mouth. In the mouth, recording taken on the maxilla (22.8%) was much better than on the mandible (13.5%) (p = 0.01). The optimum position for a microphone on the maxilla was at the highest point of the palate with 31.9% correct word identification there (p = 0.028). Further investigations have to be undertaken with forthcoming development of smartphone processing power and with development of a smartphone-based voice recognition application.

Reconstruction of articulatory movements during neutral speech from those during whispered speech

A transformation function (TF) that reconstructs neutral speech articulatory trajectories (NATs) from whispered speech articulatory trajectories (WATs) is investigated, such that the dynamic time warped (DTW) distance between the transformed whispered and the original neutral articulatory movements is minimized. Three candidate TFs are considered: an affine function with a diagonal matrix ( A_d) which reconstructs one NAT from the corresponding WAT, an affine function with a full matrix ( A_f) and a deep neural network (DNN) based nonlinear function which reconstruct each NAT from all WATs. Experiments reveal that the transformation could be approximated well by A_f, since it generalizes better across subjects and achieves the least DTW distance of 5.20 (±1.27) mm (on average), with an improvement of 7.47%, 4.76%, and 7.64% (relative) compared to that with A_d, DNN, and the best baseline scheme, respectively. Further analysis to understand the differences in neutral and whispered articulation reveals that the whispered articulators exhibit exaggerated movements in order to reconstruct the lip movements during neutral speech. It is also observed that among the articulators considered in the study, the tongue exhibits a higher precision and stability while whispering, implying that subjects control their tongue movements carefully in order to render an intelligible whispered speech.

Speech reconstruction using a deep partially supervised neural network

Statistical speech reconstruction for larynx-related dysphonia has achieved good performance using Gaussian mixture models and, more recently, restricted Boltzmann machine arrays; however, deep neural network (DNN)-based systems have been hampered by the limited amount of training data available from individual voice-loss patients. The authors propose a novel DNN structure that allows a partially supervised training approach on spectral features from smaller data sets, yielding very good results compared with the current state-of-the-art.

A preliminary study on improving the recognition of esophageal speech using a hybrid system based on statistical voice conversion

In this paper, we propose a hybrid system based on a modified statistical GMM voice conversion algorithm for improving the recognition of esophageal speech. This hybrid system aims to compensate for the distorted information present in the esophageal acoustic features by using a voice conversion method. The esophageal speech is converted into a “target” laryngeal speech using an iterative statistical estimation of a transformation function. We did not apply a speech synthesizer for reconstructing the converted speech signal, given that the converted Mel cepstral vectors are used directly as input of our speech recognition system. Furthermore the feature vectors are linearly transformed by the HLDA (heteroscedastic linear discriminant analysis) method to reduce their size in a smaller space having good discriminative properties. The experimental results demonstrate that our proposed system provides an improvement of the phone recognition accuracy with an absolute increase of 3.40 % when compared with the phone recognition accuracy obtained with neither HLDA nor voice conversion.

Reconstruction of Phonated Speech from Whispers Using Formant-Derived Plausible Pitch Modulation

Whispering is a natural, unphonated, secondary aspect of speech communications for most people. However, it is the primary mechanism of communications for some speakers who have impaired voice production mechanisms, such as partial laryngectomees, as well as for those prescribed voice rest, which often follows surgery or damage to the larynx. Unlike most people, who choose when to whisper and when not to, these speakers may have little choice but to rely on whispers for much of their daily vocal interaction.

Even though most speakers will whisper at times, and some speakers can only whisper, the majority of today’s computational speech technology systems assume or require phonated speech. This article considers conversion of whispers into natural-sounding phonated speech as a noninvasive prosthetic aid for people with voice impairments who can only whisper. As a by-product, the technique is also useful for unimpaired speakers who choose to whisper.

Speech reconstruction systems can be classified into those requiring training and those that do not. Among the latter, a recent parametric reconstruction framework is explored and then enhanced through a refined estimation of plausible pitch from weighted formant differences. The improved reconstruction framework, with proposed formant-derived artificial pitch modulation, is validated through subjective and objective comparison tests alongside state-of-the-art alternatives.