Teaching lip-reading: the efficacy of lessons on video

Design, Manufacture, and Acceptance Evaluation of APO: A Lip-syncing Social Robot Developed for Lip-reading Training Programs

Lack of educational facilities for the burgeoning world population, financial barriers, and the growing tendency in favor of inclusive education have all helped channel a general inclination toward using various educational assistive technologies, e.g., socially assistive robots. Employing social robots in diverse educational scenarios could enhance learners' achievements by motivating them and sustaining their level of engagement. This study is devoted to manufacturing and investigating the acceptance of a novel social robot named APO, designed to improve hearing-impaired individuals' lip-reading skills through an educational game. To accomplish the robot's objective, we proposed and implemented a lip-syncing system on the APO social robot. The proposed robot's potential with regard to its primary goals, tutoring and practicing lip-reading, was examined through two main experiments. The first experiment was dedicated to evaluating the clarity of the utterances articulated by the robot. The evaluation was quantified by comparing the robot's articulation of words with a video of a human teacher lip-syncing the same words. In this inspection, due to the adults' advanced skill in lip-reading compared to children, twenty-one adult participants were asked to identify the words lip-synced in the two scenarios (the articulation of the robot and the video recorded from the human teacher). Subsequently, the number of words that participants correctly recognized from the robot and the human teacher articulations was considered a metric to evaluate the caliber of the designed lip-syncing system. The outcome of this experiment revealed that no significant differences were observed between the participants' recognition of the robot and the human tutor's articulation of multisyllabic words. Following the validation of the proposed articulatory system, the acceptance of the robot by a group of hearing-impaired participants, eighteen adults and sixteen children, was scrutinized in the second experiment. The adults and the children were asked to fill in two standard questionnaires, UTAUT and SAM, respectively. Our findings revealed that the robot acquired higher scores than the lip-syncing video in most of the questionnaires' items, which could be interpreted as a greater intention of utilizing the APO robot as an assistive technology for lip-reading instruction among adults and children.

Deaf-Accessible Parenting Classes: Insights from Deaf Parents in North Wales

Parenting support services and programs develop and strengthen existing parenting skills. However, in the UK and despite the 2010 UK Equality Act’s provisions, these programs are generally not accessible for Deaf parents whose first and/or preferred language is British Sign Language (BSL) because the medium of instruction is typically spoken and written English. This small-scale qualitative interview study gauged North Walian Deaf parents’ needs and preferences for accessing parenting classes. A structured interview assessed a small group of North Walian Deaf parents’ language practices, their perceptions of parenting support and accessibility, and their needs and preferences when it comes to parenting classes. An additional case study of a Deaf parent’s experience of participating in an 11-week-long parenting course with an English-BSL interpreter provides further insight into how such classes can be made accessible to Deaf parents. The main interview findings were that the participants had substantially lower English skills than BSL skills, that face-to-face delivery was preferred over online BSL support, and that all materials should be made available in BSL. The case study further uncovered several small adjustments that should be made to face-to-face classes to make them accessible to Deaf parents. In conclusion, materials from already existing parenting classes should be translated into BSL, interpreters should be available, and small adjustments to face-to-face classes should be made, so that Deaf parents can access and participate in already existing parenting programs.

The effect of speech training experiences on speechreading skills of Chinese children with hearing impairment

Speechreading contributes significantly to effective communication, and persons with hearing impairment (HI) may need to rely more on speechreading. However, whether they may benefit from training/practice to improve their speechreading performance remains unclear. In this study, we examine the effect of speech training on speechreading performance of children with HI in China, and how such effect, if any, may be influenced by age. Fifty-nine HI children with speech training experiences, and fifty-eight HI children without speech training experiences completed tasks measuring their ability to speechread Chinese at the linguistic levels of words, phrases, and sentences. The children ranged from 7 to 14 years old, and were divided into four age groups by two-year age intervals. Both accuracy rate and response time data were collected. Results revealed three findings: (1) HI children with speech training experiences speechread more accurately but more slowly than those without speech training experiences; (2) while speechreading performance generally improved with age, age didn't alter the relative performance between the two groups; and (3) speechreading performance was best for phrases, and worst for sentences. These findings suggest that HI children benefit from speech training to improve their speechreading performance, and their speechreading performance is influenced by age, linguistic level, and the specific measure used.

Auditory-visual speech perception and aging

OBJECTIVE

This experiment was designed to assess the integration of auditory and visual information for speech perception in older adults. The integration of place and voicing information was assessed across modalities using the McGurk effect. The following questions were addressed: 1) Are older adults as successful as younger adults at integrating auditory and visual information for speech perception? 2) Is successful integration of this information related to lipreading performance?

DESIGN

The performance of three groups of participants was compared: young adults with normal hearing and vision, older adults with normal to near-normal hearing and vision, and young controls, whose hearing thresholds were shifted with noise to match the older adults. Each participant completed a lipreading test and auditory and auditory-plus-visual identification of syllables with conflicting auditory and visual cues.

RESULTS

The results show that on average older adults are as successful as young adults at integrating auditory and visual information for speech perception at the syllable level. The number of fused responses did not differ for the CV tokens across the ages tested. Although there were no significant differences between groups for integration at the syllable level, there were differences in the response alternatives chosen. Young adults with normal peripheral sensitivity often chose an auditory alternative whereas, older adults and control participants leaned toward visual alternatives. In additions, older adults demonstrated poorer lipreading performance than their younger counterparts. This was not related to successful integration of information at the syllable level.

CONCLUSIONS

Based on the findings of this study, when auditory and visual integration of speech information fails to occur, producing a nonfused response, participants select an alternative response from the modality with the least ambiguous signal.

Lipreading and audio-visual speech perception

This paper reviews progress in understanding the psychology of lipreading and audio-visual speech perception. It considers four questions. What distinguishes better from poorer lipreaders? What are the effects of introducing a delay between the acoustical and optical speech signals? What have attempts to produce computer animations of talking faces contributed to our understanding of the visual cues that distinguish consonants and vowels? Finally, how should the process of audio-visual integration in speech perception be described; that is, how are the sights and sounds of talking faces represented at their conflux?

Choosing talkers for the BKB/A Speechreading Test: a procedure with observations on talker age and gender

A procedure is described for choosing talkers for the BKB/A (BKB/Australian version) Speechreading Test. The main aims were: to select several talkers from a pool of potential talkers, to avoid adventitiously choosing a markedly atypical single talker; to assess speechreading as a general skill rather than as talker-specific; and to select talkers who were acceptable to speechreaders, relatively easy to speechread, and comparable in their speechreadability. Because of the number of variables involved and the demanding nature of the task for speechreaders, a three-stage selection procedure was adopted. In the resulting BKB/A 21-sentence list Speechreading Test, four of the 16 sentences in a list are each spoken by four talkers, chosen as follows. In Stage 1, 16 talkers (four of each age/gender set: older men, older women, younger men, younger women) were selected from an original pool of 40 (10 of each set), via rankings made by eight hearing-impaired judges with speechreading experience. In Stage 2, the final four talkers (one of each set) were selected from the 16 via the speechreading scores of further hearing-impaired subjects with speechreading experience. In Stage 3, the order of talker appearance within lists (in random order versus over blocks of four consecutive sentences) was determined. This three-stage approach to talker selection identified differences between talker candidates within sets, except for younger men, and suggested that, overall, younger women were the easiest to speechread. The discussion addresses the merits and disadvantages of this approach to talker selection, and suggests some reasons for the documented differences in speechreadability among talkers of different age and gender.

Speechreading instruction for hard-of-hearing adults. Effects of training face-to-face and with a video programme

The effects of speechreading instruction were studied using a structurally planned training programme in two types of situations: in a face-to-face situation and by using videotapes viewed at home. Instruction was given only visually. Three speechreading tests were selected to monitor the efficacy of training: CID sentences. Helen questions and the tracking method. Two of the tests (CID and Helen) showed significant improvement in all three groups training speechreading. The tracking method, however, did not prove to be a reliable method for assessing the performance of a large group of subjects. The performances of the face-to-face groups and the video group did not differ significantly. The present study showed that training speechreading is advantageous and confirmed the effectiveness of tutored self-instruction at home.

Long-term training, transfer, and retention in learning to lipread

A long-term training paradigm in lipreading was used to test the fuzzy logical model of perception (FLMP). This model has been used successfully to describe the joint contribution of audible and visible speech in bimodal speech perception. Tests of the model were extended in the present experiment to include the prediction of confusion matrices, as well as performance at several different levels of skill. The predictions of the FLMP were contrasted with the predictions of a prelabeling integration model (PRLM). Subjects were taught to lipread 22 initial consonants in three different vowel contexts. Training involved a variety of discrimination and identification lessons with the consonant-vowel syllables. Repeated testing was given on syllables, words, and sentences. The test items were presented visually, auditorily, and bimodally, at normal rate or three times normal rate. The subjects improved in their lipreading ability across all three types of test items. Replicating previous results, the present study illustrates that substantial gains in lipreading performance are possible. Relative to the PRLM, the FLMP gave a better description of the confusion matrices at both the beginning and the end of practice. One new finding from the present study is that the FLMP can account for the gains in bimodal speech perception as subjects improve their lipreading and listening abilities.