Multilevel Modeling of Gaze From Listeners With Hearing Loss Following a Realistic Conversation

PURPOSE

There is a need for tools to study real-world communication abilities in people with hearing loss. We outline a potential method for this that analyzes gaze and use it to answer the question of when and how much listeners with hearing loss look toward a new talker in a conversation.

METHOD

Twenty-two older adults with hearing loss followed a prerecorded two-person audiovisual conversation in the presence of babble noise. We compared their eye-gaze direction to the conversation in two multilevel logistic regression (MLR) analyses. First, we split the conversation into events classified by the number of active talkers within a turn or a transition, and we tested if these predicted the listener's gaze. Second, we mapped the odds that a listener gazed toward a new talker over time during a conversation transition.

RESULTS

We found no evidence that our conversation events predicted changes in the listener's gaze, but the listener's gaze toward the new talker during a silence-transition was predicted by time: The odds of looking at the new talker increased in an s-shaped curve from at least 0.4 s before to 1 s after the onset of the new talker's speech. A comparison of models with different random effects indicated that more variance was explained by differences between individual conversation events than by differences between individual listeners.

CONCLUSIONS

MLR modeling of eye-gaze during talker transitions is a promising approach to study a listener's perception of realistic conversation. Our experience provides insight to guide future research with this method.

Comparing In-ear EOG for Eye-Movement Estimation With Eye-Tracking: Accuracy, Calibration, and Speech Comprehension

This presentation details and evaluates a method for estimating the attended speaker during a two-person conversation by means of in-ear electro-oculography (EOG). Twenty-five hearing-impaired participants were fitted with molds equipped with EOG electrodes (in-ear EOG) and wore eye-tracking glasses while watching a video of two life-size people in a dialog solving a Diapix task. The dialogue was directionally presented and together with background noise in the frontal hemisphere at 60 dB SPL. During three conditions of steering (none, in-ear EOG, conventional eye-tracking), participants' comprehension was periodically measured using multiple-choice questions. Based on eye movement detection by in-ear EOG or conventional eye-tracking, the estimated attended speaker was amplified by 6 dB. In the in-ear EOG condition, the estimate was based on one selected channel pair of electrodes out of 36 possible electrodes. A novel calibration procedure introducing three different metrics was used to select the measurement channel. The in-ear EOG attended speaker estimates were compared to those of the eye-tracker. Across participants, the mean accuracy of in-ear EOG estimation of the attended speaker was 68%, ranging from 50 to 89%. Based on offline simulation, it was established that higher scoring metrics obtained for a channel with the calibration procedure were significantly associated with better data quality. Results showed a statistically significant improvement in comprehension of about 10% in both steering conditions relative to the no-steering condition. Comprehension in the two steering conditions was not significantly different. Further, better comprehension obtained under the in-ear EOG condition was significantly correlated with more accurate estimation of the attended speaker. In conclusion, this study shows promising results in the use of in-ear EOG for visual attention estimation with potential for applicability in hearing assistive devices.

Feeling the Beat: Bouncing Synchronization to Vibrotactile Music in Hearing and Early Deaf People

The ability to dance relies on the ability to synchronize movements to a perceived musical beat. Typically, beat synchronization is studied with auditory stimuli. However, in many typical social dancing situations, music can also be perceived as vibrations when objects that generate sounds also generate vibrations. This vibrotactile musical perception is of particular relevance for deaf people, who rely on non-auditory sensory information for dancing. In the present study, we investigated beat synchronization to vibrotactile electronic dance music in hearing and deaf people. We tested seven deaf and 14 hearing individuals on their ability to bounce in time with the tempo of vibrotactile stimuli (no sound) delivered through a vibrating platform. The corresponding auditory stimuli (no vibrations) were used in an additional condition in the hearing group. We collected movement data using a camera-based motion capture system and subjected it to a phase-locking analysis to assess synchronization quality. The vast majority of participants were able to precisely time their bounces to the vibrations, with no difference in performance between the two groups. In addition, we found higher performance for the auditory condition compared to the vibrotactile condition in the hearing group. Our results thus show that accurate tactile-motor synchronization in a dance-like context occurs regardless of auditory experience, though auditory-motor synchronization is of superior quality.

White matter structure in the right planum temporale region correlates with visual motion detection thresholds in deaf people

The right planum temporale region is typically involved in higher-order auditory processing. After deafness, this area reorganizes to become sensitive to visual motion. This plasticity is thought to support compensatory enhancements to visual ability. In earlier work we showed that enhanced visual motion detection abilities in early-deaf people correlate with cortical thickness in a subregion of the right planum temporale. In the current study, we build on this earlier result by examining the relationship between enhanced visual motion detection ability and white matter structure in this area in the same sample. We used diffusion-weighted magnetic resonance imaging and extracted the measures of white matter structure from a region-of-interest just below the grey matter surface where cortical thickness correlates with visual motion detection ability. We also tested control regions-of-interest in the auditory and visual cortices where we did not expect to find a relationship between visual motion detection ability and white matter. We found that in the right planum temporale subregion, and in no other tested regions, fractional anisotropy, radial diffusivity, and mean diffusivity correlated with visual motion detection thresholds. We interpret this change as further evidence of a structural correlate of cross-modal reorganization after deafness.

The Right Hemisphere Planum Temporale Supports Enhanced Visual Motion Detection Ability in Deaf People: Evidence from Cortical Thickness

After sensory loss, the deprived cortex can reorganize to process information from the remaining modalities, a phenomenon known as cross-modal reorganization. In blind people this cross-modal processing supports compensatory behavioural enhancements in the nondeprived modalities. Deaf people also show some compensatory visual enhancements, but a direct relationship between these abilities and cross-modally reorganized auditory cortex has only been established in an animal model, the congenitally deaf cat, and not in humans. Using T1-weighted magnetic resonance imaging, we measured cortical thickness in the planum temporale, Heschl's gyrus and sulcus, the middle temporal area MT+, and the calcarine sulcus, in early-deaf persons. We tested for a correlation between this measure and visual motion detection thresholds, a visual function where deaf people show enhancements as compared to hearing. We found that the cortical thickness of a region in the right hemisphere planum temporale, typically an auditory region, was greater in deaf individuals with better visual motion detection thresholds. This same region has previously been implicated in functional imaging studies as important for functional reorganization. The structure-behaviour correlation observed here demonstrates this area's involvement in compensatory vision and indicates an anatomical correlate, increased cortical thickness, of cross-modal plasticity.

Reorganization of auditory cortex in early-deaf people: functional connectivity and relationship to hearing aid use

Cross-modal reorganization after sensory deprivation is a model for understanding brain plasticity. Although it is a well-documented phenomenon, we still know little of the mechanisms underlying it or the factors that constrain and promote it. Using fMRI, we identified visual motion-related activity in 17 early-deaf and 17 hearing adults. We found that, in the deaf, the posterior superior temporal gyrus (STG) was responsive to visual motion. We compared functional connectivity of this reorganized cortex between groups to identify differences in functional networks associated with reorganization. In the deaf more than the hearing, the STG displayed increased functional connectivity with a region in the calcarine fissure. We also explored the role of hearing aid use, a factor that may contribute to variability in cross-modal reorganization. We found that both the cross-modal activity in STG and the functional connectivity between STG and calcarine cortex correlated with duration of hearing aid use, supporting the hypothesis that residual hearing affects cross-modal reorganization. We conclude that early auditory deprivation alters not only the organization of auditory regions but also the interactions between auditory and primary visual cortex and that auditory input, as indexed by hearing aid use, may inhibit cross-modal reorganization in early-deaf people.

Enhancement of visual motion detection thresholds in early deaf people

In deaf people, the auditory cortex can reorganize to support visual motion processing. Although this cross-modal reorganization has long been thought to subserve enhanced visual abilities, previous research has been unsuccessful at identifying behavioural enhancements specific to motion processing. Recently, research with congenitally deaf cats has uncovered an enhancement for visual motion detection. Our goal was to test for a similar difference between deaf and hearing people. We tested 16 early and profoundly deaf participants and 20 hearing controls. Participants completed a visual motion detection task, in which they were asked to determine which of two sinusoidal gratings was moving. The speed of the moving grating varied according to an adaptive staircase procedure, allowing us to determine the lowest speed necessary for participants to detect motion. Consistent with previous research in deaf cats, the deaf group had lower motion detection thresholds than the hearing. This finding supports the proposal that cross-modal reorganization after sensory deprivation will occur for supramodal sensory features and preserve the output functions.