3Mo: A Model for Music-Based Biofeedback

Perceptual coupling in human dyads: Kinematics does not affect interpersonal synchronization

The minimal, essential condition for individuals to interact is that they exchange information via at least one sensory channel. Once informational coupling is established, it enables basic forms of coordinated behavior to spontaneously emerge from the interaction. Our previous study revealed different coordination dynamics in dyads engaged in a joint finger-tapping task based on visual versus auditory coupling. This observation led us to propose the 'modality-dependent hypothesis', which posits that coordination dynamics are influenced by the sensory modality mediating informational coupling. However, recognizing that different modalities have inherent differences in accessing spatiotemporal features of perceived movement, we formulated the alternative 'kinematic hypothesis'. This hypothesis posits that differences in dynamics would vanish given equivalent kinematic information across modalities. The study involved forty (N = 40) participants, grouped into twenty (N = 20) dyads, who engaged in a joint finger-tapping task. This task was conducted under varying conditions of visual and auditory coupling, with manipulations in the access to kinematic information, categorized as discrete and continuous. Contrary to our initial predictions, the results strongly supported the 'modality-dependent hypothesis'. We observed that visual and auditory coupling consistently yielded distinct attractor dynamics, regardless of the access to kinematic information. Furthermore, all conditions of auditory coupling resulted in higher levels of synchronization than their visual counterparts. These findings suggest that the differences in interpersonal synchronization are predominantly influenced by the sensory modality, rather than the continuity of kinematic information. Our study highlights the significance of sensorimotor interactions in interpersonal synchronization and addresses the potential of sonification strategies in supporting motor training and rehabilitation.

Exploring crossmodal correspondences for future research in human movement augmentation

"Crossmodal correspondences" are the consistent mappings between perceptual dimensions or stimuli from different sensory domains, which have been widely observed in the general population and investigated by experimental psychologists in recent years. At the same time, the emerging field of human movement augmentation (i.e., the enhancement of an individual's motor abilities by means of artificial devices) has been struggling with the question of how to relay supplementary information concerning the state of the artificial device and its interaction with the environment to the user, which may help the latter to control the device more effectively. To date, this challenge has not been explicitly addressed by capitalizing on our emerging knowledge concerning crossmodal correspondences, despite these being tightly related to multisensory integration. In this perspective paper, we introduce some of the latest research findings on the crossmodal correspondences and their potential role in human augmentation. We then consider three ways in which the former might impact the latter, and the feasibility of this process. First, crossmodal correspondences, given the documented effect on attentional processing, might facilitate the integration of device status information (e.g., concerning position) coming from different sensory modalities (e.g., haptic and visual), thus increasing their usefulness for motor control and embodiment. Second, by capitalizing on their widespread and seemingly spontaneous nature, crossmodal correspondences might be exploited to reduce the cognitive burden caused by additional sensory inputs and the time required for the human brain to adapt the representation of the body to the presence of the artificial device. Third, to accomplish the first two points, the benefits of crossmodal correspondences should be maintained even after sensory substitution, a strategy commonly used when implementing supplementary feedback.

Designing Ecological Auditory Feedback on Lower Limb Kinematics for Hemiparetic Gait Training

Auditory feedback has earlier been explored as a tool to enhance patient awareness of gait kinematics during rehabilitation. In this study, we devised and tested a novel set of concurrent feedback paradigms on swing phase kinematics in hemiparetic gait training. We adopted a user-centered design approach, where kinematic data recorded from 15 hemiparetic patients was used to design three feedback algorithms (wading sounds, abstract, musical) based on filtered gyroscopic data from four inexpensive wireless inertial units. The algorithms were tested (hands-on) by a focus group of five physiotherapists. They recommended that the abstract and musical algorithms be discarded due to sound quality and informational ambiguity. After modifying the wading algorithm (as per their feedback), we conducted a feasibility test involving nine hemiparetic patients and seven physiotherapists, where variants of the algorithm were applied to a conventional overground training session. Most patients found the feedback meaningful, enjoyable to use, natural-sounding, and tolerable for the typical training duration. Three patients exhibited immediate improvements in gait quality when the feedback was applied. However, minor gait asymmetries were found to be difficult to perceive in the feedback, and there was variability in receptiveness and motor change among the patients. We believe that our findings can advance current research in inertial sensor-based auditory feedback for motor learning enhancement during neurorehabilitation.

The Role of a Mechanical Coupling in (Spontaneous) Interpersonal Synchronization: a Human Version of Huygens’ Clock Experiments

Interpersonal musical interaction typically relies on the mutual exchange of auditory and visual information. Inspired by the finding of Christiaan Huygens that two pendulum clocks spontaneously synchronize when hanging from a common, movable wooden beam, we explored the possible use of mechanical coupling as an alternative coupling modality between people to strengthen (spontaneous and instructed) joint (musical) synchronization. From a coupled oscillator viewpoint, we hypothesized that dyads standing on a common movable platform would cause bidirectional passive body motion (and corresponding proprioceptive, vestibular and somatosensory sensations), leading to enhanced interpersonal coordination and mutual entrainment. To test this hypothesis, we asked dyads to perform a musical synchronization–continuation task, while standing on a movable platform. Their rhythmic movements were compared under different conditions: mechanically coupled/decoupled platforms, and spontaneous/instructed synchronization. Additionally, we investigated the effects of performing an additional collaborative conversation task, and of initial tempo and phase differences in the instructed rhythms. The analysis was based on cross wavelet and synchrosqueezed transforms. The overall conclusion was that a mechanical coupling was effective in support of interpersonal synchronization, specifically when dyads were explicitly instructed to synchronize using the movable platform (instructed synchronization). On the other hand, results showed that mechanical coupling led only minimally to spontaneous interpersonal synchronization. The collaborative task and the initial phase and tempo have no strong effect. Although more research is required, possible applications can be found in the domains of music education, dance and music performance, sports, and well-being.

An Embodied Sonification Model for Sit-to-Stand Transfers

Interactive sonification of biomechanical quantities is gaining relevance as a motor learning aid in movement rehabilitation, as well as a monitoring tool. However, existing gaps in sonification research (issues related to meaning, aesthetics, and clinical effects) have prevented its widespread recognition and adoption in such applications. The incorporation of embodied principles and musical structures in sonification design has gradually become popular, particularly in applications related to human movement. In this study, we propose a general sonification model for the sit-to-stand (STS) transfer, an important activity of daily living. The model contains a fixed component independent of the use-case, which represents the rising motion of the body as an ascending melody using the physical model of a flute. In addition, a flexible component concurrently sonifies STS features of clinical interest in a particular rehabilitative/monitoring situation. Here, we chose to represent shank angular jerk and movement stoppages (freezes), through perceptually salient pitch modulations and bell sounds. We outline the details of our technical implementation of the model. We evaluated the model by means of a listening test experiment with 25 healthy participants, who were asked to identify six normal and simulated impaired STS patterns from sonified versions containing various combinations of the constituent mappings of the model. Overall, we found that the participants were able to classify the patterns accurately (86.67 ± 14.69% correct responses with the full model, 71.56% overall), confidently (64.95 ± 16.52% self-reported rating), and in a timely manner (response time: 4.28 ± 1.52 s). The amount of sonified kinematic information significantly impacted classification accuracy. The six STS patterns were also classified with significantly different accuracy depending on their kinematic characteristics. Learning effects were seen in the form of increased accuracy and confidence with repeated exposure to the sound sequences. We found no significant accuracy differences based on the participants' level of music training. Overall, we see our model as a concrete conceptual and technical starting point for STS sonification design catering to rehabilitative and clinical monitoring applications.

Modality-specific attractor dynamics in dyadic entrainment

Rhythmic joint coordination is ubiquitous in daily-life human activities. In order to coordinate their actions towards shared goals, individuals need to co-regulate their timing and move together at the collective level of behavior. Remarkably, basic forms of coordinated behavior tend to emerge spontaneously as long as two individuals are exposed to each other's rhythmic movements. The present study investigated the dynamics of spontaneous dyadic entrainment, and more specifically how they depend on the sensory modalities mediating informational coupling. By means of a novel interactive paradigm, we showed that dyadic entrainment systematically takes place during a minimalistic rhythmic task despite explicit instructions to ignore the partner. Crucially, the interaction was organized by clear dynamics in a modality-dependent fashion. Our results showed highly consistent coordination patterns in visually-mediated entrainment, whereas we observed more chaotic and more variable profiles in the auditorily-mediated counterpart. The proposed experimental paradigm yields empirical evidence for the overwhelming tendency of dyads to behave as coupled rhythmic units. In the context of our experimental design, it showed that coordination dynamics differ according to availability and nature of perceptual information. Interventions aimed at rehabilitating, teaching or training sensorimotor functions can be ultimately informed and optimized by such fundamental knowledge.

Musical Auditory Alpha Wave Neurofeedback: Validation and Cognitive Perspectives

Neurofeedback through visual, auditory, or tactile sensations improves cognitive functions and alters the activities of daily living. However, some people, such as children and the elderly, have difficulty concentrating on neurofeedback for a long time. Constant stressless neurofeedback for a long time may be achieved with auditory neurofeedback using music. The primary purpose of this study was to clarify whether music-based auditory neurofeedback increases the power of the alpha wave in healthy subjects. During neurofeedback, white noise was superimposed on classical music, with the noise level inversely correlating with normalized alpha wave power. This was a single-blind, randomized control crossover trial in which 10 healthy subjects underwent, in an assigned order, normal and random feedback (NF and RF), either of which was at least 4 weeks long. Cognitive functions were evaluated before, between, and after each neurofeedback period. The secondary purpose was to assess neurofeedback-induced changes in cognitive functions. A crossover analysis showed that normalized alpha-power was significantly higher in NF than in RF; therefore, music-based auditory neurofeedback facilitated alpha wave induction. A composite category-based analysis of cognitive functions revealed greater improvements in short-term memory in subjects whose alpha-power increased in response to NF. The present study employed a long period of auditory alpha neurofeedback and achieved successful alpha wave induction and subsequent improvements in cognitive functions. Although this was a pilot study that validated a music-based alpha neurofeedback system for healthy subjects, the results obtained are encouraging for those with difficulty in concentrating on conventional alpha neurofeedback.

Trial registration: 2018077NI, date of registration: 2018/11/27

A Breathing Sonification System to Reduce Stress During the COVID-19 Pandemic

Since sound and music are powerful forces and drivers of human behavior and physiology, we propose the use of sonification to activate healthy breathing patterns in participants to induce relaxation. Sonification is often used in the context of biofeedback as it can represent an informational, non-invasive and real-time stimulus to monitor, motivate or modify human behavior. The first goal of this study is the proposal and evaluation of a distance-based biofeedback system using a tempo- and phase-aligned sonification strategy to adapt breathing patterns and induce states of relaxation. A second goal is the evaluation of several sonification stimuli on 18 participants that were recruited online and of which we analyzed psychometric and behavioral data using, respectively questionnaires and respiration rate and ratio. Sonification stimuli consisted of filtered noise mimicking a breathing sound, nature environmental sounds and a musical phrase. Preliminary results indicated the nature stimulus as most pleasant and as leading to the most prominent decrease of respiration rate. The noise sonification had the most beneficial effect on respiration ratio. While further research is needed to generalize these findings, this study and its methodological underpinnings suggest the potential of the proposed biofeedback system to perform ecologically valid experiments at participants' homes during the COVID-19 pandemic.

Music-based biofeedback to reduce tibial shock in over-ground running: a proof-of-concept study

Methods to reduce impact in distance runners have been proposed based on real-time auditory feedback of tibial acceleration. These methods were developed using treadmill running. In this study, we extend these methods to a more natural environment with a proof-of-concept. We selected ten runners with high tibial shock. They used a music-based biofeedback system with headphones in a running session on an athletic track. The feedback consisted of music superimposed with noise coupled to tibial shock. The music was automatically synchronized to the running cadence. The level of noise could be reduced by reducing the momentary level of tibial shock, thereby providing a more pleasant listening experience. The running speed was controlled between the condition without biofeedback and the condition of biofeedback. The results show that tibial shock decreased by 27% or 2.96 g without guided instructions on gait modification in the biofeedback condition. The reduction in tibial shock did not result in a clear increase in the running cadence. The results indicate that a wearable biofeedback system aids in shock reduction during over-ground running. This paves the way to evaluate and retrain runners in over-ground running programs that target running with less impact through instantaneous auditory feedback on tibial shock.

The sonic instructor: A music-based biofeedback system for improving weightlifting technique

In this study, we assumed that correct functional movements for weightlifting can be learned with the help of a music-based biofeedback system. We compared musical feedback with verbal feedback from experienced trainers using two independent groups. The focus was on one specific movement called deadlift. Physical parameters under considerations were the spine (i.e. loss of midline stability resulting in flexion) and the forward displacement of the barbell during the repetitions relative to the mid-foot. We recruited 31 recreational weight lifters (21-42 years of age). Results revealed that both feedback types are effective in improving the movements for deadlift. No significant differences were found across the two feedback types, neither in terms of movement, nor in terms of clarity and motivation. The results suggest that the proposed feedback system is a valid tool for technology-aided training and self-training practices.

Optimizing beat synchronized running to music

The use of music and specifically tempo-matched music has been shown to affect running performance. But can we maximize the synchronization of movements to music and does maximum synchronization influence kinematics and motivation? In this study, we explore the effect of different types of music-to-movement alignment strategies on phase coherence, cadence and motivation. These strategies were compared to a control condition where the music tempo was deliberately not aligned to the running cadence. Results show that without relative phase alignment, a negative mean asynchrony (NMA) of footfall timings with respect to the beats is obtained. This means that footfalls occurred slightly before the beat and that beats were anticipated. Convergence towards this NMA or preferred relative phase angle was facilitated when the first music beat of a new song started close to the step, which means that entrainment occurred. The results also show that using tempo and phase alignment, the relative phase can be manipulated or forced in a certain angle with a high degree of accuracy. Ensuring negative angles larger than NMA (step before beat) results in increased motivation and decreasing cadence. Running in NMA or preferred relative phase angles results in a null effect on cadence. Ensuring a positive phase angle with respect to NMA results in higher motivation and higher cadence. None of the manipulations resulted in change in perceived exhaustion or a change in velocity. Results also indicate that gender plays an important role when using forced phase algorithms: effects were more pronounced for the female population than for the male population. The implementation of the proposed alignment strategies and control of beat timing while running opens possibilities optimizing the individual running cadence and motivation.