Sound masking by a low-pitch speech-shaped noise improves a social robot's talk in noisy environments

Introduction: There has been a surge in the use of social robots for providing information, persuasion, and entertainment in noisy public spaces in recent years. Considering the well-documented negative effect of noise on human cognition, masking sounds have been introduced. Masking sounds work, in principle, by making the intrusive background speeches less intelligible, and hence, less distracting. However, this reduced distraction comes with the cost of increasing annoyance and reduced cognitive performance in the users of masking sounds. Methods: In a previous study, it was shown that reducing the fundamental frequency of the speech-shaped noise as a masking sound significantly contributes to its being less annoying and more efficient. In this study, the effectiveness of the proposed masking sound was tested on the performance of subjects listening to a lecture given by a social robot in a noisy cocktail party environment. Results: The results indicate that the presence of the masking sound significantly increased speech comprehension, perceived understandability, acoustic satisfaction, and sound privacy of the individuals listening to the robot in an adverse listening condition. Discussion: To the knowledge of the authors, no previous work has investigated the application of sound masking technology in human-robot interaction designs. The future directions of this trend are discussed.

Effect of having and switching multiple avatars on the operator's right to talk and receive social support

People with communication difficulties encounter several challenges in their daily online interactions, such as a limited right to talk (RoT), insufficient social support (SS), and a low sense of being attended to (SoBA). Computer-mediated technologies are limited in addressing such problems owing to their limited capacity in transferring verbal and nonverbal cues between users. In this study, to address the limited RoT, low SS, and low SoBA challenges, we proposed a robotic video conference system with two teleoperated robot avatars. The proposed system was compared with another robotic video conference system that adopts only one teleoperated robot avatar. In the field experiment, 37 participants took part in two discussion sessions using each system type, where RoT, SS, and SoBA were adopted as the measured indices. The proposed system significantly increased the users' RoT and SS compared with other robotic video conference systems. This study contributes to the literature by demonstrating the effect exerted by the type of robotic video conference adopted on users' feelings about RoT, SS, and SoBA.

Towards a simultaneously speaking bilingual robot: Primary study on the effect of gender and pitch of the robot's voice

With fast and reliable international transportation, more people with different language backgrounds can interact now. As a result, the need for communicative agents fluent in several languages to assist those people is highlighted. The high cost of hiring human attendants fluent in several languages makes using social robots a more affordable alternative in international gatherings. A social robot capable of presenting a piece of information in more than one language at the same time to its audience is the goal of this line of study. However, the negative effect of background noise on speech comprehension in humans is well-established. Hence, presenting a piece of information in two different languages at the same time by the robot creates an adverse listening condition for both individuals listening to the speech of such a bilingual robot. In this study, we investigated whether manipulating the pitch and gender of the robot's voice could affect human subjects' memory of the presented information in the presence of background noise. The results indicate that the pitch and gender of the speaking voice do indeed affect our memory of the presented information. when a male voice was used, a higher pitch resulted in significantly better memory performance than a lower pitch. Contrarily, when a female voice was used, a lower pitch resulted in significantly better memory in participants than a higher pitch. Both male and female subjects performed significantly better with a female voice in a noisy background. In nutshell, the result of this study suggests using a female voice for robots in noisy conditions, as in the case of simultaneously speaking robots, can significantly improve the retrieval of presented information in human subjects.

Multiple Groups of Agents for Increased Movement Interference and Synchronization

We examined the influence of groups of agents and the type of avatar on movement interference. In addition, we studied the synchronization of the subject with the agent. For that, we conducted experiments utilizing human subjects to examine the influence of one, two, or three agents, as well as human or robot avatars, and finally, the agent moving biologically or linearly. We found the main effect on movement interference was the number of agents; namely, three agents had significantly more influence on movement interference than one agent. These results suggest that the number of agents is more influential on movement interference than other avatar characteristics. For the synchronization, the main effect of the type of the agent was revealed, showing that the human agent kept more synchronization compared to the robotic agent. In this experiment, we introduced an additional paradigm on the interference which we called synchronization, discovering that a group of agents is able to influence this behavioral level as well.

A Preliminary Study on Realizing Human-Robot Mental Comforting Dialogue via Sharing Experience Emotionally

Mental health issues are receiving more and more attention in society. In this paper, we introduce a preliminary study on human-robot mental comforting conversation, to make an android robot (ERICA) present an understanding of the user's situation by sharing similar emotional experiences to enhance the perception of empathy. Specifically, we create the emotional speech for ERICA by using CycleGAN-based emotional voice conversion model, in which the pitch and spectrogram of the speech are converted according to the user's mental state. Then, we design dialogue scenarios for the user to talk about his/her predicament with ERICA. In the dialogue, ERICA shares other people's similar predicaments and adopts a low-spirit voice to express empathy to the interlocutor's situation. At the end of the dialogue, ERICA tries to encourage with a positive voice. Subsequently, questionnaire-based evaluation experiments were conducted with the recorded conversation. In the questionnaire, we use the Big Five scale to evaluate ERICA's personality. In addition, the perception of emotion, empathy, and encouragement in the dialogue are evaluated. The results show that the proposed emotional expression strategy helps the android robot better present low-spirit emotion, empathy, the personality of extroversion, while making the user better feel the encouragement.

Expansion in speech time can restore comprehension in a simultaneously speaking bilingual robot

Introduction: In this study, the development of a social robot, capable of giving speech simultaneously in more than one language was in mind. However, the negative effect of background noise on speech comprehension is well-documented in previous works. This deteriorating effect is more highlighted when the background noise has speech-like properties. Hence, the presence of speech as the background noise in a simultaneously speaking bilingual robot can be fatal for the speech comprehension of each person listening to the robot. Methods: To improve speech comprehension and consequently, user experience in the intended bilingual robot, the effect of time expansion on speech comprehension in a multi-talker speech scenario was investigated. Sentence recognition, speech comprehension, and subjective evaluation tasks were implemented in the study. Results: The obtained results suggest that a reduced speech rate, leading to an expansion in the speech time, in addition to increased pause duration in both the target and background speeches can lead to statistically significant improvement in both sentence recognition, and speech comprehension of participants. More interestingly, participants got a higher score in the time-expanded multi-talker speech than in the standard-speed single-talker speech in the speech comprehension and, in the sentence recognition task. However, this positive effect could not be attributed merely to the time expansion, as we could not repeat the same positive effect in a time-expanded single-talker speech. Discussion: The results obtained in this study suggest a facilitating effect of the presence of the background speech in a simultaneously speaking bilingual robot provided that both languages are presented in a time-expanded manner. The implications of such a simultaneously speaking robot are discussed.

Communication Apprehension and Eye Contact Anxiety in Video Conferences Involving Teleoperated Robot Avatars: A Subjective Evaluation Study

Communication apprehension (CA), defined as anxiety in oral communication, and anxiety in eye contact (AEC), defined as the discomfort felt in communication while being stared at by others, limit communication effectiveness. In this study, we examined whether using a teleoperated robot avatar in a video teleconference provides communication support to people with CA and AEC. We propose a robotic telecommunication system in which a user has two options to produce utterance for own responses in online interaction with interviewer i.e., either by a robot avatar that faces the interviewer, or by self. Two imagination-based experiments were conducted, in which a total of 400 participants were asked to watch videos for interview scenes with or without the proposed system; 200 participants for each experiment. The participants then evaluated their impressions by imagining that they were the interviewee. In the first experiment, a video conference with the proposed system was compared with an ordinary video conference, where the interviewer and interviewee faced each other. In the second experiment, it was compared with an ordinary video conference where the interviewer’s attentional focus was directed away from the interviewee. A significant decrease in the expected CA and AEC of participants with the proposed system was observed in both experiments, whereas a significant increase in the expected sense of being attended (SoBA) was observed in the second experiment. This study contributes to the literature in terms of examining the expected impact of using a teleoperated robot avatar for better video conferences, especially for supporting individuals with CA and AEC.

Ostensive-Cue Sensitive Learning and Exclusive Evaluation of Policies: A Solution for Measuring Contingency of Experiences for Social Developmental Robot

Joint attention related behaviors (JARBs) are some of the most important and basic cognitive functions for establishing successful communication in human interaction. It is learned gradually during the infant's developmental process, and enables the infant to purposefully improve his/her interaction with the others. To adopt such a developmental process for building an adaptive and social robot, previous studies proposed several contingency evaluation methods, by which an infant robot becomes able to sequentially learn some primary social skills. These skills included gaze following and social referencing, and could be acquired through interacting with a human caregiver model in a computer simulation. However, to implement such methods to a real-world robot, two major problems, that were not addressed in the previous research, have remained unresearched: (1) dependency of histogram of the observed events by the robot to each other, which increases the error of the internal calculation and consequently decreases the accuracy of contingency evaluation; and (2) unsynchronized teaching/learning phase of the teaching-caregiver and the learning-robot, which leads the robot and the caregiver not to understand the suitable timing for the learning and the teaching, respectively. In this paper, we address these two problems, and propose two algorithms in order to solve them: (1) exclusive evaluation of policies (XEP) for the former, and (2) ostensive-cue sensitive learning (OsL) for the latter. To show the effect of the proposed algorithms, we conducted a real-world human-robot interaction experiment with 48 subjects, and compared the performance of the learning robot with/without proposed algorithms. Our results show that adopting proposed algorithms improves the robot's performance in terms of learning efficiency, complexity of the learned behaviors, predictability of the robot, and even the result of the subjective evaluation of the participants about the intelligence of the robot as well as the quality of the interaction.