Framework for identifying and visualising emotional atmosphere in online learning environments in the COVID-19 Era

In addition to the almost five million lives lost and millions more than that in hospitalisations, efforts to mitigate the spread of the COVID-19 pandemic, which that has disrupted every aspect of human life deserves the contributions of all and sundry. Education is one of the areas most affected by the COVID-imposed abhorrence to physical (i.e., face-to-face (F2F)) communication. Consequently, schools, colleges, and universities worldwide have been forced to transition to different forms of online and virtual learning. Unlike F2F classes where the instructors could monitor and adjust lessons and content in tandem with the learners’ perceived emotions and engagement, in online learning environments (OLE), such tasks are daunting to undertake. In our modest contribution to ameliorate disruptions to education caused by the pandemic, this study presents an intuitive model to monitor the concentration, understanding, and engagement expected of a productive classroom environment. The proposed apposite OLE (i.e., AOLE) provides an intelligent 3D visualisation of the classroom atmosphere (CA), which could assist instructors adjust and tailor both content and instruction for maximum delivery. Furthermore, individual learner status could be tracked via visualisation of his/her emotion curve at any stage of the lesson or learning cycle. Considering the enormous emotional and psychological toll caused by COVID and the attendant shift to OLE, the emotion curves could be progressively compared through the duration of the learning cycle and the semester to track learners’ performance through to the final examinations. In terms of learning within the CA, our proposed AOLE is assessed within a class of 15 students and three instructors. Correlation of the outcomes reported with those from administered questionnaires validate the potential of our proposed model as a support for learning and counselling during these unprecedentedtimes that we find ourselves.

Speaker Localization Based on Audio-Visual Bimodal Fusion

The demand for fluency in human–computer interaction is on an increase globally; thus, the active localization of the speaker by the machine has become a problem worth exploring. Considering that the stability and accuracy of the single-mode localization method are low, while the multi-mode localization method can utilize the redundancy of information to improve accuracy and anti-interference, a speaker localization method based on voice and image multimodal fusion is proposed. First, the voice localization method based on time differences of arrival (TDOA) in a microphone array and the face detection method based on the AdaBoost algorithm are presented herein. Second, a multimodal fusion method based on spatiotemporal fusion of speech and image is proposed, and it uses a coordinate system converter and frame rate tracker. The proposed method was tested by positioning the speaker stand at 15 different points, and each point was tested 50 times. The experimental results demonstrate that there is a high accuracy when the speaker stands in front of the positioning system within a certain range.

How to Combine (Dis)Utilities of Different Aspects into a Single (Dis)Utility Value, and How This Is Related to Geometric Images of Happiness

In many practical situations, a user needs our help in selecting the best out of a large number of alternatives. To be able to help, we need to understand the user’s preferences. In decision theory, preferences are described by numerical values known as utilities. It is often not feasible to ask the user to provide utilities of all possible alternatives, so we must be able to estimate these utilities based on utilities of different aspects of these alternatives. In this paper, we provide a general formula for combining utilities of aspects into a single utility value. The resulting formula turns out to be in good accordance with the known correspondence between geometric images and different degrees of happiness.

Speech Emotion Recognition Based on Selective Interpolation Synthetic Minority Over-Sampling Technique in Small Sample Environment

Speech emotion recognition often encounters the problems of data imbalance and redundant features in different application scenarios. Researchers usually design different recognition models for different sample conditions. In this study, a speech emotion recognition model for a small sample environment is proposed. A data imbalance processing method based on selective interpolation synthetic minority over-sampling technique (SISMOTE) is proposed to reduce the impact of sample imbalance on emotion recognition results. In addition, feature selection method based on variance analysis and gradient boosting decision tree (GBDT) is introduced, which can exclude the redundant features that possess poor emotional representation. Results of experiments of speech emotion recognition on three databases (i.e., CASIA, Emo-DB, SAVEE) show that our method obtains average recognition accuracy of 90.28% (CASIA), 75.00% (SAVEE) and 85.82% (Emo-DB) for speaker-dependent speech emotion recognition which is superior to some state-of-the-arts works.

Quantum Structure for Modelling Emotion Space of Robots

Utilising the properties of quantum mechanics, i.e., entanglement, parallelism, etc., a quantum structure is proposed for representing and manipulating emotion space of robots. This quantum emotion space (QES) provides a mechanism to extend emotion interpretation to the quantum computing domain whereby fewer resources are required and, by using unitary transformations, it facilitates easier tracking of emotion transitions over different intervals in the emotion space. The QES is designed as an intuitive and graphical visualisation of the emotion state as a curve in a cuboid, so that an “emotion sensor” could be used to track the emotion transition as well as its manipulation. This ability to use transition matrices to convey manipulation of emotions suggests the feasibility and effectiveness of the proposed approach. Our study is primarily influenced by two developments. First, the massive amounts of data, complexity of control, planning and reasoning required for today’s sophisticated automation processes necessitates the need to equip robots with powerful sensors to enable them adapt and operate in all kinds of environments. Second, the renewed impetus and inevitable transition to the quantum computing paradigm suggests that quantum robots will have a role to play in future data processing and human-robot interaction either as standalone units or as part of larger hybrid systems. The QES proposed in this study provides a quantum mechanical formulation for quantum emotion as well as a platform to process, track, and manipulate instantaneous transitions in a robot’s emotion. The new perspective will open broad areas, such as applications in emotion recognition and emotional intelligence for quantum robots.

Multi-Robot Behavior Adaptation to Humans’ Intention in Human-Robot Interaction Using Information-Driven Fuzzy Friend-Q Learning

A multi-robot behavior adaptation mechanism that adapts to human intention is proposed for human-robot interaction (HRI), where information-driven fuzzy friend-Q learning (IDFFQ) is used to generate an optimal behavior-selection policy, and intention is understood mainly based on human emotions. This mechanism aims to endow robots with human-oriented interaction capabilities to understand and adapt their behaviors to human intentions. It also decreases the response time (RT) of robots by embedding the human identification information such as religion for behavior selection, and increases the satisfaction of humans by considering their deep-level information, including intention and emotion, so as to make interactions run smoothly. Experiments is performed in a scenario of drinking at a bar. Results show that the learning steps of the proposal is 51 steps less than that of the fuzzy production rule based friend-Q learning (FPRFQ), and the robots’ RT is about 25% of the time consumed by FPRFQ. Additionally, emotion recognition and intention understanding achieved an accuracy of 80.36% and 85.71%, respectively. Moreover, a subjective evaluation of customers through a questionnaire obtains a reaction of “satisfied.” Based on these preliminary experiments, the proposal is being extended to service robots for behavior adaptation to customers’ intention to drink at a bar.

Deep Level Emotion Understanding Using Customized Knowledge for Human-Robot Communication

In this study, a method for acquiring deep level emotion understanding is proposed to facilitate better human-robot communication, where customized learning knowledge of an observed agent (human or robot) is used with the observed input information from a Kinect sensor device. It aims to obtain agentdependent emotion understanding by utilizing special customized knowledge of the agent rather than ordinary surface level emotion understanding that uses visual/acoustic/distance information without any customized knowledge. In the experiment employing special demonstration scenarios where a company employee’s emotion is understood by a secretary eye robot equipped with a Kinect sensor device, it is confirmed that the proposed method provides deep level emotion understanding that is different from ordinary surface level emotion understanding. The proposal is being planned to be applied to a part of the emotion understanding module in the demonstration experiments of an ongoing robotics research project titled “Multi-Agent Fuzzy Atmosfield.”

Bloch Sphere-Based Representation for Quantum Emotion Space

A Bloch Sphere-based Emotion Space (BSES), where two angles φ and θ in the Bloch sphere represent the emotion (such as happiness, surprise, anger, sadness, expectation, or relaxation in [0, 2π)) and its intensity (from neutral to maximum in [0, π]), respectively, is proposed. It exploits the psychological interpretation of color to assign a basic color to each emotion subspace such that the BSES can be visualized, and by using quantum gates, changes in emotions can be tracked and recovered. In an experimental validation, two typical human emotions, happiness and sadness, are analyzed and visualized using the BSES according to a preset emotional transmission model. A transition matrix that tracks emotional change can be used to control robots allowing them to adapt and respond to human emotions.

Adapting Multi-Robot Behavior to Communication Atmosphere in Humans-Robots Interaction Using Fuzzy Production Rule Based Friend-Q Learning

A behavior adaptation mechanism in humans-robots interaction is proposed to adjust robots’ behavior to communication atmosphere, where fuzzy production rule based friend-Q learning (FPRFQ) is introduced. It aims to shorten the response time of robots and decrease the social distance between humans and robots to realize the smooth communication of robots and humans. Experiments on robots/humans interaction are performed in a virtual communication atmosphere environment. Results show that robots adapt well by saving 44 and 482 learning steps compared to that by friend-Q learning (FQ) and independent learning (IL), respectively; additionally, the distance between human-generated atmosphere and robot-generated atmosphere is 3 times and 10 times shorter than the FQ and the IL, respectively. The proposed behavior adaptation mechanism is also applied to robots’ eye movement in the developing humans-robots interaction system, calledmascot robot system, and basic experimental results are shown in home party scenario with five eye robots and four humans.