Question asking and eye tracking during cognitive disequilibrium: comprehending illustrated texts on devices when the devices break down

Invoking and identifying task-oriented interlocutor confusion in human-robot interaction

Successful conversational interaction with a social robot requires not only an assessment of a user's contribution to an interaction, but also awareness of their emotional and attitudinal states as the interaction unfolds. To this end, our research aims to systematically trigger, but then interpret human behaviors to track different states of potential user confusion in interaction so that systems can be primed to adjust their policies in light of users entering confusion states. In this paper, we present a detailed human-robot interaction study to prompt, investigate, and eventually detect confusion states in users. The study itself employs a Wizard-of-Oz (WoZ) style design with a Pepper robot to prompt confusion states for task-oriented dialogues in a well-defined manner. The data collected from 81 participants includes audio and visual data, from both the robot's perspective and the environment, as well as participant survey data. From these data, we evaluated the correlations of induced confusion conditions with multimodal data, including eye gaze estimation, head pose estimation, facial emotion detection, silence duration time, and user speech analysis-including emotion and pitch analysis. Analysis shows significant differences of participants' behaviors in states of confusion based on these signals, as well as a strong correlation between confusion conditions and participants own self-reported confusion scores. The paper establishes strong correlations between confusion levels and these observable features, and lays the ground or a more complete social and affect oriented strategy for task-oriented human-robot interaction. The contributions of this paper include the methodology applied, dataset, and our systematic analysis.

Facial Expressions as an Index of Listening Difficulty and Emotional Response

Knowledge about listening difficulty experienced during a task can be used to better understand speech perception processes, to guide amplification outcomes, and can be used by individuals to decide whether to participate in communication. Another factor affecting these decisions is individuals' emotional response which has not been measured objectively previously. In this study, we describe a novel method of measuring listening difficulty and affect of individuals in adverse listening situations using automatic facial expression algorithm. The purpose of our study was to determine if facial expressions of confusion and frustration are sensitive to changes in listening difficulty. We recorded speech recognition scores, facial expressions, subjective listening effort scores, and subjective emotional responses in 33 young participants with normal hearing. We used the signal-to-noise ratios of -1, +2, and +5 dB SNR and quiet conditions to vary the difficulty level. We found that facial expression of confusion and frustration increased with increase in difficulty level, but not with change in each level. We also found a relationship between facial expressions and both subjective emotion ratings and subjective listening effort. Emotional responses in the form of facial expressions show promise as a measure of affect and listening difficulty. Further research is needed to determine the specific contribution of affect to communication in challenging listening environments.

Confused or not: decoding brain activity and recognizing confusion in reasoning learning using EEG

Objective.Confusion is the primary epistemic emotion in the learning process, influencing students' engagement and whether they become frustrated or bored. However, research on confusion in learning is still in its early stages, and there is a need to better understand how to recognize it and what electroencephalography (EEG) signals indicate its occurrence. The present work investigates confusion during reasoning learning using EEG, and aims to fill this gap with a multidisciplinary approach combining educational psychology, neuroscience and computer science.Approach.First, we design an experiment to actively and accurately induce confusion in reasoning. Second, we propose a subjective and objective joint labeling technique to address the label noise issue. Third, to confirm that the confused state can be distinguished from the non-confused state, we compare and analyze the mean band power of confused and unconfused states across five typical bands. Finally, we present an EEG database for confusion analysis, together with benchmark results from conventional (Naive Bayes, Support Vector Machine, Random Forest, and Artificial Neural Network) and end-to-end (Long Short Term Memory, Residual Network, and EEGNet) machine learning methods.Main results.Findings revealed: 1. Significant differences in the power of delta, theta, alpha, beta and lower gamma between confused and non-confused conditions; 2. A higher attentional and cognitive load when participants were confused; and 3. The Random Forest algorithm with time-domain features achieved a high accuracy/F1 score (88.06%/0.88 for the subject-dependent approach and 84.43%/0.84 for the subject-independent approach) in the binary classification of the confused and non-confused states.Significance.The study advances our understanding of confusion and provides practical insights for recognizing and analyzing it in the learning process. It extends existing theories on the differences between confused and non-confused states during learning and contributes to the cognitive-affective model. The research enables researchers, educators, and practitioners to monitor confusion, develop adaptive systems, and test recognition approaches.

Automatic Detection of a Student's Affective States for Intelligent Teaching Systems

AutoTutor is an automated computer tutor that simulates human tutors and holds conversations with students in natural language. Using data collected from AutoTutor, the following determinations were sought: Can we automatically classify affect states from intelligent teaching systems to aid in the detection of a learner’s emotional state? Using frequency patterns of AutoTutor feedback and assigned user emotion in a series of pairs, can the next pair of feedback/emotion series be predicted? Through a priori data mining approaches, we found dominant frequent item sets that predict the next set of responses. Thirty-four participants provided 200 turns between the student and the AutoTutor. Two series of attributes and emotions were concatenated into one row to create a record of previous and next set of emotions. Feature extraction techniques, such as multilayer-perceptron and naive Bayes, were performed on the dataset to perform classification for affective state labeling. The emotions ‘Flow’ and ‘Frustration’ had the highest classification of all the other emotions when measured against other emotions and their respective attributes. The most common frequent item sets were ‘Flow’ and ‘Confusion’.

Revealing Unconscious Consumer Reactions to Advertisements That Include Visual Metaphors. A Neurophysiological Experiment

The main challenge of advertising is to catch consumers' attention and evoke in them positive attitudes to consequently achieve product preference and higher purchase intentions. In modern advertising, visual metaphors are widely used due to their effects such as improving advertising recall, enhancing persuasiveness, and generating consumers' positive attitudes. Previous research has pointed out the existence of an "inverted U-curve" that describes a positive relationship between the conceptual complexity of metaphors and consumers' positive reactions to them, which ends where complexity outweighs comprehension. Despite the dominance of visual metaphors in modern advertising, academic research on this topic has been relatively sparse. The inverted U-curve pattern has been validated regarding ad appreciation, ad liking, and purchase intention by using declarative methods. However, at present, there is no evidence of consumers' neurophysiological responses to visual metaphors included in advertising. Given this gap, the aim of this research is to assess consumer neurophysiological responses to print advertisements that include visual metaphors, using neuroscience-based techniques. Forty-three participants (22W-21M) were exposed to 28 stimuli according to three levels of visual complexity, while their reactions were recorded with an electroencephalogram (EEG), eye tracking (ET), and galvanic skin response (GSR). The results indicated that, regardless of metaphor type, ads with metaphors evoke more positive reactions than non-metaphor ads. EEG results revealed a positive relationship between cognitive load and conceptual complexity that is not mediated by comprehension. This suggests that the cognitive load index could be a suitable indicator of complexity, as it reflects the amount of cognitive resources needed to process stimuli. ET results showed significant differences in the time dedicated to exploring the ads; however, comprehension doesn't mediate this relationship. Moreover, no cognitive load was detected from GSR. ET and GSR results suggest that neither methodology is a suitable measure of cognitive load in the case of visual metaphors. Instead, it seems that they are more related to the attention and/or emotion devoted to the stimuli. Our empirical analysis reveals the importance of using neurophysiological measures to analyze the appropriate use of visual metaphors and to find out how to maximize their impact on advertising effectiveness.

Puzzle-Solving Activity as an Indicator of Epistemic Confusion

When students perform complex cognitive activities, such as solving a problem, epistemic emotions can occur and influence the completion of the task. Confusion is one of these emotions and it can produce either negative or positive outcomes, according to the situation. For this reason, considering confusion can be an important factor for educators to evaluate students’ progression in cognitive activities. However, in digital learning environments, observing students’ confusion, as well as other epistemic emotions, can be problematic because of the remoteness of students. The study reported in this article explored new methodologies to assess emotions in a problem-solving task. The experimental task consisted of the resolution of logic puzzles presented on a computer, before, and after watching an instructional video depicting a method to solve the puzzle. In parallel to collecting self-reported confusion ratings, human-computer interaction was captured to serve as non-intrusive measures of emotions. The results revealed that the level of self-reported confusion was negatively correlated with the performance on solving the puzzles. In addition, while comparing the pre- and post-video sequences, the experience of confusion tended to differ. Before watching the instructional video, the number of clicks on the puzzle was positively correlated with the level of confusion whereas the correlation was negatively after the video. Moreover, the main emotions reported before the video (e.g., confusion, frustration, curiosity) tended to differ from the emotions reported after the videos (e.g., engagement, delight, boredom). These results provide insights into the ambivalent impact of confusion in problem-solving task, illustrating the dual effect (i.e., positive or negative) of this emotion on activity and performance, as reported in the literature. Applications of this methodology to real-world settings are discussed.

Time limits in testing: An analysis of eye movements and visual attention in spatial problem solving

Individuals with an aptitude for interpreting spatial information (high mental rotation ability: HMRA) typically master anatomy with more ease, and more quickly, than those with low mental rotation ability (LMRA). This article explores how visual attention differs with time limits on spatial reasoning tests. Participants were assorted to two groups based on their mental rotation ability scores and their eye movements were collected during these tests. Analysis of salience during testing revealed similarities between MRA groups in untimed conditions but significant differences between the groups in the timed one. Question-by-question analyses demonstrate that HMRA individuals were more consistent across the two timing conditions (κ = 0.25), than the LMRA (κ = 0.013). It is clear that the groups respond to time limits differently and their apprehension of images during spatial problem solving differs significantly. Without time restrictions, salience analysis suggests LMRA individuals attended to similar aspects of the images as HMRA and their test scores rose concomitantly. Under timed conditions however, LMRA diverge from HMRA attention patterns, adopting inflexible approaches to visual search and attaining lower test scores. With this in mind, anatomical educators may wish to revisit some evaluations and teaching approaches in their own practice. Although examinations need to evaluate understanding of anatomical relationships, the addition of time limits may induce an unforeseen interaction of spatial reasoning and anatomical knowledge. Anat Sci Educ 10: 528-537. © 2017 American Association of Anatomists.

Confusion and its dynamics during device comprehension with breakdown scenarios

The incidence and dynamics of confusion during complex learning and problem solving were investigated in an experiment where participants first read illustrated texts on everyday devices (e.g., an electric bell) followed by breakdown scenarios reflecting device malfunctions (e.g., "When a person rang the bell there was a short ding and then no sound was heard"). The breakdown scenarios were expected to trigger impasses and put participants in a state of cognitive disequilibrium where they would experience confusion and engage in effortful confusion resolution activities in order to restore equilibrium. The results confirmed that participants reported more confusion when presented with the breakdown scenarios compared to control scenarios that involved focusing on important device components in the absence of malfunctions. A second-by-second analysis of the dynamics of confusion yielded two characteristic trajectories that distinguished participants who partially resolved their confusion from those who remained confused. Participants who were successful in partial confusion resolution while processing the breakdowns outperformed their counterparts on knowledge assessments after controlling for scholastic aptitude, engagement, and frustration. This effect was amplified for those who were highly confused by the breakdowns. There was no direct breakdown vs. control effect on learning, but being actively engaged and partially resolving confusion during breakdown processing were positive predictors of increased learning with the breakdown compared to control scenarios. Implications of our findings for theories that highlight the role of impasses, cognitive disequilibrium, and confusion to learning are discussed.

Questioning and reading goals: information-seeking questions asked on scientific texts read under different task conditions

BACKGROUND

A number of studies report that few questions are asked in classrooms and that many of them are shallow questions.

AIMS

This study investigates the way in which reading goals determine questioning on scientific texts. Reading goals were manipulated through two different tasks: reading for understanding versus reading to solve a problem.

SAMPLE

A total of 183 university students.

METHODS

In the first and third questioning experiments, the participants read two short texts. Students in one condition were instructed to understand the texts, whereas in the alternative condition they had to read texts to solve a problem. Students were instructed to write down any questions they might have about the texts. The questions were categorized according to the type of underlying obstacle: associative, explanatory, or predictive. The second experiment used a think-aloud methodology to identify the mental representations generated by the students.

RESULTS AND CONCLUSIONS

The two questioning experiments show that the questions asked depend on the reading goals. Significantly more explanation questions were asked in the understanding condition than in the problem-solving condition. Also, the two conditions were found to have a different influence on the generation of association and explanation questions. Very few prediction questions were asked in either condition. The think-aloud experiment revealed that the mental representations attempted by readers under the two conditions were indeed different. In conclusion, the experiments showed that, given a certain textual input, readers' questions depend on the reading goals associated with tasks.

The half-life of cognitive-affective states during complex learning

We investigated the temporal dynamics of students' cognitive-affective states (confusion, frustration, boredom, engagement/flow, delight, and surprise) during deep learning activities. After a learning session with an intelligent tutoring system with conversational dialogue, the cognitive-affective states of the learner were classified by the learner, a peer, and two trained judges at approximately 100 points in the tutorial session. Decay rates for the cognitive-affective states were estimated by fitting exponential curves to time series of affect responses. The results partially confirmed predictions of goal-appraisal theories of emotion by supporting a tripartite classification of the states along a temporal dimension: persistent states (boredom, engagement/flow, and confusion), transitory states (delight and surprise), and an intermediate state (frustration). Patterns of decay rates were generally consistent across affect judges, except that a reversed actor-observer effect was discovered for engagement/flow and frustration. Correlations between decay rates of the cognitive-affective states and several learning measures confirmed the major predictions and uncovered some novel findings that have implications for theories of pedagogy that integrate cognition and affect during deep learning.

Depending on My Mood: Mood-Driven Influences on Text Comprehension

Reading comprehension is a critical component of success in educational settings. To date, research on text processing in educational and cognitive psychological domains has focused predominantly on cognitive influences on comprehension, and in particular, those influences that might be derived from particular tasks or strategies. However, there is growing interest in documenting the influences of emotional factors on the processes and products of text comprehension, because these factors are less likely to be associated with explicit reading strategies. The present study examines this issue by evaluating the degree to which mood can influence readers' processing of text. Participants in control, happy-induced, or sad-induced groups thought aloud while reading expository texts. Happy, sad, and neutral moods influenced the degree to which readers engaged in particular types of coherence-building processes in the service of comprehension. Although reading strategies clearly influence processing, understudied factors that are less explicitly goal-driven, such as mood, can similarly impact comprehension activity. These findings have important implications for the role of mood on reading instruction and evaluation.