Tracking the Decision-Making Process in Multiple-Choice Assessment: Evidence from Eye Movements

Did You Get That? Predicting Learners' Comprehension of a Video Lecture from Visualizations of Their Gaze Data

In online lectures, unlike in face-to-face lectures, teachers lack access to (nonverbal) cues to check if their students are still "with them" and comprehend the lecture. The increasing availability of low-cost eye-trackers provides a promising solution. These devices measure unobtrusively where students look and can visualize these data to teachers. These visualizations might inform teachers about students' level of "with-me-ness" (i.e., do students look at the information that the teacher is currently talking about) and comprehension of the lecture, provided that (1) gaze measures of "with-me-ness" are related to comprehension, (2) people not trained in eye-tracking can predict students' comprehension from gaze visualizations, (3) we understand how different visualization techniques impact this prediction. We addressed these issues in two studies. In Study 1, 36 students watched a video lecture while being eye-tracked. The extent to which students looked at relevant information and the extent to which they looked at the same location as the teacher both correlated with students' comprehension (score on an open question) of the lecture. In Study 2, 50 participants watched visualizations of students' gaze (from Study 1), using six visualization techniques (dynamic and static versions of scanpaths, heatmaps, and focus maps) and were asked to predict students' posttest performance and to rate their ease of prediction. We found that people can use gaze visualizations to predict learners' comprehension above chance level, with minor differences between visualization techniques. Further research should investigate if teachers can act on the information provided by gaze visualizations and thereby improve students' learning.

IEyeGASE: An Intelligent Eye Gaze-Based Assessment System for Deeper Insights into Learner Performance

In the current education environment, learning takes place outside the physical classroom, and tutors need to determine whether learners are absorbing the content delivered to them. Online assessment has become a viable option for tutors to establish the achievement of course learning outcomes by learners. It provides real-time progress and immediate results; however, it has challenges in quantifying learner aspects like wavering behavior, confidence level, knowledge acquired, quickness in completing the task, task engagement, inattentional blindness to critical information, etc. An intelligent eye gaze-based assessment system called IEyeGASE is developed to measure insights into these behavioral aspects of learners. The system can be integrated into the existing online assessment system and help tutors re-calibrate learning goals and provide necessary corrective actions.

Deliberative process in sharing information with different audiences: Eye-tracking correlates

Research on conversational pragmatics demonstrates how interlocutors tailor the information they share depending on the audience. Previous research showed that, in informal contexts, speakers often provide several alternative answers, whereas in formal contexts, they tend to give only a single answer; however, the psychological underpinnings of these effects remain obscure. To investigate this answer selection process, we measured participants' eye movements in different experimentally modelled social contexts. Participants answered general knowledge questions by providing responses with either single (one) or plural (three) alternatives. Then, a formal (job interview) or informal (conversation with friends) context was presented and participants decided either to report or withdraw their responses after considering the given social context. Growth curve analysis on the eye movements indicates that the selected response option attracted more eye movements. There was a discrepancy between the answer selection likelihood and the proportion of fixations to the corresponding option-but only in the formal context. These findings support a more elaborate decision-making processes in formal contexts. They also suggest that eye movements do not necessarily accompany the options considered in the decision-making processes.

Lernergebnisse und individuelle Prozesse des Physik-Lernens mit auditiven und visuellen Hinweisen

Zusammenfassung. Die vorliegende Studie untersuchte im Kontext einer videobasierten Lernumgebung, inwieweit Personen durch die Darbietung von auditiven und visuellen Hinweisen beim Lernen komplexer Inhalte unterstützt werden können. Vor dem Hintergrund von Theorien des multimedialen Lernens wurden die Wirkungen der Hinweise auf kognitive Belastung, Lernerfolg sowie Blickbewegungen der Lernenden analysiert. In einem experimentellen Zwei-Gruppen-Design mit Vor- und Nachtest lernten 46 Teilnehmerinnen und Teilnehmer mithilfe visueller ( n = 23) oder auditiver ( n = 23) Unterstützung. Den Lerngegenstand bildeten Skizzen elektrischer Schaltungen, die als kognitives Werkzeug zur Lösung physikalischer Aufgaben eingesetzt werden. Es ergaben sich zwischen den Versuchsbedingungen keine signifikanten Unterschiede in Belastungserleben und Lernerfolg (.00 ≤ ηp2 ≤ .08). Allerdings waren Maße der kognitiven Belastung signifikante Prädiktoren des Lernerfolgs. Es zeigten sich auch bedeutsame Einflüsse des Vorwissens auf Wissensabruf ( R2 = .213) und Wissenstransfer ( R2 = .257). Im Rahmen qualitativer Auswertungen wurden zudem die Blickpfade von Personen mit niedrigem und hohem Vorwissen verglichen. Hier zeigte sich in beiden Modalitäten bei Personen mit niedrigem Vorwissen eher ein detailorientiertes Blickmuster auf Einzelelemente der Schaltungen und bei Personen mit hohem Vorwissen eher ein ganzheitlich orientiertes Blickmuster über das System von Schaltungen. In Bezug auf die vorwissensabhängigen Unterschiede in den Blickmustern und die Komplexität des Lerngegenstandes erscheinen sowohl eine zeitliche als auch eine instruktionale Adaptierbarkeit der Lernumgebung für Lernende mit wenig Vorwissen in beiden Hinweismodalitäten sinnvoll. Für die zukünftige Forschung bietet es sich im Sinne eines Mixed-Method-Ansatzes an, Prozesse und Ergebnisse des Lernens stärker aufeinander zu beziehen, um zu tieferen Erkenntnissen über die Nutzung und Wirkung instruktionaler Hinweise zu gelangen.

The Effects of Different Feedback Types on Learning With Mobile Quiz Apps

Testing is an effective learning method, and it is the basis of mobile quiz apps. Quiz apps have the potential to facilitate remote and self-regulated learning. In this context, automatized feedback plays a crucial role. In two experimental studies, we examined the effects of two feedback types of quiz apps on performance, namely, the standard corrective feedback of quiz apps and a feedback that incorporates additional information related to the correct response option. We realized a controlled lab setting (n = 68, Study 1) and an unsupervised mobile setting (n = 150, Study 2). In the learning phase, participants used the quiz app and received feedback. They also completed a subsequent test as well as a follow-up test 1 week later by using the same quiz app. Irrespective of feedback type and setting, cognitive outcomes (quiz scores) and metacognitive outcomes (response certainty) increased similarly in the short term and long term. Feedback effects were not moderated by participants' overall response certainty during learning, their prior knowledge, and the difficulty of quiz items. Moreover, we found that participants perceived the quiz app to be similarly attractive, interesting, and enjoyable in both feedback conditions and that they spent slightly more time to process quiz items in the lab setting. We discuss these results in detail, including the role of moderating and mediating factors and prospects for further research and practice. Overall, our results underline that quiz apps are useful and effective tools that can support the acquisition and retention of semantic knowledge in different learning settings.

Differences in brain signal complexity between experts and novices when solving conceptual science problem: a functional near-infrared spectroscopy study

Assessing the result of conceptual change (i.e., whether an individual has come to correctly understand a science concept) is important in science education, however traditional assessment methods lack objectivity. In this study, permutation entropy (PE) based complexity, a kind of entropy used to quantify the complexity describing the uncertainty of time series, was explored by the functional near-infrared spectroscopy to seek an objective neurobiological indicator for this assessment. Two groups of participants, engineering students (classified as "experts") and humanities students (classified as "novices"), were tested on their conceptions to discriminate the speed of cars according to the animation, while the hemodynamic response was recorded over their inferior frontal gyrus (IFG). The activation analysis, PE based complexity analysis, and k-means clustering analysis were conducted. The results indicated that experts performed the task better than novices in behavioral performances, and PE values in the IFG were smaller for experts, especially in the right IFG. Furthermore, the k-means clustering analysis showed that the PE could be a feature to classify the students into two groups. It is concluded that the PE is a promising neurobiological indicator for assessment of this kind.

How Representational Pictures Enhance Students’ Performance and Test-Taking Pleasure in Low-Stakes Assessment

Abstract. Pictures are often used in standardized educational large-scale assessment (LSA), but their impact on test parameters has received little attention up until now. Even less is known about pictures’ affective effects on students in testing (i.e., test-taking pleasure and motivation). However, such knowledge is crucial for a focused application of multiple representations in LSA. Therefore, this study investigated how adding representational pictures (RPs) to text-based item stems affects (1) item difficulty and (2) students’ test-taking pleasure. An experimental study with N = 305 schoolchildren was conducted, using 48 manipulated parallel science items (text-only vs. text-picture) in a rotated multimatrix design to realize within-subject measures. Students’ general cognitive abilities, reading abilities, and background variables were assessed to consider potential interactions between RPs’ effects and students’ performance. Students also rated their item-solving pleasure for each item. Results from item-response theory (IRT) model comparisons showed that RPs only reduced item difficulty when pictures visualized information mandatory for solving the task, while RPs substantially enhanced students’ test-taking pleasure even when they visualized optional context information. Overall, our findings suggest that RPs have a positive cognitive and affective influence on students’ performance in LSA (i.e., multimedia effect in testing) and should be considered more frequently.