Exploring action dynamics as an index of paired-associate learning

Explicit and implicit cognitive processes of the public towards people who stutter

PURPOSE

The Public Opinion Survey of Human Attributes - Stuttering (POSHA-S, St. Louis, 2013) was developed as a standard measure of public attitudes about people who stutter. As with any survey-based methods, threats to validity may occur because of social desirability bias. Using computer mouse-tracking, we were interested in observing changes in cognition that are manifested in intentionality through action by evaluating underlying cognitive processes that drive social judgments of people who stutter.

METHODS

Twenty-two women, 1 non-binary person, and 47 men reported using a computer mouse to complete an online, remote, and modified version of the POSHA-S. Responses were categorized as correct/helpful or incorrect/unhelpful relative to each component of the POSHA-S and were used as measures of explicit cognitive processes. Computer-mouse trajectory metrics, including area under the curve (AUC) and reaction time (RT), were used to measure implicit cognitive processes.

RESULTS

Although participants' explicit responses were significantly more likely to be correct/helpful than incorrect/unhelpful, with endorsement of correct/helpful prompts 77 % of the time, participants also endorsed incorrect/unhelpful prompts more than half (i.e., 52 %) of the time. Familiarity with people who stutter was associated with disagreeing with incorrect/unhelpful prompts. As indicated by greater AUC, participants exhibited significantly more implicit cognitive processes indicating competition when responding "disagree" compared to "agree", regardless of whether the prompts were correct/helpful or incorrect/unhelpful. Similarly, participants took significantly longer to respond to prompts with "disagree" rather than "agree".

CONCLUSION

The findings of this study offer evidence of participants reporting cognitive processes that are overall more correct/helpful than incorrect/unhelpful, in their explicit responses to the dichotomous response tasks of the POSHA-S. However, these findings are tempered by evidence of a tendency to agree with statements in the measure and suggest the need for further research to increase understanding of how to measure and improve explicit and implicit cognitive processes related to people who stutter.

Evidence for entropy maximisation in human free choice behaviour

The freedom to choose between options is strongly linked to notions of free will. Accordingly, several studies have shown that individuals demonstrate a preference for choice, or the availability of multiple options, over and above utilitarian value. Yet we lack a decision-making framework that integrates preference for choice with traditional utility maximisation in free choice behaviour. Here we test the predictions of an inference-based model of decision-making in which an agent actively seeks states yielding entropy (availability of options) in addition to utility (economic reward). We designed a study in which participants freely navigated a virtual environment consisting of two consecutive choices leading to reward locations in separate rooms. Critically, the choice of one room always led to two final doors while, in the second room, only one door was permissible to choose. This design allowed us to separately determine the influence of utility and entropy on participants' choice behaviour and their self-evaluation of free will. We found that choice behaviour was better predicted by an inference-based model than by expected utility alone, and that both the availability of options and the value of the context positively influenced participants' perceived freedom of choice. Moreover, this consideration of options was apparent in the ongoing motion dynamics as individuals navigated the environment. In a second study, in which participants selected between rooms that gave access to three or four doors, we observed a similar pattern of results, with participants preferring the room that gave access to more options and feeling freer in it. These results suggest that free choice behaviour is well explained by an inference-based framework in which both utility and entropy are optimised and supports the idea that the feeling of having free will is tightly related to options availability.

Gender stereotypes drive perceptual differences of vocal confidence

One's ability to express confidence is critical to achieve one's goals in a social context-such as commanding respect from others, establishing higher social status, and persuading others. How individuals perceive confidence may be shaped by the socio-indexical cues produced by the speaker. In the current production/perception study, we asked four speakers (two cisgender women/men) to answer trivia questions under three speaking contexts: natural, overconfident, and underconfident (i.e., lack of confidence). An evaluation of the speakers' acoustics indicated that the speakers significantly varied their acoustic cues as a function of speaking context and that the women and men had significantly different acoustic cues. The speakers' answers to the trivia questions in the three contexts (natural, overconfident, underconfident) were then presented to listeners (N = 26) in a social judgment task using a computer mouse-tracking paradigm. Listeners were sensitive to the speakers' acoustic modulations of confidence and differentially interpreted these cues based on the perceived gender of the speaker, thereby impacting listeners' cognition and social decision making. We consider, then, how listeners' social judgments about confidence were impacted by gender stereotypes about women and men from social, heuristic-based processes.

Science with Duplo: Multilevel goal management in preschoolers' toy house constructions

Executing goal-directed action sequences is fundamental to our behavior. Planning and controlling these action sequences improves greatly over the preschool years. In this study, we examined preschoolers' ability to plan action sequences. A total of 69 3- to 5-year-olds were assessed on an action sequence planning task with a hierarchical goal structure and on several executive function tasks. Planning abilities improved with age. Improvements in inhibition were related to avoidance of actions irrelevant to the goal hierarchy. Updating skill appears to be associated with executing actions relevant to different subgoals. Using optical motion capture, we showed that children who followed the subgoals displayed less movement with their nonreaching hand within a subgoal. This effect was enhanced in children with better inhibitory skills, suggesting that such skills allow greater focus on executing the current subgoal. Thus, we provide evidence that structuring of subgoals in action sequence planning emerges during the preschool years and that improvements in performance in action sequence planning are related to executive functions.

Cognitive Processing of Miscommunication in Interactive Listening: An Evaluation of Listener Indecision and Cognitive Effort

Purpose The purpose of the current study was to evaluate the social and cognitive underpinnings of miscommunication during an interactive listening task. Method An eye and computer mouse-tracking visual-world paradigm was used to investigate how a listener's cognitive effort (local and global) and decision-making processes were affected by a speaker's use of ambiguity that led to a miscommunication. Results Experiments 1 and 2 found that an environmental cue that made a miscommunication more or less salient impacted listener language processing effort (eye-tracking). Experiment 2 also indicated that listeners may develop different processing heuristics dependent upon the speaker's use of ambiguity that led to a miscommunication, exerting a significant impact on cognition and decision making. We also found that perspective-taking effort and decision-making complexity metrics (computer mouse tracking) predict language processing effort, indicating that instances of miscommunication produced cognitive consequences of indecision, thinking, and cognitive pull. Conclusion Together, these results indicate that listeners behave both reciprocally and adaptively when miscommunications occur, but the way they respond is largely dependent upon the type of ambiguity and how often it is produced by the speaker.

Social Judgments of Digitally Manipulated Stuttered Speech: Cognitive Heuristics Drive Implicit and Explicit Bias

Purpose People who stutter are susceptible to discrimination, stemming from negative stereotypes and social misattributions. There has been a recent push to evaluate the underlying explicit and implicit cognitive mechanisms associated with social judgments, moving away from only evaluating explicit social bias about people who stutter. The purpose of the current study was to evaluate how listeners change their implicit and explicit social (mis)attributions after hearing a people who stutter produce disfluent speech. Method The current project was an adaptation of the Byrd et al. (2017) study to evaluate listener implicit/explicit social judgments of stuttered speech across five categories (i.e., confidence, friendliness, intelligence, distractibility, and extroversion) before and after a stuttering self-disclosure. This was done by implementing a modified version of the Ferguson et al. (2019) computer mouse-tracking paradigm. Results Consistent with previous findings, participants made more explicit positive social judgments of confidence, friendliness, extroversion, and intelligence after a stuttering self-disclosure, but the distractedness category was resistant to change. Also consistent with previous findings, participants experienced a higher degree of cognitive competition (i.e., higher area under the curve) shortly after self-disclosure, which lessened over time. Conclusions Explicit and implicit biases exist, but self-disclosure significantly impacts the cognitive system of listeners. Specifically, self-disclosure may reduce explicit bias through experience and explicit belief updating, but when cognitive heuristics are strong, implicit bias may be slower to change.

Social Judgments of Digitally Manipulated Stuttered Speech: An Evaluation of Self-Disclosure on Cognition

Purpose Persons who stutter (PWS) may be susceptible to discrimination because of negative judgments made by listeners. The current study sought to determine how the cognitive system's explicit (i.e., conscious) and implicit (i.e., nonconscious) biases about PWS are impacted by self-disclosure. Method A computer mouse-tracking paradigm was used to evaluate categorical social judgments about PWS. Computer mouse trajectories, which have been shown to reveal underlying cognitive pull or competition between opposing concepts, were used to measure implicit bias (i.e., nonconscious stereotypes). Participants were asked to explicitly categorize the speaker as either intelligent or unintelligent before and after listening to a speaker self-disclose. Mouse cursor trajectories during the explicit response categorization were used to evaluate implicit bias associated with the decision-making process. Results Results indicated that participants chose "intelligent" for a higher proportion of the trials in the disclosure condition compared to baseline, showing that listeners' explicit biases changed after listening to a self-disclosure that the speaker stutters. Results also indicated listeners exhibited a more negative implicit bias, based on computer mouse trajectories, when rating the PWS relative to the "persons who do not stutter" talker, but this negative implicit bias did seem to reduce over time after the disclosure was made. Conclusions These findings indicate that, even though explicit and implicit biases were evident when listeners heard stuttering, both explicit and implicit biases seemed to extinguish over time after a self-disclosure. Although the bias was not completely extinguished, these results provide promising evidence toward developing methods to reduce negative beliefs and reactions toward PWS. Supplemental Material http://osf.io/mwrp7/.

Unpacking decision difficulty: Testing action dynamics in Intertemporal, gamble, and consumer choices

The study examined the two-factor structure of decision difficulty proposed by Cheng and González-Vallejo (2017) in new domains, and the role of numeracy in relation to these factors. Using the measurement methodology of 'mouse' (cursor) movements, participants' temporal and spatial measures were recorded when making decisions in the domains of intertemporal, gamble, and consumer choices. Task manipulations designed to affect difficulty included the sign of the payoffs (gains vs. losses), the similarity of the attribute values being compared, and attribute importance. A psychometric analysis of the measures revealed three orthogonal components, two of which, conflict and wavering, described decision difficulty. The conflict component was most affected by changes in the sign of the payoffs of intertemporal and gamble choices, with greater means observed in the loss than in the gain context. By contrast, the wavering component was most affected by changes of the similarity between the options' attributes, with greater means when the options were more similar. The study also found that choosing the long-term advantageous options in an intertemporal choice task; choosing the riskier gain and safer loss in a gamble choice task; and choosing the more expensive/better-quality hotel in a consumer choice task demonstrated greater conflict and/or wavering. The study further found that numeracy, or the degree to which individuals are able to use and interpret numbers, was negatively related to the conflict component. Taken together, the study demonstrated that decision difficulty varied with contextual changes, and action-dynamic measures reflected different facets of decision difficulty.

Doing psychological science by hand

Over the past decade, mouse-tracking in choice tasks has become a popular method across psychological science. This method exploits hand movements as a measure of multiple response activations that can be tracked continuously over hundreds of milliseconds. Whereas early mouse-tracking research focused on specific debates, researchers have realized the methodology has far broader theoretical value. This more recent work demonstrates that mouse-tracking is a widely applicable measure across the field, capable of exposing the micro-structure of real-time decisions including their component processes and millisecond-resolution time-course in ways that inform theory. In the article, recent advances in the mouse-tracking approach are described, and comparisons with the gold standard measure of reaction time and other temporally-sensitive methodologies are provided. Future directions, including mapping to neural representations with brain-imaging and ways to improve our theoretical understanding of mouse-tracking methodology, are discussed.

When do arrows start to compete? A developmental mouse-tracking study

Recent work in adults has suggested that the strength of social and symbolic cues not presented at fixation (but allowing eye movements to the cue) may be determined less by their biological relevance and more by the distinctiveness of the shape of the cue. The present study examines whether these results extend to children, who may differ in their relative exposure to symbolic cues (arrows) compared to social cues. Children aged 3 to 11 were presented with congruent or incongruent pairs of cues (line drawings of gazing eyes, pointing hands, and arrows) and were asked to indicate the direction of the target cue (indicated at the start of the block) by moving the mouse towards the response box indicating its direction. Results show a similar advantage for arrows and pointing hands in young children as previously found in adults, suggesting that cue shape trumps biological relevance for cues away from fixation from an early age.

Mouse-tracking evidence for parallel anticipatory option evaluation

In fast-paced, dynamic tasks, the ability to anticipate the future outcome of a sequence of events is crucial to quickly selecting an appropriate course of action among multiple alternative options. There are two classes of theories that describe how anticipation occurs. Serial theories assume options are generated and evaluated one at a time, in order of quality, whereas parallel theories assume simultaneous generation and evaluation. The present research examined the option evaluation process during a task designed to be analogous to prior anticipation tasks, but within the domain of narrative text comprehension. Prior research has relied on indirect, off-line measurement of the option evaluation process during anticipation tasks. Because the movement of the hand can provide a window into underlying cognitive processes, online metrics such as continuous mouse tracking provide more fine-grained measurements of cognitive processing as it occurs in real time. In this study, participants listened to three-sentence stories and predicted the protagonists' final action by moving a mouse toward one of three possible options. Each story was presented with either one (control condition) or two (distractor condition) plausible ending options. Results seem most consistent with a parallel option evaluation process because initial mouse trajectories deviated further from the best option in the distractor condition compared to the control condition. It is difficult to completely rule out all possible serial processing accounts, although the results do place constraints on the time frame in which a serial processing explanation must operate.

Decision landscapes: visualizing mouse-tracking data

Computerized paradigms have enabled gathering rich data on human behaviour, including information on motor execution of a decision, e.g. by tracking mouse cursor trajectories. These trajectories can reveal novel information about ongoing decision processes. As the number and complexity of mouse-tracking studies increase, more sophisticated methods are needed to analyse the decision trajectories. Here, we present a new computational approach to generating decision landscape visualizations based on mouse-tracking data. A decision landscape is an analogue of an energy potential field mathematically derived from the velocity of mouse movement during a decision. Visualized as a three-dimensional surface, it provides a comprehensive overview of decision dynamics. Employing the dynamical systems theory framework, we develop a new method for generating decision landscapes based on arbitrary number of trajectories. This approach not only generates three-dimensional illustration of decision landscapes, but also describes mouse trajectories by a number of interpretable parameters. These parameters characterize dynamics of decisions in more detail compared with conventional measures, and can be compared across experimental conditions, and even across individuals. The decision landscape visualization approach is a novel tool for analysing mouse trajectories during decision execution, which can provide new insights into individual differences in the dynamics of decision making.

PsyGlass: Capitalizing on Google Glass for naturalistic data collection

As commercial technology moves further into wearable technologies, cognitive and psychological scientists can capitalize on these devices to facilitate naturalistic research designs while still maintaining strong experimental control. One such wearable technology is Google Glass (Google, Inc.: www.google.com/glass), which can present wearers with audio and visual stimuli while tracking a host of multimodal data. In this article, we introduce PsyGlass, a framework for incorporating Google Glass into experimental work that is freely available for download and community improvement over time (www.github.com/a-paxton/PsyGlass). As a proof of concept, we use this framework to investigate dual-task pressures on naturalistic interaction. The preliminary study demonstrates how designs from classic experimental psychology may be integrated in naturalistic interactive designs with emerging technologies. We close with a series of recommendations for using PsyGlass and a discussion of how wearable technology more broadly may contribute to new or adapted naturalistic research designs.

A Perceptual Pathway to Bias: Interracial Exposure Reduces Abrupt Shifts in Real-Time Race Perception That Predict Mixed-Race Bias

In two national samples, we examined the influence of interracial exposure in one's local environment on the dynamic process underlying race perception and its evaluative consequences. Using a mouse-tracking paradigm, we found in Study 1 that White individuals with low interracial exposure exhibited a unique effect of abrupt, unstable White-Black category shifting during real-time perception of mixed-race faces, consistent with predictions from a neural-dynamic model of social categorization and computational simulations. In Study 2, this shifting effect was replicated and shown to predict a trust bias against mixed-race individuals and to mediate the effect of low interracial exposure on that trust bias. Taken together, the findings demonstrate that interracial exposure shapes the dynamics through which racial categories activate and resolve during real-time perceptions, and these initial perceptual dynamics, in turn, may help drive evaluative biases against mixed-race individuals. Thus, lower-level perceptual aspects of encounters with racial ambiguity may serve as a foundation for mixed-race prejudice.

Decisions in Motion: Decision Dynamics during Intertemporal Choice reflect Subjective Evaluation of Delayed Rewards

People tend to discount rewards or losses that occur in the future. Such delay discounting has been linked to many behavioral and health problems, since people choose smaller short-term gains over greater long-term gains. We investigated whether the effect of delays on the subjective value of rewards is expressed in how people move when they make choices. Over 600 patrons of the RISK LAB exhibition hosted by the Science Gallery Dublin^TM played a short computer game in which they used a computer mouse to choose between amounts of money at various delays. Typical discounting effects were observed and decision dynamics indicated that choosing smaller short-term rewards became easier (i.e., shorter response times, tighter trajectories, less vacillation) as the delays until later rewards increased. Based on a sequence of choices, subjective values of delayed outcomes were estimated and decision dynamics during initial choices predicted these values. Decision dynamics are affected by subjective values of available options and thus provide a means to estimate such values.

Abrupt category shifts during real-time person perception

Previous studies have suggested that real-time person perception relies on continuous competition, in which partially active categories smoothly compete over time. Here, two studies demonstrated the involvement of a different kind of competition. In Study 1, before participants selected the correct sex category for morphed faces, their mouse trajectories often exhibited a continuous attraction toward the incorrect category that increased with sex-category ambiguity, indicating continuous competition. On other trials, however, trajectories initially pursued the incorrect category and then abruptly redirected toward the correct category, suggesting early incorrect category activation that was rapidly reversed later in processing. These abrupt category reversals also increased with ambiguity. In Study 2, participants were presented with faces containing a sex-typical or sex-atypical hair cue, in a context in which the norm was either sex-typical targets (normative context) or sex-atypical targets (counternormative context). Sex-atypical targets induced greater competition in the normative context, but sex-typical targets induced greater competition in the counternormative context. Together, these results demonstrate that categorizing others involves both smooth competition and abrupt category shifts, and that these flexibly adapt to the social context.

Advanced mouse-tracking analytic techniques for enhancing psychological science

Computer mouse-tracking is a relatively recently developed behavioral methodology that can contribute unique insight into a wide variety of psychological phenomena. By recording mouse movements en route to specific responses on a screen, researchers glean continuous information about tentative commitments to multiple response alternatives over time. This approach yields a richness of data that can be fully explored with a variety of sophisticated analytic techniques, but these approaches are relatively underutilized and can be difficult to adopt. Here we describe several techniques for researchers to examine the onset and timing of evolving decision processes; test the degree of response competition at different time points; assess trajectory complexity with spatial disorder analyses; identify qualitatively distinct psychological processes during response generation; and finally to distill unique and meaningful components from mouse-tracking data for subsequent analysis. With this guide, we hope researchers can address novel hypotheses otherwise inaccessible with more traditional methods.

Local dynamics in decision making: The evolution of preference within and across decisions

Within decisions, perceived alternatives compete until one is preferred. Across decisions, the playing field on which these alternatives compete evolves to favor certain alternatives. Mouse cursor trajectories provide rich continuous information related to such cognitive processes during decision making. In three experiments, participants learned to choose symbols to earn points in a discrimination learning paradigm and the cursor trajectories of their responses were recorded. Decisions between two choices that earned equally high-point rewards exhibited far less competition than decisions between choices that earned equally low-point rewards. Using positional coordinates in the trajectories, it was possible to infer a potential field in which the choice locations occupied areas of minimal potential. These decision spaces evolved through the experiments, as participants learned which options to choose. This visualisation approach provides a potential framework for the analysis of local dynamics in decision-making that could help mitigate both theoretical disputes and disparate empirical results.

Using action dynamics to assess competing stimulus control during stimulus equivalence testing

Previous studies have identified potential sources of competing stimulus control in tests for stimulus equivalence. The present experiment employed the Nintendo Wii remote (Wiimote®) to investigate whether such competition would affect suboperant action dynamics (e.g., topographies of equivalence responses). Following one-to-many training on conditional discriminations sufficient to establish three 3-member equivalence classes, participants were presented with a test for equivalence responding that included five different trial types. These included "traditional" equivalence trials, on which the incorrect stimulus had previously been presented as a correct comparison stimulus during training, trials on which a novel unrelated word was provided as the incorrect comparison, and trials on which the incorrect stimulus varied in orthographical and phonological similarity to the sample stimulus. The presence of phonological and orthographic distractor stimuli significantly reduced the probability of equivalence-consistent responding, relative to neutral distractors, but this effect was almost exclusively seen in participants who failed to demonstrate equivalence on traditional equivalence trials. Analyses of correct response trajectories suggested that the prior history of reinforcement for choosing the incorrect stimulus on the traditional equivalence trial gave rise to greater competition than did phonological or orthographic similarity between the sample and incorrect comparisons.

The response dynamics of preferential choice

The ubiquity of psychological process models requires an increased degree of sophistication in the methods and metrics that we use to evaluate them. We contribute to this venture by capitalizing on recent work in cognitive science analyzing response dynamics, which shows that the bearing information processing dynamics have on intended action is also revealed in the motor system. This decidedly "embodied" view suggests that researchers are missing out on potential dependent variables with which to evaluate their models-those associated with the motor response that produces a choice. The current work develops a method for collecting and analyzing such data in the domain of decision making. We first validate this method using widely normed stimuli from the International Affective Picture System (Experiment 1), and demonstrate that curvature in response trajectories provides a metric of the competition between choice options. We next extend the method to risky decision making (Experiment 2) and develop predictions for three popular classes of process model. The data provided by response dynamics demonstrate that choices contrary to the maxim of risk seeking in losses and risk aversion in gains may be the product of at least one "online" preference reversal, and can thus begin to discriminate amongst the candidate models. Finally, we incorporate attentional data collected via eye-tracking (Experiment 3) to develop a formal computational model of joint information sampling and preference accumulation. In sum, we validate response dynamics for use in preferential choice tasks and demonstrate the unique conclusions afforded by response dynamics over and above traditional methods.

Cognitive processing and motor execution in the lexical decision task: a developmental study

We investigated lexical decision making in children and adults by analyzing spatiotemporal characteristics of responses involving a hand movement. Children's and adults' movement trajectories were assessed in three tasks: a lexical decision task (LDT), a pointing task that involved minimal cognitive processing, and a symbol task requiring a simple binary decision. Cognitive interference on motor performance was quantified by analyzing movement characteristics in the LDT and symbol task relative to the pointing task. Across age groups, movements in the LDT were less smooth, slower, and more strongly curved to the opposite response option, and these interference effects decreased steadily with age. Older children showed stronger interference effects than did adults, even though their reaction times were similar to adults' performance. No comparable effects were found in the symbol task, indicating that task characteristics such as response mapping and decision selection alone are not able to explain the developmental differences observed in the LDT. Our results indicate substantial overlap between cognitive processing and motor execution in the LDT in children that is not captured by computational models of visual word recognition and cognitive development.

An assessment of the temporal dynamics of moral decisions

In the domain of moral decision making, models in which emotion and deliberation constitute competing dual-systems have become increasingly popular. Currently, the favored explanation of this interaction is what Evans (2008) termed a “default-interventionist” (DI) process where moral decisions are the result of a prepotent emotional response, which can be overridden with substantial deliberative effort. Although this “emotion-then-deliberation” sequence is often assumed, existing methods have lacked the requisite process resolution to clearly depict the nature of this interaction. The present work utilized continuous mouse tracking, or response dynamics, to develop and test predictions of these DI models of moral decision making. Study 1 utilized previously published moral dilemmas to validate the method for use with such complex stimuli. Although the data replicated typical choice and RT patterns, the process metrics provided by the response trajectories did not demonstrate the online preference reversals predicted by DI models. Study 2 utilized more rigorously constructed stimuli and an alternative presentation format to provide the strongest possible test of DI predictions, but again failed to show the predicted reversals. In summary, neither experiment provided data in accordance with the predictions of popular DI dual-systems models, which suggests that researchers should consider models allowing for concurrent activation of deliberative and emotional systems, or reconceptualize moral decisions within the typical multiattribute decision framework.

Primate drum kit: a system for studying acoustic pattern production by non-human primates using acceleration and strain sensors

The possibility of achieving experimentally controlled, non-vocal acoustic production in non-human primates is a key step to enable the testing of a number of hypotheses on primate behavior and cognition. However, no device or solution is currently available, with the use of sensors in non-human animals being almost exclusively devoted to applications in food industry and animal surveillance. Specifically, no device exists which simultaneously allows: (i) spontaneous production of sound or music by non-human animals via object manipulation, (ii) systematical recording of data sensed from these movements, (iii) the possibility to alter the acoustic feedback properties of the object using remote control. We present two prototypes we developed for application with chimpanzees (Pan troglodytes) which, while fulfilling the aforementioned requirements, allow to arbitrarily associate sounds to physical object movements. The prototypes differ in sensing technology, costs, intended use and construction requirements. One prototype uses four piezoelectric elements embedded between layers of Plexiglas and foam. Strain data is sent to a computer running Python through an Arduino board. A second prototype consists in a modified Wii Remote contained in a gum toy. Acceleration data is sent via Bluetooth to a computer running Max/MSP. We successfully pilot tested the first device with a group of chimpanzees. We foresee using these devices for a range of cognitive experiments.

Memory in motion: movement dynamics reveal memory strength

Recognition memory is typically examined as a discrete end-state, describable by static variables, such as accuracy, response time, and confidence. In the present study, we combined real-time mouse-tracking with subsequent, overt confidence estimates to examine the dynamic nature of memory decisions. By examining participants' streaming x-, y- mouse coordinates during recognition decisions, we observed that movement trajectories revealed underlying response confidence. More confident decisions were associated with shorter decision times and more linear response trajectories. Less confident decisions were made slowly, with increased trajectory curvature. Statistical indices of curvature and decision times, including area-under-the-curve and time to maximum deviation, suggested that memory strength relates to response dynamics. Whether participants were correct or incorrect, old responses showed a stronger correspondence between mouse trajectories and confidence, relative to new responses. We suggest that people subjectively experience a correspondence between feelings of memory and feelings of confidence; that subjective experience reveals itself in real-time decision processes, as suggested by sequential sampling models of recognition decisions.

Multimodal behavior and interaction as indicators of cognitive load

High cognitive load arises from complex time and safety-critical tasks, for example, mapping out flight paths, monitoring traffic, or even managing nuclear reactors, causing stress, errors, and lowered performance. Over the last five years, our research has focused on using the multimodal interaction paradigm to detect fluctuations in cognitive load in user behavior during system interaction. Cognitive load variations have been found to impact interactive behavior: by monitoring variations in specific modal input features executed in tasks of varying complexity, we gain an understanding of the communicative changes that occur when cognitive load is high. So far, we have identified specific changes in: speech, namely acoustic, prosodic, and linguistic changes; interactive gesture; and digital pen input, both interactive and freeform. As ground-truth measurements, galvanic skin response, subjective, and performance ratings have been used to verify task complexity.

The data suggest that it is feasible to use features extracted from behavioral changes in multiple modal inputs as indices of cognitive load. The speech-based indicators of load, based on data collected from user studies in a variety of domains, have shown considerable promise. Scenarios include single-user and team-based tasks; think-aloud and interactive speech; and single-word, reading, and conversational speech, among others. Pen-based cognitive load indices have also been tested with some success, specifically with pen-gesture, handwriting, and freeform pen input, including diagraming. After examining some of the properties of these measurements, we present a multimodal fusion model, which is illustrated with quantitative examples from a case study.

The feasibility of employing user input and behavior patterns as indices of cognitive load is supported by experimental evidence. Moreover, symptomatic cues of cognitive load derived from user behavior such as acoustic speech signals, transcribed text, digital pen trajectories of handwriting, and shapes pen, can be supported by well-established theoretical frameworks, including O'Donnell and Eggemeier's workload measurement [1986] Sweller's Cognitive Load Theory [Chandler and Sweller 1991], and Baddeley's model of modal working memory [1992] as well as McKinstry et al.'s [2008] and Rosenbaum's [2005] action dynamics work. The benefit of using this approach to determine the user's cognitive load in real time is that the data can be collected implicitly that is, during day-to-day use of intelligent interactive systems, thus overcomes problems of intrusiveness and increases applicability in real-world environments, while adapting information selection and presentation in a dynamic computer interface with reference to load.

Dynamics of cognition

The application of dynamical systems methods and concepts to cognitive phenomena has broadened the range of testable hypotheses and theoretical narratives available to cognitive scientists. Most research in cognitive dynamics tests the degree to which observed cognitive performance is consistent with one or another core phenomena associated with complex dynamical systems, such as tests for phase transitions, coupling among processes, or scaling laws. Early applications of dynamical systems theory to perceptual-motor performance and developmental psychology paved the way for more recent applications of dynamical systems analyses, models, and theoretical concepts in areas such as learning, memory, speech perception, decision making, problem solving, and reading, among others. Reviews of the empirical results of both foundational and contemporary cognitive dynamics are provided. WIREs Cogn Sci 2012. doi: 10.1002/wcs.1200 For further resources related to this article, please visit the WIREs website.

Response dynamics: A new window on the decision process

The history of judgment and decision making is defined by a trend toward increasingly nuanced explanations of the decision making process. Recently, process models have become incredibly sophisticated, yet the tools available to directly test these models have not kept pace. These increasingly complex process models require increasingly complex process data by which they can be adequately tested. We propose a new class of data collection that will facilitate evaluation of sophisticated process models. Tracking mouse paths during a continuous response provides an implicit measure of the growth of preference that produces a choice—rather than the current practice of recording just the button press that indicates that choice itself. Recent research in cognitive science (Spivey & Dale, 2006) has shown that cognitive processing can be revealed in these dynamic motor responses. Unlike current process methodologies, these response dynamics studies can demonstrate continuous competition between choice options and even online preference reversals. Here, in order to demonstrate the mechanics and utility of the methodology, we present an example response dynamics experiment utilizing a common multi-alternative decision task.

Hand in motion reveals mind in motion

Recently, researchers have measured hand movements en route to choices on a screen to understand the dynamics of a broad range of psychological processes. We review this growing body of research and explain how manual action exposes the real-time unfolding of underlying cognitive processing. We describe how simple hand motions may be used to continuously index participants’ tentative commitments to different choice alternatives during the evolution of a behavioral response. As such, hand-tracking can provide unusually high-fidelity, real-time motor traces of the mind. These motor traces cast novel theoretical and empirical light onto a wide range of phenomena and serve as a potential bridge between far-reaching areas of psychological science – from language, to high-level cognition and learning, to social cognitive processes.

The real-time link between person perception and action: brain potential evidence for dynamic continuity

Using event-related potentials, we investigated how the brain extracts information from another's face and translates it into relevant action in real time. In Study 1, participants made between-hand sex categorizations of sex-typical and sex-atypical faces. Sex-atypical faces evoked negativity between 250 and 550 ms (N300/N400 effects), reflecting the integration of accumulating sex-category knowledge into a coherent sex-category interpretation. Additionally, the lateralized readiness potential revealed that the motor cortex began preparing for a correct hand response while social category knowledge was still gradually evolving in parallel. In Study 2, participants made between-hand eye-color categorizations as part of go/no-go trials that were contingent on a target's sex. On no-go trials, although the hand did not actually move, information about eye color partially prepared the motor cortex to move the hand before perception of sex had finalized. Together, these findings demonstrate the dynamic continuity between person perception and action, such that ongoing results from face processing are immediately and continuously cascaded into the motor system over time. The preparation of action begins based on tentative perceptions of another's face before perceivers have finished interpreting what they just saw.

MouseTracker: software for studying real-time mental processing using a computer mouse-tracking method

In the present article, we present a software package, MouseTracker, that allows researchers to use a computer mouse-tracking method for assessing real-time processing in psychological tasks. By recording the streaming x-, y-coordinates of the computer mouse while participants move the mouse into one of multiple response alternatives, motor dynamics of the hand can reveal the time course of mental processes. MouseTracker provides researchers with fine-grained information about the real-time evolution of participant responses by sampling 60-75 times/sec the online competition between multiple response alternatives. MouseTracker allows researchers to develop and run experiments and subsequently analyze mouse trajectories in a user-interactive, graphics-based environment. Experiments may incorporate images, letter strings, and sounds. Mouse trajectories can be processed, averaged, visualized, and explored, and measures of spatial attraction/curvature, complexity, velocity, and acceleration can be computed. We describe the software and the method, and we provide details on mouse trajectory analysis. We validate the software by demonstrating the accuracy and reliability of its trajectory and reaction time data. The latest version of MouseTracker is freely available at http://mousetracker.jbfreeman.net.

Lévy-like diffusion in eye movements during spoken-language comprehension

This study explores the diffusive properties of human eye movements during a language comprehension task. In this task, adults are given auditory instructions to locate named objects on a computer screen. Although it has been convention to model visual search as standard Brownian diffusion, we find evidence that eye movements are hyperdiffusive. Specifically, we use comparisons of maximum-likelihood fit as well as standard deviation analysis and diffusion entropy analysis to show that visual search during language comprehension exhibits Lévy-like rather than Gaussian diffusion.