The planning and control model (PCM) of motorvisual priming: reconciling motorvisual impairment and facilitation effects

Modal and amodal cognition: an overarching principle in various domains of psychology

Accounting for how the human mind represents the internal and external world is a crucial feature of many theories of human cognition. Central to this question is the distinction between modal as opposed to amodal representational formats. It has often been assumed that one but not both of these two types of representations underlie processing in specific domains of cognition (e.g., perception, mental imagery, and language). However, in this paper, we suggest that both formats play a major role in most cognitive domains. We believe that a comprehensive theory of cognition requires a solid understanding of these representational formats and their functional roles within and across different domains of cognition, the developmental trajectory of these representational formats, and their role in dysfunctional behavior. Here we sketch such an overarching perspective that brings together research from diverse subdisciplines of psychology on modal and amodal representational formats so as to unravel their functional principles and their interactions.

Perception and action as viewed from the Theory of Event Coding: a multi-lab replication and effect size estimation of common experimental designs

The Theory of Event Coding (TEC) has influenced research on action and perception across the past two decades. It integrates several seminal empirical phenomena and it has continued to stimulate novel experimental approaches on the representational foundations of action control and perceptual experience. Yet, many of the most notable results surrounding TEC originate from an era of psychological research that relied on rather small sample sizes as judged by today's standards. This state hampers future research aiming to build on previous phenomena. We, therefore, provide a multi-lab re-assessment of the following six classical observations: response-effect compatibility, action-induced blindness, response-effect learning, stimulus-response binding, code occupation, and short-term response-effect binding. Our major goal is to provide precise estimates of corresponding effect sizes to facilitate future scientific endeavors. These effect sizes turned out to be considerably smaller than in the original reports, thus allowing for informed decisions on how to address each phenomenon in future work. Of note, the most relevant results of the original observations were consistently obtained in the present experiments as well.

On doing multi-act arithmetic: A multitrait-multimethod approach of performance dimensions in integrated multitasking

Here we present a systematic plan to the experimental study of test-retest reliability in the multitasking domain, adopting the multitrait-multimethod (MTMM) approach to evaluate the psychometric properties of performance in Düker-type speeded multiple-act mental arithmetic. These form of tasks capacitate the experimental analysis of integrated multi-step processing by combining multiple mental operations in flexible ways in the service of the overarching goal of completing the task. A particular focus was on scoring methodology, particularly measures of response speed variability. To this end, we present data of two experiments with regard to (a) test-retest reliability, (b) between-measures correlational structure, (c) and stability (test-retest practice effects). Finally, we compared participants with high versus low performance variability to assess ability-related differences in measurement precision (typically used as proxy to "simulate" patient populations), which is especially relevant in the applied fields of clinical neuropsychology. The participants performed two classic integrated multi-act arithmetic tasks, combining addition and verification (Exp. 1) and addition and comparison (Exp. 2). The results revealed excellent test-retest reliability for the standard and the variability measures. The analysis of between-measures correlational structure revealed the typical pattern of convergent and discriminant relationships, and also, that absolute response speed variability was highly correlated with average speed (r > 0.85), indicating that these measures mainly deliver redundant information. In contrast, speed-adjusted (relativized) variability revealed discriminant validity being correlated to a much lesser degree with average speed, indicating that this measure delivers additional information not already provided by the speed measure. Furthermore, speed-adjusted variability was virtually unaffected by test-retest practice, which makes this measure interesting in situations with repeated testing.

Partial Repetition Costs are Reduced but not Eliminated with Practice

Often, we depart from an intended course of events to react to sudden situational demands (an intervening event) before resuming the originally planned action. Executing an action to an intervening event can be delayed if the features of this action plan partly overlap with an action plan retained in working memory (WM) compared to when they completely overlap or do not overlap. This delay is referred to as a partial repetition cost (PRC). PRCs are typically attributed to code confusion between action plans in WM. We tested this by training the component action plans extensively to reduce their reliance on WM. If PRCs are caused by code confusion within WM, then PRCs should be reduced and possibly eliminated with extensive practice. To test this, participants performed a partial repetition (PR) task after 0, 4 and 8.5 sessions of stimulus-response (S-R) training. In the PR task, participants saw two visual events. They retained an action to the first event while executing a speeded action to a second (intervening) event; afterwards, they executed the retained action. The two action plans either partly overlapped or did not overlap. Results showed that extensive (S-R and PR task) practice reduced but did not eliminate PRCs. A reduction in PRCs (code confusion) with practice is compatible with memory models that assume action events become more specific and less reliant on WM with practice. These findings merit expansions of PR tasks to other domains and broader conceptions of action plans that incorporate the formal structure of memory models.

Binding and Retrieval of Response Durations: Subtle Evidence for Episodic Processing of Continuous Movement Features

Re-encountering a stimulus retrieves nominally relevant, categorical response features related to previous action decisions in response to this stimulus. Whether binding and retrieval extend to nominally irrelevant, metric features relating to an actual body movement is unknown, however. In two experiments, we thus tested whether repeating target or distractor stimuli across trials retrieves the irrelevant duration of spatial responses to these stimuli. We found subtle indication of such retrieval by task-relevant target stimuli, suggesting that binding and retrieval also operate on metric features of a motor response. In contrast, there was no sign of binding and retrieval of metric features for distractor stimuli. We discuss these observations regarding the representation of action episodes during action-related decision making and during actual movement initiation and control.

Partial repetition between action plans delays responses to ideomotor compatible stimuli

Often one must depart from an intended course of events to react to sudden situational demands before resuming his or her original action retained in working memory. Retaining an action plan in working memory (WM) can delay or facilitate the execution of an intervening action when the action features of the two action plans partly overlap (partial repetition) compared to when they do not overlap. We investigated whether partial repetition costs (PRCs) or benefits (PRBs) occur when the intervening event is an ideomotor-compatible stimulus that is a biological representation of the response required by the participant. Participants viewed two visual events and retained an action plan to the first event (A) while executing a speeded response to the second, intervening event (B). In Experiment 1A, the two visual events were ideomotor compatible, non-ideomotor compatible (abstract), or one was ideomotor compatible, and the other abstract. Results showed PRCs for all event A-B stimulus combinations with reduced PRCs for intervening, ideomotor compatible events. In contrast to previous research, there was no evidence that ideomotor-compatible actions were automatic and bypassed the selection bottleneck. Experiment 1B confirmed PRCs for ideomotor compatible stimuli that more accurately mimicked the required response. Findings suggest that mechanisms for activating, selecting, and retaining action plans are similar between ideomotor compatible and abstract visual events. We conclude that PRCs occur in response to intervening events when action plans are generated offline and rely on WM, including those for ideomotor-compatible stimuli; but PRBs may be restricted to actions generated online. This conclusion is consistent with the perceptual-motor framework by Goodale and Milner (Trends in Neuroscience 15:22-25, 1992).

Task relevance determines binding of effect features in action planning

Action planning can be construed as the temporary binding of features of perceptual action effects. While previous research demonstrated binding for task-relevant, body-related effect features, the role of task-irrelevant or environment-related effect features in action planning is less clear. Here, we studied whether task-relevance or body-relatedness determines feature binding in action planning. Participants planned an action A, but before executing it initiated an intermediate action B. Each action relied on a body-related effect feature (index vs. middle finger movement) and an environment-related effect feature (cursor movement towards vs. away from a reference object). In Experiments 1 and 2, both effects were task-relevant. Performance in action B suffered from partial feature overlap with action A compared to full feature repetition or alternation, which is in line with binding of both features while planning action A. Importantly, this cost disappeared when all features were available but only body-related features were task-relevant (Experiment 3). When only the environment-related effect of action A was known in advance, action B benefitted when it aimed at the same (vs. a different) environment-related effect (Experiment 4). Consequently, the present results support the idea that task relevance determines whether binding of body-related and environment-related effect features takes place while the pre-activation of environment-related features without binding them primes feature-overlapping actions.

On perceptual biases in virtual object manipulation: Signal reliability and action relevance matter

This study examined the role of visual reliability and action relevance in mutual visual-proprioceptive attraction in a virtual grasping task. Participants initially enclosed either the width or the height of a visual rectangular object with two cursors controlled by the movements of the index finger and thumb. Then, either the height or the width of this object or the distance between the fingers was judged. The judgments of object's size were attracted by the felt finger distance, and, vice versa, the judged finger distance was attracted by the size of the grasped object. The impact of the proprioceptive information on object judgments increased, whereas the impact of visual object information on finger judgments decreased when the reliability of the visual stimulus was reduced. Moreover, the proprioceptive bias decreased for the action-relevant stimulus dimension as compared with the action-irrelevant stimulus dimension. These results indicate sensory integration of spatially separated sensory signals in the absence of any direct spatial or kinematic relation between them. We therefore suggest that the basic principles of sensory integration apply to the broad research field on perceptual-motor interactions as well as to many virtual interactions with external objects.

Visuomotor and motorvisual priming with different types of set-level congruency: evidence in support of ideomotor theory, and the planning and control model (PCM)

Perception can prime action (visuomotor priming), and action can prime perception (motorvisual priming). According to ideomotor theory both effects rely on the overlap of mental representations between perception and action. This implies that both effects get more pronounced the more features they share. We tested this hypothesis by employing in a motorvisual (Exp. 1) and in a visuomotor (Exp. 2) setting, three different pairs of left/right target stimuli (hand pictures, arrows, and words) varying in how strongly they overlap with the pair of left/right responses. For two stimulus pairs (hands and words) the hypothesis was confirmed: hand pictures share more features with the responses than words, consequently hand pictures produced a stronger visuomotor and a stronger motorvisual priming effect than words. However, arrow stimuli showed a different pattern: the temporal dynamics of both priming effects, as well as the direction of the effect seen in motorvisual priming, were significant but opposite to that of the hand and word stimuli. This suggests that the arrows' representations were not involved in ideomotor processes, and we propose instead that they were represented in a spatial or scalar fashion, outside the representations assumed in ideomotor theory. The results are discussed in the context of ideomotor theory, and the planning and control model of motorvisual priming.

What is a task? An ideomotor perspective

Although multitasking has been the subject of a large number of papers and experiments, the term task is still not well defined. In this opinion paper, we adopt the ideomotor perspective to define the term task and distinguish it from the terms goal and action. In our opinion, actions are movements executed by an actor to achieve a concrete goal. Concrete goals are represented as anticipated sensory consequences that are associated with an action in an ideomotor manner. Concrete goals are nested in a hierarchy of more and more abstract goals, which form the context of the corresponding action. Finally, tasks are depersonalized goals, i.e., goals that should be achieved by someone. However, tasks can be assigned to a specific person or group of persons, either by a third party or by the person or the group of persons themselves. By accepting this assignment, the depersonalized task becomes a personal goal. In our opinion, research on multitasking needs to confine its scope to the analysis of concrete tasks, which result in concrete goals as anticipated sensory consequences of the corresponding action. We further argue that the distinction between dual- and single-tasking is dependent on the subjective conception of the task assignment, the goal representation and previous experience. Finally, we conclude that it is not the tasks, but the performing of the tasks, i.e. the actions that cause costs in multitasking experiments.

Intentional binding of two effects

An action that produced an effect is perceived later in time compared to an action that did not produce an effect. Likewise, the effect of an action is perceived earlier in time compared to a stimulus that was not produced by an action. Despite numerous studies on this phenomenon-referred to as Intentional Binding effect (IB)-the underlying mechanisms are still not fully understood. Typically, IB is investigated in settings where the action produces just one single effect, whereas in everyday action contexts, it rather causes a sequence of effects before leading to the desired outcome. Therefore, we investigated IB of two consecutive effects. We observed substantially more IB of a first effect tone compared to a second tone. This pattern was observed for second tones that were temporally predictable (Exp. 1) or not (Exp. 2 and 3). Interestingly, the second tone yielded stronger IB when it was less delayed (Exp. 4). Thus, also an event occurring later in an unfolding action-effect sequence can be bound to its causing action, but it might be less bound to the action than a first effect. Instead of the fact that it is the second of two consecutive effects, this, however, rather seems to be influenced by the longer delay of a second and, therefore, later occurring effect.

How You Move Is What I See: Planning an Action Biases a Partner's Visual Search

Activating action representations can modulate perceptual processing of action-relevant dimensions, indicative of a common-coding of perception and action. When two or more agents work together in joint action, individual agents often need to consider not only their own actions and their effects on the world, but also predict the actions of a co-acting partner. If in these situations the action of a partner is represented in a functionally equivalent way to the agent's own actions, one may also expect interaction effects between action and perception across jointly acting individuals. The present study investigated whether the action of a co-acting partner may modulate an agent's perception. The "performer" prepared a grasping or pointing movement toward a physical target while the "searcher" performed a visual search task. The performer's planned action impaired the searcher's perceptual performance when the search target dimension was relevant to the performer's movement execution. These results demonstrate an action-induced modulation of perceptual processes across participants and indicate that agents represent their partner's action by employing the same perceptual system they use to represent an own action. We suggest that task representations in joint action operate along multiple levels of a cross-brain predictive coding system, which provides agents with information about a partner's actions when they coordinate to reach a common goal.

Mobilizing cognition for speeded action: try-harder instructions promote motivated readiness in the constant-foreperiod paradigm

We examined the effect of motivational readiness on cognitive performance. An important but still not sufficiently elaborated question is whether individuals can voluntarily increase cognitive efficiency for an impending target event, given sufficient preparation time. Within the framework of the constant-foreperiod design (comparing reaction time performance in blocks of short and long foreperiod intervals, FPs), we examined the effect of an instruction to try harder (instructional cue: standard vs. effort) in a choice-reaction task on performance speed and variability. Proceeding from previous theoretical considerations, we expected the instruction to speed-up processing irrespective of FP length, while error rate should be increased in the short-FP but decreased in the long-FP condition. Overall, the results confirmed this prediction. Importantly, the distributional (ex-Gaussian and delta plot) analysis revealed that the instruction to try harder decreased distributional skewness (i.e., longer percentiles were more affected), indicating that mobilization ensured temporal performance stability (persistence).

Visuospatial cueing by self-caused features: Orienting of attention and action-outcome associative learning

The effect of a salient visual feature in orienting spatial attention was examined as a function of the learned association between the visual feature and the observer's action. During an initial acquisition phase, participants learned that two keypress actions consistently produced red and green visual cues. Next, in a test phase, participants' actions continued to result in singletons, but their color could be either congruent or incongruent with the learned action-color associations. Furthermore, the color singletons now functioned as valid or invalid spatial cues in a visual search, in which participants looked for a tilted line ("/" or "\") among distractors ("X"s). The results showed that an action-congruent color was more effective as a valid cue in the search task (increased benefit), but less effective as an invalid cue (reduced cost). We discuss our findings in terms of both an inhibition account and a preactivation account of action-driven sensory bias, and argue in favor of the preactivation account.

Linking perception and action by structure or process? Toward an integrative perspective

Over the past decades cognitive neuroscience's renewed interest in action has intensified the search of principles explaining how the cognitive system links perception to action and vice versa. To date, at least two seemingly alternative approaches can be distinguished. Perception and action might be linked either by common representational structures, as assumed by the ideomotor approach, or by common attentional processes, as assumed by the attention approach. This article first reviews the evidence from different paradigms supporting each approach. It becomes clear that most studies selectively focus either on actions directed at goals outside the actors' perceptual range (supporting the ideomotor approach) or on actions directed at targets within the actors' perceptual range (supporting the attention approach). In a second step, I will try to reconcile both approaches by reviewing recent eye movement studies that abolish the classical combination of approach and goals under study. Demonstrating that both approaches cover target- as well as goal-directed actions, it is proposed that operations addressed in both conceptual frameworks interact with each other.

Time-based event expectations employ relative, not absolute, representations of time

When the timing of an event is predictable, humans automatically form implicit time-based event expectations. We investigated whether these expectations rely on absolute (e.g., 800 ms) or relative (e.g., a shorter duration) representations of time. In a choice-response task with two different pre-target intervals, participants implicitly learned that targets were predictable by interval durations. In a test phase, the two intervals were either considerably shortened or lengthened. In both cases, behavioral tendencies transferred from practice to test according to relative, not absolute, interval duration. We conclude that humans employ relative representations of time periods when forming time-based event expectations. These results suggest that learned time-based event expectations (e.g., in communication and human-machine interaction) should transfer to faster or slower environments if the relative temporal distribution of events is preserved.

Impact of planned movement direction on judgments of visual locations

The present study examined if and how the direction of planned hand movements affects the perceived direction of visual stimuli. In three experiments participants prepared hand movements that deviated regarding direction ("Experiment 1" and "2") or distance relative to a visual target position ("Experiment 3"). Before actual execution of the movement, the direction of the visual stimulus had to be estimated by means of a method of adjustment. The perception of stimulus direction was biased away from planned movement direction, such that with leftward movements stimuli appeared somewhat more rightward than with rightward movements. Control conditions revealed that this effect was neither a mere response bias, nor a result of processing or memorizing movement cues. Also, shifting the focus of attention toward a cued location in space was not sufficient to induce the perceptual bias observed under conditions of movement preparation ("Experiment 4"). These results confirm that characteristics of planned actions bias visual perception, with the direction of bias (contrast or assimilation) possibly depending on the type of the representations (categorical or metric) involved.

Temporal Predictability Facilitates Action, Not Perception

Expectancy for upcoming action requirements is a fundamental prerequisite for human control of action. In the research reported here, we investigated which part of cognitive processing benefits from temporal predictability. In a binary forced-choice paradigm, visual targets were preceded by different intervals. In one condition, targets could be predicted by the length of the intervals. In other conditions, response goals or response effectors could be predicted by the length of the intervals. Behavioral advantages were observed when response effectors were temporally predictable, whereas temporal predictability of response goals and target stimuli was not sufficient. The findings thus show that temporal expectancy in speeded choice-reaction tasks facilitates late, effector-specific motor processing. These findings are of importance not only for our basic understanding of action control but also for any human-machine interaction that involves system delays.

Features of planned hand actions influence identification of graspable objects

We demonstrate that constituents of motor actions associated with handled objects play a role in identifying such objects. Subjects held in working memory action plans for reaching movements; these action plans specified both the hand to be used (left or right) and a wrist orientation (vertical or horizontal). Speeded object identification was impaired when a pictured object matched the action on only one of these two categorical dimensions (e.g., a beer mug with its handle facing left, an action plan involving the right hand and vertical wrist orientation), relative to when the object matched the action on both dimensions or neither dimension. This result implies that identification of a manipulable object leads to automatic retrieval of matching features of a planned action along with nonmatching features to which they are bound. These discrepant features conflict with those of the target object, which results in delayed identification.

Perseveration effects in reaching and grasping rely on motor priming and not perception

Perseveration effects in grasping were examined in two experiments. In both experiments, participants reached and grasped different versions of a novel object with their thumb and forefinger using either a horizontal or vertical pincer grasp. The dependent variable was the choice of grasp. In Experiment 1, trials were performed either with or without visual feedback. In Experiment 2, trials were performed either physically or using motor imagery. In both experiments, participants tended to perseverate in their choice of grip. Further, there was no evidence that either the availability of visual feedback during the preceding or current action modulated this effect; mode of responding was similarly inconsequential. The results were interpreted as evidence for a motor priming explanation of perseveration and against an account that relies on perceptual priming.

Investigating ideomotor cognition with motorvisual priming paradigms: key findings, methodological challenges, and future directions

Ideomotor theory claims that perceptual representations of action-effects are functionally involved in the planning of actions. Strong evidence for this claim comes from a phenomenon called motorvisual priming. Motorvisual priming refers to the finding that action planning directly affects perception, and that the effects are selective for stimuli that share features with the planned action. Motorvisual priming studies have provided detailed insights into the processing of perceptual representations in action planning. One important finding is that such representations in action planning have a categorical format, whereas metric representations are not anticipated in planning. Further essential findings regard the processing mechanisms and the time course of ideomotor cognition. Perceptual representations of action-effects are first activated by action planning and then bound into a compound representation of the action plan. This compound representation is stabilized throughout the course of the action by the shielding of all involved representations from other cognitive processes. Despite a rapid growth in the number of motorvisual priming studies in the current literature, there are still many aspects of ideomotor cognition which have not yet been investigated. These aspects include the scope of ideomotor processing with regard to action types and stimulus types, as well as the exact nature of the binding and shielding mechanisms involved.

Imaging when acting: picture but not word cues induce action-related biases of visual attention

In line with the Theory of Event Coding (Hommel et al., 2001a), action planning has been shown to affect perceptual processing – an effect that has been attributed to a so-called intentional weighting mechanism (Wykowska et al., 2009; Memelink and Hommel, 2012), whose functional role is to provide information for open parameters of online action adjustment (Hommel, 2010). The aim of this study was to test whether different types of action representations induce intentional weighting to various degrees. To meet this aim, we introduced a paradigm in which participants performed a visual search task while preparing to grasp or to point. The to-be performed movement was signaled either by a picture of a required action or a word cue. We reasoned that picture cues might trigger a more concrete action representation that would be more likely to activate the intentional weighting of perceptual dimensions that provide information for online action control. In contrast, word cues were expected to trigger a more abstract action representation that would be less likely to induce intentional weighting. In two experiments, preparing for an action facilitated the processing of targets in an unrelated search task if they differed from distractors on a dimension that provided information for online action control. As predicted, however, this effect was observed only if action preparation was signaled by picture cues but not if it was signaled by word cues. We conclude that picture cues are more efficient than word cues in activating the intentional weighting of perceptual dimensions, presumably by specifying not only invariant characteristics of the planned action but also the dimensions of action-specific parameters.