Visuo-motor imagery of specific manual actions: A multi-variate pattern analysis fMRI study

Why motor imagery is not really motoric: towards a re-conceptualization in terms of effect-based action control

Overt and imagined action seem inextricably linked. Both have similar timing, activate shared brain circuits, and motor imagery influences overt action and vice versa. Motor imagery is, therefore, often assumed to recruit the same motor processes that govern action execution, and which allow one to play through or simulate actions offline. Here, we advance a very different conceptualization. Accordingly, the links between imagery and overt action do not arise because action imagery is intrinsically motoric, but because action planning is intrinsically imaginistic and occurs in terms of the perceptual effects one want to achieve. Seen like this, the term 'motor imagery' is a misnomer of what is more appropriately portrayed as 'effect imagery'. In this article, we review the long-standing arguments for effect-based accounts of action, which are often ignored in motor imagery research. We show that such views provide a straightforward account of motor imagery. We review the evidence for imagery-execution overlaps through this new lens and argue that they indeed emerge because every action we execute is planned, initiated and controlled through an imagery-like process. We highlight findings that this new view can now explain and point out open questions.

Neural oscillations guiding action during effects imagery

Goal-directed acting requires the integration of sensory information but can also be performed without direct sensory input. Examples of this can be found in sports and can be conceptualized by feedforward processes. There is, however, still a lack of understanding of the temporal neural dynamics and neuroanatomical structures involved in such processes. In the current study, we used EEG beamforming methods and examined 37 healthy participants in two well-controlled experiments varying the necessity of anticipatory processes during goal-directed action. We found that alpha and beta activity in the medial and posterior cingulate cortex enabled feedforward predictions about the position of an object based on the latest sensorimotor state. On this basis, theta band activity seems more related to sensorimotor representations, while beta band activity would be more involved in setting up the structure of the neural representations themselves. Alpha band activity in sensory cortices reflects an intensified gating of the anticipated perceptual consequences of the to-be-executed action. Together, the findings indicate that goal-directed acting through the anticipation of the predicted state of an effector is based on accompanying processes in multiple frequency bands in midcingulate and sensory brain regions.

Spatio-temporal evolution of human neural activity during visually cued hand movements

Making hand movements in response to visual cues is common in daily life. It has been well known that this process activates multiple areas in the brain, but how these neural activations progress across space and time remains largely unknown. Taking advantage of intracranial electroencephalographic (iEEG) recordings using depth and subdural electrodes from 36 human subjects using the same task, we applied single-trial and cross-trial analyses to high-frequency iEEG activity. The results show that the neural activation was widely distributed across the human brain both within and on the surface of the brain, and focused specifically on certain areas in the parietal, frontal, and occipital lobes, where parietal lobes present significant left lateralization on the activation. We also demonstrate temporal differences across these brain regions. Finally, we evaluated the degree to which the timing of activity within these regions was related to sensory or motor function. The findings of this study promote the understanding of task-related neural processing of the human brain, and may provide important insights for translational applications.

Automatic imitation in youngsters with Gilles de la Tourette syndrome: A behavioral study

It is widely known that humans have a tendency to imitate each other and that appropriate modulation of automatic imitative behaviors has a crucial function in social interactions. Gilles de la Tourette syndrome (GTS) is a childhood-onset neuropsychiatric disorder characterized by motor and phonic tics. Apart from tics, patients with GTS are often reported to show an abnormal tendency to automatically imitate others' behaviors (i.e., echophenomena), which may be related to a failure in top-down inhibition of imitative response tendencies. The aim of the current study is to explore the top-down inhibitory mechanisms on automatic imitative behaviors in youngsters with GTS. Error rates and reaction times from 32 participants with GTS and 32 controls were collected in response to an automatic imitation task assessing the influence of observed movements displayed in the first-person perspective on congruent and incongruent motor responses. Results showed that participants with GTS had higher error rates than controls, and their responses were faster than those of controls in incompatible stimuli. Our findings provide novel evidence of a key difference between youngsters with GTS and typically developing participants in the ability to effectively control the production of own motor responses to sensory inputs deriving from observed actions.

Neural Representations of Procedural Knowledge

Although declarative concepts (e.g., apple) have been shown to be identifiable from their functional MRI (fMRI) signatures, the correspondence has yet to be established for executing a complex procedure such as tying a knot. In this study, 7 participants were trained to tie seven knots. Their neural representations of these seven procedures were assessed with fMRI as they imagined tying each knot. A subset of the trained participants physically tied each knot in a later fMRI session. Findings demonstrated that procedural knowledge of tying a particular knot can be reliably identified from its fMRI signature, and such procedural signatures were found here in frontal, parietal, motor, and cerebellar regions. In addition, a classifier trained on mental tying signatures was able to reliably identify when participants were planning to tie knots before they physically tied them, which suggests that the mental-tying and physical-tying procedural signatures are similar. These findings indicate that fMRI activation patterns can illuminate the representation and organization of procedural knowledge.

Visual Body Part Representation in the Lateral Occipitotemporal Cortex in Children/Adolescents and Adults

The lateral occipitotemporal cortex (LOTC) that responds to human bodies and body parts has been implicated in social development and neurodevelopmental disorders like autism spectrum disorder (ASD). Neuroimaging studies using a representational similarity analysis (RSA) revealed that body representation in the LOTC of typically developing (TD) adults is categorized into 3 clusters: action effector body parts, noneffector body parts, and face parts. However, its organization of younger people (i.e., children and adolescents) and its association with individual traits remain unclear. In this functional MRI study, TD adults and children/adolescents observed photographs of hands, feet, arms, legs, chests, waists, upper/lower faces, the whole body, and chairs. The univariate analysis showed that fewer child/adolescent participants showed left LOTC activation in response to whole-body images (relative to those of chairs) than adult participants. Contrastingly, the RSA on both age groups revealed a comparable body representation with 3 clusters of body parts in the bilateral LOTC. Hence, this result indicates that, although response to whole-body images can differ, LOTC body part representations for children/ adolescents and adults are highly similar. Furthermore, sensory atypicality is associated with spatial LOTC organization, suggesting the importance of this region for understanding individual difference, which is frequently observed in ASD.

Decoding stimulus identity in occipital, parietal and inferotemporal cortices during visual mental imagery

In the absence of input from the external world, humans are still able to generate vivid mental images. This cognitive process, known as visual mental imagery, involves a network of prefrontal, parietal, inferotemporal, and occipital regions. Using multivariate pattern analysis (MVPA), previous studies were able to distinguish between the different orientations of imagined gratings, but not between more complex imagined stimuli, such as common objects, in early visual cortex (V1). Here we asked whether letters, simple shapes, and objects can be decoded in early visual areas during visual mental imagery. In a delayed spatial judgment task, we asked participants to observe or imagine stimuli. To examine whether it is possible to discriminate between neural patterns during perception and visual mental imagery, we performed ROI-based and whole-brain searchlight-based MVPA. We were able to decode imagined stimuli in early visual (V1, V2), parietal (SPL, IPL, aIPS), inferotemporal (LOC) and prefrontal (PMd) areas. In a subset of these areas (i.e., V1, V2, LOC, SPL, IPL and aIPS), we also obtained significant cross-decoding across visual imagery and perception. Moreover, we observed a linear relationship between behavioral accuracy and the amplitude of the BOLD signal in parietal and inferotemporal cortices, but not in early visual cortex, in line with the view that these areas contribute to the ability to perform visual imagery. Together, our results suggest that in the absence of bottom-up visual inputs, patterns of functional activation in early visual cortex allow distinguishing between different imagined stimulus exemplars, most likely mediated by signals from parietal and inferotemporal areas.

The Effect of Expertise on Kinesthetic Motor Imagery of Complex Actions

The ability to mentally simulate an action by recalling the body sensations relative to the real execution is referred to as kinesthetic motor imagery (MI). Frontal and parietal motor-related brain regions are generally engaged during MI. The present study aimed to investigate the time course and neural correlates of complex action imagery and possible effects of expertise on the underlying action representation processes. Professional ballet dancers and controls were presented with effortful and effortless ballet steps and instructed to mentally reproduce each movement during EEG recording. Time-locked MI was associated with an Anterior Negativity (AN) component (400-550 ms) that was larger in dancers relative to controls. The AN was differentially modulated by the motor content (effort) as a function of ballet expertise. It was more negative in response to effortful (than effortless) movements in control participants only. This effect also had a frontal distribution in controls and a centro-parietal distribution in dancers, as shown by the topographic maps of the scalp voltage. The source reconstruction (swLORETA) of the recorded potentials in the AN time-window showed enhanced engagement of prefrontal regions in controls (BA 10/47) relative to dancers, and occipitotemporal (BA 20) and bilateral sensorimotor areas in dancers (BA6/40) compared with controls. This evidence seems to suggest that kinesthetic MI of complex action relied on visuomotor simulation processes in participants with acquired dance expertise. Simultaneously, increased cognitive demands occurred in participants lacking in motor knowledge with the specific action. Hence, professional dance training may lead to refined action representation processes.

Hierarchical Action Encoding Within the Human Brain

Humans are able to interact with objects with extreme flexibility. To achieve this ability, the brain does not only control specific muscular patterns, but it also needs to represent the abstract goal of an action, irrespective of its implementation. It is debated, however, how abstract action goals are implemented in the brain. To address this question, we used multivariate pattern analysis of functional magnetic resonance imaging data. Human participants performed grasping actions (precision grip, whole hand grip) with two different wrist orientations (canonical, rotated), using either the left or right hand. This design permitted to investigate a hierarchical organization consisting of three levels of abstraction: 1) “concrete action” encoding; 2) “effector-dependent goal” encoding (invariant to wrist orientation); and 3) “effector-independent goal” encoding (invariant to effector and wrist orientation). We found that motor cortices hosted joint encoding of concrete actions and of effector-dependent goals, while the parietal lobe housed a convergence of all three representations, comprising action goals within and across effectors. The left lateral occipito-temporal cortex showed effector-independent goal encoding, but no convergence across the three levels of representation. Our results support a hierarchical organization of action encoding, shedding light on the neural substrates supporting the extraordinary flexibility of human hand behavior.

Predictive coding of action intentions in dorsal and ventral visual stream is based on visual anticipations, memory-based information and motor preparation

Predictions of upcoming movements are based on several types of neural signals that span the visual, somatosensory, motor and cognitive system. Thus far, pre-movement signals have been investigated while participants viewed the object to be acted upon. Here, we studied the contribution of information other than vision to the classification of preparatory signals for action, even in the absence of online visual information. We used functional magnetic resonance imaging (fMRI) and multivoxel pattern analysis (MVPA) to test whether the neural signals evoked by visual, memory-based and somato-motor information can be reliably used to predict upcoming actions in areas of the dorsal and ventral visual stream during the preparatory phase preceding the action, while participants were lying still. Nineteen human participants (nine women) performed one of two actions towards an object with their eyes open or closed. Despite the well-known role of ventral stream areas in visual recognition tasks and the specialization of dorsal stream areas in somato-motor processes, we decoded action intention in areas of both streams based on visual, memory-based and somato-motor signals. Interestingly, we could reliably decode action intention in absence of visual information based on neural activity evoked when visual information was available and vice versa. Our results show a similar visual, memory and somato-motor representation of action planning in dorsal and ventral visual stream areas that allows predicting action intention across domains, regardless of the availability of visual information.

Target-oriented motor imagery for grasping action: different characteristics of brain activation between kinesthetic and visual imagery

Motor imagery (MI) for target-oriented movements, which is a basis for functional activities of daily living, can be more appropriate than non-target-oriented MI as tasks to promote motor recovery or brain-computer interface (BCI) applications. This study aimed to explore different characteristics of brain activation among target-oriented kinesthetic imagery (KI) and visual imagery (VI) in the first-person (VI-1) and third-person (VI-3) perspectives. Eighteen healthy volunteers were evaluated for MI ability, trained for the three types of target-oriented MIs, and scanned using 3 T functional magnetic resonance imaging (fMRI) under MI and perceptual control conditions, presented in a block design. Post-experimental questionnaires were administered after fMRI. Common brain regions activated during the three types of MI were the left premotor area and inferior parietal lobule, irrespective of the MI modalities or perspectives. Contrast analyses showed significantly increased brain activation only in the contrast of KI versus VI-1 and KI versus VI-3 for considerably extensive brain regions, including the supplementary motor area and insula. Neural activity in the orbitofrontal cortex and cerebellum during VI-1 and KI was significantly correlated with MI ability measured by mental chronometry and a self-reported questionnaire, respectively. These results can provide a basis in developing MI-based protocols for neurorehabilitation to improve motor recovery and BCI training in severely paralyzed individuals.

Increased similarity of neural responses to experienced and empathic distress in costly altruism

Empathy—affective resonance with others’ sensory or emotional experiences—is hypothesized to be an important precursor to altruism. However, it is not known whether real-world altruists’ heightened empathy reflects true self-other mapping of multi-voxel neural response patterns. We investigated this relationship in adults who had engaged in extraordinarily costly real-world altruism: donating a kidney to a stranger. Altruists and controls completed fMRI testing while anticipating and experiencing pain, and watching as a stranger anticipated and experienced pain. Machine learning classifiers tested for shared representation between experienced and observed distress. Altruists exhibited more similar representations of experienced and observed fearful anticipation spontaneously and following an empathy prompt in anterior insula and anterior/middle cingulate cortex, respectively, suggesting heightened empathic proclivities and abilities for fear. During pain epochs, altruists were distinguished by spontaneous empathic responses in anterior insula, anterior/mid-cingulate cortex and supplementary motor area, but showed no difference from controls after the empathy prompt. These findings (1) link shared multi-voxel representations of the distress of self and others to real-world costly altruism, (2) reinforce distinctions between empathy for sensory states like pain and anticipatory affective states like fear, and (3) highlight the importance of differentiating between the proclivity and ability to empathize.

What Perceivers Know About Their Own Affordance Perception: Post Hoc Evaluations of Perceptual Processes Do Not Relate to Accuracy

In 2 studies, we sought to investigate participants’ awareness of their own cognitive process when perceiving affordances (i.e., action capabilities) and whether their evaluations of this process relate to affordance perception accuracy. In Study 1, we combined data from 8 experiments and created a coding scheme designed to capture the unique features of different affordance perception theories within these post hoc evaluations. We then tested whether the perceptual process that participants reported was related to the accuracy of their judgments. We found that evaluations were distributed fairly evenly across the themes. We also found no relationship between the evaluations and accuracy. In Study 2, we conducted an experiment that specifically addressed our hypotheses about awareness and affordance perception accuracy, while controlling for confounds in Study 1. The greater detail participants gave about their cognitive strategy produced reports that largely did not relate to affordance perception theories. Participants used one of a few strategies aimed at estimating their body dimensions. Our results suggest that awareness of the perceptual process has no influence on the accuracy of affordance perception. It is likely that subjective reports of affordance perceptual process, though clear to the perceiver, are not related to the process itself.

Imagined and Executed Actions in the Human Motor System: Testing Neural Similarity Between Execution and Imagery of Actions with a Multivariate Approach

Simulation theory proposes motor imagery (MI) to be a simulation based on representations also used for motor execution (ME). Nonetheless, it is unclear how far they use the same neural code. We use multivariate pattern analysis (MVPA) and representational similarity analysis (RSA) to describe the neural representations associated with MI and ME within the frontoparietal motor network. During functional magnetic resonance imaging scanning, 20 volunteers imagined or executed 3 different types of right-hand actions. Results of MVPA showed that these actions as well as their modality (MI or ME) could be decoded significantly above chance from the spatial patterns of BOLD signals in premotor and posterior parietal cortices. This was also true for cross-modal decoding. Furthermore, representational dissimilarity matrices of frontal and parietal areas showed that MI and ME representations formed separate clusters, but that the representational organization of action types within these clusters was identical. For most ROIs, this pattern of results best fits with a model that assumes a low-to-moderate degree of similarity between the neural patterns associated with MI and ME. Thus, neural representations of MI and ME are neither the same nor totally distinct but exhibit a similar structural geometry with respect to different types of action.

Investigating common coding of observed and executed actions in the monkey brain using cross-modal multi-variate fMRI classification

Mirror neurons are generally described as a neural substrate hosting shared representations of actions, by simulating or 'mirroring' the actions of others onto the observer's own motor system. Since single neuron recordings are rarely feasible in humans, it has been argued that cross-modal multi-variate pattern analysis (MVPA) of non-invasive fMRI data is a suitable technique to investigate common coding of observed and executed actions, allowing researchers to infer the presence of mirror neurons in the human brain. In an effort to close the gap between monkey electrophysiology and human fMRI data with respect to the mirror neuron system, here we tested this proposal for the first time in the monkey. Rhesus monkeys either performed reach-and-grasp or reach-and-touch motor acts with their right hand in the dark or observed videos of human actors performing similar motor acts. Unimodal decoding showed that both executed or observed motor acts could be decoded from numerous brain regions. Specific portions of rostral parietal, premotor and motor cortices, previously shown to house mirror neurons, in addition to somatosensory regions, yielded significant asymmetric action-specific cross-modal decoding. These results validate the use of cross-modal multi-variate fMRI analyses to probe the representations of own and others' actions in the primate brain and support the proposed mapping of others' actions onto the observer's own motor cortices.

Domain-general and domain-specific neural changes underlying visual expertise

Visual expertise induces changes in neural processing for many different domains of expertise. However, it is unclear how expertise effects for different domains of expertise are related. In the present fMRI study, we combine large-scale univariate and multi-voxel analyses to contrast the expertise-related neural changes associated with two different domains of expertise, bird expertise (ornithology) and mineral expertise (mineralogy). Results indicated distributed expertise-related neural changes, with effects for both domains of expertise in high-level visual cortex and effects for bird expertise even extending to low-level visual regions and the frontal lobe. Importantly, a multivariate generalization analysis showed that effects in high-level visual cortex were specific to the domain of expertise. In contrast, the neural changes in the frontal lobe relating to expertise showed significant generalization, signaling the presence of domain-independent expertise effects. In conclusion, expertise is related to a combination of domain-specific and domain-general changes in neural processing.

Cross-modal and non-monotonic representations of statistical regularity are encoded in local neural response patterns

Current neurobiological models assign a central role to predictive processes calibrated to environmental statistics. Neuroimaging studies examining the encoding of stimulus uncertainty have relied almost exclusively on manipulations in which stimuli were presented in a single sensory modality, and further assumed that neural responses vary monotonically with uncertainty. This has left a gap in theoretical development with respect to two core issues: (i) are there cross-modal brain systems that encode input uncertainty in way that generalizes across sensory modalities, and (ii) are there brain systems that track input uncertainty in a non-monotonic fashion? We used multivariate pattern analysis to address these two issues using auditory, visual and audiovisual inputs. We found signatures of cross-modal encoding in frontoparietal, orbitofrontal, and association cortices using a searchlight cross-classification analysis where classifiers trained to discriminate levels of uncertainty in one modality were tested in another modality. Additionally, we found widespread systems encoding uncertainty non-monotonically using classifiers trained to discriminate intermediate levels of uncertainty from both the highest and lowest uncertainty levels. These findings comprise the first comprehensive report of cross-modal and non-monotonic neural sensitivity to statistical regularities in the environment, and suggest that conventional paradigms testing for monotonic responses to uncertainty in a single sensory modality may have limited generalizability.

Time-resolved decoding of planned delayed and immediate prehension movements

Different contexts require us either to react immediately, or to delay (or suppress) a planned movement. Previous studies that aimed at decoding movement plans typically dissociated movement preparation and execution by means of delayed-movement paradigms. Here we asked whether these results can be generalized to the planning and execution of immediate movements. To directly compare delayed, non-delayed, and suppressed reaching and grasping movements, we used a slow event-related functional magnetic resonance imaging (fMRI) design. To examine how neural representations evolved throughout movement planning, execution, and suppression, we performed time-resolved multivariate pattern analysis (MVPA). During the planning phase, we were able to decode upcoming reaching and grasping movements in contralateral parietal and premotor areas. During the execution phase, we were able to decode movements in a widespread bilateral network of motor, premotor, and somatosensory areas. Moreover, we obtained significant decoding across delayed and non-delayed movement plans in contralateral primary motor cortex. Our results demonstrate the feasibility of time-resolved MVPA and provide new insights into the dynamics of the prehension network, suggesting early neural representations of movement plans in the primary motor cortex that are shared between delayed and non-delayed contexts.

Role of medial premotor areas in action language processing in relation to motor skills

The literature reports that the supplementary motor area (SMA) and pre-supplementary motor area (pre-SMA) are involved in motor planning and execution, and in motor-related cognitive functions such as motor imagery. However, their specific role in action language processing remains unclear. In the present study, we investigated the impact of repetitive transcranial magnetic stimulation (rTMS) over SMA and pre-SMA during an action semantic analogy task (SAT) in relation with fine motor skills (i.e., manual dexterity) and motor imagery abilities in healthy non-expert adults. The impact of rTMS over SMA (but not pre-SMA) on reaction times (RT) during SAT was correlated with manual dexterity. Specifically, results show that rTMS over SMA modulated RT for those with lower dexterity skills. Our results therefore demonstrate a causal involvement of SMA in action language processing, as well as the existence of inter-individual differences in this involvement. We discuss these findings in light of neurolinguistic theories of language processing.

Development of visual category selectivity in ventral visual cortex does not require visual experience

To what extent does functional brain organization rely on sensory input? Here, we show that for the penultimate visual-processing region, ventral-temporal cortex (VTC), visual experience is not the origin of its fundamental organizational property, category selectivity. In the fMRI study reported here, we presented 14 congenitally blind participants with face-, body-, scene-, and object-related natural sounds and presented 20 healthy controls with both auditory and visual stimuli from these categories. Using macroanatomical alignment, response mapping, and surface-based multivoxel pattern analysis, we demonstrated that VTC in blind individuals shows robust discriminatory responses elicited by the four categories and that these patterns of activity in blind subjects could successfully predict the visual categories in sighted controls. These findings were confirmed in a subset of blind participants born without eyes and thus deprived from all light perception since conception. The sounds also could be decoded in primary visual and primary auditory cortex, but these regions did not sustain generalization across modalities. Surprisingly, although not as strong as visual responses, selectivity for auditory stimulation in visual cortex was stronger in blind individuals than in controls. The opposite was observed in primary auditory cortex. Overall, we demonstrated a striking similarity in the cortical response layout of VTC in blind individuals and sighted controls, demonstrating that the overall category-selective map in extrastriate cortex develops independently from visual experience.

Neural Representations of Observed Actions Generalize across Static and Dynamic Visual Input

People interact with entities in the environment in distinct and categorizable ways (e.g., kicking is making contact with foot). We can recognize these action categories across variations in actors, objects, and settings; moreover, we can recognize them from both dynamic and static visual input. However, the neural systems that support action recognition across these perceptual differences are unclear. Here, we used multivoxel pattern analysis of fMRI data to identify brain regions that support visual action categorization in a format-independent way. Human participants were scanned while viewing eight categories of interactions (e.g., pulling) depicted in two visual formats: (1) visually controlled videos of two interacting actors and (2) visually varied photographs selected from the internet involving different actors, objects, and settings. Action category was decodable across visual formats in bilateral inferior parietal, bilateral occipitotemporal, left premotor, and left middle frontal cortex. In most of these regions, the representational similarity of action categories was consistent across subjects and visual formats, a property that can contribute to a common understanding of actions among individuals. These results suggest that the identified brain regions support action category codes that are important for action recognition and action understanding.SIGNIFICANCE STATEMENT Humans tend to interpret the observed actions of others in terms of categories that are invariant to incidental features: whether a girl pushes a boy or a button and whether we see it in real-time or in a single snapshot, it is still pushing Here, we investigated the brain systems that facilitate the visual recognition of these action categories across such differences. Using fMRI, we identified several areas of parietal, occipitotemporal, and frontal cortex that exhibit action category codes that are similar across viewing of dynamic videos and still photographs. Our results provide strong evidence for the involvement of these brain regions in recognizing the way that people interact physically with objects and other people.

Is the Sensorimotor Cortex Relevant for Speech Perception and Understanding? An Integrative Review

In the neuroscience of language, phonemes are frequently described as multimodal units whose neuronal representations are distributed across perisylvian cortical regions, including auditory and sensorimotor areas. A different position views phonemes primarily as acoustic entities with posterior temporal localization, which are functionally independent from frontoparietal articulatory programs. To address this current controversy, we here discuss experimental results from functional magnetic resonance imaging (fMRI) as well as transcranial magnetic stimulation (TMS) studies. On first glance, a mixed picture emerges, with earlier research documenting neurofunctional distinctions between phonemes in both temporal and frontoparietal sensorimotor systems, but some recent work seemingly failing to replicate the latter. Detailed analysis of methodological differences between studies reveals that the way experiments are set up explains whether sensorimotor cortex maps phonological information during speech perception or not. In particular, acoustic noise during the experiment and ‘motor noise’ caused by button press tasks work against the frontoparietal manifestation of phonemes. We highlight recent studies using sparse imaging and passive speech perception tasks along with multivariate pattern analysis (MVPA) and especially representational similarity analysis (RSA), which succeeded in separating acoustic-phonological from general-acoustic processes and in mapping specific phonological information on temporal and frontoparietal regions. The question about a causal role of sensorimotor cortex on speech perception and understanding is addressed by reviewing recent TMS studies. We conclude that frontoparietal cortices, including ventral motor and somatosensory areas, reflect phonological information during speech perception and exert a causal influence on language understanding.

CoSMoMVPA: Multi-Modal Multivariate Pattern Analysis of Neuroimaging Data in Matlab/GNU Octave

Recent years have seen an increase in the popularity of multivariate pattern (MVP) analysis of functional magnetic resonance (fMRI) data, and, to a much lesser extent, magneto- and electro-encephalography (M/EEG) data. We present CoSMoMVPA, a lightweight MVPA (MVP analysis) toolbox implemented in the intersection of the Matlab and GNU Octave languages, that treats both fMRI and M/EEG data as first-class citizens. CoSMoMVPA supports all state-of-the-art MVP analysis techniques, including searchlight analyses, classification, correlations, representational similarity analysis, and the time generalization method. These can be used to address both data-driven and hypothesis-driven questions about neural organization and representations, both within and across: space, time, frequency bands, neuroimaging modalities, individuals, and species. It uses a uniform data representation of fMRI data in the volume or on the surface, and of M/EEG data at the sensor and source level. Through various external toolboxes, it directly supports reading and writing a variety of fMRI and M/EEG neuroimaging formats, and, where applicable, can convert between them. As a result, it can be integrated readily in existing pipelines and used with existing preprocessed datasets. CoSMoMVPA overloads the traditional volumetric searchlight concept to support neighborhoods for M/EEG and surface-based fMRI data, which supports localization of multivariate effects of interest across space, time, and frequency dimensions. CoSMoMVPA also provides a generalized approach to multiple comparison correction across these dimensions using Threshold-Free Cluster Enhancement with state-of-the-art clustering and permutation techniques. CoSMoMVPA is highly modular and uses abstractions to provide a uniform interface for a variety of MVP measures. Typical analyses require a few lines of code, making it accessible to beginner users. At the same time, expert programmers can easily extend its functionality. CoSMoMVPA comes with extensive documentation, including a variety of runnable demonstration scripts and analysis exercises (with example data and solutions). It uses best software engineering practices including version control, distributed development, an automated test suite, and continuous integration testing. It can be used with the proprietary Matlab and the free GNU Octave software, and it complies with open source distribution platforms such as NeuroDebian. CoSMoMVPA is Free/Open Source Software under the permissive MIT license.

Website: http://cosmomvpa.org

Source code: https://github.com/CoSMoMVPA/CoSMoMVPA

Beta band modulations underlie action representations for movement planning

To be able to interact with our environment, we need to transform incoming sensory information into goal-directed motor outputs. Whereas our ability to plan an appropriate movement based on sensory information appears effortless and simple, the underlying brain dynamics are still largely unknown. Here we used magnetoencephalography (MEG) to investigate this issue by recording brain activity during the planning of non-visually guided reaching and grasping actions, performed with either the left or right hand. Adopting a combination of univariate and multivariate analyses, we revealed specific patterns of beta power modulations underlying varying levels of neural representations during movement planning. (1) Effector-specific modulations were evident as a decrease in power in the beta band. Within both hemispheres, this decrease was stronger while planning a movement with the contralateral hand. (2) The comparison of planned grasping and reaching led to a relative increase in power in the beta band. These power changes were localized within temporal, premotor and posterior parietal cortices. Action-related modulations overlapped with effector-related beta power changes within widespread frontal and parietal regions, suggesting the possible integration of these two types of neural representations. (3) Multivariate analyses of action-specific power changes revealed that part of this broadband beta modulation also contributed to the encoding of an effector-independent neural representation of a planned action within fronto-parietal and temporal regions. Our results suggest that beta band power modulations play a central role in movement planning, within both the dorsal and ventral stream, by coding and integrating different levels of neural representations, ranging from the simple representation of the to-be-moved effector up to an abstract, effector-independent representation of the upcoming action.

MEG Multivariate Analysis Reveals Early Abstract Action Representations in the Lateral Occipitotemporal Cortex

Understanding other people's actions is a fundamental prerequisite for social interactions. Whether action understanding relies on simulating the actions of others in the observers' motor system or on the access to conceptual knowledge stored in nonmotor areas is strongly debated. It has been argued previously that areas that play a crucial role in action understanding should (1) distinguish between different actions, (2) generalize across the ways in which actions are performed (Dinstein et al., 2008; Oosterhof et al., 2013; Caramazza et al., 2014), and (3) have access to action information around the time of action recognition (Hauk et al., 2008). Whereas previous studies focused on the first two criteria, little is known about the dynamics underlying action understanding. We examined which human brain regions are able to distinguish between pointing and grasping, regardless of reach direction (left or right) and effector (left or right hand), using multivariate pattern analysis of magnetoencephalography data. We show that the lateral occipitotemporal cortex (LOTC) has the earliest access to abstract action representations, which coincides with the time point from which there was enough information to allow discriminating between the two actions. By contrast, precentral regions, though recruited early, have access to such abstract representations substantially later. Our results demonstrate that in contrast to the LOTC, the early recruitment of precentral regions does not contain the detailed information that is required to recognize an action. We discuss previous theoretical claims of motor theories and how they are incompatible with our data.

SIGNIFICANCE STATEMENT

It is debated whether our ability to understand other people's actions relies on the simulation of actions in the observers' motor system, or is based on access to conceptual knowledge stored in nonmotor areas. Here, using magnetoencephalography in combination with machine learning, we examined where in the brain and at which point in time it is possible to distinguish between pointing and grasping actions regardless of the way in which they are performed (effector, reach direction). We show that, in contrast to the predictions of motor theories of action understanding, the lateral occipitotemporal cortex has access to abstract action representations substantially earlier than precentral regions.

Decoding Internally and Externally Driven Movement Plans

During movement planning, brain activity within parietofrontal networks encodes information about upcoming actions that can be driven either externally (e.g., by a sensory cue) or internally (i.e., by a choice/decision). Here we used multivariate pattern analysis (MVPA) of fMRI data to distinguish between areas that represent (1) abstract movement plans that generalize across the way in which these were driven, (2) internally driven movement plans, or (3) externally driven movement plans. In a delayed-movement paradigm, human volunteers were asked to plan and execute three types of nonvisually guided right-handed reaching movements toward a central target object: using a precision grip, a power grip, or touching the object without hand preshaping. On separate blocks of trials, movements were either instructed via color cues (Instructed condition), or chosen by the participant (Free-Choice condition). Using ROI-based and whole-brain searchlight-based MVPA, we found abstract representations of planned movements that generalize across the way these movements are selected (internally vs externally driven) in parietal cortex, dorsal premotor cortex, and primary motor cortex contralateral to the acting hand. In addition, we revealed representations specific for internally driven movement plans in contralateral ventral premotor cortex, dorsolateral prefrontal cortex, supramarginal gyrus, and in ipsilateral posterior parietotemporal regions, suggesting that these regions are recruited during movement selection. Finally, we observed representations of externally driven movement plans in bilateral supplementary motor cortex and a similar trend in presupplementary motor cortex, suggesting a role in stimulus-response mapping.

SIGNIFICANCE STATEMENT

The way the human brain prepares the body for action constitutes an essential part of our ability to interact with our environment. Previous studies demonstrated that patterns of neuronal activity encode upcoming movements. Here we used multivariate pattern analysis of human fMRI data to distinguish between brain regions containing movement plans for instructed (externally driven) movements, areas involved in movement selection (internally driven), and areas containing abstract movement plans that are invariant to the way these were generated (i.e., that generalize across externally and internally driven movement plans). Our findings extend our understanding of the neural basis of movement planning and have the potential to contribute to the development of brain-controlled neural prosthetic devices.

Motor imagery of hand actions: Decoding the content of motor imagery from brain activity in frontal and parietal motor areas

How motor maps are organized while imagining actions is an intensely debated issue. It is particularly unclear whether motor imagery relies on action‐specific representations in premotor and posterior parietal cortices. This study tackled this issue by attempting to decode the content of motor imagery from spatial patterns of Blood Oxygen Level Dependent (BOLD) signals recorded in the frontoparietal motor imagery network. During fMRI‐scanning, 20 right‐handed volunteers worked on three experimental conditions and one baseline condition. In the experimental conditions, they had to imagine three different types of right‐hand actions: an aiming movement, an extension–flexion movement, and a squeezing movement. The identity of imagined actions was decoded from the spatial patterns of BOLD signals they evoked in premotor and posterior parietal cortices using multivoxel pattern analysis. Results showed that the content of motor imagery (i.e., the action type) could be decoded significantly above chance level from the spatial patterns of BOLD signals in both frontal (PMC, M1) and parietal areas (SPL, IPL, IPS). An exploratory searchlight analysis revealed significant clusters motor‐ and motor‐associated cortices, as well as in visual cortices. Hence, the data provide evidence that patterns of activity within premotor and posterior parietal cortex vary systematically with the specific type of hand action being imagined. Hum Brain Mapp 37:81–93, 2016. © 2015 The Authors. Human Brain Mapping Published by Wiley Periodicals, Inc.

A topographical organization for action representation in the human brain

How the human brain represents distinct motor features into a unique finalized action still remains undefined. Previous models proposed the distinct features of a motor act to be hierarchically organized in separated, but functionally interconnected, cortical areas. Here, we hypothesized that distinct patterns across a wide expanse of cortex may actually subserve a topographically organized coding of different categories of actions that represents, at a higher cognitive level and independently from the distinct motor features, the action and its final aim as a whole. Using functional magnetic resonance imaging and pattern classification approaches on the neural responses of 14 right-handed individuals passively watching short movies of hand-performed tool-mediated, transitive, and meaningful intransitive actions, we were able to discriminate with a high accuracy and characterize the category-specific response patterns. Actions are distinctively coded in distributed and overlapping neural responses within an action-selective network, comprising frontal, parietal, lateral occipital and ventrotemporal regions. This functional organization, that we named action topography, subserves a higher-level and more abstract representation of finalized actions and has the capacity to provide unique representations for multiple categories of actions.

Convergent and invariant object representations for sight, sound, and touch

We continuously perceive objects in the world through multiple sensory channels. In this study, we investigated the convergence of information from different sensory streams within the cerebral cortex. We presented volunteers with three common objects via three different modalities-sight, sound, and touch-and used multivariate pattern analysis of functional magnetic resonance imaging data to map the cortical regions containing information about the identity of the objects. We could reliably predict which of the three stimuli a subject had seen, heard, or touched from the pattern of neural activity in the corresponding early sensory cortices. Intramodal classification was also successful in large portions of the cerebral cortex beyond the primary areas, with multiple regions showing convergence of information from two or all three modalities. Using crossmodal classification, we also searched for brain regions that would represent objects in a similar fashion across different modalities of presentation. We trained a classifier to distinguish objects presented in one modality and then tested it on the same objects presented in a different modality. We detected audiovisual invariance in the right temporo-occipital junction, audiotactile invariance in the left postcentral gyrus and parietal operculum, and visuotactile invariance in the right postcentral and supramarginal gyri. Our maps of multisensory convergence and crossmodal generalization reveal the underlying organization of the association cortices, and may be related to the neural basis for mental concepts.

Common coding and dynamic interactions between observed, imagined, and experienced motor and somatosensory activity

Motor imagery and perception - considered generally as forms of motor simulation - share overlapping neural representations with motor production. While much research has focused on the extent of this "common coding," less attention has been paid to how these overlapping representations interact. How do imagined, observed, or produced actions influence one another, and how do we maintain control over our perception and behavior? In the first part of this review we describe interactions between motor production and motor simulation, and explore apparent regulatory mechanisms that balance these processes. Next, we consider the somatosensory system. Numerous studies now support a "sensory mirror system" comprised of neural representations activated by either afferent sensation or vicarious sensation. In the second part of this review we summarize evidence for shared representations of sensation and sensory simulation (including imagery and observed sensation), and suggest that similar interactions and regulation of simulation occur in the somatosensory domain as in the motor domain. We suggest that both motor and somatosensory simulations are flexibly regulated to support simulations congruent with our sensorimotor experience and goals and suppress or separate the influence of those that are not. These regulatory mechanisms are frequently revealed by cases of brain injury but can also be employed to facilitate sensorimotor rehabilitation.

Multivariate cross-classification: applying machine learning techniques to characterize abstraction in neural representations

Here we highlight an emerging trend in the use of machine learning classifiers to test for abstraction across patterns of neural activity. When a classifier algorithm is trained on data from one cognitive context, and tested on data from another, conclusions can be drawn about the role of a given brain region in representing information that abstracts across those cognitive contexts. We call this kind of analysis Multivariate Cross-Classification (MVCC), and review several domains where it has recently made an impact. MVCC has been important in establishing correspondences among neural patterns across cognitive domains, including motor-perception matching and cross-sensory matching. It has been used to test for similarity between neural patterns evoked by perception and those generated from memory. Other work has used MVCC to investigate the similarity of representations for semantic categories across different kinds of stimulus presentation, and in the presence of different cognitive demands. We use these examples to demonstrate the power of MVCC as a tool for investigating neural abstraction and discuss some important methodological issues related to its application.

Moving to music: effects of heard and imagined musical cues on movement-related brain activity

Music is commonly used to facilitate or support movement, and increasingly used in movement rehabilitation. Additionally, there is some evidence to suggest that music imagery, which is reported to lead to brain signatures similar to music perception, may also assist movement. However, it is not yet known whether either imagined or musical cueing changes the way in which the motor system of the human brain is activated during simple movements. Here, functional magnetic resonance imaging was used to compare neural activity during wrist flexions performed to either heard or imagined music with self-pacing of the same movement without any cueing. Focusing specifically on the motor network of the brain, analyses were performed within a mask of BA4, BA6, the basal ganglia (putamen, caudate, and pallidum), the motor nuclei of the thalamus, and the whole cerebellum. Results revealed that moving to music compared with self-paced movement resulted in significantly increased activation in left cerebellum VI. Moving to imagined music led to significantly more activation in pre-supplementary motor area (pre-SMA) and right globus pallidus, relative to self-paced movement. When the music and imagery cueing conditions were contrasted directly, movements in the music condition showed significantly more activity in left hemisphere cerebellum VII and right hemisphere and vermis of cerebellum IX, while the imagery condition revealed more significant activity in pre-SMA. These results suggest that cueing movement with actual or imagined music impacts upon engagement of motor network regions during the movement, and suggest that heard and imagined cues can modulate movement in subtly different ways. These results may have implications for the applicability of auditory cueing in movement rehabilitation for different patient populations.

Distinct neural circuits subserve interpersonal and non-interpersonal emotions

Emotions elicited by interpersonal versus non-interpersonal experiences have different effects on neurobiological functioning in both animals and humans. However, the extent to which the brain circuits underlying interpersonal and non-interpersonal emotions are distinct still remains unclear. The goal of our study was to assess whether different neural circuits are implicated in the processing of arousal and valence of interpersonal versus non-interpersonal emotions. During functional magnetic resonance imaging, participants imagined themselves in emotion-eliciting interpersonal or non-interpersonal situations and then rated the arousal and valence of emotions they experienced. We identified (1) separate neural circuits that are implicated in the arousal and valence dimensions of interpersonal versus non-interpersonal emotions, (2) circuits that are implicated in arousal and valence for both types of emotion, and (3) circuits that are responsive to the type of emotion, regardless of the valence or arousal level of the emotion. We found extensive recruitment of limbic (for arousal) and temporal-parietal (for valence) systems associated with processing of specifically interpersonal emotions compared to non-interpersonal ones. The neural bases of interpersonal and non-interpersonal emotions may, therefore, be largely distinct.

Multiple roles of motor imagery during action observation

Over the last 20 years, the topics of action observation (AO) and motor imagery (MI) have been largely studied in isolation from each other, despite the early integrative account by Jeannerod (1994, 2001). Recent neuroimaging studies demonstrate enhanced cortical activity when AO and MI are performed concurrently ("AO+MI"), compared to either AO or MI performed in isolation. These results indicate the potentially beneficial effects of AO+MI, and they also demonstrate that the underlying neurocognitive processes are partly shared. We separately review the evidence for MI and AO as forms of motor simulation, and present two quantitative literature analyses that indeed indicate rather little overlap between the two bodies of research. We then propose a spectrum of concurrent AO+MI states, from congruent AO+MI where the contents of AO and MI widely overlap, over coordinative AO+MI, where observed and imagined action are different but can be coordinated with each other, to cases of conflicting AO+MI. We believe that an integrative account of AO and MI is theoretically attractive, that it should generate novel experimental approaches, and that it can also stimulate a wide range of applications in sport, occupational therapy, and neurorehabilitation.

Beyond motor scheme: a supramodal distributed representation in the action-observation network

The representation of actions within the action-observation network is thought to rely on a distributed functional organization. Furthermore, recent findings indicate that the action-observation network encodes not merely the observed motor act, but rather a representation that is independent from a specific sensory modality or sensory experience. In the present study, we wished to determine to what extent this distributed and ‘more abstract’ representation of action is truly supramodal, i.e. shares a common coding across sensory modalities. To this aim, a pattern recognition approach was employed to analyze neural responses in sighted and congenitally blind subjects during visual and/or auditory presentation of hand-made actions. Multivoxel pattern analyses-based classifiers discriminated action from non-action stimuli across sensory conditions (visual and auditory) and experimental groups (blind and sighted). Moreover, these classifiers labeled as ‘action’ the pattern of neural responses evoked during actual motor execution. Interestingly, discriminative information for the action/non action classification was located in a bilateral, but left-prevalent, network that strongly overlaps with brain regions known to form the action-observation network and the human mirror system. The ability to identify action features with a multivoxel pattern analyses-based classifier in both sighted and blind individuals and independently from the sensory modality conveying the stimuli clearly supports the hypothesis of a supramodal, distributed functional representation of actions, mainly within the action-observation network.

Multimodal functional imaging of motor imagery using a novel paradigm

Neuroimaging studies have shown that the neural mechanisms of motor imagery (MI) overlap substantially with the mechanisms of motor execution (ME). Surprisingly, however, the role of several regions of the motor circuitry in MI remains controversial, a variability that may be due to differences in neuroimaging techniques, MI training, instruction types, or tasks used to evoke MI. The objectives of this study were twofold: (i) to design a novel task that reliably invokes MI, provides a reliable behavioral measure of MI performance, and is transferable across imaging modalities; and (ii) to measure the common and differential activations for MI and ME with functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). We present a task in which it is difficult to give accurate responses without the use of either motor execution or motor imagery. The behavioral results demonstrate that participants performed similarly on the task when they imagined vs. executed movements and this performance did not change over time. The fMRI results show a spatial overlap of MI and ME in a number of motor and premotor areas, sensory cortices, cerebellum, inferior frontal gyrus, and ventrolateral thalamus. MI uniquely engaged bilateral occipital areas, left parahippocampus, and other temporal and frontal areas, whereas ME yielded unique activity in motor and sensory areas, cerebellum, precuneus, and putamen. The MEG results show a robust event-related beta band desynchronization in the proximity of primary motor and premotor cortices during both ME and MI. Together, these results further elucidate the neural circuitry of MI and show that our task robustly and reliably invokes motor imagery, and thus may prove useful for interrogating the functional status of the motor circuitry in patients with motor disorders.