Manual control of the visual stimulus reduces the flash-lag effect

Memory benefits when actively, rather than passively, viewing images

Serial visual presentations of images exist both in the laboratory and increasingly on virtual platforms such as social media feeds. However, the way we interact with information differs between these. In many laboratory experiments participants view stimuli passively, whereas on social media people tend to interact with information actively. This difference could influence the way information is remembered, which carries practical and theoretical implications. In the current study, 821 participants viewed streams containing seven landscape images that were presented at either a self-paced (active) or an automatic (passive) rate. Critically, the presentation speed in each automatic trial was matched to the speed of a self-paced trial for each participant. Both memory accuracy and memory confidence were greater on self-paced compared to automatic trials. These results indicate that active, self-paced progression through images increases the likelihood of them being remembered, relative to when participants have no control over presentation speed and duration.

Motion Extrapolation in Visual Processing: Lessons from 25 Years of Flash-Lag Debate

Because of the delays inherent in neural transmission, the brain needs time to process incoming visual information. If these delays were not somehow compensated, we would consistently mislocalize moving objects behind their physical positions. Twenty-five years ago, Nijhawan used a perceptual illusion he called the flash-lag effect (FLE) to argue that the brain's visual system solves this computational challenge by extrapolating the position of moving objects (Nijhawan, 1994). Although motion extrapolation had been proposed a decade earlier (e.g., Finke et al., 1986), the proposal that it caused the FLE and functioned to compensate for computational delays was hotly debated in the years that followed, with several alternative interpretations put forth to explain the effect. Here, I argue, 25 years later, that evidence from behavioral, computational, and particularly recent functional neuroimaging studies converges to support the existence of motion extrapolation mechanisms in the visual system, as well as their causal involvement in the FLE. First, findings that were initially argued to challenge the motion extrapolation model of the FLE have since been explained, and those explanations have been tested and corroborated by more recent findings. Second, motion extrapolation explains the spatial shifts observed in several FLE conditions that cannot be explained by alternative (temporal) models of the FLE. Finally, neural mechanisms that actually perform motion extrapolation have been identified at multiple levels of the visual system, in multiple species, and with multiple different methods. I outline key questions that remain, and discuss possible directions for future research.

The Role of Hand Movement in Spatial Serial Order Memory

Research on serial order memory has traditionally used tasks where participants passively view the items. A few studies that included hand movement showed that such movement interfered with serial order memory. In the present study of three experiments, we investigated whether and how hand movements improved spatial serial order memory. Experiment 1 showed that manual tracing (i.e., hand movements that traced the presentation of stimuli on the modified eCorsi block tapping task) improved the performance of backward recall as compared to no manual tracing (the control condition). Experiment 2 showed that the facilitation effect resulted from voluntary hand movements and could not be achieved via passive viewing of another person's manual tracing. Experiment 3 showed that it was the temporal, not the spatial, signal within manual tracing that facilitated spatial serial memory.

Temporal dynamics of a perceptual decision

Previous research suggests that cognitive factors acting in a top-down manner influence the perceptual interpretation of ambiguous stimuli. To examine the temporal unfolding of these influences as a perceptual decision evolves, we have implemented a modified version of the stream-bounce display. Our novel approach allows us to track responses to stream-bounce stimuli dynamically over the entire course of the motion sequence rather than collecting a subjective report after the fact. Using a trackpad, we had participants control a cursor to track a stream-bounce target actively from start to end and measured tracking speed throughout as the dependent variable. Our paradigm replicated the typical effect of visual-only displays being associated with a streaming bias and audiovisual displays with a bouncing bias. Our main finding is a significant behavioral change preceding a perceptual decision that then predicts that decision. Specifically, for trials in which the sound was presented, tracking speeds were significantly slower starting 500 ms before the point of coincidence and presentation of the sound for bounce compared to stream responses. We suggest that behavioral response may reflect a cognitive expectation of a perceptual outcome that then biases action and the interpretation of sensory input to favor that forthcoming percept in a manner consistent with both the predictive-coding and common-coding theoretical frameworks. Our approach provides a novel behavioral corroboration of recent imaging studies that are suggestive of early brain activity in perception and action.

Don't worry, be active: how to facilitate the detection of errors in immersive virtual environments

The current research aims to study the link between the type of vision experienced in a collaborative immersive virtual environment (active vs. multiple passive), the type of error one looks for during a cooperative multi-user exploration of a design project (affordance vs. perceptual violations), and the type of setting in which multi-user perform (field in Experiment 1 vs. laboratory in Experiment 2). The relevance of this link is backed by the lack of conclusive evidence on an active vs. passive vision advantage in cooperative search tasks within software based on immersive virtual reality (IVR). Using a yoking paradigm based on the mixed usage of simultaneous active and multiple passive viewings, we found that the likelihood of error detection in a complex 3D environment was characterized by an active vs. multi-passive viewing advantage depending on: (1) the degree of knowledge dependence of the type of error the passive/active observers were looking for (low for perceptual violations, vs. high for affordance violations), as the advantage tended to manifest itself irrespectively from the setting for affordance, but not for perceptual violations; and (2) the degree of social desirability possibly induced by the setting in which the task was performed, as the advantage occurred irrespectively from the type of error in the laboratory (Experiment 2) but not in the field (Experiment 1) setting. Results are relevant to future development of cooperative software based on IVR used for supporting the design review. A multi-user design review experience in which designers, engineers and end-users all cooperate actively within the IVR wearing their own head mounted display, seems more suitable for the detection of relevant errors than standard systems characterized by a mixed usage of active and passive viewing.

The haptic and the visual flash-lag effect and the role of flash characteristics

When a short flash occurs in spatial alignment with a moving object, the moving object is seen ahead the stationary one. Similar to this visual "flash-lag effect" (FLE) it has been recently observed for the haptic sense that participants judge a moving hand to be ahead a stationary hand when judged at the moment of a short vibration ("haptic flash") that is applied when the two hands are spatially aligned. We further investigated the haptic FLE. First, we compared participants' performance in two isosensory visual or haptic conditions, in which moving object and flash were presented only in a single modality (visual: sphere and short color change, haptic: hand and vibration), and two bisensory conditions, in which the moving object was presented in both modalities (hand aligned with visible sphere), but the flash was presented only visually or only haptically. The experiment aimed to disentangle contributions of the flash's and the objects' modalities to the FLEs in haptics versus vision. We observed a FLE when the flash was visually displayed, both when the moving object was visual and visuo-haptic. Because the position of a visual flash, but not of an analogue haptic flash, is misjudged relative to a same visuo-haptic moving object, the difference between visual and haptic conditions can be fully attributed to characteristics of the flash. The second experiment confirmed that a haptic FLE can be observed depending on flash characteristics: the FLE increases with decreasing intensity of the flash (slightly modulated by flash duration), which had been previously observed for vision. These findings underline the high relevance of flash characteristics in different senses, and thus fit well with the temporal-sampling framework, where the flash triggers a high-level, supra-modal process of position judgement, the time point of which further depends on the processing time of the flash.

The buzz-lag effect

In the flash-lag illusion, a brief visual flash and a moving object presented at the same location appear to be offset with the flash trailing the moving object. A considerable amount of studies investigated the visual flash-lag effect, and flash-lag-like effects have also been observed in audition, and cross-modally between vision and audition. In the present study, we investigate whether a similar effect can also be observed when using only haptic stimuli. A fast vibration (or buzz, lasting less than 20 ms) was applied to the moving finger of the observers and employed as a "haptic flash." Participants performed a two-alternative forced-choice (2AFC) task where they had to judge whether the moving finger was located to the right or to the left of the stationary finger at the time of the buzz. We used two different movement velocities (Slow and Fast conditions). We found that the moving finger was systematically misperceived to be ahead of the stationary finger when the two were physically aligned. This result can be interpreted as a purely haptic analogue of the flash-lag effect, which we refer to as "buzz-lag effect." The buzz-lag effect can be well accounted for by the temporal-sampling explanation of flash-lag-like effects.

Naturalizing sense of agency with a hierarchical event-control approach

Unraveling the mechanisms underlying self and agency has been a difficult scientific problem. We argue for an event-control approach for naturalizing the sense of agency by focusing on the role of perception-action regularities present at different hierarchical levels and contributing to the sense of self as an agent. The amount of control at different levels of the control hierarchy determines the sense of agency. The current study investigates this approach in a set of two experiments using a scenario containing multiple agents sharing a common goal where one of the agents is partially controlled by the participant. The participant competed with other agents for achieving the goal and subsequently answered questions on identification (which agent was controlled by the participant), the degree to which they are confident about their identification (sense of identification) and the degree to which the participant believed he/she had control over his/her actions (sense of authorship). Results indicate a hierarchical relationship between goal-level control (higher level) and perceptual-motor control (lower level) for sense of agency. Sense of identification ratings increased with perceptual-motor control when the goal was not completed but did not vary with perceptual-motor control when the goal was completed. Sense of authorship showed a similar interaction effect only in experiment 2 that had only one competing agent unlike the larger number of competing agents in experiment 1. The effect of hierarchical control can also be seen in the misidentification pattern and misidentification was greater with the agent affording greater control. Results from the two studies support the event-control approach in understanding sense of agency as grounded in control. The study also offers a novel paradigm for empirically studying sense of agency and self.

Do the flash-lag effect and representational momentum involve similar extrapolations?

In the flash-lag effect (FLE) and in representational momentum (RM), the represented position of a moving target is displaced in the direction of motion. Effects of numerous variables on the FLE and on RM are briefly considered. In many cases, variables appear to have the same effect on the FLE and on RM, and this is consistent with a hypothesis that displacements in the FLE and in RM result from overlapping or similar mechanisms. In other cases, variables initially appear to have different effects on the FLE and on RM, but accounts reconciling those apparent differences with a hypothesis of overlapping or similar mechanisms are suggested. Given that RM is simpler and accounts for a wider range of findings (i.e., RM involves a single stimulus rather than the relationship between two stimuli, RM accounts for displacement in absolute position of a single stimulus and for differences in relative position of two stimuli), it is suggested that (at least some cases of) the FLE might be a special case of RM in which the position of the target is assessed relative to the position of another stimulus (i.e., the flashed object) rather than relative to the actual position of the target.

Effects of consciousness and consistency in manual control of visual stimulus on reduction of the flash-lag effect for luminance change

Four experiments investigated how observers' consciousness about their control of stimulus change affects the visual perception associated with the illusory flash-lag effect. In previous study (Ichikawa and Masakura, 2006), we found that the flash-lag effect in motion is reduced if observers were conscious that they were controlling stimulus movements by the use of computer mouse, even if the stimulus moved automatically, independently of observer's mouse control. In the other study (Ichikawa and Masakura, 2010a), we found that the consistent directional relationship between the observer's mouse control and stimulus movement, which is learned in our everyday computer use, is important for the reduction of the flash-lag effect in active observation. In the present study, we examined whether the reduction of the flash-lag effect in active observation requires the observers' consciousness about their control of stimulus change, and consistency in coupling mouse movement direction and stimulus change across trials in experiments. We used the flash-lag effect in luminance change because there is no intrinsic relationship between observer's mouse control and luminance change in our everyday computer use. We compared the illusory flash-lag effects for automatic change of the luminance with luminance change that was controlled by the observers' active manipulation of a computer mouse. Because the flash occurs randomly in time, observers could not anticipate when the flash was presented. Results suggest that the not only observer's consciousness of controlling the stimulus, but also consistency in coupling mouse movement direction with stimulus change, are required for the reduction of the flash-lag effect in active observation. The basis of the reduction of the flash-lag effect in active observation is discussed.

The effects of voluntary movements on auditory-haptic and haptic-haptic temporal order judgments

In two experiments we investigated the effects of voluntary movements on temporal haptic perception. Measures of sensitivity (JND) and temporal alignment (PSS) were obtained from temporal order judgments made on intermodal auditory-haptic (Experiment 1) or intramodal haptic (Experiment 2) stimulus pairs under three movement conditions. In the baseline, static condition, the arm of the participants remained stationary. In the passive condition, the arm was displaced by a servo-controlled motorized device. In the active condition, the participants moved voluntarily. The auditory stimulus was a short, 500Hz tone presented over headphones and the haptic stimulus was a brief suprathreshold force pulse applied to the tip of the index finger orthogonally to the finger movement. Active movement did not significantly affect discrimination sensitivity on the auditory-haptic stimulus pairs, whereas it significantly improved sensitivity in the case of the haptic stimulus pair, demonstrating a key role for motor command information in temporal sensitivity in the haptic system. Points of subjective simultaneity were by-and-large coincident with physical simultaneity, with one striking exception in the passive condition with the auditory-haptic stimulus pair. In the latter case, the haptic stimulus had to be presented 45ms before the auditory stimulus in order to obtain subjective simultaneity. A model is proposed to explain the discrimination performance.

Reduction of the flash-lag effect in terms of active observation

In the present study, we investigated how observers' control of stimulus change affects temporal and spatial aspects of visual perception. We compared the illusory flash-lag effects for automatic movement of the stimulus with stimulus movement that was controlled by the observers' active manipulation of a computer mouse (Experiments 1, 2, and 5), a keyboard (Experiment 3), or a trackball (Experiment 4). We found that the flash-lag effect was significantly reduced when the observer was familiar with the directional relationship between the mouse movement and stimulus movement on a front parallel display (Experiments 1 and 2) and that, although the unfamiliar directional relationship between the mouse movement and stimulus movement increased the flash-lag effect at the beginning of the experimental session, the repetitive observation with the same unfamiliar directional relationship reduced the flash-lag effect (Experiment 5). We found no consistent reduction of the flash-lag effect with the use of a keyboard or a trackball (Experiments 3 and 4). These results suggest that the learning of a specific directional relationship between a proprioceptive signal of hand movements and a visual signal of stimulus movements is necessary for the reduction of the flash-lag effect.

Voluntary action affects perception of bistable motion display

Hand action can bias the perceived direction of ambiguously moving objects (Wohlschlager, 2000 Vision Research 40 925-930). A common coding framework has been proposed to account for action-perception interaction, assuming goal-based feature matching between action and perception [Prinz, 2003, in Agency and Self-Awareness (New York: Oxford University Press)]. I investigated whether and how movement representation and a distal action goal are implicated in biased motion perception using a stream-bounce display as stimulus. By moving a mouse, participants controlled the progression of disks, which were initially separated and then moved toward each other. Mouse movement was found to bias perception toward the stream when the directional motion of the mouse and disk were consistent; however, this biased perception was not observed when the directions were orthogonal. Subsequent experiments revealed that this induced stream percept could not be attributed to the processes of attention, and the action goal of disk control was critical to the occurrence of biased motion perception. These results suggest that feature matching occurs at the level of movement representation, and the action goal is important in linking movement representation to perceptual representation.

Observer's control of the moving stimulus increases the flash-lag effect

The flash-lag effect (FLE) consists in perceiving a briefly presented stationary stimulus to lag behind an aligned moving stimulus. This study investigates the effects of actively controlling the moving stimulus. By means of a robotic arm, observers continuously moved a dot along a circular trajectory, and a flash was displayed closely at unpredictable times. In two experiments, we found that the FLE was larger when participants controlled the moving stimulus, compared to a computer-controlled condition. Two control conditions tested the possibility that the observed modulation of the FLE was due to visuo-spatial attention or dual-task factors. This study provides evidence that the motor system interacts with and possibly speeds up the processing of a moving visual stimulus when the observer controls its movement.

Performed or observed keyboard actions affect pianists' judgements of relative pitch

Action can affect visual perception if the action's expected sensory effects resemble a concurrent unstable or deviant event. To determine whether action can also change auditory perception, participants were required to play pairs of octave-ambiguous tones by pressing successive keys on a piano or computer keyboard and to judge whether each pitch interval was rising or falling. Both pianists and nonpianist musicians gave significantly more "rising" responses when the order of key presses was left-to-right than when it was right-to-left, in accord with the pitch mapping of the piano. However, the effect was much larger in pianists. Pianists showed a similarly large effect when they passively observed the experimenter pressing keys on a piano keyboard, as long as the keyboard faced the participant. The results suggest that acquired action-effect associations can affect auditory perceptual judgement.

Distinct position assignment mechanisms revealed by cross-order motion

Motion perception influences perceived position. It has been shown that first-order (luminance defined) motion shifts perceived position across a wide range of spatial and temporal frequencies. On the other hand, second-order (contrast defined) motion shifts perceived position over a narrow range of temporal frequencies, regardless of spatial frequency [Bressler, D. W., & Whitney, D. (2006). Second-order motion shifts perceived position. Vision Research, 46(6-7), 1120-1128]. These results suggest the presence of distinct position assignment mechanisms for first- and second-order motion. We investigated whether the first- and second-order systems independently encode and assign the position of a moving stimulus. To measure motion induced position shift we presented two horizontally offset Gabors placed above and below a central fixation point, with sine wave carriers drifting in opposite directions. Subjects judged the position of the top Gabor relative to the bottom one. We used both first-order Gabors (sinusoidal luminance modulation of a dynamic noise carrier enveloped by a static Gaussian) and second-order Gabors (sinusoidal contrast modulation of a dynamic noise carrier enveloped by a static Gaussian). Results showed a strong position shift in the direction of the carrier motion when both Gabors were first-order, a weak position shift when both Gabors were second-order, and no appreciable position shift when one Gabor was first-order and the other was second-order (cross-order motion). The absence of a position shift using cross-order motion supports the hypothesis that the two motion systems independently encode and assign the position of a moving object. These results are consistent with those of experiments investigating global spatial interactions between static first-order and second-order Gabor patches, indicating a commonality in the underlying spatial integration processes.