A naturalistic viewing paradigm using 360° panoramic video clips and real-time field-of-view changes with eye-gaze tracking

Use of functional magnetic resonance imaging to identify cortical loci for lower limb movements and their efficacy for individuals after stroke

BACKGROUND

Identification of cortical loci for lower limb movements for stroke rehabilitation is crucial for better rehabilitation outcomes via noninvasive brain stimulation by targeting the fine-grained cortical loci of the movements. However, identification of the cortical loci for lower limb movements using functional MRI (fMRI) is challenging due to head motion and difficulty in isolating different types of movement. Therefore, we developed a custom-made MR-compatible footplate and leg cushion to identify the cortical loci for lower limb movements and conducted multivariate analysis on the fMRI data. We evaluated the validity of the identified loci using both fMRI and behavioral data, obtained from healthy participants as well as individuals after stroke.

METHODS

We recruited 33 healthy participants who performed four different lower limb movements (ankle dorsiflexion, ankle rotation, knee extension, and toe flexion) using our custom-built equipment while fMRI data were acquired. A subgroup of these participants (Dataset 1; n = 21) was used to identify the cortical loci associated with each lower limb movement in the paracentral lobule (PCL) using multivoxel pattern analysis and representational similarity analysis. The identified cortical loci were then evaluated using the remaining healthy participants (Dataset 2; n = 11), for whom the laterality index (LI) was calculated for each lower limb movement using the cortical loci identified for the left and right lower limbs. In addition, we acquired a dataset from 15 individuals with chronic stroke for regression analysis using the LI and the Fugl-Meyer Assessment (FMA) scale.

RESULTS

The cortical loci associated with the lower limb movements were hierarchically organized in the medial wall of the PCL following the cortical homunculus. The LI was clearer using the identified cortical loci than using the PCL. The healthy participants (mean ± standard deviation: 0.12 ± 0.30; range: - 0.63 to 0.91) exhibited a higher contralateral LI than the individuals after stroke (0.07 ± 0.47; - 0.83 to 0.97). The corresponding LI scores for individuals after stroke showed a significant positive correlation with the FMA scale for paretic side movement in ankle dorsiflexion (R2 = 0.33, p = 0.025) and toe flexion (R2 = 0.37, p = 0.016).

CONCLUSIONS

The cortical loci associated with lower limb movements in the PCL identified in healthy participants were validated using independent groups of healthy participants and individuals after stroke. Our findings suggest that these cortical loci may be beneficial for the neurorehabilitation of lower limb movement in individuals after stroke, such as in developing effective rehabilitation interventions guided by the LI scores obtained for neuronal activations calculated from the identified cortical loci across the paretic and non-paretic sides of the brain.

Decoding the dynamic perception of risk and speed using naturalistic stimuli: A multivariate, whole-brain analysis

Time-resolved decoding of speed and risk perception in car driving is important for understanding the perceptual processes related to driving safety. In this study, we used an fMRI-compatible trackball with naturalistic stimuli to record dynamic ratings of perceived risk and speed and investigated the degree to which different brain regions were able to decode these. We presented participants with first-person perspective videos of cars racing on the same course. These videos varied in terms of subjectively perceived speed and risk profiles, as determined during a behavioral pilot. During the fMRI experiment, participants used the trackball to dynamically rate subjective risk in a first and speed in a second session and assessed overall risk and speed after watching each video. A standard multivariate correlation analysis based on these ratings revealed sparse decodability in visual areas only for the risk ratings. In contrast, the dynamic rating-based correlation analysis uncovered frontal, visual, and temporal region activation for subjective risk and dorsal visual stream and temporal region activation for subjectively perceived speed. Interestingly, further analyses showed that the brain regions for decoding risk changed over time, whereas those for decoding speed remained constant. Overall, our results demonstrate the advantages of time-resolved decoding to help our understanding of the dynamic networks associated with decoding risk and speed perception in realistic driving scenarios.

Examining the exposure-reception-retention link in realistic communication environments via VR and eye-tracking: The VR billboard paradigm

Exposure is key to message effects. No effects can ensue if a health, political, or commercial message is not noticed. Yet, existing research in communication, advertising, and related disciplines often measures 'opportunities for exposure' at an aggregate level, whereas knowing whether recipients were 'actually exposed' to a message requires a micro-level approach. Micro-level research, on the other hand, focuses on message processing and retention, takes place under highly controlled laboratory conditions with forced message exposure, and largely ignores how recipients attend selectively to messages under more natural conditions. Eye-tracking enables us to assess actual exposure, but its previous applications were restricted to screen-based reading paradigms lacking ecological validity or field studies that suffer from limited experimental control. Our solution is to measure eye-tracking within an immersive VR environment that creates the message delivery and reception context. Specifically, we simulate a car ride down a highway alongside which billboards are placed. The VR headset (HP Omnicept Pro) provides an interactive 3D view of the environment and holds a seamlessly integrated binocular eye tracker that records the drivers' gaze and detects all fixations on the billboards. This allows us to quantify the nexus between exposure and reception rigorously, and to link our measures to subsequent memory, i.e., whether messages were remembered, forgotten, or not even encoded. An empirical study shows that incidental memory for messages differs based on participants' gaze behavior while passing the billboards. The study further shows how an experimental manipulation of attentional demands directly impacts drivers' gaze behavior and memory. We discuss the large potential of this paradigm to quantify exposure and message reception in realistic communication environments and the equally promising applications in new media contexts (e.g., the Metaverse).

Neural representations of the perception of handwritten digits and visual objects from a convolutional neural network compared to humans

We investigated neural representations for visual perception of 10 handwritten digits and six visual objects from a convolutional neural network (CNN) and humans using functional magnetic resonance imaging (fMRI). Once our CNN model was fine-tuned using a pre-trained VGG16 model to recognize the visual stimuli from the digit and object categories, representational similarity analysis (RSA) was conducted using neural activations from fMRI and feature representations from the CNN model across all 16 classes. The encoded neural representation of the CNN model exhibited the hierarchical topography mapping of the human visual system. The feature representations in the lower convolutional (Conv) layers showed greater similarity with the neural representations in the early visual areas and parietal cortices, including the posterior cingulate cortex. The feature representations in the higher Conv layers were encoded in the higher-order visual areas, including the ventral/medial/dorsal stream and middle temporal complex. The neural representations in the classification layers were observed mainly in the ventral stream visual cortex (including the inferior temporal cortex), superior parietal cortex, and prefrontal cortex. There was a surprising similarity between the neural representations from the CNN model and the neural representations for human visual perception in the context of the perception of digits versus objects, particularly in the primary visual and associated areas. This study also illustrates the uniqueness of human visual perception. Unlike the CNN model, the neural representation of digits and objects for humans is more widely distributed across the whole brain, including the frontal and temporal areas.

A Case for Studying Naturalistic Eye and Head Movements in Virtual Environments

Using head mounted displays (HMDs) in conjunction with virtual reality (VR), vision researchers are able to capture more naturalistic vision in an experimentally controlled setting. Namely, eye movements can be accurately tracked as they occur in concert with head movements as subjects navigate virtual environments. A benefit of this approach is that, unlike other mobile eye tracking (ET) set-ups in unconstrained settings, the experimenter has precise control over the location and timing of stimulus presentation, making it easier to compare findings between HMD studies and those that use monitor displays, which account for the bulk of previous work in eye movement research and vision sciences more generally. Here, a visual discrimination paradigm is presented as a proof of concept to demonstrate the applicability of collecting eye and head tracking data from an HMD in VR for vision research. The current work’s contribution is 3-fold: firstly, results demonstrating both the strengths and the weaknesses of recording and classifying eye and head tracking data in VR, secondly, a highly flexible graphical user interface (GUI) used to generate the current experiment, is offered to lower the software development start-up cost of future researchers transitioning to a VR space, and finally, the dataset analyzed here of behavioral, eye and head tracking data synchronized with environmental variables from a task specifically designed to elicit a variety of eye and head movements could be an asset in testing future eye movement classification algorithms.

Individualized video recommendation modulates functional connectivity between large scale networks

With the emergence of AI‐powered recommender systems and their extensive use in the video streaming service, questions and concerns also arise. Why can recommended video content continuously capture users' attention? What is the impact of long‐term exposure to personalized video content on one's behaviors and brain functions? To address these questions, we designed an fMRI experiment presenting participants with personally recommended videos and generally recommended ones. To examine how large‐scale networks were modulated by personalized video content, graph theory analysis was applied to investigate the interaction between seven networks, including the ventral and dorsal attention networks (VAN, DAN), frontal–parietal network (FPN), salience network (SN), and three subnetworks of default mode network (dorsal medial prefrontal (dMPFC), Core, and medial temporal lobe (MTL)). Our results showed that viewing nonpersonalized video content mainly enhanced the connectivity in the DAN‐FPN‐Core pathway, whereas viewing personalized ones increased not only the connectivity in this pathway but also the DAN‐VAN‐dMPFC pathway. In addition, both personalized and nonpersonalized short videos decreased the couplings between SN and VAN as well as between two DMN subsystems, Core and MTL. Collectively, these findings uncovered distinct patterns of network interactions in response to short videos and provided insights into potential neural mechanisms by which human behaviors are biased by personally recommended content.

A cross-cultural comparison of visual search strategies and response times in road hazard perception testing

Road hazard perception is considered the most prominent higher-order cognitive skill related to traffic-accident involvement. Regional cultures and social rules that govern acceptable behavior may influence drivers' interpretation of a traffic situation and, consequently, the correct identification of potentially hazardous situations. Here, we aimed to compare hazard perception skills among four European countries that differ in their traffic culture, policies to reduce traffic risks, and fatal crashes: Ukraine, Italy, Spain, and Sweden. We developed a static hazard perception test in which driving scenes with different levels of braking affordance were presented while drivers' gaze was recorded. The test required drivers to indicate the action they would undertake: to brake vs. to keep driving. We assessed 218 young adult drivers. Multilevel models revealed that the scenes' levels of braking affordance (i.e., road hazard) modulated drivers' behavior. As the levels of braking affordance increased, drivers' responses became faster and their gaze entropy decreased (i.e., visual search strategy became less erratic). The country of origin influenced these effects. Ukrainian drivers were the fastest and Swedish drivers were the slowest to respond. For all countries, the decrement in response times was less marked in the case of experienced drivers. Also, Spanish drivers showed the most structured (least erratic) visual search strategy, whereas the Italians had the most rigid (most constant) one. These results suggest that road hazard perception can be defined cross-culturally, with cultural factors (e.g., traffic climate, legislation) modulating response times and visual search strategies. Our results also support the idea that a multimodal assessment methodology is possible for mass testing of road hazard perception and its outcomes would be relevant to understand how different traffic cultures shape driving behavior.