Predicting road scenes from brief views of driving video

Road Hazard Stimuli: Annotated naturalistic road videos for studying hazard detection and scene perception

Driving requires vision, yet there is little empirical data about how vision and cognition support safe driving. It is difficult to study perception during natural driving because the experimental rigor required would be dangerous and unethical to implement on the road. The driving environment is complex, dynamic, and immensely variable, making it extremely challenging to accurately replicate in simulation. Our proposed solution is to study vision using stimuli which reflect this inherent complexity by using footage of real driving situations. To this end, we curated a set of 750 crowd-sourced video clips (434 hazard and 316 no-hazard clips), which have been spatially, temporally, and categorically annotated. These annotations describe where the hazard appears, what it is, and when it occurs. In addition, perceived dangerousness changes from moment to moment and is not a simple binary detection judgement. To capture this more granular aspect of our stimuli, we asked 48 observers to rate the perceived hazardousness of 1356 brief video clips taken from these 750 source clips on a continuous scale. These ratings span the entire scale, have high interrater agreement, and are robust to driving history. This novel stimulus set is not only useful for understanding drivers' ability to detect hazards, but is also a tool for studying dynamic scene perception and other aspects of visual function. While this stimulus set was originally designed for behavioral studies, researchers interested in other areas such as traffic safety or computer vision may also find this dataset a useful resource.

Highly dangerous road hazards are not immune from the low prevalence effect

The low prevalence effect (LPE) is a cognitive limitation commonly found in visual search tasks, in which observers miss rare targets. Drivers looking for road hazards are also subject to the LPE. However, not all road hazards are equal; a paper bag floating down the road is much less dangerous than a rampaging moose. Here, we asked whether perceived hazardousness modulated the LPE. To examine this, we took a dataset in which 48 raters assessed the perceived dangerousness of hazards in recorded road videos (Song et al. in Behav Res Methods, 2023. https://doi.org/10.3758/s13428-023-02299-8 ) and correlated the ratings with data from a hazard detection task using the same stimuli with varying hazard prevalence rates (Kosovicheva et al. in Psychon Bull Rev 30(1):212-223, 2023. https://doi.org/10.3758/s13423-022-02159-0 ). We found that while hazard detectability increased monotonically with hazardousness ratings, the LPE was comparable across perceived hazardousness levels. Our findings are consistent with the decision criterion account of the LPE, in which target rarity induces a conservative shift in criterion. Importantly, feedback was necessary for a large and consistent LPE; when participants were not given feedback about their accuracy, the most dangerous hazards showed a non-significant LPE. However, eliminating feedback was not enough to induce the opposite of the LPE-prevalence induced concept change (Levari et al. in Science 360(6396):1465-1467, 2018. https://doi.org/10.1126/science.aap8731 ), in which participants adopt a more liberal criterion when instances of a category become rare. Our results suggest that the road hazard LPE may be somewhat affected by the inherent variability of driving situations, but is still observed for highly dangerous hazards.

Impact of Temporary Browsing Restrictions on Drivers’ Situation Awareness When Interacting with In-Vehicle Infotainment Systems

Looking away from the road during a task degrades situation awareness of potential hazards. Long glances back to the road rebuild this awareness and are thought to be critical for maintaining good vehicle control and recognizing conflicts. To further investigate the importance of rebuilding situation awareness, a controlled test-track study was performed that evaluated drivers’ hazard awareness and response performance to a surprise event after completing a task that involved pausing partway through it to look back at the road. Thirty-two drivers completed a visual-manual infotainment system secondary task. Half of the drivers were instructed to pause their browsing mid-task, while the others were not. While the task was being performed, a lead vehicle activated its hazard lights. It then unexpectedly dropped a fake muffler once drivers completed the task. Drivers’ visual attention to the road and their ability to respond to the muffler were measured. The drivers that paused their browsing were more aware of the lead vehicle’s hazard lights, showed less surprise to the dropped muffler, and executed more measured avoidance maneuvers compared to the drivers that did not pause their browsing. These findings suggest that drivers’ situation awareness can be better maintained when task interactions are paced, allowing for longer monitoring of the environment. Mechanisms that encourage drivers to take restorative on-road glances during extended browsing may be a key aspect of an overall approach to mitigating driver distraction.

Toward a Theory of Visual Information Acquisition in Driving

OBJECTIVE

The aim of this study is to describe information acquisition theory, explaining how drivers acquire and represent the information they need.

BACKGROUND

While questions of what drivers are aware of underlie many questions in driver behavior, existing theories do not directly address how drivers in particular and observers in general acquire visual information. Understanding the mechanisms of information acquisition is necessary to build predictive models of drivers' representation of the world and can be applied beyond driving to a wide variety of visual tasks.

METHOD

We describe our theory of information acquisition, looking to questions in driver behavior and results from vision science research that speak to its constituent elements. We focus on the intersection of peripheral vision, visual attention, and eye movement planning and identify how an understanding of these visual mechanisms and processes in the context of information acquisition can inform more complete models of driver knowledge and state.

RESULTS

We set forth our theory of information acquisition, describing the gap in understanding that it fills and how existing questions in this space can be better understood using it.

CONCLUSION

Information acquisition theory provides a new and powerful way to study, model, and predict what drivers know about the world, reflecting our current understanding of visual mechanisms and enabling new theories, models, and applications.

APPLICATION

Using information acquisition theory to understand how drivers acquire, lose, and update their representation of the environment will aid development of driver assistance systems, semiautonomous vehicles, and road safety overall.

Neural integration underlying naturalistic prediction flexibly adapts to varying sensory input rate

Prediction of future sensory input based on past sensory information is essential for organisms to effectively adapt their behavior in dynamic environments. Humans successfully predict future stimuli in various natural settings. Yet, it remains elusive how the brain achieves effective prediction despite enormous variations in sensory input rate, which directly affect how fast sensory information can accumulate. We presented participants with acoustic sequences capturing temporal statistical regularities prevalent in nature and investigated neural mechanisms underlying predictive computation using MEG. By parametrically manipulating sequence presentation speed, we tested two hypotheses: neural prediction relies on integrating past sensory information over fixed time periods or fixed amounts of information. We demonstrate that across halved and doubled presentation speeds, predictive information in neural activity stems from integration over fixed amounts of information. Our findings reveal the neural mechanisms enabling humans to robustly predict dynamic stimuli in natural environments despite large sensory input rate variations.

Introduction to special issue on "Prediction in Perception and Action"

The wide diversity of articles in this issue reveals an explosion of evidence for the mechanisms of prediction in the visual system. When thought of as visual priors, predictive mechanisms can be seen as tightly interwoven with incoming sensory data. Prediction is thus a fundamental and essential aspect not only of visual perception but of the actions that are guided by perception.