Comparative study of the effects of auditory, visual and multimodality displays on drivers' performance in advanced traveller information systems

Audiovisual presentation of variable message signs improves message processing in distracted drivers during partially automated driving

INTRODUCTION

Highly automated driving is expected to reduce the accident risk occurrence by human errors, but it can also increase driver distraction. Previous evidence shows that auditory signals can help drivers take over in critical situations. However, it is still uncertain whether the potential benefit of verbal auditory signals could be generalized to driving situations where drivers are visually and auditorily distracted.

METHOD

Our first objective was to compare the effectiveness of complementary audio messages (audio + visual condition) and visual only (visual condition) variable message signs (VMS) messages. The second objective was to explore the potential use of oral messages with traffic information to help highly-automated vehicle drivers identify critical situations. Eye-tracking data were also registered. Twenty-four volunteers participated in a driving simulator study, completing two tasks: (a) a TV series task, where they had to pay attention to an episode of a TV series while traveling along the route; and (b) a VMS task, where they had to recover the manual control of the car if the VMS message was a 'critical message.'

RESULTS

General results showed that, when the audio was available, the participants: (a) had a higher ability to discriminate the VMS messages, (b) were less conservative, (c) responded earlier, and (d) their pattern of fixations was more efficient. A complementary analysis showed that the counterbalance order was a moderating factor for the discrimination ability and the response distance measures. This evidence suggests a potential learning effect, not cancelled by counterbalancing the order of the conditions.

CONCLUSION

The processing of traffic messages may improve when provided as oral and visual messages.

PRACTICAL APPLICATIONS

These results would be of special interest for engineers designing highly automated cars, considering that the design of automated systems must ensure that the driver's attention is sufficient to take over control.

A comparison of visual and auditory EEG interfaces for robot multi-stage task control

Shared autonomy holds promise for assistive robotics, whereby physically-impaired people can direct robots to perform various tasks for them. However, a robot that is capable of many tasks also introduces many choices for the user, such as which object or location should be the target of interaction. In the context of non-invasive brain-computer interfaces for shared autonomy-most commonly electroencephalography-based-the two most common choices are to provide either auditory or visual stimuli to the user-each with their respective pros and cons. Using the oddball paradigm, we designed comparable auditory and visual interfaces to speak/display the choices to the user, and had users complete a multi-stage robotic manipulation task involving location and object selection. Users displayed differing competencies-and preferences-for the different interfaces, highlighting the importance of considering modalities outside of vision when constructing human-robot interfaces.

Fast & scrupulous: Gesture-based alarms improve accuracy and reaction times under various mental workload levels. An ERSP study

In high-risk environments, fast and accurate responses to warning systems are essential to efficiently handle emergency situations. The aim of the present study was twofold: 1) investigating whether hand action videos (i.e., gesture alarms) trigger faster and more accurate responses than text alarm messages (i.e., written alarms), especially when mental workload (MWL) is high; and 2) investigating the brain activity in response to both types of alarms as a function of MWL. Regardless of MWL, participants (N = 28) were found to be both faster and more accurate when responding to gesture alarms than to written alarms. Brain electrophysiological results suggest that this greater efficiency might be due to a facilitation of the action execution, reflected by the decrease in mu and beta power observed around the response time window observed at C3 and C4 electrodes. These results suggest that gesture alarms may improve operators' performances in emergency situations.

Input modality matters: A comparison of touch, speech, and gesture based in-vehicle interaction

Innovative input devices are being available for in-vehicle information systems (IVISs). While they have the potential to provide enjoyable driving by enabling drivers to perform non-driving related tasks (NDRTs) in more natural ways, the associated distracting effects should be paid with more attention. The purpose of this exploratory study was to compare the effects of three novel input modalities, i.e., touchscreen-based interaction (TBI), speech-based interaction (SBI), and gesture-based interaction (GBI), on driving performance and driver visual behaviors. Moreover, we examined if the influence of different modalities would be moderated by the difficulty level of NDRTs. A total of 36 participants were invited to a simulated driving experiment where they were randomly assigned to one of the four groups (TBI, GBI, SBI or baseline) and completed three driving trials. The results showed that TBI led to the worse driving performance, as indicated by the significantly prolonged reaction time, reduced minimum time-to-collision, and increased variations in both longitudinal and lateral vehicle control. The deteriorated driving performance could be attributed, at least partially, to the intense visual demand induced by looking towards the touchscreen, as indicated by more and longer off-the-road glances. The adverse impacts of GBI were relatively smaller, but it still posed great crash risk by leading to a shorter minimum time-to-collision and less stable vehicle control compared to the baseline. SBI, although not completely equivalent to the baseline group, showed the minimum influence on driving and visual performance. Only very few interaction effects were found, suggesting that the effects of modality were quite robust across different NDRTs. It was concluded that SBI and GBI provided safer alternatives to in-vehicle interaction than TBI.

Young Novice Drivers' Cognitive Distraction Detection: Comparing Support Vector Machines and Random Forest Model of Vehicle Control Behavior

The use of mobile phones has become one of the major threats to road safety, especially in young novice drivers. To avoid crashes induced by distraction, adaptive distraction mitigation systems have been developed that can determine how to detect a driver's distraction state. A driving simulator experiment was conducted in this paper to better explore the relationship between drivers' cognitive distractions and traffic safety, and to better analyze the mechanism of distracting effects on young drivers during the driving process. A total of 36 participants were recruited and asked to complete an n-back memory task while following the lead vehicle. Drivers' vehicle control behavior was collected, and an ANOVA was conducted on both lateral driving performance and longitudinal driving performance. Indicators from three aspects, i.e., lateral indicators only, longitudinal indicators only, and combined lateral and longitudinal indicators, were inputted into both SVM and random forest models, respectively. Results demonstrated that the SVM model with parameter optimization outperformed the random forest model in all aspects, among which the genetic algorithm had the best parameter optimization effect. For both lateral and longitudinal indicators, the identification effect of lateral indicators was better than that of longitudinal indicators, probably because drivers are more inclined to control the vehicle in lateral operation when they were cognitively distracted. Overall, the comprehensive model built in this paper can effectively identify the distracted state of drivers and provide theoretical support for control strategies of driving distraction.

A user study of directional tactile and auditory user interfaces for take-over requests in conditionally automated vehicles

General introduction of unconditionally and conditionally automated vehicles is expected to have a highly positive impact on the society, from increased accessibility to mobility and road traffic safety, to decreased environmental and economic negative impacts. However, there are several obstacles and risks slowing down the adoption of this technology, which are primarily related to the human-machine interaction (HMI) and exchange of control between the vehicle and the human driver. In this article, we present key takeaways for HMI design of take-over requests (TOR) that the vehicle issues to inform the driver to take over control of the vehicle. The key takeaways were developed based on the results of a user study, where directional tactile-ambient (visual) and auditory-ambient (visual) TOR user interfaces (UI) were evaluated with regards to commonly used take-over quality aspects (attention redirection, take-over time, correct interpretation of stimuli, off-road drive, brake application, lateral acceleration, minimal time-to-collision and occurrence of collision). 36 participants took part in the mixed design study, which was conducted in a driving simulator. The results showed that drivers' attention was statistically significantly faster redirected with the auditory-ambient UI, however using the tactile-ambient UI resulted in less off-road driving and slightly less collisions. The results also revealed that drivers correctly interpreted the directional TOR stimuli more often than the non-directional one. Based on the study results, a list of key takeaways was developed and is presented in the conclusion of the paper. The results from this study are especially relevant to the TOR UI designers and the automotive industry, which tend to provide the most usable UI for ensuring safer end efficient human-vehicle interaction during the TOR task.

Multisensory GPS impact on spatial representation in an immersive virtual reality driving game

Individuals are increasingly relying on GPS devices to orient and find their way in their environment and research has pointed to a negative impact of navigational systems on spatial memory. We used immersive virtual reality to examine whether an audio–visual navigational aid can counteract the negative impact of visual only or auditory only GPS systems. We also examined the effect of spatial representation preferences and abilities when using different GPS systems. Thirty-four participants completed an IVR driving game including 4 GPS conditions (No GPS; audio GPS; visual GPS; audio–visual GPS). After driving one of the routes in one of the 4 GPS conditions, participants were asked to drive to a target landmark they had previously encountered. The audio–visual GPS condition returned more accurate performance than the visual and no GPS condition. General orientation ability predicted the distance to the target landmark for the visual and the audio–visual GPS conditions, while landmark preference predicted performance in the audio GPS condition. Finally, the variability in end distance to the target landmark was significantly reduced in the audio–visual GPS condition when compared to the visual and audio GPS conditions. These findings support theories of spatial cognition and inform the optimisation of GPS designs.

Attention Pedestrians Ahead: Evaluating User Acceptance and Perceptions of a Cooperative Intelligent Transportation System-Warning System for Pedestrians

Warning system for pedestrians (WSP), one of cooperative intelligent transport system (C-ITS) applications, is designed to increase safety for pedestrians but also for drivers and other road users. The evaluation of end-user acceptance and perceptions of this technology is crucial before deploying it in transportation systems. Five WSP human–machine interfaces (HMIs) were designed and simulated using a driver’s first-view video footage of driving through a pedestrian crossing in Newcastle upon Tyne. The five WSP designs were evaluated with 24 younger end users (35 years old and younger). This study first evaluated the usefulness of the unified theory of acceptance and use of technology (UTAUT) in modelling end-user acceptance in terms of behavioural intentions to use WSP. The results suggest that the UTAUT can be applied to investigate the end-user acceptance of WSP, with performance expectancy and effort expectancy influencing the behavioural intentions to use WSP. Furthermore, we investigated end-user attitudes towards various WSP human–machine interface (HMI) designs. Participants showed more positive attitudes towards visual-only interfaces than towards audio-only and multi-modal (combinations of visual and audio) interfaces. Above all, the findings of this research increase our understanding of public acceptance and perceptions of this C-ITS application.

The effects of age and physical exercise on multimodal signal responses: Implications for semi-autonomous vehicle takeover requests

The present study examined whether the non-chronological age factor, engagement in physical exercise, affected responses to multimodal (combinations of visual, auditory, and/or tactile) signals differently between younger and older adults in complex environments. Forty-eight younger and older adults were divided into exercise and non-exercise groups, and rode in a simulated Level 3 autonomous vehicle under four different task conditions (baseline, video watching, headway estimation, and video-headway combination), while being asked to respond to various multimodal warning signals. Overall, bi- and trimodal warnings had faster response times for both age groups across driving conditions, but was more pronounced for older adults. Engagement in physical exercise was associated with smaller maximum braking force for younger participants only, and also corresponded to longer average fixation durations, compared to the non-exercise group. Findings from this research can help to guide decisions about the design of warning and information systems for semi-autonomous vehicles.

Influence of prompt timing and messages of an audio navigation system on driver behavior on an urban expressway with five exits

An audio navigation system is a very useful tool for driving path guidance with less distraction. However, the influence of the audio navigation system on driver behavior, especially in complicated road environments, is still not entirely clear. This study aims to investigate navigation prompt timing (NPT), navigation prompt message (NPM), and their combination in an audio navigation system on driving behavior on an urban expressway with five exits. Driving simulator technology was used to reproduce the experimental environment and obtain driver behavior data. Four indicators-speed, standard deviation of speed, absolute values of acceleration, and depth of accelerator-were selected to examine the main and interactive effects of NPT, NPM, and their combination based on repeated measures analysis of variance. The results show that the driver's psychological state and operation of the vehicle on the urban expressway were affected by the prompt timing and messages of the audio navigation system. An interaction effect existed between prompt timing and prompt messages of the system, and this effect adjusted the effect on the driver's psychological state and vehicle operation caused by these two important factors. This study contributes to our understanding of the influence of audio navigation systems on driver behavior in complex road environments and thus lays a foundation for developing standard audio navigation broadcast guidelines to improve drivers' acceptance of navigation systems, reduce drivers' cognitive workload, and improve the level of vehicle operation safety.

Allocation of Driver Attention for Varying In-Vehicle System Modalities

OBJECTIVE

This paper examines drivers' allocation of attention using response time to a tactile detection response task (TDRT) while interacting with an in-vehicle information system (IVIS) over time.

BACKGROUND

Longer TDRT response time is associated with higher cognitive workload. However, it is not clear what role is assumed by the human and system in response to varying in-vehicle environments over time.

METHOD

A driving simulator study with 24 participants was conducted with a restaurant selection task of two difficulty levels (easy and hard) presented in three modalities (audio only, visual only, hybrid). A linear mixed-effects model was applied to identify factors that affect TDRT response time. A nonparametric time-series model was also used to explore the visual attention allocation under the hybrid mode over time.

RESULTS

The visual-only mode significantly increased participants' response time compared with the audio-only mode. Females took longer to respond to the TDRT when engaged with an IVIS. The study showed that participants tend to use the visual component more toward the end of the easy tasks, whereas the visual mode was used more at the beginning of the harder tasks.

CONCLUSION

The visual-only mode of the IVIS increased drivers' cognitive workload when compared with the auditory-only mode. Drivers showed different visual attention allocation during the easy and hard restaurant selection tasks in the hybrid mode.

APPLICATION

The findings can help guide the design of automotive user interfaces and help manage cognitive workload.

Are safety and performance affected by navigation system display size, environmental illumination, and gender when driving in both urban and rural areas?

Driving with the aid of a navigation system could distract drivers. A high level of distraction influences driver performance and safety, leading to a possible increase in road crashes. The illumination level and size of the GPS display may influence the duration and frequency of a driver's glances, which in turn may affect driver distraction. In a simulated driving experiment requiring the use of a GPS, the GPS's display size and illumination level were examined, in light of the driver's experience and gender, to understand their effects on the performance and safety of young drivers on roads in urban and rural areas. Twenty young subjects, male and female between the ages of 18 and 29 years, participated in this experiment. Driving safety was evaluated by lateral control (number of pavement and lane line crossings), number of crashes, number of near misses, and the total time out of the lane. Driving performance was evaluated by the number of navigational errors, the total time making navigational errors, number of times the speed limit was exceeded and total amount of time speeding. These measures were analyzed using a repeated measures analysis of variance (ANOVA) model. Furthermore, the effects of the GPS display, considering the driving experience, were investigated with a simple linear regression. Findings suggest that driving with a small GPS display in an urban area leads to more navigational errors than driving with a large GPS display. Furthermore, more speed limit violations tend to occur in rural areas in the daytime than at night. Moreover, in urban areas, male drivers tend to have the highest number of crashes during the daytime. Furthermore, in rural areas, males tend to violate the speed limit more often and for longer periods of time during the daytime than at night and more than females do. Additionally, when navigating with a GPS system, young experienced drivers drive safer than inexperience drivers. The findings are of interest to designers and transportation researchers concerned with improving GPSs to enhance driving safety and performance.

Human Mirror Neuron System Based Alarms in the Cockpit: A Neuroergonomic Evaluation

Controlled Flight Into Terrain (CFIT) events still remain among the deadliest accidents in aviation. When facing the possible occurrence of such an event, pilots have to immediately react to the ground proximity alarm ("Pull Up" alarm) in order to avoid the impending collision. However, the pilots' reaction to this alarm is not always optimal. This may be at least partly due to the low visual saliency of the current alarm and the deleterious effects of stress that alleviate the pilot's reactions. In the present study, two experiments (in a laboratory and in a flight simulator) were conducted to (1) investigate whether hand gesture videos (a hand pulling back the sidestick) can trigger brainwave frequencies related to the mirror neuron system; (2) determine whether enhancing the visual characteristics of the "Pull Up" alarm could improve pilots' response times. Electrophysiological results suggest that hand gesture videos attracted more participants' attention (greater alpha desynchronization in the parieto-occipital area) and possibly triggered greater activity of the mirror neuron system (greater mu and beta desynchronizations at central electrodes). Results obtained in the flight simulator revealed that enhancing the visual characteristics of the original "Pull Up" alarm improved the pilots' reaction times. However, no significant difference in reaction times between an enlarged "Pull Up" inscription and the hand gesture video was found. Further work is needed to determine whether mirror neuron system based alarms could bring benefits for flight safety, in particular, these alarms should be assessed during a high stress context.

I want to brake free: The effect of connected vehicle features on driver behaviour, usability and acceptance

This study evaluated the effectiveness and acceptance of four connected vehicle features, i.e. Emergency Electronic Brake Lights (EEBL), Emergency Vehicle Warning (EVW), Roadworks warning (RWW) and Traffic Condition Warning (TCW) which were presented via a mobile phone mounted near the line of sight. A driving simulator study was conducted in which 36 drivers were exposed to different levels of urgent and critical situations. They involved the approach of an emergency vehicle, an emergency braking of a lead vehicle, a roadworks area and a congested section of a road. All these events took place in a simulated motorway scenario. In the EEBL event, the vehicle braking ahead with the brake lights on was either visible or not (between-subjects). Whereas no effect of RWW and TCW were observed on driving behaviour, results showed that drivers who were shown the EEBL warnings had shorter braking and decelerating response times, and a slower mean speed during the events, and this was independent of brake lights visibility. The EVW resulted in participants giving way to the emergency vehicle (i.e. staying on the slow lane instead of overtaking slower vehicles) more frequently than those who did not receive the warning. The mobile phone app was accepted and considered usable. Locating the mobile phone in different locations within the drivers' line of sight (i.e. dashboard, instrument cluster) did not impact significantly neither drivers' attitudes nor behaviour. Additional in-vehicle information systems could enhance safety and allow emergency vehicles to get faster to their destination.

Effects of visual complexity of in-vehicle information display: Age-related differences in visual search task in the driving context

We aimed to investigate the effects of the visual complexity of in-vehicle information display and driver's age in a driving context. A driving simulator study was conducted where participants performed visual search tasks at different visual complexity levels while driving. Two groups were included, 20 younger drivers (mean age = 28.75 years) and 14 older drivers (mean age = 54.87 years). Older drivers were found to be more vulnerable to the effects of increased visual complexity when performing a visual search task. The task completion time of the younger group increased by about 20% (from 7.69 s to 9.30 s), while the older group increased by about 47% (from 8.92 s to 13.14 s). Further, the driving performance of the older group deteriorated, unlike the younger group. The subjective workload score supported the results of the objective performance measures. These differences can be explained by glance behavior. The total off-road glance duration of older drivers was longer than that of younger drivers, but the average off-road glance duration of younger drivers was longer. In other words, older drivers have a more conservative strategy when dealing with increased visual complexity in a driving context so as not to affect their driving. The findings of this study show that the visual complexity level has a significant effect on driving behaviors, especially in older drivers, which provides insights for designing in-vehicle information displays.

The effects of auditory satellite navigation instructions and visual blur on road hazard perception

The distracting effects of mobile telephone use while driving are well known, however the effects of other sources of distraction, such as auditory navigation devices, are less well understood. Whether the effects of auditory distraction might interact with other sensory impairments, such as vision impairment, is of interest given that visual impairment is relatively common within the population, particularly as a result of uncorrected refractive error. In this experiment, 20 current drivers (mean age of 29.4 ± 3.2 years), binocularly viewed video recordings of traffic scenes presented as part of the Hazard Perception Test and responded to potential hazards within the traffic scenes. Half of the presented scenes included auditory navigation instructions as an auditory distractor. Additionally, some of the scenes were viewed through optical lenses to induce different levels of refractive blur (+0.50 DS, +1.00 DS and +2.00 DS). Hazard perception response times increased significantly (p < 0.05) with increasing blur. Participants were significantly slower in reacting to hazards for the +1.00 DS and +2.00 DS blur conditions compared to the control condition (with no blur). There was also a significant increase in response times to hazards in the presence of the auditory navigation instructions. The combined effect of blur and auditory instructions was additive, with the worst performance being in the presence of both blur and auditory instructions. These results suggest that the delivery of auditory navigation guidance for those with visual impairments, such as blur, which are relatively common in the population, should be further investigated.

Tactile Display Design for Flight Envelope Protection and Situational Awareness

Spatial disorientation and visual channel saturation are defined as critical situations encountered by military pilots. Such subjects are interesting research areas likely to create innovative systems able to surmount obstacles of this kind. The integration of new stimulation techniques (sensory substitute, adjunct for visual and audio feedback) may be considered to make the visual channel better. This contribution may help in integrating tactile stimulation to improve or substitute the visual channel. It may also help to better interpret the spatial disorientation awareness signals and the vestibulo-ocular response limitations. The innovation of the proposed approach translates in: (i) the development of the PI-Inverse dynamics controller to provide a time delay reduction of the low cost tactile actuator, and thus, high-performance tactile system; (ii) an approach based on fuzzy logic controller (FLC) is being used in order to translate the turn rate angle, the flight path climb angle and the warning messages into tactile signal features instead of a conventional approach based on direct coding of the pitch and bank angles; and (iii) the consideration of the flight envelope. The fuzzy set translation of flight parameters into tactile signals is also a pragmatic and useful way to design the system.

Preventing Emergency Vehicle Crashes: Status and Challenges of Human Factors Issues

OBJECTIVE

This study reports current status of knowledge and challenges associated with the emergency vehicle (police car, fire truck, and ambulance) crashes, with respect to the major contributing risk factors.

BACKGROUND

Emergency vehicle crashes are a serious nationwide problem, causing injury and death to emergency responders and citizens. Understanding the underlying causes of these crashes is critical for establishing effective strategies for reducing the occurrence of similar incidents.

METHOD

We reviewed the broader literature associated with the contributing factors for emergency vehicle crashes: peer-reviewed journal papers; and reports, policies, and manuals published by government agencies, universities, and research institutes.

RESULTS

Major risk factors for emergency vehicle crashes identified in this study were organized into four categories: driver, task, vehicle, and environmental factors. Also, current countermeasures and interventions to mitigate the hazards of emergency vehicle crashes were discussed, and new ideas for future studies were suggested.

CONCLUSION

Risk factors, control measures, and knowledge gaps relevant to emergency vehicle crashes were presented. Six research concepts are offered for the human factors community to address. Among the topics are emergency vehicle driver risky behavior carryover between emergency response and return from a call, distraction in emergency vehicle driving, in-vehicle driver assistance technologies, vehicle red light running, and pedestrian crash control.

APPLICATION

This information is helpful for emergency vehicle drivers, safety practitioners, public safety agencies, and research communities to mitigate crash risks. It also offers ideas for researchers to advance technologies and strategies to further emergency vehicle safety on the road.

The use of smartphones for wayfinding by people with mild dementia

The aim of this study was to explore the acceptability, feasibility and usability of older people with mild dementia to use smartphone for wayfinding. Thirty cognitively normal older people and 16 people with mild dementia were recruited to participate in a wayfinding trial in the free-living environment. Five feasibility and three acceptability markers were compared between the groups. Content analysis on the video-recorded trial processes and individual interviews was employed to identify the usability issues. The results found that there were no significant between-group differences on the feasibility markers, except that the people with mild dementia needed significantly more time to complete the wayfinding trial and workshop; or on the acceptability items. Sensory/cognitive impairment and GPS signal reliability affected their usability. Mild dementia does not limit the older people to use smartphones for wayfinding in the free-living environment. Future studies should examine the efficacy and safety of smartphone to promote outdoor independence of the people with mild dementia.

Digital, analogue, or redundant speedometers for truck driving: Impact on visual distraction, efficiency and usability

Existing literature does not draw conclusions as to which speedometer type is better for truck driving. A digital speedometer would be more beneficial when obtaining absolute and relative readings, while an analogue speedometer would be more efficient and less distracting when detecting dynamic speed changes. Redundant speedometers, which simultaneously present digital and analogue speedometers, appear increasingly in vehicles, but no information is available on their ergonomic qualities. This study compared three speedometers: digital speedometers, analogue speedometers, and redundant speedometers. This study compared the efficiency, usability and visual distraction measures for all three types of speedometers in a simulated truck driving setting. The task-dependant results were confirmed for the digital and analogue speedometer. The redundant speedometer combined the benefits of each type presented separately, which highlights interesting theoretical and applied implications.

The Effects of Continuous Driving-Related Feedback on Drivers’ Response to Automation Failures

During prolonged periods of autonomous driving, drivers tend to shift their attention away from the driving task. As a result, they require more time to regain awareness of the driving situation and to react to it. This study examined the use of informative automation that during Level-3 autonomous driving provided drivers with continuous feedback regarding the vehicle’s actions and surroundings. It was hypothesized that the operation of informative automation will trigger drivers to allocate more attention to the driving task and will improve their reaction times when resuming control of the vehicle. Sixteen participants drove manual and autonomous driving segments in a driving simulator equipped with Level-3 automation. For half of the participants, the informative automation issued alerts and messages while for the other half no messages were issued (control). The number of on-road glances served as a proxy for drivers’ attention. Drivers’ performance on handling an unexpected automation failure event was measured using their time-to-brake and time-to-steer. Results showed that drivers using the informative automation made more frequent on-road glances than drivers in the control group. Yet, there were no significant differences in reaction times to the automation failure event between the groups. Explanations and implications of these results are discussed.

Performance in Takeover and Characteristics of Non-driving Related Tasks during Highly Automated Driving in Younger and Older Drivers

This paper aims to examine the effect of age and various characteristics of non-driving related activities during highly automated driving on subsequent performance in notified takeovers among younger and older drivers. The paper presents new analyses of data collected in our earlier study (Clark & Feng, 2016). Non-driving-related activities that participants voluntarily chose to engage in during automated driving were categorized according to their cognitive dimensions in information processing. Using hierarchical multiple regressions, we analyzed the effect of driver age, total duration and number of engagement in non-driving-related activities, the duration and cognitive dimensions of the last activity prior to takeover on average speed during takeover and the response time to a takeover notification. We found that older drivers speed was negatively predicted by age while their response time to a notification was not predicted by any factor. In contrast, younger drivers showed a trend of positive relationship between age and average speed and the characteristics of the last task engagement explained a significant portion of the variance of response time to a notification.

Redundancy, Modality, and Priority in Dual Task Interference

This study examined the effects of several properties of a simulated in-vehicle-task (IVT) on interference with a concurrent tracking task, which simulated vehicle control. We compared auditory, visual and redundant delivery of IVT information, under conditions when the visual display was close and separated from the tracking task. In the first experiment we varied whether the tracking or the IVT was emphasized, and in the second experiment we added instruction in the use of redundant displays. IVT messages varied in length. The results from 20 participants in each experiment revealed (a) an effect of priority on the tracking task only, suggesting that separate resources were used for each task, (b) an advantage for auditory over visual delivery only when the visual display was separated, suggesting that visual costs relate to peripheral, not central resources, (c) no benefit and sometimes noticeable costs for the redundant display, compared to the single modality displays in experiment 1, and (d) an improvement in performance with redundant displays when training was given in experiment 2. The results have a positive bearing on the use of head-up (adjacent) displays for complex information.

Designing sensory-substitution devices: Principles, pitfalls and potential1

An exciting possibility for compensating for loss of sensory function is to augment deficient senses by conveying missing information through an intact sense. Here we present an overview of techniques that have been developed for sensory substitution (SS) for the blind, through both touch and audition, with special emphasis on the importance of training for the use of such devices, while highlighting potential pitfalls in their design. One example of a pitfall is how conveying extra information about the environment risks sensory overload. Related to this, the limits of attentional capacity make it important to focus on key information and avoid redundancies. Also, differences in processing characteristics and bandwidth between sensory systems severely constrain the information that can be conveyed. Furthermore, perception is a continuous process and does not involve a snapshot of the environment. Design of sensory substitution devices therefore requires assessment of the nature of spatiotemporal continuity for the different senses. Basic psychophysical and neuroscientific research into representations of the environment and the most effective ways of conveying information should lead to better design of sensory substitution systems. Sensory substitution devices should emphasize usability, and should not interfere with other inter- or intramodal perceptual function. Devices should be task-focused since in many cases it may be impractical to convey too many aspects of the environment. Evidence for multisensory integration in the representation of the environment suggests that researchers should not limit themselves to a single modality in their design. Finally, we recommend active training on devices, especially since it allows for externalization, where proximal sensory stimulation is attributed to a distinct exterior object.

An EEG-Based Fatigue Detection and Mitigation System

Research has indicated that fatigue is a critical factor in cognitive lapses because it negatively affects an individual’s internal state, which is then manifested physiologically. This study explores neurophysiological changes, measured by electroencephalogram (EEG), due to fatigue. This study further demonstrates the feasibility of an online closed-loop EEG-based fatigue detection and mitigation system that detects physiological change and can thereby prevent fatigue-related cognitive lapses. More importantly, this work compares the efficacy of fatigue detection and mitigation between the EEG-based and a nonEEG-based random method. Twelve healthy subjects participated in a sustained-attention driving experiment. Each participant’s EEG signal was monitored continuously and a warning was delivered in real-time to participants once the EEG signature of fatigue was detected. Study results indicate suppression of the alpha- and theta-power of an occipital component and improved behavioral performance following a warning signal; these findings are in line with those in previous studies. However, study results also showed reduced warning efficacy (i.e. increased response times (RTs) to lane deviations) accompanied by increased alpha-power due to the fluctuation of warnings over time. Furthermore, a comparison of EEG-based and nonEEG-based random approaches clearly demonstrated the necessity of adaptive fatigue-mitigation systems, based on a subject’s cognitive level, to deliver warnings. Analytical results clearly demonstrate and validate the efficacy of this online closed-loop EEG-based fatigue detection and mitigation mechanism to identify cognitive lapses that may lead to catastrophic incidents in countless operational environments.

Aging and Sensory Substitution in a Virtual Navigation Task

Virtual environments are becoming ubiquitous, and used in a variety of contexts–from entertainment to training and rehabilitation. Recently, technology for making them more accessible to blind or visually impaired users has been developed, by using sound to represent visual information. The ability of older individuals to interpret these cues has not yet been studied. In this experiment, we studied the effects of age and sensory modality (visual or auditory) on navigation through a virtual maze. We added a layer of complexity by conducting the experiment in a rotating room, in order to test the effect of the spatial bias induced by the rotation on performance. Results from 29 participants showed that with the auditory cues, it took participants a longer time to complete the mazes, they took a longer path length through the maze, they paused more, and had more collisions with the walls, compared to navigation with the visual cues. The older group took a longer time to complete the mazes, they paused more, and had more collisions with the walls, compared to the younger group. There was no effect of room rotation on the performance, nor were there any significant interactions among age, feedback modality and room rotation. We conclude that there is a decline in performance with age, and that while navigation with auditory cues is possible even at an old age, it presents more challenges than visual navigation.

A user study of auditory, head-up and multi-modal displays in vehicles

This paper describes a user study on the interaction with an in-vehicle information system (IVIS). The motivation for conducting this research was to investigate the subjectively and objectively measured impact of using a single- or multi-modal IVIS while driving. A hierarchical, list-based menu was presented using a windshield projection (head-up display), auditory display and a combination of both interfaces. The users were asked to navigate a vehicle in a driving simulator and simultaneously perform a set of tasks of varying complexity. The experiment showed that the interaction with visual and audio-visual head-up displays is faster and more efficient than with the audio-only display. All the interfaces had a similar impact on the overall driving performance. There was no significant difference between the visual only and audio-visual displays in terms of their efficiency and safety; however, the majority of test subjects clearly preferred to use the multi-modal interface while driving.

Developing an EEG-based on-line closed-loop lapse detection and mitigation system

In America, 60% of adults reported that they have driven a motor vehicle while feeling drowsy, and at least 15–20% of fatal car accidents are fatigue-related. This study translates previous laboratory-oriented neurophysiological research to design, develop, and test an On-line Closed-loop Lapse Detection and Mitigation (OCLDM) System featuring a mobile wireless dry-sensor EEG headgear and a cell-phone based real-time EEG processing platform. Eleven subjects participated in an event-related lane-keeping task, in which they were instructed to manipulate a randomly deviated, fixed-speed cruising car on a 4-lane highway. This was simulated in a 1st person view with an 8-screen and 8-projector immersive virtual-reality environment. When the subjects experienced lapses or failed to respond to events during the experiment, auditory warning was delivered to rectify the performance decrements. However, the arousing auditory signals were not always effective. The EEG spectra exhibited statistically significant differences between effective and ineffective arousing signals, suggesting that EEG spectra could be used as a countermeasure of the efficacy of arousing signals. In this on-line pilot study, the proposed OCLDM System was able to continuously detect EEG signatures of fatigue, deliver arousing warning to subjects suffering momentary cognitive lapses, and assess the efficacy of the warning in near real-time to rectify cognitive lapses. The on-line testing results of the OCLDM System validated the efficacy of the arousing signals in improving subjects' response times to the subsequent lane-departure events. This study may lead to a practical on-line lapse detection and mitigation system in real-world environments.

Shared and distinct factors driving attention and temporal processing across modalities

In addition to the classic finding that "sounds are judged longer than lights," the timing of auditory stimuli is often more precise and accurate than is the timing of visual stimuli. In cognitive models of temporal processing, these modality differences are explained by positing that auditory stimuli more automatically capture and hold attention, more efficiently closing an attentional switch that allows the accumulation of pulses marking the passage of time (Penney, Gibbon, & Meck, 2000). However, attention is a multifaceted construct, and there has been little attempt to determine which aspects of attention may be related to modality effects. We used visual and auditory versions of the Continuous Temporal Expectancy Task (CTET; O'Connell et al., 2009) a timing task previously linked to behavioral and electrophysiological measures of mind-wandering and attention lapses, and tested participants with or without the presence of a video distractor. Performance in the auditory condition was generally superior to that in the visual condition, replicating standard results in the timing literature. The auditory modality was also less affected by declines in sustained attention indexed by declines in performance over time. In contrast, distraction had an equivalent impact on performance in the two modalities. Analysis of individual differences in performance revealed further differences between the two modalities: Poor performance in the auditory condition was primarily related to boredom whereas poor performance in the visual condition was primarily related to distractibility. These results suggest that: 1) challenges to different aspects of attention reveal both modality-specific and nonspecific effects on temporal processing, and 2) different factors drive individual differences when testing across modalities.

Transfer of complex skill learning from virtual to real rowing

Simulators are commonly used to train complex tasks. In particular, simulators are applied to train dangerous tasks, to save costs, and to investigate the impact of different factors on task performance. However, in most cases, the transfer of simulator training to the real task has not been investigated. Without a proof for successful skill transfer, simulators might not be helpful at all or even counter-productive for learning the real task. In this paper, the skill transfer of complex technical aspects trained on a scull rowing simulator to sculling on water was investigated. We assume if a simulator provides high fidelity rendering of the interactions with the environment even without augmented feedback, training on such a realistic simulator would allow similar skill gains as training in the real environment. These learned skills were expected to transfer to the real environment. Two groups of four recreational rowers participated. One group trained on water, the other group trained on a simulator. Within two weeks, both groups performed four training sessions with the same licensed rowing trainer. The development in performance was assessed by quantitative biomechanical performance measures and by a qualitative video evaluation of an independent, blinded trainer. In general, both groups could improve their performance on water. The used biomechanical measures seem to allow only a limited insight into the rowers' development, while the independent trainer could also rate the rowers' overall impression. The simulator quality and naturalism was confirmed by the participants in a questionnaire. In conclusion, realistic simulator training fostered skill gains to a similar extent as training in the real environment and enabled skill transfer to the real environment. In combination with augmented feedback, simulator training can be further exploited to foster motor learning even to a higher extent, which is subject to future work.

Self-reported wayfinding ability of older drivers

Some older drivers experience difficulties driving whilst wayfinding in unfamiliar areas. Difficulties in wayfinding have been associated with poorer driving performance and reduced driving mobility. The objective of the current study was to identify cognitive and demographic predictors in older drivers of perceived wayfinding difficulty, avoidance of unfamiliar areas and the use of wayfinding strategies. Five hundred and thirty-four drivers aged 65 years and over (excluding those with dementia or Parkinson's disease) completed a mail-out survey. Drivers commonly reported difficulties with wayfinding, with 59.5% reporting their abilities as poor or fair rather than good. Those significantly more likely to report difficulty were older, reported poorer health and cognition, and had less driving experience. A small proportion of drivers reported regularly avoiding unfamiliar areas (13.8%); these drivers were significantly more likely to be female and to report poorer wayfinding abilities. The most common wayfinding strategies regularly used by older drivers were using a street directory whilst driving (61.9%) and pulling over to check the map (55.1%). Regular passenger guidance (23.9%) or use of a navigation system (9.9%) was less common. The implications of this study are wide and include collecting further information about: (1) the role of cognitive processes in wayfinding ability; (2) the relationship between perceived wayfinding difficulty and restriction of driving in unfamiliar areas; and (3) older drivers' preferences for different wayfinding strategies.

Can arousing feedback rectify lapses in driving? Prediction from EEG power spectra

OBJECTIVE

This study explores the neurophysiological changes, measured using an electroencephalogram (EEG), in response to an arousing warning signal delivered to drowsy drivers, and predicts the efficacy of the feedback based on changes in the EEG.

APPROACH

Eleven healthy subjects participated in sustained-attention driving experiments. The driving task required participants to maintain their cruising position and compensate for randomly induced lane deviations using the steering wheel, while their EEG and driving performance were continuously monitored. The arousing warning signal was delivered to participants who experienced momentary behavioral lapses, failing to respond rapidly to lane-departure events (specifically the reaction time exceeded three times the alert reaction time).

MAIN RESULTS

The results of our previous studies revealed that arousing feedback immediately reversed deteriorating driving performance, which was accompanied by concurrent EEG theta- and alpha-power suppression in the bilateral occipital areas. This study further proposes a feedback efficacy assessment system to accurately estimate the efficacy of arousing warning signals delivered to drowsy participants by monitoring the changes in their EEG power spectra immediately thereafter. The classification accuracy was up 77.8% for determining the need for triggering additional warning signals.

SIGNIFICANCE

The findings of this study, in conjunction with previous studies on EEG correlates of behavioral lapses, might lead to a practical closed-loop system to predict, monitor and rectify behavioral lapses of human operators in attention-critical settings.

Supporting interruption management and multimodal interface design: three meta-analyses of task performance as a function of interrupting task modality

OBJECTIVE

The aim of this study was to integrate empirical data showing the effects of interrupting task modality on the performance of an ongoing visual-manual task and the interrupting task itself. The goal is to support interruption management and the design of multimodal interfaces.

BACKGROUND

Multimodal interfaces have been proposed as a promising means to support interruption management.To ensure the effectiveness of this approach, their design needs to be based on an analysis of empirical data concerning the effectiveness of individual and redundant channels of information presentation.

METHOD

Three meta-analyses were conducted to contrast performance on an ongoing visual task and interrupting tasks as a function of interrupting task modality (auditory vs. tactile, auditory vs. visual, and single modality vs. redundant auditory-visual). In total, 68 studies were included and six moderator variables were considered.

RESULTS

The main findings from the meta-analyses are that response times are faster for tactile interrupting tasks in case of low-urgency messages.Accuracy is higher with tactile interrupting tasks for low-complexity signals but higher with auditory interrupting tasks for high-complexity signals. Redundant auditory-visual combinations are preferable for communication tasks during high workload and with a small visual angle of separation.

CONCLUSION

The three meta-analyses contribute to the knowledge base in multimodal information processing and design. They highlight the importance of moderator variables in predicting the effects of interruption task modality on ongoing and interrupting task performance.

APPLICATIONS

The findings from this research will help inform the design of multimodal interfaces in data-rich, event-driven domains.

Preventing lapse in performance using a drowsiness monitoring and management system

Research on public security, especially the safe manipulation and control of vehicles, has gained increasing attention in recent years. This study proposes a closed-loop drowsiness monitoring and management system that can estimate subjects' driving performance. The system observes electroencephalographic (EEG) dynamics and behavioral changes, delivers arousing feedback to individuals experiencing momentary cognitive lapses, and assesses the efficacy of the feedback. Results of this study showed that the arousing feedback immediately improved subject performance, which was accompanied by concurrent theta- and alpha-power suppression in the bilateral occipital areas. This study further demonstrated the feasibility of accurately assessing the efficacy of arousing feedback presented to drowsy participants by monitoring the changes in their EEG power spectra.

Augmented visual, auditory, haptic, and multimodal feedback in motor learning: a review

It is generally accepted that augmented feedback, provided by a human expert or a technical display, effectively enhances motor learning. However, discussion of the way to most effectively provide augmented feedback has been controversial. Related studies have focused primarily on simple or artificial tasks enhanced by visual feedback. Recently, technical advances have made it possible also to investigate more complex, realistic motor tasks and to implement not only visual, but also auditory, haptic, or multimodal augmented feedback. The aim of this review is to address the potential of augmented unimodal and multimodal feedback in the framework of motor learning theories. The review addresses the reasons for the different impacts of feedback strategies within or between the visual, auditory, and haptic modalities and the challenges that need to be overcome to provide appropriate feedback in these modalities, either in isolation or in combination. Accordingly, the design criteria for successful visual, auditory, haptic, and multimodal feedback are elaborated.

Driving with navigational instructions: Investigating user behaviour and performance

This paper reports the results of an inter-disciplinary study investigating user preferences and performance in relation to spoken in-car route guidance. In-car navigation systems are becoming increasingly popular. However, despite large amounts of research assessing the presentation of spatial information, and the usability and interaction issues surrounding the interfaces, there has been much less investigation of the impacts of auditory presentation of route information. We addressed this issue using a multi-disciplinary approach to collect both qualitative and quantitative data through questionnaires and user experiments. Our research identified a user preference for auditory presentation of route information, as well as a memory advantage for auditory over visual presentation. We also found that simple auditory route instructions could be followed without significant interference to a simulated driving task, whereas more complex auditory instructions did cause interference. Taken together, this research highlights the importance of the design of spoken route guidance instructions in minimising the cognitive demands that they impose.

Effects of age and the use of hands-free cellular phones on driving behavior and task performance

OBJECTIVE

This study used a driving simulator to investigate the effect of using a Bluetooth hands-free cellular phone earpiece on the driving behavior of two age groups.

METHODS

Forty-eight participants (24 aged 20-26 and 24 aged 65-73) were examined to assess their performance on the following divided-attention tasks under 2 driving load conditions (high and low): (1) attempting to maintain the speed limit and (2) using a cellular phone while driving. The length of the call conversation (long vs. short) and the conversational content (complex vs. simple) were manipulated as within-subject independent variables. The driving behavior of the participants, their task reaction times and accuracy, and subjective ratings were collected as dependent variables.

RESULTS

The results indicate that under low driving loads, short talk times, and simple conversational content, the driving behavior of the participants showed low variance in the vehicle's mean speed. In contrast, complex conversation had a significantly negative impact on driving behavior. Notably, under a low driving load, motorists' driving behaviors, measured in lateral acceleration, caused significantly smaller variance in complex conversations compared to no call and simple conversations. The use of a hands-free cellular phone affected the performance (acceleration, lane deviation, reaction time, and accuracy) of older drivers significantly more than younger drivers. While performing divided attention tasks, the accuracy of the older drivers was 66.3 percent and that of the younger drivers was 96.3 percent. Although this study did not find a clear impact of cellular phone use on the driving behavior of younger drivers, their divided-attention task reaction times and accuracy were better under no-call than calling conditions.

CONCLUSIONS

This study indicates that the use of hands-free cellular phones could significantly affect the safety of driving among the older and present risks, although lesser, for younger drivers.

Auditory Displays for In-Vehicle Technologies

Modern vehicle cockpits have begun to incorporate a number of information-rich techno-logies, including systems to enhance and improve driving and navigation performance and also driving-irrelevant information systems. The visually intensive nature of the driving task requires these systems to adopt primarily nonvisual means of information display, and the auditory modality represents an obvious alternative to vision for interacting with in-vehicle technologies (IVTs). Although the literature on auditory displays has grown tremendously in recent decades, to date, few guidelines or recommendations exist to aid in the design of effective auditory displays for IVTs. This chapter provides an overview of the current state of research and practice with auditory displays for IVTs. The role of basic auditory capabilities and limitations as they relate to in-vehicle auditory display design are discussed. Extant systems and prototypes are reviewed, and when possible, design recommendations are made. Finally, research needs and an iterative design process to meet those needs are discussed.

Suggestions on the applicability of visual instructions with see-through head mounted displays depending on the task

Task instructions have traditionally been communicated orally in many fields. However, recently more and more wearable displays, such as the see-through head mounted displays (HMDs) have been developed, and some studies have provided ideas on applying visual instruction using these new interfaces to particular situations. However, in some cases, where instructions are communicated amongst the workers, the data is not sufficient for field workers to choose the best way of communicating instructions depending on the situation. Thus, this study aims to clarify the cases in which it is effective to apply visual instructions with HMDs, and to provide information that suggests the applicability of such visual instructions instead of or in addition to the traditional auditory instructions in different situations. These suggestions will be a useful reference for workers in safety-critical fields, helping them make better decisions about whether, when, and where to introduce the new method of instructions. It will also address some of the unsolved problems in the field, such as errors, low efficiency, and discomfort in communication.

Tonic and phasic EEG and behavioral changes induced by arousing feedback

This study investigates brain dynamics and behavioral changes in response to arousing auditory signals presented to individuals experiencing momentary cognitive lapses during a sustained-attention task. Electroencephalographic (EEG) and behavioral data were simultaneously collected during virtual-reality (VR) based driving experiments, in which subjects were instructed to maintain their cruising position and compensate for randomly induced lane deviations using the steering wheel. 30-channel EEG data were analyzed by independent component analysis and the short-time Fourier transform. Across subjects and sessions, intermittent performance during drowsiness was accompanied by characteristic spectral augmentation or suppression in the alpha- and theta-band spectra of a bilateral occipital component, corresponding to brief periods of normal (wakeful) and hypnagogic (sleeping) awareness and behavior. Arousing auditory feedback was delivered to the subjects in half of the non-responded lane-deviation events, which immediately agitated subject's responses to the events. The improved behavioral performance was accompanied by concurrent spectral suppression in the theta- and alpha-bands of the bilateral occipital component. The effects of auditory feedback on spectral changes lasted 30s or longer. The results of this study demonstrate the amount of cognitive state information that can be extracted from noninvasively recorded EEG data and the feasibility of online assessment and rectification of brain networks exhibiting characteristic dynamic patterns in response to momentary cognitive challenges.

Combining cognitive and visual distraction: less than the sum of its parts

Driver distraction has become a leading cause of motor-vehicle crashes. Although visual and cognitive distraction has been studied extensively, relatively little research has addressed their combined effects on drivers' behavior. To fill this gap, a medium-fidelity simulator study examined the driver behavior before, during and after three types of distraction. Driving without distraction was compared to visual distraction, cognitive distraction, and combined visual and cognitive distraction. The results show that the visual and combined distraction both impaired vehicle control and hazard detection and resulted in frequent, long off-road glances. The combined distraction was less detrimental than visual distraction alone. Cognitive distraction made steering less smooth, but improved lane maintenance. All distractions caused gaze concentration and slow saccades when drivers looked at the roadway, and cognitive and combined distraction increased blink frequency. Steering neglect, under-compensation, and over-compensation were three typical steering failures that were differentially associated with the different distractions: steering neglect and over-compensation with visual distraction and under-compensation with cognitive distraction. Overall, visual distraction interferes with driving performance more than cognitive distraction, and visual distraction dominates the performance decrements during combined distraction. These results suggest that minimizing visual demand is particularly important in the design of in-vehicle systems and in the development of distraction countermeasures.

Arousing feedback rectifies lapse in performance and corresponding EEG power spectrum

This study explores electroencephalographic (EEG) dynamics and behavioral changes in response to arousing auditory signals presented to individuals experiencing momentary cognitive lapses. Arousing auditory feedback was delivered to the subjects in half of the non-responded lane-deviation events during a sustained-attention driving task, which immediately agitated subject's responses to the events. The improved behavioral performance was accompanied by concurrent power suppression in the theta- and alpha-bands in the lateral occipital cortices. This study further explores the feasibility of estimating the efficacy of arousing feedback presented to the drowsy subjects by monitoring the changes in EEG power spectra.