How much time do drivers need to obtain situation awareness? A laboratory-based study of automated driving

Effects of time interval and request modality on driver takeover responses: Identifying the optimal time interval for two-stage warning system

Two-stage warning system plays a critical role in guiding drivers to prepare for takeovers in conditional automated driving. However, the optimal time interval for this system, especially under different takeover request (TOR) modalities, remains unclear. A driving simulator experiment with 36 participants was conducted to investigate the effects of time interval and TOR modality of two-stage warning system on drivers' takeover responses from a multidimensional perspective. Each participant completed takeovers with four time intervals (3 s, 5 s, 7 s, and 9 s) and three TOR modalities (visual-only, auditory-only, and auditory-visual). Drivers' takeover performance, mental workload, situation awareness (SA), user experience, and eye movements during the takeover process were recorded. The results indicated that drivers showed faster and higher-quality takeovers as the time interval increased from 3 s to 9 s. Their ratings of satisfaction, usefulness, effectiveness, and safeness of the warning system showed the inverted U-shaped trends, with the 7 s as a turning point. The 7 s interval was also favored for drivers to regain sufficient SA while maintaining an appropriate mental workload, as evidenced by both subjective measures and eye-tracking metrics. This allowed drivers to adopt more focused visual strategies for the takeover after receiving TOR warning, thereby improving takeover performance. Additionally, the auditory-visual TOR was found to be the most effective across all measures, followed by the auditory-only TOR, and finally the visual-only TOR. No significant interaction effects between time interval and TOR modality were observed. In conclusion, regardless of TOR modality, the 7 s time interval was generally favored for young drivers with relatively limited driving experience for swift takeover responses, high takeover quality, sufficient SA, appropriate mental workload, and good satisfaction ratings. When the interval was extended to 9 s, drivers' takeover performance improved, but with the cost of reduced satisfaction and potential shift in visual attention from driving task to non-driving-related task. These findings had implications for the design and application of appropriate time interval of two-stage warning system for Level 3 automatic vehicles.

Triboelectric sensor gloves for real-time behavior identification and takeover time adjustment in conditionally automated vehicles

The takeover issue, especially the setting of the takeover time budget, is a critical factor restricting the implementation and development of conditionally automated vehicles. The general fixed takeover time budget has certain limitations, as it does not take into account the driver's non-driving behaviors. Here, we propose an intelligent takeover assistance system consisting of all-round sensing gloves, a non-driving behavior identification module, and a takeover time budget determination module. All-round sensing gloves based on triboelectric sensors seamlessly detect delicate motions of hands and interactions between hands and other objects, and then transfer the electrical signals to the non-driving behavior identification module, which achieves an accuracy of 94.72% for six non-driving behaviors. Finally, combining the identification result and its corresponding minimum takeover time budget obtained through the takeover time budget determination module, our system dynamically adjusts the takeover time budget based on the driver's current non-driving behavior, significantly improving takeover performance in terms of safety and stability. Our work presents a potential value in the application and implementation of conditionally automated vehicles.

Exploring the Timing of Disengagement From Nondriving Related Tasks in Scheduled Takeovers With Pre-Alerts: An Analysis of Takeover-Related Measures

OBJECTIVES

This study aimed to investigate drivers' disengagement from nondriving related tasks (NDRT) during scheduled takeovers and to evaluate its impact on takeover performance.

BACKGROUND

During scheduled takeovers, drivers typically have sufficient time to prepare. However, inadequate disengagement from NDRTs can introduce safety risks.

METHOD

Participants experienced scheduled takeovers using a driving simulator, undergoing two conditions, with and without an NDRT. We assessed their takeover performance and monitored their NDRT disengagement from visual, cognitive, and physical perspectives.

RESULTS

The study examined three NDRT disengagement timings (DTs): DT1 (disengaged before the takeover request), DT2 (disengaged after the request but before taking over), and DT3 (not disengaged). The impact of NDRT on takeover performance varied depending on DTs. Specifically, DT1 demonstrated no adverse effects; DT2 impaired takeover time, while DT3 impaired both takeover time and quality. Additionally, participants who displayed DT1 exhibited longer eye-off-NDRT duration and a higher eye-off-NDRT count during the prewarning stage compared to those with DT2 and DT3.

CONCLUSION

Drivers can benefit from earlier disengagement from NDRTs, demonstrating resilience to the adverse effects of NDRTs on takeover performance. The disengagement of cognition is often delayed compared to that of eyes and hands, potentially leading to DT3. Moreover, visual disengagement from NDRTs during the prewarning stage could distinguish DT1 from the other two.

APPLICATION

Our study emphasizes considering NDRT disengagement in designing systems for scheduled takeovers. Measures should be taken to promote early disengagement, facilitate cognitive disengagement, and employ visual disengagement during the prewarning period as predictive indicators of DTs.

Exploring driving anger-caused impairment of takeover performance among professional taxi drivers during partially automated driving

Partially automated systems are expected to reduce road crashes related to human error, even amongst professional drivers. Consequently, the applications of these systems into the taxi industry would potentially improve transportation safety. However, taxi drivers are prone to experiencing driving anger, which may subsequently affect their takeover performance. In this research, we explored how driving anger emotion affects taxi drivers' driving performance in various takeover scenarios, namely Mandatory Automation-Initiated transition (MAIT), Mandatory Driver-Initiated transition (MDIT), and Optional Driver-Initiated transition (ODIT). Forty-seven taxi drivers participated in this 2·3 mixed design simulator experiment (between-subjects: anger vs. calmness; within-subjects: MAIT vs. MDIT vs. ODIT). Compared to calmness, driving anger emotion led to a narrower field of attention (e.g., smaller standard deviations of horizontal fixation points position) and worse hazard perception (e.g., longer saccade latency, smaller amplitude of skin conductance responses), which resulted in longer takeover time and inferior vehicle control stability (e.g., higher standard deviations of lateral position) in MAIT and MDIT scenarios. Angry taxi drivers were more likely to deactivate vehicle automation and take over the vehicle in a more aggressive manner (e.g., higher maximal resulting acceleration, refusing to yield to other road users) in ODIT scenarios. The findings will contribute to addressing the safety concerns related to driving anger among professional taxi drivers and promote the widespread acceptance and integration of partially automated systems within the taxi industry.

Willingness to take responsibility: Self-sacrifice versus sacrificing others in takeover decisions during autonomous driving

In Level-3 autonomous driving, drivers are required to take over in an emergency upon receiving a request from an autonomous vehicle (AV). However, before the deadline for the takeover request expires, drivers are not considered fully responsible for the accident, which may make them hesitant to assume control and take on full liability before the time runs out. Therefore, to prevent problems caused by late takeover, it is important to know which factors influence a driver's willingness to take over in an emergency. To address this issue, we recruited 250 participants each for both video-based and text-based surveys to investigate the takeover decision in a dilemmatic situation that can endanger the driver, with the AV either sacrificing a group of pedestrians or the driver if the participants do not intervene. The results showed that 88.2% of respondents chose to take over when the AV intended to sacrifice the driver, while only 59.4% wanted to take over when the pedestrians would be sacrificed. Additionally, when the AV's chosen path matched the participant's intention, 77.4% chose to take over when the car intended to sacrifice the driver compared with only 34.3% when the pedestrians would be sacrificed. Furthermore, other factors such as sex, driving experience, and driving preferences partially influenced takeover decisions; however, they had a smaller effect than the situational context. Overall, our findings show that regardless of the driving intention of an AV, informing drivers that their safety is at risk can enhance their willingness to take over control of an AV in critical situations.

Effects of driving style on takeover performance during automated driving: Under the influence of warning system factors

Driving style has been proposed to be a critical factor in automated driving. However, the role of driving style in the process of taking over during automated driving needs further investigation. The main purpose of this study was to investigate the influence of driving style on takeover performance under the influence of warning system factors. In addition, this study also explored whether the impact of driving style on reaction time varies over time and the role of driving style on a comprehensive takeover quality indicator. Two driving simulation experiments with different takeover request (TOR) designs were conducted. In experiment 1, content warning information was provided in the TOR with different warning stage designs; in experiment 2, countdown warning information was provided in the TOR with different warning stage designs. Sixty-four participants (32 for experiment 1 and 32 for experiment 2) were classified into two groups based on their driving style (i.e., aggressive, or defensive) using the Chinese version of the Multidimensional Driving Style Inventory (the brief MDSI-C). The results suggested that drivers' driving style had significant effects on takeover performance, but the effects were influenced by warning system designs. Specifically, defensive participants performed better takeover performance, i.e., shorter reaction time and cautious vehicle control behaviors, than aggressive participants in most warning conditions. The content and countdown warning information and warning stage design affected the roles of driving style on takeover performance: 1) compared to the one-stage warning design, the two-stage warning design significantly shortened the reaction time of the participants with different driving styles, 2) compared to the countdown warning information design, the design of content warning information can shorten the reaction time of aggressive participants and lengthen the reaction time of defensive participants in the two-stage warning conditions, and 3) compared to the content warning information design, countdown warning information can improve the safe takeover performance of defensive participants. This study provides a better understanding of the role of driving style on takeover performance, and driving style should be considered when designing warning systems for autonomous vehicles.

A Computational Cognitive Model of Driver Response Time for Scheduled Freeway Exiting Takeovers in Conditionally Automated Vehicles

OBJECTIVE

This study develops a computational model to predict drivers' response time and understand the underlying cognitive mechanism for freeway exiting takeovers in conditionally automated vehicles (AVs).

BACKGROUND

Previous research has modeled drivers' takeover response time in emergency scenarios that demand a quick response. However, existing models may not be applicable for scheduled, non-time-critical takeovers as drivers take longer to resume control when there is no time pressure. A model of driver response time in non-time-critical takeovers is lacking.

METHOD

A computational cognitive model of driver takeover response time is developed based on Queuing Network-Model Human Processor (QN-MHP) architecture. The model quantifies gaze redirection in response to takeover request (ToR), task prioritization, driver situation awareness, and driver trust to address the complexities of drivers' takeover strategies when sufficient time budget exists.

RESULTS

Experimental data of a preliminary driving simulator study were used to validate the model. The model accounted for 97% of the experimental takeover response time for freeway exiting.

CONCLUSION

The current model can successfully predict drivers' response time for scheduled, non-time-critical freeway exiting takeovers in conditionally AVs.

APPLICATION

This model can be applied to the human-machine interface design with respect to ToR lead time for enhancing safe freeway exiting takeovers in conditionally AVs. It also provides a foundation for future modeling work towards an integrated driver model of freeway exiting takeover performance.

The monitoring requests on young driver's fatigue and take-over performance in prolonged conditional automated driving

INTRODUCTION

L3 automated vehicles can perform all dynamic driving tasks unless a take-over occurs due to operational limits. This issue is potentially important for young drivers who are vulnerable road users since they have skill deficits and easily evolve into aberrant driving. However, drivers lacking active involvement may be fatigued and drowsy. Previous research indicated that performing a voluntary non-driving-related task (NDRT) could keep drivers alert, but there was no difference in take-over performance with or without NDRT. Providing a monitoring request (MR) before a possible take-over request (TOR) exhibited better take-over performance in temporary automated driving. Therefore, the study aimed to investigate the effects of MR and voluntary NDRT on young drivers' fatigue and performance.

METHOD

Twenty-five young drivers experienced 60 min automated driving on a highway with low traffic density and a TOR prompted due to a collision event. A within-subjects was designed that comprised three conditions: NONE, TOR-only, and MR + TOR. Drivers were allowed to perform a self-paced phone NDRT during automated driving.

RESULTS

The PERCLOS and blink frequency data showed that playing phones could keep drivers vigilant. The take-over performance on whether taking phone had no difference, but with MRs condition exhibited better take-over performance including the shorter reaction time and the longer TTC. Subjective evaluations also showed the advantages of MRs with more safety, trust, acceptance, and lower workload.

CONCLUSIONS

Taking MRs had a positive effect on relieving fatigue and improving take-over performance. Furthermore, MRs could potentially improve the safety and acceptance of automated driving.

PRACTICAL APPLICATIONS

The MR design can be used in the automotive industry to ensure the safest interfaces between fatigue drivers and automation systems.

Effect of a looming visual cue on situation awareness and perceived urgency in response to a takeover request

This study aimed to investigate the effect of a looming visual cue on situation awareness and perceived urgency in response to a takeover request (TOR), and to explore the underlying mechanisms of this effect through three experiments. In Experiment 1, the optimal size and speed of a red disk were determined, which were effective in capturing looming motion and conveying the urgency of the situation. The results indicated that both looming speed and size ratio had significant effects on situation awareness and perceived urgency. In Experiment 2, the effects of looming stimuli were compared with dimming stimuli, and the results showed that the looming visual cue was more effective in promoting perceived urgency and situation awareness. The results also indicated that the looming visual cue attracted more visual attention than the dimming visual cue, in line with previous studies. Experiment 3 utilized a driving simulator to test the effectiveness of the looming visual cue in promoting fast and appropriate responses to TORs in complex driving scenarios. The results showed that the looming visual cue was more effective in promoting perceived urgency and enhancing situation awareness, especially in highly complex driving situations. Overall, the findings suggest that the looming visual cue is a powerful tool for promoting fast and appropriate responses to TORs and enhancing situation awareness, particularly in complex driving scenarios. These results have important implications for designing effective TOR systems and improving driver safety on the road.

Is driving experience all that matters? Drivers' takeover performance in conditionally automated driving

INTRODUCTION

In conditionally automated driving, drivers are allowed to engage in non-driving related tasks (NDRTs) and are occasionally requested to take over vehicle control in situations that the automation system cannot handle. Drivers may not be able to adequately perform such requests if they have limited driving experience. This study investigates the influence of driving experience on takeover performance in conditionally automated driving.

METHOD

Nineteen subjects participated in this driving simulator study. The NDRTs consisted of three tasks: writing business emails (working condition), watching videos (entertaining condition), and taking a break with eyes closed (resting condition). These three NDRTs require drivers to invest high, moderate, and low levels of mental workload, respectively. The duration of engagement in each NDRT before a takeover request (TOR) was either 5 minutes (short interval) or 30 minutes (long interval).

RESULTS

Drivers' driving experience and performance during the control period are highly correlated with their TOR performance. Furthermore, the type and duration of NDRT influence TOR performance, and inexperienced drivers exhibit poorer TOR performance than experienced drivers.

CONCLUSIONS AND PRACTICAL APPLICATIONS

These findings have relevance for the types of NDRTs that ought to be permitted during automated driving, the design of automated driving systems, and the formulation of regulations regarding the responsible use of automated vehicles.

Evaluation of Imminent Take-Over Requests With Real Automation on a Test Track

OBJECTIVE

Investigating take-over, driving, non-driving related task (NDRT) performance, and trust of conditionally automated vehicles (AVs) in critical transitions on a test track.

BACKGROUND

Most experimental results addressing driver take-over were obtained in simulators. The presented experiment aimed at validating relevant findings while uncovering potential effects of motion cues and real risk.

METHOD

Twenty-two participants responded to four critical transitions on a test track. Non-driving related task modality (reading on a handheld device vs. auditory) and take-over timing (cognitive load) were varied on two levels. We evaluated take-over and NDRT performance as well as gaze behavior. Further, trust and workload were assessed with scales and interviews.

RESULTS

Reaction times were significantly faster than in simulator studies. Further, reaction times were only barely affected by varying visual, physical, or cognitive load. Post-take-over control was significantly degraded with the handheld device. Experiencing the system reduced participants' distrust, and distrusting participants monitored the system longer and more frequently. NDRTs on a handheld device resulted in more safety-critical situations.

CONCLUSION

The results confirm that take-over performance is mainly influenced by visual-cognitive load, while physical load did not significantly affect responses. Future take-over request (TOR) studies may investigate situation awareness and post-take-over control rather than reaction times only. Trust and distrust can be considered as different dimensions in AV research.

APPLICATION

Conditionally AVs should offer dedicated interfaces for NDRTs to provide an alternative to using nomadic devices. These interfaces should be designed in a way to maintain drivers' situation awareness.

PRÉCIS

This paper presents a test track experiment addressing conditionally automated driving systems. Twenty-two participants responded to critical TORs, where we varied NDRT modality and take-over timing. In addition, we assessed trust and workload with standardized scales and interviews.

Design Factors of Shared Situation Awareness Interface in Human–Machine Co-Driving

Automated vehicles can perceive their environment and control themselves, but how to effectively transfer the information perceived by the vehicles to human drivers through interfaces, or share the awareness of the situation, is a problem to be solved in human–machine co-driving. The four elements of the shared situation awareness (SSA) interface, namely human–machine state, context, current task status, and plan, were analyzed and proposed through an abstraction hierarchy design method to guide the output of the corresponding interface design elements. The four elements were introduced to visualize the interface elements and design the interface prototype in the scenario of “a vehicle overtaking with a dangerous intention from the left rear”, and the design schemes were experimentally evaluated. The results showed that the design with the four elements of an SSA interface could effectively improve the usability of the human–machine interface, increase the levels of human drivers’ situational awareness and prediction of dangerous intentions, and boost trust in the automatic systems, thereby providing ideas for the design of human–machine collaborative interfaces that enhance shared situational awareness in similar scenarios.

Assessing Distraction Potential of Augmented Reality Head-Up Displays for Vehicle Drivers

OBJECTIVE

To develop a framework for quantifying the visual and cognitive distraction potential of augmented reality (AR) head-up displays (HUDs).

BACKGROUND

AR HUDs promise to be less distractive than traditional in-vehicle displays because they project information onto the driver's forward-looking view of the road. However, AR graphics may direct the driver's attention away from critical road elements. Moreover, current in-vehicle device assessment methods, which are based on eyes-off-road time measures, cannot capture this unique challenge.

METHOD

This article proposes a new method for the assessment of AR HUDs by measuring driver gaze behavior, situation awareness, confidence, and workload. An experimental user study (n = 24) was conducted in a driving simulator to apply the proposed method for the assessment of two AR pedestrian collision warning (PCW) design alternatives.

RESULTS

Only one of the two tested AR interfaces improved driver awareness of pedestrians without visually and cognitively distracting drivers from other road elements that were not augmented by the display but still critical for safe driving.

CONCLUSION

Our initial human-subject experiment demonstrated the potential of the proposed method in quantifying both positive and negative consequences of AR HUDs on driver cognitive processes. More importantly, the study suggests that AR interfaces can be informative or distractive depending on the perceptual forms of graphical elements presented on the displays.

APPLICATION

The proposed methods can be applied by designers of in-vehicle AR HUD interfaces and be leveraged by designers of AR user interfaces in general.

Toward a Theory of Visual Information Acquisition in Driving

OBJECTIVE

The aim of this study is to describe information acquisition theory, explaining how drivers acquire and represent the information they need.

BACKGROUND

While questions of what drivers are aware of underlie many questions in driver behavior, existing theories do not directly address how drivers in particular and observers in general acquire visual information. Understanding the mechanisms of information acquisition is necessary to build predictive models of drivers' representation of the world and can be applied beyond driving to a wide variety of visual tasks.

METHOD

We describe our theory of information acquisition, looking to questions in driver behavior and results from vision science research that speak to its constituent elements. We focus on the intersection of peripheral vision, visual attention, and eye movement planning and identify how an understanding of these visual mechanisms and processes in the context of information acquisition can inform more complete models of driver knowledge and state.

RESULTS

We set forth our theory of information acquisition, describing the gap in understanding that it fills and how existing questions in this space can be better understood using it.

CONCLUSION

Information acquisition theory provides a new and powerful way to study, model, and predict what drivers know about the world, reflecting our current understanding of visual mechanisms and enabling new theories, models, and applications.

APPLICATION

Using information acquisition theory to understand how drivers acquire, lose, and update their representation of the environment will aid development of driver assistance systems, semiautonomous vehicles, and road safety overall.

Transitions Between Highly Automated and Longitudinally Assisted Driving: The Role of the Initiator in the Fight for Authority

OBJECTIVE

A driving simulator study explored how drivers behaved depending on their initial role during transitions between highly automated driving (HAD) and longitudinally assisted driving (via adaptive cruise control).

BACKGROUND

During HAD, drivers might issue a take-over request (TOR), initiating a transition of control that was not planned. Understanding how drivers behave in this situation and, ultimately, the implications on road safety is of paramount importance.

METHOD

Sixteen participants were recruited for this study and performed transitions of control between HAD and longitudinally assisted driving in a driving simulator. While comparing how drivers behaved depending on whether or not they were the initiators, different handover strategies were presented to analyze how drivers adapted to variations in the authority level they were granted at various stages of the transitions.

RESULTS

Whenever they initiated the transition, drivers were more engaged with the driving task and less prone to follow the guidance of the proposed strategies. Moreover, initiating a transition and having the highest authority share during the handover made the drivers more engaged with the driving task and attentive toward the road.

CONCLUSION

Handover strategies that retained a larger authority share were more effective whenever the automation initiated the transition. Under driver-initiated transitions, reducing drivers' authority was detrimental for both performance and comfort.

APPLICATION

As the operational design domain of automated vehicles (Society of Automotive Engineers [SAE] Level 3/4) expands, the drivers might very well fight boredom by taking over spontaneously, introducing safety issues so far not considered but nevertheless very important.

The effects of takeover request lead time on drivers' situation awareness for manually exiting from freeways: A web-based study on level 3 automated vehicles

Conditional automation systems allow drivers to turn their attention away from the driving task in certain scenarios but still require drivers to gain situation awareness (SA) upon a takeover request (ToR) and resume manual control when the system is unable to handle the upcoming situation. Unlike time-critical takeover situations in which drivers must respond within a relatively short time frame, the ToRs for non-critical events such as exiting from a freeway can be scheduled way ahead of time. It is unknown how the ToR lead time affects driver SA for resuming manual control and when to send the ToR is most appropriate in non-critical takeover events. The present study conducted a web-based, supervised experiment with 31 participants using conditional automation systems in freeway existing scenarios while playing a mobile game. Each participant experienced 12 trials with different ToR lead times (6, 8, 10, 12, 14, 16, 18, 20, 25, 30, 45, and 60 s) for exiting from freeways in a randomized order. Driver SA was measured by using a freeze probe technique in each trial when the participant pressed the spacebar on the laptop to simulate the takeover action. Results revealed a positive effect of longer ToR lead times on driver SA for resuming control to exit from freeways and the effect leveled off at the lead time of 16-30 s. The participants tended to postpone their takeover actions further when they were given a longer ToR lead time and it did not level off up to 60 s. Nevertheless, not all drivers waited till the last moment to take over AVs even though they did not get sufficient SA. The ToR lead time of 16-30 s was recommended for better SA; and it could be narrowed down to 25-30 s if considering the subjective evaluations on takeover readiness, workload, and trust. The findings provide implications for the future design of conditional automation systems used for freeway driving.

The effect of inconsistent steering guidance during transitions from Highly Automated Driving

This driving simulator study investigated the effect of inconsistent steering guidance during system and user-initiated transitions from Highly Automated Driving (HAD). In particular, the aim of the study was to understand if steering conflicts could be achieved by adopting inconsistent steering guidance and whether these conflicts could be exploited to accelerate drivers' steering engagement within a limited time. Inconsistent steering guidance was generated by switching the guidance on and off at 3 different frequencies (0.1, 0.2 and 0.3 Hz). Results revealed that steering engagement has more to do with the initiation rather than the quality of the steering guidance. In fact, drivers were more engaged with the steering task when they initiated the transition themselves. Compared to system-initiated transitions, in user-initiated ones, drivers exerted stronger steering inputs throughout the transition, which allowed them to maintain larger Time To Lane Crossing (TTLC) values with fewer steering corrections. During system-initiated transitions, drivers started to actively engage with the steering activity only after more than 5 s from the start of the transition but were able to achieve a steering behaviour close to the one shown during user-initiated transitions at 10 s.

Impact of Age on Takeover Behavior in Automated Driving in Complex Traffic Situations: A Case Study of Beijing, China

Research on the influence of age on various automated driving conditions will contribute to an understanding of driving behavior characteristics and the development of specific automated driving systems. This study aims to analyze the relationship between age and takeover behavior in automated driving, where 16 test conditions were taken into consideration, including two driving tasks, two warning times and four driving scenarios. Forty-two drivers in Beijing, China in 2020 were recruited to participate in a static driving simulator with Level 3 (L3) conditional automation to obtain detailed test information of the recorded takeover time, mean speed and mean lateral offset. An ANOVA test was proposed to examine the significance among different age groups and conditions. The results confirmed that reaction time increased significantly with age and the driving stability of the older group was worse than the young and middle groups. It was also indicated that the older group could not adapt to complex tasks well when driving due to their limited cognitive driving ability. Additionally, the higher urgency of a scenario explained the variance in the takeover quality. According to the obtained influencing mechanisms, policy implications for the development of vehicle automation, considering the various driving behaviors of drivers, were put forward, so as to correctly identify the high-risk driving conditions in different age groups. For further research, on-road validation will be necessary in order to check for driving simulation-related effects.

Situation Awareness in Remote Operators of Autonomous Vehicles: Developing a Taxonomy of Situation Awareness in Video-Relays of Driving Scenes

Even entirely driverless vehicles will sometimes require remote human intervention. Existing SA frameworks do not acknowledge the significant human factors challenges unique to a driver in charge of a vehicle that they are not physically occupying. Remote operators will have to build up a mental model of the remote environment facilitated by monitor view and video feed. We took a novel approach to “freeze and probe” techniques to measure SA, employing a qualitative verbal elicitation task to uncover what people “see” in a remote scene when they are not constrained by rigid questioning. Participants (n = 10) watched eight videos of driving scenes randomized and counterbalanced across four road types (motorway, rural, residential and A road). Participants recorded spoken descriptions when each video stopped, detailing what was happening (SA Comprehension) and what could happen next (SA Prediction). Participant transcripts provided a rich catalog of verbal data reflecting clear interactions between different SA levels. This suggests that acquiring SA in remote scenes is a flexible and fluctuating process of combining comprehension and prediction globally rather than serially, in contrast to what has sometimes been implied by previous SA methodologies (Jones and Endsley, 1996; Endsley, 2000, 2017b). Inductive thematic analysis was used to categorize participants’ responses into a taxonomy aimed at capturing the key elements of people’s reported SA for videos of driving situations. We suggest that existing theories of SA need to be more sensitively applied to remote driving contexts such as remote operators of autonomous vehicles.

Modeling takeover behavior in level 3 automated driving via a structural equation model: Considering the mediating role of trust

The takeover process in level 3 automated driving determines the controllability of the functions of automated vehicles and thereby traffic safety. In this study, we attempted to explain drivers' takeover performance variation in a level 3 automated vehicle in consideration of the effects of trust, system characteristics, environmental characteristics, and driver characteristics with a structural equation model. The model was built by incorporating drivers' takeover time and quality as endogenous variables. A theoretical framework of the model was hypothesized on the basis of the ACT-R cognitive architecture and relevant research results. The validity of the model was confirmed using data collected from 136 driving simulator samples under the condition of voluntary non-driving-related tasks. Results revealed that takeover time budget was the most critical factor in promoting the safety and stability of takeover process, which, together with traffic density, drivers' age and manual driving experience, determined drivers' takeover quality directly. In addition, the pre-existing experience with an automated system or a similar technology and self-confidence of the driver, as well as takeover time budget, strongly influenced the takeover time directly. Apart from the direct effects mentioned above, trust, as an intermediary variable, explained a major portion of the variance in takeover time. Theoretically, these findings suggest that takeover behavior could be comprehensively evaluated from the two dimensions of takeover time and quality through the combination of trust, driver characteristics, environmental characteristics, and vehicle characteristics. The influence mechanism of the above factors is complex and multidimensional. In addition to the form of direct influence, trust, as an intermediary variable, could reflect the internal mechanism of the takeover behavior variation. Practically, the findings emphasize the crucial role of trust in the change in takeover behavior through the dimensions of subjective trust level and monitoring strategy, which may provide new insights into the function design of takeover process.

A low-cost, high-performance video-based binocular eye tracker for psychophysical research

We describe a high-performance, pupil-based binocular eye tracker that approaches the performance of a well-established commercial system, but at a fraction of the cost. The eye tracker is built from standard hardware components, and its software (written in Visual C++) can be easily implemented. Because of its fast and simple linear calibration scheme, the eye tracker performs best in the central 10 degrees of the visual field. The eye tracker possesses a number of useful features: (1) automated calibration simultaneously in both eyes while subjects fixate four fixation points sequentially on a computer screen, (2) automated realtime continuous analysis of measurement noise, (3) automated blink detection, (4) and realtime analysis of pupil centration artifacts. This last feature is critical because it is known that pupil diameter changes can be erroneously registered by pupil-based trackers as a change in eye position. We evaluated the performance of our system against that of a wellestablished commercial system using simultaneous measurements in 10 participants. We propose our low-cost eye tracker as a promising resource for studies of binocular eye movements.

Designing for the Extremes: Modeling Drivers' Response Time to Take Back Control From Automation Using Bayesian Quantile Regression

OBJECTIVE

Understanding the factors that affect drivers' response time in takeover from automation can help guide the design of vehicle systems to aid drivers. Higher quantiles of the response time distribution might indicate a higher risk of an unsuccessful takeover. Therefore, assessments of these systems should consider upper quantiles rather than focusing on the central tendency.

BACKGROUND

Drivers' responses to takeover requests can be assessed using the time it takes the driver to take over control. However, all the takeover timing studies that we could find focused on the mean response time.

METHOD

A study using an advanced driving simulator evaluated the effect of takeover request timing, event type at the onset of a takeover, and visual demand on drivers' response time. A mixed effects model was fit to the data using Bayesian quantile regression.

RESULTS

Takeover request timing, event type that precipitated the takeover, and the visual demand all affect driver response time. These factors affected the 85th percentile differently than the median. This was most evident in the revealed stopped vehicle event and conditions with a longer time budget and scenes with lower visual demand.

CONCLUSION

Because the factors affect the quantiles of the distribution differently, a focus on the mean response can misrepresent actual system performance. The 85th percentile is an important performance metric because it reveals factors that contribute to delayed responses and potentially dangerous outcomes, and it also indicates how well the system accommodates differences between drivers.

Modeling takeover time based on non-driving-related task attributes in highly automated driving

This study aims to investigate the effects of non-driving-related tasks (NDRTs) on the transition of control in highly automated driving (HAD) by investigating the effects of NDRT physical, visual, and cognitive attributes during transition of control. A conceptual model of the takeover process is proposed by dividing this process into motor and mental reactions. A laboratory experiment was conducted to evaluate the effects of each NDRT attribute on the corresponding stage of the process of taking over control. A prediction model was developed using the results of multiple linear regression analysis. Additionally, a validation experiment with nine NDRTs and a baseline condition was conducted to determine the extent to which the developed model explains the takeover time for each NDRT condition. The results showed that the timing aspects of the transition of control in HAD largely consist of participant motor reactions that are affected by the physical attributes of NDRTs.

Optimal Time Intervals in Two-Stage Takeover Warning Systems With Insight Into the Drivers' Neuroticism Personality

Conditional automated driving [level 3, Society of Automotive Engineers (SAE)] requires drivers to take over the vehicle when an automated system's failure occurs or is about to leave its operational design domain. Two-stage warning systems, which warn drivers in two steps, can be a promising method to guide drivers in preparing for the takeover. However, the proper time intervals of two-stage warning systems that allow drivers with different personalities to prepare for the takeover remain unclear. This study explored the optimal time intervals of two-stage warning systems with insights into the drivers' neuroticism personality. A total of 32 drivers were distributed into two groups according to their self-ratings in neuroticism (high vs. low). Each driver experienced takeover under the two-stage warning systems with four time intervals (i.e., 3, 5, 7, and 9 s). The takeover performance (i.e., hands-on-steering-wheel time, takeover time, and maximum resulting acceleration) and subjective opinions (i.e., appropriateness and usefulness) for time intervals and situation awareness (SA) were recorded. The results showed that drivers in the 5-s time interval had the best takeover preparation (fast hands-on steering wheel responses and sufficient SA). Furthermore, both the 5- and 7-s time intervals resulted in more rapid takeover reactions and were rated more appropriate and useful than the 3- and 9-s time intervals. In terms of personality, drivers with high neuroticism tended to take over immediately after receiving takeover messages, at the cost of SA deficiency. In contrast, drivers with low neuroticism responded safely by judging whether they gained enough SA. We concluded that the 5-s time interval was optimal for drivers in two-stage takeover warning systems. When considering personality, drivers with low neuroticism had no strict requirements for time intervals. However, the extended time intervals were favorable for drivers with high neuroticism in developing SA. The present findings have reference implications for designers and engineers to set the time intervals of two-stage warning systems according to the neuroticism personality of drivers.

Updating our understanding of situation awareness in relation to remote operators of autonomous vehicles

The introduction of autonomous vehicles (AVs) could prevent many accidents attributable to human driver error. However, even entirely driverless vehicles will sometimes require remote human intervention. Current taxonomies of automated driving do not acknowledge the possibility of remote control of AVs or the challenges that are unique to such a driver in charge of a vehicle that they are not physically occupying. Yet there are significant differences between situation awareness (SA) in normal driving contexts and SA in these remote driving operations. We argue that the established understanding of automated driving requires updating to include the context of remote operation that is likely to come in to play at higher levels of automation. It is imperative to integrate the role of the remote operator within industry standard taxonomies, so that regulatory frameworks can be established with regards to the training required for remote operation, the necessary equipment and technology, and a comprehensive inventory of the use cases under which we could expect remote operation to be carried out. We emphasise the importance of designing control interfaces in a way that will maximise remote operator (RO) SA and we identify some principles for designing systems aimed at increasing an RO's sense of embodiment in the AV that requires temporary control.

Virtual reality tour for first-time users of highly automated cars: Comparing the effects of virtual environments with different levels of interaction fidelity

Research in aviation and driving has highlighted the importance of training as an effective approach to reduce the costs associated with the supervisory role of the human in automated systems. However, only a few studies have investigated the effect of training on highly automated driving. Moreover, available interactive trainings are mostly based on automated driving simulators and the application of immersive technology such as Virtual Reality (VR) as a low-cost training solution has not been widely adopted. In this study, we developed three types of familiarization tours (low-fidelity VR, high-fidelity VR, and video) to train first-time users of highly automated cars. Then, the effectiveness of these tours was investigated on automation trust and driving performance in several critical and non-critical transition tasks in four groups: control, video, low-fidelity VR, and high-fidelity VR. The results revealed the positive impact of the tours on trust and transition performance at the first time of measurement. Takeover quality only improved when practices were presented in high-fidelity VR. After three times of exposure to transition requests, trust and transition performance of all groups converged to those of the high-fidelity VR group, demonstrating that: a) experiencing takeover transition during the training may reduce costs associated with first critical takeover request in highly automated driving, b) the VR tour with high level of interaction fidelity was superior to other training methods, and c) untrained and less-trained drivers learned about automation after a few trials. Knowledge resulting from this research could help develop cost-effective solutions for automated driving training in dealerships and car rental centers.

Estimating the out-of-the-loop phenomenon from visual strategies during highly automated driving

During highly automated driving, drivers no longer physically control the vehicle but they might need to monitor the driving scene. This is true for SAE level 2, where monitoring the external environment is required; it is also true for level 3, where drivers must react quickly and safely to a take-over request. Without such monitoring, even if only partial, drivers are considered out-of-the-loop (OOTL) and safety may be compromised. The OOTL phenomenon may be particularly important for long automated driving periods during which mind wandering can occur. This study scrutinized drivers' visual behaviour for 18 min of highly automated driving. Intersections between gaze and 13 areas of interest (AOIs) were analysed, considering both static and dynamic indicators. An estimation of self-reported mind wandering based on gaze behaviour was performed using partial least squares (PLS) regression models. The outputs of the PLS regressions allowed defining visual strategies associated with good monitoring of the driving scene. This information may enable online estimation of the OOTL phenomenon based on a driver's spontaneous visual behaviour.

Attentional Demand as a Function of Contextual Factors in Different Traffic Scenarios

OBJECTIVE

To assess the attentional demand of different contextual factors in driving.

BACKGROUND

The attentional demand on the driver varies with the situation. One approach for estimating the attentional demand, via spare capacity, is to use visual occlusion.

METHOD

Using a 3 × 5 within-subjects design, 33 participants drove in a fixed-base simulator in three scenarios (i.e., urban, rural, and motorway), combined with five fixed occlusion durations (1.0, 1.4, 1.8, 2.2, and 2.6 s). By pressing a microswitch on a finger, the driver initiated each occlusion, which lasted for the same predetermined duration within each trial. Drivers were instructed to occlude their vision as often as possible while still driving safely.

RESULTS

Stepwise logistic regression per scenario indicated that the occlusion predictors varied with scenario. In the urban environment, infrastructure-related variables had the biggest influence, whereas the distance to oncoming traffic played a major role on the rural road. On the motorway, occlusion duration and time since the last occlusion were the main determinants.

CONCLUSION

Spare capacity is dependent on the scenario, selected speed, and individual factors. This is important for developing workload managers, infrastructural design, and aspects related to transfer of control in automated driving.

APPLICATION

Better knowledge of the determinants of spare capacity in the road environment can help improve workload managers, thereby contributing to more efficient and safer interaction with additional tasks.

Task Interruption and Control Recovery Strategies After Take-Over Requests Emphasize Need for Measures of Situation Awareness

OBJECTIVE

Our objective was to determine whether there is a need to go beyond measures of automation deactivation time to understand the transition to manual driving after take-over requests (TORs) using the example of office tasks as nondriving-related tasks (NDRTs).

BACKGROUND

Office tasks are likely NDRTs during automated commutes to/from work. Complex tasks can influence how manual control and visual attention is recovered after TORs.

METHOD

N = 51 participants in a driving simulator performed either one of two office tasks or no task (between subjects). We recorded reaction times in a high-urgency and low-urgency scenario (within subjects) and analyzed task interruption strategies.

RESULTS

90% of the participants who performed an NDRT deactivated the automation after 7 to 8 s. However, 90% of the same drivers looked at the side mirror for the first time only after 11 to 14 s. Drivers with office tasks either interrupted the tasks sequentially or in parallel. Strategies were not adapted to the take-over situation or the task but appeared to be due to individual preferences.

CONCLUSION

Drivers engaged in NDRTs may neglect lower priority subtasks after a TOR, such as mirror checking. Therefore, there is a need to go beyond measures of automation deactivation time to understand the transition to manual driving. Using analyses of attentional dynamics during take-over situations may enhance the safety of future car-driver handover assistance systems.

APPLICATION

If low driver availability is detected, TORs should only be used as a fallback option if sufficient time and adaptive driver support can be provided.

Transport safety and human factors in the era of automation: What can transport modes learn from each other?

One of the main aims of introducing automation in transport is to improve safety by reducing or eliminating human errors; it is often argued however that this may induce new types of errors. There is different level of maturity with automation in different transport modes (road, aviation, maritime and rail), however no systematic research has been conducted on the lessons learned in different sectors, so that they can be exploited for the design of safer automated systems. The aim of this paper is to review the impact of key human factors on the safety of automated transport systems, with focus on relevant experiences from different transport sectors. A systematic literature review is carried out on the following topics: the level of trust in automation - in particular the impact of mis-aligned trust, i.e. mistrust vs overreliance, the resulting impact on operator situation awareness (SA), the implications for takeover control from machine to human, and the role of experience and training on using automated transport systems. The results revealed several areas where experiences from the aviation and road domain can be transferable to other sectors. Experiences from maritime and rail transport, although limited, tend to confirm the general patterns. Remarkably, in the road sector where higher levels of automation are only recently introduced, there are clearer and more quantitative approaches to human factors, while other sectors focus only on mental modes. Other sectors could use similar approaches to define their own context-specific metrics. The paper makes a synthesis of key messages on automation safety in different transport sectors, and presents an assessment of their transferability.

Predicting road scenes from brief views of driving video

If a vehicle is driving itself and asks the driver to take over, how much time does the driver need to comprehend the scene and respond appropriately? Previous work on natural-scene perception suggests that observers quickly acquire the gist, but gist-level understanding may not be sufficient to enable action. The moving road environment cannot be studied with static images alone, and safe driving requires anticipating future events. We performed two experiments to examine how quickly subjects could perceive the road scenes they viewed and make predictions based on their mental representations of the scenes. In both experiments, subjects performed a temporal-order prediction task, in which they viewed brief segments of road video and indicated which of two still frames would come next after the end of the video. By varying the duration of the previewed video clip, we determined the viewing duration required for accurate prediction of recorded road scenes. We performed an initial experiment on Mechanical Turk to explore the space, and a follow-up experiment in the lab to address questions of road type and stimulus discriminability. Our results suggest that representations which enable prediction can be developed from brief views of a road scene, and that different road environments (e.g., city versus highway driving) have a significant impact on the viewing durations drivers require to make accurate predictions of upcoming scenes.

Identifying the Causes of Drivers' Hazardous States Using Driver Characteristics, Vehicle Kinematics, and Physiological Measurements

Drivers’ hazardous physical and mental states (e.g., distraction, fatigue, stress, and high workload) have a major effect on driving performance and strongly contribute to 25–50% of all traffic accidents. They are caused by numerous factors, such as cell phone use or lack of sleep. However, while significant research has been done on detecting hazardous states, most studies have not tried to identify the causes of the hazardous states. Such information would be very useful, as it would allow intelligent vehicles to better respond to a detected hazardous state. Thus, this study examined whether the cause of a driver’s hazardous state can be automatically identified using a combination of driver characteristics, vehicle kinematics, and physiological measures. Twenty-one healthy participants took part in four 45-min sessions of simulated driving, of which they were mildly sleep-deprived for two sessions. Within each session, there were eight different scenarios with different weather (sunny or snowy), traffic density and cell phone usage (with or without cell phone). During each scenario, four physiological (respiration, electrocardiogram, skin conductance, and body temperature) and eight vehicle kinematics measures were monitored. Additionally, three self-reported driver characteristics were obtained: personality, stress level, and mood. Three feature sets were formed based on driver characteristics, vehicle kinematics, and physiological signals. All possible combinations of the three feature sets were used to classify sleep deprivation (drowsy vs. alert), traffic density (low vs. high), cell phone use, and weather conditions (foggy/snowy vs. sunny) with highest accuracies of 98.8%, 91.4%, 82.3%, and 71.5%, respectively. Vehicle kinematics were most useful for classification of weather and traffic density while physiology and driver characteristics were useful for classification of sleep deprivation and cell phone use. Furthermore, a second classification scheme was tested that also incorporates information about whether or not other causes of hazardous states are present, though this did not result in higher classification accuracy. In the future, these classifiers could be used to identify both the presence and cause of a driver’s hazardous state, which could serve as the basis for more intelligent intervention systems.

Assessment selection in human-automation interaction studies: The Failure-GAM2E and review of assessment methods for highly automated driving

Highly automated driving will change driver's behavioural patterns. Traditional methods used for assessing manual driving will only be applicable for the parts of human-automation interaction where the driver intervenes such as in hand-over and take-over situations. Therefore, driver behaviour assessment will need to adapt to the new driving scenarios. This paper aims at simplifying the process of selecting appropriate assessment methods. Thirty-five papers were reviewed to examine potential and relevant methods. The review showed that many studies still relies on traditional driving assessment methods. A new method, the Failure-GAM²E model, with purpose to aid assessment selection when planning a study, is proposed and exemplified in the paper. Failure-GAM²E includes a systematic step-by-step procedure defining the situation, failures (Failure), goals (G), actions (A), subjective methods (M), objective methods (M) and equipment (E). The use of Failure-GAM²E in a study example resulted in a well-reasoned assessment plan, a new way of measuring trust through feet movements and a proposed Optimal Risk Management Model. Failure-GAM²E and the Optimal Risk Management Model are believed to support the planning process for research studies in the field of human-automation interaction.

More than the Useful Field: Considering peripheral vision in driving

Applied research on driving and basic vision research have held similar views on central, fovea-based vision as the core of visual perception. In applied work, the concept of the Useful Field, as determined by the Useful Field of View (UFOV) test, divides vision between a "useful" region towards the center of the visual field, and the rest of the visual field. While compelling, this dichotomization is at odds with findings in vision science which demonstrate the capabilities of peripheral vision. In this paper, we examine driving research from this new perspective, and argue for the need for an updated understanding of how drivers acquire information about their operating environment using peripheral vision. The concept of the Useful Field and the UFOV test are not discarded; instead we discuss their strengths, limitations, and future directions. We discuss key findings from vision science on peripheral vision, and a theory that provides insights into its capabilities and limitations. This more complete basic science understanding of peripheral vision informs appropriate use of the UFOV and the Useful Field in driving research going forward.

The Effects of Continuous Driving-Related Feedback on Drivers’ Response to Automation Failures

During prolonged periods of autonomous driving, drivers tend to shift their attention away from the driving task. As a result, they require more time to regain awareness of the driving situation and to react to it. This study examined the use of informative automation that during Level-3 autonomous driving provided drivers with continuous feedback regarding the vehicle’s actions and surroundings. It was hypothesized that the operation of informative automation will trigger drivers to allocate more attention to the driving task and will improve their reaction times when resuming control of the vehicle. Sixteen participants drove manual and autonomous driving segments in a driving simulator equipped with Level-3 automation. For half of the participants, the informative automation issued alerts and messages while for the other half no messages were issued (control). The number of on-road glances served as a proxy for drivers’ attention. Drivers’ performance on handling an unexpected automation failure event was measured using their time-to-brake and time-to-steer. Results showed that drivers using the informative automation made more frequent on-road glances than drivers in the control group. Yet, there were no significant differences in reaction times to the automation failure event between the groups. Explanations and implications of these results are discussed.