Getting Back Into the Loop: The Perceptual-Motor Determinants of Successful Transitions out of Automated Driving

Repeated conditionally automated driving on the road: How do drivers leave the loop over time?

The goal of this on the road driving study was to investigate how drivers adapt their behavior when driving with conditional vehicle automation (SAE L3) on different occasions. Specifically, we focused on changes in how fast drivers took over control from automation and how their gaze off the road changed over time. On each of three consecutive days, 21 participants drove for 50 min, in a conditionally automated vehicle (Wizard of Oz methodology), on a typical German commuting highway. Over these rides the take-over behavior and gaze behavior were analyzed. The data show that drivers' reactions to non-critical, system initiated, take-overs took about 5.62 s and did not change within individual rides, but on average became 0.72 s faster over the three rides. After these self-paced take-over requests a final urgent take-over request was issued at the end of the third ride. In this scenario participants took over rapidly with an average of 5.28 s. This urgent take-over time was not found to be different from the self-paced take-over requests in the same ride. Regarding gaze behavior, participants' overall longest glance off the road and the percentage of time looked off the road increased within each ride, but stayed stable over the three rides. Taken together, our results suggest that drivers regularly leave the loop by gazing off the road, but multiple exposures to take-over situations in automated driving allow drivers to come back into loop faster.

Drivers' gaze patterns when resuming control with a head-up-display: Effects of automation level and time budget

The removal of drivers' active engagement in driving tasks can lead to erratic gaze patterns in SAE Level 2 (L2) and Level 3 (L3) automation, which has been linked to their subsequential degraded take-over performance. To further address how changes in gaze patterns evolve during the take-over phase, and whether they are influenced by the take-over urgency and the location of the human-machine interface, this driving simulator study used a head-up display (HUD) to relay information about the automation status and conducted take-over driving experiments where the ego car was about to exit the highway with variations in the automation level (L2, L3) and time budget (2 s, 6 s). In L2 automation, drivers were required to monitor the environment, while in L3, they were engaged with a visual non-driving related task. Manual driving was also embodied in the experiments as the baseline. Results showed that, compared to manual driving, drivers in L2 automation focused more on the HUD and Far Road (roadway beyond 2 s time headway ahead), and less on the Near Road (roadway within 2 s time headway ahead); while in L3, drivers' attention was predominantly allocated on the non-driving related task. After receiving take-over requests (TORs), there was a gradual diversion of attention from the Far Road to the Near Road in L2 take-overs. This trend changed nearly in proportion to the time within the time budget and it exaggerated given a shorter time budget of 2 s. While in L3, drivers' gaze distribution was similar in the early stage of take-overs for both time budget conditions (2 s vs. 6 s), where they prioritized their early glances to Near Road with a gradual increase in attention towards Far Road. The HUD used in the present study showed the potential to maintain drivers' attention around the road center during automation and to encourage drivers to glance the road earlier after TORs by reducing glances to the instrument cluster, which might be of significance to take-over safety. These findings were discussed based on an extended conceptual gaze control model, which advances our understanding of gaze patterns around control transitions and their underlying gaze control causations. Implications can be contributed to the design of autonomous vehicles to facilitate the transition of control by guiding drivers' attention appropriately according to drivers' attentional state and the take-over urgency.

The effect of information from dash-based human-machine interfaces on drivers' gaze patterns and lane-change manoeuvres after conditionally automated driving

The goal of this paper was to measure the effect of Human-Machine Interface (HMI) information and guidance on drivers' gaze and takeover behaviour during transitions of control from automation. The motivation for this study came from a gap in the literature, where previous research reports improved performance of drivers' takeover based on HMI information, without considering its effect on drivers' visual attention distribution, and how drivers also use the information available in the environment to guide their response. This driving simulator study investigated drivers' lane-changing behaviour after resumption of control from automation. Different levels of information were provided on a dash-based HMI, prior to each lane change, to investigate how drivers distribute their attention between the surrounding environment and the HMI. The difficulty of the lane change was also manipulated by controlling the position of approaching vehicles in drivers' offside lane. Results indicated that drivers' decision-making time was sensitive to the presence of nearby vehicles in the offside lane, but not directly influenced by the information on the HMI. In terms of gaze behaviour, the closer the position of vehicles in the offside lane, the longer drivers looked in that direction. Drivers looked more at the HMI, and less towards the road centre, when the HMI presented information about automation status, and included an advisory message indicating it was safe to change lane. Machine learning techniques showed a strong relationship between drivers' gaze to the information presented on the HMI, and decision-making time (DMT). These results contribute to our understanding of HMI design for automated vehicles, by demonstrating the attentional costs of an overly-informative HMI, and that drivers still rely on environmental information to perform a lane-change, even when the same information can be acquired by the HMI of the vehicle.

Ambient Light Conveying Reliability Improves Drivers’ Takeover Performance without Increasing Mental Workload

Drivers of L3 automated vehicles (AVs) are not required to continuously monitor the AV system. However, they must be prepared to take over when requested. Therefore, it is necessary to design an in-vehicle environment that allows drivers to adapt their levels of preparedness to the likelihood of control transition. This study evaluates ambient in-vehicle lighting that continuously communicates the current level of AV reliability, specifically on how it could influence drivers’ take-over performance and mental workload (MW). We conducted an experiment in a driving simulator with 42 participants who experienced 10 take-over requests (TORs). The experimental group experienced a four-stage ambient light display that communicated the current level of AV reliability, which was not provided to the control group. The experimental group demonstrated better take-over performance, based on lower vehicle jerks. Notably, perceived MW did not differ between the groups, and the EEG indices of MW (frontal theta power, parietal alpha power, Task–Load Index) did not differ between the groups. These findings suggest that communicating the current level of reliability using ambient light might help drivers be better prepared for TORs and perform better without increasing their MW.

Objective assessment of trait attentional control predicts driver response to emergency failures of vehicular automation

With the advent of autonomous driving, the issue of human intervention during safety-critical events is an urgent topic of research. Supervisory monitoring, taking over vehicle control during automation failures and then bringing the vehicle to safety under time pressure are cognitively demanding tasks that pose varying difficulties across the driving population. This underpins a need to investigate individual differences (i.e., how people differ in their dispositional traits) in driver responses to automation system limits, so that autonomous vehicle design can be tailored to meet the safety-critical needs of higher-risk drivers. However, few studies thus far have examined individual differences, with self-report measures showing limited ability to predict driver takeover performance. To address this gap, the present study explored the utility of an established brain activity-based objective index of trait attentional control (frontal theta/beta ratio; TBR) in predicting driver interactions with conditional automation. Frontal TBR predicted drivers' average takeover reaction time, as well as the likelihood of accident after takeover. Moving towards practical applications, this study also demonstrated the utility of streamlined estimates of frontal TBR measured from the forehead electrodes and from a single crown electrode, with the latter showing better fidelity and predictive value. Overall, TBR is behaviourally relevant, measurable with minimal sensors and easily computable, rendering it a promising candidate for practical and objective assessment of drivers' neurocognitive traits that contribute to their AV driving readiness.

Keeping the driver in the loop through semi-automated or manual lane changes in conditionally automated driving

In the current study we investigated if drivers of conditionally automated vehicles can be kept in the loop through lane change maneuvers. More specifically, we examined whether involving drivers in lane-changes during a conditionally automated ride can influence critical take-over behavior and keep drivers' gaze on the road. In a repeated measures driving simulator study (n = 85), drivers drove the same route three times, each trial containing four lane changes that were all either (1) automated, (2) semi-automated or (3) manual. Each ride ended with a critical take-over situation that could be solved by braking and/or steering. Critical take-over reactions were analyzed with a linear mixed model and parametric accelerated failure time survival analysis. As expected, semi-automated and manual lane changes throughout the ride led to 13.5% and 17.0% faster maximum deceleration compared to automated lane changes. Additionally, semi-automated and manual lane changes improved the quality of the take-over by significantly decreasing standard deviation of the steering wheel angle. Unexpectedly, drivers in the semi-automated condition were slowest to start the braking maneuver. This may have been caused by the drivers' confusion as to how the semi-automated system would react. Additionally, the percentage gaze off-the-road was significantly decreased by the semi-automated (6.0%) and manual (6.6%) lane changes. Taken together, the results suggest that semi-automated and manual transitions may be an alarm-free instrument which developers could use to help maintain drivers' perception-action loop and improve automated driving safety.

Displays for Productive Non-Driving Related Tasks: Visual Behavior and Its Impact in Conditionally Automated Driving

(1) Background: Primary driving tasks are increasingly being handled by vehicle automation so that support for non-driving related tasks (NDRTs) is becoming more and more important. In SAE L3 automation, vehicles can require the driver-passenger to take over driving controls, though. Interfaces for NDRTs must therefore guarantee safe operation and should also support productive work. (2) Method: We conducted a within-subjects driving simulator study (N=53) comparing Heads-Up Displays (HUDs) and Auditory Speech Displays (ASDs) for productive NDRT engagement. In this article, we assess the NDRT displays’ effectiveness by evaluating eye-tracking measures and setting them into relation to workload measures, self-ratings, and NDRT/take-over performance. (3) Results: Our data highlights substantially higher gaze dispersion but more extensive glances on the road center in the auditory condition than the HUD condition during automated driving. We further observed potentially safety-critical glance deviations from the road during take-overs after a HUD was used. These differences are reflected in self-ratings, workload indicators and take-over reaction times, but not in driving performance. (4) Conclusion: NDRT interfaces can influence visual attention even beyond their usage during automated driving. In particular, the HUD has resulted in safety-critical glances during manual driving after take-overs. We found this impacted workload and productivity but not driving performance.

Model-based estimation of the state of vehicle automation as derived from the driver's spontaneous visual strategies

When manually steering a car, the driver's visual perception of the driving scene and his or her motor actions to control the vehicle are closely linked. Since motor behaviour is no longer required in an automated vehicle, the sampling of the visual scene is affected. Autonomous driving typically results in less gaze being directed towards the road centre and a broader exploration of the driving scene, compared to manual driving. To examine the corollary of this situation, this study estimated the state of automation (manual or automated) on the basis of gaze behaviour. To do so, models based on partial least square regressions were computed by considering the gaze behaviour in multiple ways, using static indicators (percentage of time spent gazing at 13 areas of interests), dynamic indicators (transition matrices between areas) or both together. Analysis of the quality of predictions for the different models showed that the best result was obtained by considering both static and dynamic indicators. However, gaze dynamics played the most important role in distinguishing between manual and automated driving. This study may be relevant to the issue of driver monitoring in autonomous vehicles.

Drivers use active gaze to monitor waypoints during automated driving

Automated vehicles (AVs) will change the role of the driver, from actively controlling the vehicle to primarily monitoring it. Removing the driver from the control loop could fundamentally change the way that drivers sample visual information from the scene, and in particular, alter the gaze patterns generated when under AV control. To better understand how automation affects gaze patterns this experiment used tightly controlled experimental conditions with a series of transitions from 'Manual' control to 'Automated' vehicle control. Automated trials were produced using either a 'Replay' of the driver's own steering trajectories or standard 'Stock' trials that were identical for all participants. Gaze patterns produced during Manual and Automated conditions were recorded and compared. Overall the gaze patterns across conditions were very similar, but detailed analysis shows that drivers looked slightly further ahead (increased gaze time headway) during Automation with only small differences between Stock and Replay trials. A novel mixture modelling method decomposed gaze patterns into two distinct categories and revealed that the gaze time headway increased during Automation. Further analyses revealed that while there was a general shift to look further ahead (and fixate the bend entry earlier) when under automated vehicle control, similar waypoint-tracking gaze patterns were produced during Manual driving and Automation. The consistency of gaze patterns across driving modes suggests that active-gaze models (developed for manual driving) might be useful for monitoring driver engagement during Automated driving, with deviations in gaze behaviour from what would be expected during manual control potentially indicating that a driver is not closely monitoring the automated system.

Vision, attention, and driving

Safe driving demands the coordination of multiple sensory and cognitive functions, such as vision and attention. Patients with neurologic or ophthalmic disease are exposed to selective pathophysiologic insults to driving-critical systems, placing them at a higher risk for unsafe driving and restricted driving privileges. Here, we evaluate how vision and attention contribute to unsafe driving across different patient populations. In ophthalmic disease, we focus on macular degeneration, glaucoma, diabetic retinopathy, and cataract; in neurologic disease, we focus on Alzheimer's disease, Parkinson's disease, and multiple sclerosis. Unsafe driving is generally associated with impaired vision and attention in ophthalmic and neurologic patients, respectively. Furthermore, patients with ophthalmic disease experience some degree of impairment in attention. Similarly, patients with neurologic disease experience some degree of impairment in vision. While numerous studies have demonstrated a relationship between impaired vision and unsafe driving in neurologic disease, there remains a dearth of knowledge regarding the relationship between impaired attention and unsafe driving in ophthalmic disease. In summary, this chapter confirms-and offers opportunities for future research into-the contribution of vision and attention to safe driving.

Estimating the out-of-the-loop phenomenon from visual strategies during highly automated driving

During highly automated driving, drivers no longer physically control the vehicle but they might need to monitor the driving scene. This is true for SAE level 2, where monitoring the external environment is required; it is also true for level 3, where drivers must react quickly and safely to a take-over request. Without such monitoring, even if only partial, drivers are considered out-of-the-loop (OOTL) and safety may be compromised. The OOTL phenomenon may be particularly important for long automated driving periods during which mind wandering can occur. This study scrutinized drivers' visual behaviour for 18 min of highly automated driving. Intersections between gaze and 13 areas of interest (AOIs) were analysed, considering both static and dynamic indicators. An estimation of self-reported mind wandering based on gaze behaviour was performed using partial least squares (PLS) regression models. The outputs of the PLS regressions allowed defining visual strategies associated with good monitoring of the driving scene. This information may enable online estimation of the OOTL phenomenon based on a driver's spontaneous visual behaviour.

Predicting takeover response to silent automated vehicle failures

Current and foreseeable automated vehicles are not able to respond appropriately in all circumstances and require human monitoring. An experimental examination of steering automation failure shows that response latency, variability and corrective manoeuvring systematically depend on failure severity and the cognitive load of the driver. The results are formalised into a probabilistic predictive model of response latencies that accounts for failure severity, cognitive load and variability within and between drivers. The model predicts high rates of unsafe outcomes in plausible automation failure scenarios. These findings underline that understanding variability in failure responses is crucial for understanding outcomes in automation failures.

Looking at the Road When Driving Around Bends: Influence of Vehicle Automation and Speed

When negotiating bends car drivers perform gaze polling: their gaze shifts between guiding fixations (GFs; gaze directed 1–2 s ahead) and look-ahead fixations (LAFs; longer time headway). How might this behavior change in autonomous vehicles where the need for constant active visual guidance is removed? In this driving simulator study, we analyzed this gaze behavior both when the driver was in charge of steering or when steering was delegated to automation, separately for bend approach (straight line) and the entry of the bend (turn), and at various speeds. The analysis of gaze distributions relative to bend sections and driving conditions indicate that visual anticipation (through LAFs) is most prominent before entering the bend. Passive driving increased the proportion of LAFs with a concomitant decrease of GFs, and increased the gaze polling frequency. Gaze polling frequency also increased at higher speeds, in particular during the bend approach when steering was not performed. LAFs encompassed a wide range of eccentricities. To account for this heterogeneity two sub-categories serving distinct information requirements are proposed: mid-eccentricity LAFs could be more useful for anticipatory planning of steering actions, and far-eccentricity LAFs for monitoring potential hazards. The results support the idea that gaze and steering coordination may be strongly impacted in autonomous vehicles.