A model for naturalistic glance behavior around Tesla Autopilot disengagements

A conceptual framework for automation disengagements

A better understanding of automation disengagements can lead to improved safety and efficiency of automated systems. This study investigates the factors contributing to automation disengagements initiated by human operators and the automation itself by analyzing semi-structured interviews with 103 users of Tesla's Autopilot and FSD Beta. The factors leading to automation disengagements are represented by categories. In total, we identified five main categories, and thirty-five subcategories. The main categories include human operator states (5), human operator's perception of the automation (17), human operator's perception of other humans (3), the automation's perception of the human operator (3), and the automation incapability in the environment (7). Human operators disengaged the automation when they anticipated failure, observed unnatural or unwanted automation behavior (e.g., erratic steering, running red lights), or believed the automation is not capable to operate safely in certain environments (e.g., inclement weather, non-standard roads). Negative experiences of human operators, such as frustration, unsafe feelings, and distrust represent some of the adverse human operate states leading to automation disengagements initiated by human operators. The automation, in turn, monitored human operators and disengaged itself if it detected insufficient vigilance or speed rule violations by human operators. Moreover, human operators can be influenced by the reactions of passengers and other road users, leading them to disengage the automation if they sensed discomfort, anger, or embarrassment due to the automation's actions. The results of the analysis are synthesized into a conceptual framework for automation disengagements, borrowing ideas from the human factor's literature and control theory. This research offers insights into the factors contributing to automation disengagements, and highlights not only the concerns of human operators but also the social aspects of this phenomenon. The findings provide information on potential edge cases of automated vehicle technology, which may help to enhance the safety and efficiency of such systems.

On investigating drivers' attention allocation during partially-automated driving

The use of partially-automated systems require drivers to supervise the system functioning and resume manual control whenever necessary. Yet literature on vehicle automation show that drivers may spend more time looking away from the road when the partially-automated system is operational. In this study we answer the question of whether this pattern is a manifestation of inattentional blindness or, more dangerously, it is also accompanied by a greater attentional processing of the driving scene. Participants drove a simulated vehicle in manual or partially-automated mode. Fixations were recorded by means of a head-mounted eye-tracker. A surprise two-alternative forced-choice recognition task was administered at the end of the data collection whereby participants were quizzed on the presence of roadside billboards that they encountered during the two drives. Data showed that participants were more likely to fixate and recognize billboards when the automated system was operational. Furthermore, whereas fixations toward billboards decreased toward the end of the automated drive, the performance in the recognition task did not suffer. Based on these findings, we hypothesize that the use of the partially-automated driving system may result in an increase in attention allocation toward peripheral objects in the road scene which is detrimental to the drivers' ability to supervise the automated system and resume manual control of the vehicle.

Factors affecting drivers' off-road glance behavior while interacting with in-vehicle voice interfaces

In-vehicle voice-based interfaces have been massively embedded in modern vehicles as a countermeasure to visual-manual distractions. However, limited data are available regarding the actual visual demands imposed on the driver when interacting with such an interface. How those factors that are associated with the drivers themselves affect their visual behavior when interacting with an in-vehicle voice-based interface remains understudied. This study focused on investigating factors affecting drivers' off-road visual behavior while interacting with a voice-based interface. A secondary data analysis leveraging structural equation modeling was performed based on data collected from a large-scale field experiment wherein participants drove a vehicle on the highway and performed a series of tasks using an in-vehicle voice interface. The results indicate that while voice-based interfaces are designed to help reduce drivers' visual demands, drivers can still direct their eyes off the road for a prolonged period of time during the interaction and exhibit potentially risky visual behavior. In addition, individual-level factors can exert influence on drivers' off-road visual behavior in such a way that older drivers and drivers with higher trust in technologies are associated with more long eyes-off-road glances when interacting with a voice-based interface. To promote the general public's adoption of in-vehicle voice interfaces and support safe interactions, we recommend relevant research and system design consider drivers' trust in, perceptions, and previous usage of similar technologies, as well as their individual characteristics such as age.

Driver response and recovery following automation initiated disengagement in real-world hands-free driving

OBJECTIVE

Advanced driver assistance systems are increasingly available in consumer vehicles, making the study of drivers' behavioral adaptation and the impact of automation beneficial for driving safety. Concerns over driver's being out-of-the-loop, coupled with known limitations of automation, has led research to focus on time-critical, system-initiated disengagements. This study used real-world data to assess drivers' response to, and recovery from, automation-initiated disengagements by quantifying changes in visual attention, vehicle control, and time to steady-state behaviors.

METHODS

Fourteen drivers drove for one month each a Cadillac CT6 equipped with Super Cruise (SC), a partial automation system that, when engaged, enables hands-free driving. The vehicles were instrumented with data acquisition systems recording driving kinematics, automation use, GPS, and video. The dataset included 265 SC-initiated disengagements identified across 5,514 miles driven with SC.

RESULTS

Linear quantile mixed-effects models of glance behavior indicated that following SC-initiated disengagement, the proportions of glances to the Road decreased (Q50_Before=0.91, Q50_After=0.69; Q85_Before=1.0, Q85_After=0.79), the proportions of glances to the Instrument Cluster increased (Q50_Before=0.14, Q50_After=0.25; Q85_Before=0.34, Q85_After=0.45), and mean glance duration to the Road decreased by 4.86 sec in Q85. Multinomial logistic regression mixed-models of glance distributions indicated that the number of transitions between glance locations following disengagement increased by 43% and that glances were distributed across fewer locations. When driving hands-free, take over time was significantly longer (2.4 sec) compared to when driving with at least one hand on the steering wheel (1.8 sec). Analysis of moment-to-moment distributional properties of visual attention and steering wheel control following disengagement indicated that on average it took drivers 6.1 sec to start the recovery of glance behavior to the Road and 1.5 sec for trend-stationary proportions of at least one hand on the steering wheel.

CONCLUSIONS

Automation-initiated disengagements triggered substantial changes in driver glance behavior including shorter on-road glances and frequent transitions between Road and Instrument Cluster glance locations. This information seeking behavior may capture drivers' search for information related to the disengagement or the automation state and is likely shaped by the automation design. The study findings can inform the design of more effective driver-centric information displays for smoother transitions and faster recovery.

Multimodal driver state modeling through unsupervised learning

Naturalistic driving data (NDD) can help understand drivers' reactions to each driving scenario and provide personalized context to driving behavior. However, NDD requires a high amount of manual labor to label certain driver's state and behavioral patterns. Unsupervised analysis of NDD can be used to automatically detect different patterns from the driver and vehicle data. In this paper, we propose a methodology to understand changes in driver's physiological responses within different driving patterns. Our methodology first decomposes a driving scenario by using a Bayesian Change Point detection model. We then apply the Latent Dirichlet Allocation method on both driver state and behavior data to detect patterns. We present two case studies in which vehicles were equipped to collect exterior, interior, and driver behavioral data. Four patterns of driving behaviors (i.e., harsh brake, normal brake, curved driving, and highway driving), as well as two patterns of driver's heart rate (HR) (i.e., normal vs. abnormal high HR), and gaze entropy (i.e., low versus high), were detected in these two case studies. The findings of these case studies indicated that among our participants, the drivers' HR had a higher fraction of abnormal patterns during harsh brakes, accelerating and curved driving. Additionally, free-flow driving with close to zero accelerations on the highway was accompanied by more fraction of normal HR as well as a lower gaze entropy pattern. With the proposed methodology we can better understand variations in driver's psychophysiological states within different driving scenarios. The findings of this work, has the potential to guide future autonomous vehicles to take actions that are fit to each specific driver.

Beyond gaze fixation: Modeling peripheral vision in relation to speed, Tesla Autopilot, cognitive load, and age in highway driving

OBJECTIVE

The study aims to model driver perception across the visual field in dynamic, real-world highway driving.

BACKGROUND

Peripheral vision acquires information across the visual field and guides a driver's information search. Studies in naturalistic settings are lacking however, with most research having been conducted in controlled simulation environments with limited eccentricities and driving dynamics.

METHODS

We analyzed data from 24 participants who drove a Tesla Model S with Autopilot on the highway. While driving, participants completed the peripheral detection task (PDT) using LEDs and the N-back task to generate cognitive load. The I-DT (identification by dispersion threshold) algorithm sampled naturalistic gaze fixations during PDTs to cover a broader and continuous spectrum of eccentricity. A generalized Bayesian regression model predicted LED detection probability during the PDT-as a surrogate for peripheral vision-in relation to eccentricity, vehicle speed, driving mode, cognitive load, and age.

RESULTS

The model predicted that LED detection probability was high and stable through near-peripheral vision but it declined rapidly beyond 20°-30° eccentricity, showing a narrower useful field over a broader visual field (maximum 70°) during highway driving. Reduced speed (while following another vehicle), cognitive load, and older age were the main factors that degraded the mid-peripheral vision (20°-50°), while using Autopilot had little effect.

CONCLUSIONS

Drivers can reliably detect objects through near-peripheral vision, but their peripheral detection degrades gradually due to further eccentricity, foveal demand during low-speed vehicle following, cognitive load, and age.

APPLICATIONS

The findings encourage the development of further multivariate computational models to estimate peripheral vision and assess driver situation awareness for crash prevention.