How we can measure the non-driving-task engagement in automated driving: Comparing flow experience and workload

Exploring the Timing of Disengagement From Nondriving Related Tasks in Scheduled Takeovers With Pre-Alerts: An Analysis of Takeover-Related Measures

OBJECTIVES

This study aimed to investigate drivers' disengagement from nondriving related tasks (NDRT) during scheduled takeovers and to evaluate its impact on takeover performance.

BACKGROUND

During scheduled takeovers, drivers typically have sufficient time to prepare. However, inadequate disengagement from NDRTs can introduce safety risks.

METHOD

Participants experienced scheduled takeovers using a driving simulator, undergoing two conditions, with and without an NDRT. We assessed their takeover performance and monitored their NDRT disengagement from visual, cognitive, and physical perspectives.

RESULTS

The study examined three NDRT disengagement timings (DTs): DT1 (disengaged before the takeover request), DT2 (disengaged after the request but before taking over), and DT3 (not disengaged). The impact of NDRT on takeover performance varied depending on DTs. Specifically, DT1 demonstrated no adverse effects; DT2 impaired takeover time, while DT3 impaired both takeover time and quality. Additionally, participants who displayed DT1 exhibited longer eye-off-NDRT duration and a higher eye-off-NDRT count during the prewarning stage compared to those with DT2 and DT3.

CONCLUSION

Drivers can benefit from earlier disengagement from NDRTs, demonstrating resilience to the adverse effects of NDRTs on takeover performance. The disengagement of cognition is often delayed compared to that of eyes and hands, potentially leading to DT3. Moreover, visual disengagement from NDRTs during the prewarning stage could distinguish DT1 from the other two.

APPLICATION

Our study emphasizes considering NDRT disengagement in designing systems for scheduled takeovers. Measures should be taken to promote early disengagement, facilitate cognitive disengagement, and employ visual disengagement during the prewarning period as predictive indicators of DTs.

Correlation of Work Fatigue and Mental Workload in Train Drivers: A Cross-sectional Study

BACKGROUND

Evidence suggests that train drivers experience a high level of fatigue and mental workload. The present study aimed to assess overall, physical, and mental fatigue levels and their correlations with the mental workload in the metro train operation. Study Design: A cross-sectional study.

METHODS

This study was conducted on all 1194 train drivers in the Tehran Metro. The train drivers completed the Samn-Perelli Fatigue Scale and the Fatigue Assessment Scales at the beginning and end of the shift. In addition, they completed the National Aeronautics and Space Administration Task Load Index in the middle and at the end of the shift. Correlation and regression analyses were performed on the data to test the study hypothesis.

RESULTS

Overall, physical, and mental fatigue levels increased significantly at the end of the shift compared to the onset of the shift (P<0.001). The mental workload and related dimensions were significantly increased at the end of the shift compared to the middle of the shift (P<0.001). Mental demand was the most important workload problem among the train drivers. The highest correlation was found between overall workload and time pressure (R=0.68, P<0.001).

CONCLUSION

The mental workload had a significant correlation with work fatigue in the train drivers. Control measures should be focused on the mental workload and related dimensions, especially mental demand and time pressure.

Impact of non-driving related tasks while operating automated driving systems (ADS): A systematic review

Automated Driving Systems (ADS) (SAE, 2021), promise improved safety and comfort for drivers. Current technological advances have resulted in increased automation capabilities. However, with the increase in automation capabilities, there is a shift in how drivers interact with their vehicles. Drivers can now temporarily hand over the control of the driving task to ADS under certain conditions. However, with ADS in temporary control of the vehicle, drivers may choose to engage in non-driving related tasks (NDRT). The current capabilities of ADS do not allow drivers to hand over control of the driving task indefinitely. Drivers must remain aware and be ready to take back control if necessary. There is a need to better understand drivers' performance and behaviors when driving with ADS, especially when engaged in NDRTs. This literature review, therefore, aims to understand the state of knowledge on automated vehicle systems and driver distraction. This review was conducted as per PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. Studies found a significant increase in takeover times while engaging in NDRTs and driving with automation active. Studies also discuss a change in driver's visual attention, with more focus given to NDRTs as compared to the front roadway. The concerning effects of increasing reaction times and decreases in visual attention can be mitigated by using interventions and studies have had success in redirecting drivers attention and reorient them to the task of driving. The review, therefore, includes a discussion of ADS and NDRT engagement and its impact on driving behaviors such as take-over times, visual attention, trust, and workload. Implications on driver safety and performance are discussed in light of this synthesis.

Non-driving related tasks' effects on takeover and manual driving behavior in a real driving setting: A differentiation approach based on task switching and modality shifting

Many studies on effects of non-driving related tasks in the context of SAE Level3 automated driving have been conducted in driving simulator settings applying standardized tasks. Thereby internal validity is favored over external validity. To assess the influence of engagement in three natural non-driving related tasks on takeover behavior in the context of SAE Level3 automated driving, we conducted an experiment on a test track with a sample of naïve participants from the general public. We used a Wizard-of-Oz vehicle to simulate a SAE Level 3 traffic jam function in a real driving setting. To measure effects of compatibility between non-driving related tasks and driving task on subsequent takeover behavior and following manual driving behavior, participants played Tetris, watched a documentary film and read a text and typed a summary of it. After approx. 15 min, each non-driving related task was interrupted by a request to intervene. In the manual driving phase after the third takeover, participants encountered a balloon car positioned on their lane which they had to evade. Results show longer takeover times in the film and text condition compared to the Tetris condition. Implications on theory and practice are discussed.

A Systematic Review and Meta-Analysis of Takeover Performance During Conditionally Automated Driving

OBJECTIVE

The aim of this paper was to synthesize the experimental research on factors that affect takeover performance during conditionally automated driving.

BACKGROUND

For conditionally automated driving, the automated driving system (ADS) can handle the entire dynamic driving task but only for limited domains. When the system reaches a limit, the driver is responsible for taking over vehicle control, which may be affected by how much time they are provided to take over, what they were doing prior to the takeover, or the type of information provided to them during the takeover.

METHOD

Out of 8446 articles identified by a systematic literature search, 48 articles containing 51 experiments were included in the meta-analysis. Coded independent variables were time budget, non-driving related task engagement and resource demands, and information support during the takeover. Coded dependent variables were takeover timing and quality measures.

RESULTS

Engaging in non-driving related tasks results in degraded takeover performance, particularly if it has overlapping resource demands with the driving task. Weak evidence suggests takeover performance is impaired with shorter time budgets. Current implementations of information support did not affect takeover performance.

CONCLUSION

Future research and implementation should focus on providing the driver more time to take over while automation is active and should further explore information support.

APPLICATION

The results of the current paper indicate the need for the development and deployment of vehicle-to-everything (V2X) services and driver monitoring.

How drowsiness and distraction can interfere with take-over performance: A systematic and meta-analysis review

Drowsiness and distraction are major factors of road crashes and responsible of>35% of road fatalities. Automated driving could solve or minimize their impact, yet it is also in itself a way to promote them. Previous literature reviews and meta-analysis regarding take-overs during automated driving primarily focused on distraction rather than drowsiness. We thus present a systematic and meta-analysis literature review focused on the effect of distraction and drowsiness on take-over performance. From an initial selection of 1896 articles from databases, we obtained by applying systematic review methodology a total of 58 articles with 42 articles dedicated to distraction and 17 articles related to drowsiness. According to our analysis, we demonstrated that distraction and drowsiness increased the take-over request reaction time (TOR-RT), which could also lead to a reduction of the quality of take-overs. In addition, this longer reaction time was even more important in the case of handheld non-driving related tasks, which is important to consider as phone use is among the most frequent tasks done during automated driving. On a more methodological aspect, we also demonstrated that take-over time budget had a significant effect on TOR-RT. However, it is difficult to estimate to what extend distraction and drowsiness could impact the take-over quality, even if several elements supported safety issues. We underpinned several limits of the current methodologies applied in the study of distraction and drowsiness such as (i) the lack of additional measures related to the take-over quality (e.g., accelerations, collision rate), (ii) the many different methodologies applied to the determination of the TOR-RT (e.g., deactivation by the steering wheel, pedals, button), (iii) the high frequency of take-over requests which can lead to habituation effects, (iv) the lack of control conditions, (v) the fact that the level of drowsiness was relatively low in most studies. We thus highlighted recommendations for a better estimation of the effect of distraction and drowsiness on take-over performance. Further studies should adopt more standardized measures of TOR-RT and additional take-over quality measures, try minimizing the number of take-over requests, and carefully consider the time budget available for the use case since it influences the TOR-RT. Regarding distraction, researchers should consider the impact of tasks requiring handholding items. Concerning drowsiness, further protocols should consider the non-linearity of drowsiness and presence of micro sleeps and favor take-over requests based on drowsiness level protocols rather than on fixed duration protocols.

Effects of non-driving related tasks on readiness to take over control in conditionally automated driving

OBJECTIVE

At conditionally automated driving, the driver can temporarily engage in non-driving related tasks (NDRTs). However, they must safely take over control when the automated driving system reaches its operation limit. Thus, understanding the effects of the NDRTs on driver take-over performance is essential. The present work investigates the effects of various NDRTs on motor readiness in take-over scenarios during conditionally automated driving.

METHODS

Three driving simulator studies were conducted. 48, 49, and 22 participants were recruited in three experiments, respectively. The participants were distracted by different NDRTs (everyday task in Experiment 1, arrow task in Experiment 2, and SuRT in Experiment 3) on a tablet mounted in the vehicle. The everyday task included reading the news and watching a video, and the arrow task included a set of arrow matrices presented to the participants in sequence. The time budgets in Experiment 1 included 3 s, 4 s, and 5 s, and the time budgets in Experiment 2 and 3 included 5 s and 7 s. A take-over request (TOR) warning was issued in the automated driving condition when the participants encountered a broken-down car in front. The participants must regain control of the vehicle with the given time budget. The hands-on time was evaluated, measuring the time from the TOR until the hands touch the steering wheel.

RESULTS

The task (arrow task and SuRT), time budget (5 s and 7 s), and gender did not affect the hands-on time. However, the hands-on time for the drivers with the everyday task was significantly shorter than that for the drivers with the arrow task in the 5 s time budget.

CONCLUSIONS

In conditionally automated driving, the arrow task and SuRT imposed a similar workload on readiness to take over control. Compared to the everyday task, the engagement in the arrow tasks consumed more workload on readiness to take over control.

Towards Mixed-Initiative Human-Robot Interaction: Assessment of Discriminative Physiological and Behavioral Features for Performance Prediction

The design of human–robot interactions is a key challenge to optimize operational performance. A promising approach is to consider mixed-initiative interactions in which the tasks and authority of each human and artificial agents are dynamically defined according to their current abilities. An important issue for the implementation of mixed-initiative systems is to monitor human performance to dynamically drive task allocation between human and artificial agents (i.e., robots). We, therefore, designed an experimental scenario involving missions whereby participants had to cooperate with a robot to fight fires while facing hazards. Two levels of robot automation (manual vs. autonomous) were randomly manipulated to assess their impact on the participants’ performance across missions. Cardiac activity, eye-tracking, and participants’ actions on the user interface were collected. The participants performed differently to an extent that we could identify high and low score mission groups that also exhibited different behavioral, cardiac and ocular patterns. More specifically, our findings indicated that the higher level of automation could be beneficial to low-scoring participants but detrimental to high-scoring ones, and vice versa. In addition, inter-subject single-trial classification results showed that the studied behavioral and physiological features were relevant to predict mission performance. The highest average balanced accuracy (74%) was reached using the features extracted from all input devices. These results suggest that an adaptive HRI driving system, that would aim at maximizing performance, would be capable of analyzing such physiological and behavior markers online to further change the level of automation when it is relevant for the mission purpose.

In a heart beat: Using driver's physiological changes to determine the quality of a takeover in highly automated vehicles

Developing conditionally automated driving systems is on the rise. Vehicles with full longitudinal and latitudinal control will allow drivers to engage in secondary tasks without monitoring the roadway, but users may be required to resume vehicle control to handle critical hazards. The loss of driver's situational awareness increases the potential for accidents. Thus, the automated systems need to estimate the driver's ability to resume control of the driving task. The aim of this study was to assess the physiological behaviour (heart rate and pupil diameter) of drivers. The assessment was performed during two naturalistic secondary tasks. The tasks were the email and the twenty questions task in addition to a control group that did not perform any tasks. The study aimed at finding possible correlations between the driver's physiological data and their responses to a takeover request. A driving simulator study was used to collect data from a total of 33 participants in a repeated measures design to examine the physiological changes during driving and to measure their takeover quality and response time. Secondary tasks induced changes on physiological measures and a small influence on response time. However, there was a strong observed correlation between the physiological measures and response time. Takeover quality in this study was assessed using two new performance measures called PerSpeed and PerAngle. They are identified as the mean percentage change of vehicle's speed and heading angle starting from a take-over request time. Using linear mixed models, there was a strong interaction between task, heart rate and pupil diameter and PerSpeed, PerAngle and response time. This, in turn, provided a measurable understanding of a driver's future responses to the automated system based on the driver's physiological changes to allow better decision making. The present findings of this study emphasised the possibility of building a driver mental state model and prediction system to determine the quality of the driver's responses in a highly automated vehicle. Such results will reduce accidents and enhance the driver's experience in highly automated vehicles.

Human-based dynamics of mental workload in complicated systems

As a dynamic system in which different factors affect human performance via dynamic interactions, mental workload needs a dynamic measure to monitor its factors and evidence in a complicated system, an approach that is lacking in the literature. The present study introduces a system dynamics-based model for designing feedback mechanisms related to the mental workload through literature review and content analysis of the previous studies. A human-based archetype of mental workload was detected from the data collection process. The archetype is presented at various stages, including dynamic theory, behavior over time, leverage points and model verification. The real validation of the dynamic model was confirmed in an urban train simulator. The dynamic model can be used to analyze the long-term behavior of the mental workload. Decision-makers can benefit from the developed archetypes in evaluating the dynamic impact of their decisions on accident prevention in the complicated systems.

Toward Computational Simulations of Behavior During Automated Driving Takeovers: A Review of the Empirical and Modeling Literatures

OBJECTIVE

This article provides a review of empirical studies of automated vehicle takeovers and driver modeling to identify influential factors and their impacts on takeover performance and suggest driver models that can capture them.

BACKGROUND

Significant safety issues remain in automated-to-manual transitions of vehicle control. Developing models and computer simulations of automated vehicle control transitions may help designers mitigate these issues, but only if accurate models are used. Selecting accurate models requires estimating the impact of factors that influence takeovers.

METHOD

Articles describing automated vehicle takeovers or driver modeling research were identified through a systematic approach. Inclusion criteria were used to identify relevant studies and models of braking, steering, and the complete takeover process for further review.

RESULTS

The reviewed studies on automated vehicle takeovers identified several factors that significantly influence takeover time and post-takeover control. Drivers were found to respond similarly between manual emergencies and automated takeovers, albeit with a delay. The findings suggest that existing braking and steering models for manual driving may be applicable to modeling automated vehicle takeovers.

CONCLUSION

Time budget, repeated exposure to takeovers, silent failures, and handheld secondary tasks significantly influence takeover time. These factors in addition to takeover request modality, driving environment, non-handheld secondary tasks, level of automation, trust, fatigue, and alcohol significantly impact post-takeover control. Models that capture these effects through evidence accumulation were identified as promising directions for future work.

APPLICATION

Stakeholders interested in driver behavior during automated vehicle takeovers may use this article to identify starting points for their work.

Noncritical State Transitions During Conditionally Automated Driving on German Freeways: Effects of Non-Driving Related Tasks on Takeover Time and Takeover Quality

OBJECTIVE

This study aimed at investigating the driver's takeover performance when switching from working on different non-driving related tasks (NDRTs) while driving with a conditionally automated driving function (SAE L3), which was simulated by a Wizard of Oz vehicle, to manual vehicle control under naturalistic driving conditions.

BACKGROUND

Conditionally automated driving systems, which are currently close to market introduction, require the user to stay fallback ready. As users will be allowed to engage in more complex NDRTs during the automated drive than when driving manually, the time needed to regain full manual control could likely be increased.

METHOD

Thirty-four users engaged in different everyday NDRTs while driving automatically with a Wizard of Oz vehicle. After approximately either 5 min or 15 min of automated driving, users were requested to take back vehicle control in noncritical situations. The test drive took place in everyday traffic on German freeways in the metropolitan area of Stuttgart.

RESULTS

Particularly tasks that required users to turn away from the central road scene or hold an object in their hands led to increased takeover times. Accordingly, increased variance in the driver's lane position was found shortly after the switch to manual control. However, the drivers rated the takeover situations to be mostly "harmless."

CONCLUSION

Drivers managed to regain control over the vehicle safely, but they needed more time to prepare for the manual takeover when the NDRTs caused motoric workload.

APPLICATION

The timings found in the study can be used to design comfortable and safe takeover concepts for automated vehicles.

From partial and high automation to manual driving: Relationship between non-driving related tasks, drowsiness and take-over performance

BACKGROUND

Until the level of full vehicle automation is reached, users of vehicle automation systems will be required to take over manual control of the vehicle occasionally and stay fallback-ready to some extent during the drive. Both, drowsiness caused by inactivity and the engagement in distracting non-driving related tasks (NDRTs) such as entertainment or office work have been suggested to impair the driver's ability to safely handle these transitions of control. Thus, it is an open question whether engagement in NDRTs will impair or improve take-over performance.

METHOD

In a motion-based driving simulator, 64 participants completed an automated drive that lasted either one or two hours using either a partially or highly automated driving system. In the partially automated driving condition, a warning was issued after several seconds when drivers took both hands off the steering wheel, while the highly automated driving system allowed hands-off driving permanently. Drivers were allowed to bring along their smartphones and to use them during the drive. They engaged in a wide variety of NDRTs such as reading or using social media. At the end of the session, drivers had to react to a sudden lead vehicle braking event. In the partial automation condition, there was no take-over request (TOR) to notify the drivers of the braking vehicle, while in the highly automated condition, the situation happened right after the drivers had deactivated the automation in response to a TOR. The lead time of the TOR was set at 8 s. Driver's level of drowsiness, workload (visual, mental and motoric) from carrying out the NDRT and motivational appeal of the NDRT right before the control transition were video-coded and used to predict the outcome of the braking event (i.e., reaction and system deactivation times, minimal Time-to-collision (TTC) and self-reported criticality) with a multiple regression approach.

RESULTS

In the partial automation condition, reaction times to the braking vehicle and situation criticality as measured by the minimum TTC could be well predicted. Main predictors for increased reaction time were drowsiness and motivational appeal of the NDRT. However, visual and mental demand associated with NDRTs did decrease reaction time, suggesting that the NDRT helped the drivers to maintain alertness during the partially automated drive. Accordingly, drowsiness and motivational appeal of the NDRT increased situation criticality, while cognitive load due to the NDRT decreased it. In the highly automated condition, however, it was not possible to predict system deactivation time (in reaction to the TOR), brake reaction time to the braking vehicle and situation criticality by observed drowsiness and NDRT engagement.

DISCUSSION

The results suggest a relationship between the driver's drowsiness and NDRT engagement in partial automation but not in highly automated driving. Several explanations for this finding are discussed. It could be possible that the lead time of 8 s might have given the drivers enough time to complete the driver state transition process from executing NDRTs to manual driving, putting them in a position to be able to cope with the driving event, while this was not possible in the partial automation condition. Methodological issues that might have led to a non-detection of an effect of drowsiness or NDRT engagement in the highly automated driving condition, such as the sample size and sensitivity of the observer ratings, are also discussed.

Effect of different alcohol levels on take-over performance in conditionally automated driving

Automated driving systems are getting pushed into the consumer market, with varying degrees of automation. Most often the driver's task will consist of being available as a fall-back level when the automation reaches its limits. These so-called take-over situations have attracted a great body of research, focusing on various human factors aspects (e.g., sleepiness) that could undermine the safety of control transitions between automated and manual driving. However, a major source of accidents in manual driving, alcohol consumption, has been a non-issue so far, although a false understanding of the driver's responsibility (i.e., being available as a fallback level) might promote driving under its influence. In this experiment, N = 36 drivers were exposed to different levels of blood alcohol concentrations (BACs: placebo vs. 0.05% vs. 0.08%) in a high fidelity driving simulator, and the effect on take-over time and quality was assessed. The results point out that a 0.08% BAC increases the time needed to re-engage in the driving task and impairs several aspects of longitudinal and lateral vehicle control, whereas 0.05% BAC did only go along with descriptive impairments in fewer parameters.

Psychosocial factors associated with intended use of automated vehicles: A simulated driving study

This study applied the Theory of Planned Behavior (TPB) and the Technology Acceptance Model (TAM) to assess drivers' intended use of automated vehicles (AVs) after undertaking a simulated driving task. In addition, this study explored the potential for trust to account for additional variance to the psychosocial factors in TPB and TAM. Seventy-four participants (51% female) aged between 25 and 64 years (M = 42.8, SD = 12.9) undertook a 20 min simulated experimental drive in which participants experienced periods of automated driving and manual control. A survey task followed. A hierarchical regression analysis revealed that TPB constructs; attitude toward the behavior, subjective norms, and perceived behavioral control, were significant predictors of intentions to use AV. In addition, there was partial support for the test of TAM, with ease of use (but not usefulness) predicting intended use of AV (SAE Level 3). Trust contributed variance to both models beyond TPB or TAM constructs. The findings provide an important insight into factors that might reflect intended use of vehicles that are primarily automated (longitudinal, lateral, and manoeuvre controls) but require and allow drivers to have periods of manual control.