Target cuing in visual search: the effects of conformality and display location on the allocation of visual attention

Beyond the Wizard of Oz: Negative Effects of Imperfect Machine Learning to Examine the Impact of Reliability of Augmented Reality Cues on Visual Search Performance

Despite knowing exactly what an object looks like, searching for it in a person's visual field is a time-consuming and error-prone experience. In Augmented Reality systems, new algorithms are proposed to speed up search time and reduce human errors. However, these algorithms might not always provide 100% accurate visual cues, which might affect users' perceived reliability of the algorithm and, thus, search performance. Here, we examined the detrimental effects of automation bias caused by imperfect cues presented in the Augmented Reality head-mounted display using the YOLOv5 machine learning model. 53 participants in the two groups received either 100% accurate visual cues or 88.9% accurate visual cues. Their performance was compared with the control condition, which did not include any additional cues. The results show how cueing may increase performance and shorten search times. The results also showed that performance with imperfect automation was much worse than perfect automation and that, consistent with automation bias, participants were frequently enticed by incorrect cues.

Helicopter Pilot Performance and Workload in a Following Task in a Degraded Visual Environment

BACKGROUND: In this study, we investigated the impact of a loss of horizon due to atmospheric conditions on flight performance and workload of helicopter pilots during a low-altitude, dynamic flight task in windy conditions at sea. We also examined the potential benefits of a helmet-mounted display (HMD) for this specific task.METHODS: In a fixed-based helicopter simulator, 16 military helicopter pilots were asked to follow a maneuvering go-fast vessel in a good visual environment (GVE) and in a degraded visual environment (DVE). DVE was simulated by fog, obscuring the horizon and reducing contrast. Both visual conditions were performed once with and once without an HMD, which was simulated by projecting head-slaved symbology in the outside visuals. Objective measures included flight performance, control inputs, gaze direction, and relative positioning. Subjective measures included self-ratings on performance, situation awareness, and workload.RESULTS: The results showed that in DVE the pilots perceived higher workload and were flying closer to the go-fast vessel than in GVE. Consequently, they responded with larger control inputs to maneuvers of the vessel. The availability of an HMD hardly improved flight performance but did allow the pilots to focus their attention more outside, significantly improving their situation awareness and reducing workload. These benefits were found in DVE as well as GVE conditions.DISCUSSION: DVE negatively affects workload and flight performance of helicopter pilots in a dynamic, low-altitude following task. An HMD can help improve situation awareness and lower the workload during such a task, irrespective of the visual conditions.Ledegang WD, van der Burg E, Valk PJL, Houben MMJ, Groen EL. Helicopter pilot performance and workload in a following task in a degraded visual environment. Aerosp Med Hum Perform. 2024; 95(1):16-24.

User-defined information sharing for team situation awareness and teamwork

Team members have different roles in various scenarios to maintain situation awareness. A collaborative system should therefore provide appropriate information to the appropriate person at an appropriate time. Considering the mismatch between the designed and actually used information, this paper proposed that users should define what information to share with their team-mates. Thirty-six participants, who formed eighteen teams, used both the traditional and user-defined shared displays to perform failure diagnosis on the context of nuclear power plants. The user-defined shared display exhibited shorter diagnosis time without significant difference in correctness. Information quality, instead of quantity, was positively correlated with team mutual awareness. This study provides empirical evidence that user-defined information sharing is effective at improving operator's diagnosis performance, so the users should be able to tailor the information based on requirements.Practitioner Summary: To support team situation awareness and teamwork, the present study proposed that users should define what information to share with their team-mates. The laboratory experiment shows that user-defined information sharing shortens operator diagnosis time without degrading correctness. Information quality appears more important than information quantity in enhancing team mutual awareness.

Guidance in Cinematic Virtual Reality-Taxonomy, Research Status and Challenges

In Cinematic Virtual Reality (CVR), the viewer of an omnidirectional movie can freely choose the viewing direction when watching a movie. Therefore, traditional techniques in filmmaking for guiding the viewers’ attention cannot be adapted directly to CVR. Practices such as panning or changing the frame are no longer defined by the filmmaker; rather it is the viewer who decides where to look. In some stories, it is necessary to show certain details to the viewer, which should not be missed. At the same time, the freedom of the viewer to look around in the scene should not be destroyed. Therefore, techniques are needed which guide the attention of the spectator to visual information in the scene. Attention guiding also has the potential to improve the general viewing experience, since viewers will be less afraid to miss something when watching an omnidirectional movie where attention-guiding techniques have been applied. In recent years, there has been a lot of research about attention guiding in images, movies, virtual reality, augmented reality and also in CVR. We classify these methods and offer a taxonomy for attention-guiding methods. Discussing the different characteristics, we elaborate the advantages and disadvantages, give recommendations for use cases and apply the taxonomy to several examples of guiding methods.

Effects of Task Difficulty and Display Format on Automation Usage Strategy: A Workload Capacity Analysis

Objective An experiment used workload capacity analysis to quantify automation usage strategy across different task difficulty and display format types in a speeded task. Background Workload capacity measures the efficiency of concurrent information processing and can serve as a gauge of automation usage strategy in speeded decision tasks. The present study used workload capacity analysis to investigate automation usage strategy while information display format and task difficulty were manipulated. Method Subjects performed a speeded judgment task assisted by an automated aid that issued decision cues at varying onset times. Response time distributions were converted to measures of workload capacity. Results Two variants of a workload capacity measure, C_zOR and C_zAND, gave evidence that operators moderated their own decision times both in anticipation of and following the arrival of the aid's diagnosis under difficult task conditions regardless of display format. Conclusion Assistance from an automated decision aid may cause operators to delay their own responses in a speeded decision task, producing joint response time distributions that are slower than optimal. Application Even when it renders its own judgments quickly and with high accuracy, an automated decision aid may slow responses from a user. Automation designers should consider the relative costs and benefits of response accuracy and time when choosing whether and how to implement an automated decision aid.

Target Detection and Identification Performance Using an Automatic Target Detection System

OBJECTIVE

We investigated the effects of automatic target detection (ATD) on the detection and identification performance of soldiers.

BACKGROUND

Prior studies have shown that highlighting targets can aid their detection. We provided soldiers with ATD that was more likely to detect one target identity than another, potentially acting as an implicit identification aid.

METHOD

Twenty-eight soldiers detected and identified simulated human targets in an immersive virtual environment with and without ATD. Task difficulty was manipulated by varying scene illumination (day, night). The ATD identification bias was also manipulated (hostile bias, no bias, and friendly bias). We used signal detection measures to treat the identification results.

RESULTS

ATD presence improved detection performance, especially under high task difficulty (night illumination). Identification sensitivity was greater for cued than uncued targets. The identification decision criterion for cued targets varied with the ATD identification bias but showed a "sluggish beta" effect.

CONCLUSION

ATD helps soldiers detect and identify targets. The effects of biased ATD on identification should be considered with respect to the operational context.

APPLICATION

Less-than-perfectly-reliable ATD is a useful detection aid for dismounted soldiers. Disclosure of known ATD identification bias to the operator may aid the identification process.

From Here to Autonomy

As autonomous and semiautonomous systems are developed for automotive, aviation, cyber, robotics and other applications, the ability of human operators to effectively oversee and interact with them when needed poses a significant challenge. An automation conundrum exists in which as more autonomy is added to a system, and its reliability and robustness increase, the lower the situation awareness of human operators and the less likely that they will be able to take over manual control when needed. The human-autonomy systems oversight model integrates several decades of relevant autonomy research on operator situation awareness, out-of-the-loop performance problems, monitoring, and trust, which are all major challenges underlying the automation conundrum. Key design interventions for improving human performance in interacting with autonomous systems are integrated in the model, including human-automation interface features and central automation interaction paradigms comprising levels of automation, adaptive automation, and granularity of control approaches. Recommendations for the design of human-autonomy interfaces are presented and directions for future research discussed.

Effects of Automated Cueing on Decision Implementation in a Visual Search Task

A visual search paradigm was used to examine the effects of status information automation cueing in a target detection task. Manual and information automation conditions were manipulated with the size of the distractor set. Participants were required to respond to the presence or absence of a target in a time-limited trial. In the information automation condition, status information regarding target presence was presented to the participant. The participants were informed that the information automation was not perfectly reliable. A significant detection performance improvement was observed with the addition of the information automation. This improvement was more marked in the condition with the higher number of distractors. Additionally, detection performance declined when the information automation was invalid, without a corresponding increase in subjective measures of workload or confidence. Implications of the results and future studies are discussed.

Supporting Situation Assessment Through Attention Guidance: A Cost-Benefit and Depth of Processing Analysis

Automated support systems may be useful tools for aiding situation assessment in complex environments such as the military battlefield. These environments are marked by large amounts of information which often must be weighted and integrated into a meaningful judgment or assessment. The present research examines the effects of attention cueing on information integration tasks in static battlefield situations. Sixteen participants completed a resource allocation task for 56 battlefield scenarios (based on perceived threats). For half the trials, an automated system guided their attention to high-threat units. On 2 trials a memory probe was administered to assess the depth of processing of information, and on the final trial an automation failure was presented. Results demonstrated an overall allocation performance advantage for automation but poorer recall for automation-enhanced units. Half of the participants failed to attend to the system failure. Those participants who detected the failure were inferred to have processed the cues more deeply on the memory trials. The costs and benefits of automated cueing are discussed.

Effects of Information Automation and Decision-Aiding Cueing on Action Implementation in a Visual Search Task

A visual search paradigm was used to examine the effects of status information as well as decision-aiding automation in a target detection and processing task. Manual, information automation, and decision-aiding automation conditions were manipulated with the size of the distractor set. Participants were required to respond to the presence or absence of a target in a time-limited trial. In the information automation condition, status information regarding target presence was presented to the participant. The participants were informed that the information automation was not perfectly reliable. A significant detection performance improvement was observed in the information automation condition. This improvement was more marked in the conditions with the higher number of distractors. Additionally, response times were improved when the information automation cue was present. Effects of cue validity and incorrect responses are presented. Implications of the results and future studies are discussed.

Helmet Mounted Display Layouts

Head mounted displays (HMDs) are being explored as an alternative means of displaying relevant information to dismounted operators. The goal of this project was to examine different visual display concepts and evaluate participant’s attention allocation to information presented on their HMD. Additionally, their ability to detect potential threats in the environment was also evaluated. This information will help revamp the design of information displays for HMDs. The task in this study required participants to monitor their HMD for critical alerts and respond accordingly while also making shoot/no shoot decisions to threats in their environment. We hypothesized that as information is presented in different layouts on the HMD, it will reduce the participants’ ability to detect real world events. Accuracy of the shoot/no shoot decisions was collected along with accuracy of detection of information on the HMD. We found that shooting performance was not affected between the three HMD layouts however information detected on the HMD was worst when all information was in the center of the HMD. The data from this study will be used to help develop intelligent visual displays used by Battlefield Airmen to accomplish their mission.

Workload Capacity: A Response Time-Based Measure of Automation Dependence

OBJECTIVE

An experiment used the workload capacity measure C(t) to quantify the processing efficiency of human-automation teams and identify operators' automation usage strategies in a speeded decision task.

BACKGROUND

Although response accuracy rates and related measures are often used to measure the influence of an automated decision aid on human performance, aids can also influence response speed. Mean response times (RTs), however, conflate the influence of the human operator and the automated aid on team performance and may mask changes in the operator's performance strategy under aided conditions. The present study used a measure of parallel processing efficiency, or workload capacity, derived from empirical RT distributions as a novel gauge of human-automation performance and automation dependence in a speeded task.

METHOD

Participants performed a speeded probabilistic decision task with and without the assistance of an automated aid. RT distributions were used to calculate two variants of a workload capacity measure, COR(t) and CAND(t).

RESULTS

Capacity measures gave evidence that a diagnosis from the automated aid speeded human participants' responses, and that participants did not moderate their own decision times in anticipation of diagnoses from the aid.

CONCLUSION AND APPLICATION

Workload capacity provides a sensitive and informative measure of human-automation performance and operators' automation dependence in speeded tasks.

Inattentional blindness increased with augmented reality surgical navigation

BACKGROUND

Augmented reality (AR) surgical navigation systems, designed to increase accuracy and efficiency, have been shown to negatively impact on attention. We wished to assess the effect "head-up" AR displays have on attention, efficiency, and accuracy, while performing a surgical task, compared with the same information being presented on a submonitor (SM).

METHODS

Fifty experienced otolaryngology surgeons (n = 42) and senior otolaryngology trainees (n = 8) performed an endoscopic surgical navigation exercise on a predissected cadaveric model. Computed tomography-generated anatomic contours were fused with the endoscopic image to provide an AR view. Subjects were randomized to perform the task with a standard endoscopic monitor with the AR navigation displayed on an SM or with AR as a single display. Accuracy, task completion time, and the recognition of unexpected findings (a foreign body and a critical complication) were recorded.

RESULTS

Recognition of the foreign body was significantly better in the SM group (15/25 [60%]) compared with the AR alone group (8/25 [32%]; p = 0.02). There was no significant difference in task completion time (p = 0.83) or accuracy (p = 0.78) between the two groups.

CONCLUSION

Providing identical surgical navigation on a SM, rather than on a single head-up display, reduced the level of inattentional blindness as measured by detection of unexpected findings. These gains were achieved without any measurable impact on efficiency or accuracy. AR displays may distract the user and we caution injudicious adoption of this technology for medical procedures.

Display clutter: a review of definitions and measurement techniques

OBJECTIVE

We aimed to synthesize the literature on display clutter by reviewing definitions and measurement techniques and to develop a comprehensive, ergonomics-oriented perspective on clutter. We provide guidance for the selection of measurement approaches that can capture the various aspects and effects of clutter on visual search in particular.

BACKGROUND

There is agreement that clutter may represent a significant problem to operators. The challenge is to determine the ideal middle ground between excessive data and insufficient information. However, definitions of clutter vary widely, which explains the range of measurement approaches, including image-processing algorithms, performance evaluation, subjective evaluation, and eye tracking. It is important to understand the affordances of each technique and provide guidance for their use.

METHOD

We provide a systematic review of clutter definitions and develop a performance-oriented perspective for ergonomics research. Next, we present a critical overview of clutter measurement approaches. The benefits and limitations of each technique are detailed, and recommendations for best practice are provided.

RESULTS

From an ergonomics perspective, clutter matters to the extent that it affects performance, particularly, visual search. A combination of measurement techniques can be employed in order to assess the performance costs stemming from the multiple aspects of clutter.

CONCLUSION

Display clutter is a multifaceted construct that is a problem when it results in performance and attentional costs. The assessment of these costs is critical and requires the use of appropriate and complementary techniques.

APPLICATION

It is important that the performance costs of clutter are detected reliably to be able to better support attention management.

Validating the random search model for two targets of different difficulty

A random visual search model was fitted to 1,788 search times obtained from a nonidentical double-target search task. 30 Hong Kong Chinese (13 men, 17 women) ages 18 to 33 years (M = 23, SD = 6.8) took part in the experiment voluntarily. The overall adequacy and prediction accuracy of the model for various search time parameters (mean and median search times and response times) for both individual and pooled data show that search strategy may reasonably be inferred from search time distributions. The results also suggested the general applicability of the random search model for describing the search behavior of a large number of participants performing the type of search used here, as well as the practical feasibility of its application for determination of stopping policy for optimization of an inspection system design. Although the data generally conformed to the model the search for the more difficult target was faster than expected. The more difficult target was usually detected after the easier target and it is suggested that some degree of memory-guided searching may have been used for the second target. Some abnormally long search times were observed and it is possible that these might have been due to the characteristics of visual lobes, nonoptimum interfixation distances and inappropriate overlapping of lobes, as has been previously reported.

A visual ergonomic evaluation of different screen types and screen technologies with respect to discrimination performance

An increasing demand to work with electronic displays and to use mobile computers emphasises the need to compare visual performance while working with different screen types. In the present study, a cathode ray tube (CRT) was compared to an external liquid crystal display (LCD) and a Notebook-LCD. The influence of screen type and viewing angle on discrimination performance was studied. Physical measurements revealed that luminance and contrast values change with varying viewing angles (anisotropy). This is most pronounced in Notebook-LCDs, followed by external LCDs and CRTs. Performance data showed that LCD's anisotropy has negative impacts on completing time critical visual tasks. The best results were achieved when a CRT was used. The largest deterioration of performance resulted when participants worked with a Notebook-LCD. When it is necessary to react quickly and accurately, LCD screens have disadvantages. The anisotropy of LCD-TFTs is therefore considered to be as a limiting factor deteriorating visual performance.

Spatial Mental Representation: Implications for Navigation System Design

Similarities exist in how people process and represent spatial information and in the factors that contribute to disorientation, whether one is moving through airspace, on the ground, or surgically within the body. As such, design principles for presenting spatial information should bear similarities across these domains but also be somewhat specific to each. In this chapter, we review research in spatial cognition and its application to navigation system design for within-vehicle, aviation, and endoscopic navigation systems. Taken together, the research suggests three general principles for navigation system design consideration. First, multimedia displays should present spatial information visually and action and description information verbally. Second, display organizations should meet users' dynamic navigational goals. Third, navigation systems should be adaptable to users' spatial information preferences. Designers of adaptive navigation display technologies can maximize the effectiveness of those technologies by appealing to the basic spatial cognition processes employed by all users while conforming to user's domain-specific requirements.

Humans: still vital after all these years of automation

OBJECTIVE

The authors discuss empirical studies of human-automation interaction and their implications for automation design.

BACKGROUND

Automation is prevalent in safety-critical systems and increasingly in everyday life. Many studies of human performance in automated systems have been conducted over the past 30 years.

METHODS

Developments in three areas are examined: levels and stages of automation, reliance on and compliance with automation, and adaptive automation.

RESULTS

Automation applied to information analysis or decision-making functions leads to differential system performance benefits and costs that must be considered in choosing appropriate levels and stages of automation. Human user dependence on automated alerts and advisories reflects two components of operator trust, reliance and compliance, which are in turn determined by the threshold designers use to balance automation misses and false alarms. Finally, adaptive automation can provide additional benefits in balancing workload and maintaining the user's situation awareness, although more research is required to identify when adaptation should be user controlled or system driven.

CONCLUSIONS

The past three decades of empirical research on humans and automation has provided a strong science base that can be used to guide the design of automated systems.

APPLICATION

This research can be applied to most current and future automated systems.

New alternative methods of analyzing human behavior in cued target acquisition

Target acquisition tasks in natural environments are often augmented by cuing systems that advise human observers during the decision process. With present technological limitations, cuing systems are imperfect, so the question arises whether cuing aids should be implemented under all conditions. We examined target acquisition performance under different levels of task complexity and cuing system reliability. We introduce here two new methods to help define observer behavior trends in cued target acquisition: a quantitative measure of observer search behavior in a temporal sense and a measure of the extent of observer reliance on the cue. We found that observer reliance on the cue correlated with task difficulty and the perceived reliability of the cue. Cuing was generally helpful in complex tasks, whereas cuing reduced performance in easy tasks. Consequently, cuing systems should be implemented only when the task is difficult enough to warrant the intrusion of a cue into the task. Actual or potential applications of this research include the design and implementation of imperfect automated aids dealing with augmented reality.

Head up versus head down: the costs of imprecision, unreliability, and visual clutter on cue effectiveness for display signaling

We conducted 2 experiments to investigate the clutter-scan trade-off between the cost of increasing clutter by overlaying complex information onto the forward field of view using a helmet-mounted display (HMD) and the cost of scanning when presenting this information on a handheld display. In the first experiment, this trade-off was examined in terms of the spatial accuracy of target cuing data in a relatively sparse display; in the second, the spatial accuracy of the cue was varied more radically in an information-rich display. Participants were asked to detect and identify targets hidden in the far domain while performing a monitoring task in the near domain using either an HMD or a handheld display. The results revealed that on a sparse display, the reduced scanning from the HMD presentation of cuing out-weighed the costs of clutter for cued targets, regardless of cue precision, but no benefit was found for uncued targets. When the HMD displayed task-irrelevant information, however, target detection was hindered by the extraneous clutter in the forward field of view relative to the handheld display condition, and this cost of clutter increased as the amount of data that needed to be inspected increased. Potential applications of this research include the development of design considerations for head-up displays for aviation and military applications.

Display signaling in augmented reality: effects of cue reliability and image realism on attention allocation and trust calibration

This experiment seeks to examine the relationships among three advanced technology features (presentation of target cuing, reliability of target cuing, and level of image reality and the attention and trust given to that information). The participants were 16 military personnel who piloted an unmanned air vehicle and searched for targets camouflaged in terrain, which was presented at two levels of image realism. Cuing was available for some targets, and the reliability of this information was manipulated at two levels (100% and 75%). The results showed that the presence of cuing aided target detection for expected targets but drew attention away from the presence of unexpected targets. Cuing benefits and attentional tunneling were both reduced when cuing became less reliable. Increasing image realism was compelling but increased reliance on the cuing information when those data were reliable. Potential applications include a cost-benefit analysis of how trust modulates attention in the use of automated target recognition systems and the extent to which increased realism may influence this trust.

A model for types and levels of human interaction with automation

Technical developments in computer hardware and software now make it possible to introduce automation into virtually all aspects of human-machine systems. Given these technical capabilities, which system functions should be automated and to what extent? We outline a model for types and levels of automation that provides a framework and an objective basis for making such choices. Appropriate selection is important because automation does not merely supplant but changes human activity and can impose new coordination demands on the human operator. We propose that automation can be applied to four broad classes of functions: 1) information acquisition; 2) information analysis; 3) decision and action selection; and 4) action implementation. Within each of these types, automation can be applied across a continuum of levels from low to high, i.e., from fully manual to fully automatic. A particular system can involve automation of all four types at different levels. The human performance consequences of particular types and levels of automation constitute primary evaluative criteria for automation design using our model. Secondary evaluative criteria include automation reliability and the costs of decision/action consequences, among others. Examples of recommended types and levels of automation are provided to illustrate the application of the model to automation design.

Frames of reference for the display of battlefield information: judgment-display dependencies

In 2 experiments, U.S. Army soldiers viewed computer-generated displays that presented battlefield information from 3 different frames of reference: a 2D plan view display (with contour lines), a 3D exocentric perspective display, and an interactive 3D immersed display. In Experiment 1, soldiers made geographical judgments. The results suggested that both 3D displays suffered from ambiguity of distance estimates but that the 3D immersed display was most accurate for judging whether a location is directly visible from another. In Experiment 2, the 3D exocentric display was compared with a 3D immersed view, coupled with a small 2D inset map, in a more continuous battlefield scenario in which judgments of enemy activity were made. The findings of 3D ambiguity were replicated from Experiment 1. The accuracy of judgments of enemy activity suffered with the immersed display when information necessary to answer correctly did not appear in the initial forward view and required panning to acquire, reflecting the cognitive demands of integration across different views. This display also hindered soldiers' ability to report changes in enemy activity from one scene to the next. The results of this research will help to provide guidelines for the appropriate choice of computer display technology to assist in designing battlefield visualization aids. Caution should be exercised in choosing immersive viewpoints.