Low target prevalence is a stubborn source of errors in visual search tasks

Great expectations: minor differences in initial instructions have a major impact on visual search in the absence of feedback

Professions such as radiology and aviation security screening that rely on visual search-the act of looking for targets among distractors-often cannot provide operators immediate feedback, which can create situations where performance may be largely driven by the searchers' own expectations. For example, if searchers do not expect relatively hard-to-spot targets to be present in a given search, they may find easy-to-spot targets but systematically quit searching before finding more difficult ones. Without feedback, searchers can create self-fulfilling prophecies where they incorrectly reinforce initial biases (e.g., first assuming and then, perhaps wrongly, concluding hard-to-spot targets are rare). In the current study, two groups of searchers completed an identical visual search task but with just a single difference in their initial task instructions before the experiment started; those in the "high-expectation" condition were told that each trial could have one or two targets present (i.e., correctly implying no target-absent trials) and those in the "low-expectation" condition were told that each trial would have up to two targets (i.e., incorrectly implying there could be target-absent trials). Compared to the high-expectation group, the low-expectation group had a lower hit rate, lower false alarm rate and quit trials more quickly, consistent with a lower quitting threshold (i.e., performing less exhaustive searches) and a potentially higher target-present decision criterion. The expectation effect was present from the start and remained across the experiment-despite exposure to the same true distribution of targets, the groups' performances remained divergent, primarily driven by the different subjective experiences caused by each groups' self-fulfilling prophecies. The effects were limited to the single-targets trials, which provides insights into the mechanisms affected by the initial expectations set by the instructions. In sum, initial expectations can have dramatic influences-searchers who do not expect to find a target, are less likely to find a target as they are more likely to quit searching earlier.

Visual Perception: How Better Imaging Can Make Things Worse

Visual search is critical to daily life and to socially important tasks - from cancer screening to airport security. New research shows how a technological advancement can interact with the human visual system to improve search for one type of target while making matters worse for another. Part of the problem is that we are surprisingly bad at knowing where we have looked.

Eye movements reflect expertise development in hybrid search

Domain-specific expertise changes the way people perceive, process, and remember information from that domain. This is often observed in visual domains involving skilled searches, such as athletics referees, or professional visual searchers (e.g., security and medical screeners). Although existing research has compared expert to novice performance in visual search, little work has directly documented how accumulating experiences change behavior. A longitudinal approach to studying visual search performance may permit a finer-grained understanding of experience-dependent changes in visual scanning, and the extent to which various cognitive processes are affected by experience. In this study, participants acquired experience by taking part in many experimental sessions over the course of an academic semester. Searchers looked for 20 categories of targets simultaneously (which appeared with unequal frequency), in displays with 0-3 targets present, while having their eye movements recorded. With experience, accuracy increased and response times decreased. Fixation probabilities and durations decreased with increasing experience, but saccade amplitudes and visual span increased. These findings suggest that the behavioral benefits endowed by expertise emerge from oculomotor behaviors that reflect enhanced reliance on memory to guide attention and the ability to process more of the visual field within individual fixations.

A Design Space of Vision Science Methods for Visualization Research

A growing number of efforts aim to understand what people see when using a visualization. These efforts provide scientific grounding to complement design intuitions, leading to more effective visualization practice. However, published visualization research currently reflects a limited set of available methods for understanding how people process visualized data. Alternative methods from vision science offer a rich suite of tools for understanding visualizations, but no curated collection of these methods exists in either perception or visualization research. We introduce a design space of experimental methods for empirically investigating the perceptual processes involved with viewing data visualizations to ultimately inform visualization design guidelines. This paper provides a shared lexicon for facilitating experimental visualization research. We discuss popular experimental paradigms, adjustment types, response types, and dependent measures used in vision science research, rooting each in visualization examples. We then discuss the advantages and limitations of each technique. Researchers can use this design space to create innovative studies and progress scientific understanding of design choices and evaluations in visualization. We highlight a history of collaborative success between visualization and vision science research and advocate for a deeper relationship between the two fields that can elaborate on and extend the methodological design space for understanding visualization and vision.

A device for surveillance of vascular access sites for bleeding: results from a clinical evaluation trial

Post-procedural wound haemorrhage is a potentially life-threatening complication. For haemodialysis patients, bleeding is often encountered after vascular access procedures and fatal episodes have been reported. Visual monitoring for bleeding is manpower intensive and bleeding episodes may still be missed between inspections. A device, Blood WArning Technology with Continuous Haemoglobin sensor (BWATCH), was developed to detect bleeding from wounds. This a prospective, observational clinical trial on patients who have had a dialysis catheter inserted or removed. The battery-powered, disc-shaped device (43 mm diameter, 12 mm height) was placed over the dressing for at least six hours. The device detects reflected light with characteristics specific for haemoglobin and an alarm would be triggered if bleeding occurs. There were 250 participants (177 post-insertion, 73 post-removal) and 36 episodes of bleeding occurred. The device alarm was triggered in all instances but there were also 9 false alarms. Specificity was 95.8%, false positive rate was 4.2% and positive predictive value was 80.0%. Sensitivity and negative predictive value were 100% but detection failure may still occur due to improper application or device maintenance. The use of technological aids for monitoring improves patient safety and may reduce demand on manpower.

It's not always about me: The effects of prior beliefs and stimulus prevalence on self-other prioritisation

Although self-relevance is widely acknowledged to enhance stimulus processing, the exclusivity of this effect remains open to question. In particular, in commonly adopted experimental paradigms, the prioritisation of self-relevant (vs. other-relevant) material may reflect the operation of a task-specific strategy rather than an obligatory facet of social-cognitive functioning. By changing basic aspects of the decisional context, it may therefore be possible to generate stimulus-prioritisation effects for targets other than the self. Based on the demonstration that ownership facilitates object categorisation (i.e., self-ownership effect), here we showed that stimulus prioritisation is sensitive to prior expectations about the prevalence of forthcoming objects (owned-by-self vs. owned-by-friend) and whether these beliefs are supported during the task. Under conditions of stimulus uncertainty (i.e., no prior beliefs), replicating previous research, objects were classified more rapidly when owned-by-self compared with owned-by-friend (Experiment 1). When, however, the frequency of stimulus presentation either confirmed (Experiment 2) or disconfirmed (Experiment 3) prior expectations, stimulus prioritisation was observed for the most prevalent objects regardless of their owner. A hierarchical drift diffusion model (HDDM) analysis further revealed that decisional bias was underpinned by differences in the evidential requirements of response generation. These findings underscore the flexibility of ownership effects (i.e., stimulus prioritisation) during object processing.

Visual search errors are persistent in a laboratory analog of the incidental finding problem

When radiologists search for a specific target (e.g., lung cancer), they are also asked to report any other clinically significant “incidental findings” (e.g., pneumonia). These incidental findings are missed at an undesirably high rate. In an effort to understand and reduce these errors, Wolfe et al. (Cognitive Research: Principles and Implications 2:35, 2017) developed “mixed hybrid search” as a model system for incidental findings. In this task, non-expert observers memorize six targets: half of these targets are specific images (analogous to the suspected diagnosis in the clinical task). The other half are broader, categorically defined targets, like “animals” or “cars” (analogous to the less well-specified incidental findings). In subsequent search through displays for any instances of any of the targets, observers miss about one third of the categorical targets, mimicking the incidental finding problem. In the present paper, we attempted to reduce the number of errors in the mixed hybrid search task with the goal of finding methods that could be deployed in a clinical setting. In Experiments 1a and 1b, we reminded observers about the categorical targets by inserting non-search trials in which categorical targets were clearly marked. In Experiment 2, observers responded twice on each trial: once to confirm the presence or absence of the specific targets, and once to confirm the presence or absence of the categorical targets. In Experiment 3, observers were required to confirm the presence or absence of every target on every trial using a checklist procedure. Only Experiment 3 produced a marked decline in categorical target errors, but at the cost of a substantial increase in response time.

Exaggerated prevalence effect with the explicit prevalence information: The description-experience gap in visual search

Despite the increasing focus on target prevalence in visual search research, few papers have thoroughly examined the effect of how target prevalence is communicated. Findings in the judgment and decision-making literature have demonstrated that people behave differently depending on whether probabilistic information is made explicit or learned through experience, hence there is potential for a similar difference when communicating prevalence in visual search. Our current research examined how visual search changes depending on whether the target prevalence information was explicitly given to observers or they learned the prevalence through experience with additional manipulations of target reward and salience. We found that when the target prevalence was low, learning prevalence from experience resulted in more target-present responses and longer search times before quitting compared to when observers were explicitly informed of the target probability. The discrepancy narrowed with increased prevalence and reversed in the high target prevalence condition. Eye-tracking results indicated that search with experience consistently resulted in longer fixation durations, with the largest difference in low-prevalence conditions. Longer search time was primarily due to observers re-visited more items. Our work addressed the importance of exploring influences brought by probability communication in future prevalence visual search studies.

Cognitive and Human Factors in Expert Decision Making: Six Fallacies and the Eight Sources of Bias

Fallacies about the nature of biases have shadowed a proper cognitive understanding of biases and their sources, which in turn lead to ways that minimize their impact. Six such fallacies are presented: it is an ethical issue, only applies to "bad apples", experts are impartial and immune, technology eliminates bias, blind spot, and the illusion of control. Then, eight sources of bias are discussed and conceptualized within three categories: (A) factors that relate to the specific case and analysis, which include the data, reference materials, and contextual information, (B) factors that relate to the specific person doing the analysis, which include past experience base rates, organizational factors, education and training, and personal factors, and lastly, (C) cognitive architecture and human nature that impacts all of us. These factors can impact what the data are (e.g., how data are sampled and collected, or what is considered as noise and therefore disregarded), the actual results (e.g., decisions on testing strategies, how analysis is conducted, and when to stop testing), and the conclusions (e.g., interpretation of the results). The paper concludes with specific measures that can minimize these biases.

The confirmation and prevalence biases in visual search reflect separate underlying processes

Research by Rajsic, Wilson, and Pratt (2015, 2017) suggests that people are biased to use a target-confirming strategy when performing simple visual search. In 3 experiments, we sought to determine whether another stubborn phenomenon in visual search, the low-prevalence effect (Wolfe, Horowitz, & Kenner, 2005), would modulate this confirmatory bias. We varied the reliability of the initial cue: For some people, targets usually occurred in the cued color (high prevalence). For others, targets rarely matched the cues (low prevalence). High cue-target prevalence exacerbated the confirmation bias, indexed via search response times (RTs) and eye-tracking measures. Surprisingly, given low cue-target prevalence, people remained biased to examine cue-colored letters, even though cue-colored targets were exceedingly rare. At the same time, people were more fluent at detecting the more common, cue-mismatching targets. The findings suggest that attention is guided to "confirm" the more available cued target template, but prevalence learning over time determines how fluently objects are perceptually appreciated. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

You shall not pass: how facial variability and feedback affect the detection of low-prevalence fake IDs

In many real-world settings, individuals rarely present another person's ID, which increases the likelihood that a screener will fail to detect it. Three experiments examined how within-person variability (i.e., differences between two images of the same person) and feedback may have influenced criterion shifting, thought to be one of the sources of the low-prevalence effect (LPE). Participants made identity judgments of a target face and an ID under either high, medium, or low mismatch prevalence. Feedback appeared after every trial, only error trials, or no trials. Experiment 1 used two controlled images taken on the same day. Experiment 2 used two controlled images taken at least 6 months apart. Experiment 3 used one controlled and one ambient image taken at least 1 year apart. Importantly, receiver operating characteristic curves revealed that feedback and greater within-person variability exacerbated the LPE by affecting both criterion and discriminability. These results carry implications for many real-world settings, such as border crossings and airports, where identity screening plays a major role in securing public safety.

Redundancy gain in visual search of simulated X-ray images

Cancer diagnosis frequently relies on the interpretation of medical images such as chest X-rays and mammography. This process is error prone; misdiagnoses can reach a rate of 15% or higher. Of particular interest are false negatives-tumors that are present but missed. Previous research has identified several perceptual and attentional problems underlying inaccurate perception of these images. But how might these problems be reduced? The psychological literature has shown that presenting multiple, duplicate images can improve performance. Here we explored whether redundant image presentation can improve target detection in simulated X-ray images, by presenting four identical or similar images concurrently. Displays with redundant images, including duplicates of the same image, showed reduced false-negative rates, compared with displays with a single image. This effect held both when the target's prevalence rate was high and when it was low. Eye tracking showed that fixating on two or more images in the redundant condition speeded target detection and prolonged search, and that the latter effect was the key to reducing false negatives. The redundancy gain may result from both perceptual enhancement and an increase in the search quitting threshold.

Serial dependence in a simulated clinical visual search task

In everyday life, we continuously search for and classify objects in the environment around us. This kind of visual search is extremely important when performed by radiologists in cancer image interpretation and officers in airport security screening. During these tasks, observers often examine large numbers of uncorrelated images (tumor x-rays, checkpoint x-rays, etc.) one after another. An underlying assumption of such tasks is that search and recognition are independent of our past experience. Here, we simulated a visual search task reminiscent of medical image search and found that shape classification performance was strongly impaired by recent visual experience, biasing classification errors 7% more towards the previous image content. This perceptual attraction exhibited the three main tuning characteristics of Continuity Fields: serial dependence extended over 12 seconds back in time (temporal tuning), it occurred only between similar tumor-like shapes (feature tuning), and only within a limited spatial region (spatial tuning). Taken together, these results demonstrate that serial dependence influences shape perception and occurs in visual search tasks. They also raise the possibility of a detrimental impact of serial dependence in clinical and practically relevant settings, such as medical image perception.

Distracting Objects Induce Early Quitting in Visual Search

Task-irrelevant objects can sometimes capture attention and increase the time it takes an observer to find a target. However, less is known about how these distractors impact visual search strategies. Here, I found that salient distractors reduced rather than increased response times on target-absent trials (Experiment 1; N = 200). Combined with higher error rates on target-present trials, these results indicate that distractors can induce observers to quit search earlier than they otherwise would. These effects were replicated when target prevalence was low (Experiment 2; N = 200) and with different stimuli that elicited shallower search slopes (Experiment 3; N = 75). These results demonstrate that salient distractors can produce at least two consequences in visual search: They can capture attention, and they can cause observers to quit searching early. This novel finding has implications both for understanding visual attention and for examining distraction in real-world domains where targets are often absent, such as medical image screening.

Spotting rare items makes the brain "blink" harder: Evidence from pupillometry

In many visual search tasks (e.g., cancer screening, airport baggage inspections), the most serious search targets occur infrequently. As an ironic side effect, when observers finally encounter important objects (e.g., a weapon in baggage), they often fail to notice them, a phenomenon known as the low-prevalence effect (LPE). Although many studies have investigated LPE search errors, we investigated the attentional consequences of successful rare target detection. Using an attentional blink paradigm, we manipulated how often observers encountered the first serial target (T1), then measured its effects on their ability to detect a following target (T2). Across two experiments, we show that the LPE is more than just an inflated miss rate: When observers successfully detected rare targets, they were less likely to spot subsequent targets. Using pupillometry to index locus-coeruleus (LC) mediated attentional engagement, Experiment 2 confirmed that an LC refractory period mediates the attentional blink (`Nieuwenhuis, Gilzenrat, Holmes, & Cohen, 2005, Journal of Experimental Psychology: General, 134[3], 291-307), and that these effects emerge relatively quickly following T1 onset. Moreover, in both behavioral and pupil analyses, we found that detecting low-prevalence targets exacerbates the LC refractory period. Consequences for theories of the LPE are discussed.

Face search in CCTV surveillance

Background

We present a series of experiments on visual search in a highly complex environment, security closed-circuit television (CCTV). Using real surveillance footage from a large city transport hub, we ask viewers to search for target individuals. Search targets are presented in a number of ways, using naturally occurring images including their passports and photo ID, social media and custody images/videos. Our aim is to establish general principles for search efficiency within this realistic context.

Results

Across four studies we find that providing multiple photos of the search target consistently improves performance. Three different photos of the target, taken at different times, give substantial performance improvements by comparison to a single target. By contrast, providing targets in moving videos or with biographical context does not lead to improvements in search accuracy.

Conclusions

We discuss the multiple-image advantage in relation to a growing understanding of the importance of within-person variability in face recognition.

Electronic supplementary material

The online version of this article (10.1186/s41235-019-0193-0) contains supplementary material, which is available to authorized users.

Visual search in breast imaging

Breast cancer is the most common cancer among females worldwide and large volumes of breast images are produced and interpreted annually. As long as radiologists interpret these images, the diagnostic accuracy will be limited by human factors and both false-positive and false-negative errors might occur. By understanding visual search in breast images, we may be able to identify causes of diagnostic errors, find ways to reduce them, and also provide a better education to radiology residents. Many visual search studies in breast radiology have been devoted to mammography. These studies showed that 70% of missed lesions on mammograms attract radiologists' visual attention and that a plethora of different reasons, such as satisfaction of search, incorrect background sampling, and incorrect first impression can cause diagnostic errors in the interpretation of mammograms. Recently, highly accurate tools, which rely on both eye-tracking data and the content of the mammogram, have been proposed to provide feedback to the radiologists. Improving these tools and determining the optimal pathway to integrate them in the radiology workflow could be a possible line of future research. Moreover, in the past few years deep learning has led to improving diagnostic accuracy of computerized diagnostic tools and visual search studies will be required to understand how radiologists interact with the prompts from these tools, and to identify the best way to utilize them. Visual search in other breast imaging modalities, such as breast ultrasound and digital breast tomosynthesis, have so far received less attention, probably due to associated complexities of eye-tracking monitoring and analysing the data. For example, in digital breast tomosynthesis, scrolling through the image results in longer trials, adds a new factor to the study's complexity and makes calculation of gaze parameters more difficult. However, considering the wide utilization of three-dimensional imaging modalities, more visual search studies involving reading stack-view examinations are required in the future. To conclude, in the past few decades visual search studies provided extensive understanding about underlying reasons for diagnostic errors in breast radiology and characterized differences between experts' and novices' visual search patterns. Further visual search studies are required to investigate radiologists' interaction with relatively newer imaging modalities and artificial intelligence tools.

Detection measures for visual inspection of X-ray images of passenger baggage

In visual inspection tasks, such as airport security and medical screening, researchers often use the detection measures d' or A' to analyze detection performance independent of response tendency. However, recent studies that manipulated the frequency of targets (target prevalence) indicate that da with a slope parameter of 0.6 is more valid for such tasks than d' or A'. We investigated the validity of detection measures (d', A', and da) using two experiments. In the first experiment, 31 security officers completed a simulated X-ray baggage inspection task while response tendency was manipulated directly through instruction. The participants knew half of the prohibited items used in the study from training, whereas the other half were novel, thereby establishing two levels of task difficulty. The results demonstrated that for both levels, d' and A' decreased when the criterion became more liberal, whereas da with a slope parameter of 0.6 remained constant. Eye-tracking data indicated that manipulating response tendency affected the decision component of the inspection task rather than search errors. In the second experiment, 124 security officers completed another simulated X-ray baggage inspection task. Receiver operating characteristic (ROC) curves based on confidence ratings provided further support for da, and the estimated slope parameter was 0.5. Consistent with previous findings, our results imply that d' and A' are not valid measures of detection performance in X-ray image inspection. We recommend always calculating da with a slope parameter of 0.5 in addition to d' to avoid potentially wrong conclusions if ROC curves are not available.

"Satisfaction" in search: Individuals' own search expectations predict their errors in multiple-target search

When people search for multiple targets in a display, finding one target hinders their ability to find additional targets. These errors were originally proposed to stem from a "satisfaction" with finding a first target that leads people to prematurely stop searching. However, empirical evidence for this premise has been elusive, prompting consideration of other theories. We returned to the satisfaction proposal and assessed whether people generate expectations regarding the likelihood of multiple targets that lead to search biases that, in turn, predict the rates at which additional targets are missed. Participants searched for one or two targets among distractors. To measure accuracy, most trials allowed search to progress to completion. The remaining trials terminated when participants had found their first target. In these cases, participants guessed whether an additional (unfound) target was present. The time needed to find a first target was inversely related to the searchers' expectations that a second target would be present. These expectations underestimated objective reality, and the strength of an individual's one-target bias was directly related to his or her likelihood of missing subsequent targets. Thus, people's expectations-based on their own behavior-likely impacted search performance, providing a novel mechanistic explanation for the previously posited "satisfaction-of-search" errors.

The hazards of perception: evaluating a change blindness demonstration within a real-world driver education course

Overconfidence in one's driving ability can lead to risky decision-making and may therefore increase the accident risk. When educating people about the risks of their driving behavior, it is all too easy for individuals to assume that the message is not meant for them and so can be ignored. In this study we developed and assessed the effect of a road safety demonstration based around the phenomenon of change blindness within a real-world Driver Awareness Course. We collected quantitative and qualitative data to evaluate the effectiveness of the demonstration in both a police-led environment (Experiment 1) and a laboratory environment (Experiment 2). We also compared the change blindness intervention to two control tasks. The results showed that participants' self-reported ability to spot important visual changes was reduced after the change blindness demonstration in both experiments, but was not reduced after participation in the control tasks of Experiment 2. Furthermore, participants described the change blindness demonstrations positively and would recommend that they were shown more widely.

Traditional Visual Search vs. X-Ray Image Inspection in Students and Professionals: Are the Same Visual-Cognitive Abilities Needed?

The act of looking for targets amongst an array of distractors is a cognitive task that has been studied extensively over many decades and has many real-world applications. Research shows that specific visual-cognitive abilities are needed to efficiently and effectively locate a target among distractors. It is, however, not always clear whether the results from traditional, simplified visual search tasks conducted by students will extrapolate to an applied inspection tasks in which professionals search for targets that are more complex, ambiguous, and less salient. More concretely, there are several potential challenges when interpreting traditional visual search results in terms of their implications for the X-ray image inspection task. In this study, we tested whether a theoretical intelligence model with known facets of visual-cognitive abilities (visual processing Gv, short-term memory Gsm, and processing speed Gs) can predict performance in both a traditional visual search task and an X-ray image inspection task in both students and professionals. Results showed that visual search ability as measured with a traditional visual search task is not comparable to an applied X-ray image inspection task. Even though both tasks require aspects of the same visual-cognitive abilities, the overlap between the tasks was small. We concluded that different aspects of visual-cognitive abilities predict performance on the measured tasks. Furthermore, although our tested populations were comparable in terms of performance predictors based on visual-cognitive abilities, professionals outperformed students on an applied X-ray image inspection task. Hence, inferences from our research questions have to be treated with caution, because the comparability of the two populations depends on the task.

Detecting Bombs in X-Ray Images of Hold Baggage: 2D Versus 3D Imaging

OBJECTIVE

This study compared the visual inspection performance of airport security officers (screeners) when screening hold baggage with state-of-the-art 3D versus older 2D imaging.

BACKGROUND

3D imaging based on computer tomography features better automated detection of explosives and higher baggage throughput than older 2D X-ray imaging technology. Nonetheless, some countries and airports hesitate to implement 3D systems due to their lower image quality and the concern that screeners will need extensive and specific training before they can be allowed to work with 3D imaging.

METHOD

Screeners working with 2D imaging (2D screeners) and screeners working with 3D imaging (3D screeners) conducted a simulated hold baggage screening task with both types of imaging. Differences in image quality of the imaging systems were assessed with the standard procedure for 2D imaging.

RESULTS

Despite lower image quality, screeners' detection performance with 3D imaging was similar to that with 2D imaging. 3D screeners revealed higher detection performance with both types of imaging than 2D screeners.

CONCLUSION

Features of 3D imaging systems (3D image rotation and slicing) seem to compensate for lower image quality. Visual inspection competency acquired with one type of imaging seems to transfer to visual inspection with the other type of imaging.

APPLICATION

Replacing older 2D with newer 3D imaging systems can be recommended. 2D screeners do not need extensive and specific training to achieve comparable detection performance with 3D imaging. Current image quality standards for 2D imaging need revision before they can be applied to 3D imaging.

Expertise, Automation and Trust in X-Ray Screening of Cabin Baggage

X-ray screening of passenger baggage is a key component in aviation security. The current study investigated how experts and novices performed in an X-ray baggage screening task while being assisted by an adaptable diagnostic aid. Furthermore, it examined how both groups operated and trusted this automated system. 30 experts (certified screeners) and 31 novices (students) had to indicate whether a target item (either a knife or a gun) was present in a series of X-ray images of cabin baggage. Half of the participants could choose between three different support levels of the diagnostic aid (DA): (1) no support, (2) a cue indicating the presence of a potential target without locating it, or (3) a cue indicating the presence of a potential target by surrounding it with a red frame. As expected, experts achieved higher detection performance (d'), were more self-confident and felt more competent in achieving the task than novices. Furthermore, experts experienced less time pressure and fatigue. Although both groups used the DA in a comparable way (in terms of support level used and frequency of level switches), results showed a performance increase for novices working with the DA compared to novices without support. This benefit of DA was not observed for experts. Interestingly, despite no difference in perceived trust ratings, experts were more compliant (i.e., following DA recommendations when it indicated the presence of a target) and reliant (i.e., following DA recommendations when it indicated the absence of a target) than novices. Altogether, the results of the present study suggested that novices benefited more from a DA than experts. Furthermore, compliance and reliance on DA seemed to depend on expertise with the task. Since experts should be better at assessing the reliability of the DA than novices, they may have used the DA as 'back-up' to confirm their decisions based on expertise (confirmatory function), while novices may have used it as a guide to base their decisions on (support function). Finally, trust towards a DA was associated with the degree to which participants found the DA useful.

Chest X-ray evaluation training: impact of normal and abnormal image ratio and instructional sequence

CONTEXT

Medical image perception training generally focuses on abnormalities, whereas normal images are more prevalent in medical practice. Furthermore, instructional sequences that let students practice prior to expert instruction (inductive) may lead to improved performance compared with methods that give students expert instruction before practice (deductive). This study investigates the effects of the proportion of normal images and practice-instruction order on learning to interpret medical images. It is hypothesised that manipulation of the proportion of normal images will lead to a sensitivity-specificity trade-off and that students in practice-first (inductive) conditons need more time per practice case but will correctly identify more test cases.

METHODS

Third-year medical students (n = 103) learned radiograph interpretation by practising cases with, respectively, 30% or 70% normal radiographs prior to expert instruction (practice-first order) or after expert instruction (instruction-first order). After training, students performed a test (60% normal) and sensitivity (% of correctly identified abnormal radiographs), specificity (% of correctly identified normal radiographs), diagnostic performance (% of correct diagnoses) and case duration were measured.

RESULTS

The conditions with 30% of normal images scored higher on sensitivity but the conditions with 70% of normal images scored higher on specificity, indicating a sensitivity and specificity trade-off. Those who participated in inductive conditions took less time per practice case but more per test case. They had similar test sensitivity, but scored lower on test specificity.

CONCLUSIONS

The proportion of normal images impacted the sensitivity-specificity trade-off. This trade-off should be an important consideration for the alignment of training with future practice. Furthermore, the deductive conditions unexpectedly scored higher on specificity when participants took less time per case. An inductive approach did not lead to higher diagnostic performance, possibly because participants might already have relevant prior knowledge. Deductive approaches are therefore advised for the training of advanced learners.

Observer Variability in Breast Cancer Diagnosis between Countries with and without Breast Screening

RATIONAL AND OBJECTIVES

Image reporting is a vital component of patient management depending on individual radiologists' performance. Our objective was to explore mammographic diagnostic efficacy in a country where breast cancer screening does not exist.

MATERIALS AND METHODS

Two mammographic test sets were used: a typical screening (TS) and high-difficulty (HD) test set. Nonscreening (NS) radiologists (n = 11) read both test sets, while 52 and 49 screening radiologists read the TS and HD test sets, respectively. The screening radiologists were classified into two groups: a less experienced (LE) group with ≤5 years' experience and a more experienced (ME) group with ≥5 years' experience. A Kruskal-Wallis and Tukey-Kramer post hoc test were used to compare reading performance among reader groups, and the Wilcoxon matched pairs tests was used to compare TS and ND test sets for the NS radiologists.

RESULTS

Across the three reader groups, there were significant differences in case sensitivity (χ² [2] = 9.4, P = .008), specificity (χ² [2] = 10.3, P = .006), location sensitivity (χ² [2] = 19.8, P < .001), receiver operating characteristics, area under the curve (χ² [2] = 19.7, P < .001) and jack-knife free-response receiver operating characteristics (JAFROCs) (χ² [2] = 18.1, P < .001). NS performance for all measured scores was significantly lower than those for the ME readers (P < .006), while only location sensitivity was lower (χ² [2] = 17.5, P = .026) for the NS compared to the LE group. No other significant differences were observed.

CONCLUSION

Large variations in mammographic performance exist between radiologists from screening and nonscreening countries.

Long-Term Visual Memory and Its Role in Learning Suppression

Long-term memory is a core aspect of human learning that permits a wide range of skills and behaviors often important for survival. While this core ability has been broadly observed for procedural and declarative memory, whether similar mechanisms subserve basic sensory or perceptual processes remains unclear. Here, we use a visual learning paradigm to show that training humans to search for common visual features in the environment leads to a persistent improvement in performance over consecutive days but, surprisingly, suppresses the subsequent ability to learn similar visual features. This suppression is reversed if the memory is prevented from consolidating, while still permitting the ability to learn multiple visual features simultaneously. These findings reveal a memory mechanism that may enable salient sensory patterns to persist in memory over prolonged durations, but which also functions to prevent false-positive detection by proactively suppressing new learning.

Automation in visual inspection tasks: X-ray luggage screening supported by a system of direct, indirect or adaptable cueing with low and high system reliability

The present study evaluated three automation modes for improving performance in an X-ray luggage screening task. One hundred and forty participants were asked to detect the presence of prohibited items in X-ray images of cabin luggage. Twenty participants conducted this task without automatic support (control group), whereas the others worked with either indirect cues (system indicated the target presence without specifying its location), or direct cues (system pointed out the exact target location) or adaptable automation (participants could freely choose between no cue, direct and indirect cues). Furthermore, automatic support reliability was manipulated (low versus high). The results showed a clear advantage for direct cues regarding detection performance and response time. No benefits were observed for adaptable automation. Finally, high automation reliability led to better performance and higher operator trust. The findings overall confirmed that automatic support systems for luggage screening should be designed such that they provide direct, highly reliable cues.Practitioner summary: The present study confirmed previous findings showing better detection performance in X-ray images of luggage when supported by automation providing direct, highly reliable cues. Furthermore, participants used adaptable automation only to select their preferred level of automation. This behaviour did not provide the benefits expected under adaptable automation.

Automation in airport security X-ray screening of cabin baggage: Examining benefits and possible implementations of automated explosives detection

Bomb attacks on civil aviation make detecting improvised explosive devices and explosive material in passenger baggage a major concern. In the last few years, explosive detection systems for cabin baggage screening (EDSCB) have become available. Although used by a number of airports, most countries have not yet implemented these systems on a wide scale. We investigated the benefits of EDSCB with two different levels of automation currently being discussed by regulators and airport operators: automation as a diagnostic aid with an on-screen alarm resolution by the airport security officer (screener) or EDSCB with an automated decision by the machine. The two experiments reported here tested and compared both scenarios and a condition without automation as baseline. Participants were screeners at two international airports who differed in both years of work experience and familiarity with automation aids. Results showed that experienced screeners were good at detecting improvised explosive devices even without EDSCB. EDSCB increased only their detection of bare explosives. In contrast, screeners with less experience (tenure < 1 year) benefitted substantially from EDSCB in detecting both improvised explosive devices and bare explosives. A comparison of all three conditions showed that automated decision provided better human-machine detection performance than on-screen alarm resolution and no automation. This came at the cost of slightly higher false alarm rates on the human-machine system level, which would still be acceptable from an operational point of view. Results indicate that a wide-scale implementation of EDSCB would increase the detection of explosives in passenger bags and automated decision instead of automation as diagnostic aid with on screen alarm resolution should be considered.

"It's Not a Tumor": A Framework for Capitalizing on Individual Diversity to Boost Target Detection

Even experts routinely miss infrequent targets, such as weapons in baggage scans or tumors in mammograms, because the visual system is not equipped to notice the unusual. To date, limited progress has been made toward improving human factors that mediate such critical diagnostic tasks. Here, we present a novel framework for pairing individuals' estimates to increase target detection. Using a wisdom-of-crowds approach that capitalizes on the visual system's ability to efficiently combine information, we demonstrated how averaging two noninteracting individuals' continuous estimates of whether a briefly presented image contained a prespecified target can significantly boost detection across a range of tasks. Furthermore, we showed how pairing individuals' estimates to maximize decorrelated patterns of performance in one task can optimize performance on a separate task. These results make significant advances toward combating severe deficits in target detection using straightforward applications for maximizing performance within limited pools of observers.

Hacking the Human: The Prevalence Paradox in Cybersecurity

OBJECTIVE

This work assesses the efficacy of the "prevalence effect" as a form of cyberattack in human-automation teaming, using an email task.

BACKGROUND

Under the prevalence effect, rare signals are more difficult to detect, even when taking into account their proportionally low occurrence. This decline represents diminished human capability to both detect and respond. As signal probability (SP) approaches zero, accuracy exhibits logarithmic decay. Cybersecurity, a context in which the environment is entirely artificial, provides an opportunity to manufacture conditions enhancing or degrading human performance, such as prevalence effects. Email cybersecurity prevalence effects have not previously been demonstrated, nor intentionally manipulated.

METHOD

The Email Testbed (ET) provides a simulation of a clerical email work involving messages containing sensitive personal information. Using the ET, participants were presented with 300 email interactions and received cyberattacks at rates of either 1%, 5%, or 20%.

RESULTS

Results demonstrated the existence and power of prevalence effects in email cybersecurity. Attacks delivered at a rate of 1% were significantly more likely to succeed, and the overall pattern of accuracy across declining SP exhibited logarithmic decay.

APPLICATION

These findings suggest a "prevalence paradox" within human-machine teams. As automation reduces attack SP, the human operator becomes increasingly likely to fail in detecting and reporting attacks that remain. In the cyber realm, the potential to artificially inflict this state on adversaries, hacking the human operator rather than algorithmic defense, is considered. Specific and general information security design countermeasures are offered.

Learning in the Target Prevalence Effect

Rare or low prevalence targets are detected less well than counterparts that occur with higher probability. It stands to reason, though, that before such a deficit is apparent, information about a given target's probability of occurrence must be apprehended. In this study, we investigated how much target experience is necessary for target probabilities to be fully acquired and established within mental task representations. A central finding was that different target probability values required approximately the same amount of target sampling to learn. This was true whether learning about target probabilities from a naive start-point (Experiment 1) or when recalibrating from one probability value to another (Experiment 2). We discuss these findings in relation to how mental task representations are modified when new task-relevant information is received and the attentional consequences of such changes.

New approaches to the analysis of eye movement behaviour across expertise while viewing brain MRIs

Brain tumour detection and diagnosis requires clinicians to inspect and analyse brain magnetic resonance images. Eye-tracking is commonly used to examine observers' gaze behaviour during such medical image interpretation tasks, but analysis of eye movement sequences is limited. We therefore used ScanMatch, a novel technique that compares saccadic eye movement sequences, to examine the effect of expertise and diagnosis on the similarity of scanning patterns. Diagnostic accuracy was also recorded. Thirty-five participants were classified as Novices, Medics and Experts based on their level of expertise. Participants completed two brain tumour detection tasks. The first was a whole-brain task, which consisted of 60 consecutively presented slices from one patient; the second was an independent-slice detection task, which consisted of 32 independent slices from five different patients. Experts displayed the highest accuracy and sensitivity followed by Medics and then Novices in the independent-slice task. Experts showed the highest level of scanning pattern similarity, with medics engaging in the least similar scanning patterns, for both the whole-brain and independent-slice task. In the independent-slice task, scanning patterns were the least similar for false negatives across all expertise levels and most similar for experts when they responded correctly. These results demonstrate the value of using ScanMatch in the medical image perception literature. Future research adopting this tool could, for example, identify cases that yield low scanning similarity and so provide insight into why diagnostic errors occur and ultimately help in training radiologists.

Setting Priorities in Behavioral Interventions: An Application to Reducing Phishing Risk

Phishing risk is a growing area of concern for corporations, governments, and individuals. Given the evidence that users vary widely in their vulnerability to phishing attacks, we demonstrate an approach for assessing the benefits and costs of interventions that target the most vulnerable users. Our approach uses Monte Carlo simulation to (1) identify which users were most vulnerable, in signal detection theory terms; (2) assess the proportion of system-level risk attributable to the most vulnerable users; (3) estimate the monetary benefit and cost of behavioral interventions targeting different vulnerability levels; and (4) evaluate the sensitivity of these results to whether the attacks involve random or spear phishing. Using parameter estimates from previous research, we find that the most vulnerable users were less cautious and less able to distinguish between phishing and legitimate emails (positive response bias and low sensitivity, in signal detection theory terms). They also accounted for a large share of phishing risk for both random and spear phishing attacks. Under these conditions, our analysis estimates much greater net benefit for behavioral interventions that target these vulnerable users. Within the range of the model's assumptions, there was generally net benefit even for the least vulnerable users. However, the differences in the return on investment for interventions with users with different degrees of vulnerability indicate the importance of measuring that performance, and letting it guide interventions. This study suggests that interventions to reduce response bias, rather than to increase sensitivity, have greater net benefit.

Suboptimality in perceptual decision making

Human perceptual decisions are often described as optimal. Critics of this view have argued that claims of optimality are overly flexible and lack explanatory power. Meanwhile, advocates for optimality have countered that such criticisms single out a few selected papers. To elucidate the issue of optimality in perceptual decision making, we review the extensive literature on suboptimal performance in perceptual tasks. We discuss eight different classes of suboptimal perceptual decisions, including improper placement, maintenance, and adjustment of perceptual criteria; inadequate tradeoff between speed and accuracy; inappropriate confidence ratings; misweightings in cue combination; and findings related to various perceptual illusions and biases. In addition, we discuss conceptual shortcomings of a focus on optimality, such as definitional difficulties and the limited value of optimality claims in and of themselves. We therefore advocate that the field drop its emphasis on whether observed behavior is optimal and instead concentrate on building and testing detailed observer models that explain behavior across a wide range of tasks. To facilitate this transition, we compile the proposed hypotheses regarding the origins of suboptimal perceptual decisions reviewed here. We argue that verifying, rejecting, and expanding these explanations for suboptimal behavior - rather than assessing optimality per se - should be among the major goals of the science of perceptual decision making.

Getting satisfied with "satisfaction of search": How to measure errors during multiple-target visual search

Visual search studies are common in cognitive psychology, and the results generally focus upon accuracy, response times, or both. Most research has focused upon search scenarios where no more than 1 target will be present for any single trial. However, if multiple targets can be present on a single trial, it introduces an additional source of error because the found target can interfere with subsequent search performance. These errors have been studied thoroughly in radiology for decades, although their emphasis in cognitive psychology studies has been more recent. One particular issue with multiple-target search is that these subsequent search errors (i.e., specific errors which occur following a found target) are measured differently by different studies. There is currently no guidance as to which measurement method is best or what impact different measurement methods could have upon various results and conclusions. The current investigation provides two efforts to address these issues. First, the existing literature is reviewed to clarify the appropriate scenarios where subsequent search errors could be observed. Second, several different measurement methods are used with several existing datasets to contrast and compare how each method would have affected the results and conclusions of those studies. The evidence is then used to provide appropriate guidelines for measuring multiple-target search errors in future studies.

Can Radiologists Learn From Airport Baggage Screening?: A Survey About Using Fictional Patients for Quality Assurance

RATIONALE AND OBJECTIVES

For both airport baggage screeners and radiologists, low target prevalence is associated with low detection rate, a phenomenon known as "prevalence effect." In airport baggage screening, the target prevalence is artificially increased with fictional weapons that are digitally superimposed on real baggage. This strategy improves the detection rate of real weapons and also allows airport supervisors to monitor screener performance. A similar strategy using fictional patients could be applied in radiology. The purpose of this study was twofold: (1) to review the psychophysics literature regarding low target prevalence and (2) to survey radiologists' attitudes toward using fictional patients as a quality assurance tool.

MATERIALS AND METHODS

We reviewed the psychophysics literature on low target prevalence and airport x-ray baggage screeners. An online survey was e-mailed to all members of the Association of University Radiologists to determine their attitudes toward using fictional patients in radiology.

RESULTS

Of the 1503 Association of University Radiologists member recipients, there were 153 respondents (10% response rate). When asked whether the use of fictional patients was a good idea, the responses were as follows: disagree (44%), neutral (25%), and agree (31%). The most frequent concern was the time taken away from doing clinical work (89% of the respondents).

CONCLUSIONS

The psychophysics literature supports the use of fictional targets to mitigate the prevalence effect. However, the use of fictional patients is not a popular idea among academic radiologists.

Cutting through the MADness: Expectations about what a target is doing impact how likely it is to be found in dynamic visual displays

When searching for things in the world, we seldom encounter the static environment so often afforded by laboratory search tasks. Dynamic events tend to capture attention; however, Kunar and Watson previously found that dynamic search displays (search for vowel targets among moving and blinking letter stimuli) resulted in strikingly high miss rates. A possible explanation for the high miss rates is that the presence of dynamic features resulted in participants becoming sensitive to the likelihood of appearance for each target category and dynamic feature combination (e.g., moving A, blinking A, moving E), even though dynamic features did not define the targets. Overall target prevalence was high in these studies, but the prevalence of each target + dynamic feature combination was low, which may have led to a low-prevalence effect, whereby infrequent targets were disproportionately missed. Searchers may have preferentially searched for (or avoided) certain dynamic features, even though the dynamic features did not differentiate targets from distractors. We investigated whether the frequency with which targets possessed particular dynamic features would affect the likelihood of search misses. We found that targets possessing features which rarely accompanied a "hit" were more often missed than those with dynamic features more strongly associated with target detection. This suggests that searchers are sensitive to the prevalence of non-defining features and use this information to direct their searches.

Eye movement feedback fails to improve visual search performance

Many real-world searches (e.g., radiology and baggage screening) have rare targets. When targets are rare, observers perform rapid, incomplete searches, leading to higher miss rates. To improve search for rare (10% prevalence) targets, we provided eye movement feedback (EMF) to observers during their searches. Although the nature of the EMF varied across experiments, each method informed observers about the regions of the display that had not yet been inspected. We hypothesized that feedback would help guide attention to unsearched areas and increase the proportion of the display searched before making a target-absent response, thereby increasing accuracy. An eye tracker was used to mark fixated areas by either removing a semiopaque gray overlay (Experiments 1 and 4) as portions of the display were fixated or by adding the overlay once the eye left a segment of the image (Experiments 2 and 4). Experiment 3 provided automated EMF, such that a new region was uncovered every 540 milliseconds. Across experiments, we varied whether people searched for “Waldo” in images from “Where’s Waldo?” search books or searched for a T among offset Ls. We found weak evidence that EMF improves accuracy in Experiment 1. However, in the remaining experiments, EMF had no effect (Experiment 4), or even reduced accuracy (Experiments 2 and 3). We conclude that the one positive result we found is likely a Type I error and that the EMF method that we used is unlikely to improve visual search performance.

When is it time to move to the next map? Optimal foraging in guided visual search

Suppose that you are looking for visual targets in a set of images, each containing an unknown number of targets. How do you perform that search, and how do you decide when to move from the current image to the next? Optimal foraging theory predicts that foragers should leave the current image when the expected value from staying falls below the expected value from leaving. Here, we describe how to apply these models to more complex tasks, like search for objects in natural scenes where people have prior beliefs about the number and locations of targets in each image, and search is guided by target features and scene context. We model these factors in a guided search task and predict the optimal time to quit search. The data come from a satellite image search task. Participants searched for small gas stations in large satellite images. We model quitting times with a Bayesian model that incorporates prior beliefs about the number of targets in each map, average search efficiency (guidance), and actual search history in the image. Clicks deploying local magnification were used as surrogates for deployments of attention and, thus, for time. Leaving times (measured in mouse clicks) were well-predicted by the model. People terminated search when their expected rate of target collection fell to the average rate for the task. Apparently, people follow a rate-optimizing strategy in this task and use both their prior knowledge and search history in the image to decide when to quit searching.

Examining social facilitation in vigilance: a hit and a miss

Vigilance is the ability of an observer to maintain attention for extended periods of time; however, performance tends to decline with time on watch, a pattern referred to as the vigilance decrement. Previous research has focused on factors that attenuate the decrement; however, one factor rarely studied is the effect of social facilitation. The purpose for the present investigation was to determine how different types of social presence affected the performance, workload and stress of vigilance. It was hypothesised that the presence of a supervisory figure would increase overall performance, but may occur at the cost of increased workload and stress. Results indicated that the per cent of false alarm and response times decreased in the presence of a supervisory figure. Using social facilitation in vigilance tasks may thus have positive, as well as, negative effects depending on the dependent measure of interest and the role of the observer. Practitioner Summary: Social facilitation has rarely been examined in the context of vigilance, even though it may improve performance. Vigilance task performance was examined under social presence. The results of the present study indicated that false alarms and response times decreased in the social presence of a supervisory figure, thus improving performance.