Effects of deliberate practice on crisis decision performance

Transport safety and human factors in the era of automation: What can transport modes learn from each other?

One of the main aims of introducing automation in transport is to improve safety by reducing or eliminating human errors; it is often argued however that this may induce new types of errors. There is different level of maturity with automation in different transport modes (road, aviation, maritime and rail), however no systematic research has been conducted on the lessons learned in different sectors, so that they can be exploited for the design of safer automated systems. The aim of this paper is to review the impact of key human factors on the safety of automated transport systems, with focus on relevant experiences from different transport sectors. A systematic literature review is carried out on the following topics: the level of trust in automation - in particular the impact of mis-aligned trust, i.e. mistrust vs overreliance, the resulting impact on operator situation awareness (SA), the implications for takeover control from machine to human, and the role of experience and training on using automated transport systems. The results revealed several areas where experiences from the aviation and road domain can be transferable to other sectors. Experiences from maritime and rail transport, although limited, tend to confirm the general patterns. Remarkably, in the road sector where higher levels of automation are only recently introduced, there are clearer and more quantitative approaches to human factors, while other sectors focus only on mental modes. Other sectors could use similar approaches to define their own context-specific metrics. The paper makes a synthesis of key messages on automation safety in different transport sectors, and presents an assessment of their transferability.

Given that the detailed original criteria for deliberate practice have not changed, could the understanding of this complex concept have improved over time? A response to Macnamara and Hambrick (2020)

In their commentary, Macnamara and Hambrick (Psychol Res, 2017) accused my colleagues and me of systematically changing the definition of the concept of deliberate practice. Deliberate practice was the result of a search for characteristics of effective practice in the laboratory that was shown to improve expert professional performance in domains, such as music. In this reply, I will first describe five different criteria that defined the original concept of deliberate practice and each of them is presented with directly supporting quotes from Ericsson, Krampe, and Tesch-Römer (Psychol Rev 100:396–406, 10.1037/0033-295X.87.3.215, 1993) paper. Unfortunately, Macnamara, Hambrick, and Oswald (Psychol Sci 25:1608–1618, 10.1177/0956797614535810, 2014) misinterpreted our concept of deliberate practice, and defined it much more broadly: “as engagement in structured activities created specifically to improve performance in a domain” (p. 914). This definition led them to include activities, such as attending lectures, studying alone by students, and group activities led by a coach, where each activity does not meet one or more of our criteria for deliberate practice. In this commentary, I will argue that Macnamara and Hambrick (2020) became aware of some of the original criteria for deliberate practice, such as the role of individualized training by a teacher, and these discoveries misled them to assume that we had changed our definition. The intended meaning of sentences that Macnamara and Hambrick (2020) had carefully selected is shown to have an appropriate interpretation in Standard English that is consistent with our original definition of deliberate practice. In conclusion, I will give a proposal for how the different perspectives can be reconciled.

Practice makes better? Testing a model for training program evaluators in situation awareness

Evaluation approaches should be appropriate for their contexts. Scholars and practitioners alike have widely acknowledged this view for at least four decades, and the ability to respond to context is clearly established as a core evaluator competency. Outside of evaluation, this knowledge of context, known as situation awareness, is seen as a critical feature of good decision making and as a factor that distinguishes experts from novices across a wide range of domains. Yet there are few opportunities for evaluators to explicitly build their skills in situation awareness. This study addressed that gap by examining the potential for one online training program, informed by research on deliberate practice, to accelerate evaluators' progress towards expertise in situation awareness. Built to align with evidence on developing situation awareness skills, the EvalPractice portal combined (1) an extensive 'case bank' of real-world evaluation scenarios, (2) repetitive practice that allowed novice evaluators to practice interpreting these evaluation scenarios, and (3) immediate feedback on the accuracy of these efforts based on events from the real-life version of the scenario. Findings from a small EvalPractice pilot suggest that it may be possible to improve foundational situation awareness skills using deliberate practice, but that further research is required to understand strategies for building higher-level situation awareness skills in evaluation.

Dealing With Unexpected Events on the Flight Deck: A Conceptual Model of Startle and Surprise

OBJECTIVE

A conceptual model is proposed in order to explain pilot performance in surprising and startling situations.

BACKGROUND

Today's debate around loss of control following in-flight events and the implementation of upset prevention and recovery training has highlighted the importance of pilots' ability to deal with unexpected events. Unexpected events, such as technical malfunctions or automation surprises, potentially induce a "startle factor" that may significantly impair performance.

METHOD

Literature on surprise, startle, resilience, and decision making is reviewed, and findings are combined into a conceptual model. A number of recent flight incident and accident cases are then used to illustrate elements of the model.

RESULTS

Pilot perception and actions are conceptualized as being guided by "frames," or mental knowledge structures that were previously learned. Performance issues in unexpected situations can often be traced back to insufficient adaptation of one's frame to the situation. It is argued that such sensemaking or reframing processes are especially vulnerable to issues caused by startle or acute stress.

CONCLUSION

Interventions should focus on (a) increasing the supply and quality of pilot frames (e.g., though practicing a variety of situations), (b) increasing pilot reframing skills (e.g., through the use of unpredictability in training scenarios), and (c) improving pilot metacognitive skills, so that inappropriate automatic responses to startle and surprise can be avoided.

APPLICATION

The model can be used to explain pilot behavior in accident cases, to design experiments and training simulations, to teach pilots metacognitive skills, and to identify intervention methods.

The Effectiveness of Bioskills Training for Simulated Open Lumbar Laminectomy

STUDY DESIGN

Randomized, prospective study within an orthopedic surgery resident program at a large urban academic medical center.

OBJECTIVES

To develop an inexpensive, user-friendly, and reproducible lumbar laminectomy bioskills training module and evaluation protocol that can be readily implemented into residency training programs to augment the clinical education of orthopedic and neurosurgical physicians-in-training.

METHODS

Twenty participants comprising senior medical students and orthopedic surgical residents. Participants were randomized to control (n = 9) or intervention (n = 11) groups controlling for level of experience (medical students, junior resident, or senior resident). The intervention group underwent a 40-minute bioskills training module, while the control group spent the same time with self-directed study. Pre- and posttest performance was self-reported by each participant (Physician Performance Diagnostic Inventory Scale [PPDIS]). Objective outcome scores were obtained from a blinded fellowship-trained attending orthopedic spine surgeon using Objective Structured Assessment of Technical Skills (OSATS) and Objective Decompression Score metrics.

RESULTS

When compared with the control group, the intervention group yielded a significant mean improvement in OSATS (P = .022) and PPDIS (P = .0001) scores. The Objective Decompression Scores improved in the intervention group with a trend toward significance (P = .058).

CONCLUSIONS

We conclude that a concise lumbar laminectomy bioskills training session can be a useful educational tool for to augment clinical education. Although no direct clinical correlation can be concluded from this study, the improvement in trainee's technical and procedural skills suggests that Sawbones training modules can be an efficient and effective tool for teaching fundamental spine surgical skills outside of the operating room.

The Automation-by-Expertise-by-Training Interaction

OBJECTIVE

I introduce the automation-by-expertise-by-training interaction in automated systems and discuss its influence on operator performance.

BACKGROUND

Transportation accidents that, across a 30-year interval demonstrated identical automation-related operator errors, suggest a need to reexamine traditional views of automation.

METHOD

I review accident investigation reports, regulator studies, and literature on human computer interaction, expertise, and training and discuss how failing to attend to the interaction of automation, expertise level, and training has enabled operators to commit identical automation-related errors.

RESULTS

Automated systems continue to provide capabilities exceeding operators' need for effective system operation and provide interfaces that can hinder, rather than enhance, operator automation-related situation awareness. Because of limitations in time and resources, training programs do not provide operators the expertise needed to effectively operate these automated systems, requiring them to obtain the expertise ad hoc during system operations. As a result, many do not acquire necessary automation-related system expertise.

CONCLUSION

Integrating automation with expected operator expertise levels, and within training programs that provide operators the necessary automation expertise, can reduce opportunities for automation-related operator errors.

APPLICATION

Research to address the automation-by-expertise-by-training interaction is needed. However, such research must meet challenges inherent to examining realistic sociotechnical system automation features with representative samples of operators, perhaps by using observational and ethnographic research. Research in this domain should improve the integration of design and training and, it is hoped, enhance operator performance.

Capturing the Naturally Occurring Superior Performance of Experts in the Laboratory

Expertise researchers have traditionally shied away from studying the highest levels of achievement in favor of studying basic cognitive processes, such as memory and categorization. In this article, we present a different approach that is focused on capturing superior (expert) performance on representative tasks that reveal the essential characteristics of expertise in a given domain. In domains where expert performance is measurable, acquisition is gradual and the highest levels are only attained after 10 years of intense preparation—even for the most “talented.” Analyses of reproducibly superior performance show that it is mediated by physiological adaptations and cognitive skills acquired as a result of the cumulative effects of special practice activities (deliberate practice). It appears that the genes necessary to attain such adaptations and expert skills can be activated in healthy children—the only clear exceptions to date being genes that control body size and height. Our knowledge of how experts acquire their superior skills provides insights into the potential for human adaptation and skill acquisition and has important implications for theories of the structure of general and expert cognition, as well as for training interventions in applied psychology and education.

A Qualitative Comparison of Susceptibility and Behavior in Recreational and Occupational Risk Environments: Implications for Promoting Health and Safety

Although internal factors that influence risk are frequently studied to understand human behavior, external factors, including social, cultural, and institutional factors, should be better utilized to inform ways to efficiently target, tailor, and promote safety messaging to at-risk populations. Semi-structured interviews obtained data from 37 motorcyclists and 18 mineworkers about their risk perceptions and behaviors within their respective dynamic environments. A comparative thematic analysis revealed information about external factors that influence risk perceptions and behaviors. Results support the importance of qualitative approaches for assessing and targeting individuals' risk perceptions and behaviors. In addition, segmenting at-risk subgroups within target populations and tailoring messages for these at-risk groups is critical for safety behavior modification. Practitioners should utilize strategic, culture-centric risk communication that takes into account external factors when determining when, who, and what to communicate via health promotion activities to more accurately disseminate valid, empathetic, and engaging communication with a higher level of fidelity.

Necessity is the mother of invention: video recording firsthand perspectives of critical medical procedures to make simulated training more effective

The traditional apprenticeship model for training doctors requires ample opportunities in the clinic for trainees to learn core procedures under the supervision of skilled doctors. In this issue, Kyser and colleagues document that the learning opportunities for residents to master certain core procedures, such as forceps and vacuum deliveries, are insufficient in many teaching hospitals. To address this serious problem, this author argues that learning techniques (i.e., deliberate practice and simulator use) from other domains of expertise, such as chess, music, and sports, must be adapted for use in medicine. For example, medical procedures should be videotaped and indexed for access over the Internet. Trainees then could view recordings of rare emergency procedures and complications and practice their decision-making skills. Evidence suggests that training outside the constraints of the clinic could be more effective in improving performance because trainees are able to engage in deliberate practice and focus on their individual weaknesses in executing procedures and making decisions. For example, with video and simulator training, trainees have the opportunity to repeatedly perform only the parts of a procedure that they find challenging until they have attained a level of acceptable speed and control. More generally, training could be structured around the particular needs of individual learners and scheduled when the learners are rested and able to concentrate fully on learning. Training also should include opportunities for learners to practice repeatedly with different patient descriptions requiring the same or different, yet related, procedures to teach trainees effective discrimination and execution.

The roles of deliberate practice and innate ability in developing expertise: evidence and implications

CONTEXT

Medical education research focuses extensively on experience and deliberate practice (DP) as key factors in the development of expert performance. The research on DP minimises the role of individual ability in expert performance. This claim ignores a large body of research supporting the importance of innate individual cognitive differences. We review the relationship between DP and an innate individual ability, working memory (WM) capacity, to illustrate how both DP and individual ability predict expert performance.

METHODS

This narrative review examines the relationship between DP and WM in accounting for expert performance. Studies examining DP, WM and individual differences were identified through a targeted search.

RESULTS

Although all studies support extensive DP as a factor in explaining expertise, much research suggests individual cognitive differences, such as WM capacity, predict expert performance after controlling for DP. The extent to which this occurs may be influenced by the nature of the task under study and the cognitive processes used by experts. The importance of WM capacity is greater for tasks that are non-routine or functionally complex. Clinical reasoning displays evidence of this task-dependent importance of individual ability.

CONCLUSIONS

No single factor is both necessary and sufficient in explaining expertise, and individual abilities such as WM can be important. These individual abilities are likely to contribute to expert performance in clinical settings. Medical education research and practice should identify the individual differences in novices and experts that are important to clinical performance.

Optimizing resident training: results and recommendations of the 2009 Council of Residency Directors consensus conference

This paper reports the results of a consensus conference of the Council of Emergency Medicine Residency Directors (CORD) to discuss the experiential training component of residency education in the emergency department (ED) and to make recommendations on structuring clinical training. Self-selected emergency medicine (EM) educators discussed experiential training focusing on three topic areas: 1) methods to optimize training in the clinical setting, 2) identification of goals and objectives by training year, and 3) determination of measurable behaviors demonstrating achievement of goals and objectives by residents. Topic areas were organized into the following questions: 1) what is the optimal number and evolution of ED shifts for EM residents during their residency training, 2) what clinical skills are expected of a resident at each level of training, and 3) what objective measures should be used to provide evidence of resident competency? Participants attended a lecture on the goals of the conference, the questions to be answered, and the role and implementation of deliberate practice into experiential training. Attendees were divided into three groups, each discussing one question. Each group had two discussion leaders. All discussions were digitally recorded for accuracy. After discussion all groups reconvened and reported summaries of discussions and recommendations to ensure group agreement. There were 59 participants representing 42 training programs. Educators agree that essential features of designing the ED clinical experience include the need to: 1) structure and tailor the clinical experience to optimize learning, 2) establish expectations for clinical performance based on year of training, and 3) provide feedback that is explicit to year-specific performance expectations.

Development and evaluation of a simulation-based pediatric emergency medicine curriculum

PURPOSE

The infrequency of severe childhood illness limits opportunities for emergency medicine (EM) providers to learn from real-world experience. Simulation offers an evidence-based educational approach to develop and practice clinical skills.

METHOD

This was a two-phase, randomized trial with a wait-list control condition. The development phase (2005-2006) involved systematic curriculum and rating checklist creation, producing a six-case, simulation-based curriculum linked to three evaluation cases.In the validation phase (2006-2007), the authors randomized 69 residents from two EM residencies to either an intervention group that received the curriculum one month before the first assessment of all participants or a wait-list control group that received the identical curriculum three months later. A final assessment of all residents followed one month after that. Two raters evaluated all residents. Primary outcome measures are percentages of items completed correctly. The authors assessed rater agreement using intraclass correlation (ICC) and compared group performance using mixed-model analysis of variance.

RESULTS

ICCs surpassed 0.78. The instructional intervention produced a statistically significant effect for two of three evaluation cases for the validation phase of the study, a case x occasion interaction. Training year was significantly associated with better performance. In a multivariate analysis, training year and session correlated with score, but study group did not.

CONCLUSIONS

A one-day, simulation-based pediatric EM curriculum produced limited results. The evaluation approach is reasonable and reproducible for the population studied. Instructional dose strength and factors may have limited curriculum effectiveness. Focused, frequent, and effortful instructional interventions are necessary to achieve substantial performance improvements.

The role of expertise research and human factors in capturing, explaining, and producing superior performance

OBJECTIVES

The goal of this article is to identify some of the major trends and findings in expertise research and their connections to human factors.

BACKGROUND

Progress in the study of superior human performance has come from improved methods of measuring expertise and the development of better tools for revealing the mechanisms that support expert performance, such as protocol analysis and eye tracking.

METHODS

We review some of the challenges of capturing superior human performance in the laboratory and the means by which the expert performance approach may overcome such challenges. We then discuss applications of the expert performance approach to a handful of domains that have long been of interest to human factors researchers.

RESULTS

Experts depend heavily on domain-specific knowledge for superior performance, and such knowledge enables the expert to anticipate and prepare for future actions more efficiently. Training programs designed to focus learners' attention on task-related knowledge and skills critical to expert performance have shown promise in facilitating skill acquisition among nonexperts and in reducing errors by experts on representative tasks.

CONCLUSIONS

Although significant challenges remain, there is encouraging progress in domains such as sports, aviation, and medicine in understanding some of the mechanisms underlying human expertise and in structuring training and tools to improve skilled performance.

APPLICATIONS

Knowledge engineering techniques can capture expert knowledge and preserve it for organizations and for the development of expert systems. Understanding the mechanisms that underlie expert performance may provide insights into the structuring of better training programs for improvingskill and in designing systems to support professional expertise.

Hospital Response to Acute-Onset Disasters: The State of the Science in 2005

The level of emergency preparedness in US hospitals is a concern in light of the steady threat of natural disasters, transportation and industrial accidents, and the possibility of terror attack resulting in mass casualties. The science of hospital emergency preparedness is in an early stage of development. For research to logically expand knowledge, an accurate assessment--or examination of the state of the science--is conducted to determine the current state of knowledge, gaps in knowledge, and opportunities for future research. Milsten reviewed the literature on hospital response to acute-onset disasters from 1977 to 1999. His review of 107 articles contains research studies, case studies,and lessons learned pieces largely published in the medical literature.Milsten's analysis provides a substantial starting point. This article examines Milsten's review, identifies articles that have been published that add to this knowledge base, and identifies additional phenomena of interest.