From Teams to Teamness: Future Directions in the Science of Team Cognition

OBJECTIVE

We review the current state-of-the-art in team cognition research, but more importantly describe the limitations of existing theories, laboratory paradigms, and measures considering the increasing complexities of modern teams and the study of team cognition.

BACKGROUND

Research on, and applications of, team cognition has led to theories, data, and measures over the last several decades.

METHOD

This article is based on research questions generated in a spring 2022 seminar on team cognition at Arizona State University led by the first author.

RESULTS

Future research directions are proposed for extending the conceptualization of teams and team cognition by examining dimensions of teamness; extending laboratory paradigms to attain more realistic teaming, including nonhuman teammates; and advancing measures of team cognition in a direction such that data can be collected unobtrusively, in real time, and automatically.

CONCLUSION

The future of team cognition is one of the new discoveries, new research paradigms, and new measures.

APPLICATION

Extending the concepts of teams and team cognition can also extend the potential applications of these concepts.

Communication Strategies in Human-Autonomy Teams During Technological Failures

OBJECTIVE

This study examines low-, medium-, and high-performing Human-Autonomy Teams' (HATs') communication strategies during various technological failures that impact routine communication strategies to adapt to the task environment.

BACKGROUND

Teams must adapt their communication strategies during dynamic tasks, where more successful teams make more substantial adaptations. Adaptations in communication strategies may explain how successful HATs overcome technological failures. Further, technological failures of variable severity may alter communication strategies of HATs at different performance levels in their attempts to overcome each failure.

METHOD

HATs in a Remotely Piloted Aircraft System-Synthetic Task Environment (RPAS-STE), involving three team members, were tasked with photographing targets. Each triad had two randomly assigned participants in navigator and photographer roles, teaming with an experimenter who simulated an AI pilot in a Wizard of Oz paradigm. Teams encountered two different technological failures, automation and autonomy, where autonomy failures were more challenging to overcome.

RESULTS

High-performing HATs calibrated their communication strategy to the complexity of the different failures better than medium- and low-performing teams. Further, HATs adjusted their communication strategies over time. Finally, only the most severe failures required teams to increase the efficiency of their communication.

CONCLUSION

HAT effectiveness under degraded conditions depends on the type of communication strategies enacted by the team. Previous findings from studies of all-human teams apply here; however, novel results suggest information requests are particularly important to HAT success during failures.

APPLICATION

Understanding the communication strategies of HATs under degraded conditions can inform training protocols to help HATs overcome failures.

The Impact of Training on Human-Autonomy Team Communications and Trust Calibration

OBJECTIVE

This work examines two human-autonomy team (HAT) training approaches that target communication and trust calibration to improve team effectiveness under degraded conditions.

BACKGROUND

Human-autonomy teaming presents challenges to teamwork, some of which may be addressed through training. Factors vital to HAT performance include communication and calibrated trust.

METHOD

Thirty teams of three, including one confederate acting as an autonomous agent, received either entrainment-based coordination training, trust calibration training, or control training before executing a series of missions operating a simulated remotely piloted aircraft. Automation and autonomy failures simulating degraded conditions were injected during missions, and measures of team communication, trust, and task efficiency were collected.

RESULTS

Teams receiving coordination training had higher communication anticipation ratios, took photos of targets faster, and overcame more autonomy failures. Although autonomy failures were introduced in all conditions, teams receiving the calibration training reported that their overall trust in the agent was more robust over time. However, they did not perform better than the control condition.

CONCLUSIONS

Training based on entrainment of communications, wherein introduction of timely information exchange through one team member has lasting effects throughout the team, was positively associated with improvements in HAT communications and performance under degraded conditions. Training that emphasized the shortcomings of the autonomous agent appeared to calibrate expectations and maintain trust.

APPLICATIONS

Team training that includes an autonomous agent that models effective information exchange may positively impact team communication and coordination. Training that emphasizes the limitations of an autonomous agent may help calibrate trust.

Establishing Human Observer Criterion in Evaluating Artificial Social Intelligence Agents in a Search and Rescue Task

Artificial social intelligence (ASI) agents have great potential to aid the success of individuals, human-human teams, and human-artificial intelligence teams. To develop helpful ASI agents, we created an urban search and rescue task environment in Minecraft to evaluate ASI agents' ability to infer participants' knowledge training conditions and predict participants' next victim type to be rescued. We evaluated ASI agents' capabilities in three ways: (a) comparison to ground truth-the actual knowledge training condition and participant actions; (b) comparison among different ASI agents; and (c) comparison to a human observer criterion, whose accuracy served as a reference point. The human observers and the ASI agents used video data and timestamped event messages from the testbed, respectively, to make inferences about the same participants and topic (knowledge training condition) and the same instances of participant actions (rescue of victims). Overall, ASI agents performed better than human observers in inferring knowledge training conditions and predicting actions. Refining the human criterion can guide the design and evaluation of ASI agents for complex task environments and team composition.

State of science: models and methods for understanding and enhancing teams and teamwork in complex sociotechnical systems

This state of the science review brings together the disparate literature of effective strategies for enhancing and accelerating team performance. The review evaluates and synthesises models and proposes recommended avenues for future research. The two major models of the Input-Mediator-Output-Input (IMOI) framework and the Big Five dimensions of teamwork were reviewed and both will need significant development for application to future teams comprising non-human agents. Research suggests that a multi-method approach is appropriate for team measurements, such as the integration of methods from self-report, observer ratings, event-based measurement and automated recordings. Simulations are recommended as the most effective team-based training interventions. The impact of new technology and autonomous agents is discussed with respect to the changing nature of teamwork. In particular, whether existing teamwork models and measures are suitable to support the design, operation and evaluation of human-nonhuman teams of the future. Practitioner summary: This review recommends a multi-method approach to the measurement and evaluation of teamwork. Team models will need to be adapted to describe interaction with non-human agents, which is what the future is most likely to hold. The most effective team training interventions use simulation-based approaches.

Evaluating sociotechnical dynamics in a simulated remotely-piloted aircraft system: a layered dynamics approach

As coordination mechanisms change and technology failures occur, a sociotechnical system must reorganise itself across human and technological layers to maintain effectiveness. We present a study examining reorganisation across communication, controls and vehicle layers of a remotely-piloted aircraft system (RPAS) using a layered dynamics approach. Team members (pilot; navigator; photographer) performed 5 simulated RPAS missions using different operator configurations, including all-human and human-autonomy teams. Reorganization (operationally defined using entropy) time series measured the changing system reorganisation profiles under different operator configurations and following autonomy failures. Correlations between these reorganisation profiles and team effectiveness scores describe the manner in which the system had to be coordinated to maintain effectiveness under these changing conditions. Four unplanned autonomy failures were analysed to visualise system reorganisation following a technology failure. With its objective and real-time modelling and measurement capabilities, layered dynamics complements existing systems thinking tools for understanding sociotechnical complexity and enhancing system effectiveness. Practitioner summary: A layered dynamics approach for understanding how a sociotechnical system dynamically reorganises itself is presented. The layered dynamics of RPAS were analysed under different operator configurations and following autonomy failures. Layered dynamics complements existing system-thinking tools for modelling sociotechnical system complexity and effectiveness. Abbreviation: RPAS: remotely-piloted aircraft system; HIS: human-systems integration; EAST: event analysis of systemic teamwork; H1: hypothesis 1; H2: hypothesis 2; H3: hypothesis 3; CERTT-STE: cognitive engineering research on team tasks--synthetic task environment; AVO: air vehicle operator; PLO: payload operator; DEMPC: data exploitation, mission planning, and communications; ACT-R: adaptive control of thought-rational; sec: seconds; ANOVA: analysis of variance.

Bridging psychology and engineering to make technology work for people

Engineering grand challenges and big ideas not only demand innovative engineering solutions, but also typically involve and affect human thought, behavior, and quality of life. To solve these types of complex problems, multidisciplinary teams must bring together experts in engineering and psychological science, yet fusing these distinct areas can be difficult. This article describes how Human Systems Engineering (HSE) researchers have confronted such challenges at the interface of humans and technological systems. Two narrative cases are reported-computer game-based cognitive assessments and medical device reprocessing-and lessons learned are shared. The article then discusses 2 strategies currently being explored to enact such lessons and enhance these kinds of multidisciplinary engineering teams: a "top-down" administrative approach that supports team formation and productivity through a university research center, and a "bottom-up" engineering education approach that prepares students to work at the intersection of psychology and engineering. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Information-Pooling Bias in Collaborative Security Incident Correlation Analysis

OBJECTIVE

Incident correlation is a vital step in the cybersecurity threat detection process. This article presents research on the effect of group-level information-pooling bias on collaborative incident correlation analysis in a synthetic task environment.

BACKGROUND

Past research has shown that uneven information distribution biases people to share information that is known to most team members and prevents them from sharing any unique information available with them. The effect of such biases on security team collaborations are largely unknown.

METHOD

Thirty 3-person teams performed two threat detection missions involving information sharing and correlating security incidents. Incidents were predistributed to each person in the team based on the hidden profile paradigm. Participant teams, randomly assigned to three experimental groups, used different collaboration aids during Mission 2.

RESULTS

Communication analysis revealed that participant teams were 3 times more likely to discuss security incidents commonly known to the majority. Unaided team collaboration was inefficient in finding associations between security incidents uniquely available to each member of the team. Visualizations that augment perceptual processing and recognition memory were found to mitigate the bias.

CONCLUSION

The data suggest that (a) security analyst teams, when conducting collaborative correlation analysis, could be inefficient in pooling unique information from their peers; (b) employing off-the-shelf collaboration tools in cybersecurity defense environments is inadequate; and (c) collaborative security visualization tools developed considering the human cognitive limitations of security analysts is necessary.

APPLICATION

Potential applications of this research include development of team training procedures and collaboration tool development for security analysts.

Teaming With a Synthetic Teammate: Insights into Human-Autonomy Teaming

Objective Three different team configurations are compared with the goal of better understanding human-autonomy teaming (HAT). Background Although an extensive literature on human-automation interaction exists, much less is known about HAT in which humans and autonomous agents interact as coordinated units. Further research must be conducted to better understand how all-human teams compare to HAT. Methods In an unmanned aerial system (UAS) context, a comparison was made among three types of three-member teams: (1) synthetic teams in which the pilot role is assigned to a synthetic teammate, (2) control teams in which the pilot was an inexperienced human, and (3) experimenter teams in which an experimenter served as an experienced pilot. Ten of each type of team participated. Measures of team performance, target processing efficiency, team situation awareness, and team verbal behaviors were analyzed. Results Synthetic teams performed as well at the mission level as control (all human) teams but processed targets less efficiently. Experimenter teams performed better across all other measures compared to control and synthetic teams. Conclusion Though there is potential for a synthetic agent to function as a full-fledged teammate, further advances in autonomy are needed to improve team-level dynamics in HAT teams. Application This research contributes to our understanding of how to make autonomy a good team player.

Investigating Team Coordination in Baseball Using a Novel Joint Decision Making Paradigm

A novel joint decision making paradigm for assessing team coordination was developed and tested using baseball infielders. Balls launched onto an infield at different trajectories were filmed using four video cameras that were each placed at one of the typical positions of the four infielders. Each participant viewed temporally occluded videos for one of the four positions and were asked to say either “ball” if they would attempt to field it or the name of the bag that they would cover. The evaluation of two experienced coaches was used to assign a group coordination score for each trajectory and group decision times were calculated. Thirty groups of 4 current college baseball players were: (i) teammates (players from same team/view from own position), (ii) non-teammates (players from different teams/view from own position), or (iii) scrambled teammates (players from same team/view not from own position). Teammates performed significantly better (i.e., faster and more coordinated decisions) than the other two groups, whereas scrambled teammates performed significantly better than non-teammates. These findings suggest that team coordination is achieved through both experience with one’s teammates’ responses to particular events (e.g., a ball hit up the middle) and one’s own general action capabilities (e.g., running speed). The sensitivity of our joint decision making paradigm to group makeup provides support for its use as a method for studying team coordination.

Identification of the Emplacement of Improvised Explosive Devices by Experienced Mission Payload Operators

Improvised Explosive Devices (IEDs) have become one of the deadliest threats to military personnel, resulting in over 50% of American combat casualties in Iraq and Afghanistan. Identification of IED emplacement is conducted by mission payload operators (MPOs). Yet, experienced MPOs are limited in number, making MPO training a critical intervention. In this article, we implement a Cognitive Engineering Based on Expert Skill methodology to better understand how experienced MPOs identify the emplacement of IEDs for the purposes of improving training. First, expert knowledge was elicited through interviews and questionnaires to identify the types of perceptual cues used and how these cues are cognitively processed. Results indicate that there are many different static and dynamic cues that interact with each other over time and space. Using data from the interviews and questionnaires, an empirically grounded framework is presented that explains the cognitive process of IED emplacement detection. Using the overall findings and the framework, IED emplacement training scenarios were developed and built into a simulation.

Human-Robot Teaming in Urban Search and Rescue

Although current urban search and rescue (USAR) robots are little more than remotely controlled cameras, the end goal is for them to work alongside humans as trusted teammates. Natural language communications and performance data are collected as a team of humans works to carry out a simulated search and rescue task in an uncertain virtual environment. Conditions are tested emulating a remotely controlled robot versus an intelligent one. Differences in performance, situation awareness (SA), trust, and workload are measured. The Intelligent robot condition resulted in higher levels of performance and operator SA.

Synthetic Teammate Communication and Coordination With Humans

A synthetic teammate based on ACT-R cognitive architecture has been developed to function as an Air Vehicle Operator in the context of a three-agent Unmanned Aerial Vehicle (UAV) ground control team taking part in studies in a Synthetic Task Environment (STE). In order for the synthetic teammate to function as team player with human teammates, it needs to skillfully handle the subtleties of team communication and coordination. Data from early synthetic teammate interactions with two human teammates are presented here to illustrate team communication and coordination challenges for the synthetic teammate. In turn, the synthetic teammate limitations have highlighted the intricacies involved in effective teamwork. Communication, though a terrifically challenging problem in itself, is only a foundation for coordinated teamwork or interacting as a team player.

Measuring Speech Flow of Co-Located and Distributed Command and Control Teams During a Communication Channel Glitch

Team cognition can be observed in the flow of communications among team members. This is shown in the context of a simulated unmanned aerial vehicle ground control station. Automatic measures of low-level team communication flow were used to assess high-level constructs of team cognition. Measures show support for the expected results of manipulations in this task. Co-location and channel degradation effects were successfully predicted by CHUMS, ProNet, and a cross-correlation function-based Dominance measure. Results grant concurrent validity to the measures, and highlight substantive effects of the manipulations. In particular, in geographically distributed teams, communication patterns are less stable, and the route planner exerts less communicative influence. Some co-location effects drop with task experience. During a mission containing a five-minute one-way communication channel cut, all teams communicate more like distributed teams, and team members do create alternate pathways to retain information flow.

Changes in Team Composition after a Break: Building Adapative Command-and-Control Teams

An experiment exploring the effects of team composition on the acquisition and retention of team performance and cognitive skill is reported. Team performance was measured in the context of photographing ground targets in an unmanned aerial vehicle synthetic task environment. Team process was taken as a measure of team cognition. Experimental results include the findings that team mixing and longer retention intervals have a short lived deleterious effect on team performance immediately after the interval, while team mixing has a positive effect on team process after the interval. These findings suggest that changes in team composition and retention interval can lead to improvements in team cognition if a brief decrement in team performance post-interval can be afforded. These results are interpreted as perturbation of established coordination patterns due to team mixing leading to more flexible and adaptive teams. Implications for process-oriented research are also considered.

Coordinated Awareness of Situation by Teams (CAST): Measuring Team Situation Awareness of a Communication Glitch

A coordination-based measure of team situation awareness is presented and contrasted with knowledge-based measurement. The measure is applied to team awareness of a communication channel failure (glitch) during a simulated unmanned air vehicle reconnaissance experiment. Experimental results are reported, including the findings that not all team members should be identically aware of the glitch and that appropriate levels of coordination are an important precursor of team situation awareness. The results are discussed in terms of the application of coordination metrics to support the understanding of team situation awareness. The use of team coordination as a low-dimension variable of team functionality is scalable over a variety of team sizes and expertise distributions.

Control of an Uninhabited Air Vehicle: A Synthetic Task Environment for Teams

A synthetic team task is presented in which a three-person team controls a UAV (Uninhabited Air Vehicle) in the context of a reconnaissance mission. This synthetic task has been designed to provide a flexible research platform, yet realistic task environment, for team performance research and the measurement of team cognition.

Some Promising Results of Communication-Based Automatic Measures of Team Cognition

Some have argued that the most appropriate measure of team cognition is a holistic measure directed at the entire team. In particular, communication data are useful for measuring team cognition because of the holistic nature of the data, and because of the connection between communication and declarative cognition. In order to circumvent the logistic difficulties of communication data, the present paper proposes several relatively automatic methods of analysis. Four data types are identified, with low-level physical data vs. content data being one dimension, and sequential vs. static data being the other. Methods addressing all four of these data types are proposed, with the exception of static physical data. Latent Semantic Analysis is an automatic method used to assess content, either statically or sequentially. PRONET is useful to address either physical or content-based sequential data, and we propose CHUMS to address sequential physical data. The usefulness of each method to predict team performance data is assessed.

Broadening the Measurement of Situation Awareness through Cognitive Engineering Methods

Situation awareness (SA) and team SA are popular concepts, yet vaguely defined and inadequately measured. They involve representations of the current situation, performance resulting from those representations, and cognitive structures and processes leading to those representations. Current measures of individual and team SA focus on the assessment of performance or the accuracy of the resulting situation model at the expense of other aspects of SA, such as situation assessment, mental models, and team process behaviors. As a result, these measures fail to capture the richness of the constructs of individual and team SA, critical for applications involving training and team SA. We propose that a cognitive engineering approach to measuring SA which focuses on the elicitation of the cognition underlying SA, can extend measurement by overcoming many of the current limits. As an illustration, the measurement of situation models using this approach is presented.

Evaluation of Latent Semantic Analysis-Based Measures of Team Communications Content

Team process is thought to mediate team member inputs and team performance. Among the team behaviors identified as process variables, team communications have been widely studied. We view team communications as a team behavior and also as team information processing, or team cognition. Within the context of a Predator Uninhabited Air Vehicle (UAV) synthetic task, we have developed several methods of communications content assessment based on Latent Semantic Analysis (LSA). These methods include: Communications Density (CD) which is the average task relevance of a team's communications, Lag Coherence (LC) which measures task-relevant topic shifting over UAV missions, and Automatic Tagging (AT) which categorizes team communications. Each method is described in detail. CD and LC are related to UAV team performance. AT-human is comparable to human-human agreement on content coding. The results are promising for the assessment of teams based on LSA applied to communication content.

A Synthetic Task Environment for Team Cognition Research

A STE (Synthetic Task Environment) provides the context for a suite of synthetic tasks in which skills pertinent to the corresponding real-world task can be exercised in a controlled setting, thus bridging the gap between lab and field studies. The UAV (Uninhabited Air Vehicle)-STE described in this paper was designed to provide a flexible research platform, yet realistic task environment, for team performance research and the measurement of team cognition. The abstraction of task features from the real UAV task environment occurred with these objectives serving as a filter. In the resulting UAV-STE, three team members work together to control and navigate the UAV and to take photographs of designated targets. Whereas, there are similarities between the synthetic and real UAV task environments, the differences reflect the objectives in that the UAV-STE exaggerates team cognition, provides task and measurement flexibility, and allows for relatively rapid skill acquisition by inexperienced participants.

Exploring Measures of Team Knowledge

Team cognition is more than the aggregate cognition of team members. It is an emerging feature, worthy of study in its own right. In this paper we investigate potential metrics of team knowledge in the context of a broader exploratory study on measures of team knowledge, performance, and situation awareness. Team members assumed different roles in a three-person synthetic task in which they were presented with unique role-relevant information. Successful accomplishment of team objectives required team members to share information. The focus of this paper is on one of several measures collected which required judgments of pairwise relatedness ratings for mission-relevant terms. These data were submitted to Pathfinder network scaling and used to derive three metrics of team knowledge: knowledge accuracy, interpositional knowledge, and knowledge similarity. The metrics revealed different perspectives on team knowledge and were generally predictive of team performance and team situation awareness.

Team Cognition As a Means to Improve Care Delivery in Critically Ill Patients With Cancer After Hematopoietic Cell Transplantation

Hematopoietic cell transplantation (HCT) is an important and complex treatment modality for a variety of hematologic malignancies and some solid tumors. Although outcomes of patients who have undergone HCT and require care in intensive care units (ICUs) have improved over time, mortality rates remain high and there are significant associated costs. Lack of a team-based approach to care, especially during critical illness, is detrimental to patient autonomy and satisfaction, and to team morale, ultimately leading to poor quality of care. In this manuscript, we describe the case of a patient who had undergone HCT and was in the ICU setting, where inconsistent team interaction among the various stakeholders delivering care resulted in a lack of shared goals and poor outcomes. Team cognition is cognitive processing at the team level through interactions among team members and is reflected in dynamic communication and coordination behaviors. Although the patient received multidisciplinary care as needed in a medically complicated case, a lack of team cognition and, particularly, inconsistent communication among the dynamic teams caring for the patient, led to mixed messages being delivered with high-cost implications for the health-care system and the family. This article highlights concepts and recommendations that begin a necessary in-depth assessment of implications for clinical care and initiate a research agenda that examines the effects of team cognition on HCT teams, and, more generally, critical care of the patient with cancer.

The Synthetic Teammate as a Team Player in Command-and-Control Teams

Project overview. The current project is part of a larger effort that focuses on Human-Automation Teaming (HAT) interaction in the context of the development, integration, and validation of a computational cognitive model that acts as a full-fledged synthetic teammate for a three-agent Unmanned Aircraft System (UAS) ground control crew. Our most recent effort looked at team process and team performance within the HAT. In order to be considered a team player, the synthetic teammate must be able to communicate and coordinate with its human teammates and do so in a subtle manner (Demir et al., 2016). In this task, there were three different and interdependent team members: 1) Air Vehicle Operator (AVO) – controls the UAS’s heading, altitude, and airspeed; 2) Data Exploitation, Mission Planning, and Communications (DEMPC) – provides a dynamic flight plan as well as speed and altitude restrictions; and 3) Payload Operator (PLO) – monitors sensor equipment, negotiates with the AVO, and takes photographs of target waypoints. The communication within a three-agent UAS team occurred over a text-based communications system. In this research, there were three conditions which are differentiated by the AVO role: 1) the Synthetic - the synthetic teammate was assigned the AVO role; 2) the Control - the AVO was an inexperienced human participant; 3) the Experimenter - the AVO was one of the experimenters who was experienced with the task. The experimenter AVO asked questions of other team members to ensure timely and adaptive passing of information at target waypoints. In this current study, the coordination among the team members occurs at each target waypoint and requires a specific sequence of information passing for an optimum team performance (Cooke, Gorman, Duran, & Taylor, 2007): the information is provided by the DEMPC about the upcoming target waypoint to the AVO. After that, the PLO and the AVO negotiate regarding an appropriate altitude and airspeed for the target waypoints about required camera settings. Finally, the PLO sends feedback to other team members about the status of the target photo.

Method. Activities during this period included conducting an experiment to: 1) evaluate the synthetic teammate’s performance, and the HAT team performance in comparison to all human teams, 2) understand how team process differs between all human and human-synthetic teams and how this impacts performance, and 3) compare the human-synthetic teams and all human control teams to a team with a pilot that is experienced in pushing and pulling information across the team. For this experiment, participants were randomly assigned for the duration of the experiment. Within each of the five missions, teams were told to obtain as many “good” photos as possible while avoiding alarms and rule violations in less than 40 minutes. The overall focus of this paper is: team process that is comprised of eight verbal behaviors associated with team effectiveness; team performance that is a combination of mission variables, including the rate of successful target photographs, time spent in alarm and warning states (for each individual), and the critical waypoint acquisition rate; and target processing efficiency took into account the time spent inside a target waypoint to get a good photo.

Results and discussion. In general, findings indicate that synthetic AVOs perform more poorly than control AVOs in terms of team performance. Synthetic teams perform as well at the mission level as control teams. However, in terms of target processing efficiency, synthetic teams perform poorer than control teams. In terms of team process, synthetic teams demonstrate interaction patterns corresponding to more pulling of information than pushing with little change over time. In summary, these results indicate that there is a strong potential for using synthetic team member as a teammate in real world tasks and for training.

Team Cognition as Interaction

Teams perform cognitive activities such as making decisions and assessing situations as a unit. The team cognition behind these activities has traditionally been linked to individual knowledge and its distribution across team members. The theory of interactive team cognition instead argues that team cognition resides in team interactions and that it is an activity that takes place in a rich context that needs to be measured at the team level. This article describes this dynamic perspective on team cognition, some research that supports it, and the implications for measuring, understanding, and improving team cognition.

Defining the methodological challenges and opportunities for an effective science of sociotechnical systems and safety

An important part of the application of sociotechnical systems theory (STS) is the development of methods, tools and techniques to assess human factors and ergonomics workplace requirements. We focus in this paper on describing and evaluating current STS methods for workplace safety, as well as outlining a set of six case studies covering the application of these methods to a range of safety contexts. We also describe an evaluation of the methods in terms of ratings of their ability to address a set of theoretical and practical questions (e.g. the degree to which methods capture static/dynamic aspects of tasks and interactions between system levels). The outcomes from the evaluation highlight a set of gaps relating to the coverage and applicability of current methods for STS and safety (e.g. coverage of external influences on system functioning; method usability). The final sections of the paper describe a set of future challenges, as well as some practical suggestions for tackling these.

PRACTITIONER SUMMARY

We provide an up-to-date review of STS methods, a set of case studies illustrating their use and an evaluation of their strengths and weaknesses. The paper concludes with a 'roadmap' for future work.

Agent-Based Model of a Cyber Security Defense Analyst Team

Cyber security defense is often performed by a group of people called cyber defense analysts and yet team work and collaboration in cyber defense is almost non-existent. This study, using an agent-based model of the cyber defense analyst’s task and interactions, explored the effects of different collaboration strategies and team sizes on performance measures such as number of intrusion alerts accurately processed by the analysts and rewards they accrue from accurately processing the alerts. This study also explored the feasibility of using agent-based modeling methodologies for studying team processes in the cyber defense context. The model revealed that specific collaboration strategies lead to better performance and that large teams are detrimental to performance.

Perspectives on the Role of Cognition in Cyber Security

The cyber security task is an intensely cognitive task that is embedded in a large multi-layered sociotechnical system of analysts, computers, and networks. Effective performance in this world is hampered by enormous size and complexity of the network data, the adaptive nature of intelligent adversaries, the lack of ground truth to assess performance, the high number of false alarms presented by automated alerting systems, by organizational stove pipes thwarting collaboration, and by technology that is thrown at the problem without an adequate understanding of the human needs. Further, the consequences of effective system performance in the cyber security domain are unparalleled because our world is so dependent on its cyber infrastructure. We have assembled a panel of six experts in cognitive engineering to provide perspectives on the cyber security problem and promising solutions.

Modeling the Complex Dynamics of Teamwork from Team Cognition to Neurophysiology

Teamwork is a complex dynamic process that emerges from team member interaction. The dynamics provide a characterization of the team over time. The stability, flexibility, and resilience of team dynamics over various windows of time can change with experience, training, environmental perturbations, and technological intervention. Once patterns of team dynamics have been established for a particular team, anomalous dynamics can signify impending teamwork problems. Dynamical systems modeling has been applied to many physical and natural systems and has only recently been applied to teamwork. The panel that we have assembled has applied dynamical modeling to this problem from different perspectives and at different scales. Each panelist will overview his or her approach to this problem. We will then discuss pros and cons of each approach and possibilities for using them in a complementary fashion.

Influence of Team Communication and Coordination on the Performance of Teams at the iCTF Competition

Effective team process is critical for the performance of cyber security teams. To examine this, we observed two comparably skilled cyber security teams participating in the International Capture the Flag (iCTF) competition held in December 2011. At the conclusion of the competition, we followed up with a focus group discussion with six members from the two teams. In this paper, we present our findings from the focus group interviews, on the relationship between team level factors and team performance. Findings from the focus group discussion indicate that team level factors such as team communication, coordination, team structure, and leadership play important roles in team performance.

Measuring patterns in team interaction sequences using a discrete recurrence approach

OBJECTIVE

Recurrence-based measures of communication determinism and pattern information are described and validated using previously collected team interaction data.

BACKGROUND

Team coordination dynamics has revealed that"mixing" team membership can lead to flexible interaction processes, but keeping a team "intact" can lead to rigid interaction processes. We hypothesized that communication of intact teams would have greater determinism and higher pattern information compared to that of mixed teams.

METHOD

Determinism and pattern information were measured from three-person Uninhabited Air Vehicle team communication sequences over a series of 40-minute missions. Because team members communicated using push-to-talk buttons, communication sequences were automatically generated during each mission.

RESULTS

The Composition x Mission determinism effect was significant. Intact teams' determinism increased over missions, whereas mixed teams' determinism did not change. Intact teams had significantly higher maximum pattern information than mixed teams.

CONCLUSION

Results from these new communication analysis methods converge with content-based methods and support our hypotheses.

APPLICATION

Because they are not content based, and because they are automatic and fast, these new methods may be amenable to real-time communication pattern analysis.

Dynamical analysis in real time: detecting perturbations to team communication

Dynamical systems methods characterise patterns of change over time. Typically, such methods are applied only after data collection is complete. However, brief disturbances - perturbations - can occur as a process unfolds and can result in undesirable outcomes if not acted on. The application of dynamics in real time would be useful for detecting these sudden changes. Real-time analysis was accomplished by updating dynamical estimates simultaneously across different window sizes. We calculated the largest Lyapunov exponent, a measure of dynamical stability, to detect a perturbation to team communication in a simulated uninhabited air vehicle (UAV) reconnaissance mission. The perturbation consisted of information demands from a confederate that occurred unexpectedly during performance of a UAV mission. We demonstrate the use of real-time methods in detecting that perturbation as it occurred. In application, this technique would have enabled real-time intervention. Extensions of the real-time dynamical method to other domains of psychological inquiry are discussed.

PRACTITIONER SUMMARY

A real-time dynamical analysis method that was developed to detect unexpected perturbations in team communication is described. The use of the method is demonstrated on perturbed communication from a three-person uninhabited air vehicle command-and-control team. The generalisability of the method is considered with respect to physiological and motor coordination dynamics.

Team coordination dynamics

Team coordination consists of both the dynamics of team member interaction and the environmental dynamics to which a team is subjected. Focusing on dynamics, an approach is developed that contrasts with traditional aggregate-static concepts of team coordination as characterized by the shared mental model approach. A team coordination order parameter was developed to capture momentary fluctuations in coordination. Team coordination was observed in three-person uninhabited air vehicle teams across two experimental sessions. The dynamics of the order parameter were observed under changes of a team familiarity control parameter. Team members returned for the second session to either the same (Intact) or different (Mixed) team. 'Roadblock' perturbations, or novel changes in the task environment, were introduced in order to probe the stability of team coordination. Nonlinear dynamic methods revealed differences that a traditional approach did not: Intact and Mixed team coordination dynamics looked very different; Mixed teams were more stable than Intact teams and explored the space of solutions without the need for correction. Stability was positively correlated with the number of roadblock perturbations that were overcome successfully. The novel and non-intuitive contribution of a dynamical analysis was that Mixed teams, who did not have a long history working together, were more adaptive. Team coordination dynamics carries new implications for traditional problems such as training adaptive teams.

Training adaptive teams

OBJECTIVE

We report an experiment in which three training approaches are compared with the goal of training adaptive teams.

BACKGROUND

Cross-training is an established method in which team members are trained with the goal of building shared knowledge. Perturbation training is a new method in which team interactions are constrained to provide new coordination experiences during task acquisition. These two approaches, and a more traditional procedural approach, are compared.

METHOD

Assigned to three training conditions were 26 teams. Teams flew nine simulated uninhabited air vehicle missions; three were critical tests of the team's ability to adapt to novel situations. Team performance, response time to novel events, and shared knowledge were measured.

RESULTS

Perturbation-trained teams significantly outperformed teams in the other conditions in two out of three critical test missions. Cross-training resulted in significant increases in shared teamwork knowledge and highest mean performance in one critical test. Procedural training led to the least adaptive teams.

CONCLUSION

Perturbation training allows teams to match coordination variability during training to demands for coordination variability during posttraining performance. Although cross-training has adaptive benefits, it is suggested that process-oriented approaches, such as perturbation training, can lead to more adaptive teams.

APPLICATION

Perturbation training is amenable to simulation-based training, where perturbations provide interaction experiences that teams can transfer to novel, real-world situations.

On teams, teamwork, and team performance: discoveries and developments

OBJECTIVE

We highlight some of the key discoveries and developments in the area of team performance over the past 50 years, especially as reflected in the pages of Human Factors.

BACKGROUND

Teams increasingly have become a way of life in many organizations, and research has kept up with the pace.

METHOD

We have characterized progress in the field in terms of eight discoveries and five challenges.

RESULTS

Discoveries pertain to the importance of shared cognition, the measurement of shared cognition, advances in team training, the use of synthetic task environments for research, factors influencing team effectiveness, models of team effectiveness, a multidisciplinary perspective, and training and technological interventions designed to improve team effectiveness. Challenges that are faced in the coming decades include an increased emphasis on team cognition; reconfigurable, adaptive teams; multicultural influences; and the need for naturalistic study and better measurement.

CONCLUSION

Work in human factors has contributed significantly to the science and practice of teams, teamwork, and team performance. Future work must keep pace with the increasing use of teams in organizations.

APPLICATION

The science of teams contributes to team effectiveness in the same way that the science of individual performance contributes to individual effectiveness.