Reliability and validity of findings in ergonomics research

Testing the reliability of accident analysis methods: a comparison of AcciMap, STAMP-CAST and AcciNet

Accident analysis methods are used to model the multifactorial cause of adverse incidents. Methods such as AcciMap, STAMP-CAST and recently AcciNet, are systemic approaches that support the identification of safety interventions across sociotechnical system levels. Despite their growing popularity, little is known about how reliable systems-based methods are when used to describe, model and classify contributory factors and relationships. Here, we conducted an intra-rater and inter-rater reliability assessment of AcciMap, STAMP-CAST and AcciNet using the Signal Detection Theory (SDT) paradigm. A total of 180 hours' worth of analyses across 360 comparisons were performed by 30 expert analysts. Findings revealed that all three methods produced a weak to moderate positive correlation coefficient, however the inter-rater reliability of STAMP-CAST was significantly higher compared to AcciMap and AcciNet. No statistically significant or practically meaningful differences were found between methods in the overall intra-rater reliability analyses. Implications and future research directions are discussed.

Testing the reliability and validity of risk assessment methods in Human Factors and Ergonomics

There is growing interest in the use of systems-based risk assessment methods in Human Factors and Ergonomics (HFE). The purpose of this study was to test the intra-rater reliability and criterion-referenced concurrent validity of three systems-based risk assessment approaches: (i) the Systems-Theoretic Process Analysis (STPA) method; (ii) the Event Analysis of Systemic Teamwork Broken Links (EAST-BL) method; and, (iii) the Network Hazard Analysis and Risk Management System (Net-HARMS) method. Reliability and validity measures were obtained using the Signal Detection Theory (SDT) paradigm. Whilst STPA identified the highest number of risks, the findings indicate a weak to moderate level of reliability and validity for STPA, EAST-BL and Net-HARMS. There were no statistically significant differences between the methods across analyses. The results suggest that there is merit to the continued use of systems-based risk assessment methods following a series of methodological extensions that aim to enhance the reliability and validity of future applications. Practitioner summary The three risk assessment methods produced weak to moderate levels of stability and accuracy regarding their capability to predict risks. There is a pressing need to further test the reliability and validity of safety methods in Human Factors and Ergonomics.

Making ergonomics accountable: Reliability, validity and utility in ergonomics methods

In this paper, we discuss the ways in which Neville Stanton has challenged himself, his research colleagues, PhD students, the many co-authors and contributors to his publications, and the entire Ergonomics community to determine what it means for there to be 'consistent standards for how [Ergonomics] methods are described and reported.' Only in this way, can it be possible to make claims about whether or not a method in Ergonomics is effective. Given that he is Chartered as both an Occupational Psychologist and an Ergonomist, it is not surprising that he has been concerned with the question of the reliability and validity of Ergonomics methods. In Occupational Psychology, psychometric and personnel selection methods are expected to exhibit acceptable levels of reliability, but this is an expectation which is still somewhat alien to Ergonomics. Neville's work has been instrumental in raising this issue and in providing approaches which can be used to critically evaluate the methods we use. We think that, despite his ground-breaking work, there is still much to do in the Ergonomics community to create the situation for which he has long argued.

Evaluation of construct and criterion-referenced validity of a systems-thinking based near miss reporting form

The validity of methods is an ongoing issue in ergonomics. Inconsistent definitions and approaches to evaluation exacerbate this challenge. In this study, the construct and criterion-referenced validity of a new near miss reporting form was evaluated to determine the extent to which it comprehensively captures near miss incidents and is aligned with the systems thinking approach to accident causation. Interview data were used as the reference standard in the evaluation. Using signal detection theory (SDT), a high average hit rate (HR), predictive value (PV) and sensitivity index (SI) were found, with an almost perfect ranking for the index of concordance. The findings show that the reporting form has strong construct and criterion-referenced validity. It is proposed that the approach used in this study could be used by researchers and practitioners when testing the validity of incident data collection tools. Practitioner summary: The validity of methods is a key issue in ergonomics. In this study, we test the validity of a near miss reporting form using interview data as a standard. This approach could be used by practitioners when testing the validity of other ergonomics methods.

Applying systems ergonomics methods in sport: A systematic review

INTRODUCTION

As sports systems become increasingly more complex, competitive, and technology-centric, there is a greater need for systems ergonomics methods to consider the performance, health, and safety of athletes in context with the wider settings in which they operate. Therefore, the purpose of this systematic review was to identify and critically evaluate studies which have applied a systems ergonomics research approach in the context of sports performance and injury management.

MATERIAL AND METHODS

Five databases (PubMed, Scopus, ScienceDirect, Web of Science, and SPORTDiscus) were searched for the dates 01 January 1990 to 01 August 2017, inclusive, for original peer-reviewed journal articles and conference papers. Reported analyses were underpinned by a recognised systems ergonomics method, and study aims were related to the optimisation of sports performance (e.g. communication, playing style, technique, tactics, or equipment), and/or the management of sports injury (i.e. identification, prevention, or treatment).

RESULTS

A total of seven articles were identified. Two articles were focussed on understanding and optimising sports performance, whereas five examined sports injury management. The methods used were the Event Analysis of Systemic Teamwork, Cognitive Work Analysis (the Work Domain Analysis Abstraction Hierarchy), Rasmussen's Risk Management Framework, and the Systems Theoretic Accident Model and Processes method. The individual sport application was distance running, whereas the team sports contexts examined were cycling, football, Australian Football League, and rugby union.

CONCLUSIONS

The included systems ergonomics applications were highly flexible, covering both amateur and elite sports contexts. The studies were rated as valuable, providing descriptions of injury controls and causation, the factors influencing injury management, the allocation of responsibilities for injury prevention, as well as the factors and their interactions underpinning sports performance. Implications and future directions for research are described.

'Remixing Rasmussen': The evolution of Accimaps within systemic accident analysis

Throughout Jens Rasmussen's career there has been a continued emphasis on the development of methods, techniques and tools for accident analysis and investigation. In this paper we focus on the evolution and development of one specific example, namely Accimaps and their use for accident analysis. We describe the origins of Accimaps followed by a review of 27 studies which have applied and adapted Accimaps over the period 2000-2015 to a range of domains and types of accident. Aside from demonstrating the versatility and popularity of the method, part of the motivation for the review of the use of Accimaps is to address the question of what constitutes a sound, usable, valid and reliable approach to systemic accident analysis. The findings from the review demonstrate continuity with the work carried out by Rasmussen, as well as significant variation (e.g., changes to the Accimap, used of additional theoretical and practice-oriented perspectives on safety). We conclude the paper with some speculations regarding future extension and adaptation of the Accimap approach including the possibility of using hybrid models for accident analysis.

Empirical evaluation of the Process Overview Measure for assessing situation awareness in process plants

The Process Overview Measure is a query-based measure developed to assess operator situation awareness (SA) from monitoring process plants. A companion paper describes how the measure has been developed according to process plant properties and operator cognitive work. The Process Overview Measure demonstrated practicality, sensitivity, validity and reliability in two full-scope simulator experiments investigating dramatically different operational concepts. Practicality was assessed based on qualitative feedback of participants and researchers. The Process Overview Measure demonstrated sensitivity and validity by revealing significant effects of experimental manipulations that corroborated with other empirical results. The measure also demonstrated adequate inter-rater reliability and practicality for measuring SA in full-scope simulator settings based on data collected on process experts. Thus, full-scope simulator studies can employ the Process Overview Measure to reveal the impact of new control room technology and operational concepts on monitoring process plants. Practitioner Summary: The Process Overview Measure is a query-based measure that demonstrated practicality, sensitivity, validity and reliability for assessing operator situation awareness (SA) from monitoring process plants in representative settings.