1
|
Hulme A, Stanton NA, Walker GH, Waterson P, Salmon PM. Testing the reliability of accident analysis methods: a comparison of AcciMap, STAMP-CAST and AcciNet. ERGONOMICS 2024; 67:695-715. [PMID: 37523211 DOI: 10.1080/00140139.2023.2240048] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Accepted: 07/13/2023] [Indexed: 08/01/2023]
Abstract
Accident analysis methods are used to model the multifactorial cause of adverse incidents. Methods such as AcciMap, STAMP-CAST and recently AcciNet, are systemic approaches that support the identification of safety interventions across sociotechnical system levels. Despite their growing popularity, little is known about how reliable systems-based methods are when used to describe, model and classify contributory factors and relationships. Here, we conducted an intra-rater and inter-rater reliability assessment of AcciMap, STAMP-CAST and AcciNet using the Signal Detection Theory (SDT) paradigm. A total of 180 hours' worth of analyses across 360 comparisons were performed by 30 expert analysts. Findings revealed that all three methods produced a weak to moderate positive correlation coefficient, however the inter-rater reliability of STAMP-CAST was significantly higher compared to AcciMap and AcciNet. No statistically significant or practically meaningful differences were found between methods in the overall intra-rater reliability analyses. Implications and future research directions are discussed.
Collapse
Affiliation(s)
- Adam Hulme
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Australia
- Southern Queensland Rural Health (SQRH), Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, Australia
| | - Neville A Stanton
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Australia
- Transportation Research Group, Faculty of Engineering and Physical Sciences, University of Southampton, Southampton, UK
| | - Guy H Walker
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Australia
- Centre for Sustainable Road Freight, School of Energy, Geoscience, Infrastructure and Society, Heriot-Watt University, Scotland, UK
| | - Patrick Waterson
- Human Factors and Complex Systems Group, Design School, Loughborough University, Leicestershire, UK
| | - Paul M Salmon
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Australia
| |
Collapse
|
2
|
Hulme A, Stanton NA, Walker GH, Waterson P, Salmon PM. Testing the reliability and validity of risk assessment methods in Human Factors and Ergonomics. ERGONOMICS 2022; 65:407-428. [PMID: 34328389 DOI: 10.1080/00140139.2021.1962969] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/08/2020] [Accepted: 07/27/2021] [Indexed: 06/13/2023]
Abstract
There is growing interest in the use of systems-based risk assessment methods in Human Factors and Ergonomics (HFE). The purpose of this study was to test the intra-rater reliability and criterion-referenced concurrent validity of three systems-based risk assessment approaches: (i) the Systems-Theoretic Process Analysis (STPA) method; (ii) the Event Analysis of Systemic Teamwork Broken Links (EAST-BL) method; and, (iii) the Network Hazard Analysis and Risk Management System (Net-HARMS) method. Reliability and validity measures were obtained using the Signal Detection Theory (SDT) paradigm. Whilst STPA identified the highest number of risks, the findings indicate a weak to moderate level of reliability and validity for STPA, EAST-BL and Net-HARMS. There were no statistically significant differences between the methods across analyses. The results suggest that there is merit to the continued use of systems-based risk assessment methods following a series of methodological extensions that aim to enhance the reliability and validity of future applications. Practitioner summary The three risk assessment methods produced weak to moderate levels of stability and accuracy regarding their capability to predict risks. There is a pressing need to further test the reliability and validity of safety methods in Human Factors and Ergonomics.
Collapse
Affiliation(s)
- Adam Hulme
- Centre for Human Factors and Sociotechnical Systems, University of the Sunshine Coast, Sippy Downs, Australia
| | - Neville A Stanton
- Centre for Human Factors and Sociotechnical Systems, University of the Sunshine Coast, Sippy Downs, Australia
- Transportation Research Group, Faculty of Engineering and Physical Sciences, University of Southampton, Southampton, UK
| | - Guy H Walker
- Centre for Human Factors and Sociotechnical Systems, University of the Sunshine Coast, Sippy Downs, Australia
- Centre for Sustainable Road Freight, School of Energy, Geoscience, Infrastructure and Society, Heriot-Watt University, Scotland, UK
| | - Patrick Waterson
- Human Factors and Complex Systems Group, Design School, Loughborough University, Leicestershire, UK
| | - Paul M Salmon
- Centre for Human Factors and Sociotechnical Systems, University of the Sunshine Coast, Sippy Downs, Australia
- Transportation Research Group, Faculty of Engineering and Physical Sciences, University of Southampton, Southampton, UK
| |
Collapse
|
3
|
Baber C, Young MS. Making ergonomics accountable: Reliability, validity and utility in ergonomics methods. APPLIED ERGONOMICS 2022; 98:103583. [PMID: 34634582 DOI: 10.1016/j.apergo.2021.103583] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/16/2021] [Revised: 08/16/2021] [Accepted: 09/07/2021] [Indexed: 06/13/2023]
Abstract
In this paper, we discuss the ways in which Neville Stanton has challenged himself, his research colleagues, PhD students, the many co-authors and contributors to his publications, and the entire Ergonomics community to determine what it means for there to be 'consistent standards for how [Ergonomics] methods are described and reported.' Only in this way, can it be possible to make claims about whether or not a method in Ergonomics is effective. Given that he is Chartered as both an Occupational Psychologist and an Ergonomist, it is not surprising that he has been concerned with the question of the reliability and validity of Ergonomics methods. In Occupational Psychology, psychometric and personnel selection methods are expected to exhibit acceptable levels of reliability, but this is an expectation which is still somewhat alien to Ergonomics. Neville's work has been instrumental in raising this issue and in providing approaches which can be used to critically evaluate the methods we use. We think that, despite his ground-breaking work, there is still much to do in the Ergonomics community to create the situation for which he has long argued.
Collapse
Affiliation(s)
- Chris Baber
- School of Computer Science, University of Birmingham, Birmingham, West Midlands, B15 2TT, UK.
| | - Mark S Young
- Human Factors and Complex Systems Group, School of Design and Creative Arts, Loughborough University, Loughborough, Leicestershire, LE11 3TU, UK.
| |
Collapse
|
4
|
Thoroman B, Salmon P, Goode N. Evaluation of construct and criterion-referenced validity of a systems-thinking based near miss reporting form. ERGONOMICS 2020; 63:210-224. [PMID: 31738666 DOI: 10.1080/00140139.2019.1694707] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/16/2019] [Accepted: 11/12/2019] [Indexed: 06/10/2023]
Abstract
The validity of methods is an ongoing issue in ergonomics. Inconsistent definitions and approaches to evaluation exacerbate this challenge. In this study, the construct and criterion-referenced validity of a new near miss reporting form was evaluated to determine the extent to which it comprehensively captures near miss incidents and is aligned with the systems thinking approach to accident causation. Interview data were used as the reference standard in the evaluation. Using signal detection theory (SDT), a high average hit rate (HR), predictive value (PV) and sensitivity index (SI) were found, with an almost perfect ranking for the index of concordance. The findings show that the reporting form has strong construct and criterion-referenced validity. It is proposed that the approach used in this study could be used by researchers and practitioners when testing the validity of incident data collection tools. Practitioner summary: The validity of methods is a key issue in ergonomics. In this study, we test the validity of a near miss reporting form using interview data as a standard. This approach could be used by practitioners when testing the validity of other ergonomics methods.
Collapse
Affiliation(s)
- Brian Thoroman
- Faculty of Arts and Business, Centre for Human Factors and Sociotechnical Systems, University of the Sunshine Coast, Maroochydore, Australia
| | - Paul Salmon
- Faculty of Arts and Business, Centre for Human Factors and Sociotechnical Systems, University of the Sunshine Coast, Maroochydore, Australia
| | | |
Collapse
|
5
|
Hulme A, Thompson J, Plant KL, Read GJM, Mclean S, Clacy A, Salmon PM. Applying systems ergonomics methods in sport: A systematic review. APPLIED ERGONOMICS 2019; 80:214-225. [PMID: 29674008 DOI: 10.1016/j.apergo.2018.03.019] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/15/2017] [Revised: 01/29/2018] [Accepted: 03/31/2018] [Indexed: 06/08/2023]
Abstract
INTRODUCTION As sports systems become increasingly more complex, competitive, and technology-centric, there is a greater need for systems ergonomics methods to consider the performance, health, and safety of athletes in context with the wider settings in which they operate. Therefore, the purpose of this systematic review was to identify and critically evaluate studies which have applied a systems ergonomics research approach in the context of sports performance and injury management. MATERIAL AND METHODS Five databases (PubMed, Scopus, ScienceDirect, Web of Science, and SPORTDiscus) were searched for the dates 01 January 1990 to 01 August 2017, inclusive, for original peer-reviewed journal articles and conference papers. Reported analyses were underpinned by a recognised systems ergonomics method, and study aims were related to the optimisation of sports performance (e.g. communication, playing style, technique, tactics, or equipment), and/or the management of sports injury (i.e. identification, prevention, or treatment). RESULTS A total of seven articles were identified. Two articles were focussed on understanding and optimising sports performance, whereas five examined sports injury management. The methods used were the Event Analysis of Systemic Teamwork, Cognitive Work Analysis (the Work Domain Analysis Abstraction Hierarchy), Rasmussen's Risk Management Framework, and the Systems Theoretic Accident Model and Processes method. The individual sport application was distance running, whereas the team sports contexts examined were cycling, football, Australian Football League, and rugby union. CONCLUSIONS The included systems ergonomics applications were highly flexible, covering both amateur and elite sports contexts. The studies were rated as valuable, providing descriptions of injury controls and causation, the factors influencing injury management, the allocation of responsibilities for injury prevention, as well as the factors and their interactions underpinning sports performance. Implications and future directions for research are described.
Collapse
Affiliation(s)
- Adam Hulme
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Queensland, 4558, Australia.
| | - Jason Thompson
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Queensland, 4558, Australia; Transport, Health and Urban Design (THUD) Research Hub, Faculty of Architecture, Building and Planning, Melbourne School of Design, The University of Melbourne, Melbourne, Victoria, 3010, Australia
| | - Katherine L Plant
- Transportation Research Group, Faculty of Engineering and the Environment, University of Southampton, Southampton, SO17 1BJ, United Kingdom
| | - Gemma J M Read
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Queensland, 4558, Australia
| | - Scott Mclean
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Queensland, 4558, Australia
| | - Amanda Clacy
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Queensland, 4558, Australia
| | - Paul M Salmon
- Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Sippy Downs, Queensland, 4558, Australia
| |
Collapse
|
6
|
Waterson P, Jenkins DP, Salmon PM, Underwood P. 'Remixing Rasmussen': The evolution of Accimaps within systemic accident analysis. APPLIED ERGONOMICS 2017; 59:483-503. [PMID: 27659766 DOI: 10.1016/j.apergo.2016.09.004] [Citation(s) in RCA: 42] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/06/2015] [Revised: 12/29/2015] [Accepted: 09/11/2016] [Indexed: 06/06/2023]
Abstract
Throughout Jens Rasmussen's career there has been a continued emphasis on the development of methods, techniques and tools for accident analysis and investigation. In this paper we focus on the evolution and development of one specific example, namely Accimaps and their use for accident analysis. We describe the origins of Accimaps followed by a review of 27 studies which have applied and adapted Accimaps over the period 2000-2015 to a range of domains and types of accident. Aside from demonstrating the versatility and popularity of the method, part of the motivation for the review of the use of Accimaps is to address the question of what constitutes a sound, usable, valid and reliable approach to systemic accident analysis. The findings from the review demonstrate continuity with the work carried out by Rasmussen, as well as significant variation (e.g., changes to the Accimap, used of additional theoretical and practice-oriented perspectives on safety). We conclude the paper with some speculations regarding future extension and adaptation of the Accimap approach including the possibility of using hybrid models for accident analysis.
Collapse
Affiliation(s)
- Patrick Waterson
- Human Factors and Complex Systems Group, Loughborough University Design School, Loughborough University, Loughborough, LE11 3TU, United Kingdom.
| | - Daniel P Jenkins
- DCA Design International Ltd., 19 Church Street, Warwick, CV34 4AB, United Kingdom
| | - Paul M Salmon
- University of the Sunshine Coast Accident Research, University of the Sunshine Coast, Locked Bag 4, Maroochydore DC, QLD, 4558, Australia
| | - Peter Underwood
- Bunnyfoot, Harwell Innovation Centre, 173 Curie Avenue, Harwell, Oxfordshire, OX11 0QG, United Kingdom
| |
Collapse
|
7
|
Lau N, Jamieson GA, Skraaning G. Empirical evaluation of the Process Overview Measure for assessing situation awareness in process plants. ERGONOMICS 2016; 59:393-408. [PMID: 26398584 DOI: 10.1080/00140139.2015.1080310] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/05/2023]
Abstract
The Process Overview Measure is a query-based measure developed to assess operator situation awareness (SA) from monitoring process plants. A companion paper describes how the measure has been developed according to process plant properties and operator cognitive work. The Process Overview Measure demonstrated practicality, sensitivity, validity and reliability in two full-scope simulator experiments investigating dramatically different operational concepts. Practicality was assessed based on qualitative feedback of participants and researchers. The Process Overview Measure demonstrated sensitivity and validity by revealing significant effects of experimental manipulations that corroborated with other empirical results. The measure also demonstrated adequate inter-rater reliability and practicality for measuring SA in full-scope simulator settings based on data collected on process experts. Thus, full-scope simulator studies can employ the Process Overview Measure to reveal the impact of new control room technology and operational concepts on monitoring process plants. Practitioner Summary: The Process Overview Measure is a query-based measure that demonstrated practicality, sensitivity, validity and reliability for assessing operator situation awareness (SA) from monitoring process plants in representative settings.
Collapse
Affiliation(s)
- Nathan Lau
- a Grado Department of Industrial & Systems Engineering , Virginia Tech , Blacksburg , VA , USA
| | - Greg A Jamieson
- b Department of Mechanical and Industrial Engineering , University of Toronto , Toronto , Canada
| | - Gyrd Skraaning
- c Industrial Psychology, OECD Halden Reactor Project , Halden , Norway
| |
Collapse
|