Testing the reliability and validity of risk assessment methods in Human Factors and Ergonomics

When tomorrow comes: A prospective risk assessment of a future artificial general intelligence-based uncrewed combat aerial vehicle system

There are concerns that Artificial General Intelligence (AGI) could pose an existential threat to humanity; however, as AGI does not yet exist it is difficult to prospectively identify risks and develop requisite controls. We applied the Work Domain Analysis Broken Nodes (WDA-BN) and Event Analysis of Systemic Teamwork-Broken Links (EAST-BL) methods to identify potential risks in a future 'envisioned world' AGI-based uncrewed combat aerial vehicle system. The findings suggest five main categories of risk in this context: sub-optimal performance risks, goal alignment risks, super-intelligence risks, over-control risks, and enfeeblement risks. Two of these categories, goal alignment risks and super-intelligence risks, have not previously been encountered or dealt with in conventional safety management systems. Whereas most of the identified sub-optimal performance risks can be managed through existing defence design lifecycle processes, we propose that work is required to develop controls to manage the other risks identified. These include controls on AGI developers, controls within the AGI itself, and broader sociotechnical system controls.

Testing the reliability of accident analysis methods: a comparison of AcciMap, STAMP-CAST and AcciNet

Accident analysis methods are used to model the multifactorial cause of adverse incidents. Methods such as AcciMap, STAMP-CAST and recently AcciNet, are systemic approaches that support the identification of safety interventions across sociotechnical system levels. Despite their growing popularity, little is known about how reliable systems-based methods are when used to describe, model and classify contributory factors and relationships. Here, we conducted an intra-rater and inter-rater reliability assessment of AcciMap, STAMP-CAST and AcciNet using the Signal Detection Theory (SDT) paradigm. A total of 180 hours' worth of analyses across 360 comparisons were performed by 30 expert analysts. Findings revealed that all three methods produced a weak to moderate positive correlation coefficient, however the inter-rater reliability of STAMP-CAST was significantly higher compared to AcciMap and AcciNet. No statistically significant or practically meaningful differences were found between methods in the overall intra-rater reliability analyses. Implications and future research directions are discussed.

Using systems thinking-based risk assessment methods to assess hazardous manual tasks: a comparison of Net-HARMS, EAST-BL, FRAM and STPA

Formal risk assessment is a component of safety management relating to hazardous manual tasks (HMT). Systems thinking approaches are currently gaining interest for supporting safety management. Existing HMT risk assessment methods have been found to be limited in their ability to identify risks across the whole work system; however, systems thinking-based risk assessment (STBRA) methods were not designed for the HMT context and have not been tested in this area. The aim of this study was to compare the performance of four state-of-the-art STBRA methods: Net-HARMS, EAST-BL, FRAM and STPA to determine which would be most useful for identifying HMT risks. Each method was independently applied by one of four analysts to assess the risks associated with a hypothetical HMT system. The outcomes were assessed for alignment with a benchmark analysis. Using signal detection theory (SDT), overall STPA was found to be the best performing method having the highest hit rate, second lowest false alarm rate and highest Matthews Correlation Coefficient of the four methods.Practitioner summary: A comparison of four systems thinking risk assessment methods found that STPA had the highest level of agreement with the benchmark analysis and is the most suitable for practitioners to use to identify the risks associated with HMT systems.

Musculoskeletal-Modeling-Based, Full-Body Load-Assessment Tool for Ergonomists (MATE): Method Development and Proof of Concept Case Studies

A new ergonomic-risk-assessment tool was developed that combines musculoskeletal-model-based loading estimates with insights from fatigue failure theory to evaluate full-body musculoskeletal loading during dynamic tasks. Musculoskeletal-modeling output parameters, i.e., joint contact forces and muscle forces, were combined with tissue-specific injury thresholds that account for loading frequency to determine the injury risk for muscles, lower back, and hip cartilage. The potential of this new risk-assessment tool is demonstrated for defining ergonomic interventions in terms of lifting characteristics, back and shoulder exoskeleton assistance, box transferring, stoop lifting, and an overhead wiring task, respectively. The MATE identifies the risk of WMSDs in different anatomical regions during occupational tasks and allows for the evaluation of the impact of interventions that modify specific lifting characteristics, i.e., load weight versus task repetition. Furthermore, and in clear contrast to currently available ergonomic assessment scores, the effects of the exoskeleton assistance level on the risk of WMSDs of full-body musculoskeletal loading (in particular, the muscles, lower back, and hips) can be evaluated and shows small reductions in musculoskeletal loading but not in injury risk. Therefore, the MATE is a risk-assessment tool based on a full-body, musculoskeletal-modeling approach combined with insights from the fatigue failure theory that shows the proof of concept of a shoulder and back exoskeleton. Furthermore, it accounts for subject-specific characteristics (age and BMI), further enhancing individualized ergonomic-risk assessment.