Reliability studies of incident coding systems in high hazard industries: A narrative review of study methodology

Testing the reliability of accident analysis methods: a comparison of AcciMap, STAMP-CAST and AcciNet

Accident analysis methods are used to model the multifactorial cause of adverse incidents. Methods such as AcciMap, STAMP-CAST and recently AcciNet, are systemic approaches that support the identification of safety interventions across sociotechnical system levels. Despite their growing popularity, little is known about how reliable systems-based methods are when used to describe, model and classify contributory factors and relationships. Here, we conducted an intra-rater and inter-rater reliability assessment of AcciMap, STAMP-CAST and AcciNet using the Signal Detection Theory (SDT) paradigm. A total of 180 hours' worth of analyses across 360 comparisons were performed by 30 expert analysts. Findings revealed that all three methods produced a weak to moderate positive correlation coefficient, however the inter-rater reliability of STAMP-CAST was significantly higher compared to AcciMap and AcciNet. No statistically significant or practically meaningful differences were found between methods in the overall intra-rater reliability analyses. Implications and future research directions are discussed.

Using systems thinking-based risk assessment methods to assess hazardous manual tasks: a comparison of Net-HARMS, EAST-BL, FRAM and STPA

Formal risk assessment is a component of safety management relating to hazardous manual tasks (HMT). Systems thinking approaches are currently gaining interest for supporting safety management. Existing HMT risk assessment methods have been found to be limited in their ability to identify risks across the whole work system; however, systems thinking-based risk assessment (STBRA) methods were not designed for the HMT context and have not been tested in this area. The aim of this study was to compare the performance of four state-of-the-art STBRA methods: Net-HARMS, EAST-BL, FRAM and STPA to determine which would be most useful for identifying HMT risks. Each method was independently applied by one of four analysts to assess the risks associated with a hypothetical HMT system. The outcomes were assessed for alignment with a benchmark analysis. Using signal detection theory (SDT), overall STPA was found to be the best performing method having the highest hit rate, second lowest false alarm rate and highest Matthews Correlation Coefficient of the four methods.Practitioner summary: A comparison of four systems thinking risk assessment methods found that STPA had the highest level of agreement with the benchmark analysis and is the most suitable for practitioners to use to identify the risks associated with HMT systems.

The development of a scenario-based human-machine-environment-procedure (HMEP) classification scheme for the root cause analysis of helicopter accidents

The current study analyzed the root causes of 22 helicopter accidents/incidents that took place between 1998 and 2019. Each root cause was coded using three commonly used classification models in aviation HFACS, ATSB, and IATA to identify recurring factors for better targeting of future prevention strategies. The frequency analysis revealed that not following procedure (22 observations), training inadequate or unavailable (17), inadequate regulatory oversight (17), inadequate procedure guidance (16), company management absent or deficient (10) and incorrect manuals/charts/checklists (9) were the most frequent contributing factors. Since none of the existing models could summarize the root causes of 22 occurrences effectively, a scenario-based human-machine-environment-procedure (HMEP) classification scheme was proposed to use organizational influences, people management, technical failure, procedure and document, and environment as the first-layer subcategories. The HMEP scheme was additionally applied to the analysis and coding of 4 helicopter accidents in the USA published by the NTSB. The HMEP scheme revealed that NTSB had identified a significantly greater number of root causes in the manufacturer design, manufacturing & documentation. Overall, HMEP can be used to guide the data collection during accident investigation and subsequently to aggregate aviation accidents to derive recurring factors and compare accident patterns in an efficient manner.

Coding agricultural injury: Factors affecting coder agreement

OBJECTIVES

To determine coders' agreement level for the Occupational Injury and Illness Classification System (OIICS) source of injury and injury event codes, and the Farm and Agricultural Injury Classification (FAIC) code in the AgInjuryNews.org and to determine the effects of supplemental information and follow-up discussion in final code assignments.

METHODS

Two independent researchers initially coded 1304 injury cases from AgInjurynews.org using the OIICS and the FAIC coding schemes. Code agreement levels for injury source, event, and FAIC and the effect of supplemental information and follow-up discussions on final coding was assessed.

RESULTS

Coders' agreement levels were almost perfect for OIICS source and event categories at the 3-digit level, with lower agreement at the 4-digit level. By using supplemental information and follow-up discussion, coders improved the coding accuracy by an average 20% for FAIC. Supplemental information and follow-up discussions had helped finalize the disagreed codes 55% of the time for OIICS source coding assignments and 40% of time for OIICS event coding assignments for most detailed 4-digit levels. Five key themes emerged regarding accurate and consistent coding of the agricultural injuries: inclusion/exclusion based on industry classification system; inconsistent/discrepant reports; incomplete/nonspecific reports; effects of supplemental information on coding; and differing interpretations of code selection rules. Practical applications: Quantifying the level of agreement for agricultural injuries will lead to a better understanding of coding discrepancies and may uncover areas for improvement to coding scheme itself. High level of initial and final agreement with FAIC and OIICS codes suggest that these coding schemes are user-friendly and amenable to widespread use.

Applying AcciMap to test the common cause hypothesis using aviation near misses

The common cause hypothesis, as applied here, proposes that similar networks of influencing factors may contribute to both adverse outcomes and near misses. This hypothesis has not been evaluated using a systems-thinking perspective. The aims of this study are to evaluate whether networks of contributory and protective factors exist within aviation serious near miss reports and to determine if the common cause hypothesis is applicable in this context. Sixteen incident reports from French civil aviation crash investigation bureau were analysed using the AcciMap method. Contributory and protective factors, and relationships between both were identified via coding of the reports. The results indicate that considering protective factors support a richer picture of incidents and provide support for the common cause hypothesis as measured by similar mean factor volume and sociotechnical levels for both contributory and protective factors. However, the findings also show the direction of relationships among protective and contributory factors may be indicative of a difference among adverse outcomes, near misses, and normal work. Future research should consider how a network of relationships may impact on the common contributory and protective factors found in near misses.

Evaluation of construct and criterion-referenced validity of a systems-thinking based near miss reporting form

The validity of methods is an ongoing issue in ergonomics. Inconsistent definitions and approaches to evaluation exacerbate this challenge. In this study, the construct and criterion-referenced validity of a new near miss reporting form was evaluated to determine the extent to which it comprehensively captures near miss incidents and is aligned with the systems thinking approach to accident causation. Interview data were used as the reference standard in the evaluation. Using signal detection theory (SDT), a high average hit rate (HR), predictive value (PV) and sensitivity index (SI) were found, with an almost perfect ranking for the index of concordance. The findings show that the reporting form has strong construct and criterion-referenced validity. It is proposed that the approach used in this study could be used by researchers and practitioners when testing the validity of incident data collection tools. Practitioner summary: The validity of methods is a key issue in ergonomics. In this study, we test the validity of a near miss reporting form using interview data as a standard. This approach could be used by practitioners when testing the validity of other ergonomics methods.

What went right? An analysis of the protective factors in aviation near misses

Learning from successful safety outcomes, or what went right, is an important emerging component of maintaining safe systems. Accordingly, there are increasing calls to study normal performance in near misses as a part of safety management activities. Despite this, there is limited guidance on how to accomplish this in practice. This article presents a study in which using Rasmussen's risk management framework to analyse 16 serious incidents from the aviation domain. The findings show that a network of protective factors prevents accidents with factors identified across the sociotechnical system. These protective networks share many properties with those identified in accidents. The article demonstrates that is possible to identify these networks of protective factors from incident investigation reports. The theoretical implications of these results and future research opportunities are discussed. Practitioner Statement: The analysis of near misses is an important part of safety management activities. This article demonstrates that Rasmussen?s risk management framework can be used to identify networks of protective factors which prevent accidents. Safety practitioners can use the framework described to discover and support the system-wide networks of protective factors.

Development of a higher-order instruction coding taxonomy for observational data: Initial application to professional driving instruction

Development of a higher-order instruction taxonomy, informed by best practice in driver education (Goals for Driver Education) and self-determination theory (guiding teaching strategies), was tested. Inter-coder reliability was assessed by coding 93 data elements from 5-min clips from three driving instructors. Seventy-three instruction and 32 teaching approach codes were selected. Reliability between two independent coders was high (IOC = 94.6%). Application to data from 17 randomly-selected, 1-h lessons (n = 3 driving instructors) in a pilot study of professional learner driver lessons assessed taxonomy validity. Missed, taken, and untaken opportunities for higher-order instruction via 9 instruction and 19 teaching-approach categories were identified. Reliability assessment and taxonomy application demonstrates evidence to facilitate a comprehensive understanding of driving instruction content and quality, with implications for assessing and evaluating the impact of higher-order instruction in relation to driving and other safety-critical sectors requiring higher-order skills.

Radio Gaga? Intra-team communication of Australian Rules Football umpires - effect of radio communication on content, structure and frequency

Intra-team communication plays an important role in team effectiveness in various domains including sport. As such, it is a key consideration when introducing new tools within systems that utilise teams. The difference in intra-team communication of Australian Rules Football (AFL) umpiring teams was studied when umpiring with or without radio communications technology. A cross-sectional observational study was conducted to analyse the verbal communication of seven umpiring teams (20 participants) grouped according to their experience with radio communication. The results identified that radio communication technology increased the frequency and altered the structure of intra-team communication. Examination of the content of the intra-team communication identified impacts on the 'Big Five' teamwork behaviours and associated coordinating mechanisms. Analysis revealed that the communications utilised did not align with the closed-loop form of communication described in the Big Five model. Implications for teamwork models, coaching and training of AFL umpires are discussed. Practitioner Summary: Assessing the impact of technology on performance is of interest to ergonomics practitioners. The impact of radio communications on teamwork is explored in the highly dynamic domain of AFL umpiring. When given radio technology, intra-team communication increased which supported teamwork behaviours, such as backup behaviour and mutual performance monitoring.

Developing a contributing factor classification scheme for Rasmussen's AcciMap: Reliability and validity evaluation

One factor potentially limiting the uptake of Rasmussen's (1997) Accimap method by practitioners is the lack of a contributing factor classification scheme to guide accident analyses. This article evaluates the intra- and inter-rater reliability and criterion-referenced validity of a classification scheme developed to support the use of Accimap by led outdoor activity (LOA) practitioners. The classification scheme has two levels: the system level describes the actors, artefacts and activity context in terms of 14 codes; the descriptor level breaks the system level codes down into 107 specific contributing factors. The study involved 11 LOA practitioners using the scheme on two separate occasions to code a pre-determined list of contributing factors identified from four incident reports. Criterion-referenced validity was assessed by comparing the codes selected by LOA practitioners to those selected by the method creators. Mean intra-rater reliability scores at the system (M = 83.6%) and descriptor (M = 74%) levels were acceptable. Mean inter-rater reliability scores were not consistently acceptable for both coding attempts at the system level (M_T1 = 68.8%; M_T2 = 73.9%), and were poor at the descriptor level (M_T1 = 58.5%; M_T2 = 64.1%). Mean criterion referenced validity scores at the system level were acceptable (M_T1 = 73.9%; M_T2 = 75.3%). However, they were not consistently acceptable at the descriptor level (M_T1 = 67.6%; M_T2 = 70.8%). Overall, the results indicate that the classification scheme does not currently satisfy reliability and validity requirements, and that further work is required. The implications for the design and development of contributing factors classification schemes are discussed.

Application of classification principles to improve the reliability of incident classification systems: A test case using HFACS-ADF

Accident classification systems are important tools for safety management. Unfortunately, many of the tools available have demonstrated poor reliability of coding, making their validity and usefulness questionable. This paper demonstrates the application of four strategies to improve the reliability of accident and incident classification systems. The strategies include creating a domain-specific system with limitations on system size and careful selection of codes, specifically the reduction of abstract concepts and bias-causing terminology. Using HFACS-ADF as a test case, the system was adapted using the strategies and validated using comprehension and comprehensiveness testing. The new system was then assessed for reliability. The reliability of the system increased by at least 20% at all levels of the classification system following the changes made. The results provide evidence that the application of theoretically and empirically-derived classification principles are effective for improving the reliability of accident and incident classification systems in high hazard industries.

Assessing the coding reliability of work accidents statistical data: How coders make a difference

INTRODUCTION

This study assesses the reliability of the coding procedure for a set of variables belonging to the European Statistics of Accidents at Work (ESAW). The work focused on the Portuguese data and experience with the system. In Portugal, this task has been systematically carried out by GEP (the governmental Cabinet for Strategy and Planning), here defined as the "reference group" or "expert group." However, it is anticipated that this coding task will be performed by non-expert people, since paper-forms will be replaced by e-forms, similarly to what happened in a few EU countries.

OBJECTIVE

This study aims to: (a) assess the current situation, that is, to quantify reliability of data coded by GEP (reference group), and (b) assess the impact on the reliability level when the coding is carried out by non-experts (two different groups of coders).

METHODS

The study comprises the estimation of both intercoder and intracoder reliability for a set of 8 nominal variables. The assessment applies 3 reliability coefficients calculated by 3 software packages.

RESULTS

The results reveal that the expert group (GEP) holds good to excellent reliability (inter- and intracoder agreements), between 68-98%, while there is a considerable "loss of reliability" (-5% to -39%) when the coding process is transferred to other people, without special training or knowledge in this task.

CONCLUSIONS

This work gives quantified evidence that reliability of coding accident data is substantially affected by the coders' profile. Moreover, certain variables, regardless of the coder, systematically hold a higher level of coding reliability than others, suggesting that certain codes may need improvement. Future studies should assess coding quality across the EU countries using the ESAW protocol.

PRACTICAL APPLICATIONS

Directions for improving the quality of accident data and related statistics; data that is used by researchers and governmental decision-makers to derive prevention strategies.

Development of safety incident coding systems through improving coding reliability

This paper reviews classification theory sources to develop five research questions concerning factors associated with incident coding system development and use and how these factors affect coding reliability. Firstly, a method was developed to enable the comparison of reliability results obtained using different methods. Second, a statistical and qualitative review of reliability studies was conducted to investigate the influence of the identified factors on the reliability of incident coding systems. As a result several factors were found to have a statistically significant effect on reliability. Four recommendations for system development and use are provided to assist researchers in improving the reliability of incident coding systems in high hazard industries.