Reliability of different behavioural tests for growing pigs on-farm

Development and Validation of a Test for the Classification of Horses as Broken or Unbroken

Regulation EC 1/2005 has stricter rules for transportation of unbroken (untamed) vs. broken (tamed) horses, but does not provide adequate tools for their identification. This study aimed to develop and validate such a tool. A behavioural test (Broken/Unbroken Test (BUT)) based on approaching, haltering, and leading was applied to 100 horses. Physiological and additional behavioural data were also collected, and the horses' status (broken/unbroken) was assessed by the expert who administered the BUT. Each horse's behaviour during the BUT was scored by four trained observers blinded to the horse's history. The BUT score showed excellent inter-observer, intra-observer, and test-retest reliability (all intraclass correlation coefficients (ICCs) > 0.75). It was also negatively associated with respiratory rate, avoidance distance, and time needed to approach, halter, and lead the horse (p < 0.05 for all). The optimal BUT score cut-off for discrimination between broken and unbroken horses (gold standard: expert judgment) showed 97.8% sensitivity and 97.3% specificity. There was almost perfect agreement between BUT-based and expert classification of horses (ICC = 0.940). These findings confirm the BUT's construct and criterion validity. The BUT could provide officials with a feasible, reliable, and valid tool to identify a horse's broken/unbroken status and, consequently, direct stakeholders towards correct transport procedures.

Evaluation of Inter-Observer Reliability of Animal Welfare Indicators: Which Is the Best Index to Use?

Simple Summary

In order to be effective, on-farm welfare assessment protocols should always rely on reliable, as well as valid and feasible, indicators. Inter-observer reliability refers to the extent to which two or more observers are observing and recording data in the same way. The present study focuses on the problem of assessing inter-observer reliability in the case of dichotomous (e.g., yes/no) welfare indicators and the presence of two observers, in order to decide about the inclusion of indicators in welfare assessment protocols. We compared the performance of the most popular currently available agreement indexes. Some widely used indexes showed their inappropriateness to evaluate the inter-observer reliability when the agreement between observers was high. Other less used indexes, such as Bangdiwala’s B or Gwet’s γ(AC1), were found to perform better and are therefore suggested to assess the inter-observer reliability of dichotomous indicators.

Abstract

This study focuses on the problem of assessing inter-observer reliability (IOR) in the case of dichotomous categorical animal-based welfare indicators and the presence of two observers. Based on observations obtained from Animal Welfare Indicators (AWIN) project surveys conducted on nine dairy goat farms, and using udder asymmetry as an indicator, we compared the performance of the most popular agreement indexes available in the literature: Scott’s π, Cohen’s k, kPABAK, Holsti’s H, Krippendorff’s α, Hubert’s Γ, Janson and Vegelius’ J, Bangdiwala’s B, Andrés and Marzo’s ∆, and Gwet’s γ(AC1). Confidence intervals were calculated using closed formulas of variance estimates for π, k, kPABAK, H, α, Γ, J, ∆, and γ(AC1), while the bootstrap and exact bootstrap methods were used for all the indexes. All the indexes and closed formulas of variance estimates were calculated using Microsoft Excel. The bootstrap method was performed with R software, while the exact bootstrap method was performed with SAS software. k, π, and α exhibited a paradoxical behavior, showing unacceptably low values even in the presence of very high concordance rates. B and γ(AC1) showed values very close to the concordance rate, independently of its value. Both bootstrap and exact bootstrap methods turned out to be simpler compared to the implementation of closed variance formulas and provided effective confidence intervals for all the considered indexes. The best approach for measuring IOR in these cases is the use of B or γ(AC1), with bootstrap or exact bootstrap methods for confidence interval calculation.