A non-parametric approach to estimate the total deviation index for non-normal data

Validating a common tick survey method: cloth-dragging and line transects

Cloth-dragging is the most widely-used method for collecting and counting ticks, but there are few studies of its reliability. By using cloth-dragging, we applied a replicated line transects survey method, in two areas in Sweden with different Ixodes ricinus tick-densities (low at Grimsö and high at Bogesund) to evaluate developmental stage specific repeatability, agreement and precision in estimates of tick abundance. 'Repeatability' was expressed as the Intraclass Correlation Coefficient (ICC), 'agreement' with the Total Deviation Index (TDI) and 'precision' by the coefficient of variation (CV) for a given dragging distance. Repeatability (ICC) and agreement (TDI) were higher for the most abundant instar (nymphs) and in the area of higher abundance. At Bogesund tick counts were higher than at Grimsö and so also repeatability, with fair to substantial ICC estimates between 0.22 and 0.75, and TDI ranged between 1 and 44.5 counts of difference (thus high to moderate agreement). At Grimsö, ICC was poor to moderate and ranged between 0 and 0.59, whereas TDI remained low with estimates lower or equal to 1 count (thus high agreement). Despite a 100-fold lower abundance at Grimsö, the same level of precision for nymphs could be achieved with a 70% increase of dragging effort. We conclude that the cloth-dragging technique is useful for surveying ticks' and primarily to estimate abundance of the nymphal stage, whereas it rarely will be recommended for larvae and adults.

Using multiple agreement methods for continuous repeated measures data: a tutorial for practitioners

Background

Studies of agreement examine the distance between readings made by different devices or observers measuring the same quantity. If the values generated by each device are close together most of the time then we conclude that the devices agree. Several different agreement methods have been described in the literature, in the linear mixed modelling framework, for use when there are time-matched repeated measurements within subjects.

Methods

We provide a tutorial to help guide practitioners when choosing among different methods of assessing agreement based on a linear mixed model assumption. We illustrate the use of five methods in a head-to-head comparison using real data from a study involving Chronic Obstructive Pulmonary Disease (COPD) patients and matched repeated respiratory rate observations. The methods used were the concordance correlation coefficient, limits of agreement, total deviation index, coverage probability, and coefficient of individual agreement.

Results

The five methods generated similar conclusions about the agreement between devices in the COPD example; however, some methods emphasized different aspects of the between-device comparison, and the interpretation was clearer for some methods compared to others.

Conclusions

Five different methods used to assess agreement have been compared in the same setting to facilitate understanding and encourage the use of multiple agreement methods in practice. Although there are similarities between the methods, each method has its own strengths and weaknesses which are important for researchers to be aware of. We suggest that researchers consider using the coverage probability method alongside a graphical display of the raw data in method comparison studies. In the case of disagreement between devices, it is important to look beyond the overall summary agreement indices and consider the underlying causes. Summarising the data graphically and examining model parameters can both help with this.

Overall indices for assessing agreement among multiple raters

The need to assess agreement exists in various clinical studies where quantifying inter-rater reliability is of great importance. Use of unscaled agreement indices, such as total deviation index and coverage probability (CP), is recommended for two main reasons: (i) they are intuitive in a sense that interpretations are tied to the original measurement unit; (ii) practitioners can readily determine whether the agreement is satisfactory by directly comparing the value of the index to a prespecified tolerable CP or absolute difference. However, the unscaled indices were only defined in the context of comparing two raters or multiple raters that assume homogeneity of variances across raters. In this paper, we introduce a set of overall indices based on the root mean square of pairwise differences that are unscaled and can be used to evaluate agreement among multiple raters that often exhibit heterogeneous measurement processes in practice. Furthermore, we propose another overall agreement index based on the root mean square of pairwise differences that is scaled and extends the concept of the recently proposed relative area under CP curve in the presence of multiple raters. We present the definitions of overall indices and propose inference procedures in which bootstrap methods are used for the estimation of standard errors. We assess the performance of the proposed approach and demonstrate its superiority over the existing methods when raters exhibit heterogeneous measurement processes using simulation studies. Finally, we demonstrate the application of our methods using a renal study.

Bayesian inference for agreement measures

The agreement of different measurement methods is an important issue in several disciplines like, for example, Medicine, Metrology, and Engineering. In this article, some agreement measures, common in the literature, were analyzed from a Bayesian point of view. Posterior inferences for such agreement measures were obtained based on well-known Bayesian inference procedures for the bivariate normal distribution. As a consequence, a general, simple, and effective method is presented, which does not require Markov Chain Monte Carlo methods and can be applied considering a great variety of prior distributions. Illustratively, the method was exemplified using five objective priors for the bivariate normal distribution. A tool for assessing the adequacy of the model is discussed. Results from a simulation study and an application to a real dataset are also reported.