Another look at interrater agreement

A Non-iterative Confidence Interval Estimating Procedure for the Intraclass Kappa Statistic with Multinomial Outcomes

We obtain the asymptotic sample variance of the intraclass kappa statistic for multinomial outcome data. A modified Wald type procedure based on this theory is then used for confidence interval construction. The results of a simulation study show that the proposed non-iterative approach performs very well in terms of confidence interval coverage and width for samples as small as 50. The procedure is illustrated with two examples from previously published medical studies.

Parent-child interactions among children with juvenile fibromyalgia, arthritis, and healthy controls

Parent-child interactions during pain-inducing exercise tasks among children (11-17 years old) with fibromyalgia, juvenile rheumatoid arthritis, and pain-free controls were examined and the contribution of parent-child interactions to disability was tested. Fifteen children in each of the three diagnostic groups and their parents completed 5-min exercise tasks and completed questionnaire measures of disability (Functional Disability Inventory) and coping (Pain Coping Questionnaire). There were few group differences in parent-child interactions. After controlling for children's ratings of pain evoked by the exercise, group differences in interactions during exercise tasks were no longer significant. Sequential analyses, controlling for group and exercise task, revealed that when parents made statements discouraging coping following children's negative verbalizations about the task or pain, children were less likely to be on task, compared to when parents made statements encouraging coping or when parents made any other statements. Children's general pain coping strategies were not related to parent-child interactions. Parent-child interactions were generally not related to disability. Across the groups, more pain and less time on task during the exercises were related to Functional Disability Inventory scores and more school absences. Parent-child interaction patterns influence children's adaptation to pain during experimental tasks. Parents' discouragement of coping in response to their children's negative statements related to the pain or the pain-evoking task are counter productive to children's ability to maintain activity in a mildly painful situation.

Multirater agreement of arthroscopic meniscal lesions

BACKGROUND

Establishing the validity of classification schemes is a crucial preparatory step that should precede multicenter studies. There are no studies investigating the reproducibility of arthroscopic classification of meniscal pathology among multiple surgeons at different institutions.

HYPOTHESIS

Arthroscopic classification of meniscal pathology is reliable and reproducible and suitable for multicenter studies that involve multiple surgeons.

STUDY DESIGN

Multirater agreement study.

METHODS

Seven surgeons reviewed a video of 18 meniscal tears and completed a meniscal classification questionnaire. Multirater agreement was calculated based on the proportion of agreement, the kappa coefficient, and the intraclass correlation coefficient.

RESULTS

There was a 46% agreement on the central/peripheral location of tears (kappa = 0.30), an 80% agreement on the depth of tears (kappa = 0.46), a 72% agreement on the presence of a degenerative component (kappa = 0.44), a 71% agreement on whether lateral tears were central to the popliteal hiatus (kappa = 0.42), a 73% agreement on the type of tear (kappa = 0.63), an 87% agreement on the location of the tear (kappa = 0.61), and an 84% agreement on the treatment of tears (kappa = 0.66). There was considerable agreement among surgeons on length, with an intraclass correlation coefficient of 0.78, 95% confidence interval of 0.57 to 0.92, and P < .001.

CONCLUSIONS

Arthroscopic grading of meniscal pathology is reliable and reproducible.

CLINICAL RELEVANCE

Surgeons can reliably classify meniscal pathology and agree on treatment, which is important for multicenter trials.

Simultaneous estimation of intrarater and interrater agreement for multiple raters under order restrictions for a binary trait

It is valuable in many studies to assess both intrarater and interrater agreement. Most measures of intrarater agreement do not adjust for unequal estimates of prevalence between the separate rating occasions for a given rater and measures of interrater agreement typically ignore data from the second set of assessments when raters make duplicate assessments. In the event when both measures are assessed there are instances where interrater agreement is larger than at least one of the corresponding intrarater agreements. This implies that a rater agrees less with him/herself and more with another rater. In the situation of multiple raters making duplicate assessments on all subjects, the authors propose properties for an agreement measure based on the odds ratio for a dichotomous trait: (i) estimate a single prevalence across two reading occasions for each rater; (ii) estimate pairwise interrater agreement from all available data; (iii) bound the pairwise interrater agreement above by the corresponding intrarater agreements. Estimation of odds ratios under these properties is done by maximizing the multinomial likelihood with constraints using generalized log-linear models in combination with a generalization of the Lemke-Dykstra iterative-incremental algorithm. An example from a mammography examination reliability study is used to demonstrate the new method.

Measuring agreement between patient and proxy responses to multidimensional health-related quality-of-life measures in clinical trials. An application of psychometric profile analysis

When patients cannot provide responses to health-related quality-of-life (HRQOL) measures in clinical trials, family or friends may be asked to respond. We present a simple, comprehensive method for assessing agreement between patients with head injury and their proxy responders. In contrast to more traditional approaches, this method defines agreement separately for each patient-proxy pair, and compares HRQOL profiles along three dimensions-level, or the average of the ratings; scatter, or the variability in the ratings; and shape, or the ranks of the ratings. We demonstrate this method in the context of a clinical trial of a treatment for traumatic head injury and compare the results to those obtained using traditional analyses. Options for incorporating proxy responses into clinical trial analyses are discussed.

An examination of interrater reliability for scoring the Rorschach Comprehensive System in eight data sets

In this article, we describe interrater reliability for the Comprehensive System (CS; Exner. 1993) in 8 relatively large samples, including (a) students, (b) experienced re- searchers, (c) clinicians, (d) clinicians and then researchers, (e) a composite clinical sample (i.e., a to d), and 3 samples in which randomly generated erroneous scores were substituted for (f) 10%, (g) 20%, or (h) 30% of the original responses. Across samples, 133 to 143 statistically stable CS scores had excellent reliability, with median intraclass correlations of.85, .96, .97, .95, .93, .95, .89, and .82, respectively. We also demonstrate reliability findings from this study closely match the results derived from a synthesis of prior research, CS summary scores are more reliable than scores assigned to individual responses, small samples are more likely to generate unstable and lower reliability estimates, and Meyer's (1997a) procedures for estimating response segment reliability were accurate. The CS can be scored reliably, but because scoring is the result of coder skills clinicians must conscientiously monitor their accuracy.

Interval estimation for a difference between intraclass kappa statistics

Model-based inference procedures for the kappa statistic have developed rapidly over the last decade. However, no method has yet been developed for constructing a confidence interval about a difference between independent kappa statistics that is valid in samples of small to moderate size. In this article, we propose and evaluate two such methods based on an idea proposed by Newcombe (1998, Statistics in Medicine, 17, 873-890) for constructing a confidence interval for a difference between independent proportions. The methods are shown to provide very satisfactory results in sample sizes as small as 25 subjects per group. Sample size requirements that achieve a prespecified expected width for a confidence interval about a difference of kappa statistic are also presented.

Indexing systematic rater agreement with a latent-class model

A latent-class model of rater agreement is presented for which 1 of the model parameters can be interpreted as the proportion of systematic agreement. The latent classes of the model emerge from the factorial combination of the "true" category in which a target belongs and the ease with which raters are able to classify targets into the true category. Several constrained cases of the model are described, and the relations to other well-known agreement models and kappa-type summary coefficients are explained. The differential quality of the rating categories can be assessed on the basis of the model fit. The model is illustrated using data from diagnoses of psychiatric disorders and classifications of individuals in a persuasive communication study.

Nonasymptotic significance tests for two measures of agreement

The kappa agreement coefficient of Cohen from 1960 and Brennan and Prediger from 1981 are defined and compared. A FORTRAN program is described that computes Cohen's kappa and Brennan and Prediger's kappa and their associated probability values based on Monte Carlo resampling and the binomial distribution, respectively.

Interval estimation for Cohen's kappa as a measure of agreement

Cohen's kappa statistic is a very well known measure of agreement between two raters with respect to a dichotomous outcome. Several expressions for its asymptotic variance have been derived and the normal approximation to its distribution has been used to construct confidence intervals. However, information on the accuracy of these normal-approximation confidence intervals is not comprehensive. Under the common correlation model for dichotomous data, we evaluate 95 per cent lower confidence bounds constructed using four asymptotic variance expressions. Exact computation, rather than simulation is employed. Specific conditions under which the use of asymptotic variance formulae is reasonable are determined.

Testing the equality of two dependent kappa statistics

Procedures are developed and compared for testing the equality of two dependent kappa statistics in the case of two raters and a dichotomous outcome variable. Such problems may arise when each of a sample of subjects are rated under two distinct settings, and it is of interest to compare the observed levels of inter-observer and intra-observer agreement. The procedures compared are extensions of previously developed procedures for comparing kappa statistics computed from independent samples. The results of a Monte Carlo simulation show that adjusting for the dependency between samples tends to be worthwhile only if the between-setting correlation is comparable in magnitude to the within-setting correlations. In this case, a goodness-of-fit procedure that takes into account the dependency between samples is recommended.

Interobserver agreement, intraobserver reliability, and the Rorschach Comprehensive System

Interrater agreement and reliability for the Rorschach have recently come under increasing scrutiny. This is the second report examining methods of Comprehensive System reliability using principles derived from observational methodology and applied behavioral analysis. This study examined a previous nonpatient sample of 20 protocols (N = 412 responses) and also examined a new clinical sample of 20 protocols (N = 374 responses) diagnosed with Research Diagnostic Criteria. Reliability was analyzed at multiple levels of Comprehensive System data, including response-level individual codes and coding decisions and ratios, percentages, and derivations from the Structural Summary. With a number of exceptions, most Comprehensive System codes, coding decisions, and summary scores yield acceptable, and in many instances excellent, levels of reliability. Limitations arising from the nature of Rorschach data and Comprehensive System coding criteria are discussed.

Patterns of disease activity in systemic lupus erythematosus

OBJECTIVE

To describe patterns of systemic lupus (SLE) disease activity over time.

METHODS

Disease activity was measured in a prospective cohort of 204 consecutive SLE patients followed up quarterly for 2.0-7.5 years (911 person-years of followup). Physician's global assessment (PGA) and modified SLE Disease Activity Index (M-SLEDAI; omitting serology) scores were plotted against time for each patient. Definitions for disease activity patterns were developed by consensus using these plots, and the proportion of total follow-up time represented by each pattern was calculated.

RESULTS

Three patterns of SLE activity were apparent: relapsing-remitting (RR), chronic active (CA), and long quiescent (LQ). The CA pattern was the most frequent for both the PGA and M-SLEDAI, representing 58% and 40% of total person-years, respectively. The least common pattern was LQ (PGA 16%, M-SLEDAI 25%), while the RR pattern was intermediate in frequency (PGA 26%, M-SLEDAI 35%). Average disease activity during RR periods tended to be mild, while that during CA periods was more likely to be moderately severe. The most common discrepancy between instruments was that the PGA depicted CA when the M-SLEDAI showed an RR pattern. The M-SLEDAI did not appear to capture mild degrees of activity.

CONCLUSION

SLE activity was readily classified into 1 of 3 patterns: RR, CA, or LQ. The CA pattern was most common, suggesting that significant morbidity may arise from persistent disease activity. These findings may have important implications regarding the choice of outcome measures in SLE clinical trials, since comparison of flare rates may not account for chronic disease activity.

Reliability of reports of violent victimization and posttraumatic stress disorder among men and women with serious mental illness

Although violent victimization is highly prevalent among men and women with serious mental illness (SMI; e.g., schizophrenia, bipolar disorder), future research in this area may be impeded by controversy concerning the ability of individuals with SMI to report traumatic events reliably. This article presents the results of a study exploring the temporal consistency of reports of childhood sexual abuse, adult sexual abuse, and adult physical abuse, as well as current symptoms of posttraumatic stress disorder (PTSD) among 50 people with SMI. Results show that trauma history and PTSD assessments can, for the most part, yield reliable information essential to further research in this area. The study also demonstrates the importance of using a variety of statistical methods to assess the reliability of self-reports of trauma history.

Correlation as Probability of Common Descent

We highlight one interpretation of Pearson's r (largely unknown to behavioral scientists), inspired by the genetic measurement of inbreeding. The coefficient of inbreeding, defined as the probability that two paired alleles originate from common descent, equals the correlation between the uniting gametes. We specify the statistical conditions under which r can be interpreted as probability of identity by descent and explore the possibility of generalizing that meaning of correlation beyond the inbreeding context. Extensions to the framework of agreement between judges and to that of sequential dependencies are considered. Viewing correlation as probability is heuristically promising. We examine the implications of this approach in the case of three types of bivariate distributions and discuss potential insights and risks.

Self-reports of HIV risk behavior by injecting drug users: are they reliable?

While most studies of AIDS risk behavior rely on self-reports, few studies have assessed the reliability of these reports. The present study examines self-reports of drug-related and sexual risk behavior among pairs of injecting drug users (IDUs) recruited from the streets in New York City. Since both members of the pair were interviewed, it was possible to compare their responses in order to assess reliability. Subjects reported on their contacts' demographic data (age, gender, race/ethnicity) and on shared risk behaviors, including syringe sharing. Despite the private and/or illegal nature of AIDS risk behaviors, IDU subjects were generally reliable in their reports of both demographic and AIDS risk behaviors.

The reliability of observational data: I. Theories and methods for speech-language pathology

Much research and clinical work in speech-language pathology depends on the validity and reliability of data gathered through the direct observation of human behavior. This paper reviews several definitions of reliability, concluding that behavior observation data are reliable if they, and the experimental conclusions drawn from them, are not affected by differences among observers or by other variations in the recording context. The theoretical bases of several methods commonly used to estimate reliability for observational data are reviewed, with examples of the use of these methods drawn from a recent volume of the Journal of Speech and Hearing Research (35, 1992). Although most recent research publications in speech-language pathology have addressed the issue of reliability for their observational data to some extent, most reliability estimates do not clearly establish that the data or the experimental conclusions were replicable or unaffected by differences among observers. Suggestions are provided for improving the usefulness of the reliability estimates published in speech-language pathology research.

The reliability of staging and grading of bladder tumours. Impact of misinformation on the pathologist's diagnosis

The influence of misinformation on the reliability of the histopathological classification of bladder tumours was analysed. Four consultant pathologists assessed 40 biopsy specimens of bladder tumours staging invasion and grading the specimens according to the Bergkvist classification. A random sample of 20 specimens was accompanied by systematically distorted information ("bias"-unknown to the pathologists) about previous histological grading of the patient (bias group); the other 20 specimens were used as control group (non bias group). After 6 months a second round with the same specimens was arranged to assess the influence of bias on the intraobserver variation. Using kappa (kappa) statistics the chance corrected interobserver agreement rate was poor both in staging of invasion and grading according to the Bergkvist classification (kappa < 0.50). The kappa values in the intraobserver study ranged from poor to excellent with a tendency towards lower kappa when the observer had been biased. The kappa values in the assessment of malignancy were acceptable to excellent. False information did not affect the pathologists' diagnosis significantly.

Bias, prevalence and kappa

Since the introduction of Cohen's kappa as a chance-adjusted measure of agreement between two observers, several "paradoxes" in its interpretation have been pointed out. The difficulties occur because kappa not only measures agreement but is also affected in complex ways by the presence of bias between observers and by the distributions of data across the categories that are used ("prevalence"). In this paper, new indices that provide independent measures of bias and prevalence, as well as of observed agreement, are defined and a simple formula is derived that expresses kappa in terms of these three indices. When comparisons are made between agreement studies it can be misleading to report kappa values alone, and it is recommended that researchers also include quantitative indicators of bias and prevalence.

Modelling patterns of agreement and disagreement

This article presents a survey of ways of statistically modelling patterns of observer agreement and disagreement. Main emphasis is placed on modelling inter-observer agreement for categorical responses, both for nominal and ordinal response scales. Models discussed include (1) simple cell-probability models based on Cohen's kappa that focus on beyond-chance agreement, (2) loglinear models for square tables, such as quasi-independence and quasi-symmetry models, (3) latent class models that express the joint distribution between ratings as a mixture of clusters for homogeneous subjects, each cluster having the same 'true' rating, and 4) Rasch models, which decompose subject-by-observer rating distributions using observer and subject main effects. Models can address two distinct components of agreement--strength of association between ratings, and similarity of marginal distributions of the ratings.

The Explanatory Model Interview Catalogue (EMIC). Contribution to cross-cultural research methods from a study of leprosy and mental health

The Explanatory Model Interview Catalogue (EMIC) has been developed to elicit illness-related perceptions, beliefs, and practices in a cultural study of leprosy and mental health in Bombay. Leprosy is an especially appropriate disorder for studying the inter-relationship of culture, mental health and medical illness because of deeply rooted cultural meanings, the emotional burden, and underuse of effective therapy. Fifty per cent of 56 recently diagnosed leprosy out-patients, 37% of 19 controls with another stigmatised dermatological condition (vitiligo), but only 8% of 12 controls with a comparable non-stigmatised condition (tinea versicolor) met DSM-III-R criteria for an axis I depressive, anxiety or somatoform disorder. Belief in a humoral (traditional) cause of illness predicted better attendance at clinic.

Interscorer reliability for the Hand Test administered to children

The utility of the Hand Test as a quick, reliable measure of 100 children's personalities was assessed. The interscorer reliability of the Hand Test was estimated by both intraclass correlations and the Kappa coefficient for 100 children. Following training, satisfactory intraclass correlations were obtained for the Quantitative scores (20 of 22 above .70) and Qualitative scores (12 of 27 above .70) Kappa coefficients were generally lower. Scorers' memory overload and low response frequency are discussed as possible bases for the low reliabilities of Qualitative scores. Although the Hand Test reliability for Quantitative scores is consistent with those of other projective tests, consideration should be given to the modification of the directions of administration for young children and clarification of scoring rules.

The process of rater training for observational instruments: implications for interrater reliability

Although the process of rater training is important for establishing interrater reliability of observational instruments, there is little information available in current literature to guide the researcher. In this article, principles and procedures that can be used when rater performance is a critical element of reliability assessment are described. Three phases of the process of rater training are presented: (a) training raters to use the instrument; (b) evaluating rater performance at the end of training; and (c) determining the extent to which rater training is maintained during a reliability study. An example is presented to illustrate how these phases were incorporated in a study to examine the reliability of a measure of patient intensity called the Patient Intensity for Nursing Index (PINI).

A note on interrater agreement

We investigate the properties of a measure of interrater agreement originally proposed by Rogot and Goldberg. Unlike commonly used measures, this measure not only adjusts for chance agreement, but it also standardizes for both perfect agreement as well as for perfect disagreement. Further, one can also use this measure to assess category specific conditional agreement, and thus apply it to situations with missing main diagonal data. We provide an asymptotic method for inference with this measure.

Latent class analysis of diagnostic agreement

We describe methods based on latent class analysis for analysis and interpretation of agreement on dichotomous diagnostic ratings. This approach formulates agreement in terms of parameters directly related to diagnostic accuracy and leads to many practical applications, such as estimation of the accuracy of individual ratings and the extent to which accuracy may improve with multiple opinions. We describe refinements in the estimation of parameters for varying panel designs, and apply latent class methods successfully to examples of medical agreement data that include data previously found to be poorly fitted by two-class models. Latent class techniques provide a powerful and flexible set of tools to analyse diagnostic agreement and one should consider them routinely in the analysis of such data.