The Dependence of Chance-Corrected Weighted Agreement Coefficients on the Power Parameter of the Weighting Scheme: Analysis and Measurement

We consider the dependence of a broad class of chance-corrected weighted agreement coefficients on the weighting scheme that penalizes rater disagreements. The considered class encompasses many existing coefficients with any number of raters, and one real-valued power parameter defines the weighting scheme that includes linear, quadratic, identity, and radical weights. We obtain the first-order and second-order derivatives of the coefficients with respect to the power parameter and decompose them into components corresponding to all pairs of different category distances. Each component compares its two distances in terms of the ratio of observed to expected-by-chance frequency. A larger ratio for the smaller distance than the larger distance contributes to a positive relationship between the power parameter and the coefficient value; the opposite contributes to a negative relationship. We provide necessary and sufficient conditions for the coefficient value to increase or decrease and the relationship to intensify or weaken as the power parameter increases. We use the first-order and second-order derivatives for corresponding measurement. Furthermore, we show how these two derivatives allow other researchers to obtain quite accurate estimates of the coefficient value for unreported values of the power parameter, even without access to the original data.

Standardization of evaluation method and prognostic significance of histological response to preoperative chemotherapy in high-grade non-round cell soft tissue sarcomas

Background

Preoperative chemotherapy is widely applied to high-grade localized soft tissue sarcomas (STSs); however, the prognostic significance of histological response to chemotherapy remains controversial. This study aimed to standardize evaluation method of histological response to chemotherapy with high agreement score among pathologists, and to establish a cut-off value closely related to prognosis.

Methods

Using data and specimens from the patients who had registered in the Japan Clinical Oncology Group study, JCOG0304, a phase II trial evaluating the efficacy of perioperative chemotherapy with doxorubicin (DOX) and ifosfamide (IFO), we evaluated histological response to preoperative chemotherapy at the central review board.

Results

A total of 64 patients were eligible for this study. The percentage of viable tumor area ranged from 0.1% to 97.0%, with median value of 35.7%. Regarding concordance proportion between pathologists, the weighted kappa coefficient (κ) score in all patients was 0.71, indicating that the established evaluation method achieved substantial agreement score. When the cut-off value of the percentage of the residual tumor area was set as 25%, the p-value for the difference in overall survival showed the minimum value. Hazard ratio of the non-responder with percentage of the residual tumor < 25%, to the responder was 4.029 (95% confidence interval 0.893–18.188, p = 0.070).

Conclusion

The standardized evaluation method of pathological response to preoperative chemotherapy showed a substantial agreement in the weighted κ score. The evaluation method established here was useful for estimating of the prognosis in STS patients who were administered perioperative chemotherapy with DOX and IFO.

Trial registration

UMIN Clinical Trials Registry C000000096. Registered 30 August, 2005 (retrospectively registered).

Summary measures of agreement and association between many raters' ordinal classifications

PURPOSE

Interpretation of screening tests such as mammograms usually require a radiologist's subjective visual assessment of images, often resulting in substantial discrepancies between radiologists' classifications of subjects' test results. In clinical screening studies to assess the strength of agreement between experts, multiple raters are often recruited to assess subjects' test results using an ordinal classification scale. However, using traditional measures of agreement in some studies is challenging because of the presence of many raters, the use of an ordinal classification scale, and unbalanced data.

METHODS

We assess and compare the performances of existing measures of agreement and association as well as a newly developed model-based measure of agreement to three large-scale clinical screening studies involving many raters' ordinal classifications. We also conduct a simulation study to demonstrate the key properties of the summary measures.

RESULTS

The assessment of agreement and association varied according to the choice of summary measure. Some measures were influenced by the underlying prevalence of disease and raters' marginal distributions and/or were limited in use to balanced data sets where every rater classifies every subject. Our simulation study indicated that popular measures of agreement and association are prone to underlying disease prevalence.

CONCLUSIONS

Model-based measures provide a flexible approach for calculating agreement and association and are robust to missing and unbalanced data as well as the underlying disease prevalence.

Assessing the influence of rater and subject characteristics on measures of agreement for ordinal ratings

Widespread inconsistencies are commonly observed between physicians' ordinal classifications in screening tests results such as mammography. These discrepancies have motivated large-scale agreement studies where many raters contribute ratings. The primary goal of these studies is to identify factors related to physicians and patients' test results, which may lead to stronger consistency between raters' classifications. While ordered categorical scales are frequently used to classify screening test results, very few statistical approaches exist to model agreement between multiple raters. Here we develop a flexible and comprehensive approach to assess the influence of rater and subject characteristics on agreement between multiple raters' ordinal classifications in large-scale agreement studies. Our approach is based upon the class of generalized linear mixed models. Novel summary model-based measures are proposed to assess agreement between all, or a subgroup of raters, such as experienced physicians. Hypothesis tests are described to formally identify factors such as physicians' level of experience that play an important role in improving consistency of ratings between raters. We demonstrate how unique characteristics of individual raters can be assessed via conditional modes generated during the modeling process. Simulation studies are presented to demonstrate the performance of the proposed methods and summary measure of agreement. The methods are applied to a large-scale mammography agreement study to investigate the effects of rater and patient characteristics on the strength of agreement between radiologists. Copyright © 2017 John Wiley & Sons, Ltd.

Measures of agreement between many raters for ordinal classifications

Screening and diagnostic procedures often require a physician's subjective interpretation of a patient's test result using an ordered categorical scale to define the patient's disease severity. Because of wide variability observed between physicians' ratings, many large-scale studies have been conducted to quantify agreement between multiple experts' ordinal classifications in common diagnostic procedures such as mammography. However, very few statistical approaches are available to assess agreement in these large-scale settings. Many existing summary measures of agreement rely on extensions of Cohen's kappa. These are prone to prevalence and marginal distribution issues, become increasingly complex for more than three experts, or are not easily implemented. Here we propose a model-based approach to assess agreement in large-scale studies based upon a framework of ordinal generalized linear mixed models. A summary measure of agreement is proposed for multiple experts assessing the same sample of patients' test results according to an ordered categorical scale. This measure avoids some of the key flaws associated with Cohen's kappa and its extensions. Simulation studies are conducted to demonstrate the validity of the approach with comparison with commonly used agreement measures. The proposed methods are easily implemented using the software package R and are applied to two large-scale cancer agreement studies.

A Monte Carlo-Based Bayesian Approach for Measuring Agreement in a Qualitative Scale

Agreement analysis has been an active research area whose techniques have been widely applied in psychology and other fields. However, statistical agreement among raters has been mainly considered from a classical statistics point of view. Bayesian methodology is a viable alternative that allows the inclusion of subjective initial information coming from expert opinions, personal judgments, or historical data. A Bayesian approach is proposed by providing a unified Monte Carlo-based framework to estimate all types of measures of agreement in a qualitative scale of response. The approach is conceptually simple and it has a low computational cost. Both informative and non-informative scenarios are considered. In case no initial information is available, the results are in line with the classical methodology, but providing more information on the measures of agreement. For the informative case, some guidelines are presented to elicitate the prior distribution. The approach has been applied to two applications related to schizophrenia diagnosis and sensory analysis.

The effect of laryngoscope handle size on possible endotracheal intubation success in university football, ice hockey, and soccer players

OBJECTIVE

To assess the effectiveness of a standard long-handle laryngoscope and a short-handle laryngoscope on ease of possible intubation in football, ice hockey, and soccer players.

DESIGN

Prospective crossover study.

SETTING

University Sport Medicine Clinic.

PARTICIPANTS

Sixty-two university varsity football (62 males), 45 ice hockey (26 males and 19 females), and 39 soccer players (20 males, 19 females).

INTERVENTIONS

Athletes were assessed for different airway and physical characteristics. Three different physicians then assessed the use of laryngoscopes of different handle sizes in supine athletes who were wearing protective equipment while in-line cervical spine immobilization was maintained.

MAIN OUTCOME MEASURES

The ease of passage of a laryngoscope blade into the posterior oropharynx of a supine athlete was assessed using both a standard long-handle and a short-handle laryngoscope.

RESULTS

Use of a short-handle laryngoscope was easier for all physicians in all sports as compared with a standard-sized laryngoscope. Passage of a laryngoscope blade into the posterior oropharynx of a supine athlete was easiest in soccer players and most difficult in football and ice hockey players for both sizes of laryngoscope. Interference from chest or shoulder pads was a common cause for difficulty in passing the laryngoscope blade into the posterior oropharynx for football and ice hockey players.

CONCLUSIONS

In the rare instances that an endotracheal intubation is to be attempted on an unconscious athlete, a short-handle laryngoscope may provide the best chance for successful intubation.

The assessment of airway maneuvers and interventions in university Canadian football, ice hockey, and soccer players

CONTEXT

Managing an airway in an unconscious athlete is a lifesaving skill that may be made more difficult by the recent changes in protective equipment. Different airway maneuvers and techniques may be required to help ventilate an unconscious athlete who is wearing full protective equipment.

OBJECTIVE

To assess the effectiveness of different airway maneuvers with football, ice hockey, and soccer players wearing full protective equipment.

DESIGN

Crossover study.

SETTING

University sports medicine clinic.

PATIENTS OR OTHER PARTICIPANTS

A total of 146 university varsity athletes, consisting of 62 football, 45 ice hockey, and 39 soccer players.

INTERVENTION(S)

Athletes were assessed for different airway and physical characteristics. Three investigators then evaluated the effectiveness of different bag-valve-mask (BVM) ventilation techniques in supine athletes who were wearing protective equipment while inline cervical spine immobilization was maintained.

MAIN OUTCOME MEASURE(S)

The effectiveness of 1-person BVM ventilation (1-BVM), 2-person BVM ventilation (2-BVM), and inline immobilization and ventilation (IIV) was judged by each investigator for each athlete using a 4-point rating scale.

RESULTS

All forms of ventilation were least difficult in soccer players and most difficult in football players. When compared with 1-BVM, both 2-BVM and IIV were deemed more effective by all investigators for all athletes. Interference from the helmet and stabilizer were common reasons for difficult ventilation in football and ice hockey players.

CONCLUSIONS

Sports medicine professionals should practice and be comfortable with different ventilation techniques for athletes wearing full equipment. The use of a new ventilation technique, termed inline immobilization and ventilation, may be beneficial, especially when the number of responders is limited.

Weighted Kappa for Multiple Raters

Five procedures to calculate the probability of weighted kappa with multiple raters under the null hypothesis of independence are described and compared in terms of accuracy, ease of use, generality, and limitations. The five procedures are (1) exact variance, (2) resampling contingency, (3) intraclass correlation, (4) randomized block, and (5) resampling block. While each procedure possesses strengths and limitations, the resampling contingency procedure is shown to be the most versatile and accurate of the five procedures, provided the number of raters is not too large. The resampling contingency procedure permits any weighting scheme, accommodates both symmetrical and asymmetrical weights, is suitable for both weighted and unweighted kappa, and makes no assumptions about either the data distribution or the probability distribution.

Resampling Probability Values for Weighted Kappa with Multiple Raters

A new procedure to compute weighted kappa with multiple raters is described. A resampling procedure to compute approximate probability values for weighted kappa with multiple raters is presented. Applications of weighted kappa are illustrated with an example analysis of classifications by three independent raters.