Cohen’s kappa is a weighted average

Validating a semi-quantitative food frequency questionnaire to assess regional diet in a study of cancer in South West Nigeria

PURPOSE

Cancer burden is rising rapidly in Nigeria, the most populous country in sub-Saharan Africa; this trend may in part be attributable to diet. We developed and validated a semi-quantitative food frequency questionnaire (FFQ) to assess regional diet in Nigeria.

METHODS

We recruited 68 adult participants from rural and urban settings in South West Nigeria. We developed an FFQ administered at baseline and assessed its validity using 3 dietary recalls (baseline, 7 days post baseline, 3 months post baseline). We calculated Spearman's rank correlation coefficients and energy-adjusted de-attenuated correlation coefficients for food items and macronutrients. We evaluated cross-classification using quartiles of macronutrient intake.

RESULTS

Energy-adjusted de-attenuated food item correlations between the FFQ and dietary recalls ranged from -0.08 (smoked beef/goat) to 0.73 (fried snacks) for the average intake from the first 2 recalls (2DR) and from -0.05 (smoked beef/goat) to 0.75 (smoked fish) for the average of all 3 recalls (3DR). Macronutrient correlations ranged from 0.15 (fat) to 0.37 (fiber) for the 2DR and from 0.08 (fat) to 0.41 (carbohydrates) for the 3DR. The percent of participants classified in the same quartile ranged from 16.4% (fat) to 32.8% (fiber, protein) for the 2DR and from 25.6% (fat) to 34.9% (carbohydrates) for the 3DR. Agreement improved when including adjacent quartiles, from 65.5% (carbohydrates) to 70.5% (fat, fiber) for the 2DR and from 62.8% (protein) to 76.8% (carbohydrate) for the 3DR.

CONCLUSION

Our semi-quantitative FFQ was reasonably valid for ranking intake of certain foods and macronutrients in adults in South West Nigeria.

Reliability and Validity of a Nationwide Survey (the Korean Radiation Workers Study)

Background

This study aimed to evaluate the reliability and validity of the self-administered questionnaire for Korean radiation workers.

Methods

From May 24, 2016, to June 30, 2017, 20,608 participants completed the questionnaire, providing information on sociodemographics, lifestyle, work history and practices, medical radiation exposure, and medical history, which was linked to the National Dose Registry and the National Cancer Registry. The validity of the questionnaire was evaluated using the responses of 20,608 workers, and reliability was evaluated using the responses of 3043 workers who responded to the survey twice.

Results

Responses concerning demographic characteristics and lifestyle showed reliability with a moderate-to-high agreement (kappa: 0.43–0.99), whereas responses concerning occupation and medical radiation exposure had a wide range of agreement (kappa: 0.05–0.95), possibly owing to temporal variability during employment. Regarding validity, responses to the question about the first year of employment had an excellent agreement with the national registry (intraclass correlation coefficient = 0.9); however, responses on cancer history had a wide range of agreement (kappa: 0.22–0.85).

Conclusion

Although the reliability and validity of the questionnaire were not distinguished by demographic characteristics, they tended to be low among participants whose occupational radiation exposure was minimal. Overall, the information collected can be reliable for epidemiological studies; however, caution must be exercised when using information such as medical exposure and work practices, which are prone to temporal variability.

Tutorial: multivariate classification for vibrational spectroscopy in biological samples

Vibrational spectroscopy techniques, such as Fourier-transform infrared (FTIR) and Raman spectroscopy, have been successful methods for studying the interaction of light with biological materials and facilitating novel cell biology analysis. Spectrochemical analysis is very attractive in disease screening and diagnosis, microbiological studies and forensic and environmental investigations because of its low cost, minimal sample preparation, non-destructive nature and substantially accurate results. However, there is now an urgent need for multivariate classification protocols allowing one to analyze biologically derived spectrochemical data to obtain accurate and reliable results. Multivariate classification comprises discriminant analysis and class-modeling techniques where multiple spectral variables are analyzed in conjunction to distinguish and assign unknown samples to pre-defined groups. The requirement for such protocols is demonstrated by the fact that applications of deep-learning algorithms of complex datasets are being increasingly recognized as critical for extracting important information and visualizing it in a readily interpretable form. Hereby, we have provided a tutorial for multivariate classification analysis of vibrational spectroscopy data (FTIR, Raman and near-IR) highlighting a series of critical steps, such as preprocessing, data selection, feature extraction, classification and model validation. This is an essential aspect toward the construction of a practical spectrochemical analysis model for biological analysis in real-world applications, where fast, accurate and reliable classification models are fundamental.

Kappa coefficients for dichotomous-nominal classifications

Two types of nominal classifications are distinguished, namely regular nominal classifications and dichotomous-nominal classifications. The first type does not include an ‘absence’ category (for example, no disorder), whereas the second type does include an ‘absence’ category. Cohen’s unweighted kappa can be used to quantify agreement between two regular nominal classifications with the same categories, but there are no coefficients for assessing agreement between two dichotomous-nominal classifications. Kappa coefficients for dichotomous-nominal classifications with identical categories are defined. All coefficients proposed belong to a one-parameter family. It is studied how the coefficients for dichotomous-nominal classifications are related and if the values of the coefficients depend on the number of categories. It turns out that the values of the new kappa coefficients can be strictly ordered in precisely two ways. The orderings suggest that the new coefficients are measuring the same thing, but to a different extent. If one accepts the use of magnitude guidelines, it is recommended to use stricter criteria for the new coefficients that tend to produce higher values.

The Use of Social Digital Networks by NGDO from a Social Marketing Perspective

The social marketing paradigm has been changing due to the use of digital social networks. This causes Non-Governmental Development Organizations’ efforts to focus on achieving a greater reaction from the public in these communication channels. We propose that the way forward is to analyze aspects of messages that give rise to a greater response from the audience. In this regard, we have analyzed 3608 Facebook and Twitter publications with the combination of content analysis and correlation analysis. We have considered three aspects: purpose, theme, and quality of the message. We have also listed a breakdown of quality and purpose parameters in order to become more fully acquainted with these aspects. The objectives of this research are firstly to carry out the communication profiles of the NGDOs studied from the points of view of the organizations and the public. Secondly, to analyze the reaction from the public (interactions) measured by the sum of likes plus the number of shares for each post, on Facebook and Twitter, according the parameters considered. The results showed that the most published messages from the organizations do not usually coincide with those that have the most impact on the public. Another proven aspect is that Twitter posts about behavior have more effectiveness than informative messages. Likewise, quality aspects, such as hashtags, mentions, or links, are not succeeding in generating public reaction.

Improved simulation-optimization approach for identifying critical and developable pollution source regions and critical migration processes for pollutant load allocation

Simulation-optimization approaches are widely used in land-based pollutant load allocation and management. However, existing simulation-optimization approaches do not optimize the entire pathway of pollution (i.e., generation, transportation, and emission). Thus, the selected allocated load scheme and its reduction management are incomplete and non-optimal in supporting effective reduction measures to set in further. This study aims to establish a simulation-optimization load allocation approach covering the entire pathway of pollution and construct separate allocated loads for each migration process. By comparing the allocated loads to the actual pollutant loads, the critical and developable pollution source regions (CPSRs and DPSRs) and the critical pollution migration processes (CPMPs) are identified, and specific pollution management indicators are established for each pollution source region. This approach is applied to the Bohai Rim, which is currently a major economic development center with a most seriously deteriorating sea area in China. The CPSRs, DPSRs, and CPMPs for total nitrogen are identified, and comparisons of the management indicators with existing approaches are made. Water quality under the allocated load emissions is simulated via a simulation water quality model and compared with that of the actual pollutant loads. Quantitative results indicate that the new approach combines the specific mitigation indicators for each CPSR and CPMP when allowing the extra generation of allocated loads for DPRSs while keeping the water quality of the Bohai Sea under control.

Assessing the fitness-for-purpose of satellite multi-mission ocean color climate data records: A protocol applied to OC-CCI chlorophyll-a data

In this work, trend estimates are used as indicators to compare the multi-annual variability of different satellite chlorophyll-a (Chla) data and to assess the fitness-for-purpose of multi-mission Chla products as climate data records (CDR). Under the assumption that single-mission products are free from spurious temporal artifacts and can be used as benchmark time series, multi-mission CDRs should reproduce the main trend patterns observed by single-mission series when computed over their respective periods. This study introduces and applies quantitative metrics to compare trend distributions from different data records. First, contingency matrices compare the trend diagnostics associated with two satellite products when expressed in binary categories such as existence, significance and signs of trends. Contingency matrices can be further summarized by metrics such as Cohen's κ index that rates the overall agreement between the two distributions of diagnostics. A more quantitative measure of the discrepancies between trends is provided by the distributions of differences between trend slopes. Thirdly, maps of the level of significance P of a t-test quantifying the degree to which two trend estimates differ provide a statistical, spatially-resolved, evaluation. The proposed methodology is applied to the multi-mission Ocean Colour-Climate Change Initiative (OC-CCI) Chla data. The agreement between trend distributions associated with OC-CCI data and single-mission products usually appears as good as when single-mission products are compared. As the period of analysis is extended beyond 2012 to 2015, the level of agreement tends to be degraded, which might be at least partly due to the aging of the MODIS sensor on-board Aqua. On the other hand, the trends displayed by the OC-CCI series over the short period 2012-2015 are very consistent with those observed with VIIRS. These results overall suggest that the OC-CCI Chla data can be used for multi-annual time series analysis (including trend detection), but with some caution required if recent years are included, particularly in the central tropical Pacific. The study also recalls the challenges associated with creating a multi-mission ocean color data record suitable for climate research.

Measuring care trajectories using health administrative databases: a population-based investigation of transitions from emergency to acute care

Background

A patient’s trajectory through the healthcare system affects resource use and outcomes. Data fields in population-based administrative health databases are potentially valuable resources for constructing care trajectories for entire populations, provided they can capture patient transitions between healthcare services. This study describes patient transitions from the emergency department (ED) to other healthcare settings, and ascertains whether the discharge disposition field recorded in the ED data was a reliable source of patient transition information from the emergency to the acute care settings.

Methods

Administrative health databases from the province of Saskatchewan, Canada (population 1.1 million) were used to identify patients with at least one ED visit to provincial teaching hospitals (n = 5) between April 1, 2006 and March 31, 2012. Discharge disposition from ED was described using frequencies and percentages; and it includes categories such as home, transfer to other facilities, and died. The kappa statistic with 95 % confidence intervals (95 % CIs) was used to measure agreement between the discharge disposition field in the ED data and hospital admission records.

Results

We identified N = 1,062,861 visits for 371,480 patients to EDs over the six-year study period. Three-quarters of the discharges were to home, 16.1 % were to acute care in the same facility in which the ED was located, and 1.6 % resulted in a patient transfer to a different acute care facility. Agreement between the discharge disposition field in the ED data and hospital admission records was good when the emergency and acute care departments were in the same facility (κ = 0.77, 95 % CI 0.77, 0.77). For transfers to a different acute care facility, agreement was only fair (κ = 0.36, 95 % CI 0.35, 0.36).

Conclusions

The majority of patients who attended EDs did not transition to another healthcare setting. For those who transitioned to acute care, accuracy of the discharge disposition field depended on whether the two services were provided in the same facility. Using the hospital data as reference, we conclude that the discharge disposition field in the ED data is not reliable for measuring transitions from ED to acute care.

Inferior turbinate classification system, grades 1 to 4: development and validation study

OBJECTIVES/HYPOTHESIS

To develop a validated inferior turbinate grading scale.

STUDY DESIGN

Development and validation study.

METHODS

Phase 1 development (alpha test) consisted of a proposal of 10 different inferior turbinate grading scales (>1,000 clinic patients). Phase 2 validation (beta test) utilized 10 providers grading 27 standardized endoscopic photos of inferior turbinates using two different classification systems. Phase 3 validation (pilot study) consisted of 100 live consecutive clinic patients (n = 200 inferior turbinates) who were each prospectively graded by 18 different combinations of two independent raters, and grading was repeated by each of the same two raters, two separate times for each patient.

RESULTS

In the development phase, 25% (grades 1-4) and 33% (grades 1-4) were the most useful systems. In the validation phase, the 25% classification system was found to be the best balance between potential clinical utility and ability to grade; the photo grading demonstrated a Cohen's kappa (κ) = 0.4671 ± 0.0082 (moderate inter-rater agreement). Live-patient grading with the 25% classification system demonstrated an overall inter-rater reliability of 71.5% (95% confidence interval [CI]: 64.8-77.3), with overall substantial agreement (κ = 0.704 ± 0.028). Intrarater reliability was 91.5% (95% CI: 88.7-94.3). Distribution for the 200 inferior turbinates was as follows: 25% quartile = grade 1, 50% quartile (median) = grade 2, 75% quartile = grade 3, and 90% quartile = grade 4. Mean turbinate size was 2.22 (95% CI: 2.07-2.34; standard deviation 1.02). Categorical κ was as follows: grade 1, 0.8541 ± 0.0289; grade 2, 0.7310 ± 0.0289; grade 3, 0.6997 ± 0.0289, and grade 4, 0.7760 ± 0.0289.

CONCLUSIONS

The 25% (grades 1-4) inferior turbinate classification system is a validated grading scale with high intrarater and inter-rater reliability. This system can facilitate future research by tracking the effect of interventions on inferior turbinates.

LEVEL OF EVIDENCE

2c.

The Effect of the Raters' Marginal Distributions on Their Matched Agreement: A Rescaling Framework for Interpreting Kappa

Cohen's κ measures the improvement in classification above chance level and it is the most popular measure of interjudge agreement. Yet, there is considerable confusion about its interpretation. Specifically, researchers often ignore the fact that the observed level of matched agreement is bounded from above and below and the bounds are a function of the particular marginal distributions of the table. We propose that these bounds should be used to rescale the components of κ (observed and expected agreement). Rescaling κ in this manner results in κ', a measure that was originally proposed by Cohen (1960) and was largely ignored in both research and practice. This measure provides a common scale for agreement measures of tables with different marginal distributions. It reaches the maximal value of 1 when the judges show the highest level of agreement possible, given their marginal disagreements. We conclude that κ' should be used to measure the level of matched agreement contingent on a particular set of marginal distributions. The article provides a framework and a set of guidelines that facilitate comparisons between various types of agreement tables. We illustrate our points with simulations and real data from two studies-one involving judges' ratings of baseball players and one involving ratings of essays in high-stakes tests.

Weighted Kappas for3×3Tables

Weighted kappa is a widely used statistic for summarizing inter-rater agreement on a categorical scale. For rating scales with three categories, there are seven versions of weighted kappa. It is shown analytically how these weighted kappas are related. Several conditional equalities and inequalities between the weighted kappas are derived. The analytical analysis indicates that the weighted kappas are measuring the same thing but to a different extent. One cannot, therefore, use the same magnitude guidelines for all weighted kappas.