Comment on "School-Based Obesity Prevention Intervention in Chilean Children: Effective in Controlling, but not Reducing Obesity"

Errors in the implementation, analysis, and reporting of randomization within obesity and nutrition research: a guide to their avoidance

Randomization is an important tool used to establish causal inferences in studies designed to further our understanding of questions related to obesity and nutrition. To take advantage of the inferences afforded by randomization, scientific standards must be upheld during the planning, execution, analysis, and reporting of such studies. We discuss ten errors in randomized experiments from real-world examples from the literature and outline best practices for their avoidance. These ten errors include: representing nonrandom allocation as random, failing to adequately conceal allocation, not accounting for changing allocation ratios, replacing subjects in nonrandom ways, failing to account for non-independence, drawing inferences by comparing statistical significance from within-group comparisons instead of between-groups, pooling data and breaking the randomized design, failing to account for missing data, failing to report sufficient information to understand study methods, and failing to frame the causal question as testing the randomized assignment per se. We hope that these examples will aid researchers, reviewers, journal editors, and other readers to endeavor to a high standard of scientific rigor in randomized experiments within obesity and nutrition research.

Childhood obesity intervention studies: A narrative review and guide for investigators, authors, editors, reviewers, journalists, and readers to guard against exaggerated effectiveness claims

Being able to draw accurate conclusions from childhood obesity trials is important to make advances in reversing the obesity epidemic. However, obesity research sometimes is not conducted or reported to appropriate scientific standards. To constructively draw attention to this issue, we present 10 errors that are commonly committed, illustrate each error with examples from the childhood obesity literature, and follow with suggestions on how to avoid these errors. These errors are as follows: using self-reported outcomes and teaching to the test; foregoing control groups and risking regression to the mean creating differences over time; changing the goal posts; ignoring clustering in studies that randomize groups of children; following the forking paths, subsetting, p-hacking, and data dredging; basing conclusions on tests for significant differences from baseline; equating "no statistically significant difference" with "equally effective"; ignoring intervention study results in favor of observational analyses; using one-sided testing for statistical significance; and stating that effects are clinically significant even though they are not statistically significant. We hope that compiling these errors in one article will serve as the beginning of a checklist to support fidelity in conducting, analyzing, and reporting childhood obesity research.

Trial Characteristics and Appropriateness of Statistical Methods Applied for Design and Analysis of Randomized School-Based Studies Addressing Weight-Related Issues: A Literature Review

OBJECTIVE

To evaluate whether clustering effects, often quantified by the intracluster correlation coefficient (ICC), were appropriately accounted for in design and analysis of school-based trials.

METHODS

We searched PubMed and extracted variables concerning study characteristics, power analysis, ICC use for power analysis, applied statistical models, and the report of the ICC estimated from the observed data.

RESULTS

N=263 papers were identified, and N=121 papers were included for evaluation. Overall, only a minority (21.5%) of studies incorporated ICC values for power analysis, fewer studies (8.3%) reported the estimated ICC, and 68.6% of studies applied appropriate multilevel models. A greater proportion of studies applied the appropriate models during the past five years (2013-2017) compared to the prior years (74.1% versus 63.5%, p=0.176). Significantly associated with application of appropriate models were a larger number of schools (p=0.030), a larger sample size (p=0.002), longer follow-up (p=0.014), and randomization at a cluster level (p < 0.001) and so were studies that incorporated the ICC into power analysis (p=0.016) and reported the estimated ICC (p=0.030).

CONCLUSION

Although application of appropriate models has increased over the years, consideration of clustering effects in power analysis has been inadequate, as has report of estimated ICC. To increase rigor, future school-based trials should address these issues at both the design and analysis stages.

Overnutrition and Scholastic Achievement: Is There a Relationship? An 8-Year Follow-Up Study

OBJECTIVE

The aim of this study was to assess the association between overnutrition and scholastic achievement (SA).

METHODS

A representative and proportional sample of 477 children of the 5th elementary school grade of both genders was randomly chosen during 2010, in the Metropolitan Region of Chile. SA was measured through the 2009 Education Quality Measurement System (SIMCE) tests of language (LSA), mathematics (MSA) and understanding of the natural environment (UNESA). Current nutritional status was assessed through the body mass index Z-score (Z-BMI). Nutritional quality of diet, schedule exercise, socioeconomic status, family, and educational variables were also recorded. Four and 8 years later, SA was assessed through the 2013 SIMCE and the University Selection Test (2017 PSU), respectively.

RESULTS

Socioeconomic status, the number of repeated school years, and maternal schooling were strong predictors of 2009 SIMCE and the independent variables with the greatest explanatory power for LSA (Model R2 = 0.178; p < 0.00001) variances, besides of gender for MSA (Model R2 = 0.205; p< 0.00001) and UNESA (Model R2 = 0.272; p < 0.00001). Overnourished children did not have significantly lower 2009 and 2013 SIMCE and 2017 PSU outcomes.

CONCLUSIONS

These results confirm that overnourished children did not achieve significantly lower SA.

Common scientific and statistical errors in obesity research

This review identifies 10 common errors and problems in the statistical analysis, design, interpretation, and reporting of obesity research and discuss how they can be avoided. The 10 topics are: 1) misinterpretation of statistical significance, 2) inappropriate testing against baseline values, 3) excessive and undisclosed multiple testing and "P-value hacking," 4) mishandling of clustering in cluster randomized trials, 5) misconceptions about nonparametric tests, 6) mishandling of missing data, 7) miscalculation of effect sizes, 8) ignoring regression to the mean, 9) ignoring confirmation bias, and 10) insufficient statistical reporting. It is hoped that discussion of these errors can improve the quality of obesity research by helping researchers to implement proper statistical practice and to know when to seek the help of a statistician.