Grilling the data: application of specification curve analysis to red meat and all-cause mortality

OBJECTIVES

To present an application of specification curve analysis-a novel analytic method that involves defining and implementing all plausible and valid analytic approaches for addressing a research question-to nutritional epidemiology.

STUDY DESIGN AND SETTING

We reviewed all observational studies addressing the effect of red meat on all-cause mortality, sourced from a published systematic review, and documented variations in analytic methods (eg, choice of model, covariates, etc.). We enumerated all defensible combinations of analytic choices to produce a comprehensive list of all the ways in which the data may reasonably be analyzed. We applied specification curve analysis to data from National Health and Nutrition Examination Survey 2007 to 2014 to investigate the effect of unprocessed red meat on all-cause mortality. The specification curve analysis used a random sample of all reasonable analytic specifications we sourced from primary studies.

RESULTS

Among 15 publications reporting on 24 cohorts included in the systematic review on red meat and all-cause mortality, we identified 70 unique analytic methods, each including different analytic models, covariates, and operationalizations of red meat (eg, continuous vs quantiles). We applied specification curve analysis to National Health and Nutrition Examination Survey, including 10,661 participants. Our specification curve analysis included 1208 unique analytic specifications, of which 435 (36.0%) yielded a hazard ratio equal to or more than 1 for the effect of red meat on all-cause mortality and 773 (64.0%) less than 1. The specification curve analysis yielded a median hazard ratio of 0.94 (interquartile range: 0.83-1.05). Forty-eight specifications (3.97%) were statistically significant, 40 of which indicated unprocessed red meat to reduce all-cause mortality and eight of which indicated red meat to increase mortality.

CONCLUSION

We show that the application of specification curve analysis to nutritional epidemiology is feasible and presents an innovative solution to analytic flexibility.

A comprehensive evaluation of the longitudinal association between alcohol consumption and a measure of inflammation: Multiverse and vibration of effects analyses

BACKGROUND

Moderate alcohol consumption appears to be associated with reduced inflammation. Determining whether this association is robust to common variations in research parameters has wide-reaching implications for our understanding of disease aetiology and public health policy. We aimed to conduct comprehensive multiverse and vibration of effects analyses evaluating the associations between alcohol consumption and a measure of inflammation.

METHODS

A secondary analysis of the 1970 British Birth Cohort Study was performed, using data from 1970 through 2016. Measurements of alcohol consumption were taken in early/mid-adulthood (ages 34 and 42), and level of inflammation marker high-sensitivity C-reactive protein (hsCRP) at age 46. Multiverse analyses were applied to comparisons of low-to-moderate consumption and consumption above various international drinking guidelines with an 'abstinent' reference. Research parameters of interest related to: definitions of drinking and reference groups; alcohol consumption measurement year; outcome variable transformation; and breadth of covariate adjustment. After identifying various analytic options within these parameters and running the analysis over each unique option combination, specification curve plots, volcano plots, effect ranges, and variance decomposition metrics were used to assess consistency of results.

RESULTS

A total of 3101 individuals were included in the final analyses, with primary analyses limited to those where occasional consumers served as reference. All combinations of research specifications resulted in lower levels of inflammation amongst low-to-moderate consumers compared to occasional consumers (1st percentile effect: -0.21; 99th percentile effect: -0.04). Estimates comparing above-guidelines drinking with occasional consumers were less definitive (1st percentile effect: -0.26; 99th percentile effect: 0.43).

CONCLUSIONS

The association between low-to-moderate drinking and lower hsCRP levels is largely robust to common variations in researcher-defined parameters, warranting further research to establish whether this relationship is causal. The association between above-guidelines drinking and hsCRP levels is less definitive.

A meta-epidemiological study found that meta-analyses of the same trials may obtain drastically conflicting results

OBJECTIVES

To assess how much protocols reduce methodological flexibility and variation of results in meta-analyses comparing operative with nonoperative treatments for proximal humerus fractures.

STUDY DESIGN AND SETTING

A meta-epidemiological study. We searched four bibliographic databases for eligible meta-analyses. We contacted the authors of the meta-analyses, requesting a copy of their protocol. We identified the included studies and extracted intervention effect data for functional outcome, quality of life, and adverse events. Using the same intervention effect data for each outcome domain, we conducted multiple meta-analyses using differing combinations of methodological choices (analytical scenarios) without restricting the available methodological choices (as if no protocol was used). For each protocol, we repeated this process but restricted the available choices to those listed in the protocol. We then compared the distributions of effect estimates with and without protocols. Methodological flexibility was estimated as the difference in number of possible meta-analyses and the variation of results as the difference between the most conflicting effect estimates.

RESULTS

We identified 23 meta-analyses, included 24 primary studies, and obtained three protocols. The protocols markedly reduced methodological flexibility (range for functional outcomes 94%-99%; quality of life 58%-76%; adverse events 87%-91%). The corresponding reduction in the variation of the results varied (range for functional outcomes; 33%-78%, quality of life; 10%-24%; adverse events 10%-13%).

CONCLUSION

Although the protocols consistently reduced methodological flexibility, the reduction in the variation (bias) of results varied. Consequently, review authors may obtain conflicting results based on the same data. We advise caution when interpreting the conclusions of meta-analyses with absent or imprecise protocols.

Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations

OBJECTIVES

Model specification-what adjusting variables are analytically modeled-may influence results of observational associations. We present a standardized approach to quantify the variability of results obtained with choices of adjustments called the "vibration of effects" (VoE).

STUDY DESIGN AND SETTING

We estimated the VoE for 417 clinical, environmental, and physiological variables in association with all-cause mortality using National Health and Nutrition Examination Survey data. We selected 13 variables as adjustment covariates and computed 8,192 Cox models for each of 417 variables' associations with all-cause mortality.

RESULTS

We present the VoE by assessing the variance of the effect size and in the -log10(P-value) obtained by different combinations of adjustments. We present whether there are multimodality patterns in effect sizes and P-values and the trajectory of results with increasing adjustments. For 31% of the 417 variables, we observed a Janus effect, with the effect being in opposite direction in the 99th versus the 1st percentile of analyses. For example, the vitamin E variant α-tocopherol had a VoE that indicated higher and lower risk for mortality.

CONCLUSION

Estimating VoE offers empirical estimates of associations are under different model specifications. When VoE is large, claims for observational associations should be very cautious.