Reconciling disparate data to determine the right answer: A grounded theory of meta analysts' reasoning in meta-analysis

Empirical evaluation of SUCRA-based treatment ranks in network meta-analysis: quantifying robustness using Cohen's kappa

OBJECTIVE

To provide a framework for quantifying the robustness of treatment ranks based on Surface Under the Cumulative RAnking curve (SUCRA) in network meta-analysis (NMA) and investigating potential factors associated with lack of robustness.

METHODS

We propose the use of Cohen's kappa to quantify the agreement between SUCRA-based treatment ranks estimated through NMA of a complete data set and a subset of it. We illustrate our approach using five published NMA data sets, where robustness was assessed by removing studies one at a time.

RESULTS

Overall, SUCRA-based treatment ranks were robust to individual studies in the five data sets we considered. We observed more incidences of disagreement between ranks in the networks with larger numbers of treatments. Most treatments moved only one or two ranks up or down. The lowest quadratic weighted kappa estimate observed across all networks was in the network with the smallest number of treatments (4), where weighted kappa=40%. In the network with the largest number of treatments (12), the lowest observed quadratic weighted kappa=89%, reflecting a small shift in this network's treatment ranks overall. Preliminary observations suggest that a study's size, the number of studies making a treatment comparison, and the agreement of a study's estimated treatment effect(s) with those estimated by other studies making the same comparison(s) may explain the overall robustness of treatment ranks to studies.

CONCLUSIONS

Investigating robustness or sensitivity in an NMA may reveal outlying rank changes that are clinically or policy-relevant. Cohen's kappa is a useful measure that permits investigation into study characteristics that may explain varying sensitivity to individual studies. However, this study presents a framework as a proof of concept and further investigation is required to identify potential factors associated with the robustness of treatment ranks using more extensive empirical evaluations.

Single-track year-round education for improving academic achievement in U.S. K-12 schools: Results of a meta-analysis

Background

Research shows that over summer break, students forget approximately 1 month of learning in math and reading; furthermore, some studies find that low-income students lose ground relative to peers. Year-round education (YRE) redistributes schooldays to shorten summer. Prior analyses pooled single-track YRE (academic intervention in which all students attend school on a common calendar) and multitrack YRE (fiscal intervention countering overcrowding, in which groups of students attend school on staggered schedules).

Search Methods

Systematic search of 22 online databases in summer 2017 yielded 494 de-duplicated results; 81 warranted full-text examination. After applying selection criteria, nine studies met criteria but did not report data that allowed effect size calculation. Thirty studies constituted our analytic sample.

Selection Criteria

Studies needed to be of K-12 single-track YRE (not multitrack, not a mix of single- and multitrack, and not a study that did not specify track), with no accompanying extended instructional time. Studies needed to be from 2001 to 2016, include outcome data, and include a comparison group.

Data

We extracted 55 math g, 58 reading g, 29 math odds ratio, and 27 reading odds ratio effect sizes.

Results

Students at single-track YRE schools show modestly higher achievement in both math and reading-by a magnitude similar to estimates of summer learning loss-but comparable proficiency. Unexpectedly, the effect was no greater for historically disadvantaged students. Math effects may be larger in middle than elementary school, but the reason is unclear. Importantly, studies of schools that shortened summer to the fewest weeks showed the largest effects in both subjects.