Increasing the degrees of freedom in existing group randomized trials: the df* approach

Long-term impacts of the CARE program on teachers' self-reported social and emotional competence and well-being

Teacher stress is at an all-time high, negatively impacting the quality of education and student outcomes. In recent years, mindfulness-based interventions have been shown to promote well-being and reduce stress among healthy adults. In particular, mindfulness-based interventions enhance emotion regulation and reduce psychological distress. One such program specifically designed to address teacher stress is Cultivating Awareness and Resilience in Education (CARE). The present study examined teachers' self-reported data collected at three time points over two consecutive school years as part of a randomized controlled trial of CARE. The study involved 224 teachers in 36 elementary schools in high poverty areas of New York City. Teachers were randomly assigned within schools to receive CARE or to a waitlist control group. This study builds on previous experimental evidence of the impacts of CARE on teacher self-reported outcomes for this sample of teachers within one school year (Jennings et al., 2017). Results indicate that at the third assessment point (9.5 months after participating in the program), CARE teachers showed continued significant decreases in psychological distress, reductions in ache-related physical distress, continued significant increases in emotion regulation and some dimensions of mindfulness. Findings indicate that teachers who participated in mindfulness-based professional development through CARE reported both sustained and new benefits regarding their well-being at a follow-up assessment almost one-year post-intervention compared to teachers in the control condition. Implications for further research and policy are discussed.

Increasing the degrees of freedom in future group randomized trials: the df * method revisited

BACKGROUND

This article revisits an article published in Evaluation Review in 2005 on sample size estimation and power analysis for group-randomized trials. With help from a careful reader, we learned of an important error in the spreadsheet used to perform the calculations and generate the results presented in that article. As we studied the spreadsheet, we discovered other minor errors. When we corrected the errors, we found that the results were substantially different and that the conclusions reported in the original article were not always appropriate.

OBJECTIVE

This article corrects the errors and reports the results as they should have been reported originally.

METHOD

Using a random-effects meta-analytic model, estimates of intraclass correlation were combined from two studies to guide sample size calculations for a new study.

RESULTS

The df * method can result in improved power or smaller studies when used a priori to plan future group-randomized trials, though the improvements will be modest in larger studies and will likely be insufficient to provide adequate power to small studies.

CONCLUSION

Smaller group-randomized trials are often desirable, for example, as pilot studies to help plan for a full-scale efficacy trial, as replication studies, or in situations in which resource constraints prohibit a larger trial. We discuss the circumstances under which the df * method will be most helpful and the risks associated with conducting smaller studies.

Efficient Bayesian joint models for group randomized trials with multiple observation times and multiple outcomes

In this paper, we propose a Bayesian method for group randomized trials with multiple observation times and multiple outcomes of different types. We jointly model these outcomes using latent multivariate normal linear regression, which allows treatment effects to change with time and accounts for (i) intraclass correlation within groups; (ii) the correlation between different outcomes measured on the same subject; and (iii) the over-time correlation of each outcome. Moreover, we develop a set of innovative priors for the variance components, which yield direct inference on the correlations, avoid undesirable constraints, and allow utilization of information from previous studies. We illustrate through simulations that our model can improve estimation efficiency (lower posterior standard deviations) of intraclass correlations and treatment effects relative to single outcome models and models with diffuse priors on the variance components. We also demonstrate the methodology using body composition data collected in the Trial of Activity in Adolescent Girls.

Intraclass correlation associated with therapists: estimates and applications in planning psychotherapy research

It is essential that outcome research permit clear conclusions to be drawn about the efficacy of interventions. The common practice of nesting therapists within conditions can pose important methodological challenges that affect interpretation, particularly if the study is not powered to account for the nested design. An obstacle to the optimal design of these studies is the lack of data about the intraclass correlation coefficient (ICC), which measures the statistical dependencies introduced by nesting. To begin the development of a public database of ICC estimates, the authors investigated ICCs for a variety outcomes reported in 20 psychotherapy outcome studies. The magnitude of the 495 ICC estimates varied widely across measures and studies. The authors provide recommendations regarding how to select and aggregate ICC estimates for power calculations and show how researchers can use ICC estimates to choose the number of patients and therapists that will optimize power. Attention to these recommendations will strengthen the validity of inferences drawn from psychotherapy studies that nest therapists within conditions.

Correcting an analysis of variance for clustering

A great deal of educational and social data arises from cluster sampling designs where clusters involve schools, classrooms, or communities. A mistake that is sometimes encountered in the analysis of such data is to ignore the effect of clustering and analyse the data as if it were based on a simple random sample. This typically leads to an overstatement of the precision of results and too liberal conclusions about precision and statistical significance of mean differences. This paper gives simple corrections to the test statistics that would be computed in an analysis of variance if clustering were (incorrectly) ignored. The corrections are multiplicative factors depending on the total sample size, the cluster size, and the intraclass correlation structure. For example, the corrected F statistic has Fisher's F distribution with reduced degrees of freedom. The corrected statistic reduces to the F statistic computed by ignoring clustering when the intraclass correlations are zero. It reduces to the F statistic computed using cluster means when the intraclass correlations are unity, and it is in between otherwise. A similar adjustment to the usual statistic for testing a linear contrast among group means is described.

Intraclass correlation estimates for cancer screening outcomes: estimates and applications in the design of group-randomized cancer screening studies

BACKGROUND

Screening has become one of our best tools for early detection and prevention of cancer. The group-randomized trial is the most rigorous experimental design for evaluating multilevel interventions. However, identifying the proper sample size for a group-randomized trial requires reliable estimates of intraclass correlation (ICC) for screening outcomes, which are not available to researchers. We present crude and adjusted ICC estimates for cancer screening outcomes for various levels of aggregation (physician, clinic, and county) and provide an example of how these ICC estimates may be used in the design of a future trial.

METHODS

Investigators working in the area of cancer screening were contacted and asked to provide crude and adjusted ICC estimates using the analysis of variance method estimator.

RESULTS

Of the 29 investigators identified, estimates were obtained from 10 investigators who had relevant data. ICC estimates were calculated from 13 different studies, with more than half of the studies collecting information on colorectal screening. In the majority of cases, ICC estimates could be adjusted for age, education, and other demographic characteristics, leading to a reduction in the ICC. ICC estimates varied considerably by cancer site and level of aggregation of the groups.

CONCLUSIONS

Previously, only two articles had published ICCs for cancer screening outcomes. We have complied more than 130 crude and adjusted ICC estimates covering breast, cervical, colon, and prostate screening and have detailed them by level of aggregation, screening measure, and study characteristics. We have also demonstrated their use in planning a future trial and the need for the evaluation of the proposed interval estimator for binary outcomes under conditions typically seen in GRTs.

Efficacy of a randomized trial of a community and school-based anti-violence media intervention among small-town middle school youth

In a community randomized controlled trial, intervention middle school students from small towns were exposed to a community and school-based anti-violence intervention ("Resolve It, Solve It"). The primary intervention was a media campaign in which local high school students served as models in print, radio, and television PSAs and spearheaded local school and community activities. The media campaign was supported with school and community events that reinforced campaign messages. Tests of recognition and recall indicated widespread exposure to the media intervention. Multiple group latent growth models indicated that relative to control students, intervention students reported significant differences in rates of growth for intent for violence, physical assault against people, verbal victimization, and perceived safety at school. No differences were found for verbal assault, physical assault against objects, physical victimization, or self-efficacy for avoiding violence. When examined by sex, it was determined that results for physical assault against people were obtained only among female students, and changes in verbal victimization and perceived school safety were observed only among male students. These results suggest that a media and reinforcing community intervention led by older peers can alter rates of growth for some measures of violence and associated factors among small-town youth. Further research is indicated to determine how different campaign messages influence students by sex.

Sizing a trial to alter the trajectory of health behaviours: methods, parameter estimates, and their application

Group-randomized trials often involve repeat observations on the same participants. When there are no more than two observations from each participant, standard mixed-model regression methods for a pretest-posttest design can be used. When there are more than two observations from each participant, random coefficients models may be useful. This paper describes the random coefficients analysis appropriate to data from an extended nested cohort design and presents the methods for power analysis and sample size calculations based on that analysis. We provide estimates for the parameters required for those calculations for a number of adolescent health behaviours. We also show how the estimates can be used to plan a future trial.

School-level intraclass correlation for physical activity in sixth grade girls

PURPOSE

The Trial for Activity in Adolescent Girls (TAAG) is a group-randomized trial (GRT) to reduce the usual decline in moderate to vigorous physical activity (MVPA) among middle school girls. We report the school-level intraclass correlation (ICC) for MVPA from the TAAG baseline survey of sixth grade girls and describe the relationship between the schedule of data collection and the ICC.

METHODS

Each of six sites recruited six schools and randomly selected 60 sixth grade girls from each school; 74.2% participated. Girls were grouped in waves defined by the date measurements began and asked to wear an Actigraph accelerometer for 6 d. Occasional missing data were replaced by imputation, and counts above 1500 per 30 s were treated as MVPA, converted into metabolic equivalents (METs), and summed over 6 a.m.-midnight to provide MET-minutes per 18-h day. Mixed-model regression was used to estimate ICC.

RESULTS

The school-level ICC were higher when estimated from a single wave compared with three waves (e.g., 0.057 vs 0.022) and across weekdays compared with weekend days (e.g., 0.024 vs 0.012). Power in a new trial would be greater with some schedules (e.g., 88% given three waves and 6 d) than with others (e.g., 23% given one wave and Tuesday only).

CONCLUSIONS

The schedule of data collection can have a dramatic effect on the ICC for MVPA. In turn, this can have a dramatic effect on the standard error for an intervention effect and on power. Investigators will need to consider the expected magnitude of the ICC and the validity of the MVPA estimates associated with their data collection schedule in planning a new study.

Increasing the degrees of freedom in future group randomized trials: the df* approach

This article builds on the previous article by Blitstein et al. (2005), which showed how external estimates of intraclass correlation can be used to improve the precision for the analysis of an existing group randomized trial. The authors extend that work to sample size estimation and power analysis for future group-randomized trials. Often this approach will allow a smaller study than would otherwise be possible without sacrificing statistical power. Such studies are needed, for example, as pilot studies to help plan for a full-scale efficacy trial, as replication studies, or in situations in which resource constraints prohibit a larger trial. The authors discuss the circumstances under which this strategy will be most helpful and the risks associated with conducting smaller studies.