Design and analysis of group-randomized trials in cancer: a review of current practices

A systematic review of sample size estimation accuracy on power in malaria cluster randomised trials measuring epidemiological outcomes

INTRODUCTION

Cluster randomised trials (CRTs) are the gold standard for measuring the community-wide impacts of malaria control tools. CRTs rely on well-defined sample size estimations to detect statistically significant effects of trialled interventions, however these are often predicted poorly by triallists. Here, we review the accuracy of predicted parameters used in sample size calculations for malaria CRTs with epidemiological outcomes.

METHODS

We searched for published malaria CRTs using four online databases in March 2022. Eligible trials included those with malaria-specific epidemiological outcomes which randomised at least six geographical clusters to study arms. Predicted and observed sample size parameters were extracted by reviewers for each trial. Pair-wise Spearman's correlation coefficients (rs) were calculated to assess the correlation between predicted and observed control-arm outcome measures and effect sizes (relative percentage reductions) between arms. Among trials which retrospectively calculated an estimate of heterogeneity in cluster outcomes, we recalculated study power according to observed trial estimates.

RESULTS

Of the 1889 records identified and screened, 108 articles were eligible and comprised of 71 malaria CRTs. Among 91.5% (65/71) of trials that included sample size calculations, most estimated cluster heterogeneity using the coefficient of variation (k) (80%, 52/65) which were often predicted without using prior data (67.7%, 44/65). Predicted control-arm prevalence moderately correlated with observed control-arm prevalence (rs: 0.44, [95%CI: 0.12,0.68], p-value < 0.05], with 61.2% (19/31) of prevalence estimates overestimated. Among the minority of trials that retrospectively calculated cluster heterogeneity (20%, 13/65), empirical values contrasted with those used in sample size estimations and often compromised study power. Observed effect sizes were often smaller than had been predicted at the sample size stage (72.9%, 51/70) and were typically higher in the first, compared to the second, year of trials. Overall, effect sizes achieved by malaria interventions tested in trials decreased between 1995 and 2021.

CONCLUSIONS

Study findings reveal sample size parameters in malaria CRTs were often inaccurate and resulted in underpowered studies. Future trials must strive to obtain more representative epidemiological sample size inputs to ensure interventions against malaria are adequately evaluated.

REGISTRATION

This review is registered with PROSPERO (CRD42022315741).

Covariate adjustment in randomized clinical trials with missing covariate and outcome data

When analyzing data from randomized clinical trials, covariate adjustment can be used to account for chance imbalance in baseline covariates and to increase precision of the treatment effect estimate. A practical barrier to covariate adjustment is the presence of missing data. In this article, in the light of recent theoretical advancement, we first review several covariate adjustment methods with incomplete covariate data. We investigate the implications of the missing data mechanism on estimating the average treatment effect in randomized clinical trials with continuous or binary outcomes. In parallel, we consider settings where the outcome data are fully observed or are missing at random; in the latter setting, we propose a full weighting approach that combines inverse probability weighting for adjusting missing outcomes and overlap weighting for covariate adjustment. We highlight the importance of including the interaction terms between the missingness indicators and covariates as predictors in the models. We conduct comprehensive simulation studies to examine the finite-sample performance of the proposed methods and compare with a range of common alternatives. We find that conducting the proposed adjustment methods generally improves the precision of treatment effect estimates regardless of the imputation methods when the adjusted covariate is associated with the outcome. We apply the methods to the Childhood Adenotonsillectomy Trial to assess the effect of adenotonsillectomy on neurocognitive functioning scores.

Improving sandwich variance estimation for marginal Cox analysis of cluster randomized trials

Cluster randomized trials (CRTs) frequently recruit a small number of clusters, therefore necessitating the application of small-sample corrections for valid inference. A recent systematic review indicated that CRTs reporting right-censored, time-to-event outcomes are not uncommon and that the marginal Cox proportional hazards model is one of the common approaches used for primary analysis. While small-sample corrections have been studied under marginal models with continuous, binary, and count outcomes, no prior research has been devoted to the development and evaluation of bias-corrected sandwich variance estimators when clustered time-to-event outcomes are analyzed by the marginal Cox model. To improve current practice, we propose nine bias-corrected sandwich variance estimators for the analysis of CRTs using the marginal Cox model and report on a simulation study to evaluate their small-sample properties. Our results indicate that the optimal choice of bias-corrected sandwich variance estimator for CRTs with survival outcomes can depend on the variability of cluster sizes and can also slightly differ whether it is evaluated according to relative bias or type I error rate. Finally, we illustrate the new variance estimators in a real-world CRT where the conclusion about intervention effectiveness differs depending on the use of small-sample bias corrections. The proposed sandwich variance estimators are implemented in an R package CoxBcv.

Design and analysis of cluster randomized trials with time-to-event outcomes under the additive hazards mixed model

A primary focus of current methods for cluster randomized trials (CRTs) has been for continuous, binary, and count outcomes, with relatively less attention given to right-censored, time-to-event outcomes. In this article, we detail considerations for sample size requirement and statistical inference in CRTs with time-to-event outcomes when the intervention effect parameter is specified through the additive hazards mixed model (AHMM), which includes a frailty term to explicitly account for the dependency between the failure times. First, we discuss improved inference for the treatment effect parameter via bias-corrected sandwich variance estimators and randomization-based test under AHMM, addressing potential small-sample biases in CRTs. Next, we derive a new sample size formula for AHMM analysis of CRTs accommodating both equal and unequal cluster sizes. When the cluster sizes vary, our sample size formula depends on the mean and coefficient of variation of cluster sizes, based on which we articulate the impact of cluster size variation in CRTs with time-to-event outcomes. Furthermore, we obtain the insight that the classical variance inflation factor for CRTs with a non-censored outcome can in fact apply to CRTs with a time-to-event outcome, providing that an appropriate definition of the intraclass correlation coefficient is considered under AHMM. Simulation studies are carried out to illustrate key design and analysis considerations in CRTs with a small to moderate number of clusters. The proposed sample size procedure and analytical methods are further illustrated using the context of the STrategies to Reduce Injuries and Develop Confidence in Elders CRT.

Influential methods reports for group-randomized trials and related designs

BACKGROUND

This article identifies the most influential methods reports for group-randomized trials and related designs published through 2020. Many interventions are delivered to participants in real or virtual groups or in groups defined by a shared interventionist so that there is an expectation for positive correlation among observations taken on participants in the same group. These interventions are typically evaluated using a group- or cluster-randomized trial, an individually randomized group treatment trial, or a stepped wedge group- or cluster-randomized trial. These trials face methodological issues beyond those encountered in the more familiar individually randomized controlled trial.

METHODS

PubMed was searched to identify candidate methods reports; that search was supplemented by reports known to the author. Candidate reports were reviewed by the author to include only those focused on the designs of interest. Citation counts and the relative citation ratio, a new bibliometric tool developed at the National Institutes of Health, were used to identify influential reports. The relative citation ratio measures influence at the article level by comparing the citation rate of the reference article to the citation rates of the articles cited by other articles that also cite the reference article.

RESULTS

In total, 1043 reports were identified that were published through 2020. However, 55 were deemed to be the most influential based on their relative citation ratio or their citation count using criteria specific to each of the three designs, with 32 group-randomized trial reports, 7 individually randomized group treatment trial reports, and 16 stepped wedge group-randomized trial reports. Many of the influential reports were early publications that drew attention to the issues that distinguish these designs from the more familiar individually randomized controlled trial. Others were textbooks that covered a wide range of issues for these designs. Others were "first reports" on analytic methods appropriate for a specific type of data (e.g. binary data, ordinal data), for features commonly encountered in these studies (e.g. unequal cluster size, attrition), or for important variations in study design (e.g. repeated measures, cohort versus cross-section). Many presented methods for sample size calculations. Others described how these designs could be applied to a new area (e.g. dissemination and implementation research). Among the reports with the highest relative citation ratios were the CONSORT statements for each design.

CONCLUSIONS

Collectively, the influential reports address topics of great interest to investigators who might consider using one of these designs and need guidance on selecting the most appropriate design for their research question and on the best methods for design, analysis, and sample size.

A comparison of analytical strategies for cluster randomized trials with survival outcomes in the presence of competing risks

While statistical methods for analyzing cluster randomized trials with continuous and binary outcomes have been extensively studied and compared, little comparative evidence has been provided for analyzing cluster randomized trials with survival outcomes in the presence of competing risks. Motivated by the Strategies to Reduce Injuries and Develop Confidence in Elders trial, we carried out a simulation study to compare the operating characteristics of several existing population-averaged survival models, including the marginal Cox, marginal Fine and Gray, and marginal multi-state models. For each model, we found that adjusting for the intraclass correlations through the sandwich variance estimator effectively maintained the type I error rate when the number of clusters is large. With no more than 30 clusters, however, the sandwich variance estimator can exhibit notable negative bias, and a permutation test provides better control of type I error inflation. Under the alternative, the power for each model is differentially affected by two types of intraclass correlations-the within-individual and between-individual correlations. Furthermore, the marginal Fine and Gray model occasionally leads to higher power than the marginal Cox model or the marginal multi-state model, especially when the competing event rate is high. Finally, we provide an illustrative analysis of Strategies to Reduce Injuries and Develop Confidence in Elders trial using each analytical strategy considered.

Strengthening causal inference from randomised controlled trials of complex interventions

Researchers conducting randomised controlled trials (RCTs) of complex interventions face design and analytical challenges that are not fully addressed in existing guidelines. Further guidance is needed to help ensure that these trials of complex interventions are conducted to the highest scientific standards while maximising the evidence that can be extracted from each trial. The key challenge is how to manage the multiplicity of outcomes required for the trial while minimising false positive and false negative findings. To address this challenge, we formulate three principles to conduct RCTs: (1) outcomes chosen should be driven by the intent and programme theory of the intervention and should thus be linked to testable hypotheses; (2) outcomes should be adequately powered and (3) researchers must be explicit and fully transparent about all outcomes and hypotheses before the trial is started and when the results are reported. Multiplicity in trials of complex interventions should be managed through careful planning and interpretation rather than through post hoc analytical adjustment. For trials of complex interventions, the distinction between primary and secondary outcomes as defined in current guidelines does not adequately protect against false positive and negative findings. Primary outcomes should be defined as outcomes that are relevant based on the intervention intent and programme theory, declared (ie, registered), and adequately powered. The possibility of confirmatory causal inference is limited to these outcomes. All other outcomes (either undeclared and/or inadequately powered) are secondary and inference relative to these outcomes will be exploratory.

Clustered restricted mean survival time regression

For multicenter randomized trials or multilevel observational studies, the Cox regression model has long been the primary approach to study the effects of covariates on time-to-event outcomes. A critical assumption of the Cox model is the proportionality of the hazard functions for modeled covariates, violations of which can result in ambiguous interpretations of the hazard ratio estimates. To address this issue, the restricted mean survival time (RMST), defined as the mean survival time up to a fixed time in a target population, has been recommended as a model-free target parameter. In this article, we generalize the RMST regression model to clustered data by directly modeling the RMST as a continuous function of restriction times with covariates while properly accounting for within-cluster correlations to achieve valid inference. The proposed method estimates regression coefficients via weighted generalized estimating equations, coupled with a cluster-robust sandwich variance estimator to achieve asymptotically valid inference with a sufficient number of clusters. In small-sample scenarios where a limited number of clusters are available, however, the proposed sandwich variance estimator can exhibit negative bias in capturing the variability of regression coefficient estimates. To overcome this limitation, we further propose and examine bias-corrected sandwich variance estimators to reduce the negative bias of the cluster-robust sandwich variance estimator. We study the finite-sample operating characteristics of proposed methods through simulations and reanalyze two multicenter randomized trials.

Finite-sample adjustments in variance estimators for clustered competing risks regression

The marginal Fine-Gray proportional subdistribution hazards model is a popular approach to directly study the association between covariates and the cumulative incidence function with clustered competing risks data, which often arise in multicenter randomized trials or multilevel observational studies. To account for the within-cluster correlations between failure times, the uncertainty of the regression parameters estimators is quantified by the robust sandwich variance estimator, which may have unsatisfactory performance with a limited number of clusters. To overcome this limitation, we propose four bias-corrected variance estimators to reduce the negative bias of the usual sandwich variance estimator, extending the bias-correction techniques from generalized estimating equations with noncensored exponential family outcomes to clustered competing risks outcomes. We further compare their finite-sample operating characteristics through simulations and two real data examples. In particular, we found the Mancl and DeRouen (MD) type sandwich variance estimator generally has the smallest bias. Furthermore, with a small number of clusters, the Wald t -confidence interval with the MD sandwich variance estimator carries close to nominal coverage for the cluster-level effect parameter. The t -confidence intervals based on the sandwich variance estimator with any one of the three types of multiplicative bias correction or the z -confidence interval with the Morel, Bokossa and Neerchal (MBN) type sandwich variance estimator have close to nominal coverage for the individual-level effect parameter. Finally, we develop a user-friendly R package crrcbcv implementing the proposed sandwich variance estimators to assist practical applications.

When should matching be used in the design of cluster randomized trials?

For cluster randomized trials (CRTs) with a small number of clusters, the matched-pair (MP) design, where clusters are paired before randomizing one to each trial arm, is often recommended to minimize imbalance on known prognostic factors, add face-validity to the study, and increase efficiency, provided the analysis recognizes the matching. Little evidence exists to guide decisions on when to use matching. We used simulation to compare the efficiency of the MP design with the stratified and simple designs, based on the mean confidence interval width of the estimated intervention effect. Matched and unmatched analyses were used for the MP design; a stratified analysis was used for the stratified design; and analyses without and with post-stratification adjustment for factors that would otherwise have been used for restricted allocation were used for the simple design. Results showed the MP design was generally the most efficient for CRTs with 10 or more pairs when the correlation between cluster-level outcomes within pairs (matching correlation) was moderate to strong (0.3-0.5). There was little gain in efficiency for the MP or stratified designs compared to simple randomization when the matching correlation was weak (0.05-0.1). For trials with four pairs of clusters, the simple and stratified designs were more efficient than the MP design because greater degrees of freedom were available for the analysis, although an unmatched analysis of the MP design recovered precision for weak matching correlations. Practical guidance on choosing between the MP, stratified, and simple designs is provided.

xtgeebcv: A command for bias-corrected sandwich variance estimation for GEE analyses of cluster randomized trials

Cluster randomized trials, where clusters (for example, schools or clinics) are randomized to comparison arms but measurements are taken on individuals, are commonly used to evaluate interventions in public health, education, and the social sciences. Analysis is often conducted on individual-level outcomes, and such analysis methods must consider that outcomes for members of the same cluster tend to be more similar than outcomes for members of other clusters. A popular individual-level analysis technique is generalized estimating equations (GEE). However, it is common to randomize a small number of clusters (for example, 30 or fewer), and in this case, the GEE standard errors obtained from the sandwich variance estimator will be biased, leading to inflated type I errors. Some bias-corrected standard errors have been proposed and studied to account for this finite-sample bias, but none has yet been implemented in Stata. In this article, we describe several popular bias corrections to the robust sandwich variance. We then introduce our newly created command, xtgeebcv, which will allow Stata users to easily apply finite-sample corrections to standard errors obtained from GEE models. We then provide examples to demonstrate the use of xtgeebcv. Finally, we discuss suggestions about which finite-sample corrections to use in which situations and consider areas of future research that may improve xtgeebcv.

Beat osteoporosis - nourish and exercise skeletons (BONES): a group randomized controlled trial in children

BACKGROUND

Lifelong healthy habits developed during childhood may prevent chronic diseases in adulthood. Interventions to promote these habits must begin early. The BONES (Beat Osteoporosis - Nourish and Exercise Skeletons) project assessed whether early elementary school children participating in a multifaceted health behavior change, after-school based intervention would improve bone quality and muscular strength and engage in more bone-strengthening behaviors.

METHODS

The 2-year BONES (B) intervention included bone-strengthening physical activity (85 min/week), educational materials (2 days/week), and daily calcium-rich snacks (380 mg calcium/day) delivered by after-school program leaders. BONES plus Parent (B + P) included an additional parent education component. From 1999 to 2004, n = 83 after-school programs (N = 1434 children aged 6-9 years) in Massachusetts and Rhode Island participated in a group randomized trial with two intervention arms (B only, n = 25 programs; B + P, n = 33) and a control arm (C, n = 25). Outcome measures (primary: bone quality (stiffness index of the calcaneus) and muscular strength (grip strength and vertical jump); secondary: bone-strengthening behaviors (calcium-rich food knowledge, preference, and intake; and physical activity level (metabolic equivalent time (MET) score, and weight-bearing factor (WBF) score)) were recorded at baseline, and after years one and two. Analyses followed an intent-to-treat protocol, and focused on individual subjects' trajectories along the three time points adjusting for baseline age and race via a mixed-effects regression framework. Analyses were performed with and without sex stratification.

RESULTS

Children in B + P increased bone stiffness compared to C (p = 0.05); No significant changes were observed in muscle strength, food knowledge, or vertical jump. Children in B + P showed significant improvement in their MET and WBF scores compared to C (p < 0.01) with a stronger effect in boys in both B and B + P (all p < 0.01).

CONCLUSION

After-school programs, coupled with parental engagement, serving early elementary school children are a potentially feasible platform to deliver bone-strengthening behaviors to prevent osteoporosis in adulthood, with some encouraging bone and physical activity outcomes.

TRIAL REGISTRATION

ClinicalTrials.gov NCT00065247. Retrospectively registered. First posted July 22, 2003.

Impact of baseline covariate imbalance on bias in treatment effect estimation in cluster randomized trials: Race as an example

Individual-level baseline covariate imbalance could happen more frequently in cluster randomized trials, and may influence the observed treatment effect. Using computer and real-data simulations, this paper quantifies the extent and impact of covariate imbalance on the estimated treatment effect for both continuous and binary outcomes, and relates it to the degree of imbalance for different numbers of clusters, cluster sizes, and covariate intraclass correlation coefficients. We focused on the impact of race as a covariate, given the emphasis of regulatory and funding bodies on understanding the influence of demographic characteristics on treatment effectiveness. We found that bias in the treatment effect is proportional to both the degree of baseline covariate imbalance and the covariate effect size. Larger numbers of clusters result in lower covariate imbalance, and increasing cluster size is less effective in reducing imbalance compared to increasing the number of clusters. Models adjusted for important baseline confounders are superior to unadjusted models for minimizing bias in both model-based simulations and an innovative simulation based on real clinical trial data. Higher outcome intraclass correlation coefficients did not affect bias but resulted in greater variance in treatment estimates.

Essential Ingredients and Innovations in the Design and Analysis of Group-Randomized Trials

This article reviews the essential ingredients and innovations in the design and analysis of group-randomized trials. The methods literature for these trials has grown steadily since they were introduced to the biomedical research community in the late 1970s, and we summarize those developments. We review, in addition to the group-randomized trial, methods for two closely related designs, the individually randomized group treatment trial and the stepped-wedge group-randomized trial. After describing the essential ingredients for these designs, we review the most important developments in the evolution of their methods using a new bibliometric tool developed at the National Institutes of Health. We then discuss the questions to be considered when selecting from among these designs or selecting the traditional randomized controlled trial. We close with a review of current methods for the analysis of data from these designs, a case study to illustrate each design, and a brief summary.

The PRogram In Support of Moms (PRISM): study protocol for a cluster randomized controlled trial of two active interventions addressing perinatal depression in obstetric settings

BACKGROUND

Perinatal depression, the most common pregnancy complication, is associated with negative maternal-offspring outcomes. Despite existence of effective treatments, it is under-recognized and under-treated. Professional organizations recommend universal screening, yet multi-level barriers exist to ensuring effective diagnosis, treatment, and follow-up. Integrating mental health and obstetric care holds significant promise for addressing perinatal depression. The overall study goal is to compare the effectiveness of two active interventions: (1) the Massachusetts Child Psychiatry Access Program (MCPAP) for Moms, a state-wide, population-based program, and (2) the PRogram In Support of Moms (PRISM) which includes MCPAP for Moms plus a proactive, multifaceted, practice-level intervention with intensive implementation support.

METHODS

This study is conducted in two phases: (1) a run-in phase which has been completed and involved practice and patient participant recruitment to demonstrate feasibility for the second phase, and (2) a cluster randomized controlled trial (RCT), which is ongoing, and will compare two active interventions 1:1 with ten Ob/Gyn practices as the unit of randomization. In phase 1, rates of depressive symptoms and other demographic and clinical features among patients were examined to inform practice randomization. Patient participants to be recruited in phase 2 will be followed longitudinally until 13 months postpartum; they will have 3-5 total study visits depending on whether their initial recruitment and interview was at 4-24 or 32-40 weeks gestation, or 1-3 months postpartum. Sampling throughout pregnancy and postpartum will ensure participants with different depressive symptom onset times. Differences in depression symptomatology and treatment participation will be compared between patient participants by intervention arm.

DISCUSSION

This manuscript describes the full two-phase study protocol. The study design is innovative because it combines effectiveness with implementation research designs and integrates critical components of participatory action research. Our approach assesses the feasibility, acceptance, efficacy, and sustainability of integrating a stepped-care approach to perinatal depression care into ambulatory obstetric settings; an approach that is flexible and can be tailored and adapted to fit unique workflows of real-world practices.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT02760004, registered prospectively on May 3, 2016.

A readily available improvement over method of moments for intra-cluster correlation estimation in the context of cluster randomized trials and fitting a GEE-type marginal model for binary outcomes

BACKGROUND/AIMS

Cluster randomized trials are popular in health-related research due to the need or desire to randomize clusters of subjects to different trial arms as opposed to randomizing each subject individually. As outcomes from subjects within the same cluster tend to be more alike than outcomes from subjects within other clusters, an exchangeable correlation arises that is measured via the intra-cluster correlation coefficient. Intra-cluster correlation coefficient estimation is especially important due to the increasing awareness of the need to publish such values from studies in order to help guide the design of future cluster randomized trials. Therefore, numerous methods have been proposed to accurately estimate the intra-cluster correlation coefficient, with much attention given to binary outcomes. As marginal models are often of interest, we focus on intra-cluster correlation coefficient estimation in the context of fitting such a model with binary outcomes using generalized estimating equations. Traditionally, intra-cluster correlation coefficient estimation with generalized estimating equations has been based on the method of moments, although such estimators can be negatively biased. Furthermore, alternative estimators that work well, such as the analysis of variance estimator, are not as readily applicable in the context of practical data analyses with generalized estimating equations. Therefore, in this article we assess, in terms of bias, the readily available residual pseudo-likelihood approach to intra-cluster correlation coefficient estimation with the GLIMMIX procedure of SAS (SAS Institute, Cary, NC). Furthermore, we study a possible corresponding approach to confidence interval construction for the intra-cluster correlation coefficient.

METHODS

We utilize a simulation study and application example to assess bias in intra-cluster correlation coefficient estimates obtained from GLIMMIX using residual pseudo-likelihood. This estimator is contrasted with method of moments and analysis of variance estimators which are standards of comparison. The approach to confidence interval construction is assessed by examining coverage probabilities.

RESULTS

Overall, the residual pseudo-likelihood estimator performs very well. It has considerably less bias than moment estimators, which are its competitor for general generalized estimating equation-based analyses, and therefore, it is a major improvement in practice. Furthermore, it works almost as well as analysis of variance estimators when they are applicable. Confidence intervals have near-nominal coverage when the intra-cluster correlation coefficient estimate has negligible bias.

CONCLUSION

Our results show that the residual pseudo-likelihood estimator is a good option for intra-cluster correlation coefficient estimation when conducting a generalized estimating equation-based analysis of binary outcome data arising from cluster randomized trials. The estimator is practical in that it is simply a result from fitting a marginal model with GLIMMIX, and a confidence interval can be easily obtained. An additional advantage is that, unlike most other options for performing generalized estimating equation-based analyses, GLIMMIX provides analysts the option to utilize small-sample adjustments that ensure valid inference.

Design and analysis of group-randomized trials in cancer: A review of current practices

The purpose of this paper is to summarize current practices for the design and analysis of group-randomized trials involving cancer-related risk factors or outcomes and to offer recommendations to improve future trials. We searched for group-randomized trials involving cancer-related risk factors or outcomes that were published or online in peer-reviewed journals in 2011-15. During 2016-17, in Bethesda MD, we reviewed 123 articles from 76 journals to characterize their design and their methods for sample size estimation and data analysis. Only 66 (53.7%) of the articles reported appropriate methods for sample size estimation. Only 63 (51.2%) reported exclusively appropriate methods for analysis. These findings suggest that many investigators do not adequately attend to the methodological challenges inherent in group-randomized trials. These practices can lead to underpowered studies, to an inflated type 1 error rate, and to inferences that mislead readers. Investigators should work with biostatisticians or other methodologists familiar with these issues. Funders and editors should ensure careful methodological review of applications and manuscripts. Reviewers should ensure that studies are properly planned and analyzed. These steps are needed to improve the rigor and reproducibility of group-randomized trials. The Office of Disease Prevention (ODP) at the National Institutes of Health (NIH) has taken several steps to address these issues. ODP offers an online course on the design and analysis of group-randomized trials. ODP is working to increase the number of methodologists who serve on grant review panels. ODP has developed standard language for the Application Guide and the Review Criteria to draw investigators' attention to these issues. Finally, ODP has created a new Research Methods Resources website to help investigators, reviewers, and NIH staff better understand these issues.

cvcrand and cptest: Commands for efficient design and analysis of cluster randomized trials using constrained randomization and permutation tests

Cluster randomized trials (CRTs), where clusters (for example, schools or clinics) are randomized to comparison arms but measurements are taken on individuals, are commonly used to evaluate interventions in public health, education, and the social sciences. Because CRTs typically involve a small number of clusters (for example, fewer than 20), simple randomization frequently leads to baseline imbalance of cluster characteristics across study arms, threatening the internal validity of the trial. In CRTs with a small number of clusters, classic approaches to balancing baseline characteristics-such as matching and stratification-have several drawbacks, especially when the number of baseline characteristics the researcher desires to balance is large (Ivers et al., 2012, Trials 13: 120). An alternative design approach is covariate-constrained randomization, whereby a randomization scheme is randomly selected from a subset of all possible randomization schemes based on the value of a balancing criterion (Raab and Butcher, 2001, Statistics in Medicine 20: 351-365). Subsequently, a clustered permutation test can be used in the analysis, which provides increased power under constrained randomization compared with simple randomization (Li et al., 2016, Statistics in Medicine 35: 1565-1579). In this article, we describe covariate-constrained randomization and the permutation test for the design and analysis of CRTs and provide an example to demonstrate the use of our new commands cvcrand and cptest to implement constrained randomization and the permutation test.

25 Years of Complex Intervention Trials: Reflections on Lived and Scientific Experiences

For the past 25 years, I have led multiple group-randomized trials, each focused on a specific underserved population of youth and each one evaluated health effects of complex interventions designed to prevent high-risk behaviors. I share my reflections on issues of intervention and research design, as well as how research results fostered my evolution toward addressing fundamental social determinants of health and well-being. Reflections related to intervention design emphasize the importance of careful consideration of theory of causes and theory of change, theoretical comprehensiveness versus fundamental determinants of population health, how high to reach, and health in all policies. Flowing from these intervention design issues are reflections on implications for research design, including the importance of matching the unit of intervention to the unit of assignment, the emerging field of public health law research, and consideration of design options and design elements beyond and in combination with random assignment.

Review of Recent Methodological Developments in Group-Randomized Trials: Part 2-Analysis

In 2004, Murray et al. reviewed methodological developments in the design and analysis of group-randomized trials (GRTs). We have updated that review with developments in analysis of the past 13 years, with a companion article to focus on developments in design. We discuss developments in the topics of the earlier review (e.g., methods for parallel-arm GRTs, individually randomized group-treatment trials, and missing data) and in new topics, including methods to account for multiple-level clustering and alternative estimation methods (e.g., augmented generalized estimating equations, targeted maximum likelihood, and quadratic inference functions). In addition, we describe developments in analysis of alternative group designs (including stepped-wedge GRTs, network-randomized trials, and pseudocluster randomized trials), which require clustering to be accounted for in their design and analysis.

Review of Recent Methodological Developments in Group-Randomized Trials: Part 1-Design

In 2004, Murray et al. reviewed methodological developments in the design and analysis of group-randomized trials (GRTs). We have highlighted the developments of the past 13 years in design with a companion article to focus on developments in analysis. As a pair, these articles update the 2004 review. We have discussed developments in the topics of the earlier review (e.g., clustering, matching, and individually randomized group-treatment trials) and in new topics, including constrained randomization and a range of randomized designs that are alternatives to the standard parallel-arm GRT. These include the stepped-wedge GRT, the pseudocluster randomized trial, and the network-randomized GRT, which, like the parallel-arm GRT, require clustering to be accounted for in both their design and analysis.

LA sprouts randomized controlled nutrition, cooking and gardening programme reduces obesity and metabolic risk in Hispanic/Latino youth

BACKGROUND

Many programmes for children that involve gardening and nutrition components exist; however, none include experimental designs allowing more rigorous evaluation of their impact on obesity.

OBJECTIVES

The objective of this study is to explore the effects of a novel 12-week gardening, nutrition and cooking intervention {'LA Sprouts'} on dietary intake, obesity parameters and metabolic disease risk among low-income, primarily Hispanic/Latino youth in Los Angeles..

METHODS

This study used a randomized control trial involving four elementary schools [two randomized to intervention {172, 3rd-5th grade students}; two randomized to control {147, 3rd-5th grade students}]. Classes were taught in 90-min sessions once per week for 12 weeks. Data collected at pre-intervention and post-intervention included dietary intake via food frequency questionnaire, anthropometric measures {body mass index, waist circumference}, body fat, and fasting blood samples.

RESULTS

LA Sprouts participants compared with controls had significantly greater reductions in body mass index z-scores {-0.1 vs. -0.04, respectively; p = 0.01} and waist circumference {-1.2 vs. 0.1 cm; p < 0.001}. Fewer LA Sprouts participants had the metabolic syndrome after the intervention than before, while controls with metabolic syndrome increased. LA Sprouts participants compared with controls increased dietary fiber intake {+3.4% vs. -16.5%; p = 0.04}. All participants decreased vegetable intake, but decreases were less in LA Sprouts than controls {-3.7% vs. -26.1%; p = 0.04}. Change in fruit intake did not differ between LA Sprouts and controls.

CONCLUSIONS

LA Sprouts was effective in reducing obesity and metabolic risk; however, additional larger and longer-term studies are warranted.

Perioperative Visual Loss in Spine Fusion Surgery: Ischemic Optic Neuropathy in the United States from 1998 to 2012 in the Nationwide Inpatient Sample

BACKGROUND

Perioperative ischemic optic neuropathy (ION) causes visual loss in spinal fusion. Previous case-control studies are limited by study size and lack of a random sample. The purpose of this study was to study trends in ION incidence in spinal fusion and risk factors in a large nationwide administrative hospital database.

METHODS

In the Nationwide Inpatient Sample for 1998 to 2012, procedure codes for posterior thoracic, lumbar, or sacral spine fusion and diagnostic codes for ION were identified. ION was studied over five 3-yr periods (1998 to 2000, 2001 to 2003, 2004 to 2006, 2007 to 2009, and 2010 to 2012). National estimates were obtained using trend weights in a statistical survey procedure. Univariate and Poisson logistic regression assessed trends and risk factors.

RESULTS

The nationally estimated volume of thoracic, lumbar, and sacral spinal fusion from 1998 to 2012 was 2,511,073. ION was estimated to develop in 257 patients (1.02/10,000). The incidence rate ratio (IRR) for ION significantly decreased between 1998 and 2012 (IRR, 0.72 per 3 yr; 95% CI, 0.58 to 0.88; P = 0.002). There was no significant change in the incidence of retinal artery occlusion. Factors significantly associated with ION were age (IRR, 1.24 per 10 yr of age; 95% CI, 1.05 to 1.45; P = 0.009), transfusion (IRR, 2.72; 95% CI, 1.38 to 5.37; P = 0.004), and obesity (IRR, 2.49; 95% CI, 1.09 to 5.66; P = 0.030). Female sex was protective (IRR, 0.30; 95% CI, 0.16 to 0.56; P = 0.0002).

CONCLUSIONS

Perioperative ION in spinal fusion significantly decreased from 1998 to 2012 by about 2.7-fold. Aging, male sex, transfusion, and obesity significantly increased the risk.

An effectiveness-implementation hybrid trial study protocol targeting posttraumatic stress disorder and comorbidity

BACKGROUND

Each year in the USA, 1.5-2.5 million Americans are so severely injured that they require inpatient hospitalization. Multiple conditions including posttraumatic stress disorder (PTSD), alcohol and drug use problems, depression, and chronic medical conditions are endemic among physical trauma survivors with and without traumatic brain injuries.

METHODS/DESIGN

The trauma survivors outcomes and support (TSOS) effectiveness-implementation hybrid trial is designed to test the delivery of high-quality screening and intervention for PTSD and comorbidities across 24 US level I trauma center sites. The pragmatic trial aims to recruit 960 patients. The TSOS investigation employs a stepped wedge cluster randomized design in which sites are randomized sequentially to initiate the intervention. Patients identified by a 10-domain electronic health record screen as high risk for PTSD are formally assessed with the PTSD Checklist for study entry. Patients randomized to the intervention condition will receive stepped collaborative care, while patients randomized to the control condition will receive enhanced usual care. The intervention training begins with a 1-day on-site workshop in the collaborative care intervention core elements that include care management, medication, cognitive behavioral therapy, and motivational-interviewing elements targeting PTSD and comorbidity. The training is followed by site supervision from the study team. The investigation aims to determine if intervention patients demonstrate significant reductions in PTSD and depressive symptoms, suicidal ideation, alcohol consumption, and improvements in physical function when compared to control patients. The study uses implementation science conceptual frameworks to evaluate the uptake of the intervention model. At the completion of the pragmatic trial, results will be presented at an American College of Surgeons' policy summit. Twenty-four representative US level I trauma centers have been selected for the study, and the protocol is being rolled out nationally.

DISCUSSION

The TSOS pragmatic trial simultaneously aims to establish the effectiveness of the collaborative care intervention targeting PTSD and comorbidity while also addressing sustainable implementation through American College of Surgeons' regulatory policy. The TSOS effectiveness-implementation hybrid design highlights the importance of partnerships with professional societies that can provide regulatory mandates targeting enhanced health care system sustainability of pragmatic trial results.

TRIAL REGISTRATION

ClinicalTrials.gov NCT02655354 . Registered 27 July 2015.

Generalized estimating equations in cluster randomized trials with a small number of clusters: Review of practice and simulation study

Background/aims:

Generalized estimating equations are a common modeling approach used in cluster randomized trials to account for within-cluster correlation. It is well known that the sandwich variance estimator is biased when the number of clusters is small (≤40), resulting in an inflated type I error rate. Various bias correction methods have been proposed in the statistical literature, but how adequately they are utilized in current practice for cluster randomized trials is not clear. The aim of this study is to evaluate the use of generalized estimating equation bias correction methods in recently published cluster randomized trials and demonstrate the necessity of such methods when the number of clusters is small.

Methods:

Review of cluster randomized trials published between August 2013 and July 2014 and using generalized estimating equations for their primary analyses. Two independent reviewers collected data from each study using a standardized, pre-piloted data extraction template. A two-arm cluster randomized trial was simulated under various scenarios to show the potential effect of a small number of clusters on type I error rate when estimating the treatment effect. The nominal level was set at 0.05 for the simulation study.

Results:

Of the 51 included trials, 28 (54.9%) analyzed 40 or fewer clusters with a minimum of four total clusters. Of these 28 trials, only one trial used a bias correction method for generalized estimating equations. The simulation study showed that with four clusters, the type I error rate ranged between 0.43 and 0.47. Even though type I error rate moved closer to the nominal level as the number of clusters increases, it still ranged between 0.06 and 0.07 with 40 clusters.

Conclusions:

Our results showed that statistical issues arising from small number of clusters in generalized estimating equations is currently inadequately handled in cluster randomized trials. Potential for type I error inflation could be very high when the sandwich estimator is used without bias correction.

Culturally adaptive storytelling method to improve hypertension control in Vietnam - "We talk about our hypertension": study protocol for a feasibility cluster-randomized controlled trial

Background

Vietnam is experiencing an epidemiologic transition with an increased prevalence of non-communicable diseases. At present, the major risk factors for cardiovascular disease (CVD) are either on the rise or at alarming levels in Vietnam; inasmuch, the burden of CVD will continue to increase in this country unless effective prevention and control measures are put in place. A national survey in 2008 found that the prevalence of hypertension (HTN) was approximately 25 % among Vietnamese adults and it increased with advancing age. Therefore, novel, large-scale, and sustainable interventions for public health education to promote engagement in the process of detecting and treating HTN in Vietnam are urgently needed.

Methods

A feasibility randomized trial will be conducted in Hung Yen province, Vietnam to evaluate the feasibility and acceptability of a novel community-based intervention using the “storytelling” method to enhance the control of HTN in adults residing in four rural communities. The intervention will center on stories about living with HTN, with patients speaking in their own words. The stories will be obtained from particularly eloquent patients, or “video stars,” identified during Story Development Groups. The study will involve two phases: (i) developing a HTN intervention using the storytelling method, which is designed to empower patients to facilitate changes in their lifestyle practices, and (ii) conducting a feasibility cluster-randomized trial to investigate the feasibility, acceptability, and potential efficacy of the intervention compared with usual care in HTN control among rural residents. The trial will be conducted at four communes, and within each commune, 25 individuals 50 years or older with HTN will be enrolled in the trial resulting in a total sample size of 100 patients.

Discussion

This feasibility trial will provide the necessary groundwork for a subsequent large-scale, fully powered, cluster-randomized controlled trial to test the efficacy of our novel community-based intervention. Results from the full-scale trial will provide health policy makers with practical evidence on how to combat a key risk factor for CVD using a feasible, sustainable, and cost-effective intervention that could be used as a national program for controlling HTN in Vietnam and other developing countries.

Trial registration

ClinicalTrials.gov. Registration number: https://clinicaltrials.gov/ct2/show/NCT02483780(registration date June 22, 2015).

Recommendations for choosing an analysis method that controls Type I error for unbalanced cluster sample designs with Gaussian outcomes

We used theoretical and simulation-based approaches to study Type I error rates for one-stage and two-stage analytic methods for cluster-randomized designs. The one-stage approach uses the observed data as outcomes and accounts for within-cluster correlation using a general linear mixed model. The two-stage model uses the cluster specific means as the outcomes in a general linear univariate model. We demonstrate analytically that both one-stage and two-stage models achieve exact Type I error rates when cluster sizes are equal. With unbalanced data, an exact size α test does not exist, and Type I error inflation may occur. Via simulation, we compare the Type I error rates for four one-stage and six two-stage hypothesis testing approaches for unbalanced data. With unbalanced data, the two-stage model, weighted by the inverse of the estimated theoretical variance of the cluster means, and with variance constrained to be positive, provided the best Type I error control for studies having at least six clusters per arm. The one-stage model with Kenward-Roger degrees of freedom and unconstrained variance performed well for studies having at least 14 clusters per arm. The popular analytic method of using a one-stage model with denominator degrees of freedom appropriate for balanced data performed poorly for small sample sizes and low intracluster correlation. Because small sample sizes and low intracluster correlation are common features of cluster-randomized trials, the Kenward-Roger method is the preferred one-stage approach.

Impact of a web-based treatment decision aid for early-stage prostate cancer on shared decision-making and health outcomes: study protocol for a randomized controlled trial

Background

At an early stage, prostate cancer patients are often eligible for more than one treatment option, or may choose to defer curative treatment. Without a pre-existing superior option, a patient has to weigh his personal preferences against the risks and benefits of each alternative to select the most appropriate treatment. Given this context, in prostate cancer treatment decision-making, it is particularly suitable to follow the principles of shared decision-making (SDM), especially with the support of specific instruments like decision aids (DAs). Although several alternatives are available, present tools are not sufficiently compatible with routine clinical practice. To overcome existing barriers and to stimulate structural implementation of DAs and SDM in clinical practice, a web-based prostate cancer treatment DA was developed to fit clinical workflow. Following the structure of an existing DA, Dutch content was developed, and values clarification methods (VCMs) were added. The aim of this study is to investigate the effect of this DA on (shared) treatment choice and patient-reported outcomes.

Methods/design

Nineteen Dutch hospitals are included in a pragmatic, cluster randomized controlled trial, with an intervention and a control arm. In the intervention group, the DA will be offered after diagnosis, and a summary of the patients’ preferences, which were identified with the DA, can be discussed by the patient and his clinician during later consultation. Patients in the control group will receive information and decisional support as usual. Results from both groups on decisional conflict, treatment choice and the experience with involvement in the decision-making process are compared. Patients are requested to fill in questionnaires after treatment decision-making but before treatment is started, and 6 and 12 months later. This will allow the development of treatment satisfaction, decisional regret, and quality of life to be monitored. Clinicians from both groups will evaluate their practice of information provision and decisional support.

Discussion

This study will describe a web-based prostate cancer treatment DA with VCMs. The effect of this DA on the decision-making process and subsequent patient reported outcomes will be evaluated.

Trial registration

The Netherlands National Trial Register: NTR4554, registration date 1 May 2014.

Analytic methods for individually randomized group treatment trials and group-randomized trials when subjects belong to multiple groups

Participants in trials may be randomized either individually or in groups and may receive their treatment either entirely individually, entirely in groups, or partially individually and partially in groups. This paper concerns cases in which participants receive their treatment either entirely or partially in groups, regardless of how they were randomized. Participants in group-randomized trials are randomized in groups, and participants in individually randomized group treatment trials are individually randomized, but participants in both types of trials receive part or all of their treatment in groups or through common change agents. Participants who receive part or all of their treatment in a group are expected to have positively correlated outcome measurements. This paper addresses a situation that occurs in group-randomized trials and individually randomized group treatment trials-participants receive treatment through more than one group. As motivation, we consider trials in The Childhood Obesity Prevention and Treatment Research Consortium, in which each child participant receives treatment in at least two groups. In simulation studies, we considered several possible analytic approaches over a variety of possible group structures. A mixed model with random effects for both groups provided the only consistent protection against inflated type I error rates and did so at the cost of only moderate loss of power when intraclass correlations were not large. We recommend constraining variance estimates to be positive and using the Kenward-Roger adjustment for degrees of freedom; this combination provided additional power but maintained type I error rates at the nominal level.

A systematic review of cluster randomised trials in residential facilities for older people suggests how to improve quality

BACKGROUND

Previous reviews of cluster randomised trials have been critical of the quality of the trials reviewed, but none has explored determinants of the quality of these trials in a specific field over an extended period of time. Recent work suggests that correct conduct and reporting of these trials may require more than published guidelines. In this review, our aim was to assess the quality of cluster randomised trials conducted in residential facilities for older people, and to determine whether (1) statistician involvement in the trial and (2) strength of journal endorsement of the Consolidated Standards of Reporting Trials (CONSORT) statement influence quality.

METHODS

We systematically identified trials randomising residential facilities for older people, or parts thereof, without language restrictions, up to the end of 2010, using National Library of Medicine (Medline) via PubMed and hand-searching. We based quality assessment criteria largely on the extended CONSORT statement for cluster randomised trials. We assessed statistician involvement based on statistician co-authorship, and strength of journal endorsement of the CONSORT statement from journal websites.

RESULTS

73 trials met our inclusion criteria. Of these, 20 (27%) reported accounting for clustering in sample size calculations and 54 (74%) in the analyses. In 29 trials (40%), methods used to identify/recruit participants were judged by us to have potentially caused bias or reporting was unclear to reach a conclusion. Some elements of quality improved over time but this appeared not to be related to the publication of the extended CONSORT statement for these trials. Trials with statistician/epidemiologist co-authors were more likely to account for clustering in sample size calculations (unadjusted odds ratio 5.4, 95% confidence interval 1.1 to 26.0) and analyses (unadjusted OR 3.2, 1.2 to 8.5). Journal endorsement of the CONSORT statement was not associated with trial quality.

CONCLUSIONS

Despite international attempts to improve methods in cluster randomised trials, important quality limitations remain amongst these trials in residential facilities. Statistician involvement on trial teams may be more effective in promoting quality than further journal endorsement of the CONSORT statement. Funding bodies and journals should promote statistician involvement and co-authorship in addition to adherence to CONSORT guidelines.

Measurement in comparative effectiveness research

Comparative effectiveness research (CER) on preventive services can shape policy and help patients, their providers, and public health practitioners select regimens and programs for disease prevention. Patients and providers need information about the relative effectiveness of various regimens they may choose. Decision makers need information about the relative effectiveness of various programs to offer or recommend. The goal of this paper is to define and differentiate measures of relative effectiveness of regimens and programs for disease prevention. Cancer screening is used to demonstrate how these measures differ in an example of two hypothetical screening regimens and programs. Conceptually and algebraically defined measures of relative regimen and program effectiveness also are presented. The measures evaluate preventive services that range from individual tests through organized, population-wide prevention programs. Examples illustrate how effective screening regimens may not result in effective screening programs and how measures can vary across subgroups and settings. Both regimen and program relative effectiveness measures assess benefits of prevention services in real-world settings, but each addresses different scientific and policy questions. As the body of CER grows, a common lexicon for various measures of relative effectiveness becomes increasingly important to facilitate communication and shared understanding among researchers, healthcare providers, patients, and policymakers.

SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials

High quality protocols facilitate proper conduct, reporting, and external review of clinical trials. However, the completeness of trial protocols is often inadequate. To help improve the content and quality of protocols, an international group of stakeholders developed the SPIRIT 2013 Statement (Standard Protocol Items: Recommendations for Interventional Trials). The SPIRIT Statement provides guidance in the form of a checklist of recommended items to include in a clinical trial protocol. This SPIRIT 2013 Explanation and Elaboration paper provides important information to promote full understanding of the checklist recommendations. For each checklist item, we provide a rationale and detailed description; a model example from an actual protocol; and relevant references supporting its importance. We strongly recommend that this explanatory paper be used in conjunction with the SPIRIT Statement. A website of resources is also available (www.spirit-statement.org). The SPIRIT 2013 Explanation and Elaboration paper, together with the Statement, should help with the drafting of trial protocols. Complete documentation of key trial elements can facilitate transparency and protocol review for the benefit of all stakeholders.

Person mobility in the design and analysis of cluster-randomized cohort prevention trials

Person mobility is an inescapable fact of life for most cluster-randomized (e.g., schools, hospitals, clinic, cities, state) cohort prevention trials. Mobility rates are an important substantive consideration in estimating the effects of an intervention. In cluster-randomized trials, mobility rates are often correlated with ethnicity, poverty and other variables associated with disparity. This raises the possibility that estimated intervention effects may generalize to only the least mobile segments of a population and, thus, create a threat to external validity. Such mobility can also create threats to the internal validity of conclusions from randomized trials. Researchers must decide how to deal with persons who leave study clusters during a trial (dropouts), persons and clusters that do not comply with an assigned intervention, and persons who enter clusters during a trial (late entrants), in addition to the persons who remain for the duration of a trial (stayers). Statistical techniques alone cannot solve the key issues of internal and external validity raised by the phenomenon of person mobility. This commentary presents a systematic, Campbellian-type analysis of person mobility in cluster-randomized cohort prevention trials. It describes four approaches for dealing with dropouts, late entrants and stayers with respect to data collection, analysis and generalizability. The questions at issue are: 1) From whom should data be collected at each wave of data collection? 2) Which cases should be included in the analyses of an intervention effect? and 3) To what populations can trial results be generalized? The conclusions lead to recommendations for the design and analysis of future cluster-randomized cohort prevention trials.

When research meets reality-lessons learned from a pragmatic multisite group-randomized clinical trial on psychosocial interventions in the psychiatric and addiction field

Research on treatments for patients with co-occurring psychiatric and substance use disorders is of core importance and at the same time highly challenging as it includes patients that are normally excluded from clinical studies. Such research may require methodological adaptations which in turn create new challenges. However, the challenges that arise in such studies are insufficiently discussed in the literature. The aim of this methodology paper is, firstly, to discuss the methodological adaptations that may be required in such research; secondly, to describe how such adaptations created new challenges in a group-randomized clinical trial on Integrated Treatment amongst patients with co-occurring psychiatric and substance use disorders. We also discuss how these challenges might be understood and highlight lessons for future research in this field.

TRIAL REGISTRATION

NCT00447733.

Allocation techniques for balance at baseline in cluster randomized trials: a methodological review

Reviews have repeatedly noted important methodological issues in the conduct and reporting of cluster randomized controlled trials (C-RCTs). These reviews usually focus on whether the intracluster correlation was explicitly considered in the design and analysis of the C-RCT. However, another important aspect requiring special attention in C-RCTs is the risk for imbalance of covariates at baseline. Imbalance of important covariates at baseline decreases statistical power and precision of the results. Imbalance also reduces face validity and credibility of the trial results. The risk of imbalance is elevated in C-RCTs compared to trials randomizing individuals because of the difficulties in recruiting clusters and the nested nature of correlated patient-level data. A variety of restricted randomization methods have been proposed as way to minimize risk of imbalance. However, there is little guidance regarding how to best restrict randomization for any given C-RCT. The advantages and limitations of different allocation techniques, including stratification, matching, minimization, and covariate-constrained randomization are reviewed as they pertain to C-RCTs to provide investigators with guidance for choosing the best allocation technique for their trial.

Multilevel interventions: study design and analysis issues

Multilevel interventions, implemented at the individual, physician, clinic, health-care organization, and/or community level, increasingly are proposed and used in the belief that they will lead to more substantial and sustained changes in behaviors related to cancer prevention, detection, and treatment than would single-level interventions. It is important to understand how intervention components are related to patient outcomes and identify barriers to implementation. Designs that permit such assessments are uncommon, however. Thus, an important way of expanding our knowledge about multilevel interventions would be to assess the impact of interventions at different levels on patients as well as the independent and synergistic effects of influences from different levels. It also would be useful to assess the impact of interventions on outcomes at different levels. Multilevel interventions are much more expensive and complicated to implement and evaluate than are single-level interventions. Given how little evidence there is about the value of multilevel interventions, however, it is incumbent upon those arguing for this approach to do multilevel research that explicates the contributions that interventions at different levels make to the desired outcomes. Only then will we know whether multilevel interventions are better than more focused interventions and gain greater insights into the kinds of interventions that can be implemented effectively and efficiently to improve health and health care for individuals with cancer. This chapter reviews designs for assessing multilevel interventions and analytic ways of controlling for potentially confounding variables that can account for the complex structure of multilevel data.

Scientific rigour in psycho-oncology trials: why and how to avoid common statistical errors

OBJECTIVE

It is well documented that statistical and methodological flaws are common in much of the health research literature, including psycho-oncology. These can have far-reaching effects, including the publishing of misleading results; the wasting of time, effort, and financial resources; exposure of patients to the potential harms of research and decreased confidence in science and researchers by the public.

METHODS

Several of the most common statistical errors and methodological pitfalls that occur in the field of psycho-oncology are discussed, including those that occur at the design, analysis, reporting and conclusion stages.

RESULTS

Fourteen topics are briefly discussed, explaining why there is a problem and how to avoid it. These include proper approaches to power, clustering, missing data, categorization of continuous variables, subgroup analyses, multiple comparisons, statistical interactions, confidence intervals and correct interpretation of p-values. Extensive referencing points the reader to more in-depth explanations.

CONCLUSIONS

To increase the scientific rigour in psycho-oncology, researchers should involve a biostatistician from the beginning of the study and should commit to continuing education on best practices in the fields of statistics and reporting.

Quality of cluster randomized controlled trials in oral health: a systematic review of reports published between 2005 and 2009

OBJECTIVES

To assess the quality of methods and reporting of recently published cluster randomized trials (CRTs) in oral health.

METHODS

We searched PubMed for CRTs that included at least one oral health-related outcome and were published from 2005 to 2009 inclusive. We developed a list of criteria for assessing trial quality and reporting. This was influenced largely by the extended CONSORT statement for CRTs but also included criteria suggested by other authors. We examined the extent to which trials were consistent with these criteria.

RESULTS

Twenty-three trials were included in the review. In 15 (65%) trials, clustering had been accounted for in sample size calculations, and in 18 (78%) authors had accounted for clustering in analysis. Intraclass correlation coefficients (ICCs) were reported for eight (35%) trials; the outcome assessor was reported as having been blinded to allocation in 12 (52%) trials; 17 (74%) described eligibility criteria at individual level, but only nine (39%) described such criteria at cluster level. Sixteen of 20 trials (80%), in which individuals were recruited, reported that individual informed consent was obtained.

CONCLUSIONS

These results suggest that the quality of recent CRTs in oral health is relatively high and appears to compare favourably with other fields. However, there remains room for improvement. Authors of future trials should endeavour to ensure sample size calculations and analyses properly account for clustering (and are reported as such), consider the potential for recruitment/identification bias at the design stage, describe the steps taken to avoid this in the final report and report observed ICCs and cluster-level eligibility criteria.

Use of design effects and sample weights in complex health survey data: a review of published articles using data from 3 commonly used adolescent health surveys

OBJECTIVES

We assessed how frequently researchers reported the use of statistical techniques that take into account the complex sampling structure of survey data and sample weights in published peer-reviewed articles using data from 3 commonly used adolescent health surveys.

METHODS

We performed a systematic review of 1003 published empirical research articles from 1995 to 2010 that used data from the National Longitudinal Study of Adolescent Health (n=765), Monitoring the Future (n=146), or Youth Risk Behavior Surveillance System (n=92) indexed in ERIC, PsycINFO, PubMed, and Web of Science.

RESULTS

Across the data sources, 60% of articles reported accounting for design effects and 61% reported using sample weights. However, the frequency and clarity of reporting varied across databases, publication year, author affiliation with the data, and journal.

CONCLUSIONS

Given the statistical bias that occurs when design effects of complex data are not incorporated or sample weights are omitted, this study calls for improvement in the dissemination of research findings based on complex sample data. Authors, editors, and reviewers need to work together to improve the transparency of published findings using complex sample data.

Effects of physical activity on teen smoking cessation

OBJECTIVE

To understand the influence of physical activity on teen smoking-cessation outcomes.

METHODS

Teens (N = 233; 14-19 years of age) from West Virginia high schools who smoked >1 cigarette in the previous 30 days were included. High schools with >300 students were selected randomly and assigned to brief intervention (BI), Not on Tobacco (N-O-T) (a proven teen cessation program), or N-O-T plus a physical activity module (N-O-T+FIT). Quit rates were determined 3 and 6 months after baseline by using self-classified and 7-day point prevalence quit rates, and carbon monoxide validation was obtained at the 3-month follow-up evaluation.

RESULTS

Trends for observed and imputed self-classified and 7-day point prevalence rates indicated that teens in the N-O-T+FIT group had significantly higher cessation rates compared with those in the N-O-T and BI groups. Effect sizes were large. Overall, girls quit more successfully with N-O-T compared with BI (relative risk [RR]: >∞) 3 months after baseline, and boys responded better to N-O-T+FIT than to BI (RR: 2-3) or to N-O-T (RR: 1-2). Youths in the N-O-T+FIT group, compared with those in the N-O-T group, had greater likelihood of cessation (RR: 1.48) at 6 months. The control group included an unusually large proportion of participants in the precontemplation stage at enrollment, but there were no significant differences in outcomes between BI and N-O-T (z = 0.94; P = .17) or N-O-T+FIT (z = 1.12; P = .13) participants in the precontemplation stage.

CONCLUSIONS

Adding physical activity to N-O-T may enhance cessation success, particularly among boys.

Impact of CONSORT extension for cluster randomised trials on quality of reporting and study methodology: review of random sample of 300 trials, 2000-8

OBJECTIVE

To assess the impact of the 2004 extension of the CONSORT guidelines on the reporting and methodological quality of cluster randomised trials.

DESIGN

Methodological review of 300 randomly sampled cluster randomised trials. Two reviewers independently abstracted 14 criteria related to quality of reporting and four methodological criteria specific to cluster randomised trials. We compared manuscripts published before CONSORT (2000-4) with those published after CONSORT (2005-8). We also investigated differences by journal impact factor, type of journal, and trial setting.

DATA SOURCES

A validated Medline search strategy. Eligibility criteria for selecting studies Cluster randomised trials published in English language journals, 2000-8.

RESULTS

There were significant improvements in five of 14 reporting criteria: identification as cluster randomised; justification for cluster randomisation; reporting whether outcome assessments were blind; reporting the number of clusters randomised; and reporting the number of clusters lost to follow-up. No significant improvements were found in adherence to methodological criteria. Trials conducted in clinical rather than non-clinical settings and studies published in medical journals with higher impact factor or general medical journals were more likely to adhere to recommended reporting and methodological criteria overall, but there was no evidence that improvements after publication of the CONSORT extension for cluster trials were more likely in trials conducted in clinical settings nor in trials published in either general medical journals or in higher impact factor journals.

CONCLUSION

The quality of reporting of cluster randomised trials improved in only a few aspects since the publication of the extension of CONSORT for cluster randomised trials, and no improvements at all were observed in essential methodological features. Overall, the adherence to reporting and methodological guidelines for cluster randomised trials remains suboptimal, and further efforts are needed to improve both reporting and methodology.

The Minnesota Adolescent Community Cohort Study: design and baseline results

The Minnesota Adolescent Community Cohort (MACC) Study is a population-based, longitudinal study that enrolled 3,636 youth from Minnesota and 605 youth from comparison states ages 12 to 16 years in 2000-2001. Participants have been surveyed by telephone semi-annually about their tobacco-related attitudes and behaviors. The goals of the study are to evaluate the effects of the Minnesota Youth Tobacco Prevention Initiative and its shutdown on youth smoking patterns, and to better define the patterns of development of tobacco use in adolescents. A multilevel sample was constructed representing individuals, local jurisdictions and the entire state, and data are collected to characterize each of these levels. This paper presents the details of the multilevel study design. We also provide baseline information about MACC participants including demographics and tobacco-related attitudes and behaviors. This paper describes variability in smoking prevalence and demographic characteristics for local units, and compares MACC participants to the state as a whole.

Intraclass correlation associated with therapists: estimates and applications in planning psychotherapy research

It is essential that outcome research permit clear conclusions to be drawn about the efficacy of interventions. The common practice of nesting therapists within conditions can pose important methodological challenges that affect interpretation, particularly if the study is not powered to account for the nested design. An obstacle to the optimal design of these studies is the lack of data about the intraclass correlation coefficient (ICC), which measures the statistical dependencies introduced by nesting. To begin the development of a public database of ICC estimates, the authors investigated ICCs for a variety outcomes reported in 20 psychotherapy outcome studies. The magnitude of the 495 ICC estimates varied widely across measures and studies. The authors provide recommendations regarding how to select and aggregate ICC estimates for power calculations and show how researchers can use ICC estimates to choose the number of patients and therapists that will optimize power. Attention to these recommendations will strengthen the validity of inferences drawn from psychotherapy studies that nest therapists within conditions.

Ignoring the group in group-level HIV/AIDS intervention trials: a review of reported design and analytic methods

OBJECTIVES

Studies evaluating the efficacy of HIV/AIDS interventions often involve the random assignment of groups of participants or the treatment of participants in groups. These studies require analytic methods that take within-group correlation into account. We reviewed published studies to determine the extent to which within-group correlation was dealt with properly.

DESIGN

We reviewed group-randomized trials (GRTs) and individually randomized group treatment (IRGT) trials published in HIV/AIDS and general public health journals 2005-2009.

METHODS

At least two of the authors reviewed each article, recording descriptive characteristics, sample size estimation methods, analytic methods, and judgments about whether the methods took intraclass correlation into account properly.

RESULTS

Of those articles including sufficient information to judge whether analytic methods were correct, only 24% used only appropriate methods for dealing with the intraclass correlation. The percentages differed substantially for GRTs (41.7%) and IRGT trials (8.0%). Most of the articles (69.2%) also made no mention of a priori sample size estimation.

CONCLUSION

A majority of the articles in our review reported analyses ignoring the intraclass correlation. This practice may result in underestimated variance, inappropriately small P values, and incorrect conclusions about the effectiveness of interventions. Previous trials that were analyzed incorrectly need to be re-analyzed, and future trials should be designed and analyzed with appropriate methods. Also, journal reviewers and editors need to be aware of the special requirements for design and analysis of GRTs and IRGT trials and judge the quality of articles reporting on such trials according to appropriate standards.

Cluster randomized trials of cancer screening interventions: are appropriate statistical methods being used?

The design and analysis of cluster randomized trials can require more sophistication than individually randomized trials. However, the need for statistical methods that account for the clustered design has not always been appreciated, and past reviews have found widespread deficiencies in methodology and reporting. We reviewed cluster randomized trials of cancer screening interventions published in 1995-2010 to determine whether the use of appropriate statistical methods had increased over time. Literature searches yielded 50 articles reporting outcome analyses of cluster randomized trials of breast, cervix and colorectal cancer screening interventions. Of studies published in 1995-1999, 2000-2002, 2003-2006 and 2007-2010, 55% (6/11), 82% (9/11), 92% (12/13) and 60% (9/15) used appropriate analytic methods, respectively. Results were suggestive of a peak in 2003-2006 (p =.06) followed by a decline in 2007-2010 (p =.08). While the sample of studies was small, these results indicate that many cluster randomized trials of cancer screening interventions have had deficiencies in the application of correct statistical procedures for the outcome analysis, and that increased adoption of appropriate methods in the early and mid-2000's may not have been sustained.

Interventions to promote repeat breast cancer screening with mammography: a systematic review and meta-analysis

BACKGROUND

Various interventions to promote repeat use of mammography have been evaluated, but the efficacy of such interventions is not well understood.

METHODS

We searched electronic databases through August 15, 2009, and extracted data to calculate unadjusted effect estimates (odds ratios [ORs] and 95% confidence intervals [CIs]). Eligible studies were those that reported estimates of repeat screening for intervention and control groups. We tested homogeneity and computed summary odds ratios. To explore possible causes of heterogeneity, we performed stratified analyses, examined meta-regression models for 15 a priori explanatory variables, and conducted influence analyses. We used funnel plots and asymmetry tests to assess publication bias. Statistical tests were two-sided.

RESULTS

The 25 eligible studies (27 effect estimates) were statistically significantly heterogeneous (Q = 69.5, I(2) = 63%, P < .001). Although there were homogeneous subgroups in some categories of the 15 explanatory variables, heterogeneity persisted after stratification. For all but one explanatory variable, subgroup summary odds ratios were similar with overlapping confidence intervals. The summary odds ratio for the eight heterogeneous reminder-only studies was the largest observed (OR = 1.79, 95% CI = 1.41 to 2.29) and was statistically significantly greater than the summary odds ratio (P(diff) = .008) for the homogeneous group of 17 studies that used the more intensive strategies of education/motivation or counseling (OR = 1.27, 95% CI = 1.17 to 1.37). However, reminder-only studies remained statistically significantly heterogeneous, whereas the studies classified as education/motivation or counseling were homogeneous. Similarly, in meta-regression modeling, the only statistically significant predictor of the intervention effect size was intervention strategy (reminder-only vs the other two combined as the referent). Publication bias was not apparent.

CONCLUSIONS

The observed heterogeneity precludes a summary effect estimate. We also cannot conclude that reminder-only intervention strategies are more effective than alternate strategies. Additional studies are needed to identify methods or strategies that could increase repeat mammography.

Analysis of group randomized trials with multiple binary endpoints and small number of groups

The group randomized trial (GRT) is a common study design to assess the effect of an intervention program aimed at health promotion or disease prevention. In GRTs, groups rather than individuals are randomized into intervention or control arms. Then, responses are measured on individuals within those groups. A number of analytical problems beset GRT designs. The major problem emerges from the likely positive intraclass correlation among observations of individuals within a group. This paper provides an overview of the analytical method for GRT data and applies this method to a randomized cancer prevention trial, where multiple binary primary endpoints were obtained. We develop an index of extra variability to investigate group-specific effects on response. The purpose of the index is to understand the influence of individual groups on evaluating the intervention effect, especially, when a GRT study involves a small number of groups. The multiple endpoints from the GRT design are analyzed using a generalized linear mixed model and the stepdown Bonferroni method of Holm.