Sample size determination for three-level randomized clinical trials with randomization at the first or second level

Cluster Randomized Trials with a Pretest and Posttest: Equivalence of Three-, Two- and One-Level Analyses, and Sample Size Calculation

In a cluster randomized trial clusters of persons, for instance, schools or health centers, are assigned to treatments, and all persons in the same cluster get the same treatment. Although less powerful than individual randomization, cluster randomization is a good alternative if individual randomization is impossible or leads to severe treatment contamination (carry-over). Focusing on cluster randomized trials with a pretest and post-test of a quantitative outcome, this paper shows the equivalence of four methods of analysis: a three-level mixed (multilevel) regression for repeated measures with as levels cluster, person, and time, and allowing for unstructured between-cluster and within-cluster covariance matrices; a two-level mixed regression with as levels cluster and person, using change from baseline as outcome; a two-level mixed regression with as levels cluster and time, using cluster means as data; a one-level analysis of cluster means of change from baseline. Subsequently, similar equivalences are shown between a constrained mixed model and methods using the pretest as covariate. All methods are also compared on a cluster randomized trial on mental health in children. From these equivalences follows a simple method to calculate the sample size for a cluster randomized trial with baseline measurement, which is demonstrated step-by-step.

An overview of optimal designs under a given budget in cluster randomized trials with a binary outcome

Cluster randomized trial design may raise financial concerns because the cost to recruit an additional cluster is much higher than to enroll an additional subject in subject-level randomized trials. Therefore, it is desirable to develop an optimal design. For local optimal designs, optimization means the minimum variance of the estimated treatment effect under the total budget. The local optimal design derived from the variance needs the input of an association parameter ρ in terms of a "working" correlation structure R ( ρ ) in the generalized estimating equation models. When the range of ρ instead of an exact value is available, the parameter space is defined as the range of ρ and the design space is defined as enrollment feasibility, for example, the number of clusters or cluster size. For any value ρ within the range, the optimal design and relative efficiency for each design in the design space is obtained. Then, for each design in the design space, the minimum relative efficiency within the parameter space is calculated. MaxiMin design is the optimal design that maximizes the minimum relative efficiency among all designs in the design space. Our contributions are threefold. First, for three common measures (risk difference, risk ratio, and odds ratio), we summarize all available local optimal designs and MaxiMin designs utilizing generalized estimating equation models when the group allocation proportion is predetermined for two-level and three-level parallel cluster randomized trials. We then propose the local optimal designs and MaxiMin designs using the same models when the group allocation proportion is undecided. Second, for partially nested designs, we develop the optimal designs for three common measures under the setting of equal number of subjects per cluster and exchangeable working correlation structure in the intervention group. Third, we create three new Statistical Analysis System (SAS) macros and update two existing SAS macros for all the optimal designs. We provide two examples to illustrate our methods.

Systematic review of the characteristics of school-based feasibility cluster randomised trials of interventions for improving the health of pupils in the UK

Background

The last 20 years have seen a marked increase in the use of cluster randomised trials (CRTs) in schools to evaluate interventions for improving pupil health outcomes. Schools have limited resources and participating in full-scale trials can be challenging and costly, given their main purpose is education. Feasibility studies can be used to identify challenges with implementing interventions and delivering trials. This systematic review summarises methodological characteristics and objectives of school-based cluster randomised feasibility studies in the United Kingdom (UK).

Methods

We systematically searched MEDLINE from inception to 31 December 2020. Eligible papers were school-based feasibility CRTs that included health outcomes measured on pupils.

Results

Of 3285 articles identified, 24 were included. School-based feasibility CRTs have been increasingly used in the UK since the first publication in 2008. Five (21%) studies provided justification for the use of the CRT design. Three (13%) studies provided details of a formal sample size calculation, with only one of these allowing for clustering. The median (IQR; range) recruited sample size was 7.5 (4.5 to 9; 2 to 37) schools and 274 (179 to 557; 29 to 1567) pupils. The most common feasibility objectives were to estimate the potential effectiveness of the intervention (n = 17; 71%), assess acceptability of the intervention (n = 16; 67%), and estimate the recruitment/retention rates (n = 15; 63%). Only one study was used to assess whether cluster randomisation was appropriate, and none of the studies that randomised clusters before recruiting pupils assessed the possibility of recruitment bias. Besides potential effectiveness, cost-effectiveness, and the intra-cluster correlation coefficient, no studies quantified the precision of the feasibility parameter estimates.

Conclusions

Feasibility CRTs are increasingly used in schools prior to definitive trials of interventions for improving health in pupils. The average sample size of studies included in this review would be large enough to estimate pupil-level feasibility parameters (e.g., percentage followed up) with reasonable precision. The review highlights the need for clearer sample size justification and better reporting of the precision with which feasibility parameters are estimated. Better use could be made of feasibility CRTs to assess challenges that are specific to the cluster design.

Trial registration

PROSPERO: CRD42020218993.

Supplementary Information

The online version contains supplementary material available at 10.1186/s40814-022-01098-w.

Confidence interval-based sample size determination formulas and some mathematical properties for hierarchical data

The use of hierarchical data (also called multilevel data or clustered data) is common in behavioural and psychological research when data of lower-level units (e.g., students, clients, repeated measures) are nested within clusters or higher-level units (e.g., classes, hospitals, individuals). Over the past 25 years we have seen great advances in methods for computing the sample sizes needed to obtain the desired statistical properties for such data in experimental evaluations. The present research provides closed-form and iterative formulas for sample size determination that can be used to ensure the desired width of confidence intervals for hierarchical data. Formulas are provided for a four-level hierarchical linear model that assumes slope variances and inclusion of covariates under both balanced and unbalanced designs. In addition, we address several mathematical properties relating to sample size determination for hierarchical data via the standard errors of experimental effect estimates. These include the relative impact of several indices (e.g., random intercept or slope variance at each level) on standard errors, asymptotic standard errors, minimum required values at the highest level, and generalized expressions of standard errors for designs with any-level randomization under any number of levels. In particular, information on the minimum required values will help researchers to minimize the risk of conducting experiments that are statistically unlikely to show the presence of an experimental effect.

Improving the Youth HIV Prevention and Care Cascades: Innovative Designs in the Adolescent Trials Network for HIV/AIDS Interventions

Dramatic decreases in HIV transmission are achievable with currently available biomedical and behavioral interventions, including antiretroviral therapy and pre-exposure prophylaxis. However, such decreases have not yet been realized among adolescents and young adults. The Adolescent Medicine Trials Network (ATN) for HIV/AIDS interventions is dedicated to research addressing the needs of youth at high risk for HIV acquisition as well as youth living with HIV. This article provides an overview of an array of efficient and effective designs across the translational spectrum that are utilized within the ATN. These designs maximize methodological rigor and real-world applicability of findings while minimizing resource use. Implementation science and cost-effectiveness methods are included. Utilizing protocol examples, we demonstrate the feasibility of such designs to balance rigor and relevance to shorten the science-to-practice gap and improve the youth HIV prevention and care continua.

The design and conduct of Project RedDE: A cluster-randomized trial to reduce diagnostic errors in pediatric primary care

BACKGROUND

Diagnostic errors contribute to the large burden of healthcare-associated harm experienced by children. Primary care settings involve high diagnostic uncertainty and limited time and information, creating ideal conditions for diagnostic errors. We report on the design and conduct of Project RedDE, a stepped-wedge, cluster-randomized controlled trial of a virtual quality improvement collaborative aimed at reducing diagnostic errors in pediatric primary care.

METHODS

Project RedDE cluster-randomized pediatric primary care practices into one of three groups. Each group participated in a quality improvement collaborative targeting the same three diagnostic errors (missed diagnoses of elevated blood pressure and adolescent depression and delayed diagnoses of abnormal laboratory studies), but in a different sequence. During the quality improvement collaborative, practices worked both independently and collaboratively, leveraging general quality improvement strategies (e.g. process mapping) in addition to error-specific content (e.g. pocket guides for blood pressure norms) delivered during the intervention phase for each error. The quality improvement collaborative intervention included interactive learning sessions and webinars, quality improvement coaching at the team level, and repeated evaluation of failures via root cause analyses. Pragmatic data were collected monthly, submitted to a centralized data aggregator, and returned to the practices in the form of run charts comparing each practice's progress over time to that of the group. The primary analysis used patients as the unit of analysis and compared diagnostic error proportions between the intervention and baseline periods, while secondary analyses evaluated the sustainability of observed reductions in diagnostic errors after the intervention period ended.

RESULTS

A total of 43 practices were recruited and randomized into Project RedDE. Eleven practices withdrew before submitting any data, and one practice merged with another participating practice, leaving 31 practices that began work on Project RedDE. All but one of the diverse, national pediatric primary care practices that participated ultimately submitted complete data. Quality improvement collaborative participation was robust, with an average of 63% of practices present on quality improvement collaborative webinars and 85% of practices present for quality improvement collaborative learning sessions. Complete data included 30 months of outcome data for the first diagnostic error worked on, 24 months of outcome data for the second, and 16 months of data for the third.

LESSONS LEARNED AND LIMITATIONS

Contamination across study groups was a recurring concern; concerted efforts were made to mitigate this risk. Electronic health records played a large role in teams' success.

CONCLUSION

Project RedDE, a virtual quality improvement collaborative aimed at reducing diagnostic errors in pediatric primary care, successfully recruited and retained a diverse, national group of pediatric primary care practices. The stepped-wedge, cluster-randomized controlled trial design allowed for enhanced scientific efficiency.

Appropriate statistical methods for analysing partially nested randomised controlled trials with continuous outcomes: a simulation study

Background

In individually randomised trials we might expect interventions delivered in groups or by care providers to result in clustering of outcomes for participants treated in the same group or by the same care provider. In partially nested randomised controlled trials (pnRCTs) this clustering only occurs in one trial arm, commonly the intervention arm. It is important to measure and account for between-cluster variability in trial design and analysis. We compare analysis approaches for pnRCTs with continuous outcomes, investigating the impact on statistical inference of cluster sizes, coding of the non-clustered arm, intracluster correlation coefficient (ICCs), and differential variance between intervention and control arm, and provide recommendations for analysis.

Methods

We performed a simulation study assessing the performance of six analysis approaches for a two-arm pnRCT with a continuous outcome. These include: linear regression model; fully clustered mixed-effects model with singleton clusters in control arm; fully clustered mixed-effects model with one large cluster in control arm; fully clustered mixed-effects model with pseudo clusters in control arm; partially nested homoscedastic mixed effects model, and partially nested heteroscedastic mixed effects model. We varied the cluster size, number of clusters, ICC, and individual variance between the two trial arms.

Results

All models provided unbiased intervention effect estimates. In the partially nested mixed-effects models, methods for classifying the non-clustered control arm had negligible impact. Failure to account for even small ICCs resulted in inflated Type I error rates and over-coverage of confidence intervals. Fully clustered mixed effects models provided poor control of the Type I error rates and biased ICC estimates. The heteroscedastic partially nested mixed-effects model maintained relatively good control of Type I error rates, unbiased ICC estimation, and did not noticeably reduce power even with homoscedastic individual variances across arms.

Conclusions

In general, we recommend the use of a heteroscedastic partially nested mixed-effects model, which models the clustering in only one arm, for continuous outcomes similar to those generated under the scenarios of our simulations study. However, with few clusters (3–6), small cluster sizes (5–10), and small ICC (≤0.05) this model underestimates Type I error rates and there is no optimal model.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0559-x) contains supplementary material, which is available to authorized users.

Reducing the default dispense quantity for new opioid analgesic prescriptions: study protocol for a cluster randomised controlled trial

INTRODUCTION

As opioid analgesic consumption has grown, so have opioid use disorder and opioid-related overdoses. Reducing the quantity of opioid analgesics prescribed for acute non-cancer pain can potentially reduce risks to the individual receiving the prescription and to others who might unintentionally or intentionally consume any leftover tablets. Reducing the default dispense quantity for new opioid analgesic prescriptions in the electronic health record (EHR) is a promising intervention to reduce prescribing.

METHODS AND ANALYSIS

This study is a prospective cluster randomised controlled trial with two parallel arms. Primary care sites (n=32) and emergency departments (n=4) will be randomised in matched pairs to either a modification of the EHR so that new opioid analgesic prescriptions default to a dispense quantity of 10 tablets (intervention) or to no EHR change (control). The dispense quantity will remain fully modifiable by providers in both arms. From 6 months preintervention to 18 months postintervention, patient-level data will be analysed (ie, the patient is the unit of inference). Patient eligibility criteria are: (A) received a new opioid analgesic prescription, defined as no other opioid analgesic prescription in the prior 6 months; (B) age ≥18 years; and (C) no cancer diagnosis within 1 year prior to the new opioid analgesic prescription. The primary outcome will be the quantity of opioid analgesics prescribed in the initial prescription. Secondary outcomes will include opioid analgesic reorders and health service utilisation within 30 days after the initial prescription. Outcomes will be compared between study arms using a difference-in-differences analysis.

ETHICS AND DISSEMINATION

This study has been approved by the Montefiore Medical Center/Albert Einstein College of Medicine Institutional Review Board with a waiver of informed consent (2016-6036) and is registered on ClinicalTrials.gov (NCT03003832, 6 December 2016). Findings will be disseminated through publication, conferences and meetings with health system leaders.

TRIAL REGISTRATION NUMBER

NCT03003832; Pre-results.

Trial Characteristics and Appropriateness of Statistical Methods Applied for Design and Analysis of Randomized School-Based Studies Addressing Weight-Related Issues: A Literature Review

OBJECTIVE

To evaluate whether clustering effects, often quantified by the intracluster correlation coefficient (ICC), were appropriately accounted for in design and analysis of school-based trials.

METHODS

We searched PubMed and extracted variables concerning study characteristics, power analysis, ICC use for power analysis, applied statistical models, and the report of the ICC estimated from the observed data.

RESULTS

N=263 papers were identified, and N=121 papers were included for evaluation. Overall, only a minority (21.5%) of studies incorporated ICC values for power analysis, fewer studies (8.3%) reported the estimated ICC, and 68.6% of studies applied appropriate multilevel models. A greater proportion of studies applied the appropriate models during the past five years (2013-2017) compared to the prior years (74.1% versus 63.5%, p=0.176). Significantly associated with application of appropriate models were a larger number of schools (p=0.030), a larger sample size (p=0.002), longer follow-up (p=0.014), and randomization at a cluster level (p < 0.001) and so were studies that incorporated the ICC into power analysis (p=0.016) and reported the estimated ICC (p=0.030).

CONCLUSION

Although application of appropriate models has increased over the years, consideration of clustering effects in power analysis has been inadequate, as has report of estimated ICC. To increase rigor, future school-based trials should address these issues at both the design and analysis stages.

Sample size determination for multilevel hierarchical designs using generalized linear mixed models

A unified statistical methodology of sample size determination is developed for hierarchical designs that are frequently used in many areas, particularly in medical and health research studies. The solid foundation of the proposed methodology opens a new horizon for power analysis in presence of various conditions. Important features such as joint significance testing, unequal allocations of clusters across intervention groups, and differential attrition rates over follow up time points are integrated to address some useful questions that investigators often encounter while conducting such studies. Proposed methodology is shown to perform well in terms of maintaining type I error rates and achieving the target power under various conditions. Proposed method is also shown to be robust with respect to violation of distributional assumptions of random-effects.

Sample size determinations for stepped-wedge clinical trials from a three-level data hierarchy perspective

Stepped-wedge (SW) designs have been steadily implemented in a variety of trials. A SW design typically assumes a three-level hierarchical data structure where participants are nested within times or periods which are in turn nested within clusters. Therefore, statistical models for analysis of SW trial data need to consider two correlations, the first and second level correlations. Existing power functions and sample size determination formulas had been derived based on statistical models for two-level data structures. Consequently, the second-level correlation has not been incorporated in conventional power analyses. In this paper, we derived a closed-form explicit power function based on a statistical model for three-level continuous outcome data. The power function is based on a pooled overall estimate of stratified cluster-specific estimates of an intervention effect. The sampling distribution of the pooled estimate is derived by applying a fixed-effect meta-analytic approach. Simulation studies verified that the derived power function is unbiased and can be applicable to varying number of participants per period per cluster. In addition, when data structures are assumed to have two levels, we compare three types of power functions by conducting additional simulation studies under a two-level statistical model. In this case, the power function based on a sampling distribution of a marginal, as opposed to pooled, estimate of the intervention effect performed the best. Extensions of power functions to binary outcomes are also suggested.