Pragmatic clinical trials embedded in healthcare systems: generalizable lessons from the NIH Collaboratory

BACKGROUND

The clinical research enterprise is not producing the evidence decision makers arguably need in a timely and cost effective manner; research currently involves the use of labor-intensive parallel systems that are separate from clinical care. The emergence of pragmatic clinical trials (PCTs) poses a possible solution: these large-scale trials are embedded within routine clinical care and often involve cluster randomization of hospitals, clinics, primary care providers, etc. Interventions can be implemented by health system personnel through usual communication channels and quality improvement infrastructure, and data collected as part of routine clinical care. However, experience with these trials is nascent and best practices regarding design operational, analytic, and reporting methodologies are undeveloped.

METHODS

To strengthen the national capacity to implement cost-effective, large-scale PCTs, the Common Fund of the National Institutes of Health created the Health Care Systems Research Collaboratory (Collaboratory) to support the design, execution, and dissemination of a series of demonstration projects using a pragmatic research design.

RESULTS

In this article, we will describe the Collaboratory, highlight some of the challenges encountered and solutions developed thus far, and discuss remaining barriers and opportunities for large-scale evidence generation using PCTs.

CONCLUSION

A planning phase is critical, and even with careful planning, new challenges arise during execution; comparisons between arms can be complicated by unanticipated changes. Early and ongoing engagement with both health care system leaders and front-line clinicians is critical for success. There is also marked uncertainty when applying existing ethical and regulatory frameworks to PCTS, and using existing electronic health records for data capture adds complexity.

Mixed-effects models for the design and analysis of stepped wedge cluster randomized trials: An overview

The stepped wedge cluster randomized design has received increasing attention in pragmatic clinical trials and implementation science research. The key feature of the design is the unidirectional crossover of clusters from the control to intervention conditions on a staggered schedule, which induces confounding of the intervention effect by time. The stepped wedge design first appeared in the Gambia hepatitis study in the 1980s. However, the statistical model used for the design and analysis was not formally introduced until 2007 in an article by Hussey and Hughes. Since then, a variety of mixed-effects model extensions have been proposed for the design and analysis of these trials. In this article, we explore these extensions under a unified perspective. We provide a general model representation and regard various model extensions as alternative ways to characterize the secular trend, intervention effect, as well as sources of heterogeneity. We review the key model ingredients and clarify their implications for the design and analysis. The article serves as an entry point to the evolving statistical literatures on stepped wedge designs.

Identifying optimal threshold statistics for elimination of hookworm using a stochastic simulation model

BACKGROUND

There is an increased focus on whether mass drug administration (MDA) programmes alone can interrupt the transmission of soil-transmitted helminths (STH). Mathematical models can be used to model these interventions and are increasingly being implemented to inform investigators about expected trial outcome and the choice of optimum study design. One key factor is the choice of threshold for detecting elimination. However, there are currently no thresholds defined for STH regarding breaking transmission.

METHODS

We develop a simulation of an elimination study, based on the DeWorm3 project, using an individual-based stochastic disease transmission model in conjunction with models of MDA, sampling, diagnostics and the construction of study clusters. The simulation is then used to analyse the relationship between the study end-point elimination threshold and whether elimination is achieved in the long term within the model. We analyse the quality of a range of statistics in terms of the positive predictive values (PPV) and how they depend on a range of covariates, including threshold values, baseline prevalence, measurement time point and how clusters are constructed.

RESULTS

End-point infection prevalence performs well in discriminating between villages that achieve interruption of transmission and those that do not, although the quality of the threshold is sensitive to baseline prevalence and threshold value. Optimal post-treatment prevalence threshold value for determining elimination is in the range 2% or less when the baseline prevalence range is broad. For multiple clusters of communities, both the probability of elimination and the ability of thresholds to detect it are strongly dependent on the size of the cluster and the size distribution of the constituent communities. Number of communities in a cluster is a key indicator of probability of elimination and PPV. Extending the time, post-study endpoint, at which the threshold statistic is measured improves PPV value in discriminating between eliminating clusters and those that bounce back.

CONCLUSIONS

The probability of elimination and PPV are very sensitive to baseline prevalence for individual communities. However, most studies and programmes are constructed on the basis of clusters. Since elimination occurs within smaller population sub-units, the construction of clusters introduces new sensitivities for elimination threshold values to cluster size and the underlying population structure. Study simulation offers an opportunity to investigate key sources of sensitivity for elimination studies and programme designs in advance and to tailor interventions to prevailing local or national conditions.

A review of the use of covariates in cluster randomized trials uncovers marked discrepancies between guidance and practice

OBJECTIVES

Reviews of the handling of covariates in trials have explicitly excluded cluster randomized trials (CRTs). In this study, we review the use of covariates in randomization, the reporting of covariates, and adjusted analyses in CRTs.

STUDY DESIGN AND SETTING

We reviewed a random sample of 300 CRTs published between 2000 and 2008 across 150 English language journals.

RESULTS

Fifty-eight percent of trials used covariates in randomization. Only 69 (23%) included tables of cluster- and individual-level covariates. Fifty-eight percent reported significance tests of baseline balance. Of 207 trials that reported baseline measures of the primary outcome, 155 (75%) subsequently adjusted for these in analyses. Of 174 trials that used covariates in randomization, 30 (17%) included an analysis adjusting for all those covariates. Of 219 trial reports that included an adjusted analysis of the primary outcome, only 71 (32%) reported that covariates were chosen a priori.

CONCLUSION

There are some marked discrepancies between practice and guidance on the use of covariates in the design, analysis, and reporting of CRTs. It is essential that researchers follow guidelines on the use and reporting of covariates in CRTs, promoting the validity of trial conclusions and quality of trial reports.

High-throughput sequencing of pooled samples to determine community-level microbiome diversity

Purpose

Community-level interventions in cluster randomized controlled trials may alter the gut microbiome of individuals. The current method of estimating community diversities uses microbiome data obtained from multiple individual's specimens. Here we propose randomly pooling a number of microbiome samples from the same community into one sample before sequencing to estimate community-level microbiome diversity.

Methods

We design and analyze an experiment to compare community microbiome diversity (gamma-diversity) estimates derived from 16S rRNA gene sequencing of 1) individually sequenced specimens vs. 2) pooled specimens collected from a community. Pool sizes of 10, 20, and 40 are considered. We then compare the gamma-estimates using Pearson's correlation as well as using Bland and Altman agreement analysis for three established diversity indices including richness, Simpson's and Shannon's.

Results

The gamma-diversity estimates are highly correlated, with most being statistically significant. All correlations between all three diversity estimates are significant in the 10-pooled data. Pools comprising 40 specimens are closest to the line of agreement, but all pooled samples and individual samples fall within the 95% limits of agreement.

Conclusions

Pooling microbiome samples before DNA amplification and metagenomics sequencing to estimate community-level diversity is a viable measure to consider in population-level association research studies.

Stepped wedge trials with continuous recruitment require new ways of thinking

OBJECTIVES

There is substantial variation in the design of stepped wedge trials. Many recruit participants continuously over time, although the methodological literature has tended not to differentiate closely between continuous recruitment and discrete sampling. We argue for a deeper understanding of the special features of stepped wedge trials with continuous recruitment.

STUDY DESIGN AND SETTING

This is a commentary and informal review.

RESULTS

We discuss the scheduling of recruitment and implementation in continuous time and how contamination might be avoided. We also offer some suggestions on reporting and terminology for stepped wedge trials with continuous recruitment and comment on issues for analysis.

CONCLUSION

Repeated cross-section and continuous recruitment stepped wedge trials are not the same thing. More work is needed to develop the theory and practice of stepped wedge designs with continuous recruitment. Thoughtful approaches to design and clarity of reporting are vital.

Marginal modeling of cluster-period means and intraclass correlations in stepped wedge designs with binary outcomes

Stepped wedge cluster randomized trials (SW-CRTs) with binary outcomes are increasingly used in prevention and implementation studies. Marginal models represent a flexible tool for analyzing SW-CRTs with population-averaged interpretations, but the joint estimation of the mean and intraclass correlation coefficients (ICCs) can be computationally intensive due to large cluster-period sizes. Motivated by the need for marginal inference in SW-CRTs, we propose a simple and efficient estimating equations approach to analyze cluster-period means. We show that the quasi-score for the marginal mean defined from individual-level observations can be reformulated as the quasi-score for the same marginal mean defined from the cluster-period means. An additional mapping of the individual-level ICCs into correlations for the cluster-period means further provides a rigorous justification for the cluster-period approach. The proposed approach addresses a long-recognized computational burden associated with estimating equations defined based on individual-level observations, and enables fast point and interval estimation of the intervention effect and correlations. We further propose matrix-adjusted estimating equations to improve the finite-sample inference for ICCs. By providing a valid approach to estimate ICCs within the class of generalized linear models for correlated binary outcomes, this article operationalizes key recommendations from the CONSORT extension to SW-CRTs, including the reporting of ICCs.

Two-Stage TMLE to reduce bias and improve efficiency in cluster randomized trials

Cluster randomized trials (CRTs) randomly assign an intervention to groups of individuals (e.g., clinics or communities) and measure outcomes on individuals in those groups. While offering many advantages, this experimental design introduces challenges that are only partially addressed by existing analytic approaches. First, outcomes are often missing for some individuals within clusters. Failing to appropriately adjust for differential outcome measurement can result in biased estimates and inference. Second, CRTs often randomize limited numbers of clusters, resulting in chance imbalances on baseline outcome predictors between arms. Failing to adaptively adjust for these imbalances and other predictive covariates can result in efficiency losses. To address these methodological gaps, we propose and evaluate a novel two-stage targeted minimum loss-based estimator to adjust for baseline covariates in a manner that optimizes precision, after controlling for baseline and postbaseline causes of missing outcomes. Finite sample simulations illustrate that our approach can nearly eliminate bias due to differential outcome measurement, while existing CRT estimators yield misleading results and inferences. Application to real data from the SEARCH community randomized trial demonstrates the gains in efficiency afforded through adaptive adjustment for baseline covariates, after controlling for missingness on individual-level outcomes.

Alternatives to Logistic Regression Models when Analyzing Cluster Randomized Trials with Binary Outcomes

Binary outcomes are often encountered when analyzing cluster randomized trials (CRTs). A common approach to obtaining the average treatment effect of an intervention may involve using a logistic regression model. We outline some interpretive and statistical challenges associated with using logistic regression and discuss two alternative/supplementary approaches for analyzing clustered data with binary outcomes: the linear probability model (LPM) and the modified Poisson regression model. In our simulation and applied example, all models use a standard error adjustment that is effective even if a low number of clusters is present. Simulation results show that both the LPM and modified Poisson regression models can provide unbiased point estimates with acceptable coverage and type I error rates even with as little as 20 clusters.

Cluster randomized trials of individual-level interventions were at high risk of bias

OBJECTIVES

To describe the prevalence of risks of bias in cluster-randomized trials of individual-level interventions, according to the Cochrane Risk of Bias tool.

STUDY DESIGN AND SETTING

Review undertaken in duplicate of a random sample of 40 primary reports of cluster-randomized trials of individual-level interventions.

RESULTS

The most common reported reasons for adopting cluster randomization were the need to avoid contamination (17, 42.5%) and practical considerations (14, 35%). Of the 40 trials all but one was assessed as being at risk of bias. A majority (27, 67.5%) were assessed as at risk due to the timing of identification and recruitment of participants; many (21, 52.5%) due to an apparent lack of adequate allocation concealment; and many due to selectively reported results (22, 55%), arising from a mixture of reasons including lack of documentation of primary outcome. Other risks mostly occurred infrequently.

CONCLUSION

Many cluster-randomized trials evaluating individual-level interventions appear to be at risk of bias, mostly due to identification and recruitment biases. We recommend that investigators carefully consider the need for cluster randomization; follow recommended procedures to mitigate risks of identification and recruitment bias; and adhere to good reporting practices including clear documentation of primary outcome and allocation concealment methods.

Intracluster correlation coefficients from school-based cluster randomized trials of interventions for improving health outcomes in pupils

BACKGROUND AND OBJECTIVES

To summarize intracluster correlation coefficient (ICC) estimates for pupil health outcomes from school-based cluster randomized trials (CRTs) across world regions and describe their relationship with study design characteristics and context.

METHODS

School-based CRTs reporting ICCs for pupil health outcomes were identified through a literature search of MEDLINE (via Ovid). ICC estimates were summarized both overall and for different categories of study characteristics.

RESULTS

Two hundred and forty-six articles reporting ICC estimates were identified. The median (interquartile range) ICC was 0.031 (0.011 to 0.08) at the school level (N = 210) and 0.063 (0.024 to 0.1) at the class level (N = 46). The distribution of ICCs at the school level was well described by the beta and exponential distributions. Besides larger ICCs in definitive trials than feasibility studies, there were no clear associations between study characteristics and ICC estimates.

CONCLUSION

The distribution of school-level ICCs worldwide was similar to previous summaries from studies in the United States. The description of the distribution of ICCs will help to inform sample size calculations and assess their sensitivity when designing future school-based CRTs of health interventions.

Stepped Wedge Cluster Randomized Trials: A Methodological Overview

BACKGROUND

Stepped wedge cluster randomized trials enable rigorous evaluations of health intervention programs in pragmatic settings. In the present study, we aimed to update neurosurgeon scientists on the design of stepped wedge randomized trials.

METHODS

We have presented an overview of recent methodological developments for stepped wedge designs and included an update on the newer associated methodological tools to aid with future study designs.

RESULTS

We defined the stepped wedge trial design and reviewed the indications for the design in depth. In addition, key considerations, including mainstream methods of analysis and sample size determination, were discussed.

CONCLUSIONS

Stepped wedge designs can be attractive for study intervention programs aiming to improve the delivery of patient care, especially when examining a small number of heterogeneous clusters.

Statistical analysis and handling of missing data in cluster randomised trials: protocol for a systematic review

INTRODUCTION

Cluster randomised trials (CRTs) randomise participants in groups, rather than as individuals, and are key tools used to assess interventions in health research where treatment contamination is likely or if individual randomisation is not feasible. Missing outcome data can reduce power in trials, including in CRTs, and is a potential source of bias. The current review focuses on evaluating methods used in statistical analysis and handling of missing data with respect to the primary outcome in CRTs.

METHODS AND ANALYSIS

We will search for CRTs published between August 2013 and July 2014 using PubMed, Web of Science and PsycINFO. We will identify relevant studies by screening titles and abstracts, and examining full-text articles based on our predefined study inclusion criteria. 86 studies will be randomly chosen to be included in our review. Two independent reviewers will collect data from each study using a standardised, prepiloted data extraction template. Our findings will be summarised and presented using descriptive statistics.

ETHICS AND DISSEMINATION

This methodological systematic review does not need ethical approval because there are no data used in our study that are linked to individual patient data. After completion of this systematic review, data will be immediately analysed, and findings will be disseminated through a peer-reviewed publication and conference presentation.

Accounting for confounding by time, early intervention adoption, and time-varying effect modification in the design and analysis of stepped-wedge designs: application to a proposed study design to reduce opioid-related mortality

BACKGROUND

Beginning in 2019, stepped-wedge designs (SWDs) were being used in the investigation of interventions to reduce opioid-related deaths in communities across the United States. However, these interventions are competing with external factors such as newly initiated public policies limiting opioid prescriptions, media awareness campaigns, and the COVID-19 pandemic. Furthermore, control communities may prematurely adopt components of the intervention as they become available. The presence of time-varying external factors that impact study outcomes is a well-known limitation of SWDs; common approaches to adjusting for them make use of a mixed effects modeling framework. However, these models have several shortcomings when external factors differentially impact intervention and control clusters.

METHODS

We discuss limitations of commonly used mixed effects models in the context of proposed SWDs to investigate interventions intended to reduce opioid-related mortality, and propose extensions of these models to address these limitations. We conduct an extensive simulation study of anticipated data from SWD trials targeting the current opioid epidemic in order to examine the performance of these models in the presence of external factors. We consider confounding by time, premature adoption of intervention components, and time-varying effect modification- in which external factors differentially impact intervention and control clusters.

RESULTS

In the presence of confounding by time, commonly used mixed effects models yield unbiased intervention effect estimates, but can have inflated Type 1 error and result in under coverage of confidence intervals. These models yield biased intervention effect estimates when premature intervention adoption or effect modification are present. In such scenarios, models incorporating fixed intervention-by-time interactions with an unstructured covariance for intervention-by-cluster-by-time random effects result in unbiased intervention effect estimates, reach nominal confidence interval coverage, and preserve Type 1 error.

CONCLUSIONS

Mixed effects models can adjust for different combinations of external factors through correct specification of fixed and random time effects. Since model choice has considerable impact on validity of results and study power, careful consideration must be given to how these external factors impact study endpoints and what estimands are most appropriate in the presence of such factors.

Key considerations for designing, conducting and analysing a cluster randomized trial

Not only do cluster randomized trials require a larger sample size than individually randomized trials, they also face many additional complexities. The potential for contamination is the most commonly used justification for using cluster randomization, but the risk of contamination should be carefully weighed against the more serious problem of questionable scientific validity in settings with post-randomization identification or recruitment of participants unblinded to the treatment allocation. In this paper we provide some simple guidelines to help researchers conduct cluster trials in a way that minimizes potential biases and maximizes statistical efficiency. The overarching theme of this guidance is that methods that apply to individually randomized trials rarely apply to cluster randomized trials. We recommend that cluster randomization be only used when necessary-balancing the benefits of cluster randomization with its increased risks of bias and increased sample size. Researchers should also randomize at the lowest possible level-balancing the risks of contamination with ensuring an adequate number of randomization units-as well as exploring other options for statistically efficient designs. Clustering should always be allowed for in the sample size calculation; and the use of restricted randomization (and adjustment in the analysis for covariates used in the randomization) should be considered. Where possible, participants should be recruited before randomizing clusters and, when recruiting (or identifying) participants post-randomization, recruiters should be masked to the allocation. In the analysis, the target of inference should align with the research question, and adjustment for clustering and small sample corrections should be used when the trial includes less than about 40 clusters.

Ethics of alternative trial designs and methods in low-resource settings

This editorial introduces articles in this Special Issue, which are based on presentations given at the 2017 meeting of the Global Forum of Bioethics in Research meeting. The main themes presented at the meeting were the use of cluster randomized trials, stepped-wedge cluster randomized trials, and controlled human infection models in research conducted in low-resource settings. The editorial sets out which ethical issues may arise in the context of alternative trial designs and describes the articles in this issue that addresses some or more of the ethical issues, such as justification of the research design, risk-benefit evaluations and consent.

Intent-to-treat analysis of cluster randomized trials when clusters report unidentifiable outcome proportions

BACKGROUND

Cluster randomized trials are designed to evaluate interventions at the cluster or group level. When clusters are randomized but some clusters report no or non-analyzable data, intent-to-treat analysis, the gold standard for the analysis of randomized controlled trials, can be compromised. This article presents a very flexible statistical methodology for cluster randomized trials whose outcome is a cluster-level proportion (e.g. proportion from a cluster reporting an event) in the setting where clusters report non-analyzable data (which in general could be due to nonadherence, dropout, missingness, etc.). The approach is motivated by a previously published stratified randomized controlled trial called, "The Randomized Recruitment Intervention Trial (RECRUIT)," designed to examine the effectiveness of a trust-based continuous quality improvement intervention on increasing minority recruitment into clinical trials (ClinicalTrials.gov Identifier: NCT01911208).

METHODS

The novel approach exploits the use of generalized estimating equations for cluster-level reports, such that all clusters randomized at baseline are able to be analyzed, and intervention effects are presented as risk ratios. Simulation studies under different outcome missingness scenarios and a variety of intra-cluster correlations are conducted. A comparative analysis of the method with imputation and per protocol approaches for RECRUIT is presented.

RESULTS

Simulation results show the novel approach produces unbiased and efficient estimates of the intervention effect that maintain the nominal type I error rate. Application to RECRUIT shows similar effect sizes when compared to the imputation and per protocol approach.

CONCLUSION

The article demonstrates that an innovative bivariate generalized estimating equations framework allows one to implement an intent-to-treat analysis to obtain risk ratios or odds ratios, for a variety of cluster randomized designs.

Analysis of multiple-period group randomized trials: random coefficients model or repeated measures ANOVA?

BACKGROUND

Multiple-period parallel group randomized trials (GRTs) analyzed with linear mixed models can represent time in mean models as continuous or categorical. If time is continuous, random effects are traditionally group- and member-level deviations from condition-specific slopes and intercepts and are referred to as random coefficients (RC) analytic models. If time is categorical, random effects are traditionally group- and member-level deviations from time-specific condition means and are referred to as repeated measures ANOVA (RM-ANOVA) analytic models. Longstanding guidance recommends the use of RC over RM-ANOVA for parallel GRTs with more than two periods because RC exhibited nominal type I error rates for both time parameterizations while RM-ANOVA exhibited inflated type I error rates when applied to data generated using the RC model. However, this recommendation was developed assuming a variance components covariance matrix for the RM-ANOVA, using only cross-sectional data, and explicitly modeling time × group variation. Left unanswered were how well RM-ANOVA with an unstructured covariance would perform on data generated according to the RC mechanism, if similar patterns would be observed in cohort data, and the impact of not modeling time × group variation if such variation was present in the data-generating model.

METHODS

Continuous outcomes for cohort and cross-sectional parallel GRT data were simulated according to RM-ANOVA and RC mechanisms at five total time periods. All simulations assumed time × group variation. We varied the number of groups, group size, and intra-cluster correlation. Analytic models using RC, RM-ANOVA, RM-ANOVA with unstructured covariance, and a Saturated random effects structure were applied to the data. All analytic models specified time × group random effects. The analytic models were then reapplied without specifying random effects for time × group.

RESULTS

Results indicated the RC and saturated analytic models maintained the nominal type I error rate in all data sets, RM-ANOVA with an unstructured covariance did not avoid type I error rate inflation when applied to cohort RC data, and analytic models omitting time-varying group random effects when such variation exists in the data were prone to substantial type I error inflation unless the residual error variance is high relative to the time × group variance.

CONCLUSION

The time × group RC and saturated analytic models are recommended as the default for multiple period parallel GRTs.

The interplay between WASH practices and vaccination with oral cholera vaccines in protecting against cholera in urban Bangladesh: Reanalysis of a cluster-randomized trial

The current global initiative to end Cholera by 2030 emphasizes the use of oral cholera vaccine (OCV) combined with feasible household Water-Sanitation-Hygiene (WASH) interventions. However, little is known about how improved WASH practices and behaviors and OCV interact to reduce the risk of cholera. We reanalyzed two arms of a cluster-randomized trial in urban Bangladesh, to evaluate the effectiveness of OCV given as a 2-dose regimen. One arm (30 clusters, n = 94,675) was randomized to vaccination of persons aged one year and older with OCV, and the other arm (30 clusters, n = 80,056) to no intervention. We evaluated the prevention of cholera by household WASH, classified at baseline using a previously validated rule, and OCV over 2 years of follow-up. When analyzed by assignment to OCV clusters rather than receipt of OCV, in comparison to persons living in "Not Better WASH" households in the control clusters, reduction of severe cholera (the primary outcome) was similar for persons in "Not Better WASH" households in vaccine clusters (46%, 95% CI:24,62), for persons in "Better WASH" households in the control clusters (48%, 95% CI:25,64), and for persons in "Better WASH" households in the vaccine clusters (48%, 95% CI:16,67). In contrast, when analyzed by actual receipt of a complete OCV regimen, , in comparison to persons in "Not Better WASH" households in the control clusters, protection against severe cholera increased steadily from 39% (95% CI:13,58) in residents of "Better WASH" households in the control clusters to 57% (95% CI:35,72) in vaccinated persons in "Not Better WASH" households to 63% (95% CI:21,83) in vaccinated persons in "Better WASH" households. This analysis suggests that improved household WASH and OCV received may interact to provide greater protection against cholera. However, the divergence between findings related to intent to vaccinate versus those pertaining to actual receipt of OCV underscores the need for further research on this topic.

Methodological challenges in pragmatic trials in Alzheimer's disease and related dementias: Opportunities for improvement

BACKGROUND AND AIMS

We need more pragmatic trials of interventions to improve care and outcomes for people living with Alzheimer's disease and related dementias. However, these trials present unique methodological challenges in their design, analysis, and reporting-often, due to the presence of one or more sources of clustering. Failure to account for clustering in the design and analysis can lead to increased risks of Type I and Type II errors. We conducted a review to describe key methodological characteristics and obtain a "baseline assessment" of methodological quality of pragmatic trials in dementia research, with a view to developing new methods and practical guidance to support investigators and methodologists conducting pragmatic trials in this field.

METHODS

We used a published search filter in MEDLINE to identify trials more likely to be pragmatic and identified a subset that focused on people living with Alzheimer's disease or other dementias or included them as a defined subgroup. Pairs of reviewers extracted descriptive information and key methodological quality indicators from each trial.

RESULTS

We identified N = 62 eligible primary trial reports published across 36 different journals. There were 15 (24%) individually randomized, 38 (61%) cluster randomized, and 9 (15%) individually randomized group treatment designs; 54 (87%) trials used repeated measures on the same individual and/or cluster over time and 17 (27%) had a multivariate primary outcome (e.g. due to measuring an outcome on both the patient and their caregiver). Of the 38 cluster randomized trials, 16 (42%) did not report sample size calculations accounting for the intracluster correlation and 13 (34%) did not account for intracluster correlation in the analysis. Of the 9 individually randomized group treatment trials, 6 (67%) did not report sample size calculations accounting for intracluster correlation and 8 (89%) did not account for it in the analysis. Of the 54 trials with repeated measurements, 45 (83%) did not report sample size calculations accounting for repeated measurements and 19 (35%) did not utilize at least some of the repeated measures in the analysis. No trials accounted for the multivariate nature of their primary outcomes in sample size calculation; only one did so in the analysis.

CONCLUSION

There is a need and opportunity to improve the design, analysis, and reporting of pragmatic trials in dementia research. Investigators should pay attention to the potential presence of one or more sources of clustering. While methods for longitudinal and cluster randomized trials are well developed, accessible resources and new methods for dealing with multiple sources of clustering are required. Involvement of a statistician with expertise in longitudinal and clustered designs is recommended.

Two weights make a wrong: Cluster randomized trials with variable cluster sizes and heterogeneous treatment effects

In cluster randomized trials (CRTs), the hierarchical nesting of participants (level 1) within clusters (level 2) leads to two conceptual populations: clusters and participants. When cluster sizes vary and the goal is to generalize to a hypothetical population of clusters, the unit average treatment effect (UATE), which averages equally at the cluster level rather than equally at the participant level, is a common estimand of interest. From an analytic perspective, when a generalized estimating equations (GEE) framework is used to obtain averaged treatment effect estimates for CRTs with variable cluster sizes, it is natural to specify an inverse cluster size weighted analysis so that each cluster contributes equally and to adopt an exchangeable working correlation matrix to account for within-cluster correlation. However, such an approach essentially uses two distinct weights in the analysis (i.e. both cluster size weights and covariance weights) and, in this article, we caution that it will lead to biased and/or inefficient treatment effect estimates for the UATE estimand. That is, two weights "make a wrong" or lead to poor estimation characteristics. These findings are based on theoretical derivations, corroborated via a simulation study, and illustrated using data from a CRT of a colorectal cancer screening program. We show that, an analysis with both an independence working correlation matrix and weighting by inverse cluster size is the only approach that always provides valid results for estimation of the UATE in CRTs with variable cluster sizes.

Design and execution of a public randomization ceremony to enhance stakeholder engagement within a cluster randomized trial to improve tuberculosis diagnosis in Uganda

Public randomization ceremonies have been proposed as a strategy to strengthen stakeholder engagement and address concerns and misconceptions associated with trial randomization. However, there are few published examples that describe how to conduct a public randomization ceremony with meaningful stakeholder engagement or how such ceremonies impact stakeholder perceptions about randomization and the randomization process. Cluster randomization for the GeneXpert Performance Evaluation for Linkage to Tuberculosis Care (XPEL-TB) trial was conducted at a public randomization ceremony attended by 70 stakeholders in Kampala, Uganda. Presentations given by the Acting Assistant Commissioner from the Uganda National Tuberculosis and Leprosy Programme and trial investigators emphasized how the trial aimed to further national TB goals, as well as how stakeholders contributed to the intervention design. The purpose and process of randomization were described using simple text and visuals. Randomization was an interactive activity that required participation of stakeholders from each trial site. A survey administered to stakeholders at the end of the ceremony suggested high comprehension of randomization (98%), trust in the randomization process (96%), and satisfaction with randomization outcomes (96%). Public randomization ceremonies should be considered more routinely to engage stakeholders in and address potential concerns about the fairness and impartiality of the randomization process for community-based trials.

Multilevel Intervention Stepped Wedge Designs (MLI-SWDs)

Multilevel interventions (MLIs) hold promise for reducing health inequities by intervening at multiple types of social determinants of health consistent with the socioecological model of health. In spite of their potential, methodological challenges related to study design compounded by a lack of tools for sample size calculation inhibit their development. We help address this gap by proposing the Multilevel Intervention Stepped Wedge Design (MLI-SWD), a hybrid experimental design which combines cluster-level (CL) randomization using a Stepped Wedge design (SWD) with independent individual-level (IL) randomization. The MLI-SWD is suitable for MLIs where the IL intervention has a low risk of interference between individuals in the same cluster, and it enables estimation of the component IL and CL treatment effects, their interaction, and the combined intervention effect. The MLI-SWD accommodates cross-sectional and cohort designs as well as both incomplete (clusters are not observed in every study period) and complete observation patterns. We adapt recent work using generalized estimating equations for SWD sample size calculation to the multilevel setting and provide an R package for power and sample size calculation. Furthermore, motivated by our experiences with the ongoing NC Works 4 Health study, we consider how to apply the MLI-SWD when individuals join clusters over the course of the study. This situation arises when unemployment MLIs include IL interventions that are delivered while the individual is unemployed. This extension requires carefully considering whether the study interventions will satisfy additional causal assumptions but could permit randomization in new settings.

Changing Health-Related Behaviors 3: Lessons from Implementation Science

Health-related behavior change refers to a body of behavior change strategies that aim to align people's behavior with advances in evidence-based knowledge and decision-making. However, human behavior is complex, and changing it often requires a combination of strategies to be effective. The challenge is in choosing the combination of strategies that will work best. Implementation science, the study of behavior change, has rapidly expanded in recent years and has pioneered work in providing more transparent and theory-based methods for choosing and evaluating behavior change strategies. There are several models and frameworks that underlie the science of implementation, the most recent and comprehensive of which include the Implementation of Change Model, the COM-B (capability, motivation, and behavior) Model, and the Theoretical Domains Framework, as well as the behavior change techniques (BCTs) taxonomy. These models and frameworks can be applied to help support the development and evaluation of behavior change interventions. In this chapter, we will review the latest advances and lessons learned from implementation science as it applies to health-related behavior change.

Futility of Cluster Designs at Individual Hospitals to Study Surgical Site Infections and Interventions Involving the Installation of Capital Equipment in Operating Rooms

Anesthesia workspaces are integral components in the chains of many intraoperative bacterial transmission events resulting in surgical site infections (SSI). Matched cohort designs can be used to compare SSI rates among operating rooms (ORs) with or without capital equipment purchases (e.g., new anesthesia machines). Patients receiving care in intervention ORs (i.e., with installed capital equipment) are matched with similar patients receiving care in ORs lacking the intervention. We evaluate statistical power of an alternative design for clinical trials in which, instead, SSI incidences are compared directly among ORs (i.e., the ORs form the clusters) at single hospitals (e.g., the 5 ORs with bactericidal lights vs. the 5 other ORs). Data used for parameter estimates were SSI for 24 categories of procedures among 338 hospitals in the State of California, 2015. Estimated statistical power was ≅8.4% for detecting a reduction in the incidence of SSI from 3.6% to 2.4% over 1 year with 5 intervention ORs and 5 control ORs. For ≅80% statistical power, >20 such hospitals would be needed to complete a study in 1 year. Matched paired cluster designs pair similar ORs (e.g., 2 cardiac ORs, 1 to intervention and 1 to control). With 5 pairs, statistical power would be even less than the estimated 8.4%. Cluster designs (i.e., analyses by OR) are not suitable for comparing SSI among ORs at single hospitals. Even though matched cohort designs are non-randomized and thus have lesser validity, matching patients by their risk factors for SSI is more practical.