Impact of risk of generalizability biases in adult obesity interventions: A meta-epidemiological review and meta-analysis

Are the Risk of Generalizability Biases Generalizable? A Meta-Epidemiological Study

Background Preliminary studies (e.g., pilot/feasibility studies) can result in misleading evidence that an intervention is ready to be evaluated in a large-scale trial when it is not. Risk of Generalizability Biases (RGBs, a set of external validity biases) represent study features that influence estimates of effectiveness, often inflating estimates in preliminary studies which are not replicated in larger-scale trials. While RGBs have been empirically established in interventions targeting obesity, the extent to which RGBs generalize to other health areas is unknown. Understanding the relevance of RGBs across health behavior intervention research can inform organized efforts to reduce their prevalence. Purpose The purpose of our study was to examine whether RGBs generalize outside of obesity-related interventions. Methods A systematic review identified health behavior interventions across four behaviors unrelated to obesity that follow a similar intervention development framework of preliminary studies informing larger-scale trials (i.e., tobacco use disorder, alcohol use disorder, interpersonal violence, and behaviors related to increased sexually transmitted infections). To be included, published interventions had to be tested in a preliminary study followed by testing in a larger trial (the two studies thus comprising a study pair). We extracted health-related outcomes and coded the presence/absence of RGBs. We used meta-regression models to estimate the impact of RGBs on the change in standardized mean difference (ΔSMD) between the preliminary study and larger trial. Results We identified sixty-nine study pairs, of which forty-seven were eligible for inclusion in the analysis (k = 156 effects), with RGBs identified for each behavior. For pairs where the RGB was present in the preliminary study but removed in the larger trial the treatment effect decreased by an average of ΔSMD=-0.38 (range - 0.69 to -0.21). This provides evidence of larger drop in effectiveness for studies containing RGBs relative to study pairs with no RGBs present (treatment effect decreased by an average of ΔSMD =-0.24, range - 0.19 to -0.27). Conclusion RGBs may be associated with higher effect estimates across diverse areas of health intervention research. These findings suggest commonalities shared across health behavior intervention fields may facilitate introduction of RGBs within preliminary studies, rather than RGBs being isolated to a single health behavior field.

Outcomes Associated With Translating Evidence-Based Research Into Practice: The Physical Activity for Lifelong Success Program

BACKGROUND

Few programs assess for outcomes once translated into practice. The Physical Activity for Lifelong Success program was developed as a center-based public health intervention and shown to improve walking speed and distance among older adults with type 2 diabetes. We adapted the program for community-based delivery by lay leaders to physically inactive older adults.

METHODS

We followed the Replicating Effective Programs framework to identify community stakeholders, adapt, implement, and evaluate fidelity of delivery in community settings, and plan for maintenance and evolution. Sixteen community sites enrolled 184 adults (mean age 73.5 y, 85% female, 93% White) in 21 workshops. Baseline and postworkshop measures assessed participants' health-related quality of life, physical function, and physical fitness. Data were analyzed using Fisher exact tests, Student t test, and paired linear regression with fixed effects.

RESULTS

Fidelity testing indicated leader training was sufficient to maintain key elements with delivery. Data from 122 participants showed improvements in chair stands (P < .001), arm curls (P < .001), 2-minute step test (P < .001), sit-and-reach (P = .001), 8-foot up-and-go (P < .001), and 10-m walk (P < .001).

CONCLUSIONS

Adaptation of Physical Activity for Lifelong Success for implementation by community organizations for physically inactive older adults demonstrates that fidelity and effectiveness can be maintained after program translation.

Re-evaluating the role of pilot trials in informing effect and sample size estimates for full-scale trials: a meta-epidemiological study

BACKGROUND

Some have argued that pilot trials have little value for informing the expected effect size of a subsequent large trial. This study aims to empirically evaluate the roles of pilot trials in informing the effect and sample size estimates of a full-scale trial.

METHODS

We conducted a search in PubMed on 19 February 2022, for all pilot trials published between 2005 and 2018 and their subsequent full-scale trials. We analysed the agreement in results by comparing the direction and magnitude of the effect size in the pilot trial and full-scale trial. Logistic regression was used to explore whether a significant pilot trial and other characteristics were associated with a significant full-scale trial.

RESULTS

A total of 248 pairs of pilot and full-scale trials were analysed. Full-scale trials with a significant pilot trial were 2.72 times more likely to find a significant result for the primary efficacy outcome than those with a non-significant pilot trial (95% CI 1.52 to 4.86, p=0.001). The association remained significant irrespective of changes made to the trial design. In 73% of the pairs, the pilot trial produced a larger point estimate than the subsequent full-scale trial, but 87% of pairs had a 95% CI estimated by the pilot trial that covered the full-scale trial point estimate. Full-scale trials with a sample size estimated using the SD from the pilot trial were less likely to yield a significant result (OR=0.26, 95% CI 0.10 to 0.65, p=0.004).

CONCLUSION

Pilot trials can provide strong signals on intervention efficacy. When determining the sample size for full-scale trials, using the CI bounds from the pilot trials instead of the point estimate may improve power estimation.

Pilot trials may improve the quality of full-scale trials: a meta-research study

OBJECTIVES

Evidence on the value of pilot trials for subsequent trial's quality is scarce. This study aims to determine if a pilot trial improves the quality of the full-scale trial.

STUDY DESIGN AND SETTING

We searched PubMed for pilot trials and their subsequent full-scale trials. The meta-analysis of the full-scale trials was used to identify other full-scale trials on the same research topic but without a pilot trial. Markers of trial quality included publication outcomes and Cochrane Risk of Bias (RoB) assessment.

RESULTS

Fifty-eight full-scale trials with a pilot trial and 151 full-scale trials without were identified from 47 meta-analyses. Trials with a pilot trial were published 0.9 years sooner (mean ± standard deviation: 1.7 ± 1.0 vs. 2.6 ± 2.0, P = 0.005) and in peer-reviewed journals with higher impact factors (60.9 ± 75.0 vs. 24.8 ± 50.3, P < 0.001). A pilot trial's presence was associated with lower risk of bias in full-scale trial random sequence generation (OR [95% CI]: 4.05 [1.27-12.91]), allocation concealment (2.89 [1.07-7.83]), and participants/researchers masking (4.31 [1.37-13.50]), but not outcome assessment masking (1.03 [0.49-2.18]), incomplete outcome data (1.27 [0.47-3.42]), and selective reporting (1.23 [0.44-3.46]).

CONCLUSION

Conducting a pilot trial may enhance the quality of the subsequent full-scale trial.

Scientists' perception of pilot study quality was influenced by statistical significance and study design

OBJECTIVES

Preliminary studies play a key role in developing large-scale interventions but may be held to higher or lower scientific standards during the peer review process because of their preliminary study status.

STUDY DESIGN AND SETTING

Abstracts from 5 published obesity prevention preliminary studies were systematically modified to generate 16 variations of each abstract. Variations differed by 4 factors: sample size (n = 20 vs. n = 150), statistical significance (P < 0.05 vs. P > 0.05), study design (single group vs. randomized 2 groups), and preliminary study status (presence/absence of pilot language). Using an online survey, behavioral scientists were provided with a randomly selected variation of each of the 5 abstracts and blinded to the existence of other variations. Respondents rated each abstract on aspects of study quality.

RESULTS

Behavioral scientists (n = 271, 79.7% female, median age 34 years) completed 1,355 abstract ratings. Preliminary study status was not associated with perceived study quality. Statistically significant effects were rated as more scientifically significant, rigorous, innovative, clearly written, warranted further testing, and had more meaningful results. Randomized designs were rated as more rigorous, innovative, and meaningful.

CONCLUSION

Findings suggest reviewers place a greater value on statistically significant findings and randomized control design and may overlook other important study characteristics.

Fund behavioral science like the frameworks we endorse: the case for increased funding of preliminary studies by the National Institutes of Health

Innovative, groundbreaking science relies upon preliminary studies (aka pilot, feasibility, proof-of-concept). In the behavioral sciences, almost every large-scale intervention is supported by a series of one or more rigorously conducted preliminary studies. The importance of preliminary studies was established by the National Institutes of Health (NIH) in 2014/2015 in two translational science frameworks (NIH Stage and ORBIT models). These frameworks outline the essential role preliminary studies play in developing the next generation of evidence-based behavioral prevention and treatment interventions. Data produced from preliminary studies are essential to secure funding from the NIH's most widely used grant mechanism for large-scale clinical trials, namely the R01. Yet, despite their unquestionable importance, the resources available for behavioral scientists to conduct rigorous preliminary studies are limited. In this commentary, we discuss ways the existing funding structure at the NIH, despite its clear reliance upon high-quality preliminary studies, inadvertently discourages and disincentivizes their pursuit by systematically underfunding them. We outline how multiple complementary and pragmatic steps via a small reinvestment of funds from larger trials could result in a large increase in funding for smaller preliminary studies. We make the case such a reinvestment has the potential to increase innovative science, increase the number of investigators currently funded, and would yield lasting benefits for behavioral science and scientists alike.

Effects of female obesity on conception, pregnancy and the health of offspring

As we all know, female obesity has become a global epidemic, which is usually accompanied with endocrine and metabolic disorders. Obese women are more likely to experience reproductive problems, including infertility, embryonic developmental defects and abnormality in offspring. Female obesity is a complex multifactorial condition, where there are many mechanisms involved in the effects of overweight and obesity on the development of these reproductive disorders. The insulin resistance, hyperinsulinaemia and hyperandrogenism, lipotoxicity and inflammation are important mechanisms. However, the precise mechanism concerning their correlation is still unclear. Fortunately, weight loss methods have been found to reverse the effects of maternal obesity on the fertility, fetus and offspring.