The reverse propensity score to detect selection bias and correct for baseline imbalances

Are Most of the Published Clinical Trial Results in Restorative Dentistry Invalid? An Empirical Investigation

BACKGROUND

To establish the number of invalid clinical trial reports in restorative dentistry, due to lack of effective randomisation and/or inadequate sample size and whether this number changed, during the 1990-2019 period.

METHODS

Databases were searched up to 14 July 2019 without limitations regarding publication language. A Journal hand search and reference check were conducted for trial reports. Selection criteria were: reporting on a prospective, controlled clinical trial; relevance to placing direct tooth restorations in human vital teeth; direct comparison between restorative materials concerning tooth restoration longevity; trial report published from 1990. Randomisation reported (Yes/No) and treatment group sample size ≥ 200 were applied as criteria, using the deductive falsification approach for trial report appraisal.

RESULTS

683 trial reports were appraised. 660 lacked effective randomisation. Of the remaining 23 reports, only 2 included a sample size of more than 200 restored teeth (mean number per treatment group 87; Standard deviation = 108.51). 92.5% of all treatment groups had a sample size of < 200. Randomisation reporting increased and sample size remained essentially unchanged between 1990 and 2019.

CONCLUSION

Most of the published clinical trial results in restorative dentistry were judged invalid, due to lack of effective randomisation and adequate sample size. These results are in line with previous findings. Evidence-based recommendations on how to improve trial methodology are available in the dental/medical literature.

Statistical properties of minimal sufficient balance and minimization as methods for controlling baseline covariate imbalance at the design stage of sequential clinical trials

When the number of baseline covariates whose imbalance needs to be controlled in a sequential randomized controlled trial is large, minimization is the most commonly used method for randomizing treatment assignments. The lack of allocation randomness associated with the minimization method has been the source of controversy, and the need to reduce even minor imbalances inherent in the minimization method has been challenged. The minimal sufficient balance (MSB) method is an alternative to the minimization method. It prevents serious imbalance from a large number of covariates while maintaining a high level of allocation randomness. In this study, the two treatment allocation methods are compared with regards to the effectiveness of balancing covariates across treatment arms and allocation randomness in equal allocation clinical trials. The MSB method proves to be equal or superior in both respects. In addition, type I error rate is preserved in analyses for both balancing methods, when using a binary endpoint.

Effect of a Pedometer-Based, 24-Week Walking Intervention on Depression and Acculturative Stress among Migrant Women Workers

Little research has examined exercise-based interventions meant to alleviate depressive symptoms among Korean-Chinese migrant women workers living in Korea. Thus, this study evaluated the effectiveness of a 24-week walking program on reducing depressive symptoms and acculturative stress levels in this population. This quasi-experimental sequential walking program was conducted with 132 Korean-Chinese women over a period of 24 weeks. Participants were divided into either a standard treatment group (n = 69) or enhanced treatment group (n = 63). All participants were instructed to walk using a pedometer. The enhanced treatment group also received mobile text messages designed to encourage walking adherence and improve acculturation. Participants were asked to complete two structured questionnaires, the Centre for Epidemiologic Studies Depression Scale and the Acculturative Stress Scale, to evaluate the intervention programs. At the end of the program, both groups exhibited decreased depression scores, but the decrease in the enhanced treatment group was more significant both at weeks 12 and 24. Acculturative stress was also found to have decreased at weeks 12 and 24. Our findings show the walking program reduced the depressive symptoms and acculturative stress levels among the Korean-Chinese women in this study. Further studies will be needed to analyze the relationship between walking step count and mental health considering exercise intensity.

Interdisciplinary Weight Loss and Lifestyle Intervention for Obstructive Sleep Apnoea in Adults: Rationale, Design and Methodology of the INTERAPNEA Study

Obesity is a major risk factor for obstructive sleep apnoea (OSA), the most common sleep-disordered breathing related to neurocognitive and metabolic syndromes, type II diabetes, and cardiovascular diseases. Although strongly recommended for this condition, there are no studies on the effectiveness of an interdisciplinary weight loss and lifestyle intervention including nutrition, exercise, sleep hygiene, and smoking and alcohol cessation. INTERAPNEA is a randomised controlled trial with a two-arm parallel design aimed at determining the effects of an interdisciplinary tailored weight loss and lifestyle intervention on OSA outcomes. The study will include 84 males aged 18–65 with a body mass index of ≥25 kg/m² and severe to moderate OSA randomly assigned to usual care (i.e., continuous positive airway pressure), or interdisciplinary weight loss and lifestyle intervention combined with usual care. Outcomes will be measured at baseline, intervention end-point, and six-month post-intervention, including apnoea-hypopnoea index (primary outcome), other neurophysical and cardiorespiratory polysomnographic outcomes, sleep quality, daily functioning and mood, body weight and composition, physical fitness, blood biomarkers, health-related quality of life, and cost-effectiveness. INTERAPNEA may serve to establish a cost-effective treatment not only for the improvement of OSA and its vast and severe comorbidities, but also for a potential remission of this condition.

Re-randomization tests in clinical trials

As randomization methods use more information in more complex ways to assign patients to treatments, analysis of the resulting data becomes challenging. The treatment assignment vector and outcome vector become correlated whenever randomization probabilities depend on data correlated with outcomes. One straightforward analysis method is a re-randomization test that fixes outcome data and creates a reference distribution for the test statistic by repeatedly re-randomizing according to the same randomization method used in the trial. This article reviews re-randomization tests, especially in nonstandard settings like covariate-adaptive and response-adaptive randomization. We show that re-randomization tests provide valid inference in a wide range of settings. Nonetheless, there are simple examples demonstrating limitations.

Methodological Aspects in Studies Based on Clinical Routine Data

Randomized controlled clinical trials are regarded as the gold standard for comparing different clinical interventions, but generally their conduct is operationally cumbersome, time-consuming, and expensive. Studies and investigations based on clinical routine data on the contrary utilize existing data acquired under real-life conditions and are increasingly popular among practitioners. In this paper, methodological aspects of studies based on clinical routine data are discussed. Important limitations and considerations as well as unique strengths of these types of studies are indicated and exemplarily demonstrated in a recent real-case study based on clinical routine data. In addition two simulation studies reveal the impact of bias in studies based on clinical routine data on the type I error rate and false decision rate in favor of the inferior intervention. It is concluded that correctly analyzing clinical routine data yields a valuable addition to clinical research; however, as a result of a lack of statistical foundation, internal validity, and comparability, generalizing results and inferring properties derived from clinical routine data to all patients of interest has to be considered with extreme caution.

FUNDING

Grünenthal GmbH.

Assessing the impact of selection bias on test decisions in trials with a time-to-event outcome

If past treatment assignments are unmasked, selection bias may arise even in randomized controlled trials. The impact of such bias can be measured by considering the type I error probability. In case of a normally distributed outcome, there already exists a model accounting for selection bias that permits calculating the corresponding type I error probabilities. To model selection bias for trials with a time‐to‐event outcome, we introduce a new biasing policy for exponentially distributed data. Using this biasing policy, we derive an exact formula to compute type I error probabilities whenever an F‐test is performed and no observations are censored. Two exemplary settings, with and without random censoring, are considered in order to illustrate how our results can be applied to compare distinct randomization procedures with respect to their performance in the presence of selection bias. © 2017 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

Two-year Effects of a Classroom-based Smoking Prevention and Cessation Intervention Program

BACKGROUND

Project EX is a classroom-based smoking prevention and cessation intervention program that has been well evaluated and designed for assessing the prevention and cessation effects among adolescents in Spain. However, its long-term efficacy is still unknown. This study deals with the outcomes of a 2-year follow-up evaluation of Project EX.

METHODS

The intervention was tested using a clustered randomized controlled trial involving 1,546 Spanish students from 3 program schools and 3 control schools. At the end of the 2-year follow-up period, 722 subjects had completed the questionnaires (266 in the control condition and 456 in the program condition) administered to them.

RESULTS

Compared to the control condition, the program condition revealed a greater reduction in nicotine dependence (p = 0.04), smoking intention (p = 0.02), and in the number of cigarettes smoked during the previous month (p = 0.03). The CO monitor repeated assessments revealed a significant decrease of ppm levels in the program group (p < 0.001). Intent-to-treat quit rates were 14.28 and 0%, respectively, for the program and control conditions.

CONCLUSIONS

This study provides evidence about the long-term effectiveness of the Project EX classroom-based program for smoking prevention and cessation among adolescents in Spain.

One-year effects of Project EX: A smoking intervention pilot program with Spanish adolescents

Adolescent smoking is a major public health problem, which has led to the development of cessation programs such as Project EX. However, there is no evidence for the long-term efficacy of cessation among Spanish adolescents. This study provides a 1-year follow-up evaluation of the Project EX tobacco use cessation program among 211 smokers. The intent-to-treat 30-day smoking quit rate for the program group was 7.81 percent ( p = .04), whereas no smokers quit in the control group ( p = .02). The intervention had a significant influence on future smoking expectation, intention, motivation to quit, and overall level of 30-day smoking. Long-term outcomes of the Project EX clinic-based program are promising for adolescent smokers in Spain.

Recommendations for the Design and Analysis of Treatment Trials for Alcohol Use Disorders

BACKGROUND

Over the past 60 years, the view that "alcoholism" is a disease for which the only acceptable goal of treatment is abstinence has given way to the recognition that alcohol use disorders (AUDs) occur on a continuum of severity, for which a variety of treatment options are appropriate. However, because the available treatments for AUDs are not effective for everyone, more research is needed to develop novel and more efficacious treatments to address the range of AUD severity in diverse populations. Here we offer recommendations for the design and analysis of alcohol treatment trials, with a specific focus on the careful conduct of randomized clinical trials of medications and nonpharmacological interventions for AUDs.

METHODS

This paper provides a narrative review of the quality of published clinical trials and recommendations for the optimal design and analysis of treatment trials for AUDs.

RESULTS

Despite considerable improvements in the design of alcohol clinical trials over the past 2 decades, many studies of AUD treatments have used faulty design features and statistical methods that are known to produce biased estimates of treatment efficacy.

CONCLUSIONS

The published statistical and methodological literatures provide clear guidance on methods to improve clinical trial design and analysis. Consistent use of state-of-the-art design features and analytic approaches will enhance the internal and external validity of treatment trials for AUDs across the spectrum of severity. The ultimate result of this attention to methodological rigor is that better treatment options will be identified for patients with an AUD.

One-Year Effects of Project EX in Spain: A Classroom-Based Smoking Prevention and Cessation Intervention Program

Background

Tobacco use prevalence rates are high among Spanish adolescents. Programming to counteract tobacco use is needed.

Methods and Findings

The current study provides a one-year follow-up outcome evaluation of Project EX, an eight-session classroom-based curriculum. The intervention was tested using a randomized controlled trial with 1,546 Spanish students, involving three program and three control schools. Compared to the control condition, the program condition revealed a greater reduction in nicotine dependence (p < .05) and CO ppm levels (p < .001), and lower consumption of cigarettes at last month (p = .03).

Conclusions

Long-term outcomes of the Project EX classroom-based program are promising for adolescent prevention and possibly cessation in Spain.

Pilot clinic study of Project EX for smoking cessation with Spanish adolescents

INTRODUCTION

Despite efforts to prevent smoking, the prevalence of smoking in Spanish adolescents remains high. So far, there are no evidence-based smoking cessation programs for adolescents in Spain.

METHODS

This study describes the evaluation of Project EX, an eight-session school-based clinic smoking cessation program, with Spanish cigarette smokers 13-19 years of age, from 9 schools (four program condition schools and five control condition schools). A group-randomized controlled trial was used. There were 211 smokers at baseline (112 program group, and 99 control group). Evaluation involved an immediate pretest and posttest survey (administered five-weeks later) and six-month follow-up (after the immediate posttest).

RESULTS

At immediate posttest, Project EX significantly reduced future nicotine dependence scores (mFTQ; p<.001), and increased intention to quit smoking (p<.001), and led to a higher previous day (prior to assessment) quit rate (p<.03). At the six-month follow-up, the percentage of quitters in the program group was 14.28%, whereas no smokers quit smoking in the control group (p<.04), and Project EX had a significant influence on future smoking expectation (p=.006) and overall level of 30-day smoking.

CONCLUSIONS

Results for the Project EX school-based clinic are promising for adolescent smokers in Spain, although difficulties in recruitment and high attrition are of concern. Findings and limitations are discussed and suggestions for future research are suggested.

Immediate Effects of Project EX in Spain: A Classroom-Based Smoking Prevention and Cessation Intervention Program

Tobacco use is of high prevalence among Spanish adolescents. Programming to counteract tobacco use has been needed. There is a lack of knowledge on the efficacy of teen tobacco use cessation programming. The current study provides an immediate outcome evaluation of the Project EX tobacco use prevention and cessation program among Spanish adolescents. An eight-session, classroom-based curriculum was translated from English to Spanish and adapted to the Spanish culture. Next, it was tested using a randomized controlled trial with 1,546 Spanish students, involving three program and three control high schools. Participants at the program group provided moderately favorable process ratings of the program. Compared to the standard care control condition, the program condition revealed a greater reduction in smoking intentions and CO ppm levels.

Post-traumatic growth, stressful life events, and relationships with substance use behaviors among alternative high school students: a prospective study

A highly stressful life event (SLE) can elicit positive psychosocial growth, referred to as post-traumatic growth (PTG) among youth. We examined PTG and the number of SLEs for their influence on substance use behaviours among a sample of older, diverse alternative high school students participating in a drug prevention programme (n=564; mean age=16.8; 49% female; 65% Hispanic). Surveys assessed PTG, SLEs and substance use behaviours at the two-year follow-up. Multilevel regression models were run to examine the effect of PTG and the number of SLEs on frequency of substance use at the two-year follow-up, controlling for baseline substance use, sociodemographic variables, peer substance use, attrition propensity and treatment group. Greater PTG scores were associated with lower frequencies of alcohol use, getting drunk on alcohol, binge drinking, marijuana use and less substance abuse at the two-year follow-up, but not associated with cigarette or hard drug use. Also, PTG did not moderate the relationship between cumulative number of SLEs and substance use behaviours, rather PTG appears to be protective against negative effects of a single, life-altering SLE. Fostering PTG from a particularly poignant SLE may be useful for prevention programmes targeting alcohol, marijuana and substance abuse behaviours among high-risk youth.

Accuracy of the Berger-Exner test for detecting third-order selection bias in randomised controlled trials: a simulation-based investigation

BACKGROUND

Randomised controlled trials (RCT) are highly influential upon medical decisions. Thus RCTs must not distort the truth. One threat to internal trial validity is the correct prediction of future allocations (selection bias). The Berger-Exner test detects such bias but has not been widely utilized in practice. One reason for this non-utilisation may be a lack of information regarding its test accuracy. The objective of this study is to assess the accuracy of the Berger-Exner test on the basis of relevant simulations for RCTs with dichotomous outcomes.

METHODS

Simulated RCTs with various parameter settings were generated, using R software, and subjected to bias-free and selection bias scenarios. The effect size inflation due to bias was quantified. The test was applied in both scenarios and the pooled sensitivity and specificity, with 95% confidence intervals for alpha levels of 1%, 5%, and 20%, were computed. Summary ROC curves were generated and the relationships of parameters with test accuracy were explored.

RESULTS

An effect size inflation of 71% - 99% was established. Test sensitivity was 1.00 (95% CI: 0.99 - 1.00) for alpha level 1%, 5%, and 20%; test specificity was 0.94 (95% CI: 0.93 - 0.96); 0.82 (95% CI: 0.80 - 0.84), and 0.56 (95% CI: 0.54 - 0.58) for alpha 1%, 5%, and 20%, respectively. Test accuracy was best with the maximal procedure used with a maximum tolerated imbalance (MTI) = 2 as the randomisation method at alpha 1%.

CONCLUSIONS

The results of this simulation study suggest that the Berger-Exner test is generally accurate for identifying third-order selection bias.

Individual music therapy for mental health care clients with low therapy motivation: multicentre randomised controlled trial

BACKGROUND

Music therapy (MT) has been shown to be efficacious for mental health care clients with various disorders such as schizophrenia, depression and substance abuse. Referral to MT in clinical practice is often based on other factors than diagnosis. We aimed to examine the effectiveness of resource-oriented MT for mental health care clients with low motivation for other therapies.

METHOD

This was a pragmatic parallel trial. In specialised centres in Norway, Austria and Australia, 144 adults with non-organic mental disorders and low therapy motivation were randomised to 3 months of biweekly individual, resource-oriented MT plus treatment as usual (TAU) or TAU alone. TAU was typically intensive (71% were inpatients) and included the best combination of therapies available for each participant, excluding MT. Blinded assessments of the Scale for the Assessment of Negative Symptoms (SANS) and 15 secondary outcomes were collected before randomisation and after 1, 3 and 9 months. Changes were analysed on an intention-to-treat basis using generalised estimating equations in longitudinal linear models, controlling for diagnosis, site and time point.

RESULTS

MT was superior to TAU for total negative symptoms (SANS, d = 0.54, p < 0.001) as well as functioning, clinical global impressions, social avoidance through music, and vitality (all p < 0.01).

CONCLUSION

Individual MT as conducted in routine practice is an effective addition to usual care for mental health care clients with low motivation.

Immediate and six-month effects of Project EX Russia: a smoking cessation intervention pilot program

This study evaluates the performance of the Project EX tobacco use cessation program in Russian summer recreational camps. An eight-session clinic-based tobacco use cessation program for adolescents was tested during the summer of 2011 in an experimental pilot trial that involved different youth that rotated through camps. Conditions were nested within camps. Two rotations of unique subject groups of smokers (program and standard care control) through each of five camps provided the means of controlling for campsite by condition. Assignment of condition by rotation was random (by a flip of a coin), achieving reasonable baseline comparability (total n=164 smokers at baseline, 76 program group, 88 standard care control group). Evaluation involved an immediate pretest and posttest and a six-month telephone follow-up. At immediate posttest, Project EX was moderately well-received, significantly reduced future smoking expectation (46% reduction in EX program condition versus 8% in control, p<.0001), decreased intention to not quit smoking (-5.2% in EX versus +1.4% in control, p<.05), and increased motivation to quit smoking (0.72 versus -0.04, p<.0001). At the six-month follow-up, program subjects reported a higher intent-to-treat quit rate during the last 30days (7.5% versus 0.1%, p<.05). For the subjects who remained monthly smokers at the six-month follow-up, Project EX reduced subjects' level of nicotine dependence (-0.53 versus +0.15, p<.001). The results were quite promising for this program, which included motivation enhancement, coping skill, and alternative medicine material. However, further research on teen tobacco use cessation programming in Russia with larger sample sizes, involving other locations of the country, and with stronger research designs is needed.

One-year outcomes of a drug abuse prevention program for older teens and emerging adults: evaluating a motivational interviewing booster component

OBJECTIVE

The present study tested the efficacy of motivational interviewing-based booster sessions for Project Toward No Drug Abuse (TND), a 12-session school-based curriculum targeting youth at risk for drug abuse. In addition, generalization of effects to risky sexual behavior was assessed. The 1-year outcomes evaluation of the project is presented.

METHOD

A total of 24 schools were randomized to one of three conditions: standard care control (SCC), TND classroom program only (TND-only), and TND plus motivational interviewing booster (TND + MI). A total of 1186 participants completed baseline and 1-year follow-up surveys. Following the classroom program, youth in the TND + MI condition received up to 3 sessions of MI in person or by telephone. Effects were examined on 30-day cigarette, alcohol, marijuana, and hard drug use, as well as measures of risky sexual behavior (number of sex partners, condom use, having sex while using drugs or alcohol).

RESULTS

Collapsed across the 2 program conditions, results showed significant reductions in alcohol use, hard drug use, and cigarette smoking relative to controls. These effects held for an overall substance use index. The MI booster component failed to achieve significant incremental effects above and beyond the TND classroom program. No effects were found on risky sexual behavior.

CONCLUSIONS

While the program effects of previous studies were replicated, the study failed to demonstrate that an adequately implemented MI booster was of incremental value at 1-year follow-up.

Sensitivity designs for preventing bias replication in randomized clinical trials

It is common, after a trial is completed, to employ sensitivity analyses to test the extent to which the results depend on various assumptions or conventions. There is a comparable benefit to employing sensitivity designs when planning a trial, so that features that cannot be varied at the analysis stage can instead be varied (e.g., across centres of a multi-centre trial) during the design stage. Design features amenable to such variation include: (1) the specific randomization methods, (2) the duration of follow-up and (3) the use or non-use of a surrogate endpoint as a replacement for a clinical endpoint. Generally, all centres in a given trial, and all trials in a given program, will employ identical protocols. This means that all will be vulnerable to the same types of biases, meaning that a single bias can by itself render all results unreliable. But by varying the randomization techniques, duration and primary endpoint, one can vary also the biases to which the site-specific results are vulnerable. This means that, if a significant result is found, then one can state that either the treatment worked or there were numerous biases (not just one) at play. This of course makes the attribution of the results to the treatments much more plausible and makes the findings much more robust to violations of assumptions.

Employing complier average causal effect analytic methods to examine effects of randomized encouragement trials

AIMS

This article details the application of Complier Average Causal Effect (CACE) analysis to the examination of youth outcomes from adaptive substance use prevention trials.

METHODS

CACE analysis is illustrated using youth-reports of tobacco-use from ages 11 to 22, from the Adolescent Transitions Program, a family-focused randomized encouragement trial designed for delivery in the school setting.

RESULTS

Female gender and early peer deviance predicted family engagement with active intervention components. Further, long-term reductions in youth tobacco use from age 11 to age 22 were found for families that engaged with treatment.

CONCLUSIONS

CACE modeling techniques enable researchers to examine factors that predict engagement with core intervention components and to examine intervention effects specifically for youth who engaged with those components.

On improving research methodology in clinical trials

Research plays a vital role within biomedicine. Scientifically appropriate research provides a basis for appropriate medical decisions; conversely, inappropriate research may lead to flawed ;best medical practices' which, when followed, contribute to avoidable morbidity and mortality. Although an all-encompassing definition of ;appropriate medical research' is beyond the scope of this article, the concept clearly entails (among other things) that research methods be continually revised and updated as better methods become available. Despite the advent of evidence-based medicine, many research methods have become ;standard' even though there are legitimate scientific reasons to question the conclusions reached by such methods. We first illustrate prominent examples of inappropriate (yet regimented) research methods that are in widespread use. Second, as a way to improve the situation, we suggest a model of research that relies on standardized statistical analyses that individual researchers must consider as a default, but are free to challenge when they can marshal sufficient scientific evidence to demonstrate that the challenge is warranted. Third, we characterize the current system as analogous to ;unnatural selection' in the biological world and argue that our proposed model of research will enable ;natural' to replace ;unnatural' selection in the choice of research methodologies. Given the pervasiveness of inappropriate research methods, we believe that there are strong scientific and ethical reasons to create such a system, that, if properly designed, will both facilitate creativity and ensure methodological rigor while protecting the public at large from the threats posed by poor medical treatment decisions resulting from flawed research methodology.