Risk of bias in exercise science: A systematic review of 340 studies

Risk of bias can contribute to irreproducible science and mislead decision making. Analyses of smaller subsections of the exercise science literature suggest many exercise science studies have unclear or high risk of bias. The current review (osf.io/jznv8) assesses whether this unclear or high risk of bias is more widespread in the exercise science literature and whether this bias has decreased since the publication of the 1996 Consolidated Standards of Reporting Trials (CONSORT) guidelines. We report significant reductions in selection, performance, detection, and reporting biases in 2020 compared with 1995 in the 340 of 5,451 studies assessed using the Cochrane Risk of Bias tool. Despite these improvements, most 2020 studies still had unclear or high risks of bias. These results underscore the need for methodological vigilance, adherence to reporting standards, and education on experimental bias. Factors contributing to these improvements, such advancements in education and journal requirements, remain uncertain.

Risk of bias and reporting practices in studies comparing VO2max responses to sprint interval vs. continuous training: A systematic review and meta-analysis

BACKGROUND

It remains unclear whether studies comparing maximal oxygen uptake (VO_2max) response to sprint interval training (SIT) vs. moderate-intensity continuous training (MICT) are associated with a high risk of bias and poor reporting quality. The purpose of this study was to evaluate the risk of bias and quality of reporting in studies comparing changes in VO_2max between SIT and MICT.

METHODS

We conducted a comprehensive literature search of 4 major databases: AMED, CINAHL, EMBASE, and MEDLINE. Studies were excluded if participants were not healthy adult humans or if training protocols were unsupervised, lasted less than 2 weeks, or utilized mixed exercise modalities. We used the Cochrane Collaboration tool and the CONSORT checklist for non-pharmacological trials to evaluate the risk of bias and reporting quality, respectively.

RESULTS

Twenty-eight studies with 30 comparisons (3 studies included 2 SIT groups) were included in our meta-analysis (n = 360 SIT participants: body mass index (BMI) = 25.9 ± 3.7 kg/m², baseline VO_2max = 37.9 ± 8.0 mL/kg/min; n = 359 MICT participants: BMI = 25.5 ± 3.8 kg/m², baseline VO_2max = 38.3 ± 8.0 mL/kg/min; all mean ± SD). All studies had an unclear risk of bias and poor reporting quality.

CONCLUSION

Although we observed a lack of superiority between SIT and MICT for improving VO_2max (weighted Hedge's g = -0.004, 95% confidence interval (95%CI): -0.08 to 0.07), the overall unclear risk of bias calls the validity of this conclusion into question. Future studies using robust study designs are needed to interrogate the possibility that SIT and MICT result in similar changes in VO_2max.

Machine-Learning Rationalization and Prediction of Solid-State Synthesis Conditions

There currently exist no quantitative methods to determine the appropriate conditions for solid-state synthesis. This not only hinders the experimental realization of novel materials but also complicates the interpretation and understanding of solid-state reaction mechanisms. Here, we demonstrate a machine-learning approach that predicts synthesis conditions using large solid-state synthesis data sets text-mined from scientific journal articles. Using feature importance ranking analysis, we discovered that optimal heating temperatures have strong correlations with the stability of precursor materials quantified using melting points and formation energies (ΔG _f , ΔH _f ). In contrast, features derived from the thermodynamics of synthesis-related reactions did not directly correlate to the chosen heating temperatures. This correlation between optimal solid-state heating temperature and precursor stability extends Tamman's rule from intermetallics to oxide systems, suggesting the importance of reaction kinetics in determining synthesis conditions. Heating times are shown to be strongly correlated with the chosen experimental procedures and instrument setups, which may be indicative of human bias in the data set. Using these predictive features, we constructed machine-learning models with good performance and general applicability to predict the conditions required to synthesize diverse chemical systems.

Risk of selection bias assessment in the NINDS rt-PA stroke study

OBJECTIVES

The NINDS rt-PA Stroke Study is frequently cited in support of alteplase for acute ischemic stroke within 3 h of symptom onset. Multiple post-hoc reanalyses of this trial have been published to adjust for a baseline imbalance in stroke severity. We performed a risk of selection bias assessment and reanalyzed trial data to determine if the etiology of this baseline imbalance was more likely due to random chance or randomization errors.

METHODS

A risk of selection bias assessment was conducted using signaling questions from the Cochrane Risk of Bias 2 (ROB 2) tool. Four sensitivity analyses were conducted on the trial data based on the randomization process: assessment of imbalances in allocation in unique strata; adherence to a pre-specified restriction on randomization between time strata at each randomization center; assessment of differences in baseline computed tomography (CT) results in unique strata; and comparison of baseline characteristics between allocation groups within each time strata. A multivariable logistic regression model was used to compare reported treatment effects with revised treatment effects after adjustment of baseline imbalances identified in the sensitivity analyses.

RESULTS

Based on criteria from the ROB 2 tool, the risk of bias arising from the randomization process was high. Sensitivity analyses found 11 of 16 unique strata deviated from the expected 1:1 allocation ratio. Three randomization centers violated an apriori rule regarding a maximum difference in allocation between the time strata. Three unique strata had imbalances in baseline CT results that prognostically favored alteplase. Four imbalances in baseline characteristics were identified in the 91-180-min time stratum that all prognostically favored alteplase and were consistent with a larger alteplase treatment effect size compared to the 0-90-min time stratum. After adjustments for baseline imbalances, all reported treatment effects were reduced. Three out of seven originally positive reported results were revised to non-significant.

CONCLUSION

This risk of selection bias assessment revealed a high risk of selection bias in the NINDS rt-PA Stroke Study. Sensitivity analyses conducted based on the randomization process supported this assessment. Baseline imbalances in the trial were more likely due to randomization errors than random chance. Adjusted analyses accounting for baseline imbalances revealed a reduction in reported treatment effects supporting the presence of selection bias in the trial. Treatment decisions and guideline recommendations based on the original treatment effect reported in the NINDS rt-PA Stroke Study should be done cautiously.

Marginalization in the Medical Encounter: Ostomy Patients Experience of Perceived Stigmatizing Sentiments from Medical Clinicians

Introduction

Ostomy¹ stigma negatively impacts the health of people with an ostomy and contributes to a lower quality of life and health outcomes.

Objective

To assess whether participants experience perceived stigmatizing sentiments (SS) from medical clinicians at the time of their ostomy procedure.

Methods

Using a nonprobability sample of 312 persons with an ostomy, we conducted a retrospective descriptive study. We measured SS as patients’ self-reports of verbal and non-verbal communication from clinicians that were perceived to be negative and may contribute to ostomy stigma. We used thematic analyses to analyze open-ended written comments.

Results

Findings indicate that ostomy patients experience stigmatizing sentiments from their medical clinician before and after surgery. Sixteen percent of patients reported a SS, such as clinicians stating feelings of disgust, showing visible signs of disgust, or treating patients negatively regarding the ostomy.

Conclusion

The perceived treatment that this patient cohort experienced in healthcare likely contributes to ostomy stigmatization and may impact ostomy patients’ psychosocial adjustment. Future research should examine the specific consequences of perceived stigmatizing sentiments from medical clinicians.

Review article: the effectiveness of group and self-help hypnotherapy for irritable bowel syndrome and the implications for improving patients' choice and access to treatment

BACKGROUND

Individual hypnotherapy (IH) is a recognised treatment for irritable bowel syndrome (IBS). However, it is not widely available to patients due to its resource-intensive nature, lack of adequately trained therapists, and scepticism about hypnosis. Non-individualised hypnotherapy approaches, such as group and self-help hypnotherapy, could maximise existing therapist resources by treating more patients at the same time, thus widening patient access to treatment without incurring additional expenditure.

AIMS

To investigate the research literature for non-individualised approaches to hypnotherapy for IBS and to determine their effectiveness for reducing symptom severity and/or providing adequate relief.

METHODS

A literature review of published peer-reviewed studies was conducted. Quantitative research was selected to determine the effectiveness of the interventions.

RESULTS

Ten studies were eligible for inclusion. Three delivered group hypnotherapy, three integrated hypnosis within a group concept, and four utilised a self-help home hypnotherapy treatment using audio recordings. Both group hypnotherapy for adults and the self-help home hypnotherapy treatment for children were effective interventions that may be non-inferior to IH for patients with mild-to-moderate symptoms. Treatment benefits were long-lasting. The evidence for the integrative group concept and home treatment for adults was less compelling.

CONCLUSIONS

Group hypnotherapy for adults, and self-help hypnotherapy for children, may be cost-effective treatments that can widen access for patients with milder IBS in primary care settings. Further research is needed to determine the effectiveness of group hypnotherapy for patients with severe, refractory IBS.

Mechanisms and direction of allocation bias in randomised clinical trials

BACKGROUND

Selective allocation of patients into the compared groups of a randomised trial may cause allocation bias, but the mechanisms behind the bias and its directionality are incompletely understood. We therefore analysed the mechanisms and directionality of allocation bias in randomised clinical trials.

METHODS

Two systematic reviews and a theoretical analysis. We conducted one systematic review of empirical studies of motives/methods for deciphering patient allocation sequences; and another review of methods publications commenting on allocation bias. We theoretically analysed the mechanisms of allocation bias and hypothesised which main factors predicts its direction.

RESULTS

Three empirical studies addressed motives/methods for deciphering allocation sequences. Main motives included ensuring best care for patients and ensuring best outcome for the trial. Main methods included various manipulations with randomisation envelopes. Out of 57 methods publications 11 (19 %) mentioned explicitly that allocation bias can go in either direction. We hypothesised that the direction of allocation bias is mainly decided by the interaction between the patient allocators' motives and treatment preference.

CONCLUSION

Inadequate allocation concealment may exaggerate treatment effects in some trials while underestimate effects in others. Our hypothesis provides a theoretical overview of the main factors responsible for the direction of allocation bias.

Participation in Universal Prevention Programs

We analyze family decisions to participate in community-based universal substance-abuse prevention programs through the framework of expected utility theory. Family functioning, which has been shown to be a good indicator of child risk for substance abuse, provides a useful reference point for family decision making. Our results show that well-functioning families (with children at low risk for substance use) should have the lowest incentive to participate, but that high-risk families may also opt out of prevention programs. For programs that are most effective for high-risk youth, this could be a problem. Using data from the Strengthening Families Program and the Washington Healthy Youth Survey, we empirically test the implications of our model and find that at least for one measure of family functioning those families with children most likely to be at risk for substance use are opting out of the program.

Influence of selection bias on the test decision. A simulation study

BACKGROUND

Selection bias arises in clinical trials by reason of selective assignment of patients to treatment groups. Even in randomized clinical trials with allocation concealment this phenomenon can occur if future assignments can be predicted due to knowledge of former allocations.

OBJECTIVES

Considering unmasked randomized clinical trials with allocation concealment the impact of selection bias on type I error rate under permuted block randomization is investigated. We aimed to extend the existing research into this topic by including practical assumptions concerning misclassification of patient characteristics to get an estimate of type I error close to clinical routine. To establish an upper bound for the type I error rate different biasing strategies of the investigator are compared first. In addition, the aspect of patient availability is considered.

METHODS

To evaluate the influence of selection bias on type I error rate under several practical situations, different block sizes, selection effects, biasing strategies and success rates of patient classification were simulated using SAS.

RESULTS

Type I error rate exceeds 5 percent significance level; it reaches values up to 21 percent. More cautious biasing strategies and misclassification of patient characteristics may diminish but cannot eliminate selection bias. The number of screened patients is about three times larger than the needed number for the trial.

CONCLUSIONS

Even in unmasked randomized clinical trials using permuted block randomization with allocation concealment the influence of selection bias must not be disregarded evaluating the test decision. It should be incorporated when designing and reporting a clinical trial.

Self-medication of migraine and tension-type headache: summary of the evidence-based recommendations of the Deutsche Migräne und Kopfschmerzgesellschaft (DMKG), the Deutsche Gesellschaft für Neurologie (DGN), the Österreichische Kopfschmerzgesellschaft (ÖKSG) and the Schweizerische Kopfwehgesellschaft (SKG)

The current evidence-based guideline on self-medication in migraine and tension-type headache of the German, Austrian and Swiss headache societies and the German Society of Neurology is addressed to physicians engaged in primary care as well as pharmacists and patients. The guideline is especially concerned with the description of the methodology used, the selection process of the literature used and which evidence the recommendations are based upon. The following recommendations about self-medication in migraine attacks can be made: The efficacy of the fixed-dose combination of acetaminophen, acetylsalicylic acid and caffeine and the monotherapies with ibuprofen or naratriptan or acetaminophen or phenazone are scientifically proven and recommended as first-line therapy. None of the substances used in self-medication in migraine prophylaxis can be seen as effective. Concerning the self-medication in tension-type headache, the following therapies can be recommended as first-line therapy: the fixed-dose combination of acetaminophen, acetylsalicylic acid and caffeine as well as the fixed combination of acetaminophen and caffeine as well as the monotherapies with ibuprofen or acetylsalicylic acid or diclofenac. The four scientific societies hope that this guideline will help to improve the treatment of headaches which largely is initiated by the patients themselves without any consultation with their physicians.

Cluster size variability and imbalance in cluster randomized controlled trials

Cluster randomized controlled trials are increasingly used to evaluate medical interventions. Research has found that cluster size variability leads to a reduction in the overall effective sample size. Although reporting standards of cluster trials have started to evolve, a far greater degree of transparency is needed to ensure that robust evidence is presented. The use of the numbers of patients recruited to summarize recruitment rate should be avoided in favour of an improved metric that illustrates cumulative power and accounts for cluster variability. Data from four trials is included to show the link between cluster size variability and imbalance. Furthermore, using simulations it is demonstrated that by randomising using a two block randomization strategy and weighting the second by cluster size recruitment, chance imbalance can be minimized.

Applying permutation tests with adjustment for covariates and attrition weights to randomized trials of health-services interventions

Using a health-services study as an illustrative example of longitudinal randomized field research with the potential for participants to be lost to follow-up, we apply a permutation test where the treatment indicator variable is randomly permuted in the context of regression models with covariates and attrition weighting. The test is applied to a multi-site randomized intervention trial of a quality-improvement program for adolescent depression treatment in primary-care settings, in which regression models were used to assess intervention effects with weights used to adjust for attrition bias. The foundation and motivation for this approach to the analysis are considered with attention to the demands associated with implementing such a strategy. The results from the permutation tests were qualitatively similar to the results obtained from conventional parametric models, and in fact suggested that the significance level from the conventional t-test was understated in this application.

Potential for technical errors and subverted allocation can be reduced if certain guidelines are followed: examples from a web-based survey

OBJECTIVE

To elicit researchers' experiences and knowledge of how the randomization process can be undermined.

STUDY DESIGN AND SETTING

Web-based survey conducted in February 2006 using a convenience sample of individuals who are, or have been, involved in some aspect of randomized controlled trials.

RESULTS

Thirty responses were received that described incidences of manipulation. Seven reasons were identified for manipulation: interest of participants, demonstrating treatment efficacy, treatment preference, lack of knowledge, pressure from participants, pressure from trial workers, and practical or technical concerns. In many cases when manipulation was discovered, it was rarely mentioned in the trial publication. Twenty-three responses that described technical errors were received. Technical errors were reported for both the generation and implementation stages of the randomization process.

CONCLUSIONS

This study provides further evidence on trial subversion and highlighted that the potential for technical errors can be reduced, and in most cases eliminated, if certain guidelines are followed. Recommendations are as follows: use simple randomization where possible, use third party allocation, test computer randomization programs prior to participant recruitment and ensure that individuals are aware of the procedures needed to be performed if the treatment allocations cannot be accessed using the intended methods.

Intraocular pressure-lowering effects of all commonly used glaucoma drugs: a meta-analysis of randomized clinical trials

OBJECTIVE

To estimate the intraocular pressure (IOP) reduction achieved by the most frequently prescribed glaucoma drugs and a placebo in a meta-analysis of randomized clinical trials.

DESIGN

Meta-analysis of randomized clinical trials.

PARTICIPANTS

Twenty-seven articles reporting on 28 randomized clinical trials. These articles reported 6953 participants for the trough and 6841 for the peak.

METHODS

Articles published up to December 2003 were identified in the following data sources: Medline, Embase, and the Cochrane Controlled Trials Register, and references from relevant articles. Over 85% of the patients had to be diagnosed with primary open-angle glaucoma (POAG) or ocular hypertension (OH), and articles had to be written in English, German, French, or Dutch. Quality of trials was assessed by a Delphi list with additions. The pooled 1-month IOP-lowering effect from baseline at peak and trough was calculated by performing meta-analysis using the random effects model.

MAIN OUTCOME MEASURES

Absolute and relative change in IOP from baseline, for peak and trough moments.

RESULTS

Relative IOP reductions from baseline [mean (95% confidence interval)] were -23% (-25% to -22%) for a peak and -20% (-23% to -17%) for a trough for 0.5% betaxolol; peak, -27% (-29% to -25%), and trough, -26% (-28% to -25%), for 0.5% timolol; peak, -22% (-24% to -20%), and trough, -17% (-19% to -15%), for 2.0% dorzolamide; peak, -17% (-19% to -15%), and trough, -17% (-19% to -15%) for 1.0% brinzolamide; peak, -25% (-28% to -22%), and trough, -18% (-21% to -14%) for 0.2% brimonidine; peak, -31% (-33% to -29%), and trough, -28% (-30% to -26%) for 0.005% latanoprost; peak, -31% (-32% to -29%), and trough, -29% (-32% to -25%) for 0.004% travoprost; peak, -33% (-35% to -31%), and trough, -28% (-29% to -27%) for 0.03% bimatoprost; and peak, -5% (-9% to -1%), and trough, -5% (-10% to -0%) for the placebo. The difference in absolute IOP reduction from baseline between timolol and prostaglandin analogs or prostamide varied from -0.4 to 0.1 mmHg at trough and from 1.0 to 1.5 mmHg at peak. Quality scores of included studies were generally high, a mean of 14.2 on a scale from 0 to 20 (interquartile range, 13-16).

CONCLUSION

This meta-analysis suggests that bimatoprost, travoprost, latanoprost, and timolol are the most effective intraocular pressure-reducing agents in POAG and OH patients.

On the generation and ownership of alpha in medical studies

Much is known about how to split alpha between or among several comparisons, or how to preserve the nominal alpha level with an exact analysis, but the issue of how alpha is generated, or where it comes from, has not received a commensurate degree of attention. It would seem that there is little point in working out methods to allocate or conserve alpha if it is unlimited in supply. Moreover, there seems to be a logical inconsistency in requiring that a given amount of alpha, generally 0.05, be split among the primary comparisons performed by a given set of researchers, yet allowing other researchers to analyze the same data with a new 0.05 to work with. We will address these inconsistencies, and ask more generally where alpha comes from, how it can be generated, and under what conditions it should be one-tailed or two-tailed.

Quantifying the Magnitude of Baseline Covariate Imbalances Resulting from Selection Bias in Randomized Clinical Trials

Selection bias is most common in observational studies, when patients select their own treatments or treatments are assigned based on patient characteristics, such as disease severity. This first-order selection bias, as we call it, is eliminated by randomization, but there is residual selection bias that may occur even in randomized trials which occurs when, subconsciously or otherwise, an investigator uses advance knowledge of upcoming treatment allocations as the basis for deciding whom to enroll. For example, patients more likely to respond may be preferentially enrolled when the active treatment is due to be allocated, and patients less likely to respond may be enrolled when the control group is due to be allocated. If the upcoming allocations can be observed in their entirety, then we will call the resulting selection bias second-order selection bias. Allocation concealment minimizes the ability to observe upcoming allocations, yet upcoming allocations may still be predicted (imperfectly), or even determined with certainty, if at least some of the previous allocations are known, and if restrictions (such as randomized blocks) were placed on the randomization. This mechanism, based on prediction but not observation of upcoming allocations, is the third-order selection bias that is controlled by perfectly successful masking, but without perfect masking is not controlled even by the combination of advance randomization and allocation concealment. Our purpose is to quantify the magnitude of baseline imbalance that can result from third-order selection bias when the randomized block procedure is used. The smaller the block sizes, the more accurately one can predict future treatment assignments in the same block as known previous assignments, so this magnitude will depend on the block size, as well as on the level of certainty about upcoming allocations required to bias the patient selection. We find that a binary covariate can, on average, be up to 50% unbalanced by third-order selection bias.

Adjusting for observable selection bias in block randomized trials

In this paper, we propose a model-based approach to detect and adjust for observable selection bias in a randomized clinical trial with two treatments and binary outcomes. The proposed method was evaluated using simulations of a randomized block design in which the investigator favoured the experimental treatment by attempting to enroll stronger patients (with greater probability of treatment success) if the probability of the next treatment being experimental was high, and enroll weak patients (with less probability of treatment success) if the probability of the next treatment being experimental was low. The method allows not only testing for the presence of observable selection bias, but also testing for a difference in treatment effects, adjusting for possible selection bias.

The reverse propensity score to detect selection bias and correct for baseline imbalances

The propensity score has been proposed, and for the most part accepted, as a tool to allow for the evaluation of medical interventions in the presence of baseline imbalances arising in the context of observational studies. The lack of an analogous tool to allow for the evaluation of medical interventions in the presence of potentially systematic baseline imbalances in randomized trials has required the use of ad hoc methods. This, in turn, leads to challenges to the conclusions. For example, much of the controversy surrounding recommendations for or against mammography for some age groups stems from the fact that all the randomized trials to study mammography had baseline imbalances, to some extent, in important prognostic covariates. While some of these trials used cluster randomization, baseline imbalances are prevalent also in individually randomized trials. We provide a systematic approach for evaluating medical interventions in the presence of potentially systematic baseline imbalances in individually randomized trials with allocation concealment. Specifically, we define the reverse propensity score as the probability, conditional on all previous allocations and the allocation procedure (restrictions on the randomization), that a given patient will receive a given treatment. We demonstrate how the reverse propensity score allows for both detection of and correction for selection bias, or systematic baseline imbalances.

Ensuring the comparability of comparison groups: is randomization enough?

BACKGROUND

It is widely believed that baseline imbalances in randomized trials must necessarily be random. In fact, there is a type of selection bias that can cause substantial, systematic and reproducible baseline imbalances of prognostic covariates even in properly randomized trials. It is possible, given complete data, to quantify both the susceptibility of a given trial to this type of selection bias and the extent to which selection bias appears to have caused either observable or unobservable baseline imbalances. Yet, in articles reporting on randomized trials, it is uncommon to find either these assessments or the information that would enable a reader to conduct them. Nevertheless, there have been a few published reports that contain descriptions of either this type of selection bias or indicators that it may have occurred.

OBJECTIVE

To document that the same type of selection bias has been described in numerous randomized trials and therefore that it represents a problem deserving of greater attention.

STUDY SELECTION

Computerized searches were not useful in locating trials with one or more elements that contribute to or are indicative of selection bias in randomized trials. We limit our treatment to trials that were previously questioned for susceptibility to selection bias or for large baseline imbalances.

RESULTS

We found 14 randomized trials that appear to be suspicious for selection bias. This may represent only the tip of the iceberg, because the status of other trials is inconclusive.

CONCLUSIONS

Authors of clinical trial reports should be required to disclose sufficient details to allow for an assessment of both allocation concealment and selection bias. The extent to which a randomized study was susceptible to selection bias should be considered in determining the relative contribution it makes to any subsequent meta-analysis, policy or decision.