Pre-trial evaluation of the potential for unblinding in drug trials: a prototype example

Application of tetrad testing to the evaluation of blinding strategies for ancillary supplies used in controlled clinical trials

BACKGROUND/AIMS

Ensuring verum and placebo cannot be visually distinguished from each other is a critical aspect of blinded controlled clinical trials. Our objective was to propose a rational approach to the visual evaluation of placebo matching candidates.

METHODS

Verum and placebo samples were prepared in clear clinical ancillary supplies (intravenous bags, syringes and administration lines) covered at different levels using opaque sleeves. Triangle and tetrad tests, two sensory discriminative testing methods widely used in the food industry, were applied to assess visual differences between verum and placebo.

RESULTS

Triangle and tetrad test results allowed defining the level of opaque coverage required to ensure blinding for three biological drug molecules of therapeutic interest. While the limited number of panelists did not allow a statistically sound comparison of triangle and tetrad test methodologies, tetrad test has a theoretical higher power than triangle test, meaning fewer panelists are needed to reach the same statistical conclusion.

CONCLUSION

Tetrad test offers a rational approach to define a blinding strategy for ancillary supplies used in a controlled clinical trial.

Blinding, sham, and treatment effects in randomized controlled trials for back pain in 2000-2019: A review and meta-analytic approach

BACKGROUND

Blinding aims to minimize biases from what participants and investigators know or believe. Randomized controlled trials, despite being the gold standard to evaluate treatment effect, do not generally assess the success of blinding. We investigated the extent of blinding in back pain trials and the associations between participant guesses and treatment effects.

METHODS

We did a review with PubMed/OvidMedline, 2000-2019. Eligibility criteria were back pain trials with data available on treatment effect and participants' guess of treatment. For blinding, blinding index was used as chance-corrected measure of excessive correct guess (0 for random guess). For treatment effects, within- or between-arm effect sizes were used. Analyses of investigators' guess/blinding or by treatment modality were performed exploratorily.

RESULTS

Forty trials (3899 participants) were included. Active and sham treatment groups had mean blinding index of 0.26 (95% confidence interval: 0.12, 0.41) and 0.01 (-0.11, 0.14), respectively, meaning 26% of participants in active treatment believed they received active treatment, whereas only 1% in sham believed they received sham treatment, beyond chance, that is, random guess. A greater belief of receiving active treatment was associated with a larger within-arm effect size in both arms, and ideal blinding (namely, "random guess," and "wishful thinking" that signifies both groups believing they received active treatment) showed smaller effect sizes, with correlation of effect size and summary blinding indexes of 0.35 (p = 0.028) for between-arm comparison. We observed uniformly large sham treatment effects for all modalities, and larger correlation for investigator's (un)blinding, 0.53 (p = 0.046).

CONCLUSION

Participants in active treatments in back pain trials guessed treatment identity more correctly, while those in sham treatments tended to display successful blinding. Excessive correct guesses (that could reflect weaker blinding and/or noticeable effects) by participants and investigators demonstrated larger effect sizes. Blinding and sham treatment effects on back pain need due consideration in individual trials and meta-analyses.

A pre-trial evaluation of blinding for a Chinese herbal medicine trial

Background

Blinding is considered an important methodological characteristic in clinical trials to minimise bias and maximise the validity of a trial. Unlike pharmaceutical substances, most herbal medicines have distinctive sensory specifications, including odour and taste, which can be quite challenging when developing a placebo control to match the specific characteristics of herbal substances being examined. The present study was, therefore, designed to evaluate whether the participants could differentiate an active herbal capsule (Ganopoly combination) from a placebo material capsule. The aim of this study was to develop a suitable placebo substance for encapsulation to be used in a future herbal medicine clinical trial.

Methods

The current study was improved upon the previous investigation, and several modifications were made to the placebo substance in order to mimic the herbal substance characteristics. Prior to conducting the study, a refined placebo substance was developed using commonly consumed culinary agents. Sixty-two healthy volunteers participated in the study and were randomly provided one of the two substances. Individuals were asked to evaluate the three sensory characteristics of the allocated capsule (visual appearance, odour, and taste), and determine whether they believed the substance to be a 'herbal' or a 'placebo' substance.

Results

The study provided evidence on the success of blinding for only two sensory characteristics, namely, visual appearance (95% CI -0.15, 0.34) and odour (95% CI -0.34, 0.15). In contrast, the findings related to the taste indicated that participants correctly guessed the herbal substance compared to the placebo substance to a significantly higher proportion than would have been expected by chance alone (95% CI 0.14, 0.60).

Conclusion

The failure to blind participants for taste highlights the difficulties in preparing placebo herbal substances that match as closely as possible to a real herbal substance. Blinding is particularly challenging where herbal medicines have different sensory characteristics.

Sample size calculations for blinding assessment

Blinding is a critical component in randomized clinical trials along with treatment effect estimation and comparisons between the treatments. Various methods have been proposed for the statistical analyses of blinding-related data, but there is little guidance for determining the sample size for this type of data, especially if blinding assessment is done in pilot studies. In this paper, we try to fill this gap and provide simple methods to address sample size calculations for a "new" study with different research questions and scenarios. The proposed methods are framed in terms of estimation/precision or statistical testing to allow investigators to choose the best suited method for their goals. We illustrate the methods using worked examples with real data.

Not All Masks Are Created Equal: Masking Success in Clinical Trials of Children and Adolescents

The current study assessed the success of masking omega-3 (Ω3) and psychotherapy in clinical trials of youth with depression or bipolar spectrum disorder. Participants were youth ages 7-14 with DSM-IV-TR diagnosed depressive (n = 72) or bipolar spectrum (n = 23) disorders. Inclusion diagnoses were depressive disorder, cyclothymic disorder, or bipolar disorder not otherwise specified. Exclusion diagnoses included bipolar I or II disorder, chronic medical condition or autism. Youth participated in 2 × 2 randomized controlled trials, in which they received Ω3 or placebo (PBO) and psychoeducational psychotherapy (PEP) or active monitoring (AM). Participants and study staff (including independent interviewers) were masked to Ω3/PBO allocation. Besides the masked independent interviewers, one coprincipal investigator (Co-PI) was fully masked to both conditions and completed all consensus conference ratings postrandomization. At the endpoint assessment or last completed interview, interviewers and the masked Co-PI guessed whether each child was assigned to Ω3 or PBO and to PEP or AM. Masking failure was calculated using the degree of correct guesses above chance level using binomial tests across all participants for Ω3 versus PBO and PEP versus AM. For all guessers, Ω3 allocation was guessed correctly approximately half the time (50%-52.5%). Rates of correct guessing were higher for PEP, but only the interviewer guesses were correct significantly more often (58.5%-68.7%) than chance. Reporting of masking success should be an essential element of RCTs. Psychotherapy is generally more difficult to mask, but with attentive masking procedures reasonable masking can be achieved.

Random Guess and Wishful Thinking are the Best Blinding Scenarios

Blinding is a methodologic safeguard of treatment evaluation, yet severely understudied empirically. Mathieu et al.'s theoretical analysis (2014) provided an important message that blinding cannot eliminate potential for bias associated with belief about allocation in randomized controlled trial; just like the intent-to-treat principle does not guarantee unbiased estimation under noncompliance, the blinded randomized trial as a golden standard may produce bias. They showed possible biases but did not assess how large the bias could be in different scenarios. In this paper, we examined their findings, and numerically assessed and compared the bias in treatment effect parameters by simulation under frequently encountered blinding scenarios, aiming to identify the most ideal blinding scenarios in practice. We conclude that Random Guess and Wishful Thinking (e.g., participants tend to believe they received treatment) are the most ideal blinding scenarios, incurring minimal bias. We also find some evidence that imperfect or partial blinding can be better than no blinding.

The matching quality of experimental and control interventions in blinded pharmacological randomised clinical trials: a methodological systematic review

Background

Blinding is a pivotal method to avoid bias in randomised clinical trials. In blinded drug trials, experimental and control interventions are often designed to be matched, i.e. to appear indistinguishable. It is unknown how often matching procedures are inadequate, so we decided to systematically identify and analyse studies of matching quality in drug trials. Our primary objective was to assess the proportion of studies that concluded that the matching was inadequate; our secondary objective was to describe mechanisms for inadequate matching.

Methods

Systematic review. We searched PubMed, Google Scholar and Web of Science Citation Index for studies that assessed whether supposedly indistinguishable interventions (experimental and control) in randomized clinical drug trials could be distinguished based on physical properties (e.g. appearance or smell). Two persons decided on study eligibility and extracted data independently. Our primary analysis was based on the conclusions of each study. In supportive analyses, we defined a low and a high threshold for inadequate matching. We summarised results qualitatively.

Results

We included studies of 36 trials, of which 28 (78 %) were published before 1977. The studies differed considerably with regard to design, methodology and analysis. Sixteen of the 36 studies (44 %) concluded inadequate matching. When we adapted high or low thresholds for inadequate matching, the number of trials with inadequate matching was reduced to 12 (33 %) or increased to 26 (72 %). Inadequate matching was concluded in 7 of 22 trials (32 %) based on a defined cohort of trials. Inadequate matching was concluded in 9 of 14 trials (64 %) which were not based on a trial cohort, and therefore at a higher risk of publication bias. The proportion of inadequate matching did not seem to depend on publication year. Typical mechanisms of inadequate matching were differences in taste or colour.

Conclusion

We identified matching quality studies of 36 randomized clinical drug trials. Sixteen of the 36 studies (44 %) concluded inadequate matching. Few studies of matching quality in contemporary trials have been published, but show similar results as found for older trials. Inadequate matching in drug trials may be more prevalent than commonly believed.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-016-0111-9) contains supplementary material, which is available to authorized users.

Revisão sistemática sobre tratamentos psicológicos para problemas relacionados ao crack

OBJETIVO: O objetivo deste artigo é apresentar uma revisão sistemática da literatura sobre tratamentos psicológicos oferecidos para usuários de crack. MÉTODOS: Foi realizada uma revisão sistemática por meio de uma busca na literatura internacional e nacional, indexada nas bases de dados Medline, SciELO, Lilacs e Web of Science. Os descritores utilizados foram: crack or crack cocaine or cocaine smokers (crack) and psychosocial treatment or psycotherapy or psychosocial treatment (tratamento psicológico) e a busca incluiu artigos publicados no período de 2001 a 2011. RESULTADOS: No total foram encontrados 155 artigos por meio dos descritores utilizados. Os artigos foram agrupados em três dimensões: tratamentos psicossociais na internação e cuidados continuados, relaxamento respiratório e outras técnicas comportamentais e abordagens fundamentadas na Entrevista Motivacional, Cognitivo-Comportamental e Modelo Transteórico de Mudança. CONCLUSÃO: Com base nos estudos examinados, pode ser formulado um elenco de algumas intervenções que estão sendo estudadas para o tratamento de usuários de crack e algumas apresentam resultados satisfatórios. Os poucos esforços de comparação entre técnicas resultaram em evidências de pouca ou nenhuma diferença, ainda que se registre o benefício para os usuários na aplicação de qualquer delas. Não existe consenso acerca da efetividade no tratamento de usuários de crack. Parece oportuno e necessário o aprofundamento dos estudos nesse campo.

Blinding in randomized clinical trials: imposed impartiality

Blinding, or "masking," is a crucial method for reducing bias in randomized clinical trials. In this paper, we review important methodological aspects of blinding, emphasizing terminology, reporting, bias mechanisms, empirical evidence, and the risk of unblinding. Theoretical considerations and empirical analyses support the blinding of patients, health-care providers, and outcome assessors as to the trial intervention to which patients have been allocated. We encourage extensive pretrial testing of blinding procedures and explicit reporting of who was in the blinded condition and the methods used to ensure blinding.

Towards a proposal for assessment of blinding success in clinical trials: up-to-date review

The CONSORT statement recommended that investigators should clearly report which key trial persons were blinded to treatment allocation and test for the success of blinding. Clinical researchers, however, more often than not overlook the assessment of the success of blinding. The severe under-reporting on the success of blinding may improve with awareness of existing quantitative methods. The two statistical methods, James' blinding index (BI) and Bang's BI, are currently available. Subjects could be asked to guess their treatment assignment, possibly with an option to express the degree of certainty. Assessments of blinding at various points may serve different purposes, i.e. to evaluate comparability between experimental versus control treatments before the trial by the third party; to examine further comparability and credibility of the control treatment and patients' expectation about treatment received in early stage of the trial, and to summarize the overall maintenance of the blinding success at the end of the trial. In this article, we review BIs and how to use these methods along with discussion of other issues in blinding assessment and reporting. We contend the two BIs that were independently developed but carry complementary properties would characterize blinding behaviours in clinical trials qualitatively as well as quantitatively, and may also shed some lights on the interpretation of the study findings. Finally we urge the Item 11b of the CONSORT statement to be revised to require the assessment/reporting of blinding success for all trials that adopt blinding schemes.

Multiple-blind: Towards a new blinding protocol for future generations of clinical trials

In this hypothesis we try to suggest a new blinding protocol acting as a gold standard for design and conduction of unbiased clinical trials. All of the following five categories of key trial persons should be blinded to treatment allocation: (a) participants, (b) healthcare providers, (c) data collectors, (d) outcome assessors and (e) data analysts. Furthermore, personnel writing the manuscript should be blinded through submission two drafts of a manuscript, with draft 1 written assuming that group A is the treatment group and draft 2 assuming that group B is the treatment group. Also, double-blind peer review procedure (in which the reviewers' and authors' names and affiliations are not disclosed to one another) should be carried out. Test of the success of blinding should be used. Participants, healthcare providers and data collectors, should be tested for the success of blinding in early stage of the trial and at the end of the trial. For outcome assessors, data analysts, manuscript writers and peer reviewers one test session in the end of their work is enough.

Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding

BACKGROUND

Blinding can reduce bias in randomized clinical trials, but blinding procedures may be unsuccessful. Our aim was to assess how often randomized clinical trials test the success of blinding, the methods involved and how often blinding is reported as being successful.

METHODS

We analysed a random sample of blinded randomized clinical trials indexed in the The Cochrane Central Register of Controlled Trials and published in 2001. We identified 1599 blinded trials, and noted if they had conducted any test for the success of blinding. We also selected 200 trials randomly that did not report any such test, and sent a questionnaire to the corresponding authors asking them if they had conducted any tests.

RESULTS

Thirty-one out of 1599 trials (2%) reported tests for the success of blinding. Test methods varied, and reporting was generally incomplete. Blinding was considered successful in 14 out of the 31 trials (45%) and unclear in 10 (32%). Of the seven trials (23%) reporting unsuccessful blinding the risk of a biased trial result was either not addressed or was discounted in six cases. We received 130 questionnaires from trial authors (65%) of which 15 (12%) informed that they had conducted, but not published, tests.

CONCLUSIONS

Blinding is rarely tested. Test methods vary, and the reporting of tests, and test results, is incomplete. There is a considerable methodological uncertainty how best to assess blinding, and an urgent need for improved methodology and improved reporting.