Sample size calculations for randomized clinical trials published in anesthesiology journals: a comparison of 2010 versus 2016

Interpreting frequentist hypothesis tests: insights from Bayesian inference

Randomized controlled trials are one of the best ways of quantifying the effectiveness of medical interventions. Therefore, when the authors of a randomized superiority trial report that differences in the primary outcome between the intervention group and the control group are "significant" (i.e., P ≤ 0.05), we might assume that the intervention has an effect on the outcome. Similarly, when differences between the groups are "not significant," we might assume that the intervention does not have an effect on the outcome. Nevertheless, both assumptions are frequently incorrect.In this article, we explore the relationship that exists between real treatment effects and declarations of statistical significance based on P values and confidence intervals. We explain why, in some circumstances, the chance an intervention is ineffective when P ≤ 0.05 exceeds 25% and the chance an intervention is effective when P > 0.05 exceeds 50%.Over the last decade, there has been increasing interest in Bayesian methods as an alternative to frequentist hypothesis testing. We provide a robust but nontechnical introduction to Bayesian inference and explain why a Bayesian posterior distribution overcomes many of the problems associated with frequentist hypothesis testing.Notwithstanding the current interest in Bayesian methods, frequentist hypothesis testing remains the default method for statistical inference in medical research. Therefore, we propose an interim solution to the "significance problem" based on simplified Bayesian metrics (e.g., Bayes factor, false positive risk) that can be reported along with traditional P values and confidence intervals. We calculate these metrics for four well-known multicentre trials. We provide links to online calculators so readers can easily estimate these metrics for published trials. In this way, we hope decisions on incorporating the results of randomized trials into clinical practice can be enhanced, minimizing the chance that useful treatments are discarded or that ineffective treatments are adopted.

The Fragility Index of randomized controlled trials in pediatric anesthesiology

PURPOSE

The P value is a widely used measure of statistical importance but has many drawbacks and limitations, one being that it does not reflect the robustness of the results of a clinical trial. The Fragility Index (FI) was developed as a measure of how many outcome events would need to change to nonevents to render a significant P value nonsignificant (P ≥ 0.05). The FI of trials from other medical specialties is typically < 5. We aimed to determine the FI of pediatric anesthesiology randomized controlled trials (RCT) and to test for association with various characteristics of the included trials.

METHODS

We conducted a comprehensive systematic search of high-impact anesthesia, surgical, and medical journals from the last 25 years for trials comparing an intervention between two groups with a statistically significant P value (< 0.05) for a dichotomous outcome. We also compared FI values for variables that reflect the quality and importance of a trial.

RESULTS

The median [interquartile range] FI was 3 [1-7] and correlated positively with the number of participants (rS = 0.41; P < 0.001) and events (rS = 0.42; P < 0.001), and negatively with the P value (rPB = -0.36; P < 0.001). Other measures of trial quality and impact or importance were not strongly associated with the FI.

CONCLUSIONS

The FI of published trials in pediatric anesthesiology is similarly low as in other medical specialties. Larger trials with more events and P values ≤ 0.01 were associated with a higher FI.

The efficacy and safety of patent Foramen Ovale Closure for Refractory Epilepsy (PFOC-RE): a prospectively randomized control trial of an innovative surgical therapy for refractory epilepsy patients with PFO of high-grade right-to-left shunt

BACKGROUND

A significant proportion of patients with epilepsy have an unknown etiology and lack effective targeted therapeutic drugs. Patent Foramen Ovale (PFO) induces hypoxia and microembolism, leading to cerebral neurological dysfunction and increased epilepsy risk. This study aims to assess the efficacy and safety of PFO closure for relieving epileptic seizures in patients with refractory epilepsy associated with PFO.

METHODS/DESIGN

Recruitment takes place at the West China Hospital of Sichuan University, China, for an open-label, randomized controlled clinical trial. The trial will include 110 patients with refractory epilepsy and PFO. Disease diagnoses will conform to the diagnostic criteria of the International League Against Epilepsy (ILAE) for refractory epilepsy and the American Society of Echocardiography (ASE) for PFO. Refractory epilepsy and high-grade right-to-left shunt (RLS) of the PFO will be further diagnosed using 24-hour video electroencephalogram and transthoracic echocardiography with contrast injection, respectively. Eligible participants require a secondary or higher volume of RLS.

TRIAL REGISTRATION

Chinese Clinical Trial Registry (ChiCTR2200065681). Registered on November 11, 2022.

Minimum false-positive risk of primary outcomes and impact of reducing nominal P-value threshold from 0.05 to 0.005 in anaesthesiology randomised clinical trials: a cross-sectional study

BACKGROUND

Reproducibility of research is poor; this may be because many articles report statistically significant findings that are false positives. Two potential solutions are to lower the P-value for statistical significance testing from 0.05 to 0.005 and to report the minimum false-positive risk (minFPR). This study determined these metrics for randomised controlled trials (RCTs) in general anaesthesiology journals.

METHODS

We identified superiority RCTs published between January 1, 2019 and March 15, 2021 from seven leading anaesthesia journals. P-values for primary outcomes were collected, and minFPRs for these outcomes were calculated using a formula assuming a 50% prior probability of an intervention being effective (minFPR₅₀). The primary outcomes were the percentage of RCTs maintaining statistical significance at P<0.005 and minFPR₅₀.

RESULTS

We included 318 RCTs. P-values below 0.05 were reported in 205/318 (64%) of RCTs. Of these 205 RCTs, 119/205 (58%) maintained statistical significance at the P<0.005 threshold. The mean (standard deviation) minFPR₅₀ was 22% (20). At P=0.005, the minFPR₅₀ was approximately 5%.

CONCLUSIONS

These proposed metrics aimed at mitigating reproducibility concerns would call a significant portion of the anaesthesiology literature into question. We found a minFPR of 22% and determined that 42% of primary outcomes would not maintain statistical significance if the P-value threshold changed from 0.05 to 0.005. These findings could partially explain the lack of reproducibility of research findings.

Assessing the accuracy of multiparametric MRI to predict clinically significant prostate cancer in biopsy naïve men across racial/ethnic groups

Introduction

The Prostate Imaging Reporting and Data System (PIRADS) has shown promise in improving the detection of Gleason grade group (GG) 2–5 prostate cancer (PCa) and reducing the detection of indolent GG1 PCa. However, data on the performance of PIRADS in Black and Hispanic men is sparse. We evaluated the accuracy of PIRADS scores in detecting GG2-5 PCa in White, Black, and Hispanic men.

Methods

We performed a multicenter retrospective review of biopsy-naïve Black (n = 108), White (n = 108), and Hispanic (n = 64) men who underwent prostate biopsy (PB) following multiparametric MRI. Sensitivity and specificity of PIRADS for GG2-5 PCa were calculated. Race-stratified binary logistic regression models for GG2-5 PCa using standard clinical variables and PIRADS were used to calculate area under the receiver operating characteristics curves (AUC).

Results

Rates of GG2-5 PCa were statistically similar between Blacks, Whites, and Hispanics (52.8% vs 42.6% vs 37.5% respectively, p = 0.12). Sensitivity was lower in Hispanic men compared to White men (87.5% vs 97.8% respectively, p = 0.01). Specificity was similar in Black versus White men (21.6% vs 27.4%, p = 0.32) and White versus Hispanic men (27.4% vs 17.5%, p = 0.14).

The AUCs of the PIRADS added to standard clinical data (age, PSA and suspicious prostate exam) were similar when comparing Black versus White men (0.75 vs 0.73, p = 0.79) and White versus Hispanic men (0.73 vs 0.59, p = 0.11). The AUCs for the Base model and PIRADS model alone were statistically similar when comparing Black versus White men and White versus Hispanic men.

Conclusions

The accuracy of the PIRADS and clinical data for detecting GG2-5 PCa seems statistically similar across race. However, there is concern that PIRADS 2.0 has lower sensitivity in Hispanic men compared to White men. Prospective validation studies are needed.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12894-022-01066-9.

Optimization of adaptive designs with respect to a performance score

Adaptive designs are an increasingly popular method for the adaptation of design aspects in clinical trials, such as the sample size. Scoring different adaptive designs helps to make an appropriate choice among the numerous existing adaptive design methods. Several scores have been proposed to evaluate adaptive designs. Moreover, it is possible to determine optimal two-stage adaptive designs with respect to a customized objective score by solving a constrained optimization problem. In this paper, we use the conditional performance score by Herrmann et al. (2020) as the optimization criterion to derive optimal adaptive two-stage designs. We investigate variations of the original performance score, for example, by assigning different weights to the score components and by incorporating prior assumptions on the effect size. We further investigate a setting where the optimization framework is extended by a global power constraint, and additional optimization of the critical value function next to the stage-two sample size is performed. Those evaluations with respect to the sample size curves and the resulting design's performance can contribute to facilitate the score's usage in practice.

Apophenia and anesthesia: how we sometimes change our practice prematurely

Human beings are predisposed to identifying false patterns in statistical noise, a likely survival advantage during our evolutionary development. Moreover, humans seem to prefer "positive" results over "negative" ones. These two cognitive features lay a framework for premature adoption of falsely positive studies. Added to this predisposition is the tendency of journals to "overbid" for exciting or newsworthy manuscripts, incentives in both the academic and publishing industries that value change over truth and scientific rigour, and a growing dependence on complex statistical techniques that some reviewers do not understand. The purpose of this article is to describe the underlying causes of premature adoption and provide recommendations that may improve the quality of published science.

Reporting of statistical sample size calculations in publications of trials on age-related macular degeneration, glaucoma and cataract

Background

Transparent and complete publications of randomised controlled trials (RCT) ought to comply with the guidelines of the CONSORT Statement, which stipulates sample size calculation as an important aspect of trial planning. The objective of this study was to analyse and compare the reporting of statistical sample size calculations in RCT papers on the treatment of age-related macular degeneration (AMD), glaucoma and cataract published in 2018.

Material and methods

This study comprises a total of 113 RCT papers (RCT-P) published in 2018 (AMD: 14, glaucoma: 28, cataract: 71), in English or German, and identified through an internet-based literature search in PubMed and EMBASE. The primary outcome measure of the study was the number of trials providing a complete description of the underlying sample case calculation on the basis of the variables required (significance level, expected outcomes, power, and resulting sample size).

Results

Of the RCTs reviewed, 64% (AMD), 61% (glaucoma) and 31% (cataract) provided a justification of the number of patients included. A complete description of the described studies’ sample size calculation including all the necessary values (primary outcome measure of this study) was described by 21% of the AMD, 29% of the cataract and 18% of the glaucoma RCT publications (in total: 24 of 113 (21%) at a confidence interval of 95%: [13%; 29%]).

Conclusion

All three treatment areas analysed lacked reporting quality regarding the justification of the number of patients included in a clinical trial based on a sample size calculation required for ethical reasons. More than half of all RCT publications reviewed did not provide all of the required information on statistical sample size calculation, and thus lacked transparency and completeness. It is therefore urgently required to involve methodologists in a study’s planning and publishing processes to ensure that methodology descriptions are transparent and of high quality.

Are treatment effect assumptions in orthodontic studies overoptimistic?

Background

At the clinical trial design stage, assumptions regarding the treatment effects to be detected should be appropriate so that the required sample size can be calculated. There is evidence in the medical literature that sample size assumption can be overoptimistic. The aim of this study was to compare the distribution of the assumed effects versus that of the observed effects as a proxy for overoptimistic treatment effect assumptions at the study design stage.

Materials and method

Systematic reviews (SRs) published between 1 January 2010 and 31 December 2019 containing at least one meta-analysis on continuous outcomes were identified electronically. SR and primary study level characteristics were extracted from the SRs and the individual trials. Details on the sample size calculation process and assumptions and the observed treatment effects were extracted.

Results

Eighty-five SRs with meta-analysis containing 347 primary trials were included. The median number of SR authors was 5 (interquartile range: 4–7). At the primary study level, the majority were single centre (78.1%), utilized a parallel design (52%), and rated as an unclear/moderate level of risk of bias (34.3%). A sample size was described in only 31.7% (110/347) of studies. From this cohort of 110 studies, in only 37 studies was the assumed clinical difference that the study was designed to detect reported (37/110). The assumed treatment effect was recalculated for the remaining 73 studies (73/110). The one-sided exact signed rank test showed a significant difference between the assumed and observed treatment effects (P < 0.001) suggesting greater values for the assumed effect sizes.

Conclusions

Careful consideration of the assumptions at the design stage of orthodontic studies are necessary in order to reduce the unreliability of clinical study results and research waste.

[Influence of impact factor on reporting sample size calculations in publications on studies exemplified by AMD treatment : Cross-sectional investigation on the presence of sample size calculations in publications of RCTs on AMD treatment in journals with low and high impact factors]

BACKGROUND

For scientific and ethical reasons randomized controlled clinical trials (RCTs) should be based on a sample size calculation. The CONSORT statement, an established publication guideline for transparent study reporting, requires a sample size calculation in every study publication.

OBJECTIVE

The availability of sample size calculations in RCT publications on treatment of age-related macular degeneration (AMD) was investigated. The primary hypothesis of this investigation compared the prevalence of reported sample size calculations between journals with higher (≥5) versus lower (<5) impact factors (IF).

MATERIAL AND METHODS

It was examined whether information on sample size calculation was available in a series of 97 publications of RTCs on AMD treatment published between 2004 and 2014.

RESULTS

Only 46 out of 97 (47%) study publications provided information on the reason for the number of patients enrolled. The comparison of publications from journals with an IF ≥ 5 (63%, 30) and from journals with an IF < 5 (40%, 67) showed a statistically significant difference of 23% in the frequencies of available sample size calculations (95% confidence interval, CI 2%; 44%). Of the publications published before 2010, 43% reported a sample size calculation versus 51% of the publications afterwards.

CONCLUSION

Publications in journals with higher IF more frequently reported a sample size calculation. More than 50% of the publications did not report any sample size calculation. Authors and reviewers of publications should pay more attention to the explicit reporting of sample size calculations.

Validity of sample sizes in publications of randomised controlled trials on the treatment of age-related macular degeneration: cross-sectional evaluation

OBJECTIVE

The aim of this cross-sectional study was to examine the completeness and accuracy of the reporting of sample size calculations in randomised controlled trial (RCT) publications on the treatment of age-related macular degeneration (AMD).

METHODS

A sample of 97 RCTs published between 2004 and 2014 was reviewed for the calculation of their sample size. It was examined whether a (complete) description of the sample size calculation was presented. Furthermore, the sample size was recalculated, whenever possible based on the published details, in order to verify the reported number of patients.

PRIMARY OUTCOME MEASURE

The primary endpoint of this cross-sectional investigation was a described sample size calculation that was reproducible, complete and correct (maximum tolerated deviation between reported and replicated sample size ±2 participants per trial arm).

RESULTS

A total of 50 publications (52%) did not provide any information on the justification of the number of patients included. Only 17 publications (18%) provided all the necessary parameters for recalculation; 8 of 97 (8%, 95%-CI: 4% to 16%) publications achieved the primary endpoint. The median relative deviation between reported and recalculated sample sizes was 1%, with a range from -43% to +66%.

CONCLUSION

Although a transparent sample size legitimation is a crucial determinant of an RCT's methodological validity, more than half of the RCT publications considered failed to report them. Furthermore, reported sample size legitimations were often incomplete or incorrect. In summary, clinical authors should pay more attention to the transparent reporting of sample size calculation, and clinical journal reviewers may opt to reproduce reported sample size calculations.

SYNOPSIS

More than half of the analysed RCT publications on the treatment of AMD did not report a transparent sample size calculation. Only 8% reported a complete and correct sample size calculation.