Misconceptions, Misuses, and Misinterpretations of P Values and Significance Testing

New Anticancer Drugs: Reliably Assessing "Value" While Addressing High Prices

Countries face challenges in paying for new drugs. High prices are driven in part by exploding drug development costs, which, in turn, are driven by essential but excessive regulation. Burdensome regulation also delays drug development, and this can translate into thousands of life-years lost. We need system-wide reform that will enable less expensive, faster drug development. The speed with which COVID-19 vaccines and AIDS therapies were developed indicates this is possible if governments prioritize it. Countries also differ in how they value drugs, and generally, those willing to pay more have better, faster access. Canada is used as an example to illustrate how "incremental cost-effectiveness ratios" (ICERs) based on measures such as gains in "quality-adjusted life-years" (QALYs) may be used to determine a drug's value but are often problematic, imprecise assessments. Generally, ICER/QALY estimates inadequately consider the impact of patient crossover or long post-progression survival, therapy benefits in distinct subpopulations, positive impacts of the therapy on other healthcare or societal costs, how much governments willingly might pay for other things, etc. Furthermore, a QALY value should be higher for a lethal or uncommon disease than for a common, nonlethal disease. Compared to international comparators, Canada is particularly ineffective in initiating public funding for essential new medications. Addressing these disparities demands urgent reform.

On the use of receiver operating characteristic curve analysis to determine the most appropriate p value significance threshold

BACKGROUND

p value is the most common statistic reported in scientific research articles. Choosing the conventional threshold of 0.05 commonly used for the p value in research articles, is unfounded. Many researchers have tried to provide a reasonable threshold for the p value; some proposed a lower threshold, eg, 0.005. However, none of the proposals has gained universal acceptance. Using the analogy between the diagnostic tests with continuous results and statistical inference tests of hypothesis, I wish to present a method to calculate the most appropriate p value significance threshold using the receiver operating characteristic curve (ROC) analysis.

RESULTS

As with diagnostic tests where the most appropriate cut-off values are different depending on the situation, there is no unique cut-off for the p significance threshold. Unlike the previous proposals, which mostly suggest lowering the threshold to a fixed value (eg, from 0.05 to 0.005), the most appropriate p significance threshold proposed here, in most instances, is much less than the conventional cut-off of 0.05 and varies from study to study and from statistical test to test, even within a single study. The proposed method provides the minimum weighted sum of type I and type II errors.

CONCLUSIONS

Given the perplexity involved in using the frequentist statistics in a correct way (dealing with different p significance thresholds, even in a single study), it seems that the p value is no longer a proper statistic to be used in our research; it should be replaced by alternative methods, eg, Bayesian methods.

Detailed statistical analysis plan for a secondary Bayesian analysis of the SafeBoosC-III trial: a multinational, randomised clinical trial assessing treatment guided by cerebral oximetry monitoring versus usual care in extremely preterm infants

BACKGROUND

Extremely preterm infants have a high mortality and morbidity. Here, we present a statistical analysis plan for secondary Bayesian analyses of the pragmatic, sufficiently powered multinational, trial-SafeBoosC III-evaluating the benefits and harms of cerebral oximetry monitoring plus a treatment guideline versus usual care for such infants.

METHODS

The SafeBoosC-III trial is an investigator-initiated, open-label, randomised, multinational, pragmatic, phase III clinical trial with a parallel-group design. The trial randomised 1601 infants, and the frequentist analyses were published in April 2023. The primary outcome is a dichotomous composite outcome of death or severe brain injury. The exploratory outcomes are major neonatal morbidities associated with neurodevelopmental impairment later in life: (1) bronchopulmonary dysplasia; (2) retinopathy of prematurity; (3) late-onset sepsis; (4) necrotising enterocolitis; and (5) number of major neonatal morbidities (count of bronchopulmonary dysplasia, retinopathy of prematurity, and severe brain injury). The primary Bayesian analyses will use non-informed priors including all plausible effects. The models will use a Hamiltonian Monte Carlo sampler with 1 chain, a sampling of 10,000, and at least 25,000 iterations for the burn-in period. In Bayesian statistics, such analyses are referred to as 'posteriors' and will be presented as point estimates with 95% credibility intervals (CrIs), encompassing the most probable results based on the data, model, and priors selected. The results will be presented as probability of any benefit or any harm, Bayes factor, and the probability of clinical important benefit or harm. Two statisticians will analyse the blinded data independently following this protocol.

DISCUSSION

This statistical analysis plan presents a secondary Bayesian analysis of the SafeBoosC-III trial. The analysis and the final manuscript will be carried out and written after we publicise the primary frequentist trial report. Thus, we can interpret the findings from both the frequentists and Bayesian perspective. This approach should provide a better foundation for interpreting of our findings.

TRIAL REGISTRATION

ClinicalTrials.org, NCT03770741. Registered on 10 December 2018.

Study Design and Analysis in Hand Surgery Research: Tips for Success

Hand surgeons are constantly faced with evaluation of new evidence to identify best practices in clinical care. However, even the most rigorous study designs have limitations due to biases, generalizability, and other flaws. Here, we highlight seven common aspects of study design and analysis that should be considered by hand surgeons when interpreting findings. The evaluation of these practices can optimize the peer-review process and assess the value of evidence to be incorporated into clinical practice.

Empfehlung für Fragebögen zur Untersuchung der Fuß- und Sprunggelenkfunktion vom Research-Komitee der AGA

Das standardisierte Erheben von Behandlungsergebnissen findet immer mehr Einzug in die moderne Medizin. Hierbei ist vor allem das subjektive Ergebnis aus Patientensicht und die damit häufig einhergehende Zufriedenheit mit der medizinischen Behandlung in den Fokus gerückt. Dies hat in den letzten Jahren nicht nur in der akademischen Landschaft, sondern auch auf gesundheitspolitischer Ebene deutlich an Bedeutung gewonnen. Dies bedeutet, dass das Behandlungsergebnis immer mehr Einfluss auf die Vergütung der Leistungserbringer erhält, auch „value-based healthcare“ genannt. Insbesondere in der Orthopädie und Traumatologie gibt es dabei zahlreiche gelenkspezifische Messinstrumente (Fragebögen, Scores oder „patient-reported outcome measures“ [PROMs]), wobei in der Fuß- und Sprunggelenkchirurgie hierfür eine Vielzahl an Fragebögen verfügbar ist, die sich jedoch häufig sehr heterogen und selten pathologiespezifisch darstellen. In Anbetracht der derzeitigen Mannigfaltigkeit an PROMs in der aktuellen Literatur ist es deshalb von Interesse, die wissenschaftliche Qualität zu prüfen und gleichzeitig die Forschungskommunikation zu vereinheitlichen, indem etablierte und gemäß akademischen Gütekriterien ausgewählte Fragebögen zum Einsatz kommen. Ziel ist es, valide, verlässliche und möglichst repräsentative Daten zu gewinnen, um eine akkurate Darstellung der tatsächlichen Ergebnisse zu erreichen. Dies bedingt neben qualitativen Kriterien auch immer kompakte und leicht verständliche Fragenkataloge, um so eine möglichst hohe Rücklaufquote zu generieren. Schließlich sollte auch immer eine Einschätzung der klinischen Relevanz der erhaltenen Ergebnisse stattfinden. Hierfür hat sich der kleinste vom Patienten als klinische Veränderung wahrgenommene Wert des jeweiligen PROM („minimal clinically important difference“, MCID) durchgesetzt. Für den Fuß- und Sprunggelenkbereich werden dabei vom Research Komitee der AGA unter Berücksichtigung dieser Kriterien folgende Messinstrumente empfohlen: FAOS (Foot and Ankle Outcome Score) zur Beurteilung der allgemeinen Schmerzsymptomatik und Funktion sowie der EQ-5D-Fragebogen zur Beurteilung der Lebensqualität. Zwar kann aufgrund fehlender Datenlage keine pathologiespezifische Empfehlung abgegeben werden, jedoch ist der FAOS für eine Vielzahl von Fuß- und Sprunggelenkpathologien validiert und liegt zudem in deutscher Sprache vor.

Development and design validation of a novel network meta-analysis presentation tool for multiple outcomes: a qualitative descriptive study

OBJECTIVE

The Grades of Recommendations, Assessment, Development and Evaluation working group recently developed an innovative approach to interpreting results from network meta-analyses (NMA) through minimally and partially contextualised methods; however, the optimal method for presenting results for multiple outcomes using this approach remains uncertain. We; therefore, developed and iteratively modified a presentation method that effectively summarises NMA results of multiple outcomes for clinicians using this new interpretation approach.

DESIGN

Qualitative descriptive study.

SETTING

A steering group of seven individuals with experience in NMA and design validation studies developed two colour-coded presentation formats for evaluation. Through an iterative process, we assessed the validity of both formats to maximise their clarity and ease of interpretation.

PARTICIPANTS

26 participants including 20 clinicians who routinely provide patient care, 3 research staff/research methodologists and 3 residents.

MAIN OUTCOME MEASURES

Two team members used qualitative content analysis to independently analyse transcripts of all interviews. The steering group reviewed the analyses and responded with serial modifications of the presentation format.

RESULTS

To ensure that readers could easily discern the benefits and safety of each included treatment across all assessed outcomes, participants primarily focused on simple information presentations, with intuitive organisational decisions and colour coding. Feedback ultimately resulted in two presentation versions, each preferred by a substantial group of participants, and development of a legend to facilitate interpretation.

CONCLUSION

Iterative design validation facilitated the development of two novel formats for presenting minimally or partially contextualised NMA results for multiple outcomes. These presentation approaches appeal to audiences that include clinicians with limited familiarity with NMAs.

Cognitive Diagnostic Assessment in University Statistics Education: Valid and Reliable Skill Measurement for Actionable Feedback Using Learning Dashboards

E-learning is increasingly used to support student learning in higher education, facilitating administration of online formative assessments. Although providing diagnostic, actionable feedback is generally more effective, in current practice, feedback is often given in the form of a simple proportion of correctly solved items. This study shows the validation process of constructing detailed diagnostic information on a set of skills, abilities, and cognitive processes (so-called attributes) from students’ item response data with diagnostic classification models. Attribute measurement in the domain of statistics education is validated based on both expert judgment and empirical student data from a think-aloud study and large-scale assessment administration. The constructed assessments provide a valid and reliable measurement of the attributes. Inferences that can be drawn from the results of these formative assessments are discussed and it is demonstrated how this information can be communicated to students via learning dashboards to allow them to make more effective learning choices.

Reduced Infection Rate After Aseptic Revision Total Knee Arthroplasty With Extended Oral Antibiotic Protocol

BACKGROUND

The optimal postoperative antibiotic duration has not been determined for aseptic revision total knee arthroplasty (R-TKA) where the risk of periprosthetic joint infection (PJI) is 3%-7.5%. This study compared PJI rates in aseptic R-TKA performed with extended oral antibiotic prophylaxis (EOAP) to published rates.

METHODS

Aseptic R-TKAs consecutively performed between 2013 and 2017 at a tertiary care referral center in the American Midwest were retrospectively reviewed. All patients were administered intravenous antibiotics while hospitalized and discharged on 7-day oral antibiotic prophylaxis. Infection rates and antibiotic-related complications were assessed.

RESULTS

Sixty-seven percent of the 176 analysis patients were female, with an average age of 64 years and body mass index of 35 kg/m². Instability and aseptic loosening comprised 86% of revision diagnoses. Overall, 87.5% of intraoperative cultures were negative, and the remainder were single positive cultures considered contaminants. PJI rates were 0% at 90 days, 1.8% (95% confidence interval 0.4%-5.3%) at 1 year, and 2.2% (95% confidence interval 0.6%-5.7%) at mean follow-up of approximately 3 years (range, 7-65 months).

CONCLUSION

EOAP after aseptic R-TKA resulted in a PJI rate equivalent to primary TKA, representing a 2- to-4-fold decrease compared with published aseptic R-TKA infection rates. Further study on the benefits and costs of EOAP after aseptic R-TKA is encouraged.

Statistics decrypted-a comprehensive review and smartphone-assisted five-step approach for good statistical practice

Background

Statistic scripts are often made by mathematicians and cryptic for clinicians or non-mathematician scientists. Nevertheless, almost all research projects necessitate the application of some statistical tests or at least an understanding thereof. The present review aims on giving an overview of the most common statistical terms and concepts. It further ensures good statistical practice by providing a five-step approach guiding the reader to the correct statistical test.

Methods and results

First, different types of variables and measurements to describe a data set with means of descriptive statistics are introduced. The basic thoughts and tools of interferential statistics are presented, and different types of bias are discussed. Then in the final paragraph, the most commonly used statistical tests are described. A smartphone app accessible via QR code finally guides the reader in five steps to the correct statistical test, depending on the data used in order to avoid commonly performed mistakes.

Conclusions

The five-step approach sets a new minimal standard for good statistical practice.

Interpreting Patient-Reported Outcome Measures in Orthopaedic Surgery: A Systematic Review

BACKGROUND

The Consolidated Standards of Reporting Trials (CONSORT) Statement recommends that studies report results beyond p values and include treatment effect(s) and measures of precision (e.g., confidence intervals [CIs]) to facilitate the interpretation of results. The objective of this systematic review was to assess the reporting and interpretation of patient-reported outcome measure (PROM) results in clinical studies from high-impact orthopaedic journals, to determine the proportion of studies that (1) only reported a p value; (2) reported a treatment effect, CI, or minimal clinically important difference (MCID); and (3) offered an interpretation of the results beyond interpreting a p value.

METHODS

We included studies from 5 high-impact-factor orthopaedic journals published in 2017 and 2019 that compared at least 2 intervention groups using PROMs.

RESULTS

A total of 228 studies were analyzed, including 126 randomized controlled trials, 35 prospective cohort studies, 61 retrospective cohort studies, 1 mixed cohort study, and 5 case-control studies. Seventy-six percent of studies (174) reported p values exclusively to express and interpret between-group differences, and only 22.4% (51) reported a treatment effect (mean difference, mean change, or odds ratio) with 95% CI. Of the 54 studies reporting a treatment effect, 31 interpreted the results using an important threshold (MCID, margin, or Cohen d), but only 3 interpreted the CIs. We found an absolute improvement of 35.5% (95% CI, 20.8% to 48.4%) in the reporting of the MCID between 2017 and 2019.

CONCLUSIONS

The majority of interventional studies reporting PROMs do not report CIs around between-group differences in outcome and do not define a clinically meaningful difference. A p value cannot effectively communicate the readiness for implementation in a clinical setting and may be misleading. Thus, reporting requirements should be expanded to require authors to define and provide a rationale for between-group clinically important difference thresholds, and study findings should be communicated by comparing CIs with these thresholds.

Hdac3 deletion in myeloid progenitor cells enhances bone healing in females and limits osteoclast fusion via Pmepa1

Previous studies examining the role of the histone deacetylase Hdac3 within myeloid cells demonstrated that Hdac3 promotes M2 activation and tissue healing in inflammatory conditions. Since myeloid lineage cells are required for proper bone formation and regeneration, in this study we examined the functions of Hdac3 during bone healing. Conditional deletion of Hdac3 within myeloid progenitors accelerates healing of cortical bone defects. Moreover, reduced osteoclast numbers within the defect site are correlated with Hdac3 suppression. Ex vivo osteoclastogenesis assays further demonstrate that Hdac3 deficiency limits osteoclastogenesis, the number of nuclei per cell and bone resorption, suggesting a defect in cell fusion. High throughput RNA sequencing identified the transmembrane protein Pmepa1 as a differentially expressed gene within osteoclast progenitor cells. Knockdown of Pmepa1 partially restores defects in osteoclastogenesis induced by Hdac3 deficiency. These results show that Hdac3 is required for optimal bone healing and osteoclast fusion, potentially via its regulation of Pmepa1 expression.

What Are the Implications of Alternative Alpha Thresholds for Hypothesis Testing in Orthopaedics?

BACKGROUND

Clinical research in orthopaedics typically reports the presence of an association after rejecting a null hypothesis of no association using an alpha threshold of 0.05 at which to evaluate a calculated p value. This arbitrary value is a factor that results in the current difficulties reproducing research findings. A proposal is gaining attention to lower the alpha threshold to 0.005. However, it is currently unknown how alpha thresholds are used in orthopaedics and the distribution of p values reported.

QUESTIONS/PURPOSES

We sought to describe the use of alpha thresholds in two orthopaedic journals by asking (1) How frequently are alpha threshold values reported? (2) How frequently are power calculations reported? (3) How frequently are p values between 0.005 and 0.05 reported for the main hypothesis? (4) Are p values less than 0.005 associated with study characteristics such as design and reporting power calculations?

METHODS

The 100 most recent original clinical research articles from two leading orthopaedic journals at the time of this proposal were reviewed. For studies without a specified primary hypothesis, a main hypothesis was selected that was most consistent with the title and abstract. The p value for the main hypothesis and lowest p value for each study were recorded. Study characteristics including details of alpha thresholds, beta, and p values were recorded. Associations between study characteristics and p values were described. Of the 200 articles (100 from each journal), 23 were randomized controlled trials, 141 were cohort studies or case series (defined as a study in which authors had access to original data collected for the study purpose), 31 were database studies, and five were classified as other.

RESULTS

An alpha threshold was reported in 166 articles (83%) with all but two reporting a value 0.05. Forty-two articles (21%) reported performing a power calculation. The p value for the main hypothesis was less than 0.005 for 88 articles (44%), between 0.05 and 0.005 for 67 (34%), and greater than 0.05 for 29 (15%). The smallest p value was between 0.05 and 0.005 for 39 articles (20%), less than 0.005 for 143 (72%), and either not provided or greater than 0.05 for 18 (9%). Although 50% (65 of 130) cohort and database papers had a main hypothesis p value less than 0.005, only 26% (6 of 23) randomized controlled trials did. Only 36% (15 of 42) articles reporting a power calculation had a p value less than 0.005 compared with 51% (73 of 142) that did not report one.

CONCLUSIONS

Although a lower alpha threshold may theoretically increase the reproducibility of research findings across orthopaedics, this would preferentially select findings from lower-quality studies or increase the burden on higher quality ones. A more-nuanced approach could be to consider alpha thresholds specific to study characteristics. For example, randomized controlled trials with a prespecified primary hypothesis may still be best evaluated at 0.05 while database studies with an abundance of statistical tests may be best evaluated at a threshold even below 0.005.

CLINICAL RELEVANCE

Surgeons and scientists in orthopaedics should understand that the default alpha threshold of 0.05 represents an arbitrary value that could be lowered to help reduce type-I errors; however, it must also be appreciated that such a change could increase type-II errors, increase resource utilization, and preferentially select findings from lower-quality studies.

Bias in Prospective Research and How to Avoid it

Prospective research studies answer clinical questions via randomized control studies or prospective cohort studies. Bias is defined as any deviation from the truth when interpreting data. Broadly speaking, bias is categorized into 3 categories: selection bias, information bias, and confounding bias. Furthermore, misinterpretation of presented statistics in research studies causes acceptance of conclusions which may not necessarily be true. This paper identifies sources of bias in prospective research and instructions to avoid bias.