An Assessment of Randomized Controlled Trial Quality in The Journal of Bone & Joint Surgery: Update from 2001 to 2013

Cryoneurolysis Associated With Improved Pain, Function, and Sleep in Patients Following total Knee Arthroplasty: Use of a New Real-World Registry

BACKGROUND

Total knee arthroplasty (TKA) is performed on approximately 790,000 patients annually in the United States and is projected to increase to 1.5 million by 2050. This study aimed at assessing the use of preoperative cryoneurolysis on patients undergoing TKA by analyzing: (1) pain severity; (2) opioid use; (3) functional status; and (4) sleep disturbance (SD) over 6 months following discharge.

METHODS

Patients enrolled in the Innovations in Genicular Outcomes Registry between September 2021 and February 2024 were followed for 6 months. Our analyses included patients undergoing unilateral primary TKA with no preoperative opioid prescription, who either received cryoneurolysis, or did not. Baseline patient demographics were collected before TKA and tabulated. Pain management was assessed via the Brief Pain Inventory-Short Form instrument for pain severity. SD was measured using the patient-reported outcomes measurement information system questionnaire. Each outcome measure was assessed prior to TKA, weekly, and at monthly follow-up. Data were analyzed by a generalized linear mixed-effect regression model to compare cryoneurolysis versus control patients, with a P < .05 as significant.

RESULTS

There were 80 patients who were treated with preoperative cryoneurolysis, while 60 control patients did not have treatment. Patients receiving cryoneurolysis experienced significantly lower pain severity and SD over the 6-month follow-up than control patients (P = .046). Cryoneurolysis was also associated with a trend toward greater functional improvement that did not reach statistical significance (P = .061). Further, patients who underwent cryoneurolysis were 72% less likely than control group patients to take opioids over 6 months following discharge (P < .001).

CONCLUSIONS

Preoperative cryoneurolysis therapy in opioid-naive patients undergoing TKA is associated with improved pain, decreased opioid use, and improved SD for 6 months postoperatively. Cryoneurolysis, a nonopioid pain relief modality administered preoperatively, demonstrated substantial benefits in patients who underwent TKA.

Improved Pain and Function With Triamcinolone Acetonide Extended-Release and Cryoneurolysis for Knee Osteoarthritis: Use of a New Real-World Registry

BACKGROUND

Knee osteoarthritis (OA) affects 19% of American adults aged more than 45 years and costs $27+ billion annually. A wide range of nonoperative treatment options are available. This study compared 6 treatments: cryoneurolysis with deep genicular nerve block (Cryo-Deep/Both), cryoneurolysis with superficial nerve block (Cryo-Superficial), intra-articular hyaluronic acid (IA-HA) injections, nonsteroidal anti-inflammatory drug injections (IA-NSAIDs), IA-corticosteroids (IA-CS) injections, or IA-triamcinolone extended release (IA-TA-ER) injections over 4 months for: (1) pain severity and analgesic use; and (2) physical function (from Knee Injury and Osteoarthritis Outcome Score for Joint Replacement).

METHODS

Patients who had unilateral knee OA and received nonoperative intervention were enrolled in the Innovations in Genicular Outcomes Research registry, a novel, multicenter real-world registry, between September 2021 and February 2024. A total of 480 patients were enrolled. Both pain and functional outcomes were assessed at baseline, weekly, and monthly, which were analyzed by overall trend, magnitude changes pretreatment to post-treatment, and distribution-based minimally clinically important difference (MCID) score. Multivariate linear regressions with adjustments for 7 confounding factors were used to compare follow-up outcomes among 6 treatment groups.

RESULTS

Use of IA-TA-ER injections was associated with the lowest pain, greatest pain reduction, and highest prevalence of patients achieving MCID relative to other treatments (P < .001). Deep/Both-Cryo and IA-CS were associated with a higher prevalence of achieving MCID than IA-HA, IA-NSAIDs, and Cryo-Superficial (P ≤ .001). Use of IA-TA-ER was also associated with the greatest functional score, improvement from baseline, and highest prevalence of patients achieving MCID than other treatments (P ≤ .003).

CONCLUSIONS

The IA-TA-ER appears to outperform other treatments in terms of pain relief and functional improvement for up to 4 months following treatment. In addition, outcomes in the novel cryoneurolysis and conventional IA-CS were similar to one another and better than those in IA-HA and IA-NSAIDs.

The Rising Quality of Randomized Controlled Trials in The Journal of Bone & Joint Surgery: An Updated Analysis from 2014 to 2022

Background

Previous reports found that 40% of randomized controlled trials (RCTs) published in The Journal of Bone & Joint Surgery (JBJS) from 1988 to 2000 and 47% of those published from 2001 to 2013 were of high quality. The purpose of this study was to assess the quality of RCTs published from 2014 to 2022 in JBJS and to compare these findings with those of prior analyses in order to identify trends over time and areas for continued improvement.

Methods

PubMed was searched for the term "randomized controlled trial" to identify studies published in JBJS from 2014 to 2022. Each included RCT was evaluated with use of the Detsky score and a risk-of-bias assessment modified from the Cochrane tool. These evaluations were then compared with previous evaluations of RCTs from the 1988 to 2000 and 2001 to 2013 periods with use of independent-sample t tests. A transformed Detsky score of >75% and a modified risk-of-bias score of ≥8 were defined as being indicative of high quality.

Results

A total of 218 RCTs were published in JBJS from 2014 to 2022. An a priori sample size was calculated in 183 studies (83.9%). A total of 152 (83.1%) of the 183 studies enrolled the calculated number of patients, of which 126 (82.9%) maintained an adequate number at the time of final follow-up. Most RCTs were conducted at a single center (146 of 218; 67%), evaluated a surgical intervention (162 of 218; 74%), and reported positive results (142 of 218; 65%). The mean transformed Detsky score was 85% ± 10% (95% confidence interval, 83.7% to 86.3%), with 82% of trials (179 of 218) scored as high quality. The mean transformed Detsky score from 2014 to 2022 was higher than that from 1988 to 2000 and that from 2001 to 2013 (85% versus 76% and 68%, respectively; p < 0.001). The mean modified risk-of-bias score was 7 ± 1, with 42% of trials (92 of 218) scored as high quality. RCTs published from 2014 to 2022 had a higher mean modified risk-of-bias score than those published from 2001 to 2013 (7 ± 1 versus 6 ± 1; p < 0.001). Compared with the 2001 to 2013 and 2014 to 2022 periods, the 1988 to 2000 period had a greater proportion of trials that reported positive results (51% and 65% versus 82%, respectively; p < 0.001) and that included data from multiple centers (31% and 33% versus 67%; p < 0.001).

Conclusions

The quality of RCTs published in JBJS from 2014 to 2022 has improved from that reported previously, as demonstrated by the increases in the modified risk-of-bias score and transformed Detsky score from prior periods. This may be the result of journal policies such as the requirements of CONSORT adherence and prospective trial registration. Investigators should focus on improving the clarity of reporting, limiting attrition bias, and making efforts to blind support staff in order to increase the quality of future RCTs.

Clinical Relevance

Improving the quality of RCTs is crucial given their potential to influence current clinical practice.

Explanatory and pragmatic trials in orthopaedics - Have we done the right studies?

Recent decades have seen marked advances in the quality of clinical orthopaedic trauma research, and with this has come a rise in the number of randomised clinical trials (RCTs) being conducted in orthopaedic trauma. These trials have been largely valuable in driving evidence-based management of injuries which previously had clinical equipoise. However, though RCTs are traditionally seen as the 'gold standard' of high-quality research, this research method is comprised primarily of two entities, explanatory and pragmatic designs, each with its own strengths and limitations. Most orthopaedic trials lie within a continuum between these designs, with varying degrees of both pragmatic and explanatory features. In this narrative review we provide a summary of the nuances within orthopaedic trial design, the advantages and limitations of such designs, and suggest tools which may aid clinicians in the appropriate selection and evaluation of trial designs.

Does Surgeon Level of Expertise Correlate with Patient Outcomes?

Background  It is difficult to evaluate the results of surgical techniques as there is inherent variability between surgeons in regard to experience, skill level, and knowledge. Tang suggested a classification system in 2009 in an attempt to standardize surgeon level of expertise, with categories ranging from nonspecialist (Level I) to expert (Level V). This epidemiological analysis of all articles citing Tang's original paper examines if a surgeon's self-reported level of expertise correlates with outcomes and evaluates whether the current definition of Tang level is sufficient to account for expertise bias. Methods  In May 2021, all articles citing Tang level of expertise were identified ( N  = 222). Articles were included if they described a novel technique and provided author(s)' levels ( n  = 205). Statistical analysis was conducted, and p -values less than 0.05 were considered significant. Results  The most common specialties reporting Tang level of expertise were orthopaedic surgery (82.9%) and plastic surgery (15.5%). The most common subspecialty was hand surgery. 2020 was the year with the most studies reporting level of expertise (31.7%), followed by 2021 (20.0%) and 2019 (17.1%). The majority of studies (80.5%) reported positive results with their technique, and of these, 63.3% were statistically significant. Level of expertise was not significantly associated with a doctoral degree, type of residency completed, fellowship completion, hand fellowship, author sex, study type, or result significance. Discussion  The current Tang classification is both underreported and incomplete in its present state. To account for expertise bias, we recommend all authors report Tang level when describing surgical techniques. Studies with multiple authors should explicitly state the level of each author, as well as a weighted average accounting for the total contribution of each individual.

Assessment of 30 Years of Randomized Controlled Trials in The American Journal of Sports Medicine: 1990-2020

Background

Randomized controlled trials (RCTs) stand atop the evidence-based hierarchy of study designs for their ability to arrive at results with the lowest risk of bias. Even for RCTs, however, critical appraisal is essential before applying results to clinical practice.

Purpose

To analyze the quality of reporting of RCTs published in The American Journal of Sports Medicine (AJSM) from 1990 to 2020 and to identify trends over time and areas of improvement for future trials.

Study Design

Systematic review; Level of evidence, 1.

Methods

We queried the AJSM database for RCTs published between January 1990 and December 2020. Data pertaining to study characteristics were recorded. Quality assessments were conducted using the Detsky quality-of-reporting index and the modified Cochrane risk-of-bias (mROB) tool. Univariate and multivariable models were generated to establish factors with associations to study quality. The Fragility Index was calculated for eligible studies.

Results

A total of 277 RCTs were identified with a median sample size of 70 patients. A total of 19 RCTs were published between 1990 and 2000 (t₁); 82 RCTs between 2001 and 2010 (t₂); and 176 RCTs between 2011 and 2020 (t₃). From t₁ to t₃, significant increases were observed in the overall mean-transformed Detsky score (from 68.2% ± 9.8% to 87.4% ± 10.2%, respectively; P < .001) and mROB score (from 4.7 ± 1.6 to 6.9 ± 1.6, respectively; P < .001). Multivariable regression analysis revealed that trials with follow-up periods of <5 years clearly stated primary outcomes, and a focus on the elbow, shoulder, or knee were associated with higher mean-transformed Detsky and mROB scores. The median Fragility Index was 2 (interquartile range, 0-5) for trials with statistically significant. Studies with small sample sizes (<100 patients) were more likely to have low Fragility Index scores and less likely to have a statistically significant finding in any outcome.

Conclusion

The quantity and quality of published RCTs published in AJSM increased over the past 3 decades. However, single-center trials with small sample sizes were prone to fragile results.

Spinal Orthoses in the Treatment of Osteoporotic Thoracolumbar Vertebral Fractures in the Elderly: A Systematic Review With Quantitative Quality Assessment

STUDY DESIGN

Systematic review.

OBJECTIVES

Spinal orthoses are frequently used to non-operatively treat osteoporotic vertebral fractures (OVF), despite the available evidence is rare. Previously systematic reviews were carried out, presenting controversial recommendations. The present study aimed to systematic review the recent and current literature on available evidence for the use of orthoses in OVF.

METHODS

A systematic review was conducted using PubMed, Medline, EMBASE and CENTRAL databases. Identified articles including previous systematic reviews were screened and selected by three authors. The results of retrieved articles were presented in a narrative form, quality assessment was performed by two authors using scores according to the study type.

RESULTS

Thirteen studies (n = 5 randomized controlled trials, n = 3 non- randomized controlled trials and n = 5 prospective studies without control group) and eight systematic reviews were analyzed. Studies without comparison group reported improvements in pain, function and quality of life during the follow-up. Studies comparing different types of orthoses favor non-rigid orthoses. In comparison to patients not wearing an orthosis three studies were unable to detect beneficial effects and two studies reported about a significant improvement using an orthosis. In the obtained quality assessment, three studies yielded good to excellent results. Previous reviews detected the low evidence for spinal orthoses but recommended them.

CONCLUSION

Based on the study quality and the affection of included studies in previous systematic reviews a general recommendation for the use of a spinal orthosis when treating OVF is not possible. Currently, no superiority for spinal orthoses in OVF treatment was found.

The Importance of Pragmatic Study Design to the Scholarly Influence of Surgical Hip Fracture Randomized Controlled Trials

BACKGROUND

Surgical randomized controlled trials (RCTs) have potential drawbacks, leading some to question their role in filling the information gap in orthopaedic surgery. Pragmatism in study design was introduced to increase the clinical applicability of study results. The purpose of this study was to examine how pragmatism affects the scholarly influence of surgical RCTs.

METHODS

A search for surgical hip fracture-related RCTs published between 1995 and 2015 was done. Journal impact factor, citation number, research question, significance and type of outcome, number of centers involved, and the Pragmatic-Explanatory Continuum Indicator Summary-2 level of pragmatism score were recorded for each study. Scholarly influence was estimated by a study's inclusion into orthopaedic literature or guidelines or through the study's average yearly citation rate.

RESULTS

One hundred sixty RCTs were included in the final analysis. A multivariate logistic regression identified large study sample size as the only predictor of an RCT being used in clinical guidance texts. Large sample size and multicenter RCTs were predictors of high yearly citation rates. The level of pragmatism in study design did not predict scholarly influence.

CONCLUSIONS

Pragmatic design is not independently associated with increased scholarly influence; however, large study sample size was the most important study characteristic affecting scholarly influence.

Few Randomized Controlled Trials in Spine Surgery in the United States Include Sociodemographic Patient Data: A Systematic Review

INTRODUCTION

The importance of sociodemographic factors such as race, education, and income on spine surgery outcomes has been well established, yet the representation of sociodemographic data within randomized controlled trials (RCTs) in spine literature remains undefined in the United States (U.S).

METHODS

Medical literature was reviewed within PubMed for RCTs with "spine" in the title or abstract published within the last 8 years (2014 to 2021) in seven major spine journals. This yielded 128 results, and after application of inclusion criteria (RCTs concerning adult spine pathologies conducted in the U.S), 54 RCTs remained for analysis. Each article's journal of publication, year of publication, and spinal pathology was recorded. Pathologies included cervical degeneration, thoracolumbar degeneration, adult deformity, cervical trauma, and thoracolumbar trauma. Sociodemographic variables collected were race, ethnicity, insurance status, income, work status, and education. The Fisher's exact test was used to compare inclusion of sociodemographic data by journal, year, and spinal pathology.

RESULTS

Sociodemographic data were included in the results and in any section of 57.4% (31/54) of RCTs. RCTs reported work status in 25.9% (14/54) of results and 38.9% (21/54) of RCTs included work status in any section. Income was included in the results and mentioned in any section in 13.0% (7/54) of RCTs. Insurance status was in the results or any section of 9.3% (5/54) and 18.5% (10/54) of RCTs, respectively. There was no association with inclusion of sociodemographic data within the results of RCTs as a factor of journal (P = 0.337), year of publication (P = 0.286), or spinal pathology (P = 0.199).

DISCUSSION

Despite evidence of the importance of sociodemographic factors on the natural history and treatment outcomes of myriad spine pathologies, this study identifies a surprising absence of sociodemographic data within contemporary RCTs in spine surgery. Failure to include sociodemographic factors in RCTs potentially bias the generalizability of outcome data.

Is Our Science Representative? A Systematic Review of Racial and Ethnic Diversity in Orthopaedic Clinical Trials from 2000 to 2020

BACKGROUND

A lack of racial and ethnic representation in clinical trials may limit the generalizability of the orthopaedic evidence base as it applies to patients in underrepresented minority populations and perpetuate existing disparities in use, complications, or functional outcomes. Although some commentators have implied the need for mandatory race or ethnicity reporting across all orthopaedic trials, the usefulness of race or ethnic reporting likely depends on the specific topic, prior evidence of disparities, and individualized study hypotheses.

QUESTIONS/PURPOSES

In a systematic review, we asked: (1) What proportion of orthopaedic clinical trials report race or ethnicity data, and of studies that do, how many report data regarding social covariates or genomic testing? (2) What trends and associations exist for racial and ethnic reporting among these trials between 2000 and 2020? (3) What is the racial or ethnic representation of United States trial participants compared with that reported in the United States Census?

METHODS

We performed a systematic review of randomized controlled trials with human participants published in three leading general-interest orthopaedic journals that focus on clinical research: The Journal of Bone and Joint Surgery, American Volume; Clinical Orthopaedics and Related Research; and Osteoarthritis and Cartilage. We searched the PubMed and Embase databases using the following inclusion criteria: English-language studies, human studies, randomized controlled trials, publication date from 2000 to 2020, and published in Clinical Orthopaedics and Related Research; The Journal of Bone and Joint Surgery, American Volume; or Osteoarthritis and Cartilage. Primary outcome measures included whether studies reported participant race or ethnicity, other social covariates (insurance status, housing or homelessness, education and literacy, transportation, income and employment, and food security and nutrition), and genomic testing. The secondary outcome measure was the racial and ethnic categorical distribution of the trial participants included in the studies reporting race or ethnicity. From our search, 1043 randomized controlled trials with 184,643 enrolled patients met the inclusion criteria. Among these studies, 21% (223 of 1043) had a small (< 50) sample size, 56% (581 of 1043) had a medium (50 to 200) sample size, and 23% (239 of 1043) had a large (> 200) sample size. Fourteen percent (141 of 1043) were based in the Northeast United States, 9.2% (96 of 1043) were in the Midwest, 4.7% (49 of 1043) were in the West, 7.2% (75 of 1043) were in the South, and 65% (682 of 1043) were outside the United States. We calculated the overall proportion of studies meeting the inclusion criteria that reported race or ethnicity. Then among the subset of studies reporting race or ethnicity, we determined the overall rate and distribution of social covariates and genomic testing reporting. We calculated the proportion of studies reporting race or ethnicity that also reported a difference in outcome by race or ethnicity. We calculated the proportion of studies reporting race or ethnicity by each year in the study period. We also calculated the proportions and 95% CIs of individual patients in each racial or ethnic category of the studies meeting the inclusion criteria.

RESULTS

During the study period (2000 to 2020), 8.5% (89 of 1043) of studies reported race or ethnicity. Of the trials reporting this factor, 4.5% (four of 89) reported insurance status, 15% (13 of 89) reported income, 4.5% (four of 89) reported housing or homelessness, 18% (16 of 89) reported education and literacy, 0% (0 of 89) reported transportation, and 2.2% (two of 89) reported food security or nutrition of trial participants. Seventy-eight percent (69 of 89) of trials reported no social covariates, while 22% (20 of 89) reported at least one. However, 0% (0 of 89) of trials reported genomic testing. Additionally, 5.6% (five of 89) of these trials reported a difference in outcomes by race or ethnicity. The proportion of studies reporting race or ethnicity increased, on average, by 0.6% annually (95% CI 0.2% to 1.0%; p = 0.02). After controlling for potentially confounding variables such as funding source, we found that studies with an increased sample size were more likely to report data by race or ethnicity; location in North America overall, Europe, Asia, and Australia or New Zealand (compared with the Northeast United States) were less likely to; and specialty-topic studies (compared with general orthopaedics research) were less likely to. Our sample of United States trials contained 18.9% more white participants than that reported in the United States Census (95% CI 18.4% to 19.4%; p < 0.001), 5.0% fewer Black participants (95% CI 4.6% to 5.3%; p < 0.001), 17.0% fewer Hispanic participants (95% CI 16.8% to 17.1%; p < 0.001), 5.3% fewer Asian participants (95% CI 5.2% to 5.4%; p < 0.001), and 7.5% more participants from other groups (95% CI 7.2% to 7.9%; p < 0.001).

CONCLUSION

Reporting of race or ethnicity data in orthopaedic clinical trials is low compared with other medical fields, although the proportion of diseases warranting this reporting might be lower in orthopaedics.

CLINICAL RELEVANCE

Investigators should initiate discussions about race and ethnicity reporting in the early stages of clinical trial development by surveying available published evidence for relevant health disparities, social determinants, and, when warranted, genomic risk factors. The decision to include or exclude race and ethnicity data in study protocols should be based on specific hypotheses, necessary statistical power, and an appreciation for unmeasured confounding. Future studies should evaluate cost-efficient mechanisms for obtaining baseline social covariate data and investigate researcher perspectives on current administrative workflows and decision-making algorithms for race and ethnicity reporting.

Common errors in the design of orthopaedic trials: Has anything changed?

INTRODUCTION

The adoption of evidence-based orthopaedics has shifted the focus from expert base opinions and anecdotal evidence to a focus on integrating the best available clinical research. This shift has led to an increased focus on randomized controlled trials (RCTs) within the field. Although RCTs are considered the highest level of evidence, methodologic errors can introduce bias and limit the validity of the results. Early trials were hampered by lack of blinding, inadequate sample sizes and other design flaws. The objective of this review was to examine the current literature to determine if the design and execution of RCTs has improved.

DESIGN ERRORS

The awareness of the importance of sample size increased over time with substantially more trials reporting sample size calculations. However, many contemporary RCTs are still underpowered and fail to reach their calculated sample size. Given the challenges of surgically based RCTs, the majority of historical trials lacked blinding, increasing the risk of bias. There is evidence that there has been a concerted effort to increase the blinding in RCTs, particularly in outcome assessors. A more recent development in the design of surgical trials is the introduction of expertise-based trial designs in which patients are randomized to a surgeon with expertise in a particular intervention. These trials minimize the bias that can arise from differential expertise bias and have the potential to improve the validity and feasibility of RCTs. Finally, there has been an increased focus on the reporting of patient reported outcomes (PROs) in orthopaedic RCTs. Alongside this movement has been the development of minimal important differences (MIDs) to define the changes that are relevant and meaningful to patients. Both PROs and MIDs should be taken into consideration when calculating the sample size and study power in clinical trials.

CONCLUSIONS

Although marked improvements have been made in the design and implementation of trials, there is still considerable room for improvement. Adequately blinded and powered studies evaluating clinically important outcomes and differences should be key considerations in trial design moving forward.

A review of trauma and orthopaedic randomised clinical trials published in high-impact general medical journals

Introduction

The recent past has seen a significant increase in the number of trauma and orthopaedic randomised clinical trials published in “the big five” general medical journals. The quality of this research has, however, not yet been established.

Methods

We therefore set out to critically appraise the quality of available literature over a 10-year period (April 2010–April 2020) through a systematic search of these 5 high-impact general medical journals (JAMA, NEJM, BMJ, Lancet and Annals). A standardised data extraction proforma was utilised to gather information regarding: trial design, sample size calculation, results, study quality and pragmatism. Quality assessment was performed using the Cochrane Risk of Bias 2 tool and the modified Delphi list. Study pragmatism was assessed using the PRECIS-2 tool.

Results

A total of 25 studies were eligible for inclusion. Over half of the included trials did not meet their sample size calculation for the primary outcome, with a similar proportion of these studies at risk of type II error for their non-significant results. There was a high degree of pragmatism according to PRECIS-2. Non-significant studies had greater pragmatism that those with statistically significant results (p < 0.001). Only 56% studies provided adequate justification for the minimum clinically important difference (MCID) in the population assessed. Overall, very few studies were deemed high quality/low risk of bias.

Conclusions

These findings highlight that there are some important methodological concerns present within the current evidence base of RCTs published in high-impact medical journals. Potential strategies that may improve future trial design are highlighted.

Level of evidence

Level 1.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00590-021-03137-3.