An assessment of the methodological quality of research published in The American Journal of Sports Medicine

Assessment of 30 Years of Randomized Controlled Trials in The American Journal of Sports Medicine: 1990-2020

Background

Randomized controlled trials (RCTs) stand atop the evidence-based hierarchy of study designs for their ability to arrive at results with the lowest risk of bias. Even for RCTs, however, critical appraisal is essential before applying results to clinical practice.

Purpose

To analyze the quality of reporting of RCTs published in The American Journal of Sports Medicine (AJSM) from 1990 to 2020 and to identify trends over time and areas of improvement for future trials.

Study Design

Systematic review; Level of evidence, 1.

Methods

We queried the AJSM database for RCTs published between January 1990 and December 2020. Data pertaining to study characteristics were recorded. Quality assessments were conducted using the Detsky quality-of-reporting index and the modified Cochrane risk-of-bias (mROB) tool. Univariate and multivariable models were generated to establish factors with associations to study quality. The Fragility Index was calculated for eligible studies.

Results

A total of 277 RCTs were identified with a median sample size of 70 patients. A total of 19 RCTs were published between 1990 and 2000 (t₁); 82 RCTs between 2001 and 2010 (t₂); and 176 RCTs between 2011 and 2020 (t₃). From t₁ to t₃, significant increases were observed in the overall mean-transformed Detsky score (from 68.2% ± 9.8% to 87.4% ± 10.2%, respectively; P < .001) and mROB score (from 4.7 ± 1.6 to 6.9 ± 1.6, respectively; P < .001). Multivariable regression analysis revealed that trials with follow-up periods of <5 years clearly stated primary outcomes, and a focus on the elbow, shoulder, or knee were associated with higher mean-transformed Detsky and mROB scores. The median Fragility Index was 2 (interquartile range, 0-5) for trials with statistically significant. Studies with small sample sizes (<100 patients) were more likely to have low Fragility Index scores and less likely to have a statistically significant finding in any outcome.

Conclusion

The quantity and quality of published RCTs published in AJSM increased over the past 3 decades. However, single-center trials with small sample sizes were prone to fragile results.

Update on the Methodological Quality of Research Published in The American Journal of Sports Medicine: Comparing 2011-2013 to 10 and 20 Years Prior

BACKGROUND

In recent years, the number of articles in The American Journal of Sports Medicine (AJSM) has risen dramatically, with an increasing emphasis on evidence-based medicine in orthopaedics and sports medicine.

HYPOTHESIS

Despite the increase in the number of articles published in AJSM over the past decade, the methodological quality of articles in 2011-2013 has improved relative to those in 2001-2003 and 1991-1993.

STUDY DESIGN

Meta-analysis.

METHODS

All articles published in AJSM during 2011-2013 were reviewed and classified by study design. For each article, the use of pertinent methodologies, such as prospective data collection, randomization, control groups, and blinding, was recorded. The frequency of each article type and the use of evidence-based techniques were compared relative to 1991-1993 and 2001-2003 by use of Pearson χ(2) testing.

RESULTS

The number of research articles published in AJSM more than doubled from 402 in 1991-1993 and 423 in 2001-2003 to 953 in 2011-2013. Case reports decreased from 15.2% to 10.6% to 2.1% of articles published over the study period (P < .001). Cadaveric/human studies and meta-analysis/literature review studies increased from 5.7% to 7.1% to 12.4% (P < .001) and from 0.2% to 0.9% to 2.3% (P = .01), respectively. Randomized, prospective clinical trials increased from 2.7% to 5.9% to 7.4% (P = .007). Fewer studies used retrospective compared with prospective data collection (P < .001). More studies tested an explicit hypothesis (P < .001) and used controls (P < .001), randomization (P < .001), and blinding of those assessing outcomes (P < .001). Multi-investigator trials increased (P < .001), as did the proportion of articles citing a funding source (P < .001).

CONCLUSION

Despite a dramatic increase in the number of published articles, the research published in AJSM shifted toward more prospective, randomized, controlled, and blinded designs during 2011-2013 compared with 2001-2003 and 1991-1993, demonstrating a continued improvement in methodological quality.

Case-control studies in neurosurgery

OBJECT

Observational studies, such as cohort and case-control studies, are valuable instruments in evidence-based medicine. Case-control studies, in particular, are becoming increasingly popular in the neurosurgical literature due to their low cost and relative ease of execution; however, no one has yet systematically assessed these types of studies for quality in methodology and reporting.

METHODS

The authors performed a literature search using PubMed/MEDLINE to identify all studies that explicitly identified themselves as "case-control" and were published in the JNS Publishing Group journals (Journal of Neurosurgery, Journal of Neurosurgery: Pediatrics, Journal of Neurosurgery: Spine, and Neurosurgical Focus) or Neurosurgery. Each paper was evaluated for 22 descriptive variables and then categorized as having either met or missed the basic definition of a case-control study. All studies that evaluated risk factors for a well-defined outcome were considered true case-control studies. The authors sought to identify key features or phrases that were or were not predictive of a true case-control study. Those papers that satisfied the definition were further evaluated using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist.

RESULTS

The search detected 67 papers that met the inclusion criteria, of which 32 (48%) represented true case-control studies. The frequency of true case-control studies has not changed with time. Use of odds ratios (ORs) and logistic regression (LR) analysis were strong positive predictors of true case-control studies (for odds ratios, OR 15.33 and 95% CI 4.52-51.97; for logistic regression analysis, OR 8.77 and 95% CI 2.69-28.56). Conversely, negative predictors included focus on a procedure/intervention (OR 0.35, 95% CI 0.13-0.998) and use of the word "outcome" in the Results section (OR 0.23, 95% CI 0.082-0.65). After exclusion of nested case-control studies, the negative correlation between focus on a procedure/intervention and true case-control studies was strengthened (OR 0.053, 95% CI 0.0064-0.44). There was a trend toward a negative association between the use of survival analysis or Kaplan-Meier curves and true case-control studies (OR 0.13, 95% CI 0.015-1.12). True case-control studies were no more likely than their counterparts to use a potential study design "expert" (OR 1.50, 95% CI 0.57-3.95). The overall average STROBE score was 72% (range 50-86%). Examples of reporting deficiencies were reporting of bias (28%), missing data (55%), and funding (44%).

CONCLUSIONS

The results of this analysis show that the majority of studies in the neurosurgical literature that identify themselves as "case-control" studies are, in fact, labeled incorrectly. Positive and negative predictors were identified. The authors provide several recommendations that may reverse the incorrect and inappropriate use of the term "case-control" and improve the quality of design and reporting of true case-control studies in neurosurgery.

Using levels of evidence to compare clinical impact from research

OBJECTIVES

Impact of medical institutions on clinical decision-making globally might be estimated by the level of evidence of their research articles. The aim of this study was to compare levels of evidence of articles for Pakistan.

METHODS

We compared levels of evidence of articles from Pakistan, Nigeria, Japan, and the United States (U.S.).

RESULTS

Majority (73%) of articles in U.S. general medical journals were high levels (1-2), while majority (66% to 95%) in Japanese, Nigerian, Pakistani, and sub-specialty U.S. journals were lower levels (3-4) (P < 0.001). The number of articles from various regions of Pakistan did not correspond to their population or number of medical colleges/universities, mainly due to the skewing effect of one institution from Karachi which contributed 38% of all high-level articles.

CONCLUSIONS

A comparison of levels of evidence of articles across institutions might reflect relative potential of clinical impact, and might be useful for institutions, policy makers, and health research planners for priority setting.

Have levels of evidence improved the quality of orthopaedic research?

BACKGROUND

Since 2003 many orthopaedic journals have adopted grading systems for levels of evidence (LOE). It is unclear if the quality of orthopaedic literature has changed since LOE was introduced.

QUESTIONS/PURPOSES

We asked three questions: (1) Have the overall number and proportion of Level I and II studies increased in the orthopaedic literature since the introduction of LOE? (2) Is a similar pattern seen in individual orthopaedic subspecialty journals? (3) What is the interobserver reliability of grading LOE?

METHODS

We assigned LOE to therapeutic studies published in 2000, 2005, and 2010 in eight major orthopaedic subspecialty journals. Number and proportion of Level I and II publications were determined. Data were evaluated using log-linear models. Twenty-six reviewers (13 residents and 13 attendings) graded LOE of 20 blinded therapeutic articles from the Journal of Bone and Joint Surgery for 2009. Interobserver agreement relative to the Journal of Bone and Joint Surgery was assessed using a weighted kappa.

RESULTS

The total number of Level I and II publications in subspecialty journals increased from 150 in 2000 to 239 in 2010. The proportion of high-quality publications increased with time (p < 0.001). All subspecialty journals other than the Journal of Pediatric Orthopaedics and the Journal of Orthopaedic Trauma showed a similar behavior. Average weighted kappa was 0.791 for residents and 0.842 for faculty (p = 0.209).

CONCLUSIONS

The number and proportion of Level I and II publications have increased. LOE can be graded reliably with high interobserver agreement. The number and proportion of high-level studies should continue to increase.

Treating the Initial Anterior Shoulder Dislocation—An Evidence-based Medicine Approach

This manuscript presents the best available evidence to answer questions regarding the treatment of the patient with an initial anterior shoulder dislocation. The highest levels of evidence available offer the following conclusions: (1) of the many methods to reduce the dislocated shoulder, little data exist to identify the best method. Recommendations are based on low levels of evidence (levels 4 and 5). (2) Premedication with intra-articular lidocaine has fewer complications and requires a shorter time in the emergency room than intravenous sedation with no detectable differences in reduction success rates (level 1). (3) Postreduction immobilization in external rotation may reduce recurrence (level 2), but immobilization in internal rotation does not (level 1). (4) Arthroscopic surgery significantly reduces recurrence compared to a nonoperative approach (level 1), and (5) there are limited data on features that would allow a safe return to play. Expert opinion suggests that return is allowed when motion and strength are nearly normal, and the athletes can engage in sport-specific activities, however, the athlete is at risk for recurrence while playing (levels 4 and 5).