A systematic review of orthopaedic manual therapy randomized clinical trials quality

Disagreements in risk of bias assessment for randomized controlled trials in hypertension-related Cochrane reviews

BACKGROUND

The inter-reviewer reliability of the risk of bias (RoB) assessment lacked agreement in previous studies. It is important to analyse these disagreements to improve the repeatability of RoB assessment. The objective of the study was to evaluate the frequency and reasons for disagreements in RoB assessments for randomised controlled trials (RCTs) that were included in multiple Cochrane reviews in the field of hypertension.

METHODS

A cross-sectional study was employed. We retrieved any RCTs that had been included in multiple Cochrane reviews in the field of hypertension from ARCHIE. The results of the RoB assessments were extracted, and the distributions of agreements and possible reasons for disagreement were analyzed.

RESULTS

Twenty-six Cochrane reviews were included in this study. A total of 78 RCTs appeared in more than one Cochrane review. The level of agreement ranged from domain to domain. "Blinding of outcome assessment" showed a reasonably high level of agreement (94.9%), while "incomplete outcome data", "selective outcome reporting" and "other sources of bias" showed moderate levels of agreement (74.6%, 79.2% and 75.6%, respectively). However, the domains of "allocation concealment", "random sequence generation" and "blinding of participants and personnel" showed low levels of agreement (24.4%, 23.5%, and 47.4%, respectively). In the domains of "allocation concealment" and "blinding of participants and personnel", the agreement group had higher proportion of publication year ≤ 1996 than the disagreement group (P = 0.008 and P < 0.001, respectively). In the "blinding of participants and personnel", the impact factor was higher in the agreement group (P < 0.001). By analyzing the support text, we found that the most likely reason for disagreement was extracting different information from the same RCT.

CONCLUSION

For Cochrane reviews in the field of hypertension using the 2011 version of the RoB tool, there was a large disagreement in the RoB assessment. It is suggested that the results of RoB assessments in systematic reviews that used the 2011 version of the RoB tool need to be interpreted with caution. More accurate information from RCTs needs to be collected when we synthesize clinical evidence.

Is there 'trustworthy' evidence for using manual therapy to treat patients with shoulder dysfunction?: A systematic review

The primary objective of this review was to create a 'trustworthy,' living systematic review and meta-analysis for the application of manual therapy interventions in treating patients with shoulder dysfunction. Included studies were English-language randomized controlled trials published between 1/1/2010 and 8/3/2023, with searches performed in: PubMed, Cochrane Central Register of Controlled Trials (CENTRAL), CINHAL, ProQuest Nursing & Allied Health, EBSCO Medline, and PEDro. The population of focus included adults 18 years and older with musculoskeletal impairments related to shoulder dysfunction. Our primary outcomes included pain and region-specific outcome measures. We excluded trials, including participants having shoulder dysfunction resulting from surgery, radicular pain, instability/dislocation, fracture, lymphedema, and radiation. Our screening methodology was based upon a previously published 'trustworthy' systematic review protocol. This included the application of our PICOTS criteria in addition to screening for prospective clinical trial registration and following of prospective intent, as well as assessment of PEDro scores, risk-of-bias ratings, GRADE scoring, and examination of confidence in estimated effects. Twenty-six randomized controlled trials met our PICOTS criteria; however, only 15 of these were registered. Only three were registered prospectively. Two of these did not have discussions and conclusions that aligned with their primary outcome. The remaining single study was found to have a high risk-of-bias, meaning the remainder of the protocol could not be employed and that no randomized controlled trials could undergo further assessment or meta-analysis. The results of this systematic review indicate there are no 'trustworthy' randomized controlled trials examining the effectiveness of manual therapy interventions for the treatment of patients with shoulder dysfunction, as defined by the prospectively established methodology. Therefore, these findings signal that creating a 'trustworthy,' living systematic review on this clinically relevant topic is not yet possible due to a lack of 'trustworthy' randomized controlled trials.

Reporting quality in preclinical animal experimental research in 2009 and 2018: A nationwide systematic investigation

Lack of translation and irreproducibility challenge preclinical animal research. Insufficient reporting methodologies to safeguard study quality is part of the reason. This nationwide study investigates the reporting prevalence of these methodologies and scrutinizes the reported information’s level of detail. Publications were from two time periods to convey any reporting progress and had at least one author affiliated to a Danish University. We retrieved all relevant animal experimental studies using a predefined research protocol and a systematic search. A random sampling of 250 studies from 2009 and 2018 led to 500 publications in total. Reporting of measures known to impact study results estimates were assessed. Part I discloses a simplified two-level scoring “yes/no” to identify the presence of reporting. Part II demonstrates an additional three-level scoring to analyze the reported information’s level of detail. Overall reporting prevalence is low, although minor improvements are noted. Reporting of randomization increased from 24.0% in 2009 to 40.8% in 2018, blinded experiment conduct from 2.4% to 4.4%, blinded outcome assessment from 23.6% to 38.0%, and sample size calculation from 3.2% to 14.0%. Poor reporting of details is striking with reporting of the random allocation method to groups being only 1.2% in 2009 and 6.0% in 2018. Reporting of sample size calculation method was 2.4% in 2009 and 7.6% in 2018. Only conflict-of-interest statements reporting increased from 37.6% in 2009 to 90.4%. Measures safeguarding study quality are poorly reported in publications affiliated with Danish research institutions. Only a modest improvement was noted during the period 2009–2018, and the lack of details urgently prompts institutional strategies to accelerate this. We suggest thorough teaching in designing, conducting and reporting animal studies. Education in systematic review methodology should be implemented in this training and will increase motivation and behavior working towards quality improvements in science.

Do Prospective Intent and Established Metrics Correlate with Journal Impact Factor in Musculoskeletal Physical Therapy Trials?: A Secondary Analysis of A Methodological Review

OBJECTIVES

To determine if there are any statistically significant associations between: 1) randomized clinical trials (RCTs) investigating physical therapy musculoskeletal interventions, 2) journal impact factor (JIF), 3) frequency of RCT citation, 4) whether prospective intent was identifiable, and 5) the Physiotherapy Evidence Database (PEDro) scores.

METHODS

MEDLINE indexed RCTs addressing musculoskeletal interventions published between January 2016 and July 2020 in physical therapy journals were included. Two blinded reviewers identified the RCTs and extracted the variables of interest.

RESULTS

With a familywise alpha adjustment, there was no statistically significant correlation between JIF and number of citations (rho = 0.187; p = 0.0280). Statistically significant weak positive correlations were identified between the JIF and prospectively registered RCTs (rho = 0.240; p = 0.0046), JIF and PEDro scores (rho = 0.250; p = 0.0031), and PEDro scores and prospectively registered RCTs (rho = 0.335; p < 0.0001).

CONCLUSION

The findings of this study suggest that JIF and PEDro scores may not be accurate measures of RCT quality. Failing to ensure that published RCTs followed their prospective intent and using bibliometrics that fail to accurately measure what they propose appears to create untrustworthy preprocessed resources for practicing physical therapists during the evidence-based practice process.

LEVEL OF EVIDENCE

1a.

Comparison between 2000 and 2018 on the reporting of statistical significance and clinical relevance in physiotherapy clinical trials in six major physiotherapy journals: a meta-research design

DESIGN

Meta-research.

OBJECTIVE

To compare the prevalence of reporting p values, effect estimates and clinical relevance in physiotherapy randomised controlled trials (RCTs) published in the years 2000 and 2018.

METHODS

We performed a meta-research study of physiotherapy RCTs obtained from six major physiotherapy peer-reviewed journals that were published in the years 2000 and 2018. We searched the databases Embase, Medline and PubMed in May 2019, and extracted data on the study characteristics and whether articles reported on statistical significance, effect estimates and confidence intervals for baseline, between-group, and within-group differences, and clinical relevance. Data were presented using descriptive statistics and inferences were made based on proportions. A 20% difference between 2000 and 2018 was regarded as a meaningful difference.

RESULTS

We found 140 RCTs: 39 were published in 2000 and 101 in 2018. Overall, there was a high prevalence (>90%) of reporting p values for the main (between-group) analysis, with no difference between years. Statistical significance testing was frequently used for evaluating baseline differences, increasing from 28% in 2000 to 61.4% in 2018. The prevalence of reporting effect estimates, CIs and the mention of clinical relevance increased from 2000 to 2018 by 26.6%, 34% and 32.8% respectively. Despite an increase in use in 2018, over 40% of RCTs failed to report effect estimates, CIs and clinical relevance of results.

CONCLUSION

The prevalence of using p values remains high in physiotherapy research. Although the proportion of reporting effect estimates, CIs and clinical relevance is higher in 2018 compared to 2000, many publications still fail to report and interpret study findings in this way.

Do manual therapies have a specific autonomic effect? An overview of systematic reviews

Background

The impact of manual therapy interventions on the autonomic nervous system have been largely assessed, but with heterogeneous findings regarding the direction of these effects. We conducted an overview of systematic reviews to describe if there is a specific autonomic effect elicited by manual therapy interventions, its relation with the type of technique used and the body region where the intervention was applied.

Methods

We conducted an overview according to a publicly registered protocol. We searched the Cochrane Database of Systematic Reviews, MEDLINE, EPISTEMONIKOS and SCOPUS, from their inception to march 2021. We included systematic reviews for which the primary aim of the intervention was to assess the autonomic effect elicited by a manual therapy intervention in either healthy or symptomatic individuals. Two authors independently applied the selection criteria, assessed risk of bias from the included reviews and extracted data. An established model of generalisation guided the data analysis and interpretation.

Results

We included 12 reviews (5 rated as low risk of bias according the ROBIS tool). The findings showed that manual therapies may have an effect on both sympathetic and parasympathetic systems. However, the results from included reviews were inconsistent due to differences in their methodological rigour and how the effects were measured. The reviews with a lower risk of bias could not discriminate the effects depending on the body region to which the technique was applied.

Conclusion

The magnitude of the specific autonomic effect elicited by manual therapies and its clinical relevance is uncertain. We point out some specific recommendations in order to improve the quality and relevance of future research in this field.

Effectiveness of manual therapy in patients with distal radius fracture: a systematic review and meta-analysis

OBJECTIVE

To determine the effectiveness of manual therapy (MT) for functional outcomes in patients with distal radius fracture (DRF).

METHODS

An electronic search was performed in the Medline, Central, Embase, PEDro, Lilacs, CINAHL, SPORTDiscus, and Web of Science databases. The eligibility criteria for selecting studies included randomized clinical trials that included MT techniques with or without other therapeutic interventions in functional outcomes, such as wrist or upper limb function, pain, grip strength, and wrist range of motion in patients older than 18 years with DRF.

RESULTS

Eight clinical trials met the eligibility criteria; for the quantitative synthesis, six studies were included. For supervised physiotherapy plus joint mobilization versus home exercise program at 6 weeks follow-up, the mean difference (MD) for wrist flexion was 7.1 degrees (p = 0.20), and extension was 11.99 degrees (p = 0.16). For exercise program plus mobilization with movement versus exercise program at 12 weeks follow-up, the PRWE was -10.2 points (p = 0.02), the DASH was -9.86 points (p = 0.0001), and grip strength was 3.9 percent (p = 0.25). For conventional treatment plus manual lymph drainage versus conventional treatment, for edema the MD at 3-7 days was -14.58 ml (p = 0.03), at 17-21 days -17.96 ml (p = 0.009), at 33-42 days -15.34 ml (p = 0.003), and at 63-68 days -13.97 ml (p = 0.002).

CONCLUSION

There was very low to high evidence according to the GRADE rating. Adding mobilization with movement and manual lymphatic drainage showed statistically significant differences in wrist, upper limb function, and hand edema in patients with DRF.

Quality of reporting and risk of bias: a review of randomised trials in occupational health

Objectives

To assess the reporting quality of randomisation and allocation methods in occupational health and safety (OHS) trials in relation to Consolidated Standards of Reporting Trials (CONSORT) requirements of journals, risk of bias (RoB) and publication year.

Methods

We systematically searched for randomised controlled trials (RCTs) in PubMed between 2010 and May 2019 in 18 OHS journals. We measured reporting quality as percentage compliance with the CONSORT 2010 checklist (items 8–10) and RoB with the ROB V.2.0 tool (first domain). We tested the mean difference (MD) in % in reporting quality between CONSORT-requiring and non-requiring journals, trials with low, some concern and high RoB and publications before and after 2015.

Results

In 135 articles reporting on 129 RCTs, average reporting quality was at 37.4% compliance (95% CI 31.9% to 43.0%), with 10% of articles reaching 100% compliance. Reporting quality was significantly better in CONSORT-requiring journals than non-requiring journals (MD 31.0% (95% CI 21.4% to 40.7%)), for studies at low RoB than high RoB (MD 33.1% (95% CI 16.1% to 50.2%)) and with RoB of some concern (MD 39.8% (95% CI 30.0% to 49.7%)). Reporting quality did not improve over time (MD −5.7% (95% CI −16.8% to 5.4%).

Conclusions

Articles in CONSORT-requiring journals and of low RoB studies show better reporting quality. Low reporting quality is linked to unclear RoB judgements (some concern). Reporting quality did not improve over the last 10 years and CONSORT is insufficiently implemented. Concerted efforts by editors and authors are needed to improve CONSORT implementation.

Should low-quality evidence dominate high-level evidence? A systematic review and meta-analysis of systematic reviews of musculoskeletal physical therapy interventions

OBJECTIVES

To determine the: 1) quality of articles cited in systematic reviews (SRs); 2) methodological quality of the SRs; and 3) impact of quality on level 1A evidence.

METHODS

SRs related to musculoskeletal physical therapy interventions were identified. The methodological quality of the SRs and articles cited by the SRs were assessed by two blinded reviewers. Data analysis was performed by a third blinded researcher. Additional comparisons were made based on the Journal Impact Factor, spin, financial bias, and conflict of interest.

RESULTS

Twenty-four SRs were identified; 21/24 SRs had 'critically low' quality on the AMSTAR 2. Thirty-four percent of included studies were 'low quality,' and 58% of SRs included studies that had unreported external validity. One-half of the SRs represented 'spin,' and one-third of the SRs generated conclusions based on low-quality clinical trials.

DISCUSSION

The 'critically low' SRs methodological quality was exacerbated by low-quality research inclusion. Most SRs failed to follow best practices, including prospective registration and integration of professional librarians in the search process. Based on the high proportion of SRs that include low-quality trials and overall low methodological quality, further discussion regarding practice recommendations on level vs. quality of evidence is warranted.

LEVEL OF EVIDENCE

1a.

Sample size, study length, and inadequate controls were the most common self-acknowledged limitations in manual therapy trials: A methodological review

OBJECTIVES

The aim of this study was to quantify and analyze the presence and type of self-acknowledged limitations (SALs) in a sample of manual therapy (MT) randomized controlled trials.

STUDY DESIGN AND SETTING

We randomly selected 120 MT trials. We extracted data related to SALs from the original reports and classified them into 12 categories. After data extraction, specific limitations within each category were identified. A descriptive analysis was performed using frequencies and percentages for qualitative variables.

RESULTS

The number of SALs per trial article ranged from 0 to 8, and more than two-thirds of trials acknowledged at least two different limitations. Despite its small proportion, 9% of trials did not report SALs. The most common limitation declared, in almost half of our sample, related to sample size (47.5%) followed by limitations related to study length and follow-up (33.3%) and inadequate controls (32.5%).

CONCLUSION

Our results indicate that at least two different limitations are consistently acknowledged in MT trial reports, the most common being those related to sample size, study length, follow-up, and inadequate controls. Analysis of the reasons behind the SALs gives some insights about the main difficulties in conducting research in this field and may help develop strategies to improve future research.

The Therapeutic Benefits of Saline Solution Injection for Lateral Epicondylitis: A Meta-analysis of Randomized Controlled Trials Comparing Saline Injections With Nonsurgical Injection Therapies

PURPOSE

To quantify the effect of saline solution injections on patient-reported outcome measures (PROMs) and to determine whether this effect is clinically relevant by comparing it with minimal clinically important difference (MCID) criteria.

METHODS

A systematic search identified randomized controlled trials of lateral epicondylitis interventions comparing saline solution injections with nonsurgical injection therapies. Among included studies, saline solution was compared with platelet-rich plasma, autologous conditioned plasma, corticosteroid, and botulinum toxin injections. By use of data from included studies, a random-effects model was used to calculate overall mean differences (MDs) in pre- and post-injection PROMs in a pair-wise fashion. Calculated MDs were then compared with MCID criteria.

RESULTS

Of 458 identified studies, 10 met the inclusion criteria and encompassed 283 patients. At 1, 3, 6, and 12 months, statistically significant improvements in MDs in visual analog scale (VAS) scores were noted as follows: MD of 16.11 (95% confidence interval [CI], 8.29-23.93) at 1 month; MD of 22.50 (95% CI, 11.45-33.55) at 3 months; MD of 40.40 (95% CI, 27.48-53.32) at 6 months; and MD of 47.04 (95% CI, 39.43-54.66) at 12 months. At 6 months, Disabilities of the Arm, Shoulder and Hand scores showed a statistically significant improvement (MD, 23.92; 95% CI, 9.47-38.37).

CONCLUSIONS

Improvements in Disabilities of the Arm, Shoulder and Hand scores at 6 months (23.92) surpassed MCID criteria for conservatively managed upper-extremity musculoskeletal pathology (10.83)-suggesting that saline solution injections have a clinically relevant effect. VAS MCID criteria are poorly established, but VAS scores at 6 and 12 months surpassed MCID criteria for conservative treatments for common orthopaedic conditions. In all but 1 study, no statistically significant difference in PROMs was found between saline solution and non-saline solution injections.

LEVEL OF EVIDENCE

Level II, meta-analysis of Level I and II randomized controlled trials.

A methodological survey on reporting of pilot and feasibility trials for physiotherapy interventions: a study protocol

INTRODUCTION

Pilot and feasibility trials aim to test whether a full trial can be conducted or if any procedures must be changed for the full trial. Pilot trials must be reported in a transparent, accurate and complete way. In this report, we present a protocol for a methodological survey with the following aims: (1) to determine the percentage of physiotherapy trial reports which claim to be pilot or feasibility trials that evaluate feasibility, (2) to determine the aspect of feasibility evaluated in the primary objectives of the pilot or feasibility trials, (3) to describe the completeness of reporting of abstracts and full articles of pilot or feasibility trials using the Consolidated Standards of Reporting Trials (CONSORT) extension to randomised pilot and feasibility trials and (4) to investigate factors associated with completeness of reporting of pilot or feasibility trials.

METHODS AND ANALYSIS

Reports of randomised controlled trials indexed in the Physiotherapy Evidence Database (PEDro) that claim to be pilot or feasibility trials and published in 2011-2017 will be included. Two independent reviewers will confirm eligibility and classify the aspect of feasibility being evaluated in the objectives of the included pilot or feasibility trials. Completeness of reporting of both the abstract and the full article will be evaluated using the CONSORT extension to randomised pilot and feasibility trials. The primary analysis will be a descriptive analysis about the reporting quality of abstracts and full texts of pilot and feasibility trials. We will use generalised estimating equation analysis to explore factors associated with completeness of reporting.

ETHICS AND DISSEMINATION

The results of this study will be disseminated by presentation at conferences and will be submitted for publication in a peer-reviewed journal. Ethical approval is not necessary for this study.

Low reproducibility of randomized clinical trials methodology related to sampling: a systematic methodological review

Objectives: The reporting of sampling methods in Randomized Clinical Trials (RCTs) allows for research quality assessment, determination of sampling bias, and assures the presence of details necessary for reproducibility in future trials. The purpose of this study was to: (1) determine if sampling methodology was reproducible in RCTs related to musculoskeletal physical therapy (MSKPT) interventions to treat non-specific low back pain (NSLBP) and (2) establish if there was a relationship between sample reproducibility and established measures of research quality.Methods: Data were collected through a systematic review by a professional librarian. The identified RCTs were assessed for methodological quality by two blinded individual reviewers. Data analysis was performed by a third, blinded researcher; additional comparisons were made based on Journal Impact Factor and PEDro score.Results: Ninety-nine published peer-reviewed RCTs were identified that met inclusion criteria. Only 29% of the articles were judged to be reproducible based on the reported sampling methodology. There were meaningful correlations between two out of ten of the sampling reporting criteria and the judgement made if the sample was reported in significant detail to allow for replication. There was no relationship between sampling reporting criteria, Journal Impact Factors (JIFs), and Physiotherapy Evidence Database (PEDro) scores.Discussion: The reporting of sampling methodology needs to be considered to ensure reproducibility and avoid sampling bias. Despite the proliferation of measures of research quality, the overall reporting quality of RCTs continues to be inadequate to allow widespread reproducibility of trials.Level of Evidence: 1a.