Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials

Guidance on how to efficiently find, choose, and use available systematic reviews was developed

OBJECTIVES

The aim of this paper is to provide clinicians and authors of clinical guidelines or patient information with practical guidance on searching and choosing systematic reviews(s) (SR[s]) and, where adequate, on making use of SR(s).

STUDY DESIGN AND SETTING

At the German conference of the Evidence-Based Medicine Network (EbM Network) a workshop on the topic was held to identify the most important areas where guidance for practice appears necessary. After the workshop, we established working groups. These included SR users with different backgrounds (eg, information specialists, epidemiologists) and working areas. Each working group developed and consented a draft guidance based on their expert knowledge and experiences. The results were presented to the entire group and finalized in an iterative process.

RESULTS

We developed a practical guidance that answers questions that usually arise when choosing and using SR(s). (1) How to efficiently find high-quality SRs? (2) How to choose the most appropriate SR? (3) What to do if no SR of sufficient quality could be identified? In addition, we developed an algorithm that links these steps and accounts for their interaction. The resulting guidance is primarily directed at clinicians and developers of clinical practice guidelines or patient information resources.

CONCLUSION

We suggest practical guidance for making the best use of SRs when answering a specific research question. The guidance may contribute to the efficient use of existing SRs. Potential benefits when using existing SRs should be always weighted against potential limitations.

Disagreements in risk of bias assessment for randomized controlled trials in hypertension-related Cochrane reviews

BACKGROUND

The inter-reviewer reliability of the risk of bias (RoB) assessment lacked agreement in previous studies. It is important to analyse these disagreements to improve the repeatability of RoB assessment. The objective of the study was to evaluate the frequency and reasons for disagreements in RoB assessments for randomised controlled trials (RCTs) that were included in multiple Cochrane reviews in the field of hypertension.

METHODS

A cross-sectional study was employed. We retrieved any RCTs that had been included in multiple Cochrane reviews in the field of hypertension from ARCHIE. The results of the RoB assessments were extracted, and the distributions of agreements and possible reasons for disagreement were analyzed.

RESULTS

Twenty-six Cochrane reviews were included in this study. A total of 78 RCTs appeared in more than one Cochrane review. The level of agreement ranged from domain to domain. "Blinding of outcome assessment" showed a reasonably high level of agreement (94.9%), while "incomplete outcome data", "selective outcome reporting" and "other sources of bias" showed moderate levels of agreement (74.6%, 79.2% and 75.6%, respectively). However, the domains of "allocation concealment", "random sequence generation" and "blinding of participants and personnel" showed low levels of agreement (24.4%, 23.5%, and 47.4%, respectively). In the domains of "allocation concealment" and "blinding of participants and personnel", the agreement group had higher proportion of publication year ≤ 1996 than the disagreement group (P = 0.008 and P < 0.001, respectively). In the "blinding of participants and personnel", the impact factor was higher in the agreement group (P < 0.001). By analyzing the support text, we found that the most likely reason for disagreement was extracting different information from the same RCT.

CONCLUSION

For Cochrane reviews in the field of hypertension using the 2011 version of the RoB tool, there was a large disagreement in the RoB assessment. It is suggested that the results of RoB assessments in systematic reviews that used the 2011 version of the RoB tool need to be interpreted with caution. More accurate information from RCTs needs to be collected when we synthesize clinical evidence.

The Benefits and Risks of Receiving Investigational Solid Tumor Drugs in Randomized Trials : A Systematic Review and Meta-analysis

BACKGROUND

Many patients participate in cancer trials to access new therapies. The extent to which new treatments produce clinical benefit for trial participants is unclear.

PURPOSE

To estimate the progression-free survival (PFS) and overall survival (OS) advantage of assignment to experimental groups in randomized trials for 6 solid tumors.

DATA SOURCES

ClinicalTrials.gov was searched for trials of investigational drugs with results posted between 2017 and 2021.

STUDY SELECTION

Investigational drugs were defined as those not yet having full approval from the U.S. Food and Drug Administration for the study indication. Trials were included if they were randomized and tested drugs or biologics.

DATA EXTRACTION

Data extraction was completed by 2 independent reviewers. Data were pooled using a random-effects model.

DATA SYNTHESIS

The sample included 128 trials comprising 141 comparisons of a new drug and a comparator. These comparisons included 47 050 patients. The pooled hazard ratio for PFS was 0.80 (95% CI, 0.75 to 0.85), indicating statistically significant benefit for patients in experimental groups. This corresponded to a median PFS advantage of 1.25 months (CI, 0.80 to 1.68 months). The pooled hazard ratio for OS was 0.92 (CI, 0.88 to 0.95), corresponding to a survival gain of 1.18 months (CI, 0.72 to 1.71 months). The absolute risk for a serious adverse event for comparator group patients was 29.56% (CI, 26.64% to 32.65%), with an increase in risk of 7.40% (CI, 5.66% to 9.14%) for patients in experimental groups.

LIMITATIONS

Trials in this sample were heterogeneous. Comparator group interventions were assumed to reflect standard of care.

CONCLUSION

Assignment to experimental groups produces statistically significant survival gains. However, the absolute survival gain is small, and toxicity is statistically significantly greater. The findings of this review provide reassuring evidence that patients are not meaningfully disadvantaged by assignment to comparator groups.

PRIMARY FUNDING SOURCE

Canadian Institutes of Health Research.

Variation observed in consensus judgments between pairs of reviewers when assessing the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research

OBJECTIVES

To evaluate the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research using the Risk Of Bias due to Missing Evidence (ROB-ME) tool and determine inter-rater agreement in assessments.

STUDY DESIGN AND SETTING

We assembled a random sample of 42 meta-analyses of nutrition research. Eight assessors were randomly assigned to one of four pairs. Each pair assessed 21 randomly assigned meta-analyses, and each meta-analysis was assessed by two pairs. We calculated raw percentage agreement and chance corrected agreement using Gwet's Agreement Coefficient (AC) in consensus judgments between pairs.

RESULTS

Across the eight signaling questions in the ROB-ME tool, raw percentage agreement ranged from 52% to 100%, and Gwet's AC ranged from 0.39 to 0.76. For the risk-of-bias judgment, the raw percentage agreement was 76% (95% confidence interval 60% to 92%) and Gwet's AC was 0.47 (95% confidence interval 0.14 to 0.80). In seven (17%) meta-analyses, either one or both pairs judged the risk of bias due to missing evidence as "low risk".

CONCLUSION

Our findings indicated substantial variation in assessments in consensus judgments between pairs for the signaling questions and overall risk-of-bias judgments. More tutorials and training are needed to help researchers apply the ROB-ME tool more consistently.

What are the best parameters of low-level laser therapy to reduce pain intensity and improve mandibular function in orofacial pain? A systematic review and meta-analysis

PURPOSE

To determine the effectiveness of laser therapy for managing patients with orofacial pain (OFP). In addition, to determine which parameters provide the best treatment effects to reduce pain, improve function, and quality of life in adults with OFP.

METHODS

Systematic review. Searches were conducted in six databases; no date or language restrictions were applied. Studies involving adults with OFP treated with laser therapy were included. The risk of bias (RoB) was performed with the Revised Cochrane RoB-2. A meta-analysis was structured around the OFP type, and outcomes. Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) assessed the overall certainty of the evidence.

RESULTS

Eighty-nine studies were included. Most studies (n = 72, 80.9%) were considered to have a high RoB. The results showed that laser therapy was better than placebo in improving pain, maximal mouth open (MMO), protrusion, and tenderness at the final assessment, but with a low or moderate level of evidence. The best lasers and parameters to reduce pain are diode or gallium-aluminum-arsenide (GaAlAs) lasers, a wavelength of 400-800 or 800-1500 nm, and dosage of <25 J/cm2.

CONCLUSIONS

Laser therapy was better than placebo to improve pain, MMO, protrusion, and tenderness. Also, it was better than occlusal splint to improve pain, but not better than TENS and medication.Implications for rehabilitationLaser therapy was found to be good in improving pain, maximal mouth opening, jaw protrusion, and tenderness at the end of the treatment.For patients with all types of temporomandibular disorders (TMDs) (myogenous, arthrogenous, and mixed), the following lasers and parameters are recommended: diode or gallium-aluminum-arsenide (GaAlAs) laser, wavelength of 400-800 or 800-1500 nm, and a dosage <25 J/cm2.For patients with arthrogenous TMDs, the following lasers and parameters are recommended: Diode laser and a wavelength between 400 and 800 nm.For patients with myogenous TMDs, the following lasers and parameters are recommended: diode laser, wavelength between 800 and 1500 nm, and dosage of <25 J/cm2.For patients with mixed TMDs, the following lasers and parameters are recommended: diode, GaAlAs, or infrared laser, a wavelength of 800-1500 nm, a dosage >100 J/cm2, and an application time between 15 and 30 s or >60 seconds.

Norfloxacin versus alternative antibiotics for prophylaxis of spontaneous bacteria peritonitis in cirrhosis: a systematic review and meta-analysis

BACKGROUND

Spontaneous bacterial peritonitis (SBP) is a life-threatening complication in patients with advanced cirrhosis. Prophylactic Norfloxacin used to be considered effective in SBP prevention, but in recent years its efficacy has been partially compromised by increasing quinolone-resistant bacteria. However, whether the effects of alternative prophylactic regimens are superior to norfloxacin remains controversial. The goal of this study is to compare the effects of norfloxacin with other antibiotics in SBP prophylaxis for cirrhotic patients.

METHODS

We systematically searched Pubmed, Embase, and Cochrane Library Databases. Two reviewers independently identified relevant random control trials (RCTs) comparing the role of norfloxacin and other antibiotics in SBP prevention.

RESULTS

Eight studies comprising 1043 cirrhotic patients were included in this study. Norfloxacin and alternative antibiotics displayed comparable effects in SBP prophylaxis, survival benefit, overall infection prevention, and safety. Subgroup analyses revealed that rifaximin prophylaxis could reduce the recurrence of SBP with fewer adverse events but failed to improve overall survival compared with norfloxacin.

CONCLUSIONS

Other antibiotics are a reasonable alternative to norfloxacin in the prophylaxis of SBP. Rifaximin prophylaxis could be an alternative choose of antibiotic for SBP prevention because of its better protective effect and safety.

Interventions to facilitate the successful eruption of impacted maxillary incisor teeth due to the presence of a supernumerary: A systematic review and meta-analysis

INTRODUCTION

A failure of maxillary incisor eruption is commonly attributed to the presence of a supernumerary tooth. This systematic review aimed to assess the percentage of impacted maxillary incisors that successfully erupt after surgical removal of supernumerary teeth with or without other interventions.

METHODS

Systematic literature searches without restrictions were undertaken in 8 databases for studies reporting any intervention aimed at facilitating incisor eruption, including surgical removal of the supernumerary alone or in conjunction with additional interventions published up to September 2022. After duplicate study selection, data extraction, and risk of bias assessment according to the risk of bias in nonrandomized studies of interventions and Newcastle-Ottawa scale, random-effects meta-analyses of aggregate data were conducted.

RESULTS

Fifteen studies (14 retrospective and 1 prospective) were included with 1058 participants (68.9% male; mean age, 9.1 years). The pooled eruption prevalence for removal of the supernumerary tooth with space creation or removal of the supernumerary tooth with orthodontic traction was significantly higher at 82.4% (95% confidence interval [CI], 65.5-93.2) and 96.9% (95% CI, 83.8-99.9) respectively, compared with removal of an associated supernumerary only (57.6%; 95% CI, 47.8-67.0). The odds of successful eruption of an impacted maxillary incisor after removal of a supernumerary were more favorable if the obstruction was removed in the deciduous dentition (odds ratio [OR], 0.42; 95% CI, 0.20-0.90; P = 0.02); if the supernumeraries were conical (OR, 2.91; 95% CI, 1.98-4.28; P <0.001); if the incisor was in the correct position (OR, 2.19; 95% CI, 1.14-4.20; P = 0.02), at the level of the gingival third (OR 0.07; 95% CI, <0.01-0.97; P = 0.04) and had incomplete root formation (OR, 9.02; 95% CI, 2.04-39.78; P = 0.004). Delaying removal of the supernumerary tooth 12 months after the expected eruption time of the maxillary incisor (OR, 0.33; 95% CI, 0.10-1.03; P = 0.05) and waiting >6 months for spontaneous eruption after removal of the obstacle (OR, 0.13; 95% CI, 0.03-0.50; P = 0.003) was associated with worse odds for eruption.

CONCLUSIONS

Limited evidence indicated that the adjunctive use of orthodontic measures and removal of supernumerary teeth might be associated with greater odds of successfull impacted incisor eruption than removal of the supernumerary tooth alone. Certain characteristics related to supernumerary type and the position or developmental stage of the incisor may also influence successful eruption after removal of the supernumerary. However, these findings should be viewed with caution as our certainty is very low to low because of bias and heterogeneity. Further well-conducted and reported studies are required. The results of this systematic review have been used to inform and justify the iMAC Trial.

Effectiveness of Manual Trigger Point Therapy in Patients with Myofascial Trigger Points in the Orofacial Region-A Systematic Review

The objective was to compile, synthetize, and evaluate the quality of the evidence from randomized controlled trials (RCTs) regarding the effectiveness of manual trigger point therapy in the orofacial area in patients with or without orofacial pain. This project was registered in PROSPERO and follows the PRISMA guidelines. Searches (20 April 2021) were conducted in six databases for RCTs involving adults with active or latent myofascial trigger points (mTrPs) in the orofacial area. The data were extracted by two independent assessors. Four studies were included. According to the GRADE approach, the overall quality/certainty of the evidence was very low due to the high risk of bias of the studies included. Manual trigger point therapy showed no clear advantage over other conservative treatments. However, it was found to be an equally effective and safe therapy for individuals with myofascial trigger points in the orofacial region and better than control groups. This systematic review revealed a limited number of RCTs conducted with patients with mTrPs in the orofacial area and the methodological limitations of those RCTs. Rigorous, well-designed RCTs are still needed in this field.

The Effectiveness of Psychological Interventions Delivered in Routine Practice: Systematic Review and Meta-analysis

This review presents a comprehensive evaluation of the effectiveness of routinely delivered psychological therapies across inpatient, outpatient and University-based clinics. This was a pre-registered systematic-review of studies meeting pre-specified inclusion criteria (CRD42020175235). Eligible studies were searched in three databases: MEDLINE, CINAHL and PsycInfo. Pre-post treatment (uncontrolled) effect sizes were calculated and pooled using random effects meta-analysis to generate effectiveness benchmarks. Moderator analyses were used to examine sources of heterogeneity in effect sizes. Overall, 252 studies (k = 298 samples) were identified, of which 223 (k = 263 samples) provided sufficient data for inclusion in meta-analysis. Results showed large pre-post treatment effects for depression [d = 0.96, (CI 0.88-1.04), p ≤ 0.001, k = 122], anxiety [d = 0.8 (CI 0.71-0.9), p ≤ 0.001, k = 69], and other outcomes [d = 1.01 (CI 0.93-1.09), p ≤ 0.001, k = 158]. This review provides support for the effectiveness of routinely delivered psychological therapy. Effectiveness benchmarks are supplied to support service evaluations across multiple settings.

Extension of the Composite Quality Score (CQS) as an appraisal tool for prospective, controlled clinical therapy trials-A systematic review of meta-epidemiological evidence

AIM

To conduct a survey of current meta-epidemiological studies to identify additional trial design characteristics that may be associated with significant over- or underestimation of the treatment effect and to use such identified characteristics as a basis for the formulation of new CQS appraisal criteria.

MATERIALS AND METHODS

We retrieved eligible studies from two systematic reviews on this topic (latest search May 2015) and searched the databases PubMed and Embase for further studies from June 2015 -March 2022. All data were extracted by one author and verified by another. Sufficiently homogeneous estimates from single studies were pooled using random-effects meta-analysis. Trial design characteristics associated with statistically significant estimates from single datasets (which could not be pooled) and meta-analyses were used as a basis to formulate new or amend existing CQS criteria.

RESULTS

A total of 38 meta-epidemiological studies were identified. From these, seven trial design characteristics associated with statistically significant over- or underestimation of the true therapeutic effect were found.

CONCLUSION

One new criterion concerning double-blinding was added to the CQS, and the original criteria for concealing the random allocation sequence and for minimum sample size were amended.

What Modifies the Effect of an Exercise Treatment for Chronic Low Back Pain? A Meta-epidemiologic Regression Analysis of Risk of Bias and Comparative Effectiveness

OBJECTIVE: To investigate how risk of bias and intervention type modify effect sizes of exercise interventions that are intended to reduce chronic low back pain intensity. DESIGN: Systematic review with meta-epidemiologic regression analysis. LITERATURE SEARCH: PubMed, CENTRAL, Embase, and CINAHL (until January 31, 2021). STUDY SELECTION CRITERIA: Systematic reviews with meta-analyses of randomized controlled exercise trials. DATA SYNTHESIS: The dependent variable was pain, calculated as standardized mean difference (SMD). Potential effect modifiers were risk of bias, exercise modes, study, and meta-analyses characteristics. Multilevel meta-regressions and inverse variance-weighted meta-regressions with random intercepts were modelled. RESULTS: Data from 26 systematic reviews (k = 349 effect sizes, n = 18,879 participants) were analysed. The overall mean effect was SMD: -0.35 (k = 349, [95% CI -0.02 to -0.7]). There was a clinically relevant effect overestimation in studies with a high risk of bias due to missing outcomes (each k = 197, Beta coefficient = -1.9 [95% CI -2.9 to -.9]) and low sample size (B = 0.01 [.001 to .01], [ie, one participant more leads to an SMD decrease of 0.01]). There was a clinically relevant underestimation of the effect when studies were at high risk of bias in allocation concealment (B = 1.3 [.5 to 2.1]) and outcome measurement (B = 1.3 [.44 to 2.0]). Motor control and stabilization training (B = -1.3 [-2.3 to -.37]) had the largest effects; stretching (B = 1.3 [-.03 to .5]) had the smallest effect. CONCLUSIONS: The effects of exercise trials at high risk of bias may be overestimated or underestimated. After accounting for risk of bias, motor control and stabilization exercises may represent the most effective exercise therapies for chronic low back pain. J Orthop Sports Phys Ther 2022;52(12):792-802. Epub: 12 August 2022. doi:10.2519/jospt.2022.11149.

Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system

Background

Machine learning and automation are increasingly used to make the evidence synthesis process faster and more responsive to policymakers’ needs. In systematic reviews of randomized controlled trials (RCTs), risk of bias assessment is a resource-intensive task that typically requires two trained reviewers. One function of RobotReviewer, an off-the-shelf machine learning system, is an automated risk of bias assessment.

Methods

We assessed the feasibility of adopting RobotReviewer within a national public health institute using a randomized, real-time, user-centered study. The study included 26 RCTs and six reviewers from two projects examining health and social interventions. We randomized these studies to one of two RobotReviewer platforms. We operationalized feasibility as accuracy, time use, and reviewer acceptability. We measured accuracy by the number of corrections made by human reviewers (either to automated assessments or another human reviewer’s assessments). We explored acceptability through group discussions and individual email responses after presenting the quantitative results.

Results

Reviewers were equally likely to accept judgment by RobotReviewer as each other’s judgement during the consensus process when measured dichotomously; risk ratio 1.02 (95% CI 0.92 to 1.13; p = 0.33). We were not able to compare time use. The acceptability of the program by researchers was mixed. Less experienced reviewers were generally more positive, and they saw more benefits and were able to use the tool more flexibly. Reviewers positioned human input and human-to-human interaction as superior to even a semi-automation of this process.

Conclusion

Despite being presented with evidence of RobotReviewer’s equal performance to humans, participating reviewers were not interested in modifying standard procedures to include automation. If further studies confirm equal accuracy and reduced time compared to manual practices, we suggest that the benefits of RobotReviewer may support its future implementation as one of two assessors, despite reviewer ambivalence. Future research should study barriers to adopting automated tools and how highly educated and experienced researchers can adapt to a job market that is increasingly challenged by new technologies.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01649-y.

Are Biases Related to Attrition, Missing Data, and the Use of Intention to Treat Related to the Magnitude of Treatment Effects in Physical Therapy Trials?: A Meta-Epidemiological Study

ABSTRACT

The objective of this study was to determine the association between biases related to attrition, missing data, and the use of intention to treat and changes in effect size estimates in physical therapy randomized trials. A meta-epidemiological study was conducted. A random sample of randomized controlled trials included in meta-analyses in the physical therapy discipline were identified. Data extraction including assessments of the use of intention to treat principle, attrition-related bias, and missing data was conducted independently by two reviewers. To determine the association between these methodological issues and effect sizes, a two-level analysis was conducted using a meta-meta-analytic approach. Three hundred ninety-three trials included in 43 meta-analyses, analyzing 44,622 patients contributed to this study. Trials that did not use the intention-to-treat principle (effect size = -0.13, 95% confidence interval = -0.26 to 0.01) or that were assessed as having inappropriate control of incomplete outcome data tended to underestimate the treatment effect when compared with trials with adequate use of intention to treat and control of incomplete outcome data (effect size = -0.18, 95% confidence interval = -0.29 to -0.08).Researchers and clinicians should pay attention to these methodological issues because they could provide inaccurate effect estimates. Authors and editors should make sure that intention-to-treat and missing data are properly reported in trial reports.

Combining meta-epidemiological study datasets on commercial funding of randomised clinical trials: Database, methods, and descriptive results of the COMFIT study

Randomised trials are often funded by commercial companies and methodological studies support a widely held suspicion that commercial funding may influence trial results and conclusions. However, these studies often have a risk of confounding and reporting bias. The risk of confounding is markedly reduced in meta-epidemiological studies that compare fairly similar trials within meta-analyses, and risk of reporting bias is reduced with access to unpublished data. Therefore, we initiated the COMmercial Funding In Trials (COMFIT) study aimed at investigating the impact of commercial funding on estimated intervention effects in randomised clinical trials based on a consortium of researchers who agreed to share meta-epidemiological study datasets with information on meta-analyses and trials included in meta-epidemiological studies. Here, we describe the COMFIT study, its database, and descriptive results. We included meta-epidemiological studies with published or unpublished data on trial funding source and results or conclusions. We searched five bibliographic databases and other sources. We invited authors of eligible meta-epidemiological studies to join the COMFIT consortium and to share data. The final construction of the COMFIT database involves checking data quality, identifying trial references, harmonising variable categories, and removing non-informative meta-analyses as well as correlated meta-analyses and trial results. We included data from 17 meta-epidemiological studies, covering 728 meta-analyses and 6841 trials. Seven studies (405 meta-analyses, 3272 trials) had not published analyses on the impact of commercial funding, but shared unpublished data on funding source. On this basis, we initiated the construction of a combined database. Once completed, the database will enable comprehensive analyses of the impact of commercial funding on trial results and conclusions with increased statistical power and a markedly reduced risk of confounding and reporting bias.

Amankwah J. Diabetes self-management education interventions in the WHO African Region: A scoping review

INTRODUCTION

Diabetes mellitus (DM) is one of the commonest chronic diseases worldwide. Self-Management Education (SME) is regarded as a critical element of treatment for all people with diabetes, as well as those at risk of developing the condition. While a great variety of diabetes self-management education (DSME) interventions are available in high-income countries, limited information exists on educational programs for the prevention and management of diabetes complications in Africa. This study, therefore, aimed at synthesizing information in the literature to describe the state of the science of DSME interventions in the WHO African Region.

MATERIALS AND METHODS

The study is a scoping review, which followed the standard PRISMA guidelines for conducting and reporting scoping reviews. A systematic keyword and subject headings searches were conducted on six electronic databases (PubMed, Scopus, MEDLINE, EMBASE, PsychINFO and the Cochrane Central Register of Controlled Trials) to identify relevant English language publications on DSME from 2000 through 2020. Titles and abstracts of the search results were screened to select eligible papers for full text reading. All eligible papers were retrieved and full text screening was done by three independent reviewers to select studies for inclusion in the final analysis.

RESULTS

Nineteen studies were included in the review. The interventions identified were individually oriented, group-based, individually oriented & group-based, and information technology-based DSME programs. Outcomes of the interventions were mixed. While the majority yielded significant positive results on HbA1c, diabetes knowledge, blood pressure, blood sugar and foot care practices; few demonstrated positive outcomes on self-efficacy, BMI, physical activity; self-monitoring of blood glucose, medication adherence, smoking and alcohol consumption.

CONCLUSIONS

The limited studies available indicate that DSME interventions in the WHO African Region have mixed effects on patient behaviors and health outcomes. That notwithstanding, the majority of the interventions demonstrated statistically significant positive effects on HbA1c, the main outcome measure in most DSME intervention studies.

Effectiveness and Determinant Variables of Augmentative and Alternative Communication Interventions in Cerebral Palsy Patients with Communication Deficit: a Systematic Review

PURPOSE

Assess the effectiveness of augmentative and alternative communication (AAC) interventions in patients with CP and to reveal determinant variables of main intervention outcomes: receptive and expressive language.

RESEARCH STRATEGIES

The search was performed in following databases: MEDLINE (Ovid); PubMed (NLM); Embase (Ovid); Cochrane Database of Systematic Reviews; Cumulative Index to Nursing and Allied Health Literature; Database of Abstracts of Reviews of Effects; Cochrane Central Register of Controlled Trials; Health Technology Assessment database and PEDro.

SELECTION CRITERIA

Full-text and peer-reviewed studies in English studying the effectiveness of AAC in patients with cerebral palsy were included. Studies with patients (<18 years) diagnosed with CP were included.

DATA ANALYSIS

A narrative analysis was conducted to evaluate the efficacy of AAC methods. A random-effects model meta-analysis was used to assess determinants of AAC intervention outcomes.

RESULTS

The online database and manual reference search revealed 445 records. Nine studies investigating a total of 294 subjects with CP met predefined eligibility criteria: 4 studies with single subject, multiple baseline research designs, 3 longitudinal cohort studies, 1 case control study and 1 case series. Results revealed moderate-quality evidence that AAC interventions improve the receptive and expressive communication skills in patients with CP. The random-effects model meta-analysis revealed the power of identified determinant variables affecting the AAC intervention outcomes.

CONCLUSION

Diversity of CP patients requires proper analysis of determinant variables to ensure the efficacy of AAC assessment and intervention. More studies of high methodological and practical quality assessing the efficacy of AAC interventions are needed to clarify the evidence.

Assessing the risk of performance and detection bias in Cochrane reviews as a joint domain is less accurate compared to two separate domains

Background

Initially, the Cochrane risk of bias (RoB) tool had a domain for “blinding of participants, personnel and outcome assessors”. In the 2011 tool, the assessment of blinding was split into two domains: blinding of participants and personnel (performance bias) and blinding of outcome assessors (detection bias). The aims of this study were twofold; first, to analyze the frequency of usage of the joint blinding domain (a single domain for performance and detection bias), and second, to assess the proportion of adequate assessments made in the joint versus single RoB domains for blinding by comparing whether authors’ RoB judgments were supported by explanatory comments in line with the Cochrane Handbook recommendations.

Methods

We extracted information about the assessment of blinding from RoB tables (judgment, comment, and whether it was specified which outcome type; e.g., objective, subjective) of 729 Cochrane reviews published in 2015-2016. In the Cochrane RoB tool, judgment (low, unclear or high risk) needs to be accompanied by a transparent comment, in which authors provide a summary justifying RoB judgment, to ensure transparency in how these judgments were reached. We reassessed RoB based on the supporting comments reported in Cochrane RoB tables, in line with instructions from the Cochrane Handbook. Then, we compared our new assessments to judgments made by Cochrane authors. We compared the frequency of adequate judgments in reviews with two separate domains for blinding versus those with a joint domain for blinding.

Results

The total number of assessments for performance bias was 6918, with 8656 for detection bias and 3169 for the joint domain. The frequency of adequate assessments was 74% for performance bias, 78% for detection bias, and 59% for the joint domain. The lowest frequency of adequate assessments was found when Cochrane authors judged low risk – 47% in performance bias, 62% in detection bias, and 31% in the joint domain. The joint domain and detection bias domain had a similar proportion of specified outcome types (17% and 18%, respectively).

Conclusions

Splitting joint RoB assessment about blinding into two domains was justified because the frequency of adequate judgments was higher in separate domains. Specification of outcome types in RoB domains should be further scrutinized.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01339-1.

Tools to Assess the Risk of Bias and Reporting Quality of Randomized Controlled Trials in Rehabilitation

OBJECTIVES

(1) To determine whether new tools and items have been developed to evaluate the risk of bias (RoB) and reporting of randomized controlled trials (RCTs) in rehabilitation; (2) to determine which items are included in the existing reporting guidelines, and to create a matrix of items to report and conduct trials in rehabilitation as the first step for a starting a rigorous validation process.

DATA SOURCES

Searches were conducted in MEDLINE, EMBASE, and Cumulative Index to Nursing and Allied Health databases.

STUDY SELECTION

Studies should describe a newly developed tool to evaluate the RoB or quality of reporting for RCTs in the area of rehabilitation.

DATA EXTRACTION

(1) First, we extracted items from new tools identified by the electronic search strategies and then (2) we looked at the items provided by the Consolidated Standards of Reporting Trials statement and its relevant extensions. We determined whether these items were already included in our matrix of items. Items were classified based on methodological domains they accomplish, biases they were able to target, and whether they guide reporting or conduct.

DATA SYNTHESIS

Among the 1596 citations found, 23 articles were potentially relevant. From these, only 3 new scales (National Institute for Complementary Medicine Acupuncture Network, Quality of reports on spa and balneotherapy [SPAC], Assessment of Study Quality and Reporting in Exercise) were found. In addition, the newly updated Cochrane RoB tool (RoB 2.0) was included. Our matrix contained 122 unique items for any rehabilitation area, 46 items (37.7%) were related to conduct, and 58 (47.5%) were related to the reporting; 18 (14.8%) were related to both. Overall, 76 new items were added among all domains.

CONCLUSIONS

Many individual and diverse items have been used to guide the reporting and conduct of rehabilitation trials. This indicates a great variability in number of items and an apparent lack of consensus on a core set of items to be used in rehabilitation. Future research should look into developing a core set of items for the rehabilitation field.

Methodological Issues in Rehabilitation Research: A Scoping Review

OBJECTIVE

To identify, synthesize, and categorize the methodological issues faced by the rehabilitation field.

DATA SOURCES

A scoping review was conducted using studies identified in MEDLINE, the Cochrane Library, EMBASE, Web of Science, Scopus, Physiotherapy Evidence Database, and Google Scholar up to August 2018.

STUDY SELECTION

We included all type of publications describing methodological issues in rehabilitation research where rehabilitation is described as a multimodal process. The methodological issues have been categorized and classified.

DATA EXTRACTION

The synthesis included qualitative and quantitative analysis. To focus the attention on rehabilitation, we post hoc divided in "specific issues" (highly related to, even if not exclusive of, rehabilitation research) and "generic issues" (common in biomedical research).

DATA SYNTHESIS

Seventy-one publications were included: 68% were narrative reviews, 15% systematic reviews, 7% editorials, 4% meta-epidemiologic studies, and 5% others. Specific methodological issues include the following: problematic application of randomized controlled trials (32%), absent definition of core outcome sets (28%), poor interventions description (22%), weak methodological (conducting) and reporting quality (21%), scarce clinical practice applicability (14%), lack of blinding assessor (10%), inadequate randomization methods or inadequate allocation concealment (8%), and inadequate participants description and recruitment (8%). "Generic" issues included the following: data and statistical description (31%), authors' methodological training (7%), peer review process (6%, n=4), funding declaration (6%), ethical statement (3%), protocol registration (3%), and conflict of interest declaration (1%).

CONCLUSIONS

Methodological and reporting issues might influence the quality of the evidence produced in rehabilitation research. The next steps to move forward in the field of rehabilitation could be to evaluate the influence of all these issues on the validity of trial results through meta-epidemiologic studies and to develop specific checklists to provide guidance to authors to improve the reporting and conduct of trials in this field.

The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement

Many randomized controlled trials (RCTs) are biased and difficult to reproduce due to methodological flaws and poor reporting. There is increasing attention for responsible research practices and implementation of reporting guidelines, but whether these efforts have improved the methodological quality of RCTs (e.g., lower risk of bias) is unknown. We, therefore, mapped risk-of-bias trends over time in RCT publications in relation to journal and author characteristics. Meta-information of 176,620 RCTs published between 1966 and 2018 was extracted. The risk-of-bias probability (random sequence generation, allocation concealment, blinding of patients/personnel, and blinding of outcome assessment) was assessed using a risk-of-bias machine learning tool. This tool was simultaneously validated using 63,327 human risk-of-bias assessments obtained from 17,394 RCTs evaluated in the Cochrane Database of Systematic Reviews (CDSR). Moreover, RCT registration and CONSORT Statement reporting were assessed using automated searches. Publication characteristics included the number of authors, journal impact factor (JIF), and medical discipline. The annual number of published RCTs substantially increased over 4 decades, accompanied by increases in authors (5.2 to 7.8) and institutions (2.9 to 4.8). The risk of bias remained present in most RCTs but decreased over time for allocation concealment (63% to 51%), random sequence generation (57% to 36%), and blinding of outcome assessment (58% to 52%). Trial registration (37% to 47%) and the use of the CONSORT Statement (1% to 20%) also rapidly increased. In journals with a higher impact factor (>10), the risk of bias was consistently lower with higher levels of RCT registration and the use of the CONSORT Statement. Automated risk-of-bias predictions had accuracies above 70% for allocation concealment (70.7%), random sequence generation (72.1%), and blinding of patients/personnel (79.8%), but not for blinding of outcome assessment (62.7%). In conclusion, the likelihood of bias in RCTs has generally decreased over the last decades. This optimistic trend may be driven by increased knowledge augmented by mandatory trial registration and more stringent reporting guidelines and journal requirements. Nevertheless, relatively high probabilities of bias remain, particularly in journals with lower impact factors. This emphasizes that further improvement of RCT registration, conduct, and reporting is still urgently needed.

Many randomized controlled trials (RCTs) are biased and difficult to reproduce due to methodological flaws and poor reporting. Analysis of 176,620 RCTs published between 1966 and 2018 reveals that the risk of bias in RCTs generally decreased. Nevertheless, relatively high probabilities of bias remain, showing that further improvement of RCT registration, conduct, and reporting is still urgently needed.

Methodological quality and risk of bias in orthodontic systematic reviews using AMSTAR and ROBIS

BACKGROUND

This study aimed to compare the methodological quality and risk bias of orthodontic systematic reviews (SRs) using the AMSTAR (Assessing the Methodological Quality of Systematic Reviews) and ROBIS (Risk of Bias in Systematic Review) tools.

MATERIALS AND METHODS

A search of electronic databases (OVID and Medline) was undertaken to identify orthodontic SRs published in five primary orthodontic journals (January 2015 to December 2018) and the Cochrane Library of Systematic Reviews (January 2000 to January 2018). Full articles were reviewed by two assessors against the eligibility criteria. Methodological quality of each SR was gauged using the AMSTAR tool with a score of 0 or 1 given for each of the 11 items. Cumulative totals were calculated and scores between 4 and 8 represented poor to fair methodological quality and 9 or greater deemed to be good. As per the ROBIS tool, the risk of bias (ROB) for each domain was assessed and the overall ROB was classified as low, high, or unclear.

RESULTS

A total of 91 SRs were included. The median AMSTAR score was 8 (IQR = 3). The methodological quality of 47.3% SRs was rated good. SRs without protocol registration (Coef: -3.00, 95% CI: -3.72, -2.28, P < 0.001) and American continent SRs (Coef: -1.00, 95% CI: -1.72, -0.21, P = 0.007) were associated with lower AMSTAR scores. A total of 56.0% SRs were rated a low ROB, with a lower ROB apparent in multicentre SRs (OR: 0.27, 95% CI: 0.11, 0.64, P = 0.003) and a higher ROB evident in SRs without a registered protocol (OR: 111.81, 95% CI: 22.34, 559.62, P < 0.001). When adjusted for the effect of AMSTAR score on ROB, a higher ROB was associated with SRs without protocol registration (OR: 32.24, 95% CI: 6.03, 172.44, P ≤ 0.001). As the AMSTAR score (per unit) increased, the odds of having a high ROB rating decreased (OR: 0.31, 95% CI: 0.21, 0.45, P ≤ 0.001).

CONCLUSIONS

As the methodological quality rating of orthodontic SRs increases, a reduction in the ROB is evident.

A unified framework for bias assessment in clinical research

Methodological flaws, limitations, and inadequate practices in research are well known and pose threats to the internal validity of any research study. However, there are ways of safeguarding research conduct to reduce the chance of research producing distorted results. Numerous tools now exist to assess the incorporation of such safeguards into primary research studies (also known as quality and/or risk-of-bias assessment). These tools typically include a variety of items that are then checked against those implemented in the study. Despite a lot of research in this area, no comprehensive generic classification of safeguards across study designs exist, although attempts have been made to clarify aspects of this. We review the developments in this area as well as use preliminary data from 100 methodological studies to illustrate our proposed approach. We conclude by proposing a new framework for identifying research studies at risk of being biased and the information in this article will promote a unification of the diverse approaches to facilitating bias assessment in clinical research.

Nursing Staff Time and Care Quality in Long-Term Care Facilities: A Systematic Review

BACKGROUND AND OBJECTIVES

In long-term care (LTC) facilities, nursing staff are important contributors to resident care and well-being. Despite this, the relationships between nursing staff coverage, care hours, and quality of resident care in LTC facilities are not well understood and have implications for policy-makers. This systematic review summarizes current evidence on the relationship between nursing staff coverage, care hours, and quality of resident care in LTC facilities.

RESEARCH DESIGN AND METHODS

A structured literature search was conducted using four bibliographic databases and gray literature sources. Abstracts were screened by two independent reviewers using Covidence software. Data from the included studies were summarized using a pretested extraction form. The studies were critically appraised, and their results were synthesized narratively.

RESULTS

The systematic searched yielded 15,842 citations, of which 54 studies (all observational) were included for synthesis. Most studies (n = 53, 98%) investigated the effect of nursing staff time on resident care. Eleven studies addressed minimum care hours and quality of care. One study examined the association between different nursing staff coverage models and resident outcomes. Overall, the quality of the included studies was poor.

DISCUSSION AND IMPLICATIONS

Because the evidence was inconsistent and of low quality, there is uncertainty about the direction and magnitude of the association between nursing staff time and type of coverage on quality of care. More rigorously designed studies are needed to test the effects of different cutoffs of care hours and different nursing coverage models on the quality of resident care in LTC facilities.

The effect of the CONSORT statement on the amount of "unclear" Risk of Bias reporting in Cochrane Systematic Reviews

BACKGROUND

The Consolidated Standards of Reporting Trials (CONSORT) statement aims to improve clarity and consistency of transparency of reporting in Randomized Controlled Trials (RCTs). The Cochrane Risk of Bias (RoB) tool for RCTs helps authors to judge the RoB. as ''low", "high" or "unclear".

OBJECTIVE

In this study we aimed to assess whether the implementation and updates of the CONSORT statement influenced the trend of "unclear" RoB scores of RCTs included in Cochrane systematic reviews.

METHODS

All Cochrane reviews published in December to October 2016 were retrieved. The publication year of RCTS included in the reviews were sorted into time frames (≤1995, 1996-2000, 2001-2009 and ≥2010) based on the release- and updates of the CONSORT statement (1996, 2001 and 2010). The association between "unclear" RoB versus "low or high" RoB and the year of publication in different time frames were calculated using a binary logistic regression.

RESULTS

Data was extracted from 64 Cochrane reviews, with 989 RCTS (6471 items). The logistic regression showed that the odds of RCTs published ≥2010, compared to ≤1995 were more likely not to report an "unclear" RoB for the total data (Odds Ratio (OR) 0.69 (95% Confidence interval: 0.59-0.80)), random sequence generation (OR 0.32 (0.22-0.47), allocation concealment (0.64 (0.43-0.95)) and incomplete outcome data (OR 0.60 (0.39-0.91)).

CONCLUSION

A slight decrease of "unclear" RoB reporting over time was found. To improve quality of reporting authors are encouraged to adhere to reporting guidelines.

A systematic review of effectiveness of interventions applicable to radiotherapy that are administered to improve patient comfort, increase patient compliance, and reduce patient distress or anxiety

OBJECTIVES

The aim of this review was to search existing literature to identify comfort interventions that can be used to assist an adult patient to undergo complex radiotherapy requiring positional stability for periods greater than 10 min. The objectives of this review were to; 1) identify comfort interventions used for clinical procedures that involve sustained inactivity similar to radiotherapy; 2) define characteristics of comfort interventions for future practice; and 3) determine the effectiveness of identified comfort interventions. The Preferred Reporting Items for Systematic Reviews and meta-analyses statement and the Template-for-Intervention-Description-and Replication guide were used.

KEY FINDINGS

The literature search was performed using PICO criteria with five databases (AMED, CINAHL EMBASE, MEDLINE, PsycINFO) identifying 5269 titles. After screening, 46 randomised controlled trials met the inclusion criteria. Thirteen interventions were reported and were grouped into four categories: Audio-visual, Psychological, Physical, and Other interventions (education/information and aromatherapy). The majority of aromatherapy, one audio-visual and one educational intervention were judged to be clinically significant for improving patient comfort based on anxiety outcome measures (effect size ≥ 0.4, mean change is greater than the Minimal-Important-Difference and low-risk-of-bias). Medium to large effect sizes were reported in many interventions where differences did not exceed the Minimal-Important-Difference for the measure. These interventions were deemed worthy of further investigation.

CONCLUSION

Several interventions were identified that may improve comfort during radiotherapy assisting patients to sustain and endure the same position over time. This is crucial for the continual growth of complex radiotherapy requiring a need for comfort to ensure stability for targeted treatment.

IMPLICATIONS FOR PRACTICE

Further investigation of comfort interventions is warranted, including tailoring interventions to patient choice and determining if multiple interventions can be used concurrently to improve effectiveness.

External factors may influence Cochrane reviewers when classifying the risk of bias of original reports

OBJECTIVE

The objective of the study was to explore contextual factors associated with high or low risk-of-bias judgment in case of incomplete or unclear information in study reports.

STUDY DESIGN AND SETTING

Research-on-research study, using matched case-control design, with a sample of 304 randomized controlled trials (RCTs) included in two Cochrane reviews for which there was disagreement on the risk-of-bias judgment related to incomplete or unclear information in the study report. A case was defined as an RCT judged at high or low risk of bias; a control was the same RCT judged at unclear risk. We used a conditional logistic regression model for analysis.

RESULTS

Review authors being also authors of the RCT were more likely to assess an item at low risk of bias than unclear (OR: 11.71; 95% CI: 1.39-98.76). Earlier trials in a review were more often assigned a low risk (OR: 0.37; [0.15-0.96]). Review groups and authors that had completed a lower number of reviews slightly more often assigned a low risk, whereas others reported "unclear" (OR: 0.97, [95% CI: 0.95-0.99] for groups) and 0.97 (95% CI: 0.95-0.998) for authors).

CONCLUSIONS

Risk-of-bias assessment of RCTs in case of incomplete or unclear information may be affected by contextual factors.

Comparing machine and human reviewers to evaluate the risk of bias in randomized controlled trials

BACKGROUND

Evidence from new health technologies is growing, along with demands for evidence to inform policy decisions, creating challenges in completing health technology assessments (HTAs)/systematic reviews (SRs) in a timely manner. Software can decrease the time and burden by automating the process, but evidence validating such software is limited. We tested the accuracy of RobotReviewer, a semi-autonomous risk of bias (RoB) assessment tool, and its agreement with human reviewers.

METHODS

Two reviewers independently conducted RoB assessments on a sample of randomized controlled trials (RCTs), and their consensus ratings were compared with those generated by RobotReviewer. Agreement with the human reviewers was assessed using percent agreement and weighted kappa (κ). The accuracy of RobotReviewer was also assessed by calculating the sensitivity, specificity, and area under the curve in comparison to the consensus agreement of the human reviewers.

RESULTS

The study included 372 RCTs. Inter-rater reliability ranged from κ = -0.06 (no agreement) for blinding of participants and personnel to κ = 0.62 (good agreement) for random sequence generation (excluding overall RoB). RobotReviewer was found to use a high percentage of "irrelevant supporting quotations" to complement RoB assessments for blinding of participants and personnel (72.6%), blinding of outcome assessment (70.4%), and allocation concealment (54.3%).

CONCLUSION

RobotReviewer can help with risk of bias assessment of RCTs but cannot replace human evaluations. Thus, reviewers should check and validate RoB assessments from RobotReviewer by consulting the original article when not relevant supporting quotations are provided by RobotReviewer. This consultation is in line with the recommendation provided by the developers.

Inter-review agreement of risk-of-bias judgments varied in Cochrane reviews

OBJECTIVES

The objective of the study was to measure the level of agreement between Cochrane reviews of overlapping randomized controlled trials (RCTs) regarding risk-of-bias (RoB) judgments.

STUDY DESIGN AND SETTING

On November 5, 2017, the Cochrane Database of Systematic Reviews was searched for Cochrane reviews on tobacco. Reviews that included overlapping RCTs were included. RoB judgments were extracted from RoB tables using automated data scraping with manual verification and adjustments. Agreement between the reviews was calculated using Conger's generalized kappa coefficient (κ) and raw agreement (a).

RESULTS

We included 53 Cochrane reviews of 376 RCTs. For the RoB domain "random sequence generation," the level of agreement between the reviews was substantial with κ = 0.63 (95% confidence interval: 0.56 to 0.71; a = 0.80). There was slight-to-moderate agreement between the reviews regarding the domains "allocation concealment": κ = 0.51 (0.41 to 0.61), a = 0.75; "blinding": κ = 0.19 (0.02 to 0.37), a = 0.52; "blinding of outcome assessment": κ = 0.43 (0.14 to 0.72) a = 0.67; and "incomplete outcome data": κ = 0.15 (-0.03 to 0.32), a = 0.64. For "blinding of participants and personnel" and "selective reporting", κ could not be calculated. The raw agreement was 0.40 and 0.42, respectively.

CONCLUSION

The level of agreement between Cochrane reviews regarding RoB judgments ranged from slight to substantial depending on the RoB domain. Further investigations regarding reasons for variation and interventions to improve agreement are needed.

Risk of bias and reporting completeness of randomised controlled trials in burn care: protocol for a systematic review

INTRODUCTION

Burn care represents a healthcare and economic burden to patients internationally. Choice of the most clinically effective treatment strategies requires evidence which is best obtained through high-quality randomised controlled trials (RCT). The number of published RCTs of burn care is increasing. However, trial quality and reporting standards are unclear. This study will assess the risk of bias and adequacy of reporting in recent burn care RCTs using tools endorsed by the Cochrane Collaboration.

METHODS AND ANALYSIS

A systematic literature review will be undertaken, assessing parallel group RCTs evaluating therapeutic interventions for patients with cutaneous burns. Literature searches will use Ovid Medline, Ovid Embase, Web of Science and the Cochrane Library. Separate searches for each database will include medical subject heading and free text terms including 'burn', 'scald', 'thermal injury' and 'RCT'. Two reviewers will independently assess each study for inclusion. Risk of bias (RoB) will be assessed with the revised tool (RoB 2) and reporting completeness with the CONsolidated Standards of Reporting Trials (CONSORT) 2010 guidelines. We will report a narrative synthesis of all studies, including domain specific, and overall risk of bias for the primary outcome of each trial. Inter-rater agreement for RoB 2 will be reported using Fleiss's Kappa. For adherence to the CONSORT guidelines, we will generate a completeness of reporting index for the five domains.

ETHICS AND DISSEMINATION

No ethics approval is required because published documents will be used. Findings of the study will be disseminated in a peer-reviewed journal and presented at conferences.

PROSPERO REGISTRATION NUMBER

CRD42018111020.

Classical psychedelics for the treatment of depression and anxiety: A systematic review

BACKGROUND

Depression and anxiety are prevalent psychiatric disorders that carry significant morbidity. Pharmacological and psychosocial interventions are used to manage these conditions, but their efficacy is limited. Recent interest into the use of psychedelic-assisted therapy using ayahuasca, psilocybin or lysergic acid diethylamide (LSD) may be a promising alternative for patients unresponsive to traditional treatments. This review aims to determine the efficacy and tolerability of psychedelics in the management of resistant depression.

METHODS

Clinical trials investigating psychedelics in patients with depression and/or anxiety were searched via MEDLINE, EMBASE and PsychINFO. Efficacy was assessed by measuring symptom improvement from baseline, and tolerability was evaluated by noting the incidence and type of adverse effects reported. Risk of bias was assessed.

RESULTS

Seven studies, with 130 patients, were analysed in this review. Three were conducted in patients with depression, two in patients with anxiety and two in patients with both. In a supportive setting, ayahuasca, psilocybin, and LSD consistently produced immediate and significant anti-depressant and anxiolytic effects that were endured for several months. Psychedelics were well-tolerated. The most common adverse effects were transient anxiety, short-lived headaches, nausea and mild increases in heart rate and blood pressure.

LIMITATIONS

At present, the number of studies on this subject is very limited; and the number of participating patients within these is also limited as the treatment under investigations is a relatively novel concept.

CONCLUSIONS

Though further evidence is required, psychedelics appear to be effective in significantly reducing symptoms of depression and anxiety and are well-tolerated.

Agreement between the Cochrane risk of bias tool and Physiotherapy Evidence Database (PEDro) scale: A meta-epidemiological study of randomized controlled trials of physical therapy interventions

Background

The Cochrane risk of bias (CROB) tool and Physiotherapy Evidence Database (PEDro) scale are used to evaluate risk of bias of randomized controlled trials. We assessed the level of agreement between the instruments.

Methods

We searched the Cochrane Library to identify trials included in systematic reviews evaluating physical therapy interventions. For trials that met our inclusion criteria (primary reference in Cochrane review, review used CROB (2008 version), indexed in PEDro), CROB items were extracted from the reviews and PEDro items and total score were downloaded from PEDro. Kappa statistics were used to determine the agreement between CROB and PEDro scale items that evaluate similar constructs (e.g., randomization). The total PEDro score was compared to the CROB summary score (% of items met) using an Intraclass Correlation Coefficient. Sensitivity analyses explored the impact of the CROB “unclear” category and variants of CROB blinding items. Kappa statistics were used to determine agreement between different thresholds for “acceptable” risk of bias between CROB and PEDro scale summary scores.

Results

We included 1442 trials from 108 Cochrane reviews. Agreement was “moderate” for three of the six CROB and PEDro scale items that evaluate similar constructs (allocation concealment, participant blinding, assessor blinding; Kappa = 0.479–0.582). Agreement between the summary scores was “poor” (Intraclass Correlation Coefficient = 0.285). Agreement was highest when the CROB “unclear” category was collapsed with “high” and when participant, personnel and assessor blinding were evaluated separately in CROB. Agreement for different thresholds for “acceptable” risk of bias between CROB and PEDro summary scores was, at best, “fair”.

Conclusion

There was moderate agreement for half of the PEDro and CROB items that evaluate similar constructs. Interpretation of the CROB “unclear” category and variants of the CROB blinding items substantially influenced agreement. Either instrument can be used to quantify risk of bias, but they can’t be used interchangeably.

Is psychotherapy effective? A re-analysis of treatments for depression

AimsThe aim of this study was to reanalyse the data from Cuijpers et al.'s (2018) meta-analysis, to examine Eysenck's claim that psychotherapy is not effective. Cuijpers et al., after correcting for bias, concluded that the effect of psychotherapy for depression was small (standardised mean difference, SMD, between 0.20 and 0.30), providing evidence that psychotherapy is not as effective as generally accepted.

METHODS

The data for this study were the effect sizes included in Cuijpers et al. (2018). We removed outliers from the data set of effects, corrected for publication bias and segregated psychotherapy from other interventions. In our study, we considered wait-list (WL) controls as the most appropriate estimate of the natural history of depression without intervention.

RESULTS

The SMD for all interventions and for psychotherapy compared to WL controls was approximately 0.70, a value consistent with past estimates of the effectiveness of psychotherapy. Psychotherapy was also more effective than care-as-usual (SMD = 0.31) and other control groups (SMD = 0.43).

CONCLUSIONS

The re-analysis reveals that psychotherapy for adult patients diagnosed with depression is effective.

Usability and sensitivity of the risk of bias assessment tool for randomized controlled trials of pharmacist interventions

Background The Cochrane collaboration risk of bias assessment (RoB) tool is used in several fields to evaluate the methodological quality of studies. Its strengths and challenges are discussed. Objective To assess the sensitivity of the RoB tool in studies of pharmacist interventions. Setting DEPICT database was used to pool randomized controlled trials (RCTs) of complex interventions. Method A Guide for RoB Judgment in Pharmacy Services was created to help in the interpretation and judgment of bias criteria. The evaluation of bias (low, unclear, high risk) was performed by RCT. Sensitivity analyses were performed to assess the influence of different interpretations of eight elements of judgment in the RoB tool. Paired analysis and estimations of the effect size (95% confidence interval) of the criteria modifications compared to the original analyses were calculated. Main outcome measure Changes in the interpretations of judgment in the RoB tool. Results Overall, 8.3, 45.4, and 46.3% of the studies were determined to have low, unclear, and high risk of bias, respectively. High risk of bias was caused by attrition and detection domains. The number of studies classified with high risk of bias significantly increased for five of the eight interpretations, while unclear risk of bias increased for three interpretations (with a negligible effect size in all of them). Lack of blinding, loss of participants, and the use of subjective and self-reported outcomes were the main elements resulting in high risk of bias. Conclusion The RoB tool is useful for evaluating RCTs of pharmacist interventions if adapted criteria for judgment are used. Ignoring these adjustments produces a floor-effect with studies classified with high risk of bias.

Disagreements in risk of bias assessment for randomised controlled trials included in more than one Cochrane systematic reviews: a research on research study using cross-sectional design

OBJECTIVES

Assess the frequency and reasons for disagreements in risk of bias assessments for randomised controlled trials (RCTs) included in more than one Cochrane review.

DESIGN

Research on research study, using cross-sectional design.

DATA SOURCES

2796 Cochrane reviews published between March 2011 and September 2014.

DATA SELECTION

RCTs included in more than one review.

DATA EXTRACTION

Risk of bias assessment and support for judgement for five key risk of bias items.

DATA SYNTHESIS

For each item, we compared risk of bias assessment made in each review and calculated proportion of agreement. Two reviewers independently analysed 50% of all disagreements by comparing support for each judgement with information from study report to evaluate whether disagreements were related to a difference in information (eg, contact the study author) or a difference in interpretation (same support for judgement but different interpretation). They also identified main reasons for different interpretation.

RESULTS

1604 RCTs were included in more than one review. Proportion of agreement ranged from 57% (770/1348 trials) for incomplete outcome data to 81% for random sequence generation (1193/1466). Most common source of disagreement was difference in interpretation of the same information, ranging from 65% (88/136) for random sequence generation to 90% (56/62) for blinding of participants and personnel. Access to different information explained 32/136 (24%) disagreements for random sequence generation and 38/205 (19%) for allocation concealment. Disagreements related to difference in interpretation were frequently related to incomplete or unclear reporting in the study report (83% of disagreements related to different interpretation for random sequence generation).

CONCLUSIONS

Risk of bias judgements of RCTs included in more than one Cochrane review differed substantially. Most disagreements were related to a difference in interpretation of an incomplete or unclear description in the study report. A clearer guidance on common causes of incomplete information may improve agreement.

Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review

OBJECTIVES

The aim of the article was to identify and summarize studies assessing methodologies for study selection, data abstraction, or quality appraisal in systematic reviews.

STUDY DESIGN AND SETTING

A systematic review was conducted, searching MEDLINE, EMBASE, and the Cochrane Library from inception to September 1, 2016. Quality appraisal of included studies was undertaken using a modified Quality Assessment of Diagnostic Accuracy Studies 2, and key results on accuracy, reliability, efficiency of a methodology, or impact on results and conclusions were extracted.

RESULTS

After screening 5,600 titles and abstracts and 245 full-text articles, 37 studies were included. For screening, studies supported the involvement of two independent experienced reviewers and the use of Google Translate when screening non-English articles. For data abstraction, studies supported involvement of experienced reviewers (especially for continuous outcomes) and two independent reviewers, use of dual monitors, graphical data extraction software, and contacting authors. For quality appraisal, studies supported intensive training, piloting quality assessment tools, providing decision rules for poorly reported studies, contacting authors, and using structured tools if different study designs are included.

CONCLUSION

Few studies exist documenting common systematic review practices. Included studies support several systematic review practices. These results provide an updated evidence-base for current knowledge synthesis guidelines and methods requiring further research.

The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies

BACKGROUND

Different tools exist for assessing risk of bias of intervention studies for systematic reviews. We present a tool for assessing risk of bias across both randomized and non-randomized study designs. The tool was developed by the Evidence Project, which conducts systematic reviews and meta-analyses of behavioral interventions for HIV in low- and middle-income countries.

METHODS

We present the eight items of the tool and describe considerations for each and for the tool as a whole. We then evaluate reliability of the tool by presenting inter-rater reliability for 125 selected studies from seven published reviews, calculating a kappa for each individual item and a weighted kappa for the total count of items.

RESULTS

The tool includes eight items, each of which is rated as being present (yes) or not present (no) and, for some items, not applicable or not reported. The items include (1) cohort, (2) control or comparison group, (3) pre-post intervention data, (4) random assignment of participants to the intervention, (5) random selection of participants for assessment, (6) follow-up rate of 80% or more, (7) comparison groups equivalent on sociodemographics, and (8) comparison groups equivalent at baseline on outcome measures. Together, items (1)-(3) summarize the study design, while the remaining items consider other common elements of study rigor. Inter-rater reliability was moderate to substantial for all items, ranging from 0.41 to 0.80 (median κ = 0.66). Agreement between raters on the total count of items endorsed was also substantial (κw = 0.66).

CONCLUSIONS

Strengths of the tool include its applicability to a range of study designs, from randomized trials to various types of observational and quasi-experimental studies. It is relatively easy to use and interpret and can be applied to a range of review topics without adaptation, facilitating comparability across reviews. Limitations include the lack of potentially relevant items measured in other tools and potential threats to validity of some items. To date, the tool has been applied in over 30 reviews. We believe it is a practical option for assessing risk of bias in systematic reviews of interventions that include a range of study designs.

Needle size for vaccination procedures in children and adolescents

BACKGROUND

This is an update of a Cochrane Review first published in 2015. The conclusions have not changed.Hypodermic needles of different sizes (gauges and lengths) can be used for vaccination procedures. The gauge (G) refers to the outside diameter of the needle tubing. The higher the gauge number, the smaller the diameter of the needle (e.g. a 23 G needle is 0.6 mm in diameter, whereas a 25 G needle is 0.5 mm in diameter). Many vaccines are recommended for injection into muscle (intramuscularly), although some are delivered subcutaneously (under the skin) and intradermally (into skin). Choosing an appropriate length and gauge of a needle may be important to ensure that a vaccine is delivered to the appropriate site and produces the maximum immune response while causing the least possible harm. Guidelines conflict regarding the sizes of needles that should be used for vaccinating children and adolescents.

OBJECTIVES

To assess the effects of using needles of different sizes for administering vaccines to children and adolescents on vaccine immunogenicity (the ability of the vaccine to elicit an immune response), procedural pain, and other reactogenicity events (adverse events following vaccine administration).

SEARCH METHODS

We updated our searches of CENTRAL, MEDLINE, Embase, and CINAHL to October 2017. We also searched proceedings of vaccine conferences and two trials registers.

SELECTION CRITERIA

Randomised controlled trials evaluating the effects of using hypodermic needles of any gauge or length to administer any type of vaccine to people aged from birth to 24 years.

DATA COLLECTION AND ANALYSIS

Three review authors independently extracted trial data and assessed the risk of bias. We contacted trial authors for additional information. We rated the quality of evidence using the GRADE system.

MAIN RESULTS

We included five trials involving 1350 participants in the original review. The updated review identified no new trials. The evidence from two small trials (one trial including infants and one including adolescents) was insufficient to allow any definitive statements to be made about the effects of the needles evaluated in the trials on vaccine immunogenicity and reactogenicity.The remaining three trials (1135 participants) contributed data to comparisons between 25 G 25 mm, 23 G 25 mm, and 25 G 16 mm needles. These trials included infants predominantly aged from two to six months undergoing intramuscular vaccination in the anterolateral thigh using the World Health Organization (WHO) injection technique (skin stretched flat, needle inserted at a 90° angle and up to the needle hub in healthy infants). The vaccines administered were combination vaccines containing diphtheria, tetanus, and whole-cell pertussis antigens (DTwP). In some trials, the vaccines also contained Haemophilus influenzae type b (DTwP-Hib) and hepatitis B (DTwP-Hib-Hep B) antigen components.Primary outcomesIncidence of vaccine-preventable diseases: No trials reported this outcome.Procedural pain and crying: Using a wider gauge 23 G 25 mm needle may slightly reduce procedural pain (low-quality evidence) and probably leads to a slight reduction in the duration of crying time immediately after vaccination (moderate-quality evidence) compared with a narrower gauge 25 G 25 mm needle (one trial, 320 participants). The effects are probably not large enough to be clinically relevant.Secondary outcomesImmune response: There is probably little or no difference in immune response, defined in terms of the proportion of seroprotected infants, between use of 25 G 25 mm, 23 G 25 mm, or 25 G 16 mm needles to administer a series of three doses of a DTwP-Hib vaccine at ages two, three, and four months (moderate-quality evidence, one trial, numbers of participants in analyses range from 309 to 402. The immune response to the pertussis antigen was not measured).Severe and non-severe local reactions: 25 mm needles (either 25 G or 23 G) probably lead to fewer severe and non-severe local reactions after DTwP-Hib vaccination compared with 25 G 16 mm needles (moderate-quality evidence, one trial, 447 to 458 participants in analyses). We estimate that one fewer infant will experience a severe local reaction (extensive redness and swelling) after the first vaccine dose for every 25 infants vaccinated with the longer rather than the shorter needle (number needed to treat for an additional beneficial outcome (NNTB) with a 25 G 25 mm needle: 25 (95% confidence interval (CI) 15 to 100); NNTB with a 23 G 25 mm needle: 25 (95% CI 17 to 100)). We estimate that one fewer infant will experience a non-severe local reaction (any redness, swelling, tenderness, or hardness (composite outcome)) at 24 hours after the first vaccine dose for every 5 or 6 infants vaccinated with a 25 mm rather than a 16 mm needle (NNTB with a 25 G 25 mm needle: 5 (95% CI 4 to 10); NNTB with a 23 G 25 mm needle: 6 (95% CI 4 to 13)). The results are similar after the second and third vaccine doses.Using a narrow gauge 25 G 25 mm needle may produce a small reduction in the incidence of local reactions after each dose of a DTwP vaccine compared with a wider gauge 23 G 25 mm needle, but the effect estimates are imprecise (low-quality evidence, two trials, 100 to 459 participants in analyses).Systemic reactions: The comparative effects of 23 G 25 mm, 25 G 25 mm, and 25 G 16 mm needles on the incidence of postvaccination fever and other systemic events such as drowsiness, loss of appetite, and vomiting are uncertain due to the very low quality of the evidence.

AUTHORS' CONCLUSIONS

Using 25 mm needles (either 23 G or 25 G) for intramuscular vaccination procedures in the anterolateral thigh of infants using the WHO injection technique probably reduces the occurrence of local reactions while achieving a comparable immune response to 25 G 16 mm needles. These findings are applicable to healthy infants aged two to six months receiving combination DTwP vaccines with a reactogenic whole-cell pertussis antigen component. These vaccines are predominantly used in low- and middle-income countries. The applicability of the findings to vaccines with acellular pertussis components and other vaccines with different reactogenicity profiles is uncertain.

Exercise for rheumatoid arthritis of the hand

BACKGROUND

Rheumatoid arthritis is an inflammatory polyarthritis that frequently affects the hands and wrists. Hand exercises are prescribed to improve mobility and strength, and thereby hand function.

OBJECTIVES

To determine the benefits and harms of hand exercise in adults with rheumatoid arthritis.

SEARCH METHODS

We searched the Cochrane Central Register of Controlled Trials (CENTRAL) (the Cochrane Library), MEDLINE, Embase, CINAHL, AMED, Physiotherapy Evidence Database (PEDro), OTseeker, Web of Science, ClinicalTrials.gov and the World Health Organization International Clinical Trials Registry Platform (WHO ICTRP) up to July 2017.

SELECTION CRITERIA

We considered all randomised or quasi-randomised controlled trials that compared hand exercise with any non-exercise therapy.

DATA COLLECTION AND ANALYSIS

We used standard methodological procedures as outlined by the Cochrane Musculoskeletal Group.

MAIN RESULTS

We included seven studies involving 841 people (aged 20 to 94 years) in the review. Most studies used validated diagnostic criteria and involved home programmes.Very low-quality evidence (due to risk of bias and imprecision) from one study indicated uncertainty about whether exercise improves hand function in the short term (< 3 months). On a 0 to 80 points hand function test (higher scores mean better function), the exercise group (n = 11) scored 76.1 points and control group (n = 13) scored 75 points.Moderate-quality evidence (due to risk of bias) from one study indicated that exercise compared to usual care probably slightly improves hand function (mean difference (MD) 4.5, 95% confidence interval (CI) 1.58 to 7.42; n = 449) in the medium term (3 to 11 months) and in the long term (12 months or beyond) (MD 4.3, 95% CI 0.86 to 7.74; n = 438). The absolute change on a 0-to-100 hand function scale (higher scores mean better function) and number needed to treat for an additional beneficial outcome (NNTB) were 5% (95% CI 2% to 7%); 8 (95% CI 5 to 20) and 4% (95% CI 1% to 8%); 9 (95% CI 6 to 27), respectively. A 4% to 5% improvement indicates a minimal clinical benefit.Very low-quality evidence (due to risk of bias and imprecision) from two studies indicated uncertainty about whether exercise compared to no treatment improved pain (MD -27.98, 95% CI -48.93 to -7.03; n = 124) in the short term. The absolute change on a 0-to-100-millimetre scale (higher scores mean more pain) was -28% (95% CI -49% to -7%) and NNTB 2 (95% CI 2 to 11).Moderate-quality evidence (due to risk of bias) from one study indicated that there is probably little or no difference between exercise and usual care on pain in the medium (MD -2.8, 95% CI - 6.96 to 1.36; n = 445) and long term (MD -3.7, 95% CI -8.1 to 0.7; n = 437). On a 0-to-100 scale, the absolute changes were -3% (95% CI -7% to 2%) and -4% (95% CI -8% to 1%), respectively.Very low-quality evidence (due to risk of bias and imprecision) from three studies (n = 141) indicated uncertainty about whether exercise compared to no treatment improved grip strength in the short term. The standardised mean difference for the left hand was 0.44 (95% CI 0.11 to 0.78), re-expressed as 3.5 kg (95% CI 0.87 to 6.1); and for the right hand 0.46 (95% CI 0.13 to 0.8), re-expressed as 4 kg (95% CI 1.13 to 7).High-quality evidence from one study showed that exercise compared to usual care has little or no benefit on mean grip strength (in kg) of both hands in the medium term (MD 1.4, 95% CI -0.27 to 3.07; n = 400), relative change 11% (95% CI -2% to 13%); and in the long term (MD 1.2, 95% CI -0.62 to 3.02; n = 355), relative change 9% (95% CI -5% to 23%).Very low-quality evidence (due to risk of bias and imprecision) from two studies (n = 120) indicated uncertainty about whether exercise compared to no treatment improved pinch strength (in kg) in the short term. The MD and relative change for the left and right hands were 0.51 (95% CI 0.13 to 0.9) and 44% (95% CI 11% to 78%); and 0.82 (95% CI 0.43 to 1.21) and 68% (95% CI 36% to 101%).High-quality evidence from one study showed that exercise compared to usual care has little or no benefit on mean pinch strength of both hands in the medium (MD 0.3, 95% CI -0.14 to 0.74; n = 396) and long term (MD 0.4, 95% CI -0.08 to 0.88; n = 351). The relative changes were 8% (95% CI -4% to 19%) and 10% (95% CI -2% to 22%).No study evaluated the American College of Rheumatology 50 criteria.Moderate-quality evidence (due to risk of bias) from one study indicated that people who also received exercise with strategies for adherence were probably more adherent than those who received routine care alone in the medium term (risk ratio 1.31, 95% CI 1.15 to 1.48; n = 438) and NNTB 6 (95% CI 4 to 10). In the long term, the risk ratio was 1.09 (95% CI 0.93 to 1.28; n = 422).Moderate-quality evidence (due to risk of bias) from one study (n = 246) indicated no adverse events with exercising. The other six studies did not report adverse events.

AUTHORS' CONCLUSIONS

It is uncertain whether exercise improves hand function or pain in the short term. It probably slightly improves function but has little or no difference on pain in the medium and long term. It is uncertain whether exercise improves grip and pinch strength in the short term, and probably has little or no difference in the medium and long term. The ACR50 response is unknown. People who received exercise with adherence strategies were probably more adherent in the medium term than who did not receive exercise, but with little or no difference in the long term. Hand exercise probably does not lead to adverse events. Future research should consider hand and wrist function as their primary outcome, describe exercise following the TIDieR guidelines, and evaluate behavioural strategies.

Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions

Background

Recent methodologic evidence suggests that lack of blinding in randomized trials can result in under- or overestimation of the treatment effect size. The objective of this study is to quantify the extent of bias associated with blinding in randomized controlled trials of oral health interventions.

Methods

We selected all oral health meta-analyses that included a minimum of five randomized controlled trials. We extracted data, in duplicate, related to nine blinding-related criteria, namely: patient blinding, assessor blinding, care-provider blinding, investigator blinding, statistician blinding, blinding of both patients and assessors, study described as “double blind”, blinding of patients, assessors, and care providers concurrently, and the appropriateness of blinding. We quantified the impact of bias associated with blinding on the magnitude of effect size using a two-level meta-meta-analytic approach with a random effects model to allow for intra- and inter-meta-analysis heterogeneity.

Results

We identified 540 randomized controlled trials, included in 64 meta-analyses, analyzing data from 137,957 patients. We identified significantly larger treatment effect size estimates in trials that had inadequate patient blinding (difference in treatment effect size = 0.12; 95% CI: 0.00 to 0.23), lack of blinding of both patients and assessors (difference = 0.19; 95% CI: 0.06 to 0.32), and lack of blinding of patients, assessors, and care-providers concurrently (difference = 0.14; 95% CI: 0.03 to 0.25). In contrast, assessor blinding (difference = 0.06; 95% CI: -0.06 to 0.18), caregiver blinding (difference = 0.02; 95% CI: -0.04 to 0.09), principal-investigator blinding (difference = − 0.02; 95% CI: -0.10 to 0.06), describing a trial as “double-blind” (difference = 0.09; 95% CI: -0.05 to 0.22), and lack of an appropriate method of blinding (difference = 0.06; 95% CI: -0.06 to 0.18) were not associated with over- or underestimated treatment effect size.

Conclusions

We found significant differences in treatment effect size estimates between oral health trials based on lack of patient and assessor blinding. Treatment effect size estimates were 0.19 and 0.14 larger in trials with lack of blinding of both patients and assessors and blinding of patients, assessors, and care-providers concurrently. No significant differences were identified in other blinding criteria. Investigators of oral health systematic reviews should perform sensitivity analyses based on the adequacy of blinding in included trials.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0491-0) contains supplementary material, which is available to authorized users.

Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study

Flaws in the design of randomized trials may bias intervention effect estimates and increase between-trial heterogeneity. Empirical evidence suggests that these problems are greatest for subjectively assessed outcomes. For the Risk of Bias in Evidence Synthesis (ROBES) Study, we extracted risk-of-bias judgements (for sequence generation, allocation concealment, blinding, and incomplete data) from a large collection of meta-analyses published in the Cochrane Library (issue 4; April 2011). We categorized outcome measures as mortality, other objective outcome, or subjective outcome, and we estimated associations of bias judgements with intervention effect estimates using Bayesian hierarchical models. Among 2,443 randomized trials in 228 meta-analyses, intervention effect estimates were, on average, exaggerated in trials with high or unclear (versus low) risk-of-bias judgements for sequence generation (ratio of odds ratios (ROR) = 0.91, 95% credible interval (CrI): 0.86, 0.98), allocation concealment (ROR = 0.92, 95% CrI: 0.86, 0.98), and blinding (ROR = 0.87, 95% CrI: 0.80, 0.93). In contrast to previous work, we did not observe consistently different bias for subjective outcomes compared with mortality. However, we found an increase in between-trial heterogeneity associated with lack of blinding in meta-analyses with subjective outcomes. Inconsistency in criteria for risk-of-bias judgements applied by individual reviewers is a likely limitation of routinely collected bias assessments. Inadequate randomization and lack of blinding may lead to exaggeration of intervention effect estimates in randomized trials.

A systematic review of predictors, moderators, and mediators of augmentative and alternative communication (AAC) outcomes for children with autism spectrum disorder

Documented variability in augmentative and alternative communication (AAC) intervention outcomes makes it difficult for clinicians to select systems most likely to be effective for individual children with autism spectrum disorder (ASD). The aim of this study was to identify child-related factors associated with AAC intervention outcomes through a systematic review of the research literature. A search was conducted of peer-reviewed research articles in which AAC intervention outcomes and associated factors were reported for children with ASD. The search yielded 965 titles and abstracts, of which seven articles relating to six studies met criteria for inclusion. In total, 18 factors were examined, of which nine were assessed as predictors (e.g., cognition, ASD severity, language use), three as moderators (e.g., joint attention, object exploration), and six as mediators (e.g., frequency of therapy, communication partner knowledge). Child characteristics associated with communication outcomes were pre-intervention cognition, severity of ASD, verbal imitation, vocabulary comprehension, object use, joint attention, language use, and two multi-dimensional measures of communication competence. This study suggests emerging evidence for predictors, but less is known about what factors moderate and mediate response to AAC interventions.

Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review

BACKGROUND

Several scales, checklists and domain-based tools for assessing risk of reporting biases exist, but it is unclear how much they vary in content and guidance. We conducted a systematic review of the content and measurement properties of such tools.

METHODS

We searched for potentially relevant articles in Ovid MEDLINE, Ovid Embase, Ovid PsycINFO and Google Scholar from inception to February 2017. One author screened all titles, abstracts and full text articles, and collected data on tool characteristics.

RESULTS

We identified 18 tools that include an assessment of the risk of reporting bias. Tools varied in regard to the type of reporting bias assessed (eg, bias due to selective publication, bias due to selective non-reporting), and the level of assessment (eg, for the study as a whole, a particular result within a study or a particular synthesis of studies). Various criteria are used across tools to designate a synthesis as being at 'high' risk of bias due to selective publication (eg, evidence of funnel plot asymmetry, use of non-comprehensive searches). However, the relative weight assigned to each criterion in the overall judgement is unclear for most of these tools. Tools for assessing risk of bias due to selective non-reporting guide users to assess a study, or an outcome within a study, as 'high' risk of bias if no results are reported for an outcome. However, assessing the corresponding risk of bias in a synthesis that is missing the non-reported outcomes is outside the scope of most of these tools. Inter-rater agreement estimates were available for five tools.

CONCLUSION

There are several limitations of existing tools for assessing risk of reporting biases, in terms of their scope, guidance for reaching risk of bias judgements and measurement properties. Development and evaluation of a new, comprehensive tool could help overcome present limitations.

Applying the ROBINS-I tool to natural experiments: an example from public health

BACKGROUND

A new tool to assess Risk of Bias In Non-randomised Studies of Interventions (ROBINS-I) was published in Autumn 2016. ROBINS-I uses the Cochrane-approved risk of bias (RoB) approach and focusses on internal validity. As such, ROBINS-I represents an important development for those conducting systematic reviews which include non-randomised studies (NRS), including public health researchers. We aimed to establish the applicability of ROBINS-I using a group of NRS which have evaluated non-clinical public health natural experiments.

METHODS

Five researchers, all experienced in critical appraisal of non-randomised studies, used ROBINS-I to independently assess risk of bias in five studies which had assessed the health impacts of a domestic energy efficiency intervention. ROBINS-I assessments for each study were entered into a database and checked for consensus across the group. Group discussions were used to identify reasons underpinning lack of consensus for specific questions and bias domains.

RESULTS

ROBINS-I helped to systematically articulate sources of bias in NRS. However, the lack of consensus in assessments for all seven bias domains raised questions about ROBINS-I's reliability and applicability for natural experiment studies. The two RoB domains with least consensus were selection (Domain 2) and performance (Domain 4). Underlying the lack of consensus were difficulties in applying an intention to treat or per protocol effect of interest to the studies. This was linked to difficulties in determining whether the intervention status was classified retrospectively at follow-up, i.e. post hoc. The overall risk of bias ranged from moderate to critical; this was most closely linked to the assessment of confounders.

CONCLUSION

The ROBINS-I tool is a conceptually rigorous tool which focusses on risk of bias due to the counterfactual. Difficulties in applying ROBINS-I may be due to poor design and reporting of evaluations of natural experiments. While the quality of reporting may improve in the future, improved guidance on applying ROBINS-I is needed to enable existing evidence from natural experiments to be assessed appropriately and consistently. We hope future refinements to ROBINS-I will address some of the issues raised here to allow wider use of the tool.