Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden

The absence of data on driving under the influence of alcohol in road traffic studies: a scoping review of non-randomized studies with vote counting based on the direction of effects of alcohol policies

BACKGROUND

Data on driving under the influence of alcohol (DUIA) are not always available, accurate, or reliable, making it difficult to study the effects of alcohol policies on road traffic outcomes. The objectives of our study were twofold: 1) to describe how road traffic outcomes of alcohol policies are assessed when DUIA data are missing, and 2) to explore the effects of alcohol policies when DUIA data are missing.

METHODS

We conducted a scoping review of non-randomized studies that assessed the road traffic outcomes of alcohol policies when DUIA data are missing. Until November 2021, we searched studies published between 2000 and 2021, in English or French, via MEDLINE, APA PsycInfo, CINAHL, and SocINDEX. We assessed the risk of bias in the included studies with the Quality Assessment Tool for Before-After (Pre-Post) Studies With No Control Group. The selection process, data extraction, and the risk of bias assessment were conducted independently and in duplicate. We used vote counting based on the direction of the effects of alcohol policies as a synthesis method. The protocol for this review was published in PROSPERO under record number CRD42021266744.

RESULTS

Twenty-four eligible studies were included. Regarding objective 1, most studies used uncontrolled interrupted time series designs to assess road traffic fatalities resulting from night-time crashes. The reasons for missing DUIA data were generally not reported. Regarding objective 2, we found evidence for an association between alcohol policies and decreased road traffic fatalities. Subgroup analyses found no evidence for an association between methodological modifiers and positive effect directions for road traffic fatalities.

CONCLUSION

Caution is needed when interpreting road traffic outcomes associated with alcohol policies when DUIA data are missing. Greater efforts should be made to improve the reporting of outcomes assessments. Future studies must address several methodological issues (e.g., more granular data, well-defined intervention and implementation, and controlled designs). Our results should be compared to those from others reviews where DUIA data were available to confirm or recalibrate the associations found in studies where DUIA data were missing.

PROTOCOL: Assessment of outcome reporting bias in studies included in Campbell systematic reviews

This is the protocol for a Campbell systematic review. The objectives are as follows: To identify methods used to assess the risk of outcome reporting bias (ORB) in studies included in recent Campbell systematic reviews of intervention effects. The review will answer the following questions: What proportion of recent Campbell reviews included assessment of ORB? How did recent reviews define levels of risk of ORB (what categories, labels, and definitions did they use)? To what extent and how did these reviews use study protocols as sources of data on ORB? To what extent and how did reviews document reasons for judgments about risk of ORB? To what extent and how did reviews assess the inter-rater reliability of ORB ratings? To what extent and how were issues of ORB considered in the review's abstract, plain language summary, and conclusions?

Levels of Evidence, Quality Assessment, and Risk of Bias: Evaluating the Internal Validity of Primary Research

Clinical decisions in human and veterinary medicine should be based on the best available evidence. The results of primary research are an important component of that evidence base. Regardless of whether assessing studies for clinical case management, developing clinical practice guidelines, or performing systematic reviews, evidence from primary research should be evaluated for internal validity i.e., whether the results are free from bias (reflect the truth). Three broad approaches to evaluating internal validity are available: evaluating the potential for bias in a body of literature based on the study designs employed (levels of evidence), evaluating whether key study design features associated with the potential for bias were employed (quality assessment), and applying a judgement as to whether design elements of a study were likely to result in biased results given the specific context of the study (risk of bias assessment). The level of evidence framework for assessing internal validity assumes that internal validity can be determined based on the study design alone, and thus makes the strongest assumptions. Risk of bias assessments involve an evaluation of the potential for bias in the context of a specific study, and thus involve the least assumptions about internal validity. Quality assessment sits somewhere between the assumptions of these two. Because risk of bias assessment involves the least assumptions, this approach should be used to assess internal validity where possible. However, risk of bias instruments are not available for all study designs, some clinical questions may be addressed using multiple study designs, and some instruments that include an evaluation of internal validity also include additional components (e.g., evaluation of comprehensiveness of reporting, assessments of feasibility or an evaluation of external validity). Therefore, it may be necessary to embed questions related to risk of bias within existing quality assessment instruments. In this article, we overview the approaches to evaluating internal validity, highlight the current complexities, and propose ideas for approaching assessments of internal validity.

How to assess applicability and methodological quality of comparative studies of operative interventions in orthopedic trauma surgery

PURPOSE

It is challenging to generate and subsequently implement high-quality evidence in surgical practice. A first step would be to grade the strengths and weaknesses of surgical evidence and appraise risk of bias and applicability. Here, we described items that are common to different risk-of-bias tools. We explained how these could be used to assess comparative operative intervention studies in orthopedic trauma surgery, and how these relate to applicability of results.

METHODS

We extracted information from the Cochrane risk-of-bias-2 (RoB-2) tool, Risk Of Bias In Non-randomised Studies-of Interventions tool (ROBINS-I), and Methodological Index for Non-Randomized Studies (MINORS) criteria and derived a concisely formulated set of items with signaling questions tailored to operative interventions in orthopedic trauma surgery.

RESULTS

The established set contained nine items: population, intervention, comparator, outcome, confounding, missing data and selection bias, intervention status, outcome assessment, and pre-specification of analysis. Each item can be assessed using signaling questions and was explained using good practice examples of operative intervention studies in orthopedic trauma surgery.

CONCLUSION

The set of items will be useful to form a first judgment on studies, for example when including them in a systematic review. Existing risk of bias tools can be used for further evaluation of methodological quality. Additionally, the proposed set of items and signaling questions might be a helpful starting point for peer reviewers and clinical readers.

Assessing the causal relationship between income inequality and mortality and self-rated health: protocol for systematic review and meta-analysis

BACKGROUND

Income inequality has been linked to health and mortality. While there has been extensive research exploring the relationship, the evidence for whether the relationship is causal remains disputed. We describe the methods for a systematic review that will transparently assess whether a causal relationship exists between income inequality and mortality and self-rated health.

METHODS

We will identify relevant studies using search terms relating to income inequality, mortality, and self-rated health (SRH). Four databases will be searched: MEDLINE, ISI Web of Science, EMBASE, and the National Bureau of Economic Research. The inclusion criteria have been developed to identify the study designs best suited to assess causality: multilevel studies that have conditioned upon individual income (or a comparable measure, such as socioeconomic position) and natural experiment studies. Risk of bias assessment of included studies will be conducted using ROBINS-I. Where possible, we will convert all measures of income inequality into Gini coefficients and standardize the effect estimate of income inequality on mortality/SRH. We will conduct random-effects meta-analysis to estimate pooled effect estimates when possible. We will assess causality using modified Bradford Hill viewpoints and assess certainty using GRADE.

DISCUSSION

This systematic review protocol lays out the complexity of the relationship between income inequality and individual health, as well as our approach for assessing causality. Understanding whether income inequality impacts the health of individuals within a population has major policy implications. By setting out our methods and approach as transparently as we can, we hope this systematic review can provide clarity to an important topic for public policy and public health, as well as acting as an exemplar for other "causal reviews".

Assessments of risk of bias in systematic reviews of observational nutritional epidemiologic studies are often not appropriate or comprehensive: a methodological study

BACKGROUND

An essential component of systematic reviews is the assessment of risk of bias. To date, there has been no investigation of how reviews of non-randomised studies of nutritional exposures (called 'nutritional epidemiologic studies') assess risk of bias.

OBJECTIVE

To describe methods for the assessment of risk of bias in reviews of nutritional epidemiologic studies.

METHODS

We searched MEDLINE, EMBASE and the Cochrane Database of Systematic Reviews (Jan 2018-Aug 2019) and sampled 150 systematic reviews of nutritional epidemiologic studies.

RESULTS

Most reviews (n=131/150; 87.3%) attempted to assess risk of bias. Commonly used tools neglected to address all important sources of bias, such as selective reporting (n=25/28; 89.3%), and frequently included constructs unrelated to risk of bias, such as reporting (n=14/28; 50.0%). Most reviews (n=66/101; 65.3%) did not incorporate risk of bias in the synthesis. While more than half of reviews considered biases due to confounding and misclassification of the exposure in their interpretation of findings, other biases, such as selective reporting, were rarely considered (n=1/150; 0.7%).

CONCLUSION

Reviews of nutritional epidemiologic studies have important limitations in their assessment of risk of bias.

Cochrane's risk of bias tool for non-randomized studies (ROBINS-I) is frequently misapplied: A methodological systematic review

OBJECTIVES

We aimed to review how 'Risk of Bias In Non-randomized Studies-of Interventions' (ROBINS-I), a Cochrane risk of bias assessment tool, has been used in recent systematic reviews.

STUDY DESIGN AND SETTING

Database and citation searches were conducted in March 2020 to identify recently published reviews using ROBINS-I. Reported ROBINS-I assessments and data on how ROBINS-I was used were extracted from each review. Methodological quality of reviews was assessed using AMSTAR 2 ('A MeaSurement Tool to Assess systematic Reviews').

RESULTS

Of 181 hits, 124 reviews were included. Risk of bias was serious/critical in 54% of assessments on average, most commonly due to confounding. Quality of reviews was mostly low, and modifications and incorrect use of ROBINS-I were common, with 20% reviews modifying the rating scale, 20% understating overall risk of bias, and 19% including critical-risk of bias studies in evidence synthesis. Poorly conducted reviews were more likely to report low/moderate risk of bias (predicted probability 57% [95% CI: 47-67] in critically low-quality reviews, 31% [19-46] in high/moderate-quality reviews).

CONCLUSION

Low-quality reviews frequently apply ROBINS-I incorrectly, and may thus inappropriately include or give too much weight to uncertain evidence. Readers should be aware that such problems can lead to incorrect conclusions in reviews.

Resource use during systematic review production varies widely: a scoping review

OBJECTIVE

We aimed to map the resource use during systematic review (SR) production and reasons why steps of the SR production are resource intensive to discover where the largest gain in improving efficiency might be possible.

STUDY DESIGN AND SETTING

We conducted a scoping review. An information specialist searched multiple databases (e.g., Ovid MEDLINE, Scopus) and implemented citation-based and grey literature searching. We employed dual and independent screenings of records at the title/abstract and full-text levels and data extraction.

RESULTS

We included 34 studies. Thirty-two reported on the resource use-mostly time; four described reasons why steps of the review process are resource intensive. Study selection, data extraction, and critical appraisal seem to be very resource intensive, while protocol development, literature search, or study retrieval take less time. Project management and administration required a large proportion of SR production time. Lack of experience, domain knowledge, use of collaborative and SR-tailored software, and good communication and management can be reasons why SR steps are resource intensive.

CONCLUSION

Resource use during SR production varies widely. Areas with the largest resource use are administration and project management, study selection, data extraction, and critical appraisal of studies.

Customized guidance/training improved the psychometric properties of methodologically rigorous risk of bias instruments for non-randomized studies

OBJECTIVES

To evaluate the impact of guidance and training on the inter-rater reliability (IRR), inter-consensus reliability (ICR) and evaluator burden of the Risk of Bias (RoB) in Non-randomized Studies (NRS) of Interventions (ROBINS-I) tool, and the RoB instrument for NRS of Exposures (ROB-NRSE).

STUDY DESIGN AND SETTING

In a before-and-after study, seven reviewers appraised the RoB using ROBINS-I (n = 44) and ROB-NRSE (n = 44), before and after guidance and training. We used Gwet's AC₁ statistic to calculate IRR and ICR.

RESULTS

After guidance and training, the IRR and ICR of the overall bias domain of ROBINS-I and ROB-NRSE improved significantly; with many individual domains showing either a significant (IRR and ICR of ROB-NRSE; ICR of ROBINS-I), or nonsignificant improvement (IRR of ROBINS-I). Evaluator burden significantly decreased after guidance and training for ROBINS-I, whereas for ROB-NRSE there was a slight nonsignificant increase.

CONCLUSION

Overall, there was benefit for guidance and training for both tools. We highly recommend guidance and training to reviewers prior to RoB assessments and that future research investigate aspects of guidance and training that are most effective.