How well can we assess the validity of non-randomised studies of medications? A systematic review of assessment tools

Tools for assessing quality of studies investigating health interventions using real-world data: a literature review and content analysis

OBJECTIVES

We aimed to identify existing appraisal tools for non-randomised studies of interventions (NRSIs) and to compare the criteria that the tools provide at the quality-item level.

DESIGN

Literature review through three approaches: systematic search of journal articles, snowballing search of reviews on appraisal tools and grey literature search on websites of health technology assessment (HTA) agencies.

DATA SOURCES

Systematic search: Medline; Snowballing: starting from three articles (D'Andrea et al, Quigley et al and Faria et al); Grey literature: websites of European HTA agencies listed by the International Network of Agencies for Health Technology Assessment. Appraisal tools were searched through April 2022.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

We included a tool, if it addressed quality concerns of NRSIs and was published in English (unless from grey literature). A tool was excluded, if it was only for diagnostic, prognostic, qualitative or secondary studies.

DATA EXTRACTION AND SYNTHESIS

Two independent researchers searched, screened and reviewed all included studies and tools, summarised quality items and scored whether and to what extent a quality item was described by a tool, for either methodological quality or reporting.

RESULTS

Forty-nine tools met inclusion criteria and were included for the content analysis. Concerns regarding the quality of NRSI were categorised into 4 domains and 26 items. The Research Triangle Institute Item Bank (RTI Item Bank) and STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) were the most comprehensive tools for methodological quality and reporting, respectively, as they addressed (n=20; 17) and sufficiently described (n=18; 13) the highest number of items. However, none of the tools covered all items.

CONCLUSION

Most of the tools have their own strengths, but none of them could address all quality concerns relevant to NRSIs. Even the most comprehensive tools can be complemented by several items. We suggest decision-makers, researchers and tool developers consider the quality-item level heterogeneity, when selecting a tool or identifying a research gap.

OSF REGISTRATION NUMBER

OSF registration DOI (https://doi.org/10.17605/OSF.IO/KCSGX).

Comparing Effectiveness and Safety of SGLT2 Inhibitors vs DPP-4 Inhibitors in Patients With Type 2 Diabetes and Varying Baseline HbA1c Levels

IMPORTANCE

Sodium-glucose cotransporter 2 inhibitor (SGLT2i) therapy has been associated with cardiovascular benefits and a few adverse events; however, whether the comparative effectiveness and safety profiles vary with differences in baseline hemoglobin A1c (HbA1c) levels is unknown.

OBJECTIVE

To compare cardiovascular effectiveness and safety of treatment with SGLT2i vs dipeptidyl peptidase 4 inhibitor (DPP-4i) in adults with type 2 diabetes (T2D) (1) overall and (2) at varying baseline HbA1c levels.

DESIGN, SETTING, AND PARTICIPANTS

A new-user comparative effectiveness and safety research study was conducted among 144 614 commercially insured adults, initiating treatment with SGLT2i or DPP-4i and with a recorded T2D diagnosis at baseline and at least 1 HbA1c laboratory result recorded within 3 months before treatment initiation.

INTERVENTIONS

The intervention consisted of the initiation of treatment with SGLT2i or DPP-4i.

MAIN OUTCOMES AND MEASURES

Primary outcomes were a composite of myocardial infarction, stroke, or all-cause death (modified major adverse cardiovascular events [MACE]) and hospitalization for heart failure (HHF). Safety outcomes were hypovolemia, fractures, falls, genital infections, diabetic ketoacidosis (DKA), acute kidney injury (AKI), and lower-limb amputation. Incidence rate (IR) per 1000 person-years, hazard ratios (HR) and rate differences (RD) with their 95% CIs were estimated controlling for 128 covariates.

RESULTS

A total of 144 614 eligible adults (mean [SD] age, 62 [12.4] years; 54% male participants) with T2D initiating treatment with a SGLT2i (n = 60 523) or a DPP-4i (n = 84 091) were identified; 44 099 had an HbA1c baseline value of less than 7.5%, 52 986 between 7.5% and 9%, and 47 529 greater than 9%. Overall, 87 274 eligible patients were 1:1 propensity score-matched: 24 052 with HbA1c less than 7.5%; 32 290 with HbA1c between 7.5% and 9%; and 30 932 with HbA1c greater than 9% (to convert percentage of total hemoglobin to proportion of total hemoglobin, multiply by 0.01). The initiation of SGLT2i vs DPP-4i was associated with a reduction in the risk of modified MACE (IR per 1000 person-years 17.13 vs 20.18, respectively; HR, 0.85; 95% CI, 0.75-0.95; RD, -3.02; 95% CI, -5.23 to -0.80) and HHF (IR per 1000 person-years 3.68 vs 8.08, respectively; HR, 0.46; 95% CI, 0.35 to 0.57; RD -4.37; 95% CI, -5.62 to -3.12) over a mean follow-up of 8 months, with no evidence of treatment effect heterogeneity across the HbA1c levels. Treatment with SGLT2i showed an increased risk of genital infections and DKA and a reduced AKI risk compared with DPP-4i. Findings were consistent by HbA1c levels, except for a more pronounced risk of genital infections associated with SGLT2i for HbA1c levels of 7.5% to 9% (IR per 1000 person-years 68.5 vs 22.8, respectively; HR, 3.10; 95% CI, 2.68-3.58; RD, 46.22; 95% CI, 40.54-51.90).

CONCLUSIONS AND RELEVANCE

In this comparative effectiveness and safety research study among adults with T2D, SGLT2i vs DPP-4i treatment initiators had a reduced risk of modified MACE and HHF, an increased risk of genital infections and DKA, and a lower risk of AKI, regardless of baseline HbA1c.

Maternal Multiple Sclerosis and Health Outcomes Among the Children: A Systematic Review

Objective

To summarize the available literature and provide an overview of in utero exposure to maternal multiple sclerosis (MS) and the influence on offspring health outcomes.

Methods

We conducted a systematic review by searching Embase, Medline and PubMed.gov databases, and we used covidence.org to conduct a thorough sorting of the articles into three groups; 1) women with MS and the influence on birth outcomes; 2) women with MS treated with disease-modifying therapy (DMT) during pregnancy and the influence on birth outcomes; and 3) women with MS and the influence on long-term health outcomes in the children.

Results

In total, 22 cohort studies were identified. Ten studies reported on MS without DMT and compared with a control group without MS, and nine studies on women with MS and DMT prior to or during pregnancy met the criteria. We found only four studies reporting on long-term child health outcomes. One study had results belonging to more than one group.

Conclusion

The studies pointed towards an increased risk of preterm birth and small for gestational age among women with MS. In terms of women with MS treated with DMT prior to or during pregnancy, no clear conclusions could be reached. The few studies on long-term child outcomes all had different outcomes within the areas of neurodevelopment and psychiatric impairment. In this systematic review, we have highlighted the research gaps on the impact of maternal MS on offspring health.

How to assess applicability and methodological quality of comparative studies of operative interventions in orthopedic trauma surgery

PURPOSE

It is challenging to generate and subsequently implement high-quality evidence in surgical practice. A first step would be to grade the strengths and weaknesses of surgical evidence and appraise risk of bias and applicability. Here, we described items that are common to different risk-of-bias tools. We explained how these could be used to assess comparative operative intervention studies in orthopedic trauma surgery, and how these relate to applicability of results.

METHODS

We extracted information from the Cochrane risk-of-bias-2 (RoB-2) tool, Risk Of Bias In Non-randomised Studies-of Interventions tool (ROBINS-I), and Methodological Index for Non-Randomized Studies (MINORS) criteria and derived a concisely formulated set of items with signaling questions tailored to operative interventions in orthopedic trauma surgery.

RESULTS

The established set contained nine items: population, intervention, comparator, outcome, confounding, missing data and selection bias, intervention status, outcome assessment, and pre-specification of analysis. Each item can be assessed using signaling questions and was explained using good practice examples of operative intervention studies in orthopedic trauma surgery.

CONCLUSION

The set of items will be useful to form a first judgment on studies, for example when including them in a systematic review. Existing risk of bias tools can be used for further evaluation of methodological quality. Additionally, the proposed set of items and signaling questions might be a helpful starting point for peer reviewers and clinical readers.

Cardiovascular Risks of Hydroxychloroquine vs Methotrexate in Patients With Rheumatoid Arthritis

BACKGROUND

Hydroxychloroquine is often used as a first-line treatment of rheumatoid arthritis despite limited evidence on its cardiovascular risk.

OBJECTIVES

We conducted a cardiovascular safety evaluation comparing hydroxychloroquine to methotrexate among patients with rheumatoid arthritis.

METHODS

Using Medicare data (2008-2016), we identified 54,462 propensity score-matched patients with rheumatoid arthritis, aged ≥65 years, who initiated hydroxychloroquine or methotrexate. Primary outcomes were sudden cardiac arrest or ventricular arrythmia (SCA/VA) and major adverse cardiovascular event (MACE). Secondary outcomes were cardiovascular mortality, all-cause mortality, myocardial infarction, stroke, and hospitalized heart failure (HF). We also examined treatment effect modification by history of HF.

RESULTS

Hydroxychloroquine was not associated with risk of SCA/VA (HR: 1.03; 95% CI: 0.79-1.35) or MACE (HR: 1.07; 95% CI: 0.97-1.18) compared with methotrexate. In patients with history of HF, hydroxychloroquine initiators had a higher risk of MACE (HR: 1.30; 95% CI: 1.08-1.56), cardiovascular mortality (HR: 1.34; 95% CI: 1.06-1.70), all-cause mortality (HR: 1.22; 95% CI: 1.04-1.43), myocardial infarction (HR: 1.74; 95% CI: 1.25-2.42), and hospitalized HF (HR: 1.29; 95% CI: 1.07-1.54) compared to methotrexate initiators. Cardiovascular risks were not different in patients without history of HF except for an increased hospitalized HF risk (HR: 1.57; 95% CI: 1.30-1.90) among hydroxychloroquine initiators.

CONCLUSIONS

In older patients with rheumatoid arthritis, hydroxychloroquine and methotrexate showed similar SCA/VA and MACE risks; however, hydroxychloroquine initiators with history of HF had higher risks of MACE, cardiovascular mortality, all-cause mortality, and myocardial infarction. An increased hospitalized HF risk was observed among hydroxychloroquine initiators regardless of an HF history.

Risk of bias in non-randomized observational studies assessing the relationship between proton-pump inhibitors and adverse kidney outcomes: a systematic review

BACKGROUND

Proton-pump inhibitors (PPIs) are widely prescribed as acid-suppression therapy. Some observational studies suggest that long-term use of PPIs is potentially associated with certain adverse kidney outcomes. We conducted a systematic literature review to assess potential bias in non-randomized studies reporting on putative associations between PPIs and adverse kidney outcomes (acute kidney injury, acute interstitial nephritis, chronic interstitial nephritis, acute tubular necrosis, chronic kidney disease, and end-stage renal disease).

METHODS

We searched the medical literature within 10 years of 17 December 2020. Pre-specified criteria guided identification of relevant English language articles for assessment. Risk of bias on an outcome-specific basis was evaluated using the Risk of Bias in Non-randomized Studies of Interventions (ROBINS-I) tool by two independent reviewers.

RESULTS

Of 620 initially identified records, 26 studies met a priori eligibility criteria and underwent risk of bias assessment. Nineteen studies were judged as having a moderate risk of bias for reported adverse kidney outcomes, while six studies were judged as having a serious risk of bias (mainly due to inadequate control of confounders and selection bias). We were unable to determine the overall risk of bias in two studies (one of which was assessed as having a moderate risk of bias for a different adverse kidney outcome) due to insufficient information presented. Effect estimates for PPIs in relation to adverse kidney outcomes varied widely (0.24-7.34) but associations mostly showed increased risk.

CONCLUSION

Using ROBINS-I, we found that non-randomized observational studies suggesting kidney harm by PPIs have moderate to serious risk of bias, making it challenging to establish causality. Additional high-quality, real-world evidence among generalizable populations are needed to better understand the relation between PPI treatment and acute and chronic kidney outcomes, accounting for the effects of varying durations of PPI treatment, self-treatment with over-the-counter PPIs, and potential critical confounders.

Estimating Cardiovascular Impacts of Drugs for Psoriatic Disease: A Long Way to Go

Patients with psoriasis and psoriatic arthritis are at an increased risk of cardiovascular (CV) events. A recent systematic review and meta-analysis by González Cantero et al. (2021) evaluated the effects of biologics on CV imaging and biomarkers in patients with psoriasis. In this commentary, we discuss the clinical and management implications of these and the related results for patients with psoriatic disease and the need for further pharmacoepidemiological research.

The use of nonrandomized evidence to estimate treatment effects in health technology assessment

Health technology assessment (HTA) is increasingly informed by nonrandomized studies, but there is limited guidance from HTA bodies on expectations around evidence quality and study conduct. We developed recommendations to support the appropriate use of such evidence based on a pragmatic literature review and a workshop involving 16 experts from eight countries as part of the EU's Horizon-2020 IMPACT-HTA program (work package six). To ensure HTA processes remain rigorous and robust, HTA bodies should demand clear, extensive and structured reporting of nonrandomized studies, including an in-depth assessment of the risk of bias. In recognition of the additional uncertainty imparted by nonrandomized designs in estimates of treatment effects, HTA bodies should strengthen early scientific advice and engage in collaborative efforts to improve use of real-world data.