Comparison of All-Cause Mortality Following VTE Treatment Between Propensity Score-Adjusted Observational Studies and Matched Randomized Controlled Trials: Meta-Epidemiologic Study

Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials: a meta-epidemiological study

BACKGROUND

Researchers and decision-makers often use evidence from randomised controlled trials (RCTs) to determine the efficacy or effectiveness of a treatment or intervention. Studies with observational designs are often used to measure the effectiveness of an intervention in 'real world' scenarios. Numerous study designs and their modifications (including both randomised and observational designs) are used for comparative effectiveness research in an attempt to give an unbiased estimate of whether one treatment is more effective or safer than another for a particular population. An up-to-date systematic analysis is needed to identify differences in effect estimates from RCTs and observational studies. This updated review summarises the results of methodological reviews that compared the effect estimates of observational studies with RCTs from evidence syntheses that addressed the same health research question.

OBJECTIVES

To assess and compare synthesised effect estimates by study type, contrasting RCTs with observational studies. To explore factors that might explain differences in synthesised effect estimates from RCTs versus observational studies (e.g. heterogeneity, type of observational study design, type of intervention, and use of propensity score adjustment). To identify gaps in the existing research comparing effect estimates across different study types.

SEARCH METHODS

We searched MEDLINE, the Cochrane Database of Systematic Reviews, Web of Science databases, and Epistemonikos to May 2022. We checked references, conducted citation searches, and contacted review authors to identify additional reviews.

SELECTION CRITERIA

We included systematic methodological reviews that compared quantitative effect estimates measuring the efficacy or effectiveness of interventions tested in RCTs versus in observational studies. The included reviews compared RCTs to observational studies (including retrospective and prospective cohort, case-control and cross-sectional designs). Reviews were not eligible if they compared RCTs with studies that had used some form of concurrent allocation.

DATA COLLECTION AND ANALYSIS

Using results from observational studies as the reference group, we examined the relative summary effect estimates (risk ratios (RRs), odds ratios (ORs), hazard ratios (HRs), mean differences (MDs), and standardised mean differences (SMDs)) to evaluate whether there was a relatively larger or smaller effect in the ratio of odds ratios (ROR) or ratio of risk ratios (RRR), ratio of hazard ratios (RHR), and difference in (standardised) mean differences (D(S)MD). If an included review did not provide an estimate comparing results from RCTs with observational studies, we generated one by pooling the estimates for observational studies and RCTs, respectively. Across all reviews, we synthesised these ratios to produce a pooled ratio of ratios comparing effect estimates from RCTs with those from observational studies. In overviews of reviews, we estimated the ROR or RRR for each overview using observational studies as the reference category. We appraised the risk of bias in the included reviews (using nine criteria in total). To receive an overall low risk of bias rating, an included review needed: explicit criteria for study selection, a complete sample of studies, and to have controlled for study methodological differences and study heterogeneity. We assessed reviews/overviews not meeting these four criteria as having an overall high risk of bias. We assessed the certainty of the evidence, consisting of multiple evidence syntheses, with the GRADE approach.

MAIN RESULTS

We included 39 systematic reviews and eight overviews of reviews, for a total of 47. Thirty-four of these contributed data to our primary analysis. Based on the available data, we found that the reviews/overviews included 2869 RCTs involving 3,882,115 participants, and 3924 observational studies with 19,499,970 participants. We rated 11 reviews/overviews as having an overall low risk of bias, and 36 as having an unclear or high risk of bias. Our main concerns with the included reviews/overviews were that some did not assess the quality of their included studies, and some failed to account appropriately for differences between study designs - for example, they conducted aggregate analyses of all observational studies rather than separate analyses of cohort and case-control studies. When pooling RORs and RRRs, the ratio of ratios indicated no difference or a very small difference between the effect estimates from RCTs versus from observational studies (ratio of ratios 1.08, 95% confidence interval (CI) 1.01 to 1.15). We rated the certainty of the evidence as low. Twenty-three of 34 reviews reported effect estimates of RCTs and observational studies that were on average in agreement. In a number of subgroup analyses, small differences in the effect estimates were detected: - pharmaceutical interventions only (ratio of ratios 1.12, 95% CI 1.04 to 1.21); - RCTs and observational studies with substantial or high heterogeneity; that is, I2 ≥ 50% (ratio of ratios 1.11, 95% CI 1.04 to 1.18); - no use (ratio of ratios 1.07, 95% CI 1.03 to 1.11) or unclear use (ratio of ratios 1.13, 95% CI 1.03 to 1.25) of propensity score adjustment in observational studies; and - observational studies without further specification of the study design (ratio of ratios 1.06, 95% CI 0.96 to 1.18). We detected no clear difference in other subgroup analyses.

AUTHORS' CONCLUSIONS

We found no difference or a very small difference between effect estimates from RCTs and observational studies. These findings are largely consistent with findings from recently published research. Factors other than study design need to be considered when exploring reasons for a lack of agreement between results of RCTs and observational studies, such as differences in the population, intervention, comparator, and outcomes investigated in the respective studies. Our results underscore that it is important for review authors to consider not only study design, but the level of heterogeneity in meta-analyses of RCTs or observational studies. A better understanding is needed of how these factors might yield estimates reflective of true effectiveness.

Design, implementation, and inferential issues associated with clinical trials that rely on data in electronic medical records: a narrative review

Real world evidence is now accepted by authorities charged with assessing the benefits and harms of new therapies. Clinical trials based on real world evidence are much less expensive than randomized clinical trials that do not rely on "real world evidence" such as contained in electronic health records (EHR). Consequently, we can expect an increase in the number of reports of these types of trials, which we identify here as 'EHR-sourced trials.' 'In this selected literature review, we discuss the various designs and the ethical issues they raise. EHR-sourced trials have the potential to improve/increase common data elements and other aspects of the EHR and related systems. Caution is advised, however, in drawing causal inferences about the relationships among EHR variables. Nevertheless, we anticipate that EHR-CTs will play a central role in answering research and regulatory questions.

Improved Outcomes After Pulmonary Valve Replacement in Repaired Tetralogy of Fallot

BACKGROUND

The impact of pulmonary valve replacement (PVR) on major adverse clinical outcomes in patients with repaired tetralogy of Fallot (rTOF) is unknown.

OBJECTIVES

The purpose of this study was to determine whether PVR is associated with improved survival and freedom from sustained ventricular tachycardia (VT) in rTOF.

METHODS

A PVR propensity score was created to adjust for baseline differences between PVR and non-PVR patients enrolled in INDICATOR (International Multicenter TOF Registry). The primary outcome was time to the earliest occurrence of death or sustained VT. PVR and non-PVR patients were matched 1:1 on PVR propensity score (matched cohort) and in the full cohort, modeling was performed with propensity score as a covariate adjustment.

RESULTS

Among 1,143 patients with rTOF (age 27 ± 14 years, 47% PVR, follow-up 8.3 ± 5.2 years), the primary outcome occurred in 82. The adjusted HR for the primary outcome for PVR vs no-PVR (matched cohort n = 524) was 0.41 (95% CI: 0.21-0.81; multivariable model P = 0.010). Full cohort analysis revealed similar results. Subgroup analysis suggested beneficial effects in patients with advanced right ventricular (RV) dilatation (interaction P = 0.046; full cohort). In patients with RV end-systolic volume index >80 mL/m2, PVR was associated with a lower primary outcome risk (HR: 0.32; 95% CI: 0.16-0.62; P < 0.001). There was no association between PVR and the primary outcome in patients with RV end-systolic volume index ≤80 mL/m2 (HR: 0.86; 95% CI: 0.38-1.92; P = 0.70).

CONCLUSIONS

Compared with rTOF patients who did not receive PVR, propensity score-matched individuals receiving PVR had lower risk of a composite endpoint of death or sustained VT.

Direct Oral Anticoagulants vs. Vitamin-K Antagonists in the Elderly With Atrial Fibrillation: A Systematic Review Comparing Benefits and Harms Between Observational Studies and Randomized Controlled Trials

Background: The publication of high-quality observational studies (OSs) has fueled reassessment of the treatment effects of direct oral anticoagulants (DOACs) in the elderly with atrial fibrillation (AF). Methods: The MEDLINE, EMBASE, and Cochrane Library databases were systematically searched (through July 1, 2019) for eligible OSs and randomized controlled trials (RCTs) that reported effectiveness outcomes [stroke or systemic embolism (SE)] or safety outcomes [intracranial hemorrhage (ICH), major bleeding, gastrointestinal bleeding (GIB), myocardial infarction (MI), and all-cause mortality] for DOACs and vitamin-K antagonists (VKAs) in elderly AF patients. A random-effects model was applied to calculate adjusted hazard ratios (HRs) for OSs and relative risks (RRs) for RCTs. Interaction analyses and the ratio of HR (RHR) were used to assess and compare OSs and RCTs. Results: A total of 32 studies involving 547,419 patients were included. No significant difference in treatment effect estimates was found between 27 OSs and 5 RCTs [P _interaction > 0.05 for each and all 95% confidence interval (CI) of RHR crossed 1.0]. Compared with VKAs, DOACs significantly reduced risk for stroke/SE (OSs, HR: 0.87, 95% CI: 0.81-0.94; RCT, RR: 0.82, 95% CI: 0.67-0.96), and ICH (OSs: 0.47 [0.37-0.57]; RCTs: 0.47 [0.31-0.63]), without increasing risk for GIB (OSs: 1.21 [0.98-1.43]; RCTs: 1.34 [0.91-1.77]), and all-cause mortality (OSs: 1.01 [0.92-1.11]; RCTs: 0.94 [0.87-1.00]). Among OSs, DOACs significantly decreased risk for major bleeding (0.87 [0.77-0.98]) and MI (0.89 [0.79-0.99]). It was found that dabigatran, but not other DOACs, significantly increased risk for GIB (1.48 [1.23-1.72]). Conclusions: DOACs were demonstrated to be more effective and safer than VKAs in elderly AF patients, whereas dabigatran users had a 48% increase in risk for GIB.

Comparison of effectiveness and safety of direct oral anticoagulants versus vitamin-k antagonists in elderly patients with atrial fibrillation: a systematic review and cost-effectiveness analysis protocol

Background

Current evidence regarding the effectiveness and safety of direct oral anticoagulants (DOACs) in the elderly with atrial fibrillation (AF) remains scarce. Based on the emerging evidence from real-world studies (RWSs) associated with DOACs, we will perform a systematic review and meta-analysis of data from RWSs and randomized controlled trials (RCTs) to compare the effectiveness, safety and cost of DOACs versus Vitamin K antagonists (VKAs) in elderly patients with AF.

Methods

The MEDLINE, EMBASE and Cochrane Library databases will be systematically searched until June 30, 2019 for eligible RWSs and RCTs that reported the clinical outcomes between DOACs and VKAs in elderly patients with AF. The effectiveness outcome is stroke or systemic embolism (SE), and the safety outcomes are major bleeding, intracranial haemorrhage (ICH), gastrointestinal bleeding (GIB), myocardial infarction (MI) and all-cause mortality. A random-effects model will be used to calculate adjusted hazard ratios (HRs) for RWSs and relative risks (RRs) for RCTs, separately. The interaction analysis and the ratio of HRs (RHRs) will be applied to compare the treatment effect difference between RWSs and RCTs. A Markov model will be constructed to evaluate the cost-effectiveness of DOACs versus VKAs in elderly AF patients in real-world setting.

Discussion

This study will summarize all available evidences from RWSs and RCTs for a comprehensive and rigorous systematic review on the effectiveness and safety associated with DOACs, as well as perform a cost-effectiveness analysis to evaluate the price performance of DOACs among elderly AF patients in real clinical setting.

Trial registration

PROSPERO register platform (CRD42019142881, www.crd.york.ac.uk/PROSPERO/display_record.php?RecordID =142881).