Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey

Empirical carbapenems or piperacillin/tazobactam for infections in intensive care: An international retrospective cohort study

BACKGROUND

Critically ill patients in intensive care units (ICU) are frequently administered broad-spectrum antibiotics (e.g., carbapenems or piperacillin/tazobactam) for suspected or confirmed infections. This retrospective cohort study aimed to describe the use of carbapenems and piperacillin/tazobactam in two international, prospectively collected datasets.

METHODS

We conducted a post hoc analysis of data from the "Adjunctive Glucocorticoid Therapy in Patients with Septic Shock" (ADRENAL) trial (n = 3713) and the "Antimicrobial de-escalation in the critically ill patient and assessment of clinical cure" (DIANA) study (n = 1488). The primary outcome was the proportion of patients receiving initial antibiotic treatment with carbapenems and piperacillin/tazobactam. Secondary outcomes included mortality, days alive and out of ICU and ICU length of stay at 28 days.

RESULTS

In the ADRENAL trial, carbapenems were used in 648 out of 3713 (17%), whereas piperacillin/tazobactam was used in 1804 out of 3713 (49%) participants. In the DIANA study, carbapenems were used in 380 out of 1480 (26%), while piperacillin/tazobactam was used in 433 out of 1488 (29%) participants. Mortality at 28 days was 23% for patients receiving carbapenems and 24% for those receiving piperacillin/tazobactam in ADRENAL and 23% and 19%, respectively, in DIANA. We noted variations in secondary outcomes; in DIANA, patients receiving carbapenems had a median of 13 days alive and out of ICU compared with 18 days among those receiving piperacillin/tazobactam. In ADRENAL, the median hospital length of stay was 27 days for patients receiving carbapenems and 21 days for those receiving piperacillin/tazobactam.

CONCLUSIONS

In this post hoc analysis of ICU patients with infections, we found widespread initial use of carbapenems and piperacillin/tazobactam in international ICUs, with the latter being more frequently used. Randomized clinical trials are needed to assess if the observed variations in outcomes may be drug-related effects or due to confounders.

Design differences and variation in results between randomised trials and non-randomised emulations: meta-analysis of RCT-DUPLICATE data

Objective

To explore how design emulation and population differences relate to variation in results between randomised controlled trials (RCT) and non-randomised real world evidence (RWE) studies, based on the RCT-DUPLICATE initiative (Randomised, Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology).

Design

Meta-analysis of RCT-DUPLICATE data.

Data sources

Trials included in RCT-DUPLICATE, a demonstration project that emulated 32 randomised controlled trials using three real world data sources: Optum Clinformatics Data Mart, 2004-19; IBM MarketScan, 2003-17; and subsets of Medicare parts A, B, and D, 2009-17.

Eligibility criteria for selecting studies

Trials where the primary analysis resulted in a hazard ratio; 29 RCT-RWE study pairs from RCT-DUPLICATE.

Results

Differences and variation in effect sizes between the results from randomised controlled trials and real world evidence studies were investigated. Most of the heterogeneity in effect estimates between the RCT-RWE study pairs in this sample could be explained by three emulation differences in the meta-regression model: treatment started in hospital (which does not appear in health insurance claims data), discontinuation of some baseline treatments at randomisation (which would have been an unusual care decision in clinical practice), and delayed onset of drug effects (which would be under-reported in real world clinical practice because of the relatively short persistence of the treatment). Adding the three emulation differences to the meta-regression reduced heterogeneity from 1.9 to almost 1 (absence of heterogeneity).

Conclusions

This analysis suggests that a substantial proportion of the observed variation between results from randomised controlled trials and real world evidence studies can be attributed to differences in design emulation.

Validation and impact of algorithms for identifying variables in observational studies of routinely collected data

BACKGROUND AND OBJECTIVES

Among observational studies of routinely collected health data (RCD) for exploring treatment effects, algorithms are used to identify study variables. However, the extent to which algorithms are reliable and impact the credibility of effect estimates is far from clear. This study aimed to investigate the validation of algorithms for identifying study variables from RCD, and examine the impact of alternative algorithms on treatment effects.

METHODS

We searched PubMed for observational studies published in 2018 that used RCD to explore drug treatment effects. Information regarding the reporting, validation, and interpretation of algorithms was extracted. We summarized the reporting and methodological characteristics of algorithms and validation. We also assessed the divergence in effect estimates given alternative algorithms by calculating the ratio of estimates of the primary vs. alternative analyses.

RESULTS

A total of 222 studies were included, of which 93 (41.9%) provided a complete list of algorithms for identifying participants, 36 (16.2%) for exposure, and 132 (59.5%) for outcomes, and 15 (6.8%) for all study variables including population, exposure, and outcomes. Fifty-nine (26.6%) studies stated that the algorithms were validated, and 54 (24.3%) studies reported methodological characteristics of 66 validations, among which 61 validations in 49 studies were from the cross-referenced validation studies. Of those 66 validations, 22 (33.3%) reported sensitivity and 16 (24.2%) reported specificity. A total of 63.6% of studies reporting sensitivity and 56.3% reporting specificity used test-result-based sampling, an approach that potentially biases effect estimates. Twenty-eight (12.6%) studies used alternative algorithms to identify study variables, and 24 reported the effects estimated by primary analyses and sensitivity analyses. Of these, 20% had differential effect estimates when using alternative algorithms for identifying population, 18.2% for identifying exposure, and 45.5% for classifying outcomes. Only 32 (14.4%) studies discussed how the algorithms may affect treatment estimates.

CONCLUSION

In observational studies of RCD, the algorithms for variable identification were not regularly validated, and-even if validated-the methodological approach and performance of the validation were often poor. More seriously, different algorithms may yield differential treatment effects, but their impact is often ignored by researchers. Strong efforts, including recommendations, are warranted to improve good practice.

Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials: a meta-epidemiological study

BACKGROUND

Researchers and decision-makers often use evidence from randomised controlled trials (RCTs) to determine the efficacy or effectiveness of a treatment or intervention. Studies with observational designs are often used to measure the effectiveness of an intervention in 'real world' scenarios. Numerous study designs and their modifications (including both randomised and observational designs) are used for comparative effectiveness research in an attempt to give an unbiased estimate of whether one treatment is more effective or safer than another for a particular population. An up-to-date systematic analysis is needed to identify differences in effect estimates from RCTs and observational studies. This updated review summarises the results of methodological reviews that compared the effect estimates of observational studies with RCTs from evidence syntheses that addressed the same health research question.

OBJECTIVES

To assess and compare synthesised effect estimates by study type, contrasting RCTs with observational studies. To explore factors that might explain differences in synthesised effect estimates from RCTs versus observational studies (e.g. heterogeneity, type of observational study design, type of intervention, and use of propensity score adjustment). To identify gaps in the existing research comparing effect estimates across different study types.

SEARCH METHODS

We searched MEDLINE, the Cochrane Database of Systematic Reviews, Web of Science databases, and Epistemonikos to May 2022. We checked references, conducted citation searches, and contacted review authors to identify additional reviews.

SELECTION CRITERIA

We included systematic methodological reviews that compared quantitative effect estimates measuring the efficacy or effectiveness of interventions tested in RCTs versus in observational studies. The included reviews compared RCTs to observational studies (including retrospective and prospective cohort, case-control and cross-sectional designs). Reviews were not eligible if they compared RCTs with studies that had used some form of concurrent allocation.

DATA COLLECTION AND ANALYSIS

Using results from observational studies as the reference group, we examined the relative summary effect estimates (risk ratios (RRs), odds ratios (ORs), hazard ratios (HRs), mean differences (MDs), and standardised mean differences (SMDs)) to evaluate whether there was a relatively larger or smaller effect in the ratio of odds ratios (ROR) or ratio of risk ratios (RRR), ratio of hazard ratios (RHR), and difference in (standardised) mean differences (D(S)MD). If an included review did not provide an estimate comparing results from RCTs with observational studies, we generated one by pooling the estimates for observational studies and RCTs, respectively. Across all reviews, we synthesised these ratios to produce a pooled ratio of ratios comparing effect estimates from RCTs with those from observational studies. In overviews of reviews, we estimated the ROR or RRR for each overview using observational studies as the reference category. We appraised the risk of bias in the included reviews (using nine criteria in total). To receive an overall low risk of bias rating, an included review needed: explicit criteria for study selection, a complete sample of studies, and to have controlled for study methodological differences and study heterogeneity. We assessed reviews/overviews not meeting these four criteria as having an overall high risk of bias. We assessed the certainty of the evidence, consisting of multiple evidence syntheses, with the GRADE approach.

MAIN RESULTS

We included 39 systematic reviews and eight overviews of reviews, for a total of 47. Thirty-four of these contributed data to our primary analysis. Based on the available data, we found that the reviews/overviews included 2869 RCTs involving 3,882,115 participants, and 3924 observational studies with 19,499,970 participants. We rated 11 reviews/overviews as having an overall low risk of bias, and 36 as having an unclear or high risk of bias. Our main concerns with the included reviews/overviews were that some did not assess the quality of their included studies, and some failed to account appropriately for differences between study designs - for example, they conducted aggregate analyses of all observational studies rather than separate analyses of cohort and case-control studies. When pooling RORs and RRRs, the ratio of ratios indicated no difference or a very small difference between the effect estimates from RCTs versus from observational studies (ratio of ratios 1.08, 95% confidence interval (CI) 1.01 to 1.15). We rated the certainty of the evidence as low. Twenty-three of 34 reviews reported effect estimates of RCTs and observational studies that were on average in agreement. In a number of subgroup analyses, small differences in the effect estimates were detected: - pharmaceutical interventions only (ratio of ratios 1.12, 95% CI 1.04 to 1.21); - RCTs and observational studies with substantial or high heterogeneity; that is, I2 ≥ 50% (ratio of ratios 1.11, 95% CI 1.04 to 1.18); - no use (ratio of ratios 1.07, 95% CI 1.03 to 1.11) or unclear use (ratio of ratios 1.13, 95% CI 1.03 to 1.25) of propensity score adjustment in observational studies; and - observational studies without further specification of the study design (ratio of ratios 1.06, 95% CI 0.96 to 1.18). We detected no clear difference in other subgroup analyses.

AUTHORS' CONCLUSIONS

We found no difference or a very small difference between effect estimates from RCTs and observational studies. These findings are largely consistent with findings from recently published research. Factors other than study design need to be considered when exploring reasons for a lack of agreement between results of RCTs and observational studies, such as differences in the population, intervention, comparator, and outcomes investigated in the respective studies. Our results underscore that it is important for review authors to consider not only study design, but the level of heterogeneity in meta-analyses of RCTs or observational studies. A better understanding is needed of how these factors might yield estimates reflective of true effectiveness.

Real-world data analysis on effectiveness of integrative therapies: A practical guide to study design and data analysis using healthcare databases

Real world data (RWD) is increasingly used to investigate health outcomes and treatment efficacy in the field of integrative medicine. Due to the fact that the majority of RWDs are not intended for research, their secondary use in research necessitates complex study designs to account for bias and confounding. To conduct a robust analysis of RWD in integrative medicine, a comprehensive study design process that reflects the characteristics of integrative therapies is necessary. In this paper, we present a guide for designing comparative effectiveness RWE research in integrative medicine. We discuss key factors to consider when selecting RWDs for research on integrative medicine. We provide practical steps for developing a research question, formulating the PICOT objectives (population, intervention, comparator, outcome, and time horizon), and selecting and defining covariates with a summary table. Specific study designs are depicted with corresponding diagrams. Finally, data analysis procedures are introduced. We hope this article clarifies the importance of RWE research design and related processes in order to improve the rigor of RWD studies in the field of integrative medicine research.

Reporting, handling, and interpretation of time-varying drug treatments in observational studies using routinely collected healthcare data

BACKGROUND

Time-varying drug treatments are common in studies using routinely collected health data (RCD) for assessing treatment effects. This study aimed to examine how these studies reported, handled, and interpreted time-varying drug treatments.

METHODS

A systematic search was conducted on PubMed from 2018 to 2020. Eligible studies were those used RCD to explore drug treatment effects. We summarized the reporting characteristics and methods employed for handling time-varying treatments. Logistic regressions were performed to investigate the association between study characteristics and the reporting of time-varying treatments.

RESULTS

Two hundred and fifty-six studies were included, and 225 (87.9%) studies involved time-varying treatments. Of these, 24 (10.7%) reported the proportion of time-varying treatments and 105 (46.7%) reported methods used to handle time-varying treatments. Multivariable logistic regression showed that medical studies, prespecified protocol, and involvement of methodologists were associated with a higher likelihood of reporting the methods applied to handle time-varying treatments. Among the 105 studies that reported methods, as-treated analyses were the most commonly used analysis sets, which were employed in 73.9%, 75.3% and 88.2% of studies that reported approaches for treatment discontinuation, treatment switching and treatment add-on. Among the 225 studies involved time-varying treatments, 27 (12.0%) acknowledged the potential bias introduced by treatment change, of which 14 (51.9%) suggested that potential biases may impact acceptance or rejection of the null hypothesis.

CONCLUSIONS

Among observational studies using RCD, the underreporting about the presence and methods for handling time-varying treatments was largely common. The potential biases due to time-varying treatments have frequently been disregarded. Collaborative endeavors are strongly needed to enhance the prevailing practices.

Observational studies of antidepressant use and suicide risk are selectively published in psychiatric journals

OBJECTIVES

To investigate if observational studies showing favorable results for antidepressants on suicidal behavior (reduced risk) are preferably and more easily published in psychiatric journals and cited more often compared to studies with unfavorable results (increased risk).

STUDY DESIGN AND SETTING

Prespecified secondary analysis, including 27 original studies selected through a systematic review of observational studies reporting associations between the use of newer antidepressant drugs and suicide risk.

RESULTS

Independent of study quality, studies reporting favorable results were more frequently published in psychiatric than nonpsychiatric journals and were more often conducted by lead authors with financial conflicts of interest (fCOI). Within psychiatric journals, lead authors with fCOI published in journals with a higher impact factor (IF) and ranking. Within psychiatric journals, favorability of results also correlated with citation frequency, IF, and journal ranking, but these associations became weaker and inconclusive after adjusting for study quality. Results for ease of publishing were inconclusive.

CONCLUSION

Studies reporting unfavorable results (increased suicide risk with antidepressant exposure) are less likely to be published in psychiatric journals. Lead authors with fCOI report more favorable results, and their studies are published in the most prestigious psychiatric journals. This may create a biased evidence base and an unbalanced dissemination and appraisal of findings within psychiatry.

New-onset atrial fibrillation in critically ill adult patients-an SSAI clinical practice guideline

BACKGROUND

Acute or new-onset atrial fibrillation (NOAF) is the most common cardiac arrhythmia in critically ill adult patients, and observational data suggests that NOAF is associated to adverse outcomes.

METHODS

We prepared this guideline according to the Grading of Recommendations Assessment, Development and Evaluation methodology. We posed the following clinical questions: (1) what is the better first-line pharmacological agent for the treatment of NOAF in critically ill adult patients?, (2) should we use direct current (DC) cardioversion in critically ill adult patients with NOAF and hemodynamic instability caused by atrial fibrillation?, (3) should we use anticoagulant therapy in critically ill adult patients with NOAF?, and (4) should critically ill adult patients with NOAF receive follow-up after discharge from hospital? We assessed patient-important outcomes, including mortality, thromboembolic events, and adverse events. Patients and relatives were part of the guideline panel.

RESULTS

The quantity and quality of evidence on the management of NOAF in critically ill adults was very limited, and we did not identify any relevant direct or indirect evidence from randomized clinical trials for the prespecified PICO questions. We were able to propose one weak recommendation against routine use of therapeutic dose anticoagulant therapy, and one best practice statement for routine follow-up by a cardiologist after hospital discharge. We were not able to propose any recommendations on the better first-line pharmacological agent or whether to use DC cardioversion in critically ill patients with hemodynamic instability induced by NOAF. An electronic version of this guideline in layered and interactive format is available in MAGIC: https://app.magicapp.org/#/guideline/7197.

CONCLUSIONS

The body of evidence on the management of NOAF in critically ill adults is very limited and not informed by direct evidence from randomized clinical trials. Practice variation appears considerable.

DESIGN DIFFERENCES EXPLAIN VARIATION IN RESULTS BETWEEN RANDOMIZED TRIALS AND THEIR NON-RANDOMIZED EMULATIONS

Objectives

While randomized controlled trials (RCTs) are considered a standard for evidence on the efficacy of medical treatments, non-randomized real-world evidence (RWE) studies using data from health insurance claims or electronic health records can provide important complementary evidence. The use of RWE to inform decision-making has been questioned because of concerns regarding confounding in non-randomized studies and the use of secondary data. RCT-DUPLICATE was a demonstration project that emulated the design of 32 RCTs with non-randomized RWE studies. We sought to explore how emulation differences relate to variation in results between the RCT-RWE study pairs.

Methods

We include all RCT-RWE study pairs from RCT-DUPLICATE where the measure of effect was a hazard ratio and use exploratory meta-regression methods to explain differences and variation in the effect sizes between the results from the RCT and the RWE study. The considered explanatory variables are related to design and population differences.

Results

Most of the observed variation in effect estimates between RCT-RWE study pairs in this sample could be explained by three emulation differences in the meta-regression model: (i) in-hospital start of treatment (not observed in claims data), (ii) discontinuation of certain baseline therapies at randomization (not part of clinical practice), (iii) delayed onset of drug effects (missed by short medication persistence in clinical practice).

Conclusions

This analysis suggests that a substantial proportion of the observed variation between results from RCTs and RWE studies can be attributed to design emulation differences. (238 words).

Emulation of Randomized Clinical Trials With Nonrandomized Database Analyses: Results of 32 Clinical Trials

Importance

Nonrandomized studies using insurance claims databases can be analyzed to produce real-world evidence on the effectiveness of medical products. Given the lack of baseline randomization and measurement issues, concerns exist about whether such studies produce unbiased treatment effect estimates.

Objective

To emulate the design of 30 completed and 2 ongoing randomized clinical trials (RCTs) of medications with database studies using observational analogues of the RCT design parameters (population, intervention, comparator, outcome, time [PICOT]) and to quantify agreement in RCT-database study pairs.

Design, Setting, and Participants

New-user cohort studies with propensity score matching using 3 US claims databases (Optum Clinformatics, MarketScan, and Medicare). Inclusion-exclusion criteria for each database study were prespecified to emulate the corresponding RCT. RCTs were explicitly selected based on feasibility, including power, key confounders, and end points more likely to be emulated with real-world data. All 32 protocols were registered on ClinicalTrials.gov before conducting analyses. Emulations were conducted from 2017 through 2022.

Exposures

Therapies for multiple clinical conditions were included.

Main Outcomes and Measures

Database study emulations focused on the primary outcome of the corresponding RCT. Findings of database studies were compared with RCTs using predefined metrics, including Pearson correlation coefficients and binary metrics based on statistical significance agreement, estimate agreement, and standardized difference.

Results

In these highly selected RCTs, the overall observed agreement between the RCT and the database emulation results was a Pearson correlation of 0.82 (95% CI, 0.64-0.91), with 75% meeting statistical significance, 66% estimate agreement, and 75% standardized difference agreement. In a post hoc analysis limited to 16 RCTs with closer emulation of trial design and measurements, concordance was higher (Pearson r, 0.93; 95% CI, 0.79-0.97; 94% meeting statistical significance, 88% estimate agreement, 88% standardized difference agreement). Weaker concordance occurred among 16 RCTs for which close emulation of certain design elements that define the research question (PICOT) with data from insurance claims was not possible (Pearson r, 0.53; 95% CI, 0.00-0.83; 56% meeting statistical significance, 50% estimate agreement, 69% standardized difference agreement).

Conclusions and Relevance

Real-world evidence studies can reach similar conclusions as RCTs when design and measurements can be closely emulated, but this may be difficult to achieve. Concordance in results varied depending on the agreement metric. Emulation differences, chance, and residual confounding can contribute to divergence in results and are difficult to disentangle.

Implementation of Ticagrelor Reduced Mortality in Routine Clinical Care: Evidence From a Natural Experiment Including 109 995 Patients With Myocardial Infarction in Sweden

Background Effectiveness estimates from observational studies on ticagrelor use in routine clinical care are conflicting, with some contrary to the results of the pivotal randomized controlled trial of ticagrelor in acute coronary syndrome. The aim of this study was to estimate the effect of implementing and using ticagrelor in routine clinical care in patients with myocardial infarction using a natural experimental approach. Methods and Results This is a retrospective cohort study including patients hospitalized for myocardial infarction in Sweden between 2009 and 2015. The study exploited differences in the timing and speed of ticagrelor implementation between treatment centers as a source of random treatment assignment. The effect of implementing and using ticagrelor was estimated based on the admitting center's likelihood of treating patients with ticagrelor, measured as the proportion of patients treated in the 90 days before patient admission. The main outcome was 12-month mortality. The study included 109 955 patients, of whom 30 773 were treated with ticagrelor. Being admitted to a treatment center with higher past ticagrelor use was associated with a reduction in 12-month mortality (2.5 percentage points for 100% versus 0% past use [95% CI, 0.2-4.8]). The results are in line with the findings from the ticagrelor pivotal trial. Conclusions Using a natural experiment, this study finds that the implementation and use of ticagrelor in routine clinical care has reduced 12-month mortality in patients admitted to the hospital with myocardial infarction in Sweden and supports the external validity of randomized evidence on ticagrelor effectiveness.

An empirical comparison of the harmful effects for randomized controlled trials and non-randomized studies of interventions

Introduction: Randomized controlled trials (RCTs) are the gold standard to evaluate the efficacy of interventions (e.g., drugs and vaccines), yet the sample size of RCTs is often limited for safety assessment. Non-randomized studies of interventions (NRSIs) had been proposed as an important alternative source for safety assessment. In this study, we aimed to investigate whether there is any difference between RCTs and NRSIs in the evaluation of adverse events. Methods: We used the dataset of systematic reviews with at least one meta-analysis including both RCTs and NRSIs and collected the 2 × 2 table information (i.e., numbers of cases and sample sizes in intervention and control groups) of each study in the meta-analysis. We matched RCTs and NRSIs by their sample sizes (ratio: 0.85/1 to 1/0.85) within a meta-analysis. We estimated the ratio of the odds ratios (RORs) of an NRSI against an RCT in each pair and used the inverse variance as the weight to combine the natural logarithm of ROR (lnROR). Results: We included systematic reviews with 178 meta analyses, from which we confirmed 119 pairs of RCTs and NRSIs. The pooled ROR of NRSIs compared to that of RCTs was estimated to be 0.96 (95% confidence interval: 0.87 and 1.07). Similar results were obtained with different sample size subgroups and treatment subgroups. With the increase in sample size, the difference in ROR between RCTs and NRSIs decreased, although not significantly. Discussion: There was no substantial difference in the effects between RCTs and NRSIs in safety assessment when they have similar sample sizes. Evidence from NRSIs might be considered a supplement to RCTs for safety assessment.

Quality of observational studies of clinical interventions: a meta-epidemiological review

BACKGROUND

This meta-epidemiological study aimed to assess methodological quality of a sample of contemporary non-randomised clinical studies of clinical interventions.

METHODS

This was a cross-sectional study of observational studies published between January 1, 2012 and December 31, 2018. Studies were identified in PubMed using search terms 'association', 'observational,' 'non-randomised' 'comparative effectiveness' within titles or abstracts. Each study was appraised against 35 quality criteria by two authors independently, with each criterion rated fully, partially or not satisfied. These quality criteria were grouped into 6 categories: justification for observational design (n = 2); minimisation of bias in study design and data collection (n = 11); use of appropriate methods to create comparable groups (n = 6); appropriate adjustment of observed effects (n = 5); validation of observed effects (n = 9); and authors interpretations (n = 2).

RESULTS

Of 50 unique studies, 49 (98%) were published in two US general medical journals. No study fully satisfied all applicable criteria; the mean (+/-SD) proportion of applicable criteria fully satisfied across all studies was 72% (+/- 10%). The categories of quality criteria demonstrating the lowest proportions of fully satisfied criteria were measures used to adjust observed effects (criteria 20, 23, 24) and validate observed effects (criteria 25, 27, 33). Criteria associated with ≤50% of full satisfaction across studies, where applicable, comprised: imputation methods to account for missing data (50%); justification for not performing an RCT (42%); interaction analyses in identifying independent prognostic factors potentially influencing intervention effects (42%); use of statistical correction to minimise type 1 error in multiple outcome analyses (33%); clinically significant effect sizes (30%); residual bias analyses for unmeasured or unknown confounders (14%); and falsification tests for residual confounding (8%). The proportions of fully satisfied criteria did not change over time.

CONCLUSIONS

Recently published observational studies fail to fully satisfy more than one in four quality criteria. Criteria that were not or only partially satisfied were identified which serve as remediable targets for researchers and journal editors.

Ferroptosis: a double-edged sword mediating immune tolerance of cancer

The term ferroptosis was put forward in 2012 and has been researched exponentially over the past few years. Ferroptosis is an unconventional pattern of iron-dependent programmed cell death, which belongs to a type of necrosis and is distinguished from apoptosis and autophagy. Actuated by iron-dependent phospholipid peroxidation, ferroptosis is modulated by various cellular metabolic and signaling pathways, including amino acid, lipid, iron, and mitochondrial metabolism. Notably, ferroptosis is associated with numerous diseases and plays a double-edged sword role. Particularly, metastasis-prone or highly-mutated tumor cells are sensitive to ferroptosis. Hence, inducing or prohibiting ferroptosis in tumor cells has vastly promising potential in treating drug-resistant cancers. Immunotolerant cancer cells are not sensitive to the traditional cell death pathway such as apoptosis and necroptosis, while ferroptosis plays a crucial role in mediating tumor and immune cells to antagonize immune tolerance, which has broad prospects in the clinical setting. Herein, we summarized the mechanisms and delineated the regulatory network of ferroptosis, emphasized its dual role in mediating immune tolerance, proposed its significant clinical benefits in the tumor immune microenvironment, and ultimately presented some provocative doubts. This review aims to provide practical guidelines and research directions for the clinical practice of ferroptosis in treating immune-resistant tumors.

Long-term Effect of Lifestyle Interventions on the Cardiovascular and All-Cause Mortality of Subjects With Prediabetes and Type 2 Diabetes: A Systematic Review and Meta-analysis

BACKGROUND

Lifestyle interventions improve the metabolic control of individuals with hyperglycemia.

PURPOSE

We aimed to determine the effect of lifestyle interventions on cardiovascular and all-cause mortality in this population.

DATA SOURCES

Searches were made through MEDLINE, Cochrane CENTRAL, Embase, and Web of Science (no date/language restriction, until 15 May 2022).

STUDY SELECTION

We included randomized clinical trials (RCTs) of subjects with prediabetes and type 2 diabetes, comparing intensive lifestyle interventions with usual care, with a minimum of 2 years of active intervention.

DATA EXTRACTION

Data from the 11 RCTs selected were extracted in duplicate. A frequentist and arm-based meta-analysis was performed with random-effects models to estimate relative risk (RR) for mortality, and heterogeneity was assessed through I2 metrics. A generalized linear mixed model (GLMM) was used to confirm the findings.

DATA SYNTHESIS

Lifestyle interventions were not superior to usual care in reducing cardiovascular (RR 0.99; 95% CI 0.79-1.23) or all-cause (RR 0.93; 95% CI 0.85-1.03) mortality. Subgroup, sensitivity, and meta-regression analyses showed no influence of type of intervention, mean follow-up, age, glycemic status, geographical location, risk of bias, or weight change. All of these results were confirmed with the GLMM. Most studies had a low risk of bias according to the RoB 2.0 tool and the certainty of evidence was moderate for both outcomes.

LIMITATIONS

Most studies had a low risk of bias according to the RoB 2.0 tool, and the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach resulted in moderate certainty of evidence for both outcomes. Differences in lifestyle programs and in usual care between the studies should be considered in the interpretation of our results.

CONCLUSIONS

Intensive lifestyle interventions implemented so far did not show superiority to usual care in reducing cardiovascular or all-cause mortality for subjects with prediabetes and type 2 diabetes.

Quadriceps Tendon Autograft Versus Bone-Patellar Tendon-Bone and Hamstring Tendon Autografts for Anterior Cruciate Ligament Reconstruction: A Systematic Review and Meta-analysis

BACKGROUND

The best type of autograft for anterior cruciate ligament (ACL) reconstruction remains debatable.

HYPOTHESIS

Compared with bone-patellar tendon-bone (BPTB) and hamstring tendon (HT) autografts, the quadriceps tendon (QT) autograft has comparable graft survival as well as clinical function and pain outcomes.

STUDY DESIGN

Meta-analysis; Level of evidence, 4.

METHODS

A systematic literature search was conducted in PubMed, Embase, Scopus, and the Cochrane Library to July 2020. Randomized controlled trials (RCTs) and observational studies reporting comparisons of QT versus BPTB or HT autografts for ACL reconstruction were included. All analyses were stratified according to study design: RCTs or observational studies.

RESULTS

A total of 24 studies were included: 7 RCTs and 17 observational studies. The 7 RCTs included 388 patients, and the 17 observational studies included 19,196 patients. No significant differences in graft failure (P = .36), the International Knee Documentation Committee (IKDC) subjective score (P = .39), or the side-to-side difference in stability (P = .60) were noted between QT and BPTB autografts. However, a significant reduction in donor site morbidity was noted in the QT group compared with the BPTB group (risk ratio [RR], 0.17 [95% CI, 0.09-0.33]; P < .001). No significant differences in graft failure (P = .57), the IKDC subjective score (P = .25), or the side-to-side stability difference (P = .98) were noted between QT and HT autografts. However, the QT autograft was associated with a significantly lower rate of donor site morbidity than the HT autograft (RR, 0.60 [95% CI, 0.39-0.93]; P = .02). A similar graft failure rate between the QT and control groups was observed after both early and late full weightbearing, after early and late full range of motion, and after using the QT autograft with a bone plug and all soft tissue QT grafts. However, a significantly lower rate of donor site morbidity was observed in the QT group compared with the control group after both early and late full weightbearing, after early and late full range of motion, and after using the QT autograft with a bone plug and all soft tissue QT grafts. No difference in effect estimates was seen between RCTs and observational studies.

CONCLUSION

The QT autograft had comparable graft survival, functional outcomes, and stability outcomes compared with BPTB and HT autografts. However, donor site morbidity was significantly lower with the QT autograft than with BPTB and HT autografts.

Reproducibility of real-world evidence studies using clinical practice data to inform regulatory and coverage decisions

Studies that generate real-world evidence on the effects of medical products through analysis of digital data collected in clinical practice provide key insights for regulators, payers, and other healthcare decision-makers. Ensuring reproducibility of such findings is fundamental to effective evidence-based decision-making. We reproduce results for 150 studies published in peer-reviewed journals using the same healthcare databases as original investigators and evaluate the completeness of reporting for 250. Original and reproduction effect sizes were positively correlated (Pearson’s correlation = 0.85), a strong relationship with some room for improvement. The median and interquartile range for the relative magnitude of effect (e.g., hazard ratio_original/hazard ratio_reproduction) is 1.0 [0.9, 1.1], range [0.3, 2.1]. While the majority of results are closely reproduced, a subset are not. The latter can be explained by incomplete reporting and updated data. Greater methodological transparency aligned with new guidance may further improve reproducibility and validity assessment, thus facilitating evidence-based decision-making. Study registration number: EUPAS19636.

Analyses of real-world evidence from digital clinical practice data provide important insights for healthcare decision makers. Here, authors test reproducibility of 150 peer-reviewed studies, reporting strong reproducibility, which could be further improved through more complete reporting in future original studies

How do cancer clinicians perceive real-world data and the evidence derived therefrom? Findings from an international survey of the European Organisation for Research and Treatment of Cancer

Background: The role of real-world evidence (RWE) in the development of anticancer therapies has been gradually growing over time. Regulators, payers and health technology assessment agencies, spurred by the rise of the precision medicine model, are increasingly incorporating RWE into their decision-making regarding the authorization and reimbursement of novel antineoplastic treatments. However, it remains unclear how this trend is viewed by clinicians in the field. This study aimed to investigate the opinions of these stakeholders with respect to RWE and its suitability for informing regulatory, reimbursement-related and clinical decisions in oncology.

Methods: An online survey was disseminated to clinicians belonging to the network of the European Organisation for Research and Treatment of Cancer between May and July 2021.

Results: In total, 557 clinicians across 30 different countries participated in the survey, representing 13 distinct cancer domains. Despite seeing the methodological challenges associated with its interpretation as difficult to overcome, the respondents mostly (75.0%) perceived RWE positively, and believed such evidence could be relatively strong, depending on the designs and data sources of the studies from which it is produced. Few (4.6%) saw a future expansion of its influence on decision-makers as a negative evolution. Furthermore, nearly all (94.0%) participants were open to the idea of sharing anonymized or pseudonymized electronic health data of their patients with external parties for research purposes. Nevertheless, most clinicians (77.0%) still considered randomized controlled trials (RCTs) to be the gold standard for generating clinical evidence in oncology, and a plurality (49.2%) thought that RWE cannot fully address the knowledge gaps that remain after a new antitumor intervention has entered the market. Moreover, a majority of respondents (50.7%) expressed that they relied more heavily on RCT-derived evidence than on RWE for their own decision-making.

Conclusion: While cancer clinicians have positive opinions about RWE and want to contribute to its generation, they also continue to hold RCTs in high regard as sources of actionable evidence.

Cancer-related fatigue in patients treated with mistletoe extracts: a systematic review and meta-analysis

PURPOSE

Cancer-related fatigue remains one of the most prevalent and distressing symptoms experienced by cancer patients. Effective treatments for cancer-related fatigue are needed. The objective of this meta-analysis is to determine the impact of mistletoe extracts as a pharmacological treatment for the management of cancer-related fatigue.

METHODS

We included randomized clinical trials (RCTs) and non-randomized studies of interventions (NRSIs) in cancer patients. Inclusion criteria were cancer-related fatigue severity or prevalence as an outcome and testing of mistletoe extracts compared to control groups. We searched Medline (EuropePMC), Embase, the Cochrane Central Register of Controlled Trials, Clinicaltrials.gov, and opengrey.org through October 2020. We assessed the risk of bias using the Cochrane risk of bias tools for RCTs and NRSIs and conducted a meta-analysis.

RESULTS

We performed one meta-analysis with 12 RCTs, including 1494 participants, and one meta-analysis with seven retrospective NRSIs, including 2668 participants. Heterogeneity between the studies was high in both meta-analyses. Most studies had a high risk of bias. A random-effects model showed for RCTs a standardized mean difference of -0.48 (95% confidence interval -0.82 to -0.14; p = 0.006) and for NRSIs an odds ratio of 0.36 (95% confidence interval 0.20 to 0.66; p = 0.0008).

CONCLUSION

Treatment with mistletoe extracts shows a moderate effect on cancer-related fatigue of similar size to physical activity. These results need to be confirmed by more placebo-controlled trials. Future trials should investigate different treatment durations and their effect on cancer-related fatigue in post-treatment cancer survivors.

TRIAL REGISTRATION

This meta-analysis has been registered under the PROSPERO registration number CRD42020191967 on October 7, 2020.

Agreement of treatment effects from observational studies and randomized controlled trials evaluating hydroxychloroquine, lopinavir-ritonavir, or dexamethasone for covid-19: meta-epidemiological study

OBJECTIVE

To systematically identify, match, and compare treatment effects and study demographics from individual or meta-analysed observational studies and randomized controlled trials (RCTs) evaluating the same covid-19 treatments, comparators, and outcomes.

DESIGN

Meta-epidemiological study.

DATA SOURCES

National Institutes of Health Covid-19 Treatment Guidelines, a living review and network meta-analysis published in The BMJ, a living systematic review with meta-analysis and trial sequential analysis in PLOS Medicine (The LIVING Project), and the Epistemonikos "Living OVerview of Evidence" (L·OVE) evidence database.

ELIGIBILITY CRITERIA FOR SELECTION OF STUDIES

RCTs in The BMJ's living review that directly compared any of the three most frequently studied therapeutic interventions for covid-19 across all data sources (that is, hydroxychloroquine, lopinavir-ritonavir, or dexamethasone) for any safety and efficacy outcomes. Observational studies that evaluated the same interventions, comparisons, and outcomes that were reported in The BMJ's living review.

DATA EXTRACTION AND SYNTHESIS

Safety and efficacy outcomes from observational studies were identified and treatment effects for dichotomous (odds ratios) or continuous (mean differences or ratios of means) outcomes were calculated and, when possible, meta-analyzed to match the treatment effects from individual RCTs or meta-analyses of RCTs reported in The BMJ's living review with the same interventions, comparisons, and outcomes (that is, matched pairs). The analysis compared the distribution of study demographics and the agreement between treatment effects from matched pairs. Matched pairs were in agreement if both observational and RCT treatment effects were significantly increasing or decreasing (P<0.05) or if both treatment effects were not significant (P≥0.05).

RESULTS

17 new, independent meta-analyses of observational studies were conducted that compared hydroxychloroquine, lopinavir-ritonavir, or dexamethasone with an active or placebo comparator for any safety or efficacy outcomes in covid-19 treatment. These studies were matched and compared with 17 meta-analyses of RCTs reported in The BMJ's living review. 10 additional matched pairs with only one observational study and/or one RCT were identified. Across all 27 matched pairs, 22 had adequate reporting of demographical and clinical data for all individual studies. All 22 matched pairs had studies with overlapping distributions of sex, age, and disease severity. Overall, 21 (78%) of the 27 matched pairs had treatment effects that were in agreement. Among the 17 matched pairs consisting of meta-analyses of observational studies and meta-analyses of RCTs, 14 (82%) were in agreement; seven (70%) of the 10 matched pairs consisting of at least one observational study or one RCT were in agreement. The 18 matched pairs with treatment effects for dichotomous outcomes had a higher proportion of agreement (n=16, 89%) than did the nine matched pairs with treatment effects for continuous outcomes (n=5, 56%).

CONCLUSIONS

Meta-analyses of observational studies and RCTs evaluating treatments for covid-19 have summary treatment effects that are generally in agreement. Although our evaluation is limited to three covid-19 treatments, these findings suggest that meta-analyzed evidence from observational studies might complement, but should not replace, evidence collected from RCTs.

Favorable safety profile of moderate hypofractionated over normofractionated radiotherapy in breast cancer patients: a multicentric prospective real-life data farming analysis

Background

Moderately hypofractionated whole-breast radiotherapy (HFRT) has proven to be as safe and efficient as normofractionated radiotherapy (NFRT) in randomized trials resulting in major changes in clinical practice. Toxicity rates observed in selected clinical trial patients may differ from those observed in unselected patients with possible comorbidities and frailty in real-life. This study aimed to examine the influence of HFRT versus NFRT on acute toxicity and identify risks factors of dermatitis in real-life patients.

Materials and methods

Prospective data from breast cancer patients, treated with locoregional radiotherapy were collected between November 2015 and February 2020 in 3 comprehensive cancer centers. Through a systematic data-farming strategy, acute toxicity evaluation forms (CTCAEv4.0) were prospectively completed and extracted electronically. The results from each center were then anonymously merged into a single database for analysis. A Chi-2 test was used to compare HFRT and NFRT. Furthermore, risk factors of dermatitis were identified in a sub-study (622 patients) by multivariate logistic regression analysis.

Results

In total, 3518 T0-4 N0-3 mostly M0 (85.8%) breast cancer patients with a median age of 60.7 (24–96 years old) were analyzed. Acute grade 2–3 dermatitis, grade 1–3 breast oedema, and grade 1–2 hyperpigmentation were less frequent with HFRT versus NFRT: respectively 8.9% versus 35.1% (Chi-2 = 373.7; p < 0.001), 29.0% versus 37.0% (Chi-2 = 23.1; p < 0.001) and 27.0% versus 55.8% (Chi-2 = 279.2; p < 0.001). Fewer patients experienced pain with HFRT versus NFRT: 33.4% versus 53.7% respectively (Chi-2 = 137.1; p < 0.001). Factors such as high BMI (OR = 2.30 [95% CI, 1.28–4.26], p < 0.01), large breast size (OR = 1.88 [95% CI, 1.07–3.28], p < 0.01) and lumpectomy over mastectomy (OR = 0.52 [95% CI, 0.27–0.97], p < 0.05) were associated with greater risk factors of grade 2–3 dermatitis in multivariate analysis regardless of NFRT or HFRT.

Conclusion

The results of this study suggests that breast HFRT may be a better option even for patients with a high BMI or large breast size. Acute toxicity was low to mild, and lower with HFRT compared to NFRT. Results from real-life data were robust, and support the use of HFRT beyond randomized study populations. Long-term real-life data awaits further investigation.

COVID-19: A Catalyst for Transforming Randomized Trials

The coronavirus disease 2019 (COVID-19) pandemic incited a global clinical trial research agenda of unprecedented speed and high volume. This expedited research activity in a time of crisis produced both successes and failures that offer valuable learning opportunities for the scientific community to consider. Successes include the implementation of large adaptive and pragmatic trials as well as burgeoning efforts toward rapid data synthesis and open science principles. Conversely, notable failures include: (1) inadequate study design and execution; (2) data reversal, fraud, and retraction; and (3) research duplication and waste. Other challenges that became highlighted were the need to find unbiased designs for investigating complex, nonpharmaceutical interventions and the use of routinely collected data for outcomes assessment. This article discusses these issues juxtaposing the COVID-19 trials experience against trials in anesthesiology and other fields. These lessons may serve as a positive catalyst for transforming future clinical trial research.

High variability in results and methodological quality among overlapping systematic reviews on the same topics in surgery: a meta-epidemiological study

BACKGROUND

Redundant publication of systematic reviews and meta-analyses (SRs/MAs) on the same topic presents an increasing burden for clinicians. The aim of this study was to describe variabilities in effect size and methodological quality of overlapping surgery-related SRs/MAs and to investigate factors associated with their postpublication citations.

METHODS

PubMed/MEDLINE was searched to identify SRs/MAs of RCTs on thoracoabdominal surgeries published in 2015. Previous SRs/MAs on the same topics published within the preceding 5 years (2011-2015) were identified and 5-year citation counts (through to 2020) were evaluated. Discrepancies in pooled effect sizes and their methodological quality using A Measurement Tool to Assess Systematic Reviews (AMSTAR) among overlapping SRs/MAs were assessed. The SR/MA-level factors associated with 5-year citation counts were explored, using a mixed-effects regression model with a random intercept for surgical topics.

RESULTS

A total of 57 surgery-related SRs/MAs (48 topics) published in 2015 were identified, and 146 SRs/MAs had overlapping publications on 29 topics (60.4 per cent of all topics) in the preceding 5 years. There was considerable variability in methodological quality of SRs/MAs and coverage probability for relevant RCTs, resulting in discrepant effect size estimates for the same topic. High quality (AMSTAR score 8-11) was independently associated with higher 5-year citation counts (coefficient = 32.82; 95 per cent c.i. 15.63 to 50.02; P < 0.001).

CONCLUSION

Overlapping SRs/MAs with high variability in results and methodological quality were common in surgery. A high-quality SR/MA score was an independent predictor of more frequent citations. Researchers and journal editors should concentrate their efforts on limiting publications to higher-quality reviews.

Development and validation of the PORTRET tool to predict recurrence, overall survival, and other-cause mortality in older patients with breast cancer in the Netherlands: a population-based study

BACKGROUND

Current prediction tools for breast cancer outcomes are not tailored to the older patient, in whom competing risk strongly influences treatment effects. We aimed to develop and validate a prediction tool for 5-year recurrence, overall mortality, and other-cause mortality for older patients (aged ≥65 years) with early invasive breast cancer and to estimate individualised expected benefits of adjuvant systemic treatment.

METHODS

We selected surgically treated patients with early invasive breast cancer (stage I-III) aged 65 years or older from the population-based FOCUS cohort in the Netherlands. We developed prediction models for 5-year recurrence, overall mortality, and other-cause mortality using cause-specific Cox proportional hazard models. External validation was performed in a Dutch Cancer registry cohort. Performance was evaluated with discrimination accuracy and calibration plots.

FINDINGS

We included 2744 female patients in the development cohort and 13631 female patients in the validation cohort. Median age was 74·8 years (range 65-98) in the development cohort and 76·0 years (70-101) in the validation cohort. 5-year follow-up was complete for more than 99% of all patients. We observed 343 and 1462 recurrences, and 831 and 3594 deaths, of which 586 and 2565 were without recurrence, in the development and validation cohort, respectively. The area under the receiver-operating-characteristic curve at 5 years in the external dataset was 0·76 (95% CI 0·75-0·76) for overall mortality, 0·76 (0·76-0·77) for recurrence, and 0·75 (0·74-0·75) for other-cause mortality.

INTERPRETATION

The PORTRET tool can accurately predict 5-year recurrence, overall mortality, and other-cause mortality in older patients with breast cancer. The tool can support shared decision making, especially since it provides individualised estimated benefits of adjuvant treatment.

FUNDING

Dutch Cancer Foundation and ZonMw.

Prevalence of periodontitis in people clinically diagnosed with diabetes mellitus: a meta-analysis of epidemiologic studies

AIMS

The relationship between diabetes mellitus and periodontitis has long been discussed, but the conclusion of this relationship was controversial. We conducted this meta-analysis to explore the association between diabetes mellitus and the prevalence of periodontitis.

METHODS

PubMed, Embase, Web of Science, and the Cochrane Library were searched for studies reporting associations of periodontitis with diabetes. Gray literature was integrated from the Open-Gray database. Prevalence estimates and odds ratios were extracted by two independent reviewers and synthesized through meta-analyses.

RESULTS

Twenty-seven papers met the inclusion criteria involving 3092 diabetic patients and 23,494 controls. Overall prevalence of periodontitis was 67.8% in diabetics (combined types) and 35.5% in patients without diabetes (OR, 1.85; 95%CI, 1.61-2.11). Probing depth (MD 0.23; 95% CI, 0.17-0.29; I² = 25%) and plaque index (MD, 0.20; 95% CI, 0.18-0.23; P = 0.59; I² = 0%) severity scores were significantly greater among the diabetic group.

CONCLUSIONS

The prevalence and severity of periodontitis are greater in patients with diabetes than in non-diabetic populations.

The PRolaCT studies - a study protocol for a combined randomised clinical trial and observational cohort study design in prolactinoma

Background

First-line treatment for prolactinomas is a medical treatment with dopamine agonists (DAs), which effectively control hyperprolactinaemia in most patients, although post-withdrawal remission rates are approximately 34%. Therefore, many patients require prolonged DA treatment, while side effects negatively impact health-related quality of life (HRQoL). Endoscopic transsphenoidal resection is reserved for patients with severe side effects, or with DA-resistant prolactinoma. Surgery has a good safety profile and high probability of remission and may thus deserve a more prominent place in prolactinoma treatment. The hypothesis for this study is that early or upfront surgical resection is superior to DA treatment both in terms of HRQoL and remission rate in patients with a non-invasive prolactinoma of limited size.

Methods

We present a combined randomised clinical trial and observational cohort study design, which comprises three unblinded randomised controlled trials (RCTs; PRolaCT-1, PRolaCT-2, PRolaCT-3), and an observational study arm (PRolaCT-O) that compare neurosurgical counselling, and potential subsequent endoscopic transsphenoidal adenoma resection, with current standard care. Patients with a non-invasive prolactinoma (< 25 mm) will be eligible for one of three RCTs based on the duration of pre-treatment with DAs: PRolaCT-1: newly diagnosed, treatment-naïve patients; PRolaCT-2: patients with limited duration of DA treatment (4–6 months); and PRolaCT-3: patients with persisting prolactinoma after DA treatment for > 2 years. PRolaCT-O will include patients who decline randomisation, due to e.g. a clear treatment preference. Primary outcomes are disease remission after 36 months and HRQoL after 12 months.

Discussion

Early or upfront surgical resection for patients with a limited-sized prolactinoma may be a reasonable alternative to the current standard practice of DA treatment, which we will investigate in three RCTs and an observational cohort study. Within the three RCTs, patients will be randomised between neurosurgical counselling and standard care. The observational study arm will recruit patients who refuse randomisation and have a pronounced treatment preference. PRolaCT will collect randomised and observational data, which may facilitate a more individually tailored practice of evidence-based medicine.

Trial registration

US National Library of Medicine registry (ClinicalTrials.gov) NCT04107480. Registered on 27 September 2019, registered retrospectively (by 2 months).

Supplementary Information

The online version contains supplementary material available at 10.1186/s13063-021-05604-y.

[Relevance of different study types in benefit assessment: Results from expert interviews]

BACKGROUND

Careful evaluation of new interventions is essential. Randomized controlled trials (RCTs) and systematic reviews of RCTs provide the most reliable evidence base in comparative evaluations of treatments and interventions. Observational study designs are found at the lower levels of the classical evidence hierarchy. In light of policy efforts to accelerate the availability of innovations, the use of evidence from non-randomized studies for reimbursement decisions is being discussed. The present study addresses the question of the value and feasibility of RCTs and observational studies for benefit assessment after an intervention has been introduced into practice.

METHODS

Experts from German-speaking countries were interviewed using qualitative, semi-structured interviews. Participants included experts in health economics, health services and health services research, health communication, ethics, and health policy. The interviews were conducted by one researcher and then analyzed using a deductive approach. Main categories and subcategories were extracted.

RESULTS

Twenty-six experts were invited to participate, and 15 agreed to be interviewed. The duration of the individual interviews varied between 35 and 80minutes. The interviewed experts considered it possible and necessary to conduct RCTs when an intervention is already available in care in order to demonstrate that an intervention is more effective than existing alternatives. Experts considered good study design, methodological knowledge of those conducting the study, infrastructure, and funding as underlying conditions. They emphasized that the benefit of an intervention cannot be conclusively clarified on the basis of observational studies and that RCTs can therefore only be dispensed with in exceptional cases. Therefore, observational studies do not represent an alternative to RCTs in benefit assessment in general. In the opinion of the experts, the requirements for observational studies included sensible criteria and quality assurance measures. As alternatives to RCTs, the experts named studies based on routine data or high-quality, prospective, comparative cohort studies. Individual experts were very critical of studies based on data from registries.

DISCUSSION AND CONCLUSION

The results of the interviews are underlined by findings from the scientific literature. For example, international scientists advocate for conducting RCTs instead of observational studies and for a reduction in hurdles and barriers for conducting RCTs. In international methodological standards for the preparation of health technology assessments, RCTs are also generally preferred for benefit assessment. Nevertheless, there is still a need for research about the optimal use of observational studies, alternative RCT designs and the use of data from routine care or registries, as policy makers, among others, believe that observational studies should be used for benefit assessment. The results show that experts still adhere to the established principles of clinical epidemiology. Observational studies should only be used as an alternative to RCTs in exceptional cases to examine whether one intervention is more effective than existing alternatives.

Addressing the issue of bias in observational studies: Using instrumental variables and a quasi-randomization trial in an ESME research project

Purpose

Observational studies using routinely collected data are faced with a number of potential shortcomings that can bias their results. Many methods rely on controlling for measured and unmeasured confounders. In this work, we investigate the use of instrumental variables (IV) and quasi-trial analysis to control for unmeasured confounders in the context of a study based on the retrospective Epidemiological Strategy and Medical Economics (ESME) database, which compared overall survival (OS) with paclitaxel plus bevacizumab or paclitaxel alone as first-line treatment in patients with HER2-negative metastatic breast cancer (MBC).

Patients and methods

Causal interpretations and estimates can be made from observation data using IV and quasi-trial analysis. Quasi-trial analysis has the same conceptual basis as IV, however, instead of using IV in the analysis, a “superficial” or “pseudo” randomized trial is used in a Cox model. For instance, in a multicenter trial, instead of using the treatment variable, quasi-trial analysis can consider the treatment preference in each center, which can be informative, and then comparisons of results between centers or clinicians can be informative.

Results

In the original analysis, the OS adjusted for major factors was significantly longer with paclitaxel and bevacizumab than with paclitaxel alone. Using the center-treatment preference as an instrument yielded to concordant results. For the quasi-trial analysis, a Cox model was used, adjusted on all factors initially used. The results consolidate those obtained with a conventional multivariate Cox model.

Conclusion

Unmeasured confounding is a major concern in observational studies, and IV or quasi-trial analysis can be helpful to complement analysis of studies of this nature.

[Benefit assessment of digital health applications-challenges and opportunities]

Digital health applications promise to improve patient health and medical care. This analysis provides a brief overview of evidence-based benefit assessment and the challenges to the underlying evidence as prerequisites for optimal patient-oriented decision making. Classical concepts in study design, recent developments, and innovative approaches are described with the aim of highlighting future areas of development in innovative study designs and strategic evaluation concepts for digital health applications. A special focus is on pragmatic study designs.Evidence-based benefit assessment has fundamental requirements and criteria regardless of the type of treatments evaluated. Reliable evidence is essential. Fast, efficient, reliable, and practice-relevant evaluation of digital health applications is not achieved by turning to nonrandomized trials, but rather by better pragmatic randomized trials. They are feasible and combine the characteristics of digital health applications, classical methodological concepts, and new approaches to study conduct. Routinely collected data, low-contact study conduct (remote trials, virtual trials), and digital biomarkers promote useful randomized real-world evidence as solid evidence base for digital health applications. Continuous learning evaluation with randomized designs embedded in routine care is key to sustainable and efficient benefit assessment of digital health applications and may be crucial for strategic improvement of healthcare.

Approaches to Assessing the Quality of Observational Studies of Clinical Practice Based on the Big Data Analysis

The article is devoted to the discussion of the problems of assessing the quality of observational studies in real clinical practice and determining their place in the hierarchy of evidence-based information. The concept of “big data” and the acceptability of using such a term to refer to large observational studies is being discussed. Data on the limitations of administrative and claims databases when performing observational studies to assess the effects of interventions are presented. The concept of confounding factors influencing the results of observational studies is discussed. Modern approaches to reducing the severity of bias in real-life clinical practice studies are presented. The criteria for assessing the quality of observational pharmacoepidemiological studies and the fundamental differences between such studies and randomized clinical trials are presented. The results of systematic reviews of real-life clinical trials to assess the effects of direct oral anticoagulants are discussed.

 

The effect of sodium-glucose transport protein 2 inhibitors on mortality and heart failure in randomized trials versus observational studies

AIM

Randomized clinical trials (RCTs) allocating type 2 diabetes patients to treatment with sodium-glucose transport protein 2 (SGLT-2) inhibitors or placebo have found significant effects on the risk of heart failure and modest effects on mortality. In the wake of the first trials, a number of observational studies have been conducted, some of these reporting a mortality reduction of 50% compared to active comparators. In this review, we systematically assess and compare the results on all-cause mortality, cardiovascular mortality and heart failure hospitalization observed in RCTs with the results obtained in observational studies.

METHOD

We performed a systematic bibliographical search including cardiovascular outcome trials and observational studies assessing the effect of SGLT-2 inhibitors on mortality and heart failure.

RESULTS

Seven RCTs and 23 observational studies were included in the current review. The observed heterogeneity between study results for all-cause mortality (p-interaction < 0.001) and cardiovascular mortality (p-interaction < 0.001) was explained by study type, whereas this was not the case for heart failure (p-interaction = 0.18).

CONCLUSION

Methodological considerations such as the omission of important confounders, immortal-time bias and residual confounding such as unmeasured social economic inequality may be the cause of the inflated results observed in observational studies and that calls for caution when observational studies are used to guide treatment of patients with type 2 diabetes.

Validity of observational evidence on putative risk and protective factors: appraisal of 3744 meta-analyses on 57 topics

BACKGROUND

The validity of observational studies and their meta-analyses is contested. Here, we aimed to appraise thousands of meta-analyses of observational studies using a pre-specified set of quantitative criteria that assess the significance, amount, consistency, and bias of the evidence. We also aimed to compare results from meta-analyses of observational studies against meta-analyses of randomized controlled trials (RCTs) and Mendelian randomization (MR) studies.

METHODS

We retrieved from PubMed (last update, November 19, 2020) umbrella reviews including meta-analyses of observational studies assessing putative risk or protective factors, regardless of the nature of the exposure and health outcome. We extracted information on 7 quantitative criteria that reflect the level of statistical support, the amount of data, the consistency across different studies, and hints pointing to potential bias. These criteria were level of statistical significance (pre-categorized according to 10^-6, 0.001, and 0.05 p-value thresholds), sample size, statistical significance for the largest study, 95% prediction intervals, between-study heterogeneity, and the results of tests for small study effects and for excess significance.

RESULTS

3744 associations (in 57 umbrella reviews) assessed by a median number of 7 (interquartile range 4 to 11) observational studies were eligible. Most associations were statistically significant at P < 0.05 (61.1%, 2289/3744). Only 2.6% of associations had P < 10^-6, ≥1000 cases (or ≥20,000 participants for continuous factors), P < 0.05 in the largest study, 95% prediction interval excluding the null, and no large between-study heterogeneity, small study effects, or excess significance. Across the 57 topics, large heterogeneity was observed in the proportion of associations fulfilling various quantitative criteria. The quantitative criteria were mostly independent from one another. Across 62 associations assessed in both RCTs and in observational studies, 37.1% had effect estimates in opposite directions and 43.5% had effect estimates differing beyond chance in the two designs. Across 94 comparisons assessed in both MR and observational studies, such discrepancies occurred in 30.8% and 54.7%, respectively.

CONCLUSIONS

Acknowledging that no gold-standard exists to judge whether an observational association is genuine, statistically significant results are common in observational studies, but they are rarely convincing or corroborated by randomized evidence.

The Rollercoaster of Paclitaxel in the Lower Limbs and Skeletons in the Closet: An Opinion Review

There have been concerns about the long-term risk of all-cause death with the use of paclitaxel-coated devices in the lower limbs. Results from a 2018 meta-analysis were corroborated by the US Food and Drug Administration and later confirmed by an individual patient data meta-analysis. However, population-based observational studies have produced contradictory results and often claimed a survival benefit with the use of paclitaxel. The recently published Swedish drug-elution trial in peripheral arterial disease did not confirm a significant mortality risk. In this review, the authors discuss the key elements of the identified mortality signal and stress important facts and figures that remain underrecognized and elusive. They also highlight the important types of epidemiological bias that pertain to the ongoing debate on paclitaxel.

Challenges and Lessons Learned From COVID-19 Trials: Should We Be Doing Clinical Trials Differently?

The COVID-19 crisis led to a flurry of clinical trials activity. The COVID-evidence database shows 2814 COVID-19 randomized trials registered as of February 16, 2021. Most were small (only 18% have a planned sample size > 500) and the rare completed ones have not provided published results promptly (only 283 trial publications as of February 2021). Small randomized trials and observational, nonrandomized analyses have not had a successful track record and have generated misleading expectations. Different large trials on the same intervention have generally been far more efficient in producing timely and consistent evidence. The rapid generation of evidence and accelerated dissemination of results have led to new challenges for systematic reviews and meta-analyses (eg, rapid, living, and scoping reviews). Pressure to regulatory agencies has also mounted with massive emergency authorizations, but some of them have had to be revoked. Pandemic circumstances have disrupted the way trials are conducted; therefore, new methods have been developed and adopted more widely to facilitate recruitment, consent, and overall trial conduct. On the basis of the COVID-19 experience and its challenges, planning of several large, efficient trials, and wider use of adaptive designs might change the future of clinical research. Pragmatism, integration in clinical care, efficient administration, promotion of collaborative structures, and enhanced integration of existing data and facilities might be several of the legacies of COVID-19 on future randomized trials.

Confounding and bias in observational studies in inflammatory bowel disease: a meta-epidemiological study

BACKGROUND

Observational research concerning inflammatory bowel disease (IBD) is highly susceptible to spurious findings because of confounding and bias.

AIM

To investigate how these issues were reported in this research field.

METHODS

We identified and appraised a random sample of 160 observational studies concerning IBD published in high-impact gastroenterology journals and the most respected specialty journals of the condition. We applied a standardised methodology to assess how confounding and bias were reported and discussed, and investigated the association between yearly citations and study characteristics using mixed-effect multivariable regression analysis.

RESULTS

The authors of 67 out of 160 articles (41.9%) mentioned confounding, and in 89 cases (55.6%) reported any bias. Although most authors applied strategies to minimise confounding or bias (n = 139; 86.9%) and acknowledged at least one unadjusted confounder (n = 116; 72.5%), a minority commented about whether the main findings could have been affected (n = 60; 37.5%). Very few authors (n = 7; 4.4%) called for caution in interpreting the results in the discussion. Reporting of confounding and bias was particularly lacking for case-control studies, those not using routinely collected data, those employing laboratory analyses as the primary method of assessment and studies investigating non-modifiable exposures. In adjusted analyses, mentioning or alluding to confounding was positively associated with yearly citations (P = 0.010), whereas calling for a cautious interpretation of the findings was not.

CONCLUSIONS

Reporting of confounding is inadequate and its acknowledgement is often neglected in interpreting high-impact observational research in IBD. These results encourage a more careful evaluation of the consequences of confounding and bias.

No inexplicable disagreements between real-world data-based nonrandomized controlled studies and randomized controlled trials were found

OBJECTIVES

We assessed disagreements between nonrandomized controlled studies based on real-world data (NRCS-RWDs) and randomized controlled trials (RCTs).

STUDY DESIGN AND SETTING

We systematically searched for studies that compared treatment effect estimates from NRCS-RWDs and RCTs on the same clinical question. We assessed the potential difference between NRCS-RWDs and RCTs related to internal and external validity. We calculated various meta-epidemiological measures to assess agreement. In case of disagreements, we tried to identify the probable causes of disagreements.

RESULTS

We included 12 studies comparing 15 treatment effect estimates of NRCS-RWDs and RCTs. There were many potential causes of disagreement. Ninety-five percent confidence intervals overlapped for 12 of 15 treatment effect estimates. Our analysis on predicted vs. observed overlap showed that there were no more disagreements than expected by chance. We observed only two substantial differences between the 15 treatment effect estimates. In both cases, we identified risk of bias in the NRCS-RWDs as the most probable cause of disagreement.

CONCLUSION

Our findings suggest that there are clinical questions where the difference in risk of bias between a well-conducted NRCS-RWD and an RCT is negligible. In our analysis, threats to external validity appeared to have no or only a weak impact on the disagreements of treatment effect estimates.

Emulating Randomized Clinical Trials With Nonrandomized Real-World Evidence Studies: First Results From the RCT DUPLICATE Initiative

BACKGROUND

Regulators are evaluating the use of noninterventional real-world evidence (RWE) studies to assess the effectiveness of medical products. The RCT DUPLICATE initiative (Randomized, Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology) uses a structured process to design RWE studies emulating randomized, controlled trials (RCTs) and compare results. We report findings of the first 10 trial emulations, evaluating cardiovascular outcomes of antidiabetic or antiplatelet medications.

METHODS

We selected 3 active-controlled and 7 placebo-controlled RCTs for replication. Using patient-level claims data from US commercial and Medicare payers, we implemented inclusion and exclusion criteria, selected primary end points, and comparator populations to emulate those of each corresponding RCT. Within the trial-mimicking populations, we conducted propensity score matching to control for >120 preexposure confounders. All study measures were prospectively defined and protocols registered before hazard ratios and 95% CIs were computed. Success criteria for the primary analysis were prespecified for each replication.

RESULTS

Despite attempts to emulate RCT design as closely as possible, differences between the RCT and corresponding RWE study populations remained. The regulatory conclusions were equivalent in 6 of 10. The RWE emulations achieved a hazard ratio estimate that was within the 95% CI from the corresponding RCT in 8 of 10 studies. In 9 of 10, either the regulatory or estimate agreement success criteria were fulfilled. The largest differences in effect estimates were found for RCTs where second-generation sulfonylureas were used as a proxy for placebo regarding cardiovascular effects. Nine of 10 replications had a standardized difference between effect estimates of <2, which suggests differences within expected random variation.

CONCLUSIONS

Agreement between RCT and RWE findings varies depending on which agreement metric is used. Interim findings indicate that selection of active comparator therapies with similar indications and use patterns enhances the validity of RWE. Even in the context of active comparators, concordance between RCT and RWE findings is not guaranteed, partially because trials are not emulated exactly. More trial emulations are needed to understand how often and in what contexts RWE findings match RCTs. Registration: URL: https://www.clinicaltrials.gov; Unique identifiers: NCT03936049, NCT04215523, NCT04215536, NCT03936010, NCT03936036, NCT03936062, NCT03936023, NCT03648424, NCT04237935, NCT04237922.

The IDEAL framework in neurosurgery: a bibliometric analysis

BACKGROUND

The Idea, Development, Exploration, Assessment and Long-term study (IDEAL) framework was created to provide a structured way for assessing and evaluating novel surgical techniques and devices.

OBJECTIVES

The aim of this paper was to investigate the utilization of the IDEAL framework within neurosurgery, and to identify factors influencing implementation.

METHODS

A bibliometric analysis of the 7 key IDEAL papers on Scopus, PubMed, Embase, Web of Science, and Google Scholar databases (2009-2019) was performed. A second journal-specific search then identified additional papers citing the IDEAL framework. Publications identified were screened by two independent reviewers to select neurosurgery-specific articles.

RESULTS

The citation search identified 1336 articles. The journal search identified another 16 articles. Following deduplication and review, 51 relevant articles remained; 14 primary papers (27%) and 37 secondary papers (73%). Of the primary papers, 5 (36%) papers applied the IDEAL framework to their research correctly; two were aligned to the pre-IDEAL stage, one to the Idea and Development stages, and two to the Exploration stage. Of the secondary papers, 21 (57%) explicitly discussed the IDEAL framework. Eighteen (86%) of these were supportive of implementing the framework, while one was not, and two were neutral.

CONCLUSION

The adoption of the IDEAL framework in neurosurgery has been slow, particularly for early-stage neurosurgical techniques and inventions. However, the largely positive reviews in secondary literature suggest potential for increased use that may be achieved with education and publicity.

Replication of randomized clinical trial results using real-world data: paving the way for effectiveness decisions

The FDA is preparing guidance about using real-world evidence (RWE) to support decisions about product effectiveness. Several ongoing efforts aim to replicate randomized clinical trial (RCT) results using RWE with the intent of identifying circumstances and methods that provide valid evidence of drug effects. Lack of agreement may not be due to faulty methods but rather to the challenges with emulating RCTs, differences in healthcare settings and patient populations, differences in effect measures and data analysis, bias, and/or the efficacy-effectiveness gap. In fact, for some decisions, RWE may lead to better understanding of how treatments work in usual care settings than a more constrained view from RCTs. Efforts to reconcile the role and opportunities for generating complementary evidence from RWE and RCTs will advance regulatory science.