Reproducibility of real-world evidence studies using clinical practice data to inform regulatory and coverage decisions

App Engagement as a Predictor of Weight Loss in Blended-Care Interventions: Retrospective Observational Study Using Large-Scale Real-World Data

BACKGROUND

Early weight loss is an established predictor for treatment outcomes in weight management interventions for people with obesity. However, there is a paucity of additional, reliable, and clinically actionable early predictors in weight management interventions. Novel blended-care weight management interventions combine coach and app support and afford new means of structured, continuous data collection, informing research on treatment adherence and outcome prediction.

OBJECTIVE

Against this backdrop, this study analyzes app engagement as a predictor for weight loss in large-scale, real-world, blended-care interventions. We hypothesize that patients who engage more frequently in app usage in blended-care treatment (eg, higher logging activity) lose more weight than patients who engage comparably less frequently at 3 and 6 months of intervention.

METHODS

Real-world data from 19,211 patients in obesity treatment were analyzed retrospectively. Patients were treated with 3 different blended-care weight management interventions, offered in Switzerland, the United Kingdom, and Germany by a digital behavior change provider. The principal component analysis identified an overarching metric for app engagement based on app usage. A median split informed a distinction in higher and lower engagers among the patients. Both groups were matched through optimal propensity score matching for relevant characteristics (eg, gender, age, and start weight). A linear regression model, combining patient characteristics and app-derived data, was applied to identify predictors for weight loss outcomes.

RESULTS

For the entire sample (N=19,211), mean weight loss was -3.24% (SD 4.58%) at 3 months and -5.22% (SD 6.29%) at 6 months. Across countries, higher app engagement yielded more weight loss than lower engagement after 3 but not after 6 months of intervention (P3 months<.001 and P6 months=.59). Early app engagement within the first 3 months predicted percentage weight loss in Switzerland and Germany, but not in the United Kingdom (PSwitzerland<.001, PUnited Kingdom=.12, and PGermany=.005). Higher age was associated with stronger weight loss in the 3-month period (PSwitzerland=.001, PUnited Kingdom=.002, and PGermany<.001) and, for Germany, also in the 6-month period (PSwitzerland=.09, PUnited Kingdom=.46, and PGermany=.03). In Switzerland, higher numbers of patients' messages to coaches were associated with higher weight loss (P3 months<.001 and P6 months<.001). Messages from coaches were not significantly associated with weight loss (all P>.05).

CONCLUSIONS

Early app engagement is a predictor of weight loss, with higher engagement yielding more weight loss than lower engagement in this analysis. This new predictor lends itself to automated monitoring and as a digital indicator for needed or adapted clinical action. Further research needs to establish the reliability of early app engagement as a predictor for treatment adherence and outcomes. In general, the obtained results testify to the potential of app-derived data to inform clinical monitoring practices and intervention design.

Real-World Evidence to Reinforce Clinical Trial Evidence in Health Technology Assessment: A Critical Review of Real-World Evidence Requirements from Seven Countries and Recommendations to Improve Acceptance

BACKGROUND

Real-world evidence (RWE) can reinforce clinical trial evidence in health technology assessment (HTA).

OBJECTIVES

Review HTA bodies' (HTAbs) requirements for RWE, real uses, and acceptance across seven countries (Brazil, Canada, France, Germany, Italy, Spain, and the United Kingdom) and outline recommendations that may improve acceptance of RWE in efficacy/effectiveness assessments and appraisals processes.

METHODS

RWE requirements were summarized based on HTAbs' guidelines. Acceptance by HTAbs was evaluated based on industry experience and case studies.

RESULTS

As of June 2022, RWE methodological guidelines were in place in three of the seven countries. HTAbs typically requested analyses based on local data sources, but the preferred study design and data sources differed. HTAbs had individual submission, assessment, and appraisal processes; some allowed early meetings for the protocol and/or results validation, though few involved external experts or medical societies to provide input to assessment and appraisal. The extent of submission, assessment, and appraisal requirements did not necessarily reflect the degree of acceptance.

CONCLUSION

All the countries reviewed face common challenges regarding the use of RWE. Our proposals address the need to facilitate collaboration and communication with industry and regulatory agencies and the need for specific guidelines describing RWE design and criteria of acceptance throughout the assessment and appraisal processes.

Markov modeling for cost-effectiveness using federated health data network

OBJECTIVE

To introduce 2 R-packages that facilitate conducting health economics research on OMOP-based data networks, aiming to standardize and improve the reproducibility, transparency, and transferability of health economic models.

MATERIALS AND METHODS

We developed the software tools and demonstrated their utility by replicating a UK-based heart failure data analysis across 5 different international databases from Estonia, Spain, Serbia, and the United States.

RESULTS

We examined treatment trajectories of 47 163 patients. The overall incremental cost-effectiveness ratio (ICER) for telemonitoring relative to standard of care was 57 472 €/QALY. Country-specific ICERs were 60 312 €/QALY in Estonia, 58 096 €/QALY in Spain, 40 372 €/QALY in Serbia, and 90 893 €/QALY in the US, which surpassed the established willingness-to-pay thresholds.

DISCUSSION

Currently, the cost-effectiveness analysis lacks standard tools, is performed in ad-hoc manner, and relies heavily on published information that might not be specific for local circumstances. Published results often exhibit a narrow focus, central to a single site, and provide only partial decision criteria, limiting their generalizability and comprehensive utility.

CONCLUSION

We created 2 R-packages to pioneer cost-effectiveness analysis in OMOP CDM data networks. The first manages state definitions and database interaction, while the second focuses on Markov model learning and profile synthesis. We demonstrated their utility in a multisite heart failure study, comparing telemonitoring and standard care, finding telemonitoring not cost-effective.

Prevalence of Pretreatment Testing Recommended for Patients With Chronic Inflammatory Skin Diseases

Importance

Laboratory testing for the presence of tuberculosis, hepatitis, and other conditions before starting most systemic immunomodulatory agents is recommended in patients with chronic inflammatory skin diseases (CISD) but current testing patterns in the US are unclear.

Objective

To determine the prevalence of pretreatment testing that is recommended for patients with CISD (psoriasis, hidradenitis suppurativa, or atopic dermatitis).

Design, Setting, and Participants

This descriptive analysis of US commercial insurance claims databases from December 31, 2002, to December 31, 2020, included adult patients with CISD (psoriasis, hidradenitis suppurativa, or atopic dermatitis) who started an immunomodulatory agent, including methotrexate, tumor necrosis factor α inhibitors, interleukin (IL)-17Ai, ustekinumab, IL-23i, dupilumab, or apremilast.

Main Outcomes and Measures

The proportion of patients who underwent the screening tests as suggested by professional societies-including for tuberculosis, hepatitis, and liver function; complete blood cell counts; and lipid panels-were determined within 6 months before and during 2 years after treatment start.

Results

A total of 122 308 patients with CISDs (median [IQR] age, 49 [38-58] years; 63 663 [52.1%] male) starting systemic immunomodulatory treatment in the US were included. Treatment for patients with CISDs comprised methotrexate (28 684), tumor necrosis factor α inhibitors (40 965), ustekinumab (12 841), IL-23i (6116), IL-17Ai (9799), dupilumab (7787), or apremilast (16 116). Complete blood cell count was the most common test, performed in 41% (3161/7787) to 69% (19 659/28 684) of individuals before initiation across treatments. Between 11% (889/7787) and 59% (3613/6116) of patients had tuberculosis screening within 6 months before treatment, and 3% (149/4577) to 26% (1559/6097) had updated tests 1 year later. Between 13% (1006/7787) and 41% (16 728/40 965) had hepatitis panels before treatment. Low pretreatment testing levels before apremilast (15% [2331/16 116] to 45% [7253/16 116]) persisted a year into treatment (9% [816/8496] to 36% [2999/8496]) and were similar to dupilumab (11% [850/7787] to 41% [3161/7787] vs 3% [149/4577] to 25% [1160/4577]).

Conclusions and Relevance

In this descriptive analysis of patients with CISDs starting systemic immunomodulatory treatment in the US, less than 60% received the recommended pretreatment testing. Additional research is required to understand whether variations in testing affect patient outcomes.

Sodium-Glucose Cotransporter 2 Inhibitors and Nephrolithiasis Risk in Patients With Type 2 Diabetes

Importance

Type 2 diabetes (T2D) is associated with an increased risk of kidney stones. Sodium-glucose cotransporter 2 inhibitors (SGLT2is) might lower the risk of nephrolithiasis by altering urine composition. However, no studies have investigated the association between SGLT2i use and nephrolithiasis risk in patients receiving routine care in the US.

Objective

To investigate the association between SGLT2i use and nephrolithiasis risk in clinical practice.

Design, Setting, and Participants

This new-user, active comparator cohort study used data from commercially insured adults (aged ≥18 years) with T2D who initiated treatment with SGLT2is, glucagon-like peptide 1 receptor agonists (GLP-1RAs), or dipeptidyl peptidase 4 inhibitors (DPP4is) between April 1, 2013, and December 31, 2020. The data were analyzed from July 2021 through June 2023.

Exposure

New initiation of an SGLT2i, GLP-1RA, or DPP4i.

Main Outcomes and Measures

The primary outcome was nephrolithiasis diagnosed by International Classification of Diseases codes in the inpatient or outpatient setting. New SGLT2i users were 1:1 propensity score matched to new users of a GLP-1RA or DPP4i in pairwise comparisons. Incidence rates, rate differences (RDs), and estimated hazard ratios (HRs) with 95% CIs were calculated.

Results

After 1:1 propensity score matching, a total of 716 406 adults with T2D (358 203 pairs) initiating an SGLT2i or a GLP-1RA (mean [SD] age, 61.4 [9.7] years for both groups; 51.4% vs 51.2% female; 48.6% vs 48.5% male) and 662 056 adults (331 028 pairs) initiating an SGLT2i or a DPP4i (mean [SD] age, 61.8 [9.3] vs 61.7 [10.1] years; 47.4% vs 47.3% female; 52.6% vs 52.7% male) were included. Over a median follow-up of 192 (IQR, 88-409) days, the risk of nephrolithiasis was lower in patients initiating an SGLT2i than among those initiating a GLP-1RA (14.9 vs 21.3 events per 1000 person-years; HR, 0.69 [95% CI, 0.67-0.72]; RD, -6.4 [95% CI, -7.1 to -5.7]) or a DPP4i (14.6 vs 19.9 events per 1000 person-years; HR, 0.74 [95% CI, 0.71-0.77]; RD, -5.3 [95% CI, -6.0 to -4.6]). The association between SGLT2i use and nephrolithiasis risk was similar by sex, race and ethnicity, history of chronic kidney disease, and obesity. The magnitude of the risk reduction with SGLT2i use was larger among adults aged younger than 70 years vs aged 70 years or older (HR, 0.85 [95% CI, 0.79-0.91]; RD, -3.46 [95% CI, -4.87 to -2.05] per 1000 person-years; P for interaction <.001).

Conclusions and Relevance

These findings suggest that in adults with T2D, SGLT2i use may lower the risk of nephrolithiasis compared with GLP-1RAs or DPP4is and could help to inform decision-making when prescribing glucose-lowering agents for patients who may be at risk for developing nephrolithiasis.

Cardiorenal effectiveness of empagliflozin vs. glucagon-like peptide-1 receptor agonists: final-year results from the EMPRISE study

BACKGROUND

No randomized clinical trials have directly compared the cardiorenal effectiveness of empagliflozin and GLP-1RA agents with demonstrated cardioprotective effects in patients with a broad spectrum of cardiovascular risk. We reported the final-year results of the EMPRISE study, a monitoring program designed to evaluate the cardiorenal effectiveness of empagliflozin across broad patient subgroups.

METHODS

We identified patients ≥ 18 years old with type 2 diabetes who initiated empagliflozin or GLP-1RA from 2014 to 2019 using US Medicare and commercial claims databases. After 1:1 propensity score matching using 143 baseline characteristics, we evaluated risks of outcomes including myocardial infarction (MI) or stroke, hospitalization for heart failure (HHF), major adverse cardiovascular events (MACE - MI, stroke, or cardiovascular mortality), a composite of HHF or cardiovascular mortality, and progression to end-stage kidney disease (ESKD) (in patients with chronic kidney disease stages 3-4). We estimated hazard ratios (HR) and rate differences (RD) per 1,000 person-years, overall and within subgroups of age, sex, baseline atherosclerotic cardiovascular disease (ASCVD), and heart failure (HF).

RESULTS

We identified 141,541 matched pairs. Compared with GLP-1RA, empagliflozin was associated with similar risks of MI or stroke [HR: 0.99 (0.92, 1.07); RD: -0.23 (-1.25, 0.79)], and lower risks of HHF [HR: 0.50 (0.44, 0.56); RD: -2.28 (-2.98, -1.59)], MACE [HR: 0.90 (0.82, 0.99); RD: -2.54 (-4.76, -0.32)], cardiovascular mortality or HHF [HR: 0.77 (0.69, 0.86); RD: -4.11 (-5.95, -2.29)], and ESKD [0.75 (0.60, 0.94); RD: -6.77 (-11.97, -1.61)]. Absolute risk reductions were larger in older patients and in those with baseline ASCVD/HF. They did not differ by sex.

CONCLUSIONS

The cardiovascular benefits of empagliflozin vs. cardioprotective GLP-1RA agents were larger in older patients and in patients with history of ASCVD or HF, while they did not differ by sex. In patients with advanced CKD, empagliflozin was associated with risk reductions of progression to ESKD.

Implementation of a Taxonomy-Based Framework for the Selection of Appropriate Drugs and Outcomes for Real-World Data Signal Detection Studies

INTRODUCTION

For signal detection studies investigating either drug safety or method evaluation, the choice of drug-outcome pairs needs to be tailored to the planned study design and vice versa. While this is well understood in hypothesis-testing epidemiology, it should be as important in signal detection, but this has not widely been considered. There is a need for a taxonomy framework to provide guidance and a systematic reproducible approach to the selection of appropriate drugs and outcomes for signal detection studies either investigating drug safety or assessing method performance using real-world data.

OBJECTIVE

The aim was to design a general framework for the selection of appropriate drugs and outcomes for signal detection studies given a study design of interest. As a motivating example, we illustrate how the framework is applied to build a reference set for a study aiming to assess the performance of the self-controlled case series with active comparators.

METHODS

We reviewed criteria presented in two published studies which aimed to provide practical advice for choosing the appropriate signal evaluation methodology, and assessed their relevance for signal detection. Further characteristics specific to signal detection were added. The final framework is based on: the application of study design requirements, the database(s) of interest, and the clinical importance of the drug(s) and outcome(s) under consideration. This structure was applied by selecting drug-outcome pairs as a reference set (i.e. list of drug-outcome pairs classified as positive or negative controls) for which the method is expected to work well for a signal detection study aiming to assess the performance of self-controlled case series. Eight criteria were used, related to the application of self-controlled case series assumptions, choice of active comparators, coverage in the database of interest and clinical importance of the outcomes.

RESULTS

After application of the framework, two classes of antibiotics (seven drugs) were selected for the study, and 28 outcomes from all organ classes were chosen from the drug labels, out of the 273 investigated. In total, this corresponds to 104 positive controls (drug-outcome pairs) and 58 negative controls.

CONCLUSIONS

We proposed and applied a framework for the selection of drugs and outcomes for both drug safety signal detection and method assessment used in signal detection to optimise their performance given a study design. This framework will eliminate part of the bias relating to drugs and outcomes not being suited to the method or database. The main difficulty lies in the choice of the criteria and their application to ensure systematic selection, especially as some information remains unknown in signal detection, and clinical judgement was needed on occasions. The same framework could be adapted for other methods.

Quality of systematic reviews with meta-analyses of resveratrol: A methodological systematic review

Recently, several meta-analyses (MAs) have focused on the health effects of resveratrol. However, the methodological and reporting quality of these MAs has not yet been fully evaluated so far. Therefore, the present study evaluated the quality of these MAs through a methodological systematic review. Systematic searches were conducted in PubMed, Embase, Web of Science, and Cochrane Library from inception until May 20, 2022, and PubMed was used to update the search until September 6, 2023. The methodological and reporting quality of the selected MAs was evaluated using AMSTAR-2 and PRISMA 2009. Fifty-one MAs published during 2013-2023 were included. In each review, the number of primary studies ranged from 3 to 37, and the number of participants ranged from 50 to 2114. Among the first-listed primary outcomes, only 23 (45.10%) were "positive." As for the methodological quality, most MAs (44, 86.27%) on resveratrol were rated critically low. Inadequate reporting of the included MAs mainly involved items 2 ("Structured summary"), 5 ("Protocol and registration"), 8 ("Search"), 9 ("Study selection"), 10 ("Data collection process"), 12 ("Risk of bias in individual studies"), and 24 ("Summary of evidence") based on the PRISMA 2009. Additionally, journal's impact factor, number of authors, and funding support were positively associated with the overall methodological quality but were not statistically significant (p > 0.05). Future MAs on resveratrol require better design, implementation, and reporting by following the Cochrane Handbook, AMSTAR-2, and PRISMA.

Stratified analysis in comparative effectiveness studies that emulate randomized trials

PURPOSE

For observational cohort studies that employ matching by propensity scores (PS), preliminary stratification by consequential predictors of outcome better emulates stratified randomization and potentially reduces variance and bias through relaxed dependence on modeling assumptions. We assessed the impact of pre-stratification in two real-life examples. For both, prior evidence from placebo-controlled randomized clinical trials (RCTs) suggested small or no risk reduction, but observational analysis suggested protection, presumably the result of confounding bias.

STUDY DESIGN AND SETTING

The study populations consisted of Medicare beneficiaries (2014-18) with type 2 diabetes initiating either (i) empagliflozin versus dipeptidyl peptidase-4 inhibitors (DPP-4i) or (ii) empagliflozin versus glucagon-like peptide-1 receptor agonists (GLP-1RA). The outcome was myocardial infarction or stroke. We estimated hazard ratios (HR) and rate differences (RD) after controlling for 143 pre-exposure covariates via 1:1 PS matching after (1) PS estimation in the total cohort (total-cohort PS-matching) and (2) PS estimation separately by baseline cardiovascular disease (stratified PS matching).

RESULTS

Stratified PS matching resulted in HRs that exceeded those from total-cohort PS-matching by 13% and 9%, respectively, for the comparisons of empagliflozin to DPP-4i and GLP-1RA. Against both comparators, HRs and RDs after stratified PS matching were closer to the null, with slightly higher variances (2%-3%) than those after total-cohort PS matching.

CONCLUSION

Stratified PS matching produced effect estimates closer to the expected trial findings than total-cohort PS matching. The price paid in increased variance was minimal.

High-Dose vs. Standard-Dose Influenza Vaccine and Cardiopulmonary Hospitalization or Mortality: Emulating the INVESTED Trial Using Insurance Claims Data

The INVESTED trial did not show benefits of high-dose (HD) vaccine vs. standard-dose (SD) for a primary composite outcome of cardiopulmonary hospitalization or all-cause mortality (hazard ratio (HR) = 1.05, 95% confidence interval (CI) = 0.96-1.15) and its components (all-cause mortality HR = 1.01, 95% CI = 0.84-1.21, cardiopulmonary hospitalization HR = 1.05, 95% CI = 0.96-1.16) during three influenza seasons (2016-2019) among participants with recent myocardial infarction or hospitalization for heart failure (HHF). We emulated INVESTED using Medicare claims data to assess whether the real-world evidence (RWE) study reached similar conclusions. We identified 1:1 propensity score (PS)-matched trial-eligible Medicare beneficiaries aged > 65 years and with prior HHF who received an HD or SD vaccine for the 2016-2019 seasons. We also re-analyzed the INVESTED trial data restricting to participants > 65 years with prior HHF to align eligibility criteria more closely with the RWE study. We compared HRs from the trial and RWE study for the main outcomes. Among 53,393 pairs of PS-matched Medicare beneficiaries, the HD vaccine group showed lower risk of the primary composite outcome (HR = 0.96, 95% CI = 0.95-0.98) and all-cause mortality (HR = 0.93, 95% CI = 0.91-0.95), and similar risk of cardiopulmonary hospitalization (HR = 0.98, 95% CI = 0.96-1.00), compared with SD. The RWE and trial results were closely concordant after the trial population was limited to participants > 65 years with prior HHF: trial-based results for the primary composite outcome (HR = 1.02, 95% CI = 0.89-1.17), all-cause mortality (HR = 0.92, 95% CI = 0.72-1.16), and cardiopulmonary hospitalization (HR = 1.02, 95% CI = 0.88-1.18). Although similar to the main trial results, the RWE was closer to the results from trial participants with aligned eligibility criteria. This study affirms the importance of considering different distributions of baseline patient characteristics when comparing trial findings to RWE.

Replicability of simulation studies for the investigation of statistical methods: the RepliSims project

Results of simulation studies evaluating the performance of statistical methods can have a major impact on the way empirical research is implemented. However, so far there is limited evidence of the replicability of simulation studies. Eight highly cited statistical simulation studies were selected, and their replicability was assessed by teams of replicators with formal training in quantitative methodology. The teams used information in the original publications to write simulation code with the aim of replicating the results. The primary outcome was to determine the feasibility of replicability based on reported information in the original publications and supplementary materials. Replicasility varied greatly: some original studies provided detailed information leading to almost perfect replication of results, whereas other studies did not provide enough information to implement any of the reported simulations. Factors facilitating replication included availability of code, detailed reporting or visualization of data-generating procedures and methods, and replicator expertise. Replicability of statistical simulation studies was mainly impeded by lack of information and sustainability of information sources. We encourage researchers publishing simulation studies to transparently report all relevant implementation details either in the research paper itself or in easily accessible supplementary material and to make their simulation code publicly available using permanent links.

Utilization of anonymization techniques to create an external control arm for clinical trial data

BACKGROUND

Subject-level real-world data (RWD) collected during daily healthcare practices are increasingly used in medical research to assess questions that cannot be addressed in the context of a randomized controlled trial (RCT). A novel application of RWD arises from the need to create external control arms (ECAs) for single-arm RCTs. In the analysis of ECAs against RCT data, there is an evident need to manage and analyze RCT data and RWD in the same technical environment. In the Nordic countries, legal requirements may require that the original subject-level data be anonymized, i.e., modified so that the risk to identify any individual is minimal. The aim of this study was to conduct initial exploration on how well pseudonymized and anonymized RWD perform in the creation of an ECA for an RCT.

METHODS

This was a hybrid observational cohort study using clinical data from the control arm of the completed randomized phase II clinical trial (PACIFIC-AF) and RWD cohort from Finnish healthcare data sources. The initial pseudonymized RWD were anonymized within the (k, ε)-anonymity framework (a model for protecting individuals against identification). Propensity score matching and weighting methods were applied to the anonymized and pseudonymized RWD, to balance potential confounders against the RCT data. Descriptive statistics for the potential confounders and overall survival analyses were conducted prior to and after matching and weighting, using both the pseudonymized and anonymized RWD sets.

RESULTS

Anonymization affected the baseline characteristics of potential confounders only marginally. The greatest difference was in the prevalence of chronic obstructive pulmonary disease (4.6% vs. 5.4% in the pseudonymized compared to the anonymized data, respectively). Moreover, the overall survival changed in anonymization by only 8% (95% CI 4-22%). Both the pseudonymized and anonymized RWD were able to produce matched ECAs for the RCT data. Anonymization after matching impacted overall survival analysis by 22% (95% CI -21-87%).

CONCLUSIONS

Anonymization may be a viable technique for cases where flexible data transfer and sharing are required. As anonymization necessarily affects some aspects of the original data, further research and careful consideration of anonymization strategies are needed.

Concepts of Designing and Implementing Pharmacoepidemiology Studies on the Safety of Systemic Treatments in Dermatology Practice

The U.S. Food and Drug Administration and clinical guidelines use evidence from pharmacoepidemiology studies to inform prescribing decisions and fill evidence gaps left by randomized controlled trials (RCTs). The long-term safety and infrequent adverse reactions are not well-understood when RCTs are short and involve few patients, as is the case for most systemic immunomodulating drugs in dermatology. A better understanding of the design and implementation of pharmacoepidemiology studies will help practitioners assess the accuracy of etiologic findings and use them with confidence in clinical practice. Conducting pharmacoepidemiology studies follows a structured approach, which we discuss in this article: (i) a design layer connects the research question with the appropriate study design, and considering which hypothetical RCT one ideally would want to conduct reduces inadvertent investigator errors; (ii) a measurement layer transforms longitudinal patient-level data into variables that identify the study population, patient characteristics, treatment, and outcomes; and (iii) the analysis focuses on the causal treatment effect estimation. The review and interpretation of pharmacoepidemiology studies should consider issues beyond a typical review of RCTs, chiefly the lack of baseline randomization and the use of secondary data. Well-designed and well-conducted pharmacoepidemiologic studies complement dermatology practice with critical information on prescribing systemic medications.

Evidence-Based Surgery: What Can Intra-Operative Images Contribute?

Evidence-based medicine integrates results from randomized controlled trials (RCTs) and meta-analyses, combining the best external evidence with individual clinical expertise and patients' preferences. However, RCTs of surgery differ from those of medicine in that surgical performance is often assumed to be consistent. Yet, evaluating whether each surgery is performed to the same standard is quite challenging. As a primary issue, the novelty of this review is to emphasize-with a focus on orthopedic trauma-the advantage of having complete intra-operative image documentation, allowing the direct evaluation of the quality of the intra-operative technical performance. The absence of complete intra-operative image documentation leads to the inhomogeneity of case series, yielding inconsistent results due to the impossibility of a secondary analysis. Thus, comparisons and the reproduction of studies are difficult. Access to complete intra-operative image data in surgical RCTs allows not only secondary analysis but also comparisons with similar cases. Such complete data can be included in electronic papers. Offering these data to peers-in an accessible link-when presenting papers facilitates the selection process and improves publications for readers. Additionally, having access to the full set of image data for all presented cases serves as a rich resource for learning. It enables the reader to sift through the information and pinpoint the details that are most relevant to their individual needs, allowing them to potentially incorporate this knowledge into daily practice. A broad use of the concept of complete intra-operative image documentation is pivotal for bridging the gap between clinical research findings and real-world applications. Enhancing the quality of surgical RCTs would facilitate the equalization of evidence acquisition in both internal medicine and surgery. Joint effort by surgeons, scientific societies, publishers, and healthcare authorities is needed to support the ideas, implement economic requirements, and overcome the mental obstacles to its realization.

Predicting Treatment Effects of a New-to-Market Drug in Clinical Practice Based on Phase III Randomized Trial Results

Trial results may not be generalizable to target populations treated in clinical practice with different distributions of baseline characteristics that modify the treatment effect. We used outcome models developed with trial data to predict treatment effects in Medicare populations. We used data from the Randomized Evaluation of Long-Term Anticoagulation Therapy trial (RE-LY), which investigated the effect of dabigatran vs. warfarin on stroke or systemic embolism (stroke/SE) among patients with atrial fibrillation. We developed outcome models by fitting proportional hazards models in trial data. Target populations were trial-eligible Medicare beneficiaries who initiated dabigatran or warfarin in 2010-2011 ("early") and 2010-2017 ("extended"). We predicted 2-year risk ratios (RRs) and risk differences (RDs) for stroke/SE, major bleeding, and all-cause death in the Medicare populations using the observed baseline characteristics. The trial and early target populations had similar mean (SD) CHADS2 scores (2.15 (SD 1.13) vs. 2.15 (SD 0.91)) but different mean ages (71 vs. 79 years). Compared with RE-LY, the early Medicare population had similar predicted benefit of dabigatran vs. warfarin for stroke/SE (trial RR = 0.63, 95% confidence interval (CI) = 0.50 to 0.76 and RD = -1.37%, -1.96% to -0.77%, Medicare RR = 0.73, 0.65 to 0.82 and RD = -0.92%, -1.26% to -0.59%) and risks for major bleeding and all-cause death. The time-extended target population showed similar results. Outcome model-based prediction facilitates estimating the average treatment effects of a drug in different target populations when treatment and outcome data are unreliable or unavailable. The predicted effects may inform payers' coverage decisions for patients, especially shortly after a drug's launch when observational data are scarce.

Using public clinical trial reports to probe non-experimental causal inference methods

BACKGROUND

Non-experimental studies (also known as observational studies) are valuable for estimating the effects of various medical interventions, but are notoriously difficult to evaluate because the methods used in non-experimental studies require untestable assumptions. This lack of intrinsic verifiability makes it difficult both to compare different non-experimental study methods and to trust the results of any particular non-experimental study.

METHODS

We introduce TrialProbe, a data resource and statistical framework for the evaluation of non-experimental methods. We first collect a dataset of pseudo "ground truths" about the relative effects of drugs by using empirical Bayesian techniques to analyze adverse events recorded in public clinical trial reports. We then develop a framework for evaluating non-experimental methods against that ground truth by measuring concordance between the non-experimental effect estimates and the estimates derived from clinical trials. As a demonstration of our approach, we also perform an example methods evaluation between propensity score matching, inverse propensity score weighting, and an unadjusted approach on a large national insurance claims dataset.

RESULTS

From the 33,701 clinical trial records in our version of the ClinicalTrials.gov dataset, we are able to extract 12,967 unique drug/drug adverse event comparisons to form a ground truth set. During our corresponding methods evaluation, we are able to use that reference set to demonstrate that both propensity score matching and inverse propensity score weighting can produce estimates that have high concordance with clinical trial results and substantially outperform an unadjusted baseline.

CONCLUSIONS

We find that TrialProbe is an effective approach for probing non-experimental study methods, being able to generate large ground truth sets that are able to distinguish how well non-experimental methods perform in real world observational data.

Venous Thromboembolism in Patients With Bullous Pemphigoid

Importance

Studies have linked bullous pemphigoid (BP) with venous thromboembolism (VTE) across several data sources finding 6-fold to 15-fold increased incidence rates.

Objective

To determine the incidence of VTE in patients with BP compared with similar controls.

Design, Setting, and Participants

This cohort study used insurance claims data from a nationwide US health care database from January 1, 2004, through January 1, 2020. Patients with dermatologist-recorded BP were identified (≥2 diagnoses of BP [International Classification of Diseases, Ninth Revision (ICD-9) 694.5 and ICD-10 L12.0] recorded by dermatologists within 1 year). Risk-set sampling identified comparator patients without BP and free of other chronic inflammatory skin diseases. Patients were followed-up until the first of the following events occurred: VTE, death, disenrollment, or end of data stream.

Exposures

Patients with BP compared with those without BP and free of other chronic inflammatory skin diseases (CISD).

Main outcome

Venous thromboembolism events were identified and incidence rates were computed before and after propensity-score (PS) matching to account for VTE risk factors. Hazard ratios (HRs) compared the incidence of VTE in BP vs non-CISD.

Results

Overall, 2654 patients with BP and 26 814 comparator patients without BP or another CISD were identified. The mean (SD) age in the BP group was 73.0 (12.6) years and 55.0 (18.9) years in the non-CSID group. With a median follow-up time was 2 years, the unadjusted incidence rate (per 1000 person-years) of outpatient or inpatient VTE was 8.5 in the BP group compared with 1.8 in patients without a CISD. Adjusted rates were 6.7 in the BP group compared with 3.0 in the non-CISD group. Age-specific adjusted incidence rates (per 1000 person-years) in patients aged 50 to 74 years was 6.0 (vs 2.9 in the non-CISD group) and in those aged 75 years or older was 7.1 (vs 4.53 in the non-CISD group). After 1:1 propensity-score matching including 60 VTE risk factors and severity markers, BP was associated with a 2-fold increased risk of VTE (2.24 [1.26-3.98]) vs those in the non-CISD group. When restricting to patients aged 50 years or older, the adjusted relative risk of VTE was 1.82 (1.05-3.16) for the BP vs non-CISD groups.

Conclusions

In this nationwide US cohort study, BP was associated with a 2-fold increased incidence of VTE after controlling for VTE risk factors in a dermatology patient population.

Evaluation of reporting quality of cohort studies using real-world data based on RECORD: systematic review

OBJECTIVE

Real-world data (RWD) and real-world evidence (RWE) have been paid more and more attention in recent years. We aimed to evaluate the reporting quality of cohort studies using real-world data (RWD) published between 2013 and 2021 and analyze the possible factors.

METHODS

We conducted a comprehensive search in Medline and Embase through the OVID interface for cohort studies published from 2013 to 2021 on April 29, 2022. Studies aimed at comparing the effectiveness or safety of exposure factors in the real-world setting were included. The evaluation was based on the REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement. Agreement for inclusion and evaluation was calculated using Cohen's kappa. Pearson chi-square test or Fisher's exact test and Mann-Whitney U test were used to analyze the possible factors, including the release of RECORD, journal IFs, and article citations. Bonferroni's correction was conducted for multiple comparisons. Interrupted time series analysis was performed to display the changes in report quality over time.

RESULTS

187 articles were finally included. The mean ± SD of the percentage of adequately reported items in the 187 articles was 44.7 ± 14.3 with a range of 11.1-87%. Of 23 items, the adequate reporting rate of 10 items reached 50%, and the reporting rate of some vital items was inadequate. After Bonferroni's correction, the reporting of only one item significantly improved after the release of RECORD and there was no significant improvement in the overall report quality. For interrupted time series analysis, there were no significant changes in the slope (p = 0.42) and level (p = 0.12) of adequate reporting rate. The journal IFs and citations were respectively related to 2 areas and the former significantly higher in high-reporting quality articles.

CONCLUSION

The endorsement of the RECORD cheklist was generally inadequate in cohort studies using RWD and has not improved in recent years. We encourage researchers to endorse relevant guidelines when utilizing RWD for research.

The IHI Rochester Report 2022 on Healthcare Informatics Research: Resuming After the CoViD-19

In 2020, the CoViD-19 pandemic spread worldwide in an unexpected way and suddenly modified many life issues, including social habits, social relationships, teaching modalities, and more. Such changes were also observable in many different healthcare and medical contexts. Moreover, the CoViD-19 pandemic acted as a stress test for many research endeavors, and revealed some limitations, especially in contexts where research results had an immediate impact on the social and healthcare habits of millions of people. As a result, the research community is called to perform a deep analysis of the steps already taken, and to re-think steps for the near and far future to capitalize on the lessons learned due to the pandemic. In this direction, on June 09th-11th, 2022, a group of twelve healthcare informatics researchers met in Rochester, MN, USA. This meeting was initiated by the Institute for Healthcare Informatics-IHI, and hosted by the Mayo Clinic. The goal of the meeting was to discuss and propose a research agenda for biomedical and health informatics for the next decade, in light of the changes and the lessons learned from the CoViD-19 pandemic. This article reports the main topics discussed and the conclusions reached. The intended readers of this paper, besides the biomedical and health informatics research community, are all those stakeholders in academia, industry, and government, who could benefit from the new research findings in biomedical and health informatics research. Indeed, research directions and social and policy implications are the main focus of the research agenda we propose, according to three levels: the care of individuals, the healthcare system view, and the population view.

Strategies to Address Current Challenges in Real-World Evidence Generation in Japan

The generation of real-world evidence (RWE), which describes patient characteristics or treatment patterns using real-world data (RWD), is rapidly growing more popular as a tool for decision-making in Japan. The aim of this review was to summarize challenges to RWE generation in Japan related to pharmacoepidemiology, and to propose strategies to address some of these challenges. We first focused on data-related issues, including the lack of transparency of RWD sources, linkage across different care settings, definitions of clinical outcomes, and the overall assessment framework of RWD when used for research purposes. Next the study reviewed methodology-related challenges. As lack of design transparency impairs study reproducibility, transparent reporting of study design is critical for stakeholders. For this review, we considered different sources of biases and time-varying confounding, along with potential study design and methodological solutions. Additionally, the implementation of robust assessment of definition uncertainty, misclassification, and unmeasured confounders would enhance RWE credibility in light of RWD source-related limitations, and is being strongly considered by task forces in Japan. Overall, the development of guidance for best practices on data source selection, design transparency, and analytical methods to address different sources of biases and robustness in the process of RWE generation will enhance credibility for stakeholders and local decision-makers.

R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 11

In this latest update we highlight a study from the REPEAT initiative that evaluates the reproducibility of real-world data studies, the publication of the HARPER Protocol Template developed by a joint ISPE/ISPOR taskforce, and discuss recent US FDA guidance on external control arms.

An overview of methodological flaws of real-world studies investigating drug safety in the post-marketing setting

INTRODUCTION

The evaluation of the post-marketing safety profile of drugs is a continuous monitoring process for approved and marketed medicines and it is crucial for detecting new adverse drug reactions. As such, real-world studies are essential to complement pre-marketing evidence with information concerning drug risk-benefit profile and use in wider patient populations and they have a great potential to support post-marketing drug safety evaluations.

AREAS COVERED

A detailed description of the main limitations of real-world data sources (i.e. claims databases, electronic healthcare records, drug/disease registers and spontaneous reporting system databases) and of the main methodological challenges of real-world studies in generating real-world evidence is provided.

EXPERT OPINION

Real-world evidence biases can be ascribed to both the methodological approach and the specific limitations of the different real-world data sources used to carry out the study. As such, it is crucial to characterize the quality of real-world data, by establishing guidelines and best practices for the assessment of data fitness for purpose. On the other hand, it is important that real-world studies are conducted using a rigorous methodology, aimed at minimizing the risk of bias.