Assessing Covariate Balance with Small Sample Sizes

Propensity score adjustment addresses confounding by balancing covariates in subject treatment groups through matching, stratification, inverse probability weighting, etc. Diagnostics ensure that the adjustment has been effective. A common technique is to check whether the standardized mean difference for each relevant covariate is less than a threshold like 0.1. For small sample sizes, the probability of falsely rejecting the validity of a study because of chance imbalance when no underlying balance exists approaches 1. We propose an alternative diagnostic that checks whether the standardized mean difference statistically significantly exceeds the threshold. Through simulation and real-world data, we find that this diagnostic achieves a better trade-off of type 1 error rate and power than standard nominal threshold tests and not testing for sample sizes from 250 to 4000 and for 20 to 100,000 covariates. In network studies, meta-analysis of effect estimates must be accompanied by meta-analysis of the diagnostics or else systematic confounding may overwhelm the estimated effect. Our procedure for statistically testing balance at both the database level and the meta-analysis level achieves the best balance of type-1 error rate and power. Our procedure supports the review of large numbers of covariates, enabling more rigorous diagnostics.

An open source knowledge graph ecosystem for the life sciences

Translational research requires data at multiple scales of biological organization. Advancements in sequencing and multi-omics technologies have increased the availability of these data, but researchers face significant integration challenges. Knowledge graphs (KGs) are used to model complex phenomena, and methods exist to construct them automatically. However, tackling complex biomedical integration problems requires flexibility in the way knowledge is modeled. Moreover, existing KG construction methods provide robust tooling at the cost of fixed or limited choices among knowledge representation models. PheKnowLator (Phenotype Knowledge Translator) is a semantic ecosystem for automating the FAIR (Findable, Accessible, Interoperable, and Reusable) construction of ontologically grounded KGs with fully customizable knowledge representation. The ecosystem includes KG construction resources (e.g., data preparation APIs), analysis tools (e.g., SPARQL endpoint resources and abstraction algorithms), and benchmarks (e.g., prebuilt KGs). We evaluated the ecosystem by systematically comparing it to existing open-source KG construction methods and by analyzing its computational performance when used to construct 12 different large-scale KGs. With flexible knowledge representation, PheKnowLator enables fully customizable KGs without compromising performance or usability.

Similar Risk of Kidney Failure among Patients with Blinding Diseases Who Receive Ranibizumab, Aflibercept, and Bevacizumab: An Observational Health Data Sciences and Informatics Network Study

PURPOSE

To characterize the incidence of kidney failure associated with intravitreal anti-VEGF exposure; and compare the risk of kidney failure in patients treated with ranibizumab, aflibercept, or bevacizumab.

DESIGN

Retrospective cohort study across 12 databases in the Observational Health Data Sciences and Informatics (OHDSI) network.

SUBJECTS

Subjects aged ≥ 18 years with ≥ 3 monthly intravitreal anti-VEGF medications for a blinding disease (diabetic retinopathy, diabetic macular edema, exudative age-related macular degeneration, or retinal vein occlusion).

METHODS

The standardized incidence proportions and rates of kidney failure while on treatment with anti-VEGF were calculated. For each comparison (e.g., aflibercept versus ranibizumab), patients from each group were matched 1:1 using propensity scores. Cox proportional hazards models were used to estimate the risk of kidney failure while on treatment. A random effects meta-analysis was performed to combine each database's hazard ratio (HR) estimate into a single network-wide estimate.

MAIN OUTCOME MEASURES

Incidence of kidney failure while on anti-VEGF treatment, and time from cohort entry to kidney failure.

RESULTS

Of the 6.1 million patients with blinding diseases, 37 189 who received ranibizumab, 39 447 aflibercept, and 163 611 bevacizumab were included; the total treatment exposure time was 161 724 person-years. The average standardized incidence proportion of kidney failure was 678 per 100 000 persons (range, 0-2389), and incidence rate 742 per 100 000 person-years (range, 0-2661). The meta-analysis HR of kidney failure comparing aflibercept with ranibizumab was 1.01 (95% confidence interval [CI], 0.70-1.47; P = 0.45), ranibizumab with bevacizumab 0.95 (95% CI, 0.68-1.32; P = 0.62), and aflibercept with bevacizumab 0.95 (95% CI, 0.65-1.39; P = 0.60).

CONCLUSIONS

There was no substantially different relative risk of kidney failure between those who received ranibizumab, bevacizumab, or aflibercept. Practicing ophthalmologists and nephrologists should be aware of the risk of kidney failure among patients receiving intravitreal anti-VEGF medications and that there is little empirical evidence to preferentially choose among the specific intravitreal anti-VEGF agents.

FINANCIAL DISCLOSURES

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Comparative Effectiveness of Second-line Antihyperglycemic Agents for Cardiovascular Outcomes: A Large-scale, Multinational, Federated Analysis of the LEGEND-T2DM Study

Background

SGLT2 inhibitors (SGLT2is) and GLP-1 receptor agonists (GLP1-RAs) reduce major adverse cardiovascular events (MACE) in patients with type 2 diabetes mellitus (T2DM). However, their effectiveness relative to each other and other second-line antihyperglycemic agents is unknown, without any major ongoing head-to-head trials.

Methods

Across the LEGEND-T2DM network, we included ten federated international data sources, spanning 1992-2021. We identified 1,492,855 patients with T2DM and established cardiovascular disease (CVD) on metformin monotherapy who initiated one of four second-line agents (SGLT2is, GLP1-RAs, dipeptidyl peptidase 4 inhibitor [DPP4is], sulfonylureas [SUs]). We used large-scale propensity score models to conduct an active comparator, target trial emulation for pairwise comparisons. After evaluating empirical equipoise and population generalizability, we fit on-treatment Cox proportional hazard models for 3-point MACE (myocardial infarction, stroke, death) and 4-point MACE (3-point MACE + heart failure hospitalization) risk, and combined hazard ratio (HR) estimates in a random-effects meta-analysis.

Findings

Across cohorts, 16·4%, 8·3%, 27·7%, and 47·6% of individuals with T2DM initiated SGLT2is, GLP1-RAs, DPP4is, and SUs, respectively. Over 5·2 million patient-years of follow-up and 489 million patient-days of time at-risk, there were 25,982 3-point MACE and 41,447 4-point MACE events. SGLT2is and GLP1-RAs were associated with a lower risk for 3-point MACE compared with DPP4is (HR 0·89 [95% CI, 0·79-1·00] and 0·83 [0·70-0·98]), and SUs (HR 0·76 [0·65-0·89] and 0·71 [0·59-0·86]). DPP4is were associated with a lower 3-point MACE risk versus SUs (HR 0·87 [0·79-0·95]). The pattern was consistent for 4-point MACE for the comparisons above. There were no significant differences between SGLT2is and GLP1-RAs for 3-point or 4-point MACE (HR 1·06 [0·96-1·17] and 1·05 [0·97-1·13]).

Interpretation

In patients with T2DM and established CVD, we found comparable cardiovascular risk reduction with SGLT2is and GLP1-RAs, with both agents more effective than DPP4is, which in turn were more effective than SUs. These findings suggest that the use of GLP1-RAs and SGLT2is should be prioritized as second-line agents in those with established CVD.

Funding

National Institutes of Health, United States Department of Veterans Affairs.

Bayesian safety surveillance with adaptive bias correction

Postmarket safety surveillance is an integral part of mass vaccination programs. Typically relying on sequential analysis of real-world health data as they accrue, safety surveillance is challenged by sequential multiple testing and by biases induced by residual confounding in observational data. The current standard approach based on the maximized sequential probability ratio test (MaxSPRT) fails to satisfactorily address these practical challenges and it remains a rigid framework that requires prespecification of the surveillance schedule. We develop an alternative Bayesian surveillance procedure that addresses both aforementioned challenges using a more flexible framework. To mitigate bias, we jointly analyze a large set of negative control outcomes that are adverse events with no known association with the vaccines in order to inform an empirical bias distribution, which we then incorporate into estimating the effect of vaccine exposure on the adverse event of interest through a Bayesian hierarchical model. To address multiple testing and improve on flexibility, at each analysis timepoint, we update a posterior probability in favor of the alternative hypothesis that vaccination induces higher risks of adverse events, and then use it for sequential detection of safety signals. Through an empirical evaluation using six US observational healthcare databases covering more than 360 million patients, we benchmark the proposed procedure against MaxSPRT on testing errors and estimation accuracy, under two epidemiological designs, the historical comparator and the self-controlled case series. We demonstrate that our procedure substantially reduces Type 1 error rates, maintains high statistical power and fast signal detection, and provides considerably more accurate estimation than MaxSPRT. Given the extensiveness of the empirical study which yields more than 7 million sets of results, we present all results in a public R ShinyApp. As an effort to promote open science, we provide full implementation of our method in the open-source R package EvidenceSynthesis.

Scalable and interpretable alternative to chart review for phenotype evaluation using standardized structured data from electronic health records

OBJECTIVES

Chart review as the current gold standard for phenotype evaluation cannot support observational research on electronic health records and claims data sources at scale. We aimed to evaluate the ability of structured data to support efficient and interpretable phenotype evaluation as an alternative to chart review.

MATERIALS AND METHODS

We developed Knowledge-Enhanced Electronic Profile Review (KEEPER) as a phenotype evaluation tool that extracts patient's structured data elements relevant to a phenotype and presents them in a standardized fashion following clinical reasoning principles. We evaluated its performance (interrater agreement, intermethod agreement, accuracy, and review time) compared to manual chart review for 4 conditions using randomized 2-period, 2-sequence crossover design.

RESULTS

Case ascertainment with KEEPER was twice as fast compared to manual chart review. 88.1% of the patients were classified concordantly using charts and KEEPER, but agreement varied depending on the condition. Missing data and differences in interpretation accounted for most of the discrepancies. Pairs of clinicians agreed in case ascertainment in 91.2% of the cases when using KEEPER compared to 76.3% when using charts. Patient classification aligned with the gold standard in 88.1% and 86.9% of the cases respectively.

CONCLUSION

Structured data can be used for efficient and interpretable phenotype evaluation if they are limited to relevant subset and organized according to the clinical reasoning principles. A system that implements these principles can achieve noninferior performance compared to chart review at a fraction of time.

Evaluating the impact of alternative phenotype definitions on incidence rates across a global data network

Objective

Developing accurate phenotype definitions is critical in obtaining reliable and reproducible background rates in safety research. This study aims to illustrate the differences in background incidence rates by comparing definitions for a given outcome.

Materials and Methods

We used 16 data sources to systematically generate and evaluate outcomes for 13 adverse events and their overall background rates. We examined the effect of different modifications (inpatient setting, standardization of code set, and code set changes) to the computable phenotype on background incidence rates.

Results

Rate ratios (RRs) of the incidence rates from each computable phenotype definition varied across outcomes, with inpatient restriction showing the highest variation from 1 to 11.93. Standardization of code set RRs ranges from 1 to 1.64, and code set changes range from 1 to 2.52.

Discussion

The modification that has the highest impact is requiring inpatient place of service, leading to at least a 2-fold higher incidence rate in the base definition. Standardization showed almost no change when using source code variations. The strength of the effect in the inpatient restriction is highly dependent on the outcome. Changing definitions from broad to narrow showed the most variability by age/gender/database across phenotypes and less than a 2-fold increase in rate compared to the base definition.

Conclusion

Characterization of outcomes across a network of databases yields insights into sensitivity and specificity trade-offs when definitions are altered. Outcomes should be thoroughly evaluated prior to use for background rates for their plausibility for use across a global network.

Development and evaluation of an algorithm to link mothers and infants in two US commercial healthcare claims databases for pharmacoepidemiology research

BACKGROUND

Administrative healthcare claims databases are used in drug safety research but are limited for investigating the impacts of prenatal exposures on neonatal and pediatric outcomes without mother-infant pair identification. Further, existing algorithms are not transportable across data sources. We developed a transportable mother-infant linkage algorithm and evaluated it in two, large US commercially insured populations.

METHODS

We used two US commercial health insurance claims databases during the years 2000 to 2021. Mother-infant links were constructed where persons of female sex 12-55 years of age with a pregnancy episode ending in live birth were associated with a person who was 0 years of age at database entry, who shared a common insurance plan ID, had overlapping insurance coverage time, and whose date of birth was within ± 60-days of the mother's pregnancy episode live birth date. We compared the characteristics of linked vs. non-linked mothers and infants to assess similarity.

RESULTS

The algorithm linked 3,477,960 mothers to 4,160,284 infants in the two databases. Linked mothers and linked infants comprised 73.6% of all mothers and 49.1% of all infants, respectively. 94.9% of linked infants' dates of birth were within ± 30-days of the associated mother's pregnancy episode end dates. Characteristics were largely similar in linked vs. non-linked mothers and infants. Differences included that linked mothers were older, had longer pregnancy episodes, and had greater post-pregnancy observation time than mothers with live births who were not linked. Linked infants had less observation time and greater healthcare utilization than non-linked infants.

CONCLUSIONS

We developed a mother-infant linkage algorithm and applied it to two US commercial healthcare claims databases that achieved a high linkage proportion and demonstrated that linked and non-linked mother and infant cohorts were similar. Transparent, reusable algorithms applied to large databases enable large-scale research on exposures during pregnancy and pediatric outcomes with relevance to drug safety. These features suggest studies using this algorithm can produce valid and generalizable evidence to inform clinical, policy, and regulatory decisions.

Multinational patterns of second line antihyperglycaemic drug initiation across cardiovascular risk groups: federated pharmacoepidemiological evaluation in LEGEND-T2DM

Objective

To assess the uptake of second line antihyperglycaemic drugs among patients with type 2 diabetes mellitus who are receiving metformin.

Design

Federated pharmacoepidemiological evaluation in LEGEND-T2DM.

Setting

10 US and seven non-US electronic health record and administrative claims databases in the Observational Health Data Sciences and Informatics network in eight countries from 2011 to the end of 2021.

Participants

4.8 million patients (≥18 years) across US and non-US based databases with type 2 diabetes mellitus who had received metformin monotherapy and had initiated second line treatments.

Exposure

The exposure used to evaluate each database was calendar year trends, with the years in the study that were specific to each cohort.

Main outcomes measures

The outcome was the incidence of second line antihyperglycaemic drug use (ie, glucagon-like peptide-1 receptor agonists, sodium-glucose cotransporter-2 inhibitors, dipeptidyl peptidase-4 inhibitors, and sulfonylureas) among individuals who were already receiving treatment with metformin. The relative drug class level uptake across cardiovascular risk groups was also evaluated.

Results

4.6 million patients were identified in US databases, 61 382 from Spain, 32 442 from Germany, 25 173 from the UK, 13 270 from France, 5580 from Scotland, 4614 from Hong Kong, and 2322 from Australia. During 2011-21, the combined proportional initiation of the cardioprotective antihyperglycaemic drugs (glucagon-like peptide-1 receptor agonists and sodium-glucose cotransporter-2 inhibitors) increased across all data sources, with the combined initiation of these drugs as second line drugs in 2021 ranging from 35.2% to 68.2% in the US databases, 15.4% in France, 34.7% in Spain, 50.1% in Germany, and 54.8% in Scotland. From 2016 to 2021, in some US and non-US databases, uptake of glucagon-like peptide-1 receptor agonists and sodium-glucose cotransporter-2 inhibitors increased more significantly among populations with no cardiovascular disease compared with patients with established cardiovascular disease. No data source provided evidence of a greater increase in the uptake of these two drug classes in populations with cardiovascular disease compared with no cardiovascular disease.

Conclusions

Despite the increase in overall uptake of cardioprotective antihyperglycaemic drugs as second line treatments for type 2 diabetes mellitus, their uptake was lower in patients with cardiovascular disease than in people with no cardiovascular disease over the past decade. A strategy is needed to ensure that medication use is concordant with guideline recommendations to improve outcomes of patients with type 2 diabetes mellitus.

Serially Combining Epidemiological Designs Does Not Improve Overall Signal Detection in Vaccine Safety Surveillance

INTRODUCTION

Vaccine safety surveillance commonly includes a serial testing approach with a sensitive method for 'signal generation' and specific method for 'signal validation.' The extent to which serial testing in real-world studies improves or hinders overall performance in terms of sensitivity and specificity remains unknown.

METHODS

We assessed the overall performance of serial testing using three administrative claims and one electronic health record database. We compared type I and II errors before and after empirical calibration for historical comparator, self-controlled case series (SCCS), and the serial combination of those designs against six vaccine exposure groups with 93 negative control and 279 imputed positive control outcomes.

RESULTS

The historical comparator design mostly had fewer type II errors than SCCS. SCCS had fewer type I errors than the historical comparator. Before empirical calibration, the serial combination increased specificity and decreased sensitivity. Type II errors mostly exceeded 50%. After empirical calibration, type I errors returned to nominal; sensitivity was lowest when the methods were combined.

CONCLUSION

While serial combination produced fewer false-positive signals compared with the most specific method, it generated more false-negative signals compared with the most sensitive method. Using a historical comparator design followed by an SCCS analysis yielded decreased sensitivity in evaluating safety signals relative to a one-stage SCCS approach. While the current use of serial testing in vaccine surveillance may provide a practical paradigm for signal identification and triage, single epidemiological designs should be explored as valuable approaches to detecting signals.

Ontologizing health systems data at scale: making translational discovery a reality

Common data models solve many challenges of standardizing electronic health record (EHR) data but are unable to semantically integrate all of the resources needed for deep phenotyping. Open Biological and Biomedical Ontology (OBO) Foundry ontologies provide computable representations of biological knowledge and enable the integration of heterogeneous data. However, mapping EHR data to OBO ontologies requires significant manual curation and domain expertise. We introduce OMOP2OBO, an algorithm for mapping Observational Medical Outcomes Partnership (OMOP) vocabularies to OBO ontologies. Using OMOP2OBO, we produced mappings for 92,367 conditions, 8611 drug ingredients, and 10,673 measurement results, which covered 68-99% of concepts used in clinical practice when examined across 24 hospitals. When used to phenotype rare disease patients, the mappings helped systematically identify undiagnosed patients who might benefit from genetic testing. By aligning OMOP vocabularies to OBO ontologies our algorithm presents new opportunities to advance EHR-based deep phenotyping.

Reproducible variability: assessing investigator discordance across 9 research teams attempting to reproduce the same observational study

OBJECTIVE

Observational studies can impact patient care but must be robust and reproducible. Nonreproducibility is primarily caused by unclear reporting of design choices and analytic procedures. This study aimed to: (1) assess how the study logic described in an observational study could be interpreted by independent researchers and (2) quantify the impact of interpretations' variability on patient characteristics.

MATERIALS AND METHODS

Nine teams of highly qualified researchers reproduced a cohort from a study by Albogami et al. The teams were provided the clinical codes and access to the tools to create cohort definitions such that the only variable part was their logic choices. We executed teams' cohort definitions against the database and compared the number of subjects, patient overlap, and patient characteristics.

RESULTS

On average, the teams' interpretations fully aligned with the master implementation in 4 out of 10 inclusion criteria with at least 4 deviations per team. Cohorts' size varied from one-third of the master cohort size to 10 times the cohort size (2159-63 619 subjects compared to 6196 subjects). Median agreement was 9.4% (interquartile range 15.3-16.2%). The teams' cohorts significantly differed from the master implementation by at least 2 baseline characteristics, and most of the teams differed by at least 5.

CONCLUSIONS

Independent research teams attempting to reproduce the study based on its free-text description alone produce different implementations that vary in the population size and composition. Sharing analytical code supported by a common data model and open-source tools allows reproducing a study unambiguously thereby preserving initial design choices.

A standardized framework for risk-based assessment of treatment effect heterogeneity in observational healthcare databases

Treatment effects are often anticipated to vary across groups of patients with different baseline risk. The Predictive Approaches to Treatment Effect Heterogeneity (PATH) statement focused on baseline risk as a robust predictor of treatment effect and provided guidance on risk-based assessment of treatment effect heterogeneity in a randomized controlled trial. The aim of this study is to extend this approach to the observational setting using a standardized scalable framework. The proposed framework consists of five steps: (1) definition of the research aim, i.e., the population, the treatment, the comparator and the outcome(s) of interest; (2) identification of relevant databases; (3) development of a prediction model for the outcome(s) of interest; (4) estimation of relative and absolute treatment effect within strata of predicted risk, after adjusting for observed confounding; (5) presentation of the results. We demonstrate our framework by evaluating heterogeneity of the effect of thiazide or thiazide-like diuretics versus angiotensin-converting enzyme inhibitors on three efficacy and nine safety outcomes across three observational databases. We provide a publicly available R software package for applying this framework to any database mapped to the Observational Medical Outcomes Partnership Common Data Model. In our demonstration, patients at low risk of acute myocardial infarction receive negligible absolute benefits for all three efficacy outcomes, though they are more pronounced in the highest risk group, especially for acute myocardial infarction. Our framework allows for the evaluation of differential treatment effects across risk strata, which offers the opportunity to consider the benefit-harm trade-off between alternative treatments.

International cohort study indicates no association between alpha-1 blockers and susceptibility to COVID-19 in benign prostatic hyperplasia patients

Purpose: Alpha-1 blockers, often used to treat benign prostatic hyperplasia (BPH), have been hypothesized to prevent COVID-19 complications by minimising cytokine storm release. The proposed treatment based on this hypothesis currently lacks support from reliable real-world evidence, however. We leverage an international network of large-scale healthcare databases to generate comprehensive evidence in a transparent and reproducible manner.

Methods: In this international cohort study, we deployed electronic health records from Spain (SIDIAP) and the United States (Department of Veterans Affairs, Columbia University Irving Medical Center, IQVIA OpenClaims, Optum DOD, Optum EHR). We assessed association between alpha-1 blocker use and risks of three COVID-19 outcomes—diagnosis, hospitalization, and hospitalization requiring intensive services—using a prevalent-user active-comparator design. We estimated hazard ratios using state-of-the-art techniques to minimize potential confounding, including large-scale propensity score matching/stratification and negative control calibration. We pooled database-specific estimates through random effects meta-analysis.

Results: Our study overall included 2.6 and 0.46 million users of alpha-1 blockers and of alternative BPH medications. We observed no significant difference in their risks for any of the COVID-19 outcomes, with our meta-analytic HR estimates being 1.02 (95% CI: 0.92–1.13) for diagnosis, 1.00 (95% CI: 0.89–1.13) for hospitalization, and 1.15 (95% CI: 0.71–1.88) for hospitalization requiring intensive services.

Conclusion: We found no evidence of the hypothesized reduction in risks of the COVID-19 outcomes from the prevalent-use of alpha-1 blockers—further research is needed to identify effective therapies for this novel disease.

PheValuator 2.0: Methodological improvements for the PheValuator approach to semi-automated phenotype algorithm evaluation

PURPOSE

Phenotype algorithms are central to performing analyses using observational data. These algorithms translate the clinical idea of a health condition into an executable set of rules allowing for queries of data elements from a database. PheValuator, a software package in the Observational Health Data Sciences and Informatics (OHDSI) tool stack, provides a method to assess the performance characteristics of these algorithms, namely, sensitivity, specificity, and positive and negative predictive value. It uses machine learning to develop predictive models for determining a probabilistic gold standard of subjects for assessment of cases and non-cases of health conditions. PheValuator was developed to complement or even replace the traditional approach of algorithm validation, i.e., by expert assessment of subject records through chart review. Results in our first PheValuator paper suggest a systematic underestimation of the PPV compared to previous results using chart review. In this paper we evaluate modifications made to the method designed to improve its performance.

METHODS

The major changes to PheValuator included allowing all diagnostic conditions, clinical observations, drug prescriptions, and laboratory measurements to be included as predictors within the modeling process whereas in the prior version there were significant restrictions on the included predictors. We also have allowed for the inclusion of the temporal relationships of the predictors in the model. To evaluate the performance of the new method, we compared the results from the new and original methods against results found from the literature using traditional validation of algorithms for 19 phenotypes. We performed these tests using data from five commercial databases.

RESULTS

In the assessment aggregating all phenotype algorithms, the median difference between the PheValuator estimate and the gold standard estimate for PPV was reduced from -21 (IQR -34, -3) in Version 1.0 to 4 (IQR -3, 15) using Version 2.0. We found a median difference in specificity of 3 (IQR 1, 4.25) for Version 1.0 and 3 (IQR 1, 4) for Version 2.0. The median difference between the two versions of PheValuator and the gold standard for estimates of sensitivity was reduced from -39 (-51, -20) to -16 (-34, -6).

CONCLUSION

PheValuator 2.0 produces estimates for the performance characteristics for phenotype algorithms that are significantly closer to estimates from traditional validation through chart review compared to version 1.0. With this tool in researcher's toolkits, methods, such as quantitative bias analysis, may now be used to improve the reliability and reproducibility of research studies using observational data.

Large-scale evidence generation and evaluation across a network of databases for type 2 diabetes mellitus (LEGEND-T2DM): a protocol for a series of multinational, real-world comparative cardiovascular effectiveness and safety studies

INTRODUCTION

Therapeutic options for type 2 diabetes mellitus (T2DM) have expanded over the last decade with the emergence of cardioprotective novel agents, but without such data for older drugs, leaving a critical gap in our understanding of the relative effects of T2DM agents on cardiovascular risk.

METHODS AND ANALYSIS

The large-scale evidence generations across a network of databases for T2DM (LEGEND-T2DM) initiative is a series of systematic, large-scale, multinational, real-world comparative cardiovascular effectiveness and safety studies of all four major second-line anti-hyperglycaemic agents, including sodium-glucose co-transporter-2 inhibitor, glucagon-like peptide-1 receptor agonist, dipeptidyl peptidase-4 inhibitor and sulfonylureas. LEGEND-T2DM will leverage the Observational Health Data Sciences and Informatics (OHDSI) community that provides access to a global network of administrative claims and electronic health record data sources, representing 190 million patients in the USA and about 50 million internationally. LEGEND-T2DM will identify all adult, patients with T2DM who newly initiate a traditionally second-line T2DM agent. Using an active comparator, new-user cohort design, LEGEND-T2DM will execute all pairwise class-versus-class and drug-versus-drug comparisons in each data source, producing extensive study diagnostics that assess reliability and generalisability through cohort balance and equipoise to examine the relative risk of cardiovascular and safety outcomes. The primary cardiovascular outcomes include a composite of major adverse cardiovascular events and a series of safety outcomes. The study will pursue data-driven, large-scale propensity adjustment for measured confounding, a large set of negative control outcome experiments to address unmeasured and systematic bias.

ETHICS AND DISSEMINATION

The study ensures data safety through a federated analytic approach and follows research best practices, including prespecification and full disclosure of results. LEGEND-T2DM is dedicated to open science and transparency and will publicly share all analytic code from reproducible cohort definitions through turn-key software, enabling other research groups to leverage our methods, data and results to verify and extend our findings.

Learning patient-level prediction models across multiple healthcare databases: evaluation of ensembles for increasing model transportability

Background

Prognostic models that are accurate could help aid medical decision making. Large observational databases often contain temporal medical data for large and diverse populations of patients. It may be possible to learn prognostic models using the large observational data. Often the performance of a prognostic model undesirably worsens when transported to a different database (or into a clinical setting). In this study we investigate different ensemble approaches that combine prognostic models independently developed using different databases (a simple federated learning approach) to determine whether ensembles that combine models developed across databases can improve model transportability (perform better in new data than single database models)?

Methods

For a given prediction question we independently trained five single database models each using a different observational healthcare database. We then developed and investigated numerous ensemble models (fusion, stacking and mixture of experts) that combined the different database models. Performance of each model was investigated via discrimination and calibration using a leave one dataset out technique, i.e., hold out one database to use for validation and use the remaining four datasets for model development. The internal validation of a model developed using the hold out database was calculated and presented as the ‘internal benchmark’ for comparison.

Results

In this study the fusion ensembles generally outperformed the single database models when transported to a previously unseen database and the performances were more consistent across unseen databases. Stacking ensembles performed poorly in terms of discrimination when the labels in the unseen database were limited. Calibration was consistently poor when both ensembles and single database models were applied to previously unseen databases.

Conclusion

A simple federated learning approach that implements ensemble techniques to combine models independently developed across different databases for the same prediction question may improve the discriminative performance in new data (new database or clinical setting) but will need to be recalibrated using the new data. This could help medical decision making by improving prognostic model performance.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-022-01879-6.

Factors Influencing Background Incidence Rate Calculation: Systematic Empirical Evaluation Across an International Network of Observational Databases

Objective: Background incidence rates are routinely used in safety studies to evaluate an association of an exposure and outcome. Systematic research on sensitivity of rates to the choice of the study parameters is lacking. Materials and Methods: We used 12 data sources to systematically examine the influence of age, race, sex, database, time-at-risk, season and year, prior observation and clean window on incidence rates using 15 adverse events of special interest for COVID-19 vaccines as an example. For binary comparisons we calculated incidence rate ratios and performed random-effect meta-analysis. Results: We observed a wide variation of background rates that goes well beyond age and database effects previously observed. While rates vary up to a factor of 1,000 across age groups, even after adjusting for age and sex, the study showed residual bias due to the other parameters. Rates were highly influenced by the choice of anchoring (e.g., health visit, vaccination, or arbitrary date) for the time-at-risk start. Anchoring on a healthcare encounter yielded higher incidence comparing to a random date, especially for short time-at-risk. Incidence rates were highly influenced by the choice of the database (varying by up to a factor of 100), clean window choice and time-at-risk duration, and less so by secular or seasonal trends. Conclusion: Comparing background to observed rates requires appropriate adjustment and careful time-at-risk start and duration choice. Results should be interpreted in the context of study parameter choices.

The association between 2,4-D and serum testosterone levels: NHANES 2013-2014

BACKGROUND

Previous studies have investigated associations between herbicides such as 2,4-Dichlorophenoxyacetic acid (2,4-D) and dyshormonogenesis, specifically low testosterone, in human, rodent, and cell models, but results have been conflicting and inconclusive.

METHODS

Using data from a cross-sectional study of 456 adult men in the 2013-2014 NHANES survey cycle, we examined the relationship between urinary concentrations of 2,4-D and serum testosterone levels.

RESULTS

Multivariable regression models adjusting for potential confounders revealed a significant, negative association between urinary 2,4-D and mean serum testosterone among U.S. adult males (β =  - 11.4 ng/dL, p = 0.02). Multivariable logistic regression models using a cutoff defining abnormally low testosterone (i.e., serum testosterone < 300 ng/dL) revealed no significant associations between 2,4-D and the odds of low testosterone.

CONCLUSION

These findings expand on previous literature implicating a role for 2,4-D in the etiology of low testosterone and dyshormonogenesis. Future studies are warranted to corroborate these findings, determine clinical significance, and to investigate the proposed potential biological mechanisms underlying this association.

DLMM as a lossless one-shot algorithm for collaborative multi-site distributed linear mixed models

Linear mixed models are commonly used in healthcare-based association analyses for analyzing multi-site data with heterogeneous site-specific random effects. Due to regulations for protecting patients’ privacy, sensitive individual patient data (IPD) typically cannot be shared across sites. We propose an algorithm for fitting distributed linear mixed models (DLMMs) without sharing IPD across sites. This algorithm achieves results identical to those achieved using pooled IPD from multiple sites (i.e., the same effect size and standard error estimates), hence demonstrating the lossless property. The algorithm requires each site to contribute minimal aggregated data in only one round of communication. We demonstrate the lossless property of the proposed DLMM algorithm by investigating the associations between demographic and clinical characteristics and length of hospital stay in COVID-19 patients using administrative claims from the UnitedHealth Group Clinical Discovery Database. We extend this association study by incorporating 120,609 COVID-19 patients from 11 collaborative data sources worldwide.

A lossless, one-shot and privacy-preserving distributed algorithm was revealed for fitting linear mixed models on multi-site data. The algorithm was applied to a study of 120,609 COVID-19 patients using only minimal aggregated data from each of 14 sites.

Unraveling COVID-19: A Large-Scale Characterization of 4.5 Million COVID-19 Cases Using CHARYBDIS

Purpose

Routinely collected real world data (RWD) have great utility in aiding the novel coronavirus disease (COVID-19) pandemic response. Here we present the international Observational Health Data Sciences and Informatics (OHDSI) Characterizing Health Associated Risks and Your Baseline Disease In SARS-COV-2 (CHARYBDIS) framework for standardisation and analysis of COVID-19 RWD.

Patients and Methods

We conducted a descriptive retrospective database study using a federated network of data partners in the United States, Europe (the Netherlands, Spain, the UK, Germany, France and Italy) and Asia (South Korea and China). The study protocol and analytical package were released on 11th June 2020 and are iteratively updated via GitHub. We identified three non-mutually exclusive cohorts of 4,537,153 individuals with a clinical COVID-19 diagnosis or positive test, 886,193 hospitalized with COVID-19, and 113,627 hospitalized with COVID-19 requiring intensive services.

Results

We aggregated over 22,000 unique characteristics describing patients with COVID-19. All comorbidities, symptoms, medications, and outcomes are described by cohort in aggregate counts and are readily available online. Globally, we observed similarities in the USA and Europe: more women diagnosed than men but more men hospitalized than women, most diagnosed cases between 25 and 60 years of age versus most hospitalized cases between 60 and 80 years of age. South Korea differed with more women than men hospitalized. Common comorbidities included type 2 diabetes, hypertension, chronic kidney disease and heart disease. Common presenting symptoms were dyspnea, cough and fever. Symptom data availability was more common in hospitalized cohorts than diagnosed.

Conclusion

We constructed a global, multi-centre view to describe trends in COVID-19 progression, management and evolution over time. By characterising baseline variability in patients and geography, our work provides critical context that may otherwise be misconstrued as data quality issues. This is important as we perform studies on adverse events of special interest in COVID-19 vaccine surveillance.

Seek COVER: using a disease proxy to rapidly develop and validate a personalized risk calculator for COVID-19 outcomes in an international network

Background

We investigated whether we could use influenza data to develop prediction models for COVID-19 to increase the speed at which prediction models can reliably be developed and validated early in a pandemic. We developed COVID-19 Estimated Risk (COVER) scores that quantify a patient’s risk of hospital admission with pneumonia (COVER-H), hospitalization with pneumonia requiring intensive services or death (COVER-I), or fatality (COVER-F) in the 30-days following COVID-19 diagnosis using historical data from patients with influenza or flu-like symptoms and tested this in COVID-19 patients.

Methods

We analyzed a federated network of electronic medical records and administrative claims data from 14 data sources and 6 countries containing data collected on or before 4/27/2020. We used a 2-step process to develop 3 scores using historical data from patients with influenza or flu-like symptoms any time prior to 2020. The first step was to create a data-driven model using LASSO regularized logistic regression, the covariates of which were used to develop aggregate covariates for the second step where the COVER scores were developed using a smaller set of features. These 3 COVER scores were then externally validated on patients with 1) influenza or flu-like symptoms and 2) confirmed or suspected COVID-19 diagnosis across 5 databases from South Korea, Spain, and the United States. Outcomes included i) hospitalization with pneumonia, ii) hospitalization with pneumonia requiring intensive services or death, and iii) death in the 30 days after index date.

Results

Overall, 44,507 COVID-19 patients were included for model validation. We identified 7 predictors (history of cancer, chronic obstructive pulmonary disease, diabetes, heart disease, hypertension, hyperlipidemia, kidney disease) which combined with age and sex discriminated which patients would experience any of our three outcomes. The models achieved good performance in influenza and COVID-19 cohorts. For COVID-19 the AUC ranges were, COVER-H: 0.69–0.81, COVER-I: 0.73–0.91, and COVER-F: 0.72–0.90. Calibration varied across the validations with some of the COVID-19 validations being less well calibrated than the influenza validations.

Conclusions

This research demonstrated the utility of using a proxy disease to develop a prediction model. The 3 COVER models with 9-predictors that were developed using influenza data perform well for COVID-19 patients for predicting hospitalization, intensive services, and fatality. The scores showed good discriminatory performance which transferred well to the COVID-19 population. There was some miscalibration in the COVID-19 validations, which is potentially due to the difference in symptom severity between the two diseases. A possible solution for this is to recalibrate the models in each location before use.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01505-z.

90-Day all-cause mortality can be predicted following a total knee replacement: an international, network study to develop and validate a prediction model

PURPOSE

The purpose of this study was to develop and validate a prediction model for 90-day mortality following a total knee replacement (TKR). TKR is a safe and cost-effective surgical procedure for treating severe knee osteoarthritis (OA). Although complications following surgery are rare, prediction tools could help identify high-risk patients who could be targeted with preventative interventions. The aim was to develop and validate a simple model to help inform treatment choices.

METHODS

A mortality prediction model for knee OA patients following TKR was developed and externally validated using a US claims database and a UK general practice database. The target population consisted of patients undergoing a primary TKR for knee OA, aged ≥ 40 years and registered for ≥ 1 year before surgery. LASSO logistic regression models were developed for post-operative (90-day) mortality. A second mortality model was developed with a reduced feature set to increase interpretability and usability.

RESULTS

A total of 193,615 patients were included, with 40,950 in The Health Improvement Network (THIN) database and 152,665 in Optum. The full model predicting 90-day mortality yielded AUROC of 0.78 when trained in OPTUM and 0.70 when externally validated on THIN. The 12 variable model achieved internal AUROC of 0.77 and external AUROC of 0.71 in THIN.

CONCLUSIONS

A simple prediction model based on sex, age, and 10 comorbidities that can identify patients at high risk of short-term mortality following TKR was developed that demonstrated good, robust performance. The 12-feature mortality model is easily implemented and the performance suggests it could be used to inform evidence based shared decision-making prior to surgery and targeting prophylaxis for those at high risk.

LEVEL OF EVIDENCE

III.

Using Iterative Pairwise External Validation to Contextualize Prediction Model Performance: A Use Case Predicting 1-Year Heart Failure Risk in Patients with Diabetes Across Five Data Sources

INTRODUCTION

External validation of prediction models is increasingly being seen as a minimum requirement for acceptance in clinical practice. However, the lack of interoperability of healthcare databases has been the biggest barrier to this occurring on a large scale. Recent improvements in database interoperability enable a standardized analytical framework for model development and external validation. External validation of a model in a new database lacks context, whereby the external validation can be compared with a benchmark in this database. Iterative pairwise external validation (IPEV) is a framework that uses a rotating model development and validation approach to contextualize the assessment of performance across a network of databases. As a use case, we predicted 1-year risk of heart failure in patients with type 2 diabetes mellitus.

METHODS

The method follows a two-step process involving (1) development of baseline and data-driven models in each database according to best practices and (2) validation of these models across the remaining databases. We introduce a heatmap visualization that supports the assessment of the internal and external model performance in all available databases. As a use case, we developed and validated models to predict 1-year risk of heart failure in patients initializing a second pharmacological intervention for type 2 diabetes mellitus. We leveraged the power of the Observational Medical Outcomes Partnership common data model to create an open-source software package to increase the consistency, speed, and transparency of this process.

RESULTS

A total of 403,187 patients from five databases were included in the study. We developed five models that, when assessed internally, had a discriminative performance ranging from 0.73 to 0.81 area under the receiver operating characteristic curve with acceptable calibration. When we externally validated these models in a new database, three models achieved consistent performance and in context often performed similarly to models developed in the database itself. The visualization of IPEV provided valuable insights. From this, we identified the model developed in the Commercial Claims and Encounters (CCAE) database as the best performing model overall.

CONCLUSION

Using IPEV lends weight to the model development process. The rotation of development through multiple databases provides context to model assessment, leading to improved understanding of transportability and generalizability. The inclusion of a baseline model in all modelling steps provides further context to the performance gains of increasing model complexity. The CCAE model was identified as a candidate for clinical use. The use case demonstrates that IPEV provides a huge opportunity in a new era of standardised data and analytics to improve insight into and trust in prediction models at an unprecedented scale.

Predictors of diagnostic transition from major depressive disorder to bipolar disorder: a retrospective observational network study

Many patients with bipolar disorder (BD) are initially misdiagnosed with major depressive disorder (MDD) and are treated with antidepressants, whose potential iatrogenic effects are widely discussed. It is unknown whether MDD is a comorbidity of BD or its earlier stage, and no consensus exists on individual conversion predictors, delaying BD's timely recognition and treatment. We aimed to build a predictive model of MDD to BD conversion and to validate it across a multi-national network of patient databases using the standardization afforded by the Observational Medical Outcomes Partnership (OMOP) common data model. Five "training" US databases were retrospectively analyzed: IBM MarketScan CCAE, MDCR, MDCD, Optum EHR, and Optum Claims. Cyclops regularized logistic regression models were developed on one-year MDD-BD conversion with all standard covariates from the HADES PatientLevelPrediction package. Time-to-conversion Kaplan-Meier analysis was performed up to a decade after MDD, stratified by model-estimated risk. External validation of the final prediction model was performed across 9 patient record databases within the Observational Health Data Sciences and Informatics (OHDSI) network internationally. The model's area under the curve (AUC) varied 0.633-0.745 (µ = 0.689) across the five US training databases. Nine variables predicted one-year MDD-BD transition. Factors that increased risk were: younger age, severe depression, psychosis, anxiety, substance misuse, self-harm thoughts/actions, and prior mental disorder. AUCs of the validation datasets ranged 0.570-0.785 (µ = 0.664). An assessment algorithm was built for MDD to BD conversion that allows distinguishing as much as 100-fold risk differences among patients and validates well across multiple international data sources.

Bias, Precision and Timeliness of Historical (Background) Rate Comparison Methods for Vaccine Safety Monitoring: An Empirical Multi-Database Analysis

Using real-world data and past vaccination data, we conducted a large-scale experiment to quantify bias, precision and timeliness of different study designs to estimate historical background (expected) compared to post-vaccination (observed) rates of safety events for several vaccines. We used negative (not causally related) and positive control outcomes. The latter were synthetically generated true safety signals with incident rate ratios ranging from 1.5 to 4. Observed vs. expected analysis using within-database historical background rates is a sensitive but unspecific method for the identification of potential vaccine safety signals. Despite good discrimination, most analyses showed a tendency to overestimate risks, with 20%-100% type 1 error, but low (0% to 20%) type 2 error in the large databases included in our study. Efforts to improve the comparability of background and post-vaccine rates, including age-sex adjustment and anchoring background rates around a visit, reduced type 1 error and improved precision but residual systematic error persisted. Additionally, empirical calibration dramatically reduced type 1 to nominal but came at the cost of increasing type 2 error.

COVID-19 in patients with autoimmune diseases: characteristics and outcomes in a multinational network of cohorts across three countries

OBJECTIVE

Patients with autoimmune diseases were advised to shield to avoid coronavirus disease 2019 (COVID-19), but information on their prognosis is lacking. We characterized 30-day outcomes and mortality after hospitalization with COVID-19 among patients with prevalent autoimmune diseases, and compared outcomes after hospital admissions among similar patients with seasonal influenza.

METHODS

A multinational network cohort study was conducted using electronic health records data from Columbia University Irving Medical Center [USA, Optum (USA), Department of Veterans Affairs (USA), Information System for Research in Primary Care-Hospitalization Linked Data (Spain) and claims data from IQVIA Open Claims (USA) and Health Insurance and Review Assessment (South Korea). All patients with prevalent autoimmune diseases, diagnosed and/or hospitalized between January and June 2020 with COVID-19, and similar patients hospitalized with influenza in 2017-18 were included. Outcomes were death and complications within 30 days of hospitalization.

RESULTS

We studied 133 589 patients diagnosed and 48 418 hospitalized with COVID-19 with prevalent autoimmune diseases. Most patients were female, aged ≥50 years with previous comorbidities. The prevalence of hypertension (45.5-93.2%), chronic kidney disease (14.0-52.7%) and heart disease (29.0-83.8%) was higher in hospitalized vs diagnosed patients with COVID-19. Compared with 70 660 hospitalized with influenza, those admitted with COVID-19 had more respiratory complications including pneumonia and acute respiratory distress syndrome, and higher 30-day mortality (2.2-4.3% vs 6.32-24.6%).

CONCLUSION

Compared with influenza, COVID-19 is a more severe disease, leading to more complications and higher mortality.

Characteristics and Outcomes of Over 300,000 Patients with COVID-19 and History of Cancer in the United States and Spain

BACKGROUND

We described the demographics, cancer subtypes, comorbidities, and outcomes of patients with a history of cancer and coronavirus disease 2019 (COVID-19). Second, we compared patients hospitalized with COVID-19 to patients diagnosed with COVID-19 and patients hospitalized with influenza.

METHODS

We conducted a cohort study using eight routinely collected health care databases from Spain and the United States, standardized to the Observational Medical Outcome Partnership common data model. Three cohorts of patients with a history of cancer were included: (i) diagnosed with COVID-19, (ii) hospitalized with COVID-19, and (iii) hospitalized with influenza in 2017 to 2018. Patients were followed from index date to 30 days or death. We reported demographics, cancer subtypes, comorbidities, and 30-day outcomes.

RESULTS

We included 366,050 and 119,597 patients diagnosed and hospitalized with COVID-19, respectively. Prostate and breast cancers were the most frequent cancers (range: 5%-18% and 1%-14% in the diagnosed cohort, respectively). Hematologic malignancies were also frequent, with non-Hodgkin's lymphoma being among the five most common cancer subtypes in the diagnosed cohort. Overall, patients were aged above 65 years and had multiple comorbidities. Occurrence of death ranged from 2% to 14% and from 6% to 26% in the diagnosed and hospitalized COVID-19 cohorts, respectively. Patients hospitalized with influenza (n = 67,743) had a similar distribution of cancer subtypes, sex, age, and comorbidities but lower occurrence of adverse events.

CONCLUSIONS

Patients with a history of cancer and COVID-19 had multiple comorbidities and a high occurrence of COVID-19-related events. Hematologic malignancies were frequent.

IMPACT

This study provides epidemiologic characteristics that can inform clinical care and etiologic studies.

A systematic assessment of the epidemiologic literature regarding an association between acetaminophen exposure and cancer

Introduced in the 1950s, acetaminophen is one of the most widely used antipyretics and analgesics worldwide. In 1999, the International Agency for Research on Cancer (IARC) reviewed the epidemiologic studies of acetaminophen and the data were judged to be "inadequate" to conclude that it is carcinogenic. In 2019 the California Office of Environmental Health Hazard Assessment initiated a review process on the carcinogenic hazard potential of acetaminophen. To inform this review process, the authors performed a comprehensive literature search and identified 136 epidemiologic studies, which for most cancer types suggest no alteration in risk associated with acetaminophen use. For 3 cancer types, renal cell, liver, and some forms of lymphohematopoietic, some studies suggest an increased risk; however, multiple factors unique to acetaminophen need to be considered to determine if these results are real and clinically meaningful. The objective of this publication is to analyze the results of these epidemiologic studies using a framework that accounts for the inherent challenge of evaluating acetaminophen, including, broad population-wide use in multiple disease states, challenges with exposure measurement, protopathic bias, channeling bias, and recall bias. When evaluated using this framework, the data do not support a causal association between acetaminophen use and cancer.

From clinical trials to clinical practice: How long are drugs tested and then used by patients?

Objective

Evidence is scarce regarding the safety of long-term drug use, especially for drugs treating chronic diseases. To bridge this knowledge gap, this research investigated the differences in drug exposure between clinical trials and clinical practice.

Materials and Methods

We extracted drug follow-up times from clinical trials in ClinicalTrials.gov and compared the difference between clinical trials and real-world usage data for 914 drugs taken by 96 645 927 patients.

Results

A total of 17.5% of drugs had longer median exposure in practice than in trials, 6% of patients had extended exposure to at least 1 drug, and drugs treating nervous system disorders and cardiovascular diseases were the most common among drugs with high rates of extended exposure.

Conclusions

For most of patients, the drug use length is shorter than the tested length in clinical trials. Still, a remarkable number of patients experienced extended drug exposure, particularly for drugs treating nervous system disorders or cardiovascular disorders.

Drawing Reproducible Conclusions from Observational Clinical Data with OHDSI

OBJECTIVE

The current observational research literature shows extensive publication bias and contradiction. The Observational Health Data Sciences and Informatics (OHDSI) initiative seeks to improve research reproducibility through open science.

METHODS

OHDSI has created an international federated data source of electronic health records and administrative claims that covers nearly 10% of the world's population. Using a common data model with a practical schema and extensive vocabulary mappings, data from around the world follow the identical format. OHDSI's research methods emphasize reproducibility, with a large-scale approach to addressing confounding using propensity score adjustment with extensive diagnostics; negative and positive control hypotheses to test for residual systematic error; a variety of data sources to assess consistency and generalizability; a completely open approach including protocol, software, models, parameters, and raw results so that studies can be externally verified; and the study of many hypotheses in parallel so that the operating characteristics of the methods can be assessed.

RESULTS

OHDSI has already produced findings in areas like hypertension treatment that are being incorporated into practice, and it has produced rigorous studies of COVID-19 that have aided government agencies in their treatment decisions, that have characterized the disease extensively, that have estimated the comparative effects of treatments, and that the predict likelihood of advancing to serious complications.

CONCLUSIONS

OHDSI practices open science and incorporates a series of methods to address reproducibility. It has produced important results in several areas, including hypertension therapy and COVID-19 research.

Increasing trust in real-world evidence through evaluation of observational data quality

Objective

Advances in standardization of observational healthcare data have enabled methodological breakthroughs, rapid global collaboration, and generation of real-world evidence to improve patient outcomes. Standardizations in data structure, such as use of common data models, need to be coupled with standardized approaches for data quality assessment. To ensure confidence in real-world evidence generated from the analysis of real-world data, one must first have confidence in the data itself.

Materials and Methods

We describe the implementation of check types across a data quality framework of conformance, completeness, plausibility, with both verification and validation. We illustrate how data quality checks, paired with decision thresholds, can be configured to customize data quality reporting across a range of observational health data sources. We discuss how data quality reporting can become part of the overall real-world evidence generation and dissemination process to promote transparency and build confidence in the resulting output.

Results

The Data Quality Dashboard is an open-source R package that reports potential quality issues in an OMOP CDM instance through the systematic execution and summarization of over 3300 configurable data quality checks.

Discussion

Transparently communicating how well common data model-standardized databases adhere to a set of quality measures adds a crucial piece that is currently missing from observational research.

Conclusion

Assessing and improving the quality of our data will inherently improve the quality of the evidence we generate.

Comparative First-Line Effectiveness and Safety of ACE (Angiotensin-Converting Enzyme) Inhibitors and Angiotensin Receptor Blockers: A Multinational Cohort Study

[Figure: see text].

Medications for attention-deficit/hyperactivity disorder in Japan: A retrospective cohort study of label compliance

Aim

To assess label compliance in prescription of medications approved for treatment of attention‐deficit/hyperactivity disorder (ADHD) in Japan at the time of this study: methylphenidate (MPH), atomoxetine, and guanfacine.

Methods

Retrospective descriptive study was conducted in prevalent‐user cohorts from the Japan Medical Data Center database. Patients who were prescribed a study drug between January 1, 2013 and September 30, 2018 and were in the database for ≥30 days were included. A prescription was considered compliant if all 4 criteria were satisfied: appropriate age, daily dose not exceeding the approved maximum, no contraindicated concurrent medications, and no contraindicated conditions.

Results

Among 17 418 patients who were prescribed a study drug during 2013‐2018, 73% were male and 53% were children (aged <18 years). Fewer than 2% of prescriptions were for patients outside the approved age, 10%‐13% of patients in the atomoxetine and MPH cohorts received ≥1 prescription exceeding maximum approved dose, no patients were co‐prescribed a contraindicated medication, and 16%–18% of patients in the MPH cohorts had ≥1 contraindicated condition. During their first 500 days of use, for approximately 73%‐86% of patients, all prescriptions were compliant with all label requirements.

Conclusions

Among patients exposed to ADHD medications in Japan during 2013‐2018, nearly all prescriptions for these medications were label‐compliant for age. For >85% of patients, all prescriptions were label‐compliant for dose, and for approximately 80%, all prescriptions were label‐compliant for contraindicated conditions. We did not find evidence of widespread abuse or noncompliant use of prescribed ADHD medications.

To assess label compliance in prescription of medications approved for the treatment of ADHD in Japan at the time of this study (2013‐2018), a retrospective descriptive cohort study was conducted in patients from the JMDC database who were prescribed a study drug (methylphenidate, atomoxetine, or guanfacine) and were in the database for at least 30 days. Among 17 418 patients eligible for the study, nearly all prescriptions were label‐compliant for age; for more than 85% of patients, all prescriptions were label‐compliant for dose, and for approximately 80%, all prescriptions were label‐compliant for contraindicated conditions. We did not find evidence of widespread abuse or noncompliant use of prescribed ADHD medications.

Characterising the background incidence rates of adverse events of special interest for covid-19 vaccines in eight countries: multinational network cohort study

OBJECTIVE

To quantify the background incidence rates of 15 prespecified adverse events of special interest (AESIs) associated with covid-19 vaccines.

DESIGN

Multinational network cohort study.

SETTING

Electronic health records and health claims data from eight countries: Australia, France, Germany, Japan, the Netherlands, Spain, the United Kingdom, and the United States, mapped to a common data model.

PARTICIPANTS

126 661 070 people observed for at least 365 days before 1 January 2017, 2018, or 2019 from 13 databases.

MAIN OUTCOME MEASURES

Events of interests were 15 prespecified AESIs (non-haemorrhagic and haemorrhagic stroke, acute myocardial infarction, deep vein thrombosis, pulmonary embolism, anaphylaxis, Bell's palsy, myocarditis or pericarditis, narcolepsy, appendicitis, immune thrombocytopenia, disseminated intravascular coagulation, encephalomyelitis (including acute disseminated encephalomyelitis), Guillain-Barré syndrome, and transverse myelitis). Incidence rates of AESIs were stratified by age, sex, and database. Rates were pooled across databases using random effects meta-analyses and classified according to the frequency categories of the Council for International Organizations of Medical Sciences.

RESULTS

Background rates varied greatly between databases. Deep vein thrombosis ranged from 387 (95% confidence interval 370 to 404) per 100 000 person years in UK CPRD GOLD data to 1443 (1416 to 1470) per 100 000 person years in US IBM MarketScan Multi-State Medicaid data among women aged 65 to 74 years. Some AESIs increased with age. For example, myocardial infarction rates in men increased from 28 (27 to 29) per 100 000 person years among those aged 18-34 years to 1400 (1374 to 1427) per 100 000 person years in those older than 85 years in US Optum electronic health record data. Other AESIs were more common in young people. For example, rates of anaphylaxis among boys and men were 78 (75 to 80) per 100 000 person years in those aged 6-17 years and 8 (6 to 10) per 100 000 person years in those older than 85 years in Optum electronic health record data. Meta-analytic estimates of AESI rates were classified according to age and sex.

CONCLUSION

This study found large variations in the observed rates of AESIs by age group and sex, showing the need for stratification or standardisation before using background rates for safety surveillance. Considerable population level heterogeneity in AESI rates was found between databases.

Use of repurposed and adjuvant drugs in hospital patients with covid-19: multinational network cohort study

OBJECTIVE

To investigate the use of repurposed and adjuvant drugs in patients admitted to hospital with covid-19 across three continents.

DESIGN

Multinational network cohort study.

SETTING

Hospital electronic health records from the United States, Spain, and China, and nationwide claims data from South Korea.

PARTICIPANTS

303 264 patients admitted to hospital with covid-19 from January 2020 to December 2020.

MAIN OUTCOME MEASURES

Prescriptions or dispensations of any drug on or 30 days after the date of hospital admission for covid-19.

RESULTS

Of the 303 264 patients included, 290 131 were from the US, 7599 from South Korea, 5230 from Spain, and 304 from China. 3455 drugs were identified. Common repurposed drugs were hydroxychloroquine (used in from <5 (<2%) patients in China to 2165 (85.1%) in Spain), azithromycin (from 15 (4.9%) in China to 1473 (57.9%) in Spain), combined lopinavir and ritonavir (from 156 (<2%) in the VA-OMOP US to 2,652 (34.9%) in South Korea and 1285 (50.5%) in Spain), and umifenovir (0% in the US, South Korea, and Spain and 238 (78.3%) in China). Use of adjunctive drugs varied greatly, with the five most used treatments being enoxaparin, fluoroquinolones, ceftriaxone, vitamin D, and corticosteroids. Hydroxychloroquine use increased rapidly from March to April 2020 but declined steeply in May to June and remained low for the rest of the year. The use of dexamethasone and corticosteroids increased steadily during 2020.

CONCLUSIONS

Multiple drugs were used in the first few months of the covid-19 pandemic, with substantial geographical and temporal variation. Hydroxychloroquine, azithromycin, lopinavir-ritonavir, and umifenovir (in China only) were the most prescribed repurposed drugs. Antithrombotics, antibiotics, H2 receptor antagonists, and corticosteroids were often used as adjunctive treatments. Research is needed on the comparative risk and benefit of these treatments in the management of covid-19.

Prediction of 90-day mortality after surgery for colorectal cancer using standardized nationwide quality-assurance data

Background

Personalized risk assessment provides opportunities for tailoring treatment, optimizing healthcare resources and improving outcome. The aim of this study was to develop a 90-day mortality-risk prediction model for identification of high- and low-risk patients undergoing surgery for colorectal cancer.

Methods

This was a nationwide cohort study using records from the Danish Colorectal Cancer Group database that included all patients undergoing surgery for colorectal cancer between 1 January 2004 and 31 December 2015. A least absolute shrinkage and selection operator logistic regression prediction model was developed using 121 pre- and intraoperative variables and internally validated in a hold-out test data set. The accuracy of the model was assessed in terms of discrimination and calibration.

Results

In total, 49 607 patients were registered in the database. After exclusion of 16 680 individuals, 32 927 patients were included in the analysis. Overall, 1754 (5.3 per cent) deaths were recorded. Targeting high-risk individuals, the model identified 5.5 per cent of all patients facing a risk of 90-day mortality exceeding 35 per cent, corresponding to a 6.7 times greater risk than the average population. Targeting low-risk individuals, the model identified 20.9 per cent of patients facing a risk less than 0.3 per cent, corresponding to a 17.7 times lower risk compared with the average population. The model exhibited discriminatory power with an area under the receiver operating characteristics curve of 85.3 per cent (95 per cent c.i. 83.6 to 87.0) and excellent calibration with a Brier score of 0.04 and 32 per cent average precision.

Conclusion

Pre- and intraoperative data, as captured in national health registries, can be used to predict 90-day mortality accurately after colorectal cancer surgery.

Implementation of the COVID-19 Vulnerability Index Across an International Network of Health Care Data Sets: Collaborative External Validation Study

Background

SARS-CoV-2 is straining health care systems globally. The burden on hospitals during the pandemic could be reduced by implementing prediction models that can discriminate patients who require hospitalization from those who do not. The COVID-19 vulnerability (C-19) index, a model that predicts which patients will be admitted to hospital for treatment of pneumonia or pneumonia proxies, has been developed and proposed as a valuable tool for decision-making during the pandemic. However, the model is at high risk of bias according to the “prediction model risk of bias assessment” criteria, and it has not been externally validated.

Objective

The aim of this study was to externally validate the C-19 index across a range of health care settings to determine how well it broadly predicts hospitalization due to pneumonia in COVID-19 cases.

Methods

We followed the Observational Health Data Sciences and Informatics (OHDSI) framework for external validation to assess the reliability of the C-19 index. We evaluated the model on two different target populations, 41,381 patients who presented with SARS-CoV-2 at an outpatient or emergency department visit and 9,429,285 patients who presented with influenza or related symptoms during an outpatient or emergency department visit, to predict their risk of hospitalization with pneumonia during the following 0-30 days. In total, we validated the model across a network of 14 databases spanning the United States, Europe, Australia, and Asia.

Results

The internal validation performance of the C-19 index had a C statistic of 0.73, and the calibration was not reported by the authors. When we externally validated it by transporting it to SARS-CoV-2 data, the model obtained C statistics of 0.36, 0.53 (0.473-0.584) and 0.56 (0.488-0.636) on Spanish, US, and South Korean data sets, respectively. The calibration was poor, with the model underestimating risk. When validated on 12 data sets containing influenza patients across the OHDSI network, the C statistics ranged between 0.40 and 0.68.

Conclusions

Our results show that the discriminative performance of the C-19 index model is low for influenza cohorts and even worse among patients with COVID-19 in the United States, Spain, and South Korea. These results suggest that C-19 should not be used to aid decision-making during the COVID-19 pandemic. Our findings highlight the importance of performing external validation across a range of settings, especially when a prediction model is being extrapolated to a different population. In the field of prediction, extensive validation is required to create appropriate trust in a model.

Comprehensive Comparative Effectiveness and Safety of First-Line β-Blocker Monotherapy in Hypertensive Patients: A Large-Scale Multicenter Observational Study

[Figure: see text].

Large-scale evidence generation and evaluation across a network of databases (LEGEND): assessing validity using hypertension as a case study

Objectives

To demonstrate the application of the Large-scale Evidence Generation and Evaluation across a Network of Databases (LEGEND) principles described in our companion article to hypertension treatments and assess internal and external validity of the generated evidence.

Materials and Methods

LEGEND defines a process for high-quality observational research based on 10 guiding principles. We demonstrate how this process, here implemented through large-scale propensity score modeling, negative and positive control questions, empirical calibration, and full transparency, can be applied to compare antihypertensive drug therapies. We assess internal validity through covariate balance, confidence-interval coverage, between-database heterogeneity, and transitivity of results. We assess external validity through comparison to direct meta-analyses of randomized controlled trials (RCTs).

Results

From 21.6 million unique antihypertensive new users, we generate 6 076 775 effect size estimates for 699 872 research questions on 12 946 treatment comparisons. Through propensity score matching, we achieve balance on all baseline patient characteristics for 75% of estimates, observe 95.7% coverage in our effect-estimate 95% confidence intervals, find high between-database consistency, and achieve transitivity in 84.8% of triplet hypotheses. Compared with meta-analyses of RCTs, our results are consistent with 28 of 30 comparisons while providing narrower confidence intervals.

Conclusion

We find that these LEGEND results show high internal validity and are congruent with meta-analyses of RCTs. For these reasons we believe that evidence generated by LEGEND is of high quality and can inform medical decision-making where evidence is currently lacking. Subsequent publications will explore the clinical interpretations of this evidence.

Alpha-1 blockers and susceptibility to COVID-19 in benign prostate hyperplasia patients : an international cohort study

Alpha-1 blockers, often used to treat benign prostate hyperplasia (BPH), have been hypothesized to prevent COVID-19 complications by minimising cytokine storms release. We conducted a prevalent-user active-comparator cohort study to assess association between alpha-1 blocker use and risks of three COVID-19 outcomes: diagnosis, hospitalization, and hospitalization requiring intensive services. Our study included 2.6 and 0.46 million users of alpha-1 blockers and of alternative BPH therapy during the period between November 2019 and January 2020, found in electronic health records from Spain (SIDIAP) and the United States (Department of Veterans Affairs, Columbia University Irving Medical Center, IQVIA OpenClaims, Optum DOD, Optum EHR). We estimated hazard ratios using state-of-the-art techniques to minimize potential confounding, including large-scale propensity score matching/stratification and negative control calibration. We found no differential risk for any of COVID-19 outcome, pointing to the need for further research on potential COVID-19 therapies.

An empirical analysis of dealing with patients who are lost to follow-up when developing prognostic models using a cohort design

Background

Researchers developing prediction models are faced with numerous design choices that may impact model performance. One key decision is how to include patients who are lost to follow-up. In this paper we perform a large-scale empirical evaluation investigating the impact of this decision. In addition, we aim to provide guidelines for how to deal with loss to follow-up.

Methods

We generate a partially synthetic dataset with complete follow-up and simulate loss to follow-up based either on random selection or on selection based on comorbidity. In addition to our synthetic data study we investigate 21 real-world data prediction problems. We compare four simple strategies for developing models when using a cohort design that encounters loss to follow-up. Three strategies employ a binary classifier with data that: (1) include all patients (including those lost to follow-up), (2) exclude all patients lost to follow-up or (3) only exclude patients lost to follow-up who do not have the outcome before being lost to follow-up. The fourth strategy uses a survival model with data that include all patients. We empirically evaluate the discrimination and calibration performance.

Results

The partially synthetic data study results show that excluding patients who are lost to follow-up can introduce bias when loss to follow-up is common and does not occur at random. However, when loss to follow-up was completely at random, the choice of addressing it had negligible impact on model discrimination performance. Our empirical real-world data results showed that the four design choices investigated to deal with loss to follow-up resulted in comparable performance when the time-at-risk was 1-year but demonstrated differential bias when we looked into 3-year time-at-risk. Removing patients who are lost to follow-up before experiencing the outcome but keeping patients who are lost to follow-up after the outcome can bias a model and should be avoided.

Conclusion

Based on this study we therefore recommend (1) developing models using data that includes patients that are lost to follow-up and (2) evaluate the discrimination and calibration of models twice: on a test set including patients lost to follow-up and a test set excluding patients lost to follow-up.

Towards clinical data-driven eligibility criteria optimization for interventional COVID-19 clinical trials

Objective

This research aims to evaluate the impact of eligibility criteria on recruitment and observable clinical outcomes of COVID-19 clinical trials using electronic health record (EHR) data.

Materials and Methods

On June 18, 2020, we identified frequently used eligibility criteria from all the interventional COVID-19 trials in ClinicalTrials.gov (n = 288), including age, pregnancy, oxygen saturation, alanine/aspartate aminotransferase, platelets, and estimated glomerular filtration rate. We applied the frequently used criteria to the EHR data of COVID-19 patients in Columbia University Irving Medical Center (CUIMC) (March 2020–June 2020) and evaluated their impact on patient accrual and the occurrence of a composite endpoint of mechanical ventilation, tracheostomy, and in-hospital death.

Results

There were 3251 patients diagnosed with COVID-19 from the CUIMC EHR included in the analysis. The median follow-up period was 10 days (interquartile range 4–28 days). The composite events occurred in 18.1% (n = 587) of the COVID-19 cohort during the follow-up. In a hypothetical trial with common eligibility criteria, 33.6% (690/2051) were eligible among patients with evaluable data and 22.2% (153/690) had the composite event.

Discussion

By adjusting the thresholds of common eligibility criteria based on the characteristics of COVID-19 patients, we could observe more composite events from fewer patients.

Conclusions

This research demonstrated the potential of using the EHR data of COVID-19 patients to inform the selection of eligibility criteria and their thresholds, supporting data-driven optimization of participant selection towards improved statistical power of COVID-19 trials.

Renin-angiotensin system blockers and susceptibility to COVID-19: an international, open science, cohort analysis

BACKGROUND

Angiotensin-converting enzyme inhibitors (ACEIs) and angiotensin receptor blockers (ARBs) have been postulated to affect susceptibility to COVID-19. Observational studies so far have lacked rigorous ascertainment adjustment and international generalisability. We aimed to determine whether use of ACEIs or ARBs is associated with an increased susceptibility to COVID-19 in patients with hypertension.

METHODS

In this international, open science, cohort analysis, we used electronic health records from Spain (Information Systems for Research in Primary Care [SIDIAP]) and the USA (Columbia University Irving Medical Center data warehouse [CUIMC] and Department of Veterans Affairs Observational Medical Outcomes Partnership [VA-OMOP]) to identify patients aged 18 years or older with at least one prescription for ACEIs and ARBs (target cohort) or calcium channel blockers (CCBs) and thiazide or thiazide-like diuretics (THZs; comparator cohort) between Nov 1, 2019, and Jan 31, 2020. Users were defined separately as receiving either monotherapy with these four drug classes, or monotherapy or combination therapy (combination use) with other antihypertensive medications. We assessed four outcomes: COVID-19 diagnosis; hospital admission with COVID-19; hospital admission with pneumonia; and hospital admission with pneumonia, acute respiratory distress syndrome, acute kidney injury, or sepsis. We built large-scale propensity score methods derived through a data-driven approach and negative control experiments across ten pairwise comparisons, with results meta-analysed to generate 1280 study effects. For each study effect, we did negative control outcome experiments using a possible 123 controls identified through a data-rich algorithm. This process used a set of predefined baseline patient characteristics to provide the most accurate prediction of treatment and balance among patient cohorts across characteristics. The study is registered with the EU Post-Authorisation Studies register, EUPAS35296.

FINDINGS

Among 1 355 349 antihypertensive users (363 785 ACEI or ARB monotherapy users, 248 915 CCB or THZ monotherapy users, 711 799 ACEI or ARB combination users, and 473 076 CCB or THZ combination users) included in analyses, no association was observed between COVID-19 diagnosis and exposure to ACEI or ARB monotherapy versus CCB or THZ monotherapy (calibrated hazard ratio [HR] 0·98, 95% CI 0·84-1·14) or combination use exposure (1·01, 0·90-1·15). ACEIs alone similarly showed no relative risk difference when compared with CCB or THZ monotherapy (HR 0·91, 95% CI 0·68-1·21; with heterogeneity of >40%) or combination use (0·95, 0·83-1·07). Directly comparing ACEIs with ARBs demonstrated a moderately lower risk with ACEIs, which was significant with combination use (HR 0·88, 95% CI 0·79-0·99) and non-significant for monotherapy (0·85, 0·69-1·05). We observed no significant difference between drug classes for risk of hospital admission with COVID-19, hospital admission with pneumonia, or hospital admission with pneumonia, acute respiratory distress syndrome, acute kidney injury, or sepsis across all comparisons.

INTERPRETATION

No clinically significant increased risk of COVID-19 diagnosis or hospital admission-related outcomes associated with ACEI or ARB use was observed, suggesting users should not discontinue or change their treatment to decrease their risk of COVID-19.

FUNDING

Wellcome Trust, UK National Institute for Health Research, US National Institutes of Health, US Department of Veterans Affairs, Janssen Research & Development, IQVIA, South Korean Ministry of Health and Welfare Republic, Australian National Health and Medical Research Council, and European Health Data and Evidence Network.

Quantifying bias in epidemiologic studies evaluating the association between acetaminophen use and cancer

Many observational studies explore the association between acetaminophen and cancer, but known limitations such as vulnerability to channeling, protopathic bias, and uncontrolled confounding hamper the interpretability of results. To help understand the potential magnitude of bias, we identify key design choices in these observational studies and specify 10 study design variants that represent different combinations of these design choices. We evaluate these variants by applying them to 37 negative controls - outcome presumed not to be caused by acetaminophen - as well as 4 cancer outcomes in the Clinical Practice Research Datalink (CPRD) database. The estimated odds and hazards ratios for the negative controls show substantial bias in the evaluated design variants, with far fewer of the 95% confidence intervals containing 1 than the nominal 95% expected for negative controls. The effect-size estimates for the cancer outcomes are comparable to those observed for the negative controls. A comparison of exposed and unexposed reveals many differences at baseline for which most studies do not correct. We observe that the design choices made in many of the published observational studies can lead to substantial bias. Thus, caution in the interpretation of published studies of acetaminophen and cancer is recommended.

Characteristics, outcomes, and mortality amongst 133,589 patients with prevalent autoimmune diseases diagnosed with, and 48,418 hospitalised for COVID-19: a multinational distributed network cohort analysis

OBJECTIVE

Patients with autoimmune diseases were advised to shield to avoid COVID-19, but information on their prognosis is lacking. We characterised 30-day outcomes and mortality after hospitalisation with COVID-19 among patients with prevalent autoimmune diseases, and compared outcomes after hospital admissions among similar patients with seasonal influenza.

DESIGN

Multinational network cohort study.

SETTING

Electronic health records data from Columbia University Irving Medical Center (CUIMC) (NYC, United States [US]), Optum [US], Department of Veterans Affairs (VA) (US), Information System for Research in Primary Care-Hospitalisation Linked Data (SIDIAP-H) (Spain), and claims data from IQVIA Open Claims (US) and Health Insurance and Review Assessment (HIRA) (South Korea).

PARTICIPANTS

All patients with prevalent autoimmune diseases, diagnosed and/or hospitalised between January and June 2020 with COVID-19, and similar patients hospitalised with influenza in 2017-2018 were included.

MAIN OUTCOME MEASURES

30-day complications during hospitalisation and death.

RESULTS

We studied 133,589 patients diagnosed and 48,418 hospitalised with COVID-19 with prevalent autoimmune diseases. The majority of participants were female (60.5% to 65.9%) and aged ≥50 years. The most prevalent autoimmune conditions were psoriasis (3.5 to 32.5%), rheumatoid arthritis (3.9 to 18.9%), and vasculitis (3.3 to 17.6%). Amongst hospitalised patients, Type 1 diabetes was the most common autoimmune condition (4.8% to 7.5%) in US databases, rheumatoid arthritis in HIRA (18.9%), and psoriasis in SIDIAP-H (26.4%).Compared to 70,660 hospitalised with influenza, those admitted with COVID-19 had more respiratory complications including pneumonia and acute respiratory distress syndrome, and higher 30-day mortality (2.2% to 4.3% versus 6.3% to 24.6%).

CONCLUSIONS

Patients with autoimmune diseases had high rates of respiratory complications and 30-day mortality following a hospitalization with COVID-19. Compared to influenza, COVID-19 is a more severe disease, leading to more complications and higher mortality. Future studies should investigate predictors of poor outcomes in COVID-19 patients with autoimmune diseases.

WHAT IS ALREADY KNOWN ABOUT THIS TOPIC

Patients with autoimmune conditions may be at increased risk of COVID-19 infection andcomplications.There is a paucity of evidence characterising the outcomes of hospitalised COVID-19 patients with prevalent autoimmune conditions.

WHAT THIS STUDY ADDS

Most people with autoimmune diseases who required hospitalisation for COVID-19 were women, aged 50 years or older, and had substantial previous comorbidities.Patients who were hospitalised with COVID-19 and had prevalent autoimmune diseases had higher prevalence of hypertension, chronic kidney disease, heart disease, and Type 2 diabetes as compared to those with prevalent autoimmune diseases who were diagnosed with COVID-19.A variable proportion of 6% to 25% across data sources died within one month of hospitalisation with COVID-19 and prevalent autoimmune diseases.For people with autoimmune diseases, COVID-19 hospitalisation was associated with worse outcomes and 30-day mortality compared to admission with influenza in the 2017-2018 season.