1
|
Chalkou K, Hamza T, Benkert P, Kuhle J, Zecca C, Simoneau G, Pellegrini F, Manca A, Egger M, Salanti G. Combining randomized and non-randomized data to predict heterogeneous effects of competing treatments. Res Synth Methods 2024; 15:641-656. [PMID: 38501273 DOI: 10.1002/jrsm.1717] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2022] [Revised: 01/26/2024] [Accepted: 02/16/2024] [Indexed: 03/20/2024]
Abstract
Some patients benefit from a treatment while others may do so less or do not benefit at all. We have previously developed a two-stage network meta-regression prediction model that synthesized randomized trials and evaluates how treatment effects vary across patient characteristics. In this article, we extended this model to combine different sources of types in different formats: aggregate data (AD) and individual participant data (IPD) from randomized and non-randomized evidence. In the first stage, a prognostic model is developed to predict the baseline risk of the outcome using a large cohort study. In the second stage, we recalibrated this prognostic model to improve our predictions for patients enrolled in randomized trials. In the third stage, we used the baseline risk as effect modifier in a network meta-regression model combining AD, IPD randomized clinical trial to estimate heterogeneous treatment effects. We illustrated the approach in the re-analysis of a network of studies comparing three drugs for relapsing-remitting multiple sclerosis. Several patient characteristics influence the baseline risk of relapse, which in turn modifies the effect of the drugs. The proposed model makes personalized predictions for health outcomes under several treatment options and encompasses all relevant randomized and non-randomized evidence.
Collapse
Affiliation(s)
- Konstantina Chalkou
- Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
- Graduate School for Health Sciences, University of Bern, Bern, Switzerland
- Department of Clinical Research, University of Bern, Bern, Switzerland
| | - Tasnim Hamza
- Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
- Graduate School for Health Sciences, University of Bern, Bern, Switzerland
| | - Pascal Benkert
- Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland
| | - Jens Kuhle
- Multiple Sclerosis Centre, Neurologic Clinic and Policlinic, Department of Head, Spine and Neuromedicine, University Hospital Basel, University of Basel, Basel, Switzerland
- Multiple Sclerosis Centre, Neurologic Clinic and Policlinic, Department of Biomedicine, University Hospital Basel, University of Basel, Basel, Switzerland
- Multiple Sclerosis Centre, Neurologic Clinic and Policlinic, Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland
- Research Center for Clinical Neuroimmunology and Neuroscience (RC2NB), University Hospital, University of Basel, Basel, Switzerland
| | - Chiara Zecca
- Multiple Sclerosis Center, Neurocenter of Southern Switzerland, EOC, Lugano, Switzerland
- Faculty of Biomedical Sciences, Università della Svizzera Italiana, Lugano, Switzerland
| | | | | | - Andrea Manca
- Centre for Health Economics, University of York, York, UK
| | - Matthias Egger
- Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
- Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
| | - Georgia Salanti
- Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
| |
Collapse
|
2
|
Seo M, Debray TP, Ruffieux Y, Gsteiger S, Bujkiewicz S, Finckh A, Egger M, Efthimiou O. Combining individual patient data from randomized and non-randomized studies to predict real-world effectiveness of interventions. Stat Methods Med Res 2022; 31:1355-1373. [PMID: 35469504 PMCID: PMC9251754 DOI: 10.1177/09622802221090759] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Meta-analysis of randomized controlled trials is generally considered the most reliable source of estimates of relative treatment effects. However, in the last few years, there has been interest in using non-randomized studies to complement evidence from randomized controlled trials. Several meta-analytical models have been proposed to this end. Such models mainly focussed on estimating the average relative effects of interventions. In real-life clinical practice, when deciding on how to treat a patient, it might be of great interest to have personalized predictions of absolute outcomes under several available treatment options. This paper describes a general framework for developing models that combine individual patient data from randomized controlled trials and non-randomized study when aiming to predict outcomes for a set of competing medical interventions applied in real-world clinical settings. We also discuss methods for measuring the models' performance to identify the optimal model to use in each setting. We focus on the case of continuous outcomes and illustrate our methods using a data set from rheumatoid arthritis, comprising patient-level data from three randomized controlled trials and two registries from Switzerland and Britain.
Collapse
Affiliation(s)
- Michael Seo
- Institute of Social and Preventive Medicine, 27210University of Bern, Bern, Switzerland.,Graduate School for Health Sciences, 27210University of Bern, Bern, Switzerland
| | - Thomas Pa Debray
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, 8125Utrecht University, Utrecht, The Netherlands.,Smart Data Analysis and Statistics B.V., Utrecht, The Netherlands
| | - Yann Ruffieux
- Institute of Social and Preventive Medicine, 27210University of Bern, Bern, Switzerland
| | - Sandro Gsteiger
- Pharmaceuticals Division, Global Access, F. Hoffmann-La Roche, Basel, Switzerland
| | - Sylwia Bujkiewicz
- Biostatistics Research Group, Department of Health Sciences, 4488University of Leicester, Leicester, UK
| | - Axel Finckh
- Division of Rheumatology, 30576University Hospitals of Geneva, Geneva, Switzerland
| | - Matthias Egger
- Institute of Social and Preventive Medicine, 27210University of Bern, Bern, Switzerland.,Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
| | - Orestis Efthimiou
- Institute of Social and Preventive Medicine, 27210University of Bern, Bern, Switzerland.,Department of Psychiatry, 6396University of Oxford, Oxford, UK
| |
Collapse
|
3
|
Hogervorst MA, Pontén J, Vreman RA, Mantel-Teeuwisse AK, Goettsch WG. Real World Data in Health Technology Assessment of Complex Health Technologies. Front Pharmacol 2022; 13:837302. [PMID: 35222045 PMCID: PMC8866967 DOI: 10.3389/fphar.2022.837302] [Citation(s) in RCA: 14] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2021] [Accepted: 01/24/2022] [Indexed: 11/13/2022] Open
Abstract
The available evidence on relative effectiveness and risks of new health technologies is often limited at the time of health technology assessment (HTA). Additionally, a wide variety in real-world data (RWD) policies exist among HTA organizations. This study assessed which challenges, related to the increasingly complex nature of new health technologies, make the acceptance of RWD most likely. A questionnaire was disseminated among 33 EUnetHTA member HTA organizations. The questions focused on accepted data sources, circumstances that allowed for RWD acceptance and barriers to acceptance. The questionnaire was validated and tested for reliability by an expert panel, and pilot-tested before dissemination via LimeSurvey. Twenty-two HTA organizations completed the questionnaire (67%). All reported accepting randomized clinical trials. The most accepted RWD source were patient registries (19/22, 86%), the least accepted were editorials and expert opinions (8/22, 36%). With orphan treatments or companion diagnostics, organizations tended to be most likely to accept RWD sources, 4.3-3.2 on a 5-point Likert scale, respectively. Additional circumstances were reported to accept RWD (e.g., a high disease burden). The two most important barriers to accepting RWD were lacking necessary RWD sources and existing policy structures. European HTA organizations seem positive toward the (wider) use of RWD in HTA of complex therapies. Expanding the use of patient registries could be potentially useful, as a large share of the organizations already accepts this source. However, many barriers still exist to the widespread use of RWD. Our results can be used to prioritize circumstances in which RWD might be accepted.
Collapse
Affiliation(s)
- Milou A. Hogervorst
- Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences (UIPS), Utrecht University, Utrecht, Netherlands
- National Health Care Institute (ZIN), Diemen, Netherlands
| | - Johan Pontén
- The Dental and Pharmaceutical Benefits Agency (TLV), Stockholm, Sweden
| | - Rick A. Vreman
- Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences (UIPS), Utrecht University, Utrecht, Netherlands
- National Health Care Institute (ZIN), Diemen, Netherlands
| | - Aukje K. Mantel-Teeuwisse
- Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences (UIPS), Utrecht University, Utrecht, Netherlands
| | - Wim G. Goettsch
- Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences (UIPS), Utrecht University, Utrecht, Netherlands
- National Health Care Institute (ZIN), Diemen, Netherlands
| |
Collapse
|
4
|
Nikolakopoulou A, Trelle S, Sutton AJ, Egger M, Salanti G. Synthesizing existing evidence to design future trials: survey of methodologists from European institutions. Trials 2019; 20:334. [PMID: 31174597 PMCID: PMC6555919 DOI: 10.1186/s13063-019-3449-6] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2018] [Accepted: 05/13/2019] [Indexed: 12/26/2022] Open
Abstract
Background ‘Conditional trial design’ is a framework for efficiently planning new clinical trials based on a network of relevant existing trials. The framework considers whether new trials are required and how the existing evidence can be used to answer the research question and plan future research. The potential of this approach has not been fully realized. Methods We conducted an online survey among trial statisticians, methodologists, and users of evidence synthesis research using referral sampling to capture opinions about the conditional trial design framework and current practices among clinical researchers. The questions included in the survey were related to the decision of whether a meta-analysis answers the research question, the optimal way to synthesize available evidence, which relates to the acceptability of network meta-analysis, and the use of evidence synthesis in the planning of new studies. Results In total, 76 researchers completed the survey. Two out of three survey participants (65%) were willing to possibly or definitely consider using evidence synthesis to design a future clinical trial and around half of the participants would give priority to such a trial design. The median rating of the frequency of using such a trial design was 0.41 on a scale from 0 (never) to 1 (always). Major barriers to adopting conditional trial design include the current regulatory paradigm and the policies of funding agencies and sponsors. Conclusions Participants reported moderate interest in using evidence synthesis methods in the design of future trials. They indicated that a major paradigm shift is required before the use of network meta-analysis is regularly employed in the design of trials. Electronic supplementary material The online version of this article (10.1186/s13063-019-3449-6) contains supplementary material, which is available to authorized users.
Collapse
Affiliation(s)
- Adriani Nikolakopoulou
- Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, Switzerland.
| | - Sven Trelle
- CTU Bern, University of Bern, Bern, Switzerland
| | - Alex J Sutton
- Department of Health Sciences, College of Medicine, Biological Sciences and Psychology, University of Leicester, Leicester, UK
| | - Matthias Egger
- Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, Switzerland
| | - Georgia Salanti
- Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, Switzerland
| |
Collapse
|
5
|
Pellegrini F, Copetti M, Bovis F, Cheng D, Hyde R, de Moor C, Kieseier BC, Sormani MP. A proof-of-concept application of a novel scoring approach for personalized medicine in multiple sclerosis. Mult Scler 2019; 26:1064-1073. [PMID: 31144577 DOI: 10.1177/1352458519849513] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
BACKGROUND Stratified medicine methodologies based on subgroup analyses are often insufficiently powered. More powerful personalized medicine approaches are based on continuous scores. OBJECTIVE We deployed a patient-specific continuous score predicting treatment response in patients with relapsing-remitting multiple sclerosis (RRMS). METHODS Data from two independent randomized controlled trials (RCTs) were used to build and validate an individual treatment response (ITR) score, regressing annualized relapse rates (ARRs) on a set of baseline predictors. RESULTS The ITR score for the combined treatment groups versus placebo detected differential clinical response in both RCTs. High responders in one RCT had a cross-validated ARR ratio of 0.29 (95% confidence interval (CI) = 0.13-0.55) versus 0.62 (95% CI = 0.47-0.83) for all other responders (heterogeneity p = 0.038) and were validated in the other RCT, with the corresponding ARR ratios of 0.31 (95% CI = 0.18-0.56) and 0.61 (95% CI = 0.47-0.79; heterogeneity p = 0.036). The strongest treatment effect modifiers were the Short Form-36 Physical Component Summary, age, Visual Function Test 2.5%, prior MS treatment and Expanded Disability Status Scale. CONCLUSION Our modelling strategy detects and validates an ITR score and opens up avenues for building treatment response calculators that are also applicable in routine clinical practice.
Collapse
Affiliation(s)
| | - Massimiliano Copetti
- Unit of Biostatistics, Fondazione IRCCS Casa Sollievo della Sofferenza Hospital, San Giovanni Rotondo, Italy
| | - Francesca Bovis
- Department of Health Sciences (DISSAL), University of Genoa, Genoa, Italy
| | - David Cheng
- Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Robert Hyde
- Biogen International GmbH, Baar, Switzerland
| | | | - Bernd C Kieseier
- Biogen Inc., Cambridge, MA, USA; Department of Neurology, Medical Faculty, Heinrich-Heine University, Düsseldorf, Germany
| | - Maria Pia Sormani
- Department of Health Sciences (DISSAL), University of Genoa, Genoa, Italy; IRCCS Ospedale Policlinico San Martino, Genoa, Italy
| |
Collapse
|
6
|
Egger M, Johnson L, Althaus C, Schöni A, Salanti G, Low N, Norris SL. Developing WHO guidelines: Time to formally include evidence from mathematical modelling studies. F1000Res 2017; 6:1584. [PMID: 29552335 PMCID: PMC5829466 DOI: 10.12688/f1000research.12367.2] [Citation(s) in RCA: 26] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 02/19/2018] [Indexed: 12/15/2022] Open
Abstract
In recent years, the number of mathematical modelling studies has increased steeply. Many of the questions addressed in these studies are relevant to the development of World Health Organization (WHO) guidelines, but modelling studies are rarely formally included as part of the body of evidence. An expert consultation hosted by WHO, a survey of modellers and users of modelling studies, and literature reviews informed the development of recommendations on when and how to incorporate the results of modelling studies into WHO guidelines. In this article, we argue that modelling studies should routinely be considered in the process of developing WHO guidelines, but particularly in the evaluation of public health programmes, long-term effectiveness or comparative effectiveness. There should be a systematic and transparent approach to identifying relevant published models, and to commissioning new models. We believe that the inclusion of evidence from modelling studies into the Grading of Recommendations Assessment, Development and Evaluation (GRADE) process is possible and desirable, with relatively few adaptations. No single "one-size-fits-all" approach is appropriate to assess the quality of modelling studies. The concept of the 'credibility' of the model, which takes the conceptualization of the problem, model structure, input data, different dimensions of uncertainty, as well as transparency and validation into account, is more appropriate than 'risk of bias'.
Collapse
Affiliation(s)
- Matthias Egger
- Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, 3012, Switzerland.,Centre for Infectious Disease Epidemiology and Research (CIDER), University of Cape Town, Cape Town, 7925, South Africa
| | - Leigh Johnson
- Centre for Infectious Disease Epidemiology and Research (CIDER), University of Cape Town, Cape Town, 7925, South Africa
| | - Christian Althaus
- Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, 3012, Switzerland
| | - Anna Schöni
- Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, 3012, Switzerland
| | - Georgia Salanti
- Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, 3012, Switzerland
| | - Nicola Low
- Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, 3012, Switzerland
| | | |
Collapse
|
7
|
Egger M, Johnson L, Althaus C, Schöni A, Salanti G, Low N, Norris SL. Developing WHO guidelines: Time to formally include evidence from mathematical modelling studies. F1000Res 2017; 6:1584. [PMID: 29552335 DOI: 10.12688/f1000research.12367.1] [Citation(s) in RCA: 33] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 08/11/2017] [Indexed: 12/15/2022] Open
Abstract
In recent years, the number of mathematical modelling studies has increased steeply. Many of the questions addressed in these studies are relevant to the development of World Health Organization (WHO) guidelines, but modelling studies are rarely formally included as part of the body of evidence. An expert consultation hosted by WHO, a survey of modellers and users of modelling studies, and literature reviews informed the development of recommendations on when and how to incorporate the results of modelling studies into WHO guidelines. In this article, we argue that modelling studies should routinely be considered in the process of developing WHO guidelines, but particularly in the evaluation of public health programmes, long-term effectiveness or comparative effectiveness. There should be a systematic and transparent approach to identifying relevant published models, and to commissioning new models. We believe that the inclusion of evidence from modelling studies into the Grading of Recommendations Assessment, Development and Evaluation (GRADE) process is possible and desirable, with relatively few adaptations. No single "one-size-fits-all" approach is appropriate to assess the quality of modelling studies. The concept of the 'credibility' of the model, which takes the conceptualization of the problem, model structure, input data, different dimensions of uncertainty, as well as transparency and validation into account, is more appropriate than 'risk of bias'.
Collapse
Affiliation(s)
- Matthias Egger
- Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, 3012, Switzerland.,Centre for Infectious Disease Epidemiology and Research (CIDER), University of Cape Town, Cape Town, 7925, South Africa
| | - Leigh Johnson
- Centre for Infectious Disease Epidemiology and Research (CIDER), University of Cape Town, Cape Town, 7925, South Africa
| | - Christian Althaus
- Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, 3012, Switzerland
| | - Anna Schöni
- Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, 3012, Switzerland
| | - Georgia Salanti
- Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, 3012, Switzerland
| | - Nicola Low
- Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, 3012, Switzerland
| | | |
Collapse
|