1
|
Otten TM, Grimm SE, Ramaekers B, Joore MA. Comprehensive Review of Methods to Assess Uncertainty in Health Economic Evaluations. PHARMACOECONOMICS 2023; 41:619-632. [PMID: 36943674 PMCID: PMC10163110 DOI: 10.1007/s40273-023-01242-1] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 01/11/2023] [Indexed: 05/06/2023]
Abstract
Uncertainty assessment is a cornerstone in model-based health economic evaluations (HEEs) that inform reimbursement decisions. No comprehensive overview of available uncertainty assessment methods currently exists. We aimed to review methods for uncertainty assessment for use in model-based HEEs, by conducting a snowballing review. We categorised all methods according to their stage of use relating to uncertainty assessment (identification, analysis, communication). Additionally, we classified identification methods according to sources of uncertainty, and subdivided analysis and communication methods according to their purpose. The review identified a total of 80 uncertainty methods: 30 identification, 28 analysis, and 22 communication methods. Uncertainty identification methods exist to address uncertainty from different sources. Most identification methods were developed with the objective to assess related concepts such as validity, model quality, and relevance. Almost all uncertainty analysis and communication methods required uncertainty to be quantified and inclusion of uncertainties in probabilistic analysis. Our review can help analysts and decision makers in selecting uncertainty assessment methods according to their aim and purpose of the assessment. We noted a need for further clarification of terminology and guidance on the use of (combinations of) methods to identify uncertainty and related concepts such as validity and quality. A key finding is that uncertainty assessment relies heavily on quantification, which may necessitate increased use of expert elicitation and/or the development of methods to assess unquantified uncertainty.
Collapse
Affiliation(s)
- Thomas Michael Otten
- Department of Clinical Epidemiology and Medical Technology Assessment (KEMTA), P. Debyelaan 25, Oxford Building, PO Box 5800a, Maastricht, Limburg, The Netherlands.
| | - Sabine E Grimm
- Department of Clinical Epidemiology and Medical Technology Assessment (KEMTA), P. Debyelaan 25, Oxford Building, PO Box 5800a, Maastricht, Limburg, The Netherlands
| | - Bram Ramaekers
- Department of Clinical Epidemiology and Medical Technology Assessment (KEMTA), P. Debyelaan 25, Oxford Building, PO Box 5800a, Maastricht, Limburg, The Netherlands
| | - Manuela A Joore
- Department of Clinical Epidemiology and Medical Technology Assessment (KEMTA), P. Debyelaan 25, Oxford Building, PO Box 5800a, Maastricht, Limburg, The Netherlands
| |
Collapse
|
2
|
Imam AA. Remarkably reproducible psychological (memory) phenomena in the classroom: some evidence for generality from small-N research. BMC Psychol 2022; 10:274. [PMID: 36419180 PMCID: PMC9685964 DOI: 10.1186/s40359-022-00982-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2021] [Accepted: 11/09/2022] [Indexed: 11/27/2022] Open
Abstract
BACKGROUND Mainstream psychology is experiencing a crisis of confidence. Many of the methodological solutions offered in response have focused largely on statistical alternatives to null hypothesis statistical testing, ignoring nonstatistical remedies that are readily available within psychology; namely, use of small-N designs. In fact, many classic memory studies that have passed the test of replicability used them. That methodological legacy warranted a retrospective look at nonexperimental data to explore the generality of the reported effects. METHOD Various classroom demonstrations were conducted over multiple semesters in introductory psychology courses with typical, mostly freshman students from a predominantly white private Catholic university in the US Midwest based on classic memory experiments on immediate memory span, chunking, and depth of processing. RESULTS Students tended to remember 7 ± 2 digits, remembered more digits of π following an attached meaningful story, and remembered more words after elaborative rehearsal than after maintenance rehearsal. These results amount to replications under uncontrolled classroom environments of the classic experiments originally conducted largely outside of null hypothesis statistical testing frameworks. CONCLUSIONS In light of the ongoing replication crisis in psychology, the results are remarkable and noteworthy, validating these historically important psychological findings. They are testament to the reliability of reproducible effects as the hallmark of empirical findings in science and suggest an alternative approach to commonly proffered solutions to the replication crisis.
Collapse
Affiliation(s)
- Abdulrazaq A. Imam
- grid.258192.50000 0001 2295 5682Department of Psychology, John Carroll University, 1 John Carroll Blvd, University Heights, OH 44118 USA
| |
Collapse
|
3
|
Schwander B, Kaier K, Hiligsmann M, Evers S, Nuijten M. Does the Structure Matter? An External Validation and Health Economic Results Comparison of Event Simulation Approaches in Severe Obesity. PHARMACOECONOMICS 2022; 40:901-915. [PMID: 35771486 PMCID: PMC9363367 DOI: 10.1007/s40273-022-01162-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Accepted: 05/29/2022] [Indexed: 06/15/2023]
Abstract
OBJECTIVES As obesity-associated events impact long-term survival, health economic (HE) modelling is commonly applied, but modelling approaches are diverse. This research aimed to compare the events simulation and the HE outcomes produced by different obesity modelling approaches. METHODS An external validation, using the Swedish obesity subjects (SOS) study, of three main structural event modelling approaches was performed: (1) continuous body mass index (BMI) approach; (2) risk equation approach; and (3) categorical BMI-related approach. Outcomes evaluated were mortality, cardiovascular events, and type 2 diabetes (T2D) for both the surgery and the control arms. Concordance between modelling results and the SOS study were investigated by different state-of-the-art measurements, and categorized by the grade of deviation observed (grades 1-4 expressing mild, moderate, severe, and very severe deviations). Furthermore, the costs per quality-adjusted life-year (QALY) gained of surgery versus controls were compared. RESULTS Overall and by study arm, the risk equation approach presented the lowest average grade of deviation (overall grade 2.50; control arm 2.25; surgery arm 2.75), followed by the continuous BMI approach (overall 3.25; control 3.50; surgery 3.00) and by the categorial BMI approach (overall 3.63; control 3.50; surgery 3.75). Considering different confidence interval limits, the costs per QALY gained were fairly comparable between all structural approaches (ranging from £2,055 to £6,206 simulating a lifetime horizon). CONCLUSION None of the structural approaches provided perfect external event validation, although the risk equation approach showed the lowest overall deviations. The economic outcomes resulting from the three approaches were fairly comparable.
Collapse
Affiliation(s)
- Björn Schwander
- Department of Health Services Research, CAPHRI-Care and Public Health Research Institute, Maastricht University, Maastricht, the Netherlands
- AHEAD GmbH-Agency for Health Economic Assessment and Dissemination, Wilhelm-Leibl-Str. 7, D-74321 Bietigheim-Bissingen, Germany
| | - Klaus Kaier
- Institute of Medical Biometry and Statistics (IMBI), University of Freiburg, Freiburg im Breisgau, Germany
| | - Mickaël Hiligsmann
- Department of Health Services Research, CAPHRI-Care and Public Health Research Institute, Maastricht University, Maastricht, the Netherlands
| | - Silvia Evers
- Department of Health Services Research, CAPHRI-Care and Public Health Research Institute, Maastricht University, Maastricht, the Netherlands
- Trimbos Institute-Netherlands Institute of Mental Health and Addiction, Utrecht, the Netherlands
| | - Mark Nuijten
- a2m-Ars Accessus Medica, Amsterdam, the Netherlands
| |
Collapse
|
4
|
Schwander B, Nuijten M, Evers S, Hiligsmann M. Replication of Published Health Economic Obesity Models: Assessment of Facilitators, Hurdles and Reproduction Success. PHARMACOECONOMICS 2021; 39:433-446. [PMID: 33751452 PMCID: PMC8009773 DOI: 10.1007/s40273-021-01008-7] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Accepted: 02/15/2021] [Indexed: 06/12/2023]
Abstract
OBJECTIVES This research aims to (1) replicate published health economic models, (2) compare reproduced results with original results, (3) identify facilitators and hurdles to model replicability and determine reproduction success, and (4) suggest model replication reporting standards to enhance model reproducibility, in the context of health economic obesity models. METHODS Four health economic obesity models simulating an adult UK population were identified, selected for replication, and evaluated using the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist. Reproduction results were compared to original results, focusing on cost-effectiveness outcomes, and the resulting reproduction success was assessed by published criteria. Replication facilitators and hurdles were identified and transferred into related reporting standards. RESULTS All four case studies were state-transition models simulating costs and quality-adjusted life-years (QALYs). Comparing original versus reproduction outcomes, the following deviation ranges were observed: costs - 3.9 to 16.1% (mean over all model simulations 3.78%), QALYs - 3.7 to 2.1% (mean - 0.11%), and average cost-utility ratios - 3.0 to 17.9% (mean 4.28%). Applying different published criteria, an overall reproduction success was observed for three of four models. Key replication facilitators were input data tables and model diagrams, while missing standard deviations and missing formulas for equations were considered as key hurdles. CONCLUSIONS This study confirms the feasibility of rebuilding health economic obesity models, but minor to major assumptions were needed to fill reporting gaps. Model replications can help to assess the quality of health economic model documentation and can be used to validate current model reporting practices. Simple changes to actual CHEERS reporting criteria may solve identified replication hurdles.
Collapse
Affiliation(s)
- Björn Schwander
- Department of Health Services Research, CAPHRI-Care and Public Health Research Institute, Maastricht University, Maastricht, The Netherlands
- AHEAD GmbH-Agency for Health Economic Assessment and Dissemination, Waschhausgasse 17, 79540 Lörrach, Germany
| | - Mark Nuijten
- a2m-Ars Accessus Medica, Amsterdam, The Netherlands
| | - Silvia Evers
- Department of Health Services Research, CAPHRI-Care and Public Health Research Institute, Maastricht University, Maastricht, The Netherlands
- Trimbos Institute-Netherlands Institute of Mental Health and Addiction, Utrecht, The Netherlands
| | - Mickaël Hiligsmann
- Department of Health Services Research, CAPHRI-Care and Public Health Research Institute, Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
5
|
Zawadzki NK, Hay JW. Characterizing the Validity and Real-World Utility of Health Technology Assessments in Healthcare: Future Directions Comment on "Problems and Promises of Health Technologies: The Role of Early Health Economic Modelling". Int J Health Policy Manag 2020; 9:352-355. [PMID: 32613807 PMCID: PMC7500389 DOI: 10.15171/ijhpm.2019.132] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2019] [Accepted: 11/30/2019] [Indexed: 12/04/2022] Open
Abstract
With their article, Grutters et al raise an important question: What do successful health technology assessments (HTAs) look like, and what is their real-world utility in decision-making? While many HTAs are published in peer-reviewed journals, many are considered proprietary and their attributes remain confidential, limiting researchers’ ability to answer these questions. Models for economic evaluations like cost-effectiveness analyses (CEAs) synthesize a wide range of evidence, are often statistically and mathematically sophisticated, and require untestable assumptions. As such, there is nearly universal agreement among researchers that enhancing transparency is an important issue in health economic modeling. However, the definition of transparency and guidelines for its implementation vary. Model registration combined with a linked database of model-based economic evaluations has been proposed as a solution, whereby registered models and their accompanying technical and nontechnical documentation are sourced into a single publicly-available repository, ideally in a standardized format to ensure consistent and complete representation of features, code, data sources, results, validation exercises, and policy recommendations. When such a repository is ultimately created, modelers will not have to reinvent the wheel for every new drug launched or new treatment pathway. These more open and transparent approaches will have substantial implications for model accuracy, reliability, and validity, improving trust and acceptance by healthcare decision-makers.
Collapse
Affiliation(s)
- Nadine K Zawadzki
- Schaeffer Center for Health Policy and Economics, Department of Pharmaceutical and Health Economics, School of Pharmacy, University of Southern California, Los Angeles, CA, USA
| | - Joel W Hay
- USC Clinical Economics Research and Education Program (CEREP), Los Angeles, CA, USA.,Schaeffer Center for Health Policy and Economics, Department of Pharmaceutical and Health Economics, School of Pharmacy, University of Southern California, Los Angeles, CA, USA
| |
Collapse
|
6
|
Tappenden P, Caro JJ. Improving Transparency in Decision Models: Current Issues and Potential Solutions. PHARMACOECONOMICS 2019; 37:1303-1304. [PMID: 31642021 DOI: 10.1007/s40273-019-00850-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Affiliation(s)
| | - J Jaime Caro
- Department of Health Policy, The London School of Economics and Political Science, London, UK.
- Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montreal, QC, Canada.
- Evidera, Waltham, MA, USA.
| |
Collapse
|