Weighing Clinical Evidence Using Patient Preferences: An Application of Probabilistic Multi-Criteria Decision Analysis

Personalization of Medical Treatment Decisions: Simplifying Complex Models while Maintaining Patient Health Outcomes

BACKGROUND

Personalizing medical treatments based on patient-specific risks and preferences can improve patient health. However, models to support personalized treatment decisions are often complex and difficult to interpret, limiting their clinical application.

METHODS

We present a new method, using machine learning to create meta-models, for simplifying complex models for personalizing medical treatment decisions. We consider simple interpretable models, interpretable ensemble models, and noninterpretable ensemble models. We use variable selection with a penalty for patient-specific risks and/or preferences that are difficult, risky, or costly to obtain. We interpret the meta-models to the extent permitted by their model architectures. We illustrate our method by applying it to simplify a previously developed model for personalized selection of antipsychotic drugs for patients with schizophrenia.

RESULTS

The best simplified interpretable, interpretable ensemble, and noninterpretable ensemble models contained at most half the number of patient-specific risks and preferences compared with the original model. The simplified models achieved 60.5% (95% credible interval [crI]: 55.2-65.4), 60.8% (95% crI: 55.5-65.7), and 83.8% (95% crI: 80.8-86.6), respectively, of the net health benefit of the original model (quality-adjusted life-years gained). Important variables in all models were similar and made intuitive sense. Computation time for the meta-models was orders of magnitude less than for the original model.

LIMITATIONS

The simplified models share the limitations of the original model (e.g., potential biases).

CONCLUSIONS

Our meta-modeling method is disease- and model- agnostic and can be used to simplify complex models for personalization, allowing for variable selection in addition to improved model interpretability and computational performance. Simplified models may be more likely to be adopted in clinical settings and can help improve equity in patient outcomes.

A comparison of various aggregation functions in multi-criteria decision analysis for drug benefit-risk assessment

Multi-criteria decision analysis is a quantitative approach to the drug benefit-risk assessment which allows for consistent comparisons by summarising all benefits and risks in a single score. The multi-criteria decision analysis consists of several components, one of which is the utility (or loss) score function that defines how benefits and risks are aggregated into a single quantity. While a linear utility score is one of the most widely used approach in benefit-risk assessment, it is recognised that it can result in counter-intuitive decisions, for example, recommending a treatment with extremely low benefits or high risks. To overcome this problem, alternative approaches to the scores construction, namely, product, multi-linear and Scale Loss Score models, were suggested. However, to date, the majority of arguments concerning the differences implied by these models are heuristic. In this work, we consider four models to calculate the aggregated utility/loss scores and compared their performance in an extensive simulation study over many different scenarios, and in a case study. It is found that the product and Scale Loss Score models provide more intuitive treatment recommendation decisions in the majority of scenarios compared to the linear and multi-linear models, and are more robust to the correlation in the criteria.

Partial Personalization of Medical Treatment Decisions: Adverse Effects and Possible Solutions

BACKGROUND

Personalizing medical treatment decisions based on patient-specific risks and/or preferences can improve health outcomes. Decision makers frequently select treatments based on partial personalization (e.g., personalization based on risks but not preferences or vice versa) due to a lack of data about patient-specific risks and preferences. However, partially personalizing treatment decisions based on a subset of patient risks and/or preferences can result in worse population-level health outcomes than no personalization and can increase the variance of population-level health outcomes.

METHODS

We develop a new method for partially personalizing treatment decisions that avoids these problems. Using a case study of antipsychotic treatment for schizophrenia, as well as 4 additional illustrative examples, we demonstrate the adverse effects and our method for avoiding them.

RESULTS

For the schizophrenia treatment case study, using a previously proposed modeling approach for personalizing treatment decisions and using only a subset of patient preferences regarding treatment efficacy and side effects, mean population-level health outcomes decreased by 0.04 quality-adjusted life-years (QALYs; 95% credible interval [crI]: 0.02-0.06) per patient compared with no personalization. Using our new method and considering the same subset of patient preferences, mean population-level health outcomes increased by 0.01 QALYs (95% crI: 0.00-0.03) per patient as compared with no personalization, and the variance decreased.

LIMITATIONS

We assumed a linear and additive utility function.

CONCLUSIONS

Selecting personalized treatments for patients should be done in a way that does not decrease expected population-level health outcomes and does not increase their variance, thereby resulting in worse risk-adjusted, population-level health outcomes compared with treatment selection with no personalization. Our method can be used to ensure this, thereby helping patients realize the benefits of treatment personalization without the potential harms.

Comprehensive Benefit-Risk Assessment of Noninferior Treatments Using Multicriteria Decision Analysis

OBJECTIVES

To develop a simple approach for evaluating the overall benefit-risk of a new noninferiority treatment compared with a standard of care.

METHODS

We propose using multicriteria decision analysis that accounts for uncertainty associated with both clinical outcomes and patient preference data. Because patients' preferences are likely to be influenced by their baseline characteristics, we suggest carrying out a preference study at the beginning of a trial. To reduce the burden of an additional study questionnaire, preference elicitation could be done on a small sample of trial participants. To restore preferences for all trial participants, we propose using multiple imputation (MI). Using simulations, we examine whether 3 different MI procedures lead to the same benefit-risk assessment conclusion, as if all trial participant preferences were obtained. We also compare MI results to complete case analysis, where only preferences of the small sample of trial participants are considered.

RESULTS

We show that the MI procedure successfully restores patients' preferences for the trial participants using different outcome criteria and preferences. For example, using 3 outcome criteria with only 10% of the trial participants providing their preferences, complete case analysis demonstrated a new noninferior treatment as favorable only 5.1% of the time, whereas MI procedures did so between 16.2% and 17.9% of the time. Given that 17.6% correspond to the fully observed weights, the MI methods demonstrate favorable results.

CONCLUSIONS

The MI procedure can help facilitate a simple comprehensive benefit-risk assessment for new noninferior treatments.

Personalizing Medical Treatment Decisions: Integrating Meta-analytic Treatment Comparisons with Patient-Specific Risks and Preferences

Background. Network meta-analyses (NMAs) that compare treatments for a given condition allow physicians to identify which treatments have higher or lower probabilities of reducing the risks of disease complications or increasing the risks of treatment side effects. Translating these data into personalized treatment plans requires integration of NMA data with patient-specific pretreatment risk estimates and preferences regarding treatment objectives and acceptable risks. Methods. We introduce a modeling framework to integrate data probabilistically from NMAs with data on individualized patient risk estimates for disease outcomes, treatment preferences (such as willingness to incur greater side effects for increased life expectancy), and risk preferences. We illustrate the modeling framework by creating personalized plans for antipsychotic drug treatment and evaluating their effectiveness and cost-effectiveness. Results. Compared with treating all patients with the drug that yields the greatest quality-adjusted life-years (QALYs) on average (amisulpride), personalizing the selection of antipsychotic drugs for schizophrenia patients over the next 5 years would be expected to yield 0.33 QALYs (95% credible interval [crI]: 0.30-0.37) per patient at an incremental cost of $4849/QALY gained (95% crI: dominant-$12,357), versus 0.29 and 0.04 QALYs per patient when accounting for only risks or preferences, respectively, but not both. Limitations. The analysis uses a linear, additive utility function to reflect patient treatment preferences and does not consider potential variations in patient time discounting. Conclusions. Our modeling framework rigorously computes what physicians normally have to do mentally. By integrating 3 key components of personalized medicine-evidence on efficacy, patient risks, and patient preferences-the modeling framework can provide personalized treatment decisions to improve patient health outcomes.

A novel measure of drug benefit-risk assessment based on Scale Loss Score

Quantitative methods have been proposed to assess and compare the benefit-risk balance of treatments. Among them, multicriteria decision analysis (MCDA) is a popular decision tool as it permits to summarise the benefits and the risks of a drug in a single utility score, accounting for the preferences of the decision-makers. However, the utility score is often derived using a linear model which might lead to counter-intuitive conclusions; for example, drugs with no benefit or extreme risk could be recommended. Moreover, it assumes that the relative importance of benefits against risks is constant for all levels of benefit or risk, which might not hold for all drugs. We propose Scale Loss Score (SLoS) as a new tool for the benefit-risk assessment, which offers the same advantages as the linear multicriteria decision analysis utility score but has, in addition, desirable properties permitting to avoid recommendations of non-effective or extremely unsafe treatments, and to tolerate larger increases in risk for a given increase in benefit when the amount of benefit is small than when it is high. We present an application to a real case study on telithromycin in Community Acquired Pneumonia and Acute Bacterial Sinusitis, and we investigated the patterns of behaviour of Scale Loss Score, as compared to the linear multicriteria decision analysis, in a comprehensive simulation study.

Multi-criteria decision analysis for health technology assessment: addressing methodological challenges to improve the state of the art

BACKGROUND

Multi-criteria decision analysis (MCDA) concepts, models and tools have been used increasingly in health technology assessment (HTA), with several studies pointing out practical and theoretical issues related to its use. This study provides a critical review of published studies on MCDA in the context of HTA by assessing their methodological quality and summarising methodological challenges.

METHODS

A systematic review was conducted to identify studies discussing, developing or reviewing the use of MCDA in HTA using aggregation approaches. Studies were classified according to publication time and type, country of study, technology type and study type. The PROACTIVE-S approach was constructed and used to analyse methodological quality. Challenges and limitations reported in eligible studies were collected and summarised; this was followed by a critical discussion on research requirements to address the identified challenges.

RESULTS

129 journal articles were eligible for review, 56% of which were published in 2015-2017; 42% focused on pharmaceuticals; 36, 26 and 18% reported model applications, issues regarding MCDA implementation analyses, and proposing frameworks, respectively. Poor compliance with good methodological practice (< 25% complying studies) was found regarding behavioural analyses, discussion of model assumptions and uncertainties, modelling of value functions, and dealing with judgment inconsistencies. The five most reported challenges related to evidence and data synthesis; value system differences and participant selection issues; participant difficulties; methodological complexity and resource balance; and criteria and attributes modelling. A critical discussion on ways to address these challenges ensues.

DISCUSSION

Results highlight the need for advancement in robust methodologies, procedures and tools to improve methodological quality of MCDA in HTA studies. Research pathways include developing new model features, good practice guidelines, technologies to enable participation and behavioural research.

Patient-Focused Drug Development Methods for Benefit-Risk Assessments: A Case Study Using a Discrete Choice Experiment for Antiepileptic Drugs

Regulatory decisions may be enhanced by incorporating patient preferences for drug benefit and harms. This study demonstrates a method of weighting clinical evidence by patients’ benefit–risk preferences. Preference weights, derived from discrete choice experiments, were applied to clinical trial data to estimate the expected utility of alternative drugs. In a case study, the rank ordering of antiepileptic drugs (AEDs), as indicated from clinical studies, was compared with ordering based on weighting clinical evidence by patients’ preferences. A statistically significant change in rank ordering of AEDs was observed for women of childbearing potential who were prescribed monotherapy for generalized or unclassified epilepsy. Rank ordering inferred from trial data, valproate > topiramate > lamotrigine, was reversed. Modeling the expected utility of drugs might address the need to use more systematic, methodologically sound approaches to collect patient input that can further inform regulatory decision making.

AMERICAN ASSOCIATION OF CLINICAL ENDOCRINOLOGISTS AND AMERICAN COLLEGE OF ENDOCRINOLOGY PROTOCOL FOR STANDARDIZED PRODUCTION OF CLINICAL PRACTICE GUIDELINES, ALGORITHMS, AND CHECKLISTS - 2017 UPDATE

Clinical practice guideline (CPG), clinical practice algorithm (CPA), and clinical checklist (CC, collectively CPGAC) development is a high priority of the American Association of Clinical Endocrinologists (AACE) and American College of Endocrinology (ACE). This 2017 update in CPG development consists of (1) a paradigm change wherein first, environmental scans identify important clinical issues and needs, second, CPA construction focuses on these clinical issues and needs, and third, CPG provide CPA node/edge-specific scientific substantiation and appended CC; (2) inclusion of new technical semantic and numerical descriptors for evidence types, subjective factors, and qualifiers; and (3) incorporation of patient-centered care components such as economics and transcultural adaptations, as well as implementation, validation, and evaluation strategies. This third point highlights the dominating factors of personal finances, governmental influences, and third-party payer dictates on CPGAC implementation, which ultimately impact CPGAC development. The AACE/ACE guidelines for the CPGAC program is a successful and ongoing iterative exercise to optimize endocrine care in a changing and challenging healthcare environment.

ABBREVIATIONS

AACE = American Association of Clinical Endocrinologists ACC = American College of Cardiology ACE = American College of Endocrinology ASeRT = ACE Scientific Referencing Team BEL = best evidence level CC = clinical checklist CPA = clinical practice algorithm CPG = clinical practice guideline CPGAC = clinical practice guideline, algorithm, and checklist EBM = evidence-based medicine EHR = electronic health record EL = evidence level G4GAC = Guidelines for Guidelines, Algorithms, and Checklists GAC = guidelines, algorithms, and checklists HCP = healthcare professional(s) POEMS = patient-oriented evidence that matters PRCT = prospective randomized controlled trial.