The challenge of obtaining information necessary for multi-criteria decision analysis implementation: the case of physiotherapy services in Canada

Development of a Prioritization Framework to Aid Healthcare Funding Decision Making in Health Technology Assessment in Australia: Application of Multicriteria Decision Analysis

OBJECTIVES

This study develops a prioritization framework to aid healthcare funding decision making in health technology assessment (HTA) in Australia using a multiple criteria decision analysis (MCDA) approach.

METHODS

MCDA frameworks for HTAs were reviewed through literature survey to identify the initial criteria and levels within each criterion. Key stakeholders and experts were consulted to confirm these criteria and levels. A conjoint analysis using 1000Minds was undertaken with policy makers from the Department of Health to establish ranking criteria and weighting scores. Monte Carlo simulations were used to examine the sensitivity of findings to factors affecting the ranking and weighting scores. The MCDA was then applied to 6 examples of chronic care models or technologies projects to demonstrate the performance of this approach.

RESULTS

Five criteria (clinical efficacy/effectiveness, safety and tolerability, severity of the condition, quality/uncertainty, and direct impact on healthcare costs) were consistently ranked highest by healthcare decision makers. Among the criteria, patient-level health outcomes were considered the most important, followed by social and ethical values. The analyses were robust to inform the uncertainty in the parameter.

CONCLUSIONS

This study has developed an MCDA tool that effectively integrates key priorities for HTA reviews, reflecting the values and preferences of healthcare stakeholders in Australia. Although this tool aims to align the assessment process more closely with health benefits, it also highlights the importance of considering other criteria.

Application of Multi-Criteria Decision Analysis (MCDA) to Prioritize Real-World Evidence Studies for Health Technology Management: Outcomes and Lessons Learned by the Canadian Real-World Evidence for Value of Cancer Drugs (CanREValue) Collaboration

Multi-criteria decision analysis (MCDA) is a value assessment tool designed to help support complex decision-making by incorporating multiple factors and perspectives in a transparent, structured approach. We developed an MCDA rating tool, consisting of seven criteria evaluating the importance and feasibility of conducting potential real-world evidence (RWE) studies aimed at addressing uncertainties stemming from initial cancer drug funding recommendations. In collaboration with the Canadian Agency for Drugs and Technologies in Health's Provincial Advisory Group, a validation exercise was conducted to further evaluate the application of the rating tool using RWE proposals varying in complexity. Through this exercise, we aimed to gain insight into consensus building and deliberation processes and to identify efficiencies in the application of the rating tool. An experienced facilitator led a multidisciplinary committee, consisting of 11 Canadian experts, through consensus building, deliberation, and prioritization. A total of nine RWE proposals were evaluated and prioritized as low (n = 4), medium (n = 3), or high (n = 2) priority. Through an iterative process, efficiencies and recommendations to improve the rating tool and associated procedures were identified. The refined MCDA rating tool can help decision-makers prioritize important and feasible RWE studies for research and can enable the use of RWE for the life-cycle evaluation of cancer drugs.

Supporting rehabilitation stakeholders in making service delivery decisions: a rapid review of multi-criteria decision analysis methods

PURPOSE

This review aimed to synthesize knowledge about multi-criteria decision analysis methods for supporting rehabilitation service design and delivery decisions, including: (1) describing the use of these methods within rehabilitation, (2) identifying decision types that can be supported by these methods, (3) describing client and family involvement, and (4) identifying implementation considerations.

METHODS

We conducted a rapid review in collaboration with a knowledge partner, searching four databases for peer-reviewed articles reporting primary research. We extracted relevant data from included studies and synthesized it descriptively and with conventional content analysis.

RESULTS

We identified 717 records, of which 54 met inclusion criteria. Multi-criteria decision analysis methods were primarily used to understand the strength of clients' and clinicians' preferences (n = 44), and five focused on supporting decision making. Shared decision making with stakeholders was evident in only two studies. Clients and families were mostly engaged in data collection and sometimes in selecting the relevant criteria. Good practices for supporting external validity were inconsistently reported. Implementation considerations included managing cognitive complexity and offering authentic choices.

CONCLUSIONS

Multi-criteria decision analysis methods are promising for better understanding client and family preferences and priorities across rehabilitation professions, contexts, and caseloads. Further work is required to use these methods in shared decision making, for which increased use of qualitative methods and stakeholder engagement is recommended. IMPLICATIONS FOR REHABILITATIONMulti-criteria decision analysis methods are promising for evidence-based, shared decision making for rehabilitation.However, most studies to date have focused on estimating stakeholder preferences, not supporting shared decision making.Cognitive complexity and modelling authentic and realistic decision choices are major barriers to implementation.Stakeholder-engagement and qualitative methods are recommended to address these barriers.

Multi-criteria decision analysis for health technology assessment: addressing methodological challenges to improve the state of the art

BACKGROUND

Multi-criteria decision analysis (MCDA) concepts, models and tools have been used increasingly in health technology assessment (HTA), with several studies pointing out practical and theoretical issues related to its use. This study provides a critical review of published studies on MCDA in the context of HTA by assessing their methodological quality and summarising methodological challenges.

METHODS

A systematic review was conducted to identify studies discussing, developing or reviewing the use of MCDA in HTA using aggregation approaches. Studies were classified according to publication time and type, country of study, technology type and study type. The PROACTIVE-S approach was constructed and used to analyse methodological quality. Challenges and limitations reported in eligible studies were collected and summarised; this was followed by a critical discussion on research requirements to address the identified challenges.

RESULTS

129 journal articles were eligible for review, 56% of which were published in 2015-2017; 42% focused on pharmaceuticals; 36, 26 and 18% reported model applications, issues regarding MCDA implementation analyses, and proposing frameworks, respectively. Poor compliance with good methodological practice (< 25% complying studies) was found regarding behavioural analyses, discussion of model assumptions and uncertainties, modelling of value functions, and dealing with judgment inconsistencies. The five most reported challenges related to evidence and data synthesis; value system differences and participant selection issues; participant difficulties; methodological complexity and resource balance; and criteria and attributes modelling. A critical discussion on ways to address these challenges ensues.

DISCUSSION

Results highlight the need for advancement in robust methodologies, procedures and tools to improve methodological quality of MCDA in HTA studies. Research pathways include developing new model features, good practice guidelines, technologies to enable participation and behavioural research.

Effects of waiting for outpatient physiotherapy services in persons with musculoskeletal disorders: a systematic review

PURPOSE

This systematic review aimed to assess the scientific evidence on the effects of waiting for outpatient physiotherapy services in persons with musculoskeletal disorders.

METHODS

A literature search was conducted in three databases (Medline, CINAHL, and Embase) for articles assessing the effects of waiting for outpatient physiotherapy services in persons with musculoskeletal disorders. Clinical and health system outcomes were analyzed.

RESULTS

Sixteen studies met the inclusion criteria for this review. The studies varied in designs, settings, and populations. The definition of waiting also varied between studies. The studies were of low to high methodological quality. Waiting for outpatient physiotherapy services was shown to have mixed results on clinical and health system outcomes. Results from included studies suggest the possible detrimental effects of waiting on pain, disability, quality of life, and psychological symptoms in persons with musculoskeletal disorders. There was also evidence of higher healthcare utilization and costs for patients who wait longer before physiotherapy services.

CONCLUSIONS

This review provides mixed evidence that suggest potential detrimental effects on the health of individuals with MSDs and at the health system level. Further high-quality studies are needed, such as longitudinal studies specifically addressing the effects of waiting due to lack of access to physiotherapy services.IMPLICATIONS FOR REHABILITATIONThe findings from this review suggest potential detrimental effects on health outcomes when patients wait longer before receiving physiotherapy services.The findings also suggest higher healthcare utilization and costs for patients with longer wait times compared to those who receive physiotherapy services more rapidly.This review suggests the need to assess and implement strategies and policies to ensure timely access to physiotherapy.

General group exercise in low back pain management in a military population, a comparison with specific spine group exercise: a service evaluation

Objective

To investigate whether general group exercise (GGE) offers the same outcomes compared with a specific spinal group exercise (SSGE) for chronic low back pain (CLBP) in a military population.

Design

Retrospective service evaluation using routine service activity data.

Setting

A UK military rehabilitation centre.

Participants

A total of 106 patients with CLBP.

Interventions

Three-week intensive (5 days per week, 15-day intervention) rehabilitation course for patients with CLBP. Six SSGE groups (n=64); CLBP only. Six GGE groups (n=42); CLBP patients grouped with chronic lower limb (LL) injuries.

Outcome measures

Oswestry Disability Index (ODI), Numerical Pain-Rating Scores and the Modified Multi-Stage Fitness Test (Mod-MSFT). Long-term effects were measured by Medical Employment Standard (MES) status and physiotherapy follow-up at 3 and 12 months.

Results

A between-group analysis showed no significant difference in GGE compared with SSGE. Mean changes (SD) in pain were −2.71±2.35 and −1.20±1.99 (p=0.018), ODI were −3.6±5.7 and −4±8.5 respectively (p=0.649) and Mod-MSFT 28.4±30.8 and 29.7±31.7 respectively (p=0.792). At 3 months, a greater proportion of the GGE were having ongoing physiotherapy; GGE=50%, SSGE=30.2%, (p=0.016) although some differences were evident across MES with 32.5 % of GGE compared with 20.6 % of SSGE being medically fit with no restrictions. At 12 months, groups were largely comparable for follow-up physiotherapy and MES; 22.5% of GGE and 20.6% of SSGE continued to have physiotherapy input; 47.5% of GGE and 50.8% of SSGE were medically fit with no restrictions.

Conclusion

Patients with CLBP who completed a 3-week rehabilitation programme had comparable outcomes when grouped with patients with LL, although only improvements in pain in the GGE group achieved a meaningful change. Further evaluation of potential costs and savings to service costs is now required.

How have systematic priority setting approaches influenced policy making? A synthesis of the current literature

BACKGROUND

There is a growing body of literature on systematic approaches to healthcare priority setting from various countries and different levels of decision making. This paper synthesizes the current literature in order to assess the extent to which program budgeting and marginal analysis (PBMA), burden of disease & cost-effectiveness analysis (BOD/CEA), multi-criteria decision analysis (MCDA), and accountability for reasonableness (A4R), are reported to have been institutionalized and influenced policy making and practice.

METHODS

We searched for English language publications on health care priority setting approaches (2000-2017). Our sources of literature included PubMed and Ovid databases (including Embase, Global Health, Medline, PsycINFO, EconLit).

FINDINGS

Of the four approaches PBMA and A4R were commonly applied in high income countries while BOD/CEA was exclusively applied in low income countries. PBMA and BOD/CEA were most commonly reported to have influenced policy making. The explanations for limited adoption of an approach were related to its complexity, poor policy maker understanding and resource requirements.

CONCLUSIONS

While systematic approaches have the potential to improve healthcare priority setting; most have not been adopted in routine policy making. The identified barriers call for sustained knowledge exchange between researchers and policy-makers and development of practical guidelines to ensure that these frameworks are more accessible, applicable and sustainable in informing policy making.

A prospective 2-site parallel intervention trial of a research-based film to increase exercise amongst older hemodialysis patients

Background

Evidence suggests that exercise training for hemodialysis patients positively improves morbidity and mortality outcomes, yet exercise programs remain rare and are not systematically incorporated into care. We developed a research-based film, Fit for Dialysis, designed to introduce, motivate, and sustain exercise for wellness amongst older hemodialysis patients, and exercise counseling and support by nephrologists, nurses, and family caregivers. The objective of this clinical trial is to determine whether and in what ways Fit for Dialysis improves outcomes and influences knowledge/attitudes regarding the importance of exercise for wellness in the context of end-stage renal disease.

Methods/Design

This 2-site parallel intervention trial will recruit 60 older hemodialysis patients from two urban hospitals. The trial will compare the film + a 16-week exercise program in one hospital, with a 16-week exercise-only program in another hospital. Physical fitness and activity measures will be performed at baseline, 8 and 16 weeks, and 12 weeks after the end of the program. These include the 2-min Walk Test, Grip Strength, Duke Activity Status Index, and the Timed Up-and-Go Test, as well as wearing a pedometer for one week. Throughout the 16-week exercise program, and at 12 weeks after, we will record patients’ exercise using the Godin Leisure-time Exercise Questionnaire. Patients will also keep a diary of the exercise that they do at home on non-dialysis days. Qualitative interviews, conducted at baseline, 8, and 16 weeks, will explore the impact of Fit for Dialysis on the knowledge/attitudes of patients, family caregivers, and nephrology staff regarding exercise for wellness, and in what ways the film is effective in educating, motivating, or sustaining patient exercise during dialysis, at home, and in the community.

Discussion

This research will determine for whom Fit for Dialysis is effective, why, and under what conditions. If Fit for Dialysis is proven beneficial to patients, nephrology staff and family caregivers, research-based film as a model to support exercise promotion and adherence could be used to support the National Kidney Foundation’s guideline recommendation (NKF-KDOQI) that exercise be incorporated into the care and treatment of dialysis patients.

Trial registration

NCT02754271 (ClinicalTrials.gov), retroactively registered on April 21, 2016.

Using clinical practice variations as a method for commissioners and clinicians to identify and prioritise opportunities for disinvestment in health care: a cross-sectional study, systematic reviews and qualitative study

Background

NHS expenditure has stagnated since the economic crisis of 2007, resulting in financial pressures. One response is for policy-makers to regulate use of existing health-care technologies and disinvest from inefficiently used health technologies. A key challenge to disinvestment is to identify existing health technologies with uncertain cost-effectiveness.

Objectives

We aimed to explore if geographical variation in procedure rates is a marker of clinical uncertainty and might be used by local commissioners to identify procedures that are potential candidates for disinvestment. We also explore obstacles and solutions to local commissioners achieving disinvestment, and patient and clinician perspectives on regulating access to procedures.

Methods

We used Hospital Episode Statistics to measure geographical variation in procedure rates from 2007/8 to 2011/12. Expected procedure numbers for each primary care trust (PCT) were calculated adjusting for proxies of need. Random effects Poisson regression quantified the residual inter-PCT procedure rate variability. We benchmarked local procedure rates in two PCTs against national rates. We conducted rapid systematic reviews of two high-use procedures selected by the PCTs [carpal tunnel release (CTR) and laser capsulotomy], searching bibliographical databases to identify systematic reviews and randomised controlled trials (RCTs). We conducted non-participant overt observations of commissioning meetings and semistructured interviews with stakeholders about disinvestment in general and with clinicians and patients about one disinvestment case study. Transcripts were analysed thematically using constant comparison methods derived from grounded theory.

Results

There was large inter-PCT variability in procedure rates for many common NHS procedures. Variation in procedure rates was highest where the diffusion or discontinuance was rapidly evolving and where substitute procedures were available, suggesting that variation is a proxy for clinical uncertainty about appropriate use. In both PCTs we identified procedures where high local use might represent an opportunity for disinvestment. However, there were barriers to achieving disinvestment in both procedure case studies. RCTs comparing CTR with conservative care indicated that surgery was clinically effective and cost-effective on average but provided limited evidence on patient subgroups to inform commissioning criteria and achieve savings. We found no RCTs of laser capsulotomy. The apparently high rate of capsulotomy was probably due to the coding inaccuracy; some savings might be achieved by greater use of outpatient procedures. Commissioning meetings were dominated by new funding requests. Benchmarking did not appear to be routinely carried out because of capacity issues and concerns about data reliability. Perceived barriers to disinvestment included lack of collaboration, central support and tools for disinvestment. Clinicians felt threshold criteria had little impact on their practice and that prior approval systems would not be cost-effective. Most patients were unaware of rationing.

Conclusions

Policy-makers could use geographical variation as a starting point to identify procedures where health technology reassessment or RCTs might be needed to inform policy. Commissioners can use benchmarking to identify procedures with high local use, possibly indicating overtreatment. However, coding inconsistency and limited evidence are major barriers to achieving disinvestment through benchmarking. Increased central support for commissioners to tackle disinvestment is needed, including tools, accurate data and relevant evidence. Early engagement with patients and clinicians is essential for successful local disinvestment.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

What constitutes high performance in priority setting and resource allocation? Decision maker narratives identified from a survey and qualitative study in Canadian healthcare organizations

Priority setting and resource allocation are key management functions; however, there may be different understandings as to what makes for a high-performing organization in this area. To interpret how decision makers actually approach this question, our research looks at what might contribute to one's reputation as such. Two sets of qualitative data are used. Senior healthcare leaders were asked to nominate organizations which they considered high performers in priority setting and resource allocation and to justify their choices. This open-ended question was analyzed to identify themes. Rigorous process was most often cited. Six case studies were subsequently conducted; respondents were asked to comment upon why they thought their organization might be named by others as a high performer. These replies were analyzed qualitatively to identify prominent storylines: three distinctive narratives are summarized here. These help us to understand how organization leaders in particular contexts bring together stakeholders to pursue locally appropriate strategies for achieving contextually defined high performance.

Balancing efficiency, equity and feasibility of HIV treatment in South Africa - development of programmatic guidance

South Africa, the country with the largest HIV epidemic worldwide, has been scaling up treatment since 2003 and is rapidly expanding its eligibility criteria. The HIV treatment programme has achieved significant results, and had 1.8 million people on treatment per 2011. Despite these achievements, it is now facing major concerns regarding (i) efficiency: alternative treatment policies may save more lives for the same budget; (ii) equity: there are large inequalities in who receives treatment; (iii) feasibility: still only 52% of the eligible population receives treatment.Hence, decisions on the design of the present HIV treatment programme in South Africa can be considered suboptimal. We argue there are two fundamental reasons to this. First, while there is a rapidly growing evidence-base to guide priority setting decisions on HIV treatment, its included studies typically consider only one criterion at a time and thus fail to capture the broad range of values that stakeholders have. Second, priority setting on HIV treatment is a highly political process but it seems no adequate participatory processes are in place to incorporate stakeholders' views and evidences of all sorts.We propose an alternative approach that provides a better evidence base and outlines a fair policy process to improve priority setting in HIV treatment. The approach integrates two increasingly important frameworks on health care priority setting: accountability for reasonableness (A4R) to foster procedural fairness, and multi-criteria decision analysis (MCDA) to construct an evidence-base on the feasibility, efficiency, and equity of programme options including trade-offs. The approach provides programmatic guidance on the choice of treatment strategies at various decisions levels based on a sound conceptual framework, and holds large potential to improve HIV priority setting in South Africa.