AMEE Guide No. 123 - How to read studies of educational costs

Cost identification in health professions education: A scoping review

BACKGROUND

Resource scarcity in health professions education (HPE) demands rigorous consideration of costs. Yet, thus far, we have been unable to completely and consistently identify the costs of HPE. To address this knowledge gap and enable use of economic evidence in decision making, a detailed overview of all existing costs and approaches to cost identification in HPE is needed.

OBJECTIVES

This review summarises the diversity of costs, cost perspectives and cost identification methods used in economic research of HPE to answer the following questions: which educational topics, costs, cost perspectives and cost identification methods are being investigated in HPE literature?

METHODS

This investigation followed the Joanna Briggs Institute guidelines for scoping reviews. PubMed, ERIC, CINAHL and PsycINFO were iteratively searched for English language publications between 2012 and 2022 that reported costs of HPE. Data was extracted for study characteristics, educational context and economic methodology.

RESULTS

A total of 136 articles of original research on the costs of educating all major licence healthcare professionals were included. A diffuse interest in costs is reflected in publications from 93 distinct journals and by the diversity of educational topics and cost types explored. However, the majority of investigations failed to define the cost perspective (86, 63%) and cost identification methods (117, 86%) and did not demonstrate common reporting structure or cost terminology, all of which contribute to persistent inconsistencies in cost identification.

CONCLUSION

The source of disharmonious cost identification in HPE is fuelled by diverse interests and divergent concepts of costs and costing methodology. Left in its current state, disharmonious cost identification will continue to limit transparency, comparison and synthesis of evidence, hamper objectivity in implementation and policy decisions and impede the efficient and sustainable allocation of resources. A research agenda must be developed, prioritised, and validated by the broader community to develop a theoretical framework for HPE cost research, define stakeholders, elicit values and preferences and chart a path toward harmonised costing.

Identifying costs in health professions education: a scoping review protocol

INTRODUCTION

Profound changes in the healthcare sector and the growing, global shortage of healthcare workers are driving the demand to update and scale-up health professions education (HPE). At the same time, educational institutions are faced with increasing resource scarcity and decisions of where and how to allocate resources must be made conscious of costs. However, costs are known to be inaccurately and incompletely identified in HPE research. Therefore, to improve rigour in cost evidence and enable its use in decision-making, it is necessary to have a better understanding of the costs, stakeholders and cost identification methods being used in HPE research. This protocol has been developed to outline the rationale and methods for a scoping review of the literature intended to map the current state of cost and cost identification evidence in HPE.

METHODS AND ANALYSIS

This protocol is developed in accordance with the Joanna Briggs Institute Manual for Evidence Synthesis. PubMed, ERIC, CINAHL and PsycInfo will be scoped for all types of English language publications from 2012 to 2022 that investigate costs associated with the education of health professionals. Two independent reviewers will assess publications for eligibility. Data will be extracted on the educational topic and context, the costs, cost types, perspectives and methods of cost identification as represented in the literature.

ETHICS AND DISSEMINATION

No human participants are involved in this scoping review. All evidence for this study are sourced from public databases; therefore, ethical approval was not required. Final results will be disseminated through peer-reviewed publications and presented at conferences.

TRIAL REGISTRATION NUMBER

A preliminary version of this protocol was prospectively registered on Open Science Framework (OSF) on 13 June 2022 and is available at https://doi.org/10.17605/OSF.IO/ET4XB.

Evaluating education innovations rapidly with build-measure-learn: Applying lean startup to health professions education

PURPOSE

The Lean Startup approach allows innovators (including innovative educators) to rapidly identify and refine promising ideas into models that actually work. Our aim is to outline key principles of Lean Startup, apply these to health professions education, and illustrate these using personal experience.

METHODS AND RESULTS

All innovations are grounded in numerous assumptions; these assumptions should be explicitly identified, prioritized, and empirically tested ('validated learning'). To identify and test assumptions, innovators need to get out of the office and interact with customers (learners, teachers, administrators, etc). Assumptions are tested using multiple quick cycles of Build (a 'minimal viable product' [MVP]), Measure (using metrics that relate meaningfully to the assumption), and Learn (interpret data and decide to persevere with further refinements, or pivot to a new direction). The MVP is a product version that allows testing of one or more key assumptions with the least effort. We describe a novel 'Lean Education Canvas' that synopsizes an innovation and its business model on one page, to help identify assumptions and monitor progress. We illustrate these principles using three cases from health professions education.

CONCLUSIONS

Lean Startup has tremendous potential for rapid, robust innovation and evaluation in education.[Box: see text].

Economic evaluation of CPD activities for healthcare professionals: A scoping review

CONTEXT

Continuing professional development (CPD) activities for healthcare professionals are central to the optimisation of patient safety and person-centred care. Although there is some evidence on the economics of healthcare professionals training, very little is known about the costs and benefits of CPD.

METHODS

This study aimed to review the research evidence on economic evaluations of CPD activities for healthcare professionals. CINAHL, MEDLINE/PubMed, Scopus, Econlit and Web of Science databases were used to identify articles published between 2010 and 2021.

RESULTS

Of the 6791 titles identified, 119 articles met the inclusion criteria and were included in this scoping review. The majority of articles were partial economic evaluations of CPD programmes (n = 70); half were from the USA. Studies that included multiple professions were most prevalent (n = 54), followed by nurses (n = 34) and doctors (n = 23). Patient outcomes were the most commonly reported outcome (n = 51), followed by change in clinical practice (n = 38) and healthcare professionals' knowledge gain (n = 19).

CONCLUSIONS

There is an urgent call for more evidence regarding the economic evaluations of CPD. This is particularly important in view of the rising costs of healthcare globally. The majority of studies included in this review did not provide detailed information on the evaluations and many focused exclusively on the cost of CPD activities rather than outcomes.

Costs of Physician Continuous Professional Development: A Systematic Review

PURPOSE

An essential yet oft-neglected step in cost evaluations is the selection of resources (ingredients) to include in cost estimates. The ingredients that most influence the cost of physician continuous professional development (CPD) are unknown, as are the relative costs of instructional modalities. This study's purpose was to estimate the costs of cost ingredients and instructional modalities in physician CPD.

METHOD

The authors conducted a systematic review in April 2020, searching MEDLINE, Embase, PsycInfo, and the Cochrane Library for comparative cost evaluations of CPD for practicing physicians. Two reviewers, working independently, screened articles for inclusion and extracted information on costs (converted to 2021 U.S. dollars) for each intervention overall, each ingredient, and each modality.

RESULTS

Of 3,338 eligible studies, 62 were included, enumerating costs for 86 discrete training interventions or instructional modalities. The most frequently reported ingredients were faculty time (25 of 86 interventions), materials (24), administrator/staff time (23), and travel (20). Ingredient costs varied widely, ranging from a per-physician median of $4 for postage (10 interventions) to $525 for learner time (13); equipment (9) and faculty time were also relatively expensive (median > $170). Among instructional modalities (≤ 11 interventions per modality), audit and feedback performed by physician learners, computer-based modules, computer-based virtual patients, in-person lectures, and experiences with real patients were relatively expensive (median > $1,000 per physician). Mailed paper materials, video clips, and audit and feedback performed by others were relatively inexpensive (median ≤ $62 per physician). Details regarding ingredient selection (10 of 62 studies), quantitation (10), and pricing (26) were reported infrequently.

CONCLUSIONS

Some ingredients, including time, are more important (i.e., contribute more to total costs) than others and should be prioritized in cost evaluations. Data on the relative costs of instructional modalities are insightful but limited. The methods and reporting of cost valuations merit improvement.

"Important but risky": attitudes of global thought leaders towards cost and value research in health professions education

Studies of cost and value can inform educational decision making, yet our understanding of the barriers to such research is incomplete. To address this gap, our aim was to explore the attitudes of global thought leaders in HPE towards cost and value research. This was a qualitative virtual interview study underpinned by social constructionism. In telephone or videoconference interviews in 2018-2019, we asked global healthcare professional thought leaders their views regarding HPE cost and value research, outstanding research questions in this area and why addressing these questions was important. Analysis was inductive and thematic, and incorporated review and comments from the original interviewees (member checking). We interviewed 11 thought leaders, nine of whom gave later feedback on our data interpretation (member checking). We identified four themes: Cost research is really important but potentially risky (quantifying and reporting costs provides evidence for decision-making but could lead to increased accountability and loss of autonomy); I don't have the knowledge and skills (lack of economic literacy); it's not what I went into education research to do (professional identity); and it's difficult to generate generalizable findings (the importance of context). This study contributes to a wider conversation in the literature about cost and value research by bringing in the views of global HPE thought leaders. Our findings provide insight to inform how best to engage and empower educators and researchers in the processes of asking and answering meaningful, acceptable and relevant cost and value questions in HPE.

Quality of cost evaluations of physician continuous professional development: Systematic review of reporting and methods

INTRODUCTION

We sought to evaluate the reporting and methodological quality of cost evaluations of physician continuing professional development (CPD).

METHODS

We conducted a systematic review, searching MEDLINE, Embase, PsycInfo, and the Cochrane Database for studies comparing the cost of physician CPD (last update 23 April 2020). Two reviewers, working independently, screened all articles for inclusion. Two reviewers extracted information on reporting quality using the Consolidated Health Economic Evaluation Reporting Standards (CHEERS), and on methodological quality using the Medical Education Research Study Quality Instrument (MERSQI) and a published reference case.

RESULTS

Of 3338 potentially eligible studies, 62 were included. Operational definitions of methodological and reporting quality elements were iteratively revised. Articles reported mean (SD) 43% (20%) of CHEERS elements for the Title/Abstract, 56% (34%) for Introduction, 66% (19%) for Methods, 61% (17%) for Results, and 66% (30%) for Discussion, with overall reporting index 292 (83) (maximum 500). Valuation methods were reported infrequently (resource selection 10 of 62 [16%], resource quantitation 10 [16%], pricing 26 [42%]), as were descriptions/discussion of the physicians trained (42 [68%]), training setting (42 [68%]), training intervention (40 [65%]), sensitivity analyses of uncertainty (9 [15%]), and generalizability (30 [48%]). MERSQI scores ranged from 6.0 to 16.0 (mean 11.2 [2.4]). Changes over time in reporting index (initial 241 [105], final 321 [52]) and MERSQI scores (initial 9.8 [2.7], final 11.9 [1.9]) were not statistically significant (p ≥ 0.08).

DISCUSSION

Methods and reporting of HPE cost evaluations fall short of current standards. Gaps exist in the valuation, analysis, and contextualization of cost outcomes.

Effects of basic endoscopic handling and care training on gastrointestinal endoscopy logistics

Background and study aims  The current practice of endoscopists is undergoing a dramatic revolution due to emerging endoscopy practices. Increasing use of gastrointestinal endoscopy has led to hospital budgets setting aside funds specifically related to damage to endoscopic instruments. Therefore, training in understanding endoscopic equipment, handling techniques, and equipment care can be helpful in addressing this issue. The aim of this study was to investigate the effects of educational courses and training about basic endoscopic handling and care in gastrointestinal endoscopic care and services. Methods  A number of new endoscopists, nurses, and nurse assistants were enrolled in a course for training in basic endoscopic handling and care. Data on the type of damage, cause, cost, and timing of endoscopic repair were prospectively collected. Data from the post-training period then were compared with retrospective data from the pre-training period. Results  This study demonstrated that after training, there was less damage to endoscopes, lower costs associated with it, and repair times were shorter for endoscopes than before the training course. Post-training results indicated savings of a total of $ 40,617.21 or £ 29,539.78 and 102.6 days per damaged endoscope. Conclusions  Basic endoscopic handling and care training plays an important role for both endoscopists and nurses, as well as in endoscopy facilities, specifically in avoiding the nuisance of unwanted and broken endoscopes. This could be beneficial for both hospital finances and endoscopic services.

Cost-effectiveness and Economic Benefit of Continuous Professional Development for Drug Prescribing: A Systematic Review

Importance

The economic impact of continuous professional development (CPD) education is incompletely understood.

Objective

To systematically identify and synthesize published research examining the costs associated with physician CPD for drug prescribing.

Evidence Review

MEDLINE, Embase, PsycInfo, and the Cochrane Database were searched from inception to April 23, 2020, for comparative studies that evaluated the cost of CPD focused on drug prescribing. Two reviewers independently screened all articles for inclusion and reviewed all included articles to extract data on participants, educational interventions, study designs, and outcomes (costs and effectiveness). Results were synthesized for educational costs, health care costs, and cost-effectiveness.

Findings

Of 3338 articles screened, 38 were included in this analysis. These studies included at least 15 659 health care professionals and 1 963 197 patients. Twelve studies reported on educational costs, ranging from $281 to $183 554 (median, $15 664). When economic outcomes were evaluated, 31 of 33 studies (94%) comparing CPD with no intervention found that CPD was associated with reduced health care costs (drug costs), ranging from $4731 to $6 912 000 (median, $79 373). Four studies found reduced drug costs for 1-on-1 outreach compared with other CPD approaches. Regarding cost-effectiveness, among 5 studies that compared CPD with no intervention, the incremental cost-effectiveness ratio for a 10% improvement in prescribing ranged from $15 390 to $437 027 to train all program participants. Four comparisons of alternative CPD approaches found that 1-on-1 educational outreach was more effective but more expensive than group education or mailed materials (incremental cost-effectiveness ratio, $18-$4105 per physician trained).

Conclusions and Relevance

In this systematic review, CPD for drug prescribing was associated with reduced health care (drug) costs. The educational costs and cost-effectiveness of CPD varied widely. Several CPD instructional approaches (including educational outreach) were more effective but more costly than comparators.

How to conduct cost and value analyses in health professions education: AMEE Guide No. 139

Growing demand for accountability, transparency, and efficiency in health professions education is expected to drive increased demand for, and use of, cost and value analyses. In this AMEE Guide, we introduce key concepts, methods, and literature that will enable novices in economics to conduct simple cost and value analyses, hold informed discussions with economic specialists, and undertake further learning on more advanced economic topics. The practical structure for conducting analyses provided in this guide will enable researchers to produce robust results that are meaningful and useful for improving educational practice. Key steps include defining the economic research question, identifying an appropriate economic study design, carefully identifying cost ingredients, quantifying, and pricing the ingredients consumed, and conducting sensitivity analyses to explore uncertainties in the results.

Spending Wisely: The Role of Cost and Value Research in the Pursuit of Advancing Anatomical Sciences Education

Studies of "cost and value" in anatomical sciences education examine not only what works, but at what cost, thus evaluating the inputs and outputs of education. This research provides insights into how to use available resources (e.g., academic time, budgets, infrastructure) as a mechanism to obtaining the maximum outcomes available. The purpose of this viewpoint article is to expand on the application of cost and value concepts to anatomical sciences education, contextualizing these concepts through a deeper dive into the more costly educational approaches of human donor dissection. In doing so, both questions and opportunities are raised for the discipline of anatomical sciences going forward. Educational decisions, inclusive of cost and value appraisals, consider the range of outcomes for which the activity is designed to achieve, and the activity's integration with the philosophy of the educational program it is contributing to; these decisions, thus, evaluate more than just cost alone. Healthcare students' engagement with human donor dissection pedagogy offers an array of reported non-economic benefits, including non-traditional discipline-independent skill (NDIS) development (e.g., professionalism, teamwork skills). These skills are often harder to measure, but are no less important to the final pedagogical decision-making process. The goal of cost and value research is to create an evidence-base toward education that delivers maximum value for a given spend. Anatomy educators, researchers, and decision makers who embrace cost and value dialogue, and interpret and apply findings from studies of educational costs, are best positioned to improve the educational value for their learners and provide effective outputs for all stakeholders.

Efficiency of Increasing Prospective Resident Entrustment in the Operating Room

BACKGROUND

Prospective resident entrustment (i.e., trust an attending surgeon intends to give to a resident in the near future) in the operating room (OR) closely associates with granted future autonomy. However, the process of determining resident entrustment takes time and effort. Thus, this study aimed to assess the efficiency of granting incremental resident entrustment for upcoming surgical cases.

METHODS

We analyzed prospective resident entrustment of 6 chief residents in 76 cases of laparoscopic cholecystectomy, laparoscopic colectomy, ventral hernia, and inguinal hernia scored by attending surgeon, resident, and a surgeon observer. Matched direct costs and operative time were extracted from hospital billing. We assessed the efficiency of granting incremental prospective resident entrustment with direct cost per minute incurred in the evaluated case. Effect size was computed to assess the differences between groups.

RESULTS

Sixty-three cases (82.9%) were matched; 47.6% (30/63) of matched cases received prospective resident entrustment score ≥ 4. The direct cost per minute increased in three procedures (laparoscopic cholecystectomy, laparoscopic colectomy, and ventral hernia) with increased intention of granting incremental resident entrustment. Inguinal hernia was the only procedure in which chiefs were entrusted with future independence while the direct cost per minute decreased.

CONCLUSIONS

Our findings demonstrate more time and effort are required (except for inguinal hernia) for residents to be entrusted with increased independence in the future. Faculty and resident development programs are recommended to improve the efficiency of the process of granting incremental operative entrustment to optimize resident training quality and cost of care delivery.

"You can't always get what you want…": economic thinking, constrained optimization and health professions education

Every choice we make in health professions education has a cost, whether it be financial or otherwise; by choosing one action (e.g., integrating more simulation, studying more for a summative examination) we lose the opportunity to take an alternative action (e.g., freeing up time for other teaching, leisure time). Economics significantly shapes the way we behave and think as educators and learners and so there is increasing interest in using economic ways of thinking and approaches to examine and understand how choices are made, the influence of constraints and boundaries in educational decision making, and how costs are felt. Thus, in this article, we provide a brief historical overview of modern economics, to illustrate how the core concepts of economics-scarcity (and desirability), rationality, and optimization-developed over time. We explain the important concept of bounded rationality, which explains how individual, meso-factors and contextual factors influence decision making. We then consider the opportunities that these concepts afford for health professions education and research. We conclude by proposing that embracing economic thinking opens up new questions and new ways of approaching old questions which can add knowledge about how choice is enacted in contemporary health professions education.

The economic costs of selecting medical students: An Australian case study

CONTEXT

The design of selection methods must balance, amongst a range of factors, the desire to select the best possible future doctors with the reality of our resource-constrained environment. Examining the cost of selection processes enables us to identify areas in which efficiencies may be gained.

METHODS

A cost description study was conducted based on selection for 2018 entry into medical school directly from secondary school. The perspectives of applicants, volunteer interviewers and the admitting institution were considered. Costs were modelled based on the Monash University (Australia) selection process, which uses a combination of secondary school matriculation score, aptitude test score (Undergraduate Medicine and Health Sciences Admission Test) and multiple mini-interview score. A variety of data sources were utilised, including bespoke surveys, audit data and existing literature. All costs are expressed in 2018 Australian dollars (AU$). Applicant behaviours in preparing for selection tests were also evaluated.

RESULTS

A total of 381 of 383 applicants returned the survey. Over 70% of applicants had utilised commercial preparation materials. The median total cost to applicants was AU$2586 (interquartile range [IQR] AU$1574-3999), including costs to both prepare for and attend selection tests. Of 217 volunteer interviewers, 108 returned the survey. These were primarily health professional clinicians at a mid-career stage. The median total cost to interviewers was AU$452 (IQR AU$252-715) for participation in a half-day interview session, largely due to the loss of income. The cost to the admitting institution was AU$269 per applicant, accounted for by the costs of equipment and consumables (52%), personnel (34%) and facilities (14%).

CONCLUSIONS

The costs of student selection for medical school are substantial. Understanding costs facilitates achievement of the objective of selecting the desired future medical workforce within the constraints of the resources available. Opportunities for change may arise from changes in applicant preparation behaviours, opportunities for economies of scale, and efficiencies driven by technological solutions.

COVID 19: Designing and conducting an online mini-multiple interview (MMI) in a dynamic landscape

Introduction: The COVID-19 pandemic presented numerous, significant challenges for medical schools, including how to select the best candidates from a pool of applicants when social distancing and other measures prevented "business as usual" admissions processes. However, selection into medical school is the gateway to medicine in many countries, and it is critical to use processes which are evidence-based, valid and reliable even under challenging circumstances. Our challenge was to plan and conduct a multiple-mini interview (MMI) in a dynamic and stringent safe distancing context.Methods: This paper reports a case study of how to plan, re-plan and conduct MMIs in an environment where substantially tighter safe distancing measures were introduced just before the MMI was due to be delivered.Results: We report on how to design and implement a fully remote, online MMI which ensured the safety of candidates and assessors.Discussion: We discuss the challenges of this approach and also reflect on broader issues associated with selection into medical school during a pandemic. The aim of the paper is to provide broadly generalizable guidance to other medical schools faced with the challenge of selecting future students under difficult conditions.

Managing the tension: From innovation to application in health professions education

Innovations in education are essential for solving problems and introducing new ways of thinking. However, implementation of these innovations must take several factors into consideration, including the context, the environment, the stakeholders, the technology needed, the cost, the pace of implementation, appropriateness, and available resources. When these factors are not balanced and considered, tensions arise. This paper describes tensions in five major educational domains, namely curriculum, instruction, assessment, accreditation/regulation, and faculty development. For each of these domains, the typical problems are described, and solutions are presented to manage the tension.

Increasing value in research: cost evaluations in health professions education

The authors use Foo et al.'s discussion of the value of economic evaluations to consider how such techniques might advance the practice of selection for medical school.

Cost evaluations in health professions education: a systematic review of methods and reporting quality

CONTEXT

High-quality research into education costs can inform better decision making. Improvements to cost research can be guided by information about the research questions, methods and reporting of studies evaluating costs in health professions education (HPE). Our objective was to appraise the overall state of the field and evaluate temporal trends in the methods and reporting quality of cost evaluations in HPE research.

METHODS

We searched the MEDLINE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), EMBASE, Business Source Complete and ERIC (Education Resources Information Centre) databases on 31 July 2017. To evaluate trends over time, we sampled research reports at 5-year intervals (2001, 2006, 2011 and 2016). All original research studies in HPE that reported a cost outcome were included. The Medical Education Research Study Quality Instrument (MERSQI) and the BMJ economic checklist were used to appraise methodological and reporting quality, respectively. Trends in quality over time were analysed.

RESULTS

A total of 78 studies were included, of which 16 were published in 2001, 15 in 2006, 20 in 2011 and 27 in 2016. The region most commonly represented was the USA (n = 43). The profession most commonly referred to was that of the physician (n = 46). The mean ± standard deviation (SD) MERSQI score was 10.9 ± 2.6 out of 18, with no significant change over time (p = 0.55). The mean ± SD BMJ score was 13.5 ± 7.1 out of 35, with no significant change over time (p = 0.39). A total of 49 (63%) studies stated a cost-related research question, 23 (29%) stated the type of cost evaluation used, and 31 (40%) described the method of estimating resource quantities and unit costs. A total of 16 studies compared two or more interventions and reported both cost and learning outcomes.

CONCLUSIONS

The absolute number of cost evaluations in HPE is increasing. However, there are shortcomings in the quality of methodology and reporting, and these are not improving over time.

These may not be the courses you are seeking: a systematic review of open online courses in health professions education

INTRODUCTION

Open Online Courses (OOCs) are increasingly presented as a possible solution to the many challenges of higher education. However, there is currently little evidence available to support decisions around the use of OOCs in health professions education. The aim of this systematic review was to summarise the available evidence describing the features of OOCs in health professions education and to analyse their utility for decision-making using a self-developed framework consisting of point scores around effectiveness, learner experiences, feasibility, pedagogy and economics.

METHODS

Electronic searches of PubMed, Medline, Embase, PsychInfo and CINAHL were made up to April 2019 using keywords related to OOC variants and health professions. We accepted any type of full text English publication with no exclusions made on the basis of study quality. Data were extracted using a custom-developed, a priori critical analysis framework comprising themes relating to effectiveness, economics, pedagogy, acceptability and learner experience.

RESULTS

54 articles were included in the review and 46 were of the lowest levels of evidence, and most were offered by institutions based in the United States (n = 11) and United Kingdom (n = 6). Most studies provided insufficient course detail to make any confident claims about participant learning, although studies published from 2016 were more likely to include information around course aims and participant evaluation. In terms of the five categories identified for analysis, few studies provided sufficiently robust evidence to be used in formal decision making in undergraduate or postgraduate curricula.

CONCLUSION

This review highlights a poor state of evidence to support or refute claims regarding the effectiveness of OOCs in health professions education. Health professions educators interested in developing courses of this nature should adopt a critical and cautious position regarding their adoption.