Technology-enabled assessment of health professions education: consensus statement and recommendations from the Ottawa 2010 Conference

Physical and biophysical markers of assessment in medical training: A scoping review of the literature

PURPOSE

Assessment in medical education has changed over time to measure the evolving skills required of current medical practice. Physical and biophysical markers of assessment attempt to use technology to gain insight into medical trainees' knowledge, skills, and attitudes. The authors conducted a scoping review to map the literature on the use of physical and biophysical markers of assessment in medical training.

MATERIALS AND METHODS

The authors searched seven databases on 1 August 2022, for publications that utilized physical or biophysical markers in the assessment of medical trainees (medical students, residents, fellows, and synonymous terms used in other countries). Physical or biophysical markers included: heart rate and heart rate variability, visual tracking and attention, pupillometry, hand motion analysis, skin conductivity, salivary cortisol, functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS). The authors mapped the relevant literature using Bloom's taxonomy of knowledge, skills, and attitudes and extracted additional data including study design, study environment, and novice vs. expert differentiation from February to June 2023.

RESULTS

Of 6,069 unique articles, 443 met inclusion criteria. The majority of studies assessed trainees using heart rate variability (n = 160, 36%) followed by visual attention (n = 143, 32%), hand motion analysis (n = 67, 15%), salivary cortisol (n = 67, 15%), fMRI (n = 29, 7%), skin conductivity (n = 26, 6%), fNIRs (n = 19, 4%), and pupillometry (n = 16, 4%). The majority of studies (n = 167, 38%) analyzed non-technical skills, followed by studies that analyzed technical skills (n = 155, 35%), knowledge (n = 114, 26%), and attitudinal skills (n = 61, 14%). 169 studies (38%) attempted to use physical or biophysical markers to differentiate between novice and expert.

CONCLUSION

This review provides a comprehensive description of the current use of physical and biophysical markers in medical education training, including the current technology and skills assessed. Additionally, while physical and biophysical markers have the potential to augment current assessment in medical education, there remains significant gaps in research surrounding reliability, validity, cost, practicality, and educational impact of implementing these markers of assessment.

Synthesis and perspectives from the Ottawa 2022 conference on the assessment of competence

INTRODUCTION

The Ottawa Conference on the Assessment of Competence in Medicine and the Healthcare Professions was first convened in 1985 in Ottawa. Since then, what has become known as the Ottawa conference has been held in various locations around the world every 2 years. It has become an important conference for the community of assessment - including researchers, educators, administrators and leaders - to share contemporary knowledge and develop international standards for assessment in medical and health professions education.

METHODS

The Ottawa 2022 conference was held in Lyon, France, in conjunction with the AMEE 2022 conference. A diverse group of international assessment experts were invited to present a symposium at the AMEE conference to summarise key concepts from the Ottawa conference. This paper was developed from that symposium.

RESULTS AND DISCUSSION

This paper summarises key themes and issues that emerged from the Ottawa 2022 conference. It highlights the importance of the consensus statements and discusses challenges for assessment such as issues of equity, diversity, and inclusion, shifts in emphasis to systems of assessment, implications of 'big data' and analytics, and challenges to ensure published research and practice are based on contemporary theories and concepts.

Simulation-based summative assessment in healthcare: an overview of key principles for practice

BACKGROUND

Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start.

METHODS

First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee.

RESULTS

Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted.

CONCLUSION

Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.

Technology enhanced assessment: Ottawa consensus statement and recommendations

INTRODUCTION

In 2011, a consensus report was produced on technology-enhanced assessment (TEA), its good practices, and future perspectives. Since then, technological advances have enabled innovative practices and tools that have revolutionised how learners are assessed. In this updated consensus, we bring together the potential of technology and the ultimate goals of assessment on learner attainment, faculty development, and improved healthcare practices.

METHODS

As a material for the report, we used the scholarly publications on TEA in both HPE and general higher education, feedback from 2020 Ottawa Conference workshops, and scholarly publications on assessment technology practices during the Covid-19 pandemic.

RESULTS AND CONCLUSION

The group identified areas of consensus that remained to be resolved and issues that arose in the evolution of TEA. We adopted a three-stage approach (readiness to adopt technology, application of assessment technology, and evaluation/dissemination). The application stage adopted an assessment 'lifecycle' approach and targeted five key foci: (1) Advancing authenticity of assessment, (2) Engaging learners with assessment, (3) Enhancing design and scheduling, (4) Optimising assessment delivery and recording learner achievement, and (5) Tracking learner progress and faculty activity and thereby supporting longitudinal learning and continuous assessment.

Impact of COVID-19 on digital medical education: compatibility of digital teaching and examinations with integrity and ethical principles

The COVID-19 (coronavirus disease 2019) pandemic in 2020 has had a lasting impact on all areas of personal life. However, the political, economic, legal and healthcare system, as well as the education system have also experienced the effects. Universities had to face new challenges and requirements in teaching and examinations as quickly as possible in order to be able to guarantee high-quality education for their students.

This study aims to examine how the German-speaking medical faculties of the Umbrella Consortium of Assessment Network (UCAN) have dealt with the challenges but also the opportunities that the outbreak of the COVID-19 pandemic created in medical education and whether digitalisation has been driven forward as a result. In an initial online-survey we focused our questions on the current teaching situation with regard to digitised teaching content, the support or establishment of adequate framework conditions by the medical faculties and IT facilities and also the execution of examinations during the summer semester 2020.

Between August and September 2020, a total of 88 examiners, educators, dean of study and/or technical admins from 32 partner faculties took part in the survey. Students were not included in our survey. Most respondents stated that a switch to a digital semester had worked, the use of e-learning increased compared to previous semesters and that most courses could be converted, with the exception of practical courses, which were largely cancelled. The respondents also indicated that most examinations could still be taken, with the exception of practical examination formats, like Objective Structured Clinical Examinations (OSCEs). However, in the case of face-to-face examinations, strict distance and hygiene conditions had to be obeyed or there had to be a switch to distance-online examinations, which raised many open issues such as equal opportunities of students (technical equipment, internet access, premises) and attempts at deception (third-party help with the exam, mutual exchange between students, web search).

In conclusion, we identified several issues regarding the rapid transition to a digital semester due to COVID-19 which were categorised into the following topics: Face-to-face teaching could not take place, (2) know-how of educators, (3) integrity aspects, (4) technical aspects, (5) additional personnel required, (6) additional time and effort required for implementation of digital teaching. Our study shows that a switch to digital teaching and distance online examinations is feasible, but many problems were encountered concerning academic integrity and basic ethical principles still need to be solved. In order to investigate whether above mentioned issued could be solved one year after the transition to a digital semester, we conducted a second survey in which the 32 initially surveyed institutions were questioned again.

Understanding and developing procedures for video-based assessment in medical education

INTRODUCTION

Novel uses of video aim to enhance assessment in health-professionals education. Whilst these uses presume equivalence between video and live scoring, some research suggests that poorly understood variations could challenge validity. We aimed to understand examiners' and students' interaction with video whilst developing procedures to promote its optimal use.

METHODS

Using design-based research we developed theory and procedures for video use in assessment, iteratively adapting conditions across simulated OSCE stations. We explored examiners' and students' perceptions using think-aloud, interviews and focus group. Data were analysed using constructivist grounded-theory methods.

RESULTS

Video-based assessment produced detachment and reduced volitional control for examiners. Examiners ability to make valid video-based judgements was mediated by the interaction of station content and specifically selected filming parameters. Examiners displayed several judgemental tendencies which helped them manage videos' limitations but could also bias judgements in some circumstances. Students rarely found carefully-placed cameras intrusive and considered filming acceptable if adequately justified.

DISCUSSION

Successful use of video-based assessment relies on balancing the need to ensure station-specific information adequacy; avoiding disruptive intrusion; and the degree of justification provided by video's educational purpose. Video has the potential to enhance assessment validity and students' learning when an appropriate balance is achieved.

COVID 19: Designing and conducting an online mini-multiple interview (MMI) in a dynamic landscape

Introduction: The COVID-19 pandemic presented numerous, significant challenges for medical schools, including how to select the best candidates from a pool of applicants when social distancing and other measures prevented "business as usual" admissions processes. However, selection into medical school is the gateway to medicine in many countries, and it is critical to use processes which are evidence-based, valid and reliable even under challenging circumstances. Our challenge was to plan and conduct a multiple-mini interview (MMI) in a dynamic and stringent safe distancing context.Methods: This paper reports a case study of how to plan, re-plan and conduct MMIs in an environment where substantially tighter safe distancing measures were introduced just before the MMI was due to be delivered.Results: We report on how to design and implement a fully remote, online MMI which ensured the safety of candidates and assessors.Discussion: We discuss the challenges of this approach and also reflect on broader issues associated with selection into medical school during a pandemic. The aim of the paper is to provide broadly generalizable guidance to other medical schools faced with the challenge of selecting future students under difficult conditions.

Medical assessment in the age of digitalisation

BACKGROUND

Digital assessment is becoming more and more popular within medical education. To analyse the dimensions of this digital trend, we investigated how exam questions (items) are created and designed for use in digital medical assessments in Germany. Thus, we want to explore whether different types of media are used for item creation and if a digital trend in medical assessment can be observed.

METHODS

In a cross-sectional descriptive study, we examined data of 30 German medical faculties stored within a common assessment platform. More precise, 23,008 exams which contained 847,137 items were analysed concerning the exam type (paper-, computer- or tablet-based) and their respective media content (picture, video and/or audio). Out of these, 5252 electronic exams with 12,214 questions were evaluated. The media types per individual question were quantified.

RESULTS

The amount of computer- and tablet-based exams were rapidly increasing from 2012 until 2018. Computer- and tablet-based written exams showed with 45 and 66% a higher percentage of exams containing media in comparison to paper-based exams (33%). Analysis on the level of individual questions showed that 90.8% of questions had one single picture. The remaining questions contained either more than one picture (2.9%), video (2.7%), audio (0.2%) or 3.3% of questions had picture as well as video added. The main question types used for items with one picture are TypeA (54%) and Long_Menu (31%). In contrast, questions with video content contain only 11% TypeA questions, whereas Long_Menu is represented by 66%. Nearly all questions containing both picture and video are Long_Menu questions.

CONCLUSIONS

It can be stated that digital assessment formats are indeed on the raise. Moreover, our data indicates that electronic assessments formats have easier options to embed media items and thus show a higher frequency of media addition. We even identified the usage of different media types in the same question and this innovative item design could be a useful feature for the creation of medical assessments. Moreover, the choice of media type seems to depend on the respective question type.

Medical assessment in the age of digitalisation

BACKGROUND

Digital assessment is becoming more and more popular within medical education. To analyse the dimensions of this digital trend, we investigated how exam questions (items) are created and designed for use in digital medical assessments in Germany. Thus, we want to explore whether different types of media are used for item creation and if a digital trend in medical assessment can be observed.

METHODS

In a cross-sectional descriptive study, we examined data of 30 German medical faculties stored within a common assessment platform. More precise, 23,008 exams which contained 847,137 items were analysed concerning the exam type (paper-, computer- or tablet-based) and their respective media content (picture, video and/or audio). Out of these, 5252 electronic exams with 12,214 questions were evaluated. The media types per individual question were quantified.

RESULTS

The amount of computer- and tablet-based exams were rapidly increasing from 2012 until 2018. Computer- and tablet-based written exams showed with 45 and 66% a higher percentage of exams containing media in comparison to paper-based exams (33%). Analysis on the level of individual questions showed that 90.8% of questions had one single picture. The remaining questions contained either more than one picture (2.9%), video (2.7%), audio (0.2%) or 3.3% of questions had picture as well as video added. The main question types used for items with one picture are TypeA (54%) and Long_Menu (31%). In contrast, questions with video content contain only 11% TypeA questions, whereas Long_Menu is represented by 66%. Nearly all questions containing both picture and video are Long_Menu questions.

CONCLUSIONS

It can be stated that digital assessment formats are indeed on the raise. Moreover, our data indicates that electronic assessments formats have easier options to embed media items and thus show a higher frequency of media addition. We even identified the usage of different media types in the same question and this innovative item design could be a useful feature for the creation of medical assessments. Moreover, the choice of media type seems to depend on the respective question type.

Balancing Assessment with "In-Service Practical Training": A Case Report on Collaborative Curriculum Design for Delivery in the Practice Setting

Three Higher Education Institutions (HEIs) in Ireland are accredited to provide education and training, successful completion of which, entitles one to register as a pharmacist with the Pharmaceutical Society of Ireland (PSI). Legislation (2014) mandated that these HEIs replace their existing structure (four-year degree followed by a one-year internship), with a five-year ‘integrated Master’s programme’. Integration includes ‘in-service practical training’ (placement) at the beginning of Year 4 (four months), and the end of Year 5 (eight months). Year 4 placements do not have to be ‘patient-facing’. Students receive a Bachelor’s degree at the end of Year 4. The Affiliation for Pharmacy Practice Experiential Learning (APPEL), established by the HEIs, manages student placements, training establishments, preceptor training, the preceptors’ competency assessment process, and the virtual learning environment (VLE) that enables delivery of co-developed online modules aligned with placements in Years 4 and 5. This case report aims to describe the process by which this integration has taken place across and within these HEIs and the challenges faced by educators, students, preceptors, and other stakeholders along the way.

Distance learning in antimicrobial stewardship: innovation in medical education

BACKGROUND

This study aimed to analyse the impact that web-based distance learning has on knowledge gain in medical students, as well as student perceptions of the methodology.

METHODS

This was an educational intervention study conducted at a tertiary teaching hospital in the city of São Paulo, Brazil. From 2008 to 2014, we offered a free web-based distance learning course, covering antimicrobial use and microbial resistance, to fifth-year medical students. The course encompassed 100 h of activities, with five theoretical modules, exercises and simulations, within a virtual learning environment. The students were tutored in their online activities, and some classes were conducted in real time for live discussions. In addition, students underwent face-to-face assessments of their knowledge of the topic before and after the course. Statistical analysis was performed and the means of the overall scores were obtained, as were the respective 95% confidence intervals (CIs). The means were compared by two-tailed paired t-tests and by the paired Wilcoxon test.

RESULTS

Of 814 eligible medical students, 606 (74.45%) completed the entire course during the study period. The mean score for knowledge of the topic was significantly higher on the final assessment than on the initial assessment (p <  0.001). We found that dedication (in hours) was directly proportional to the level of participation, as reflected in the mean final score (p = 0.009) and in the proportion of students who passed (p = 0.028). All of the participants considered their knowledge adequate or insufficient before the course, stating that it is quite important or important to address the topic during medical education. Although dedication levels were low, 70.5% stated that they had learned "quite a lot" or "more than expected" about the topic and would dedicate more time to it if they could.

CONCLUSIONS

The use of a virtual learning environment can promote teaching and learning in the infectious diseases field, specifically for antimicrobial stewardship, increasing knowledge significantly, and should be considered for inclusion in the final stages of medical education.

Simulation versus real-world performance: a direct comparison of emergency medicine resident resuscitation entrustment scoring

Background

Simulation is increasingly being used in postgraduate medical education as an opportunity for competency assessment. However, there is limited direct evidence that supports performance in the simulation lab as a surrogate of workplace-based clinical performance for non-procedural tasks such as resuscitation in the emergency department (ED). We sought to directly compare entrustment scoring of resident performance in the simulation environment to clinical performance in the ED.

Methods

The resuscitation assessment tool (RAT) was derived from the previously implemented and studied Queen's simulation assessment tool (QSAT) via a modified expert review process. The RAT uses an anchored global assessment scale to generate an entrustment score and narrative comments. Emergency medicine (EM) residents were assessed using the RAT on cases in simulation-based examinations and in the ED during resuscitation cases from July 2016 to June 2017. Resident mean entrustment scores were compared using Pearson's correlation coefficient to determine the relationship between entrustment in simulation cases and in the ED. Inductive thematic analysis of written commentary was conducted to compare workplace-based with simulation-based feedback.

Results

There was a moderate, positive correlation found between mean entrustment scores in the simulated and workplace-based settings, which was statistically significant (r = 0.630, n = 17, p < 0.01). Further, qualitative analysis demonstrated overall management and leadership themes were more common narratives in the workplace, while more specific task-based feedback predominated in the simulation-based assessment. Both workplace-based and simulation-based narratives frequently commented on communication skills.

Conclusions

In this single-center study with a limited sample size, assessment of residents using entrustment scoring in simulation settings was demonstrated to have a moderate positive correlation with assessment of resuscitation competence in the workplace. This study suggests that resuscitation performance in simulation settings may be an indicator of competence in the clinical setting. However, multiple factors contribute to this complicated and imperfect relationship. It is imperative to consider narrative comments in supporting the rationale for numerical entrustment scores in both settings and to include both simulation and workplace-based assessment in high-stakes decisions of progression.

An Assessment of Pathology Resident Access to and Use of Technology: A Nationwide Survey

Context.—

Current technologies including digital slide scanners and handheld devices can revolutionize clinical practice and pathology graduate medical education (GME). The extent to which these technologies are used in pathology GME is unknown.

Objectives.—

To determine the types of technologies used, usage amount, and how they are integrated into pathology residency/fellowship programs nationwide.

Design.—

A 40-question online survey for residents/fellows was developed and administered via the Research Electronic Data Capture System after institutional review board approval.

Results.—

Fifty-two program directors (37%) gave permission for participation. One-hundred seventy-one responses were received (18% response rate). Most respondents have access to personal technology (laptop = 78% [134 of 171]), smartphone = 81% [139 of 171], tablet = 49% [84 of 171]), and Web-based digital slide collections (82%, 141 of 171). Few residents are provided electronic devices by their programs (laptop = 22% [38 of 171], smartphone = 0.5% [1 of 171], and tablet = 12% [21 of 171]). Fifty-nine percent have access to digital slide scanners, 33% have access to a program-created database of digitized slides, and 52% use telepathology. Fifteen percent have access to asynchronous learning. Of those with access to video-recorded conferences, 89% review them. Program size was significantly positively correlated with resident access to program-provided laptops (P = .02) and tablets (P &lt; .001), digital slide scanners (P = .01), and telepathology (P = .001). Of all devices, program-provided laptops are used most for professional work (60.5% use this device for more than 5 hours per day).

Conclusions.—

Most residents report access to multiple types of innovative technology, but incorporation of these tools within pathology training programs is highly variable. Opportunities for incorporating innovative technologies exist and could be further explored.

Validation of educational assessments: a primer for simulation and beyond

BACKGROUND

Simulation plays a vital role in health professions assessment. This review provides a primer on assessment validation for educators and education researchers. We focus on simulation-based assessment of health professionals, but the principles apply broadly to other assessment approaches and topics.

KEY PRINCIPLES

Validation refers to the process of collecting validity evidence to evaluate the appropriateness of the interpretations, uses, and decisions based on assessment results. Contemporary frameworks view validity as a hypothesis, and validity evidence is collected to support or refute the validity hypothesis (i.e., that the proposed interpretations and decisions are defensible). In validation, the educator or researcher defines the proposed interpretations and decisions, identifies and prioritizes the most questionable assumptions in making these interpretations and decisions (the "interpretation-use argument"), empirically tests those assumptions using existing or newly-collected evidence, and then summarizes the evidence as a coherent "validity argument." A framework proposed by Messick identifies potential evidence sources: content, response process, internal structure, relationships with other variables, and consequences. Another framework proposed by Kane identifies key inferences in generating useful interpretations: scoring, generalization, extrapolation, and implications/decision. We propose an eight-step approach to validation that applies to either framework: Define the construct and proposed interpretation, make explicit the intended decision(s), define the interpretation-use argument and prioritize needed validity evidence, identify candidate instruments and/or create/adapt a new instrument, appraise existing evidence and collect new evidence as needed, keep track of practical issues, formulate the validity argument, and make a judgment: does the evidence support the intended use?

CONCLUSIONS

Rigorous validation first prioritizes and then empirically evaluates key assumptions in the interpretation and use of assessment scores. Validation science would be improved by more explicit articulation and prioritization of the interpretation-use argument, greater use of formal validation frameworks, and more evidence informing the consequences and implications of assessment.

Key Points to Facilitate the Adoption of Computer-Based Assessments

There are strong pedagogical arguments in favor of adopting computer-based assessment. The risks of technical failure can be managed and are offset by improvements in cost-effectiveness and quality assurance capability. Academic, administrative, and technical leads at an appropriately senior level within an institution need to be identified, so that they can act as effective advocates. All stakeholder groups need to be represented in undertaking a detailed appraisal of requirements and shortlisting software based on core functionality, summative assessment life cycle needs, external compatibility, security, and usability. Any software that is a candidate for adoption should be trialed under simulated summative conditions, with all stakeholders having a voice in agreeing the optimum solution. Transfer to a new system should be carefully planned and communicated, with a programme of training established to maximize the success of adoption.

Electronic acquisition of OSCE performance using tablets

BACKGROUND

Objective Structured Clinical Examinations (OSCEs) often involve a considerable amount of resources in terms of materials and organization since the scores are often recorded on paper. Computer-assisted administration is an alternative with which the need for material resources can be reduced. In particular, the use of tablets seems sensible because these are easy to transport and flexible to use.

AIM

User acceptance concerning the use of tablets during OSCEs has not yet been extensively investigated. The aim of this study was to evaluate tablet-based OSCEs from the perspective of the user (examiner) and the student examinee.

METHOD

For two OSCEs in Internal Medicine at the University of Heidelberg, user acceptance was analyzed regarding tablet-based administration (satisfaction with functionality) and the subjective amount of effort as perceived by the examiners. Standardized questionnaires and semi-standardized interviews were conducted (complete survey of all participating examiners). In addition, for one OSCE, the subjective evaluation of this mode of assessment was gathered from a random sample of participating students in semi-standardized interviews.

RESULTS

Overall, the examiners were very satisfied with using tablets during the assessment. The subjective amount of effort to use the tablet was found on average to be "hardly difficult". The examiners identified the advantages of this mode of administration as being in particular the ease of use and low rate of error. During the interviews of the examinees, acceptance for the use of tablets during the assessment was also detected.

DISCUSSION

Overall, it was found that the use of tablets during OSCEs was well accepted by both examiners and examinees. We expect that this mode of assessment also offers advantages regarding assessment documentation, use of resources, and rate of error in comparison with paper-based assessments; all of these aspects should be followed up on in further studies.

Validation of the one pass measure for motivational interviewing competence

OBJECTIVE

This paper examines the psychometric properties of the OnePass coding system: a new, user-friendly tool for evaluating practitioner competence in motivational interviewing (MI). We provide data on reliability and validity with the current gold-standard: Motivational Interviewing Treatment Integrity tool (MITI).

METHODS

We compared scores from 27 videotaped MI sessions performed by student counselors trained in MI and simulated patients using both OnePass and MITI, with three different raters for each tool. Reliability was estimated using intra-class coefficients (ICCs), and validity was assessed using Pearson's r.

RESULTS

OnePass had high levels of inter-rater reliability with 19/23 items found from substantial to almost perfect agreement. Taking the pair of scores with the highest inter-rater reliability on the MITI, the concurrent validity between the two measures ranged from moderate to high. Validity was highest for evocation, autonomy, direction and empathy.

CONCLUSION

OnePass appears to have good inter-rater reliability while capturing similar dimensions of MI as the MITI. Despite the moderate concurrent validity with the MITI, the OnePass shows promise in evaluating both traditional and novel interpretations of MI.

PRACTICE IMPLICATIONS

OnePass may be a useful tool for developing and improving practitioner competence in MI where access to MITI coders is limited.

The value of online learning and MRI: finding a niche for expensive technologies

BACKGROUND

The benefits of online learning come at a price. How can we optimize the overall value?

AIMS

Critically appraise the value of online learning.

METHODS

Narrative review.

RESULTS

Several prevalent myths overinflate the value of online learning. These include that online learning is cheap and easy (it is usually more expensive), that it is more efficient (efficiency depends on the instructional design, not the modality), that it will transform education (fundamental learning principles have not changed), and that the Net Generation expects it (there is no evidence of pent-up demand). However, online learning does add real value by enhancing flexibility, control and analytics. Costs may also go down if disruptive innovations (e.g. low-cost, low-tech, but instructionally sound "good enough" online learning) supplant technically superior but more expensive online learning products. Cost-lowering strategies include focusing on core principles of learning rather than technologies, using easy-to-learn authoring tools, repurposing content (organizing and sequencing existing resources rather than creating new content) and using course templates.

CONCLUSIONS

Online learning represents just one tool in an educator's toolbox, as does the MRI for clinicians. We need to use the right tool(s) for the right learner at the right dose, time and route.

Advance care planning norms may contribute to hospital variation in end-of-life ICU use: a simulation study

BACKGROUND

There is wide variation in end-of-life (EOL) intensive care unit (ICU) use among academic medical centers (AMCs). Our objective was to develop hypotheses regarding medical decision-making factors underlying this variation.

METHODS

This was a high-fidelity simulation experiment involving a critically and terminally ill elder, followed by a survey and debriefing cognitive interview and evaluated using triangulated quantitative-qualitative comparative analysis. The study was conducted in 2 AMCs in the same state and health care system with disparate EOL ICU use. Subjects were hospital-based physicians responsible for ICU admission decisions. Measurements included treatment plan, prognosis, diagnosis, qualitative case perceptions, and clinical reasoning.

RESULTS

Sixty-seven of 111 (60%) eligible physicians agreed to participate; 48 (72%) could be scheduled. There were no significant between-AMC differences in 3-month prognosis or treatment plan, but there were systematic differences in perceptions of the case. Case perceptions at the low-intensity AMC seemed to be influenced by the absence of a do-not-resuscitate order in the context of norms of universal code status discussion and documentation upon admission, whereas case perceptions at the high-intensity AMC seemed to be influenced by the patient's known metastatic gastric cancer in the context of norms of oncologists' avoiding code status discussions.

CONCLUSIONS

In this simulation study of 2 AMCs, hospital-based physicians had different perceptions of an identical case. We hypothesize that different advance care planning norms may have influenced their decision-making heuristics.

Reconsidering fidelity in simulation-based training

In simulation-based health professions education, the concept of simulator fidelity is usually understood as the degree to which a simulator looks, feels, and acts like a human patient. Although this can be a useful guide in designing simulators, this definition emphasizes technological advances and physical resemblance over principles of educational effectiveness. In fact, several empirical studies have shown that the degree of fidelity appears to be independent of educational effectiveness. The authors confronted these issues while conducting a recent systematic review of simulation-based health professions education, and in this Perspective they use their experience in conducting that review to examine key concepts and assumptions surrounding the topic of fidelity in simulation.Several concepts typically associated with fidelity are more useful in explaining educational effectiveness, such as transfer of learning, learner engagement, and suspension of disbelief. Given that these concepts more directly influence properties of the learning experience, the authors make the following recommendations: (1) abandon the term fidelity in simulation-based health professions education and replace it with terms reflecting the underlying primary concepts of physical resemblance and functional task alignment; (2) make a shift away from the current emphasis on physical resemblance to a focus on functional correspondence between the simulator and the applied context; and (3) focus on methods to enhance educational effectiveness using principles of transfer of learning, learner engagement, and suspension of disbelief. These recommendations clarify underlying concepts for researchers in simulation-based health professions education and will help advance this burgeoning field.

Good exams made easy: the item management system for multiple examination formats

BACKGROUND

The development, implementation and evaluation of assessments require considerable resources and often cannot be carried out by a single faculty/institution. Therefore some medical faculties have founded cooperation projects which mainly focus on the exchange of multiple choice questions (MCQs).

METHODS

Since these cooperation projects do not entirely support all relevant processes in terms of preparation, implementation and evaluation of assessment, in 2006 the Medical Assessment Alliance (MAA) was founded for mutual support. In addition to MCQs the MAA started to develop innovative assessment formats and facilitate content through a coordinated exchange of experiences. To support cooperation within this network, the web-based Item Management System (IMS) was developed which supports all processes of the assessment workflow as an all-in-one working platform.

RESULTS

At present, the Alliance has 28 partner faculties in Europe. More than 2.800 users in 750 working groups are collaborating. Currently 90.000 questions have been stored in the IMS. Since 2007, nearly 4.600 examinations have been successfully conducted.

CONCLUSION

This article describes in detail the unique features of the IMS and contrasts it with the item management systems of other associations.

Rater training to support high-stakes simulation-based assessments

Competency-based assessment and an emphasis on obtaining higher-level outcomes that reflect physicians' ability to demonstrate their skills has created a need for more advanced assessment practices. Simulation-based assessments provide medical education planners with tools to better evaluate the 6 Accreditation Council for Graduate Medical Education (ACGME) and American Board of Medical Specialties (ABMS) core competencies by affording physicians opportunities to demonstrate their skills within a standardized and replicable testing environment, thus filling a gap in the current state of assessment for regulating the practice of medicine. Observational performance assessments derived from simulated clinical tasks and scenarios enable stronger inferences about the skill level a physician may possess, but also introduce the potential of rater errors into the assessment process. This article reviews the use of simulation-based assessments for certification, credentialing, initial licensure, and relicensing decisions and describes rater training strategies that may be used to reduce rater errors, increase rating accuracy, and enhance the validity of simulation-based observational performance assessments.