The utility of mini-Clinical Evaluation Exercise in undergraduate and postgraduate medical education: A BEME review: BEME Guide No. 59

Authentic assessment in medical education: exploring AI integration and student-as-partners collaboration

BACKGROUND

Traditional assessments often lack flexibility, personalized feedback, real-world applicability, and the ability to measure skills beyond rote memorization. These may not adequately accommodate diverse learning styles and preferences, nor do they always foster critical thinking or creativity. The inclusion of Artificial Intelligence (AI), especially Generative Pre-trained Transformers, in medical education marks a significant shift, offering both exciting opportunities and notable challenges for authentic assessment practices. Various fields, including anatomy, physiology, pharmacy, dentistry, and pathology, are anticipated to employ the metaverse for authentic assessments increasingly. This innovative approach will likely enable students to engage in immersive, project-based learning experiences, facilitating interdisciplinary collaboration and providing a platform for real-world application of knowledge and skills.

METHODS

This commentary paper explores how AI, authentic assessment, and Student-as-Partners (SaP) methodologies can work together to reshape assessment practices in medical education.

RESULTS

The paper provides practical insights into effectively utilizing AI tools to create authentic assessments, offering educators actionable guidance to enhance their teaching practices. It also addresses the challenges and ethical considerations inherent in implementing AI-driven assessments, emphasizing the need for responsible and inclusive practices within medical education. Advocating for a collaborative approach between AI and SaP methodologies, the commentary proposes a robust plan to ensure ethical use while upholding academic integrity.

CONCLUSION

Through navigating emerging assessment paradigms and promoting genuine evaluation of medical knowledge and proficiency, this collaborative effort aims to elevate the quality of medical education and better prepare learners for the complexities of clinical practice.

Enhancing clinical skills in pediatric trainees: a comparative study of ChatGPT-assisted and traditional teaching methods

BACKGROUND

As artificial intelligence (AI) increasingly integrates into medical education, its specific impact on the development of clinical skills among pediatric trainees needs detailed investigation. Pediatric training presents unique challenges which AI tools like ChatGPT may be well-suited to address.

OBJECTIVE

This study evaluates the effectiveness of ChatGPT-assisted instruction versus traditional teaching methods on pediatric trainees' clinical skills performance.

METHODS

A cohort of pediatric trainees (n = 77) was randomly assigned to two groups; one underwent ChatGPT-assisted training, while the other received conventional instruction over a period of two weeks. Performance was assessed using theoretical knowledge exams and Mini-Clinical Evaluation Exercises (Mini-CEX), with particular attention to professional conduct, clinical judgment, patient communication, and overall clinical skills. Trainees' acceptance and satisfaction with the AI-assisted method were evaluated through a structured survey.

RESULTS

Both groups performed similarly in theoretical exams, indicating no significant difference (p > 0.05). However, the ChatGPT-assisted group showed a statistically significant improvement in Mini-CEX scores (p < 0.05), particularly in patient communication and clinical judgment. The AI-teaching approach received positive feedback from the majority of trainees, highlighting the perceived benefits in interactive learning and skill acquisition.

CONCLUSION

ChatGPT-assisted instruction did not affect theoretical knowledge acquisition but did enhance practical clinical skills among pediatric trainees. The positive reception of the AI-based method suggests that it has the potential to complement and augment traditional training approaches in pediatric education. These promising results warrant further exploration into the broader applications of AI in medical education scenarios.

Evaluation of the Workplace-Based Assessment Anaesthesia-Clinical Evaluation Exercise (A-CEX) and Its Role in the Royal College of Anaesthetists 2021 Curriculum

The workplace-based assessment (WPBA) Anaesthesia-Clinical Evaluation Exercise (A-CEX) is used in anaesthetic training in the Royal College of Anaesthetists 2021 curriculum. WBPAs are part of a multimodal approach to assess competencies, but can be limited by their granularity. They are an essential component of assessment and are used in both a formative and summative capacity. The A-CEX is a form of WBPA which evaluates knowledge, behaviours and skill of anaesthetists in training across a variety of 'real world' situations. An entrustment scale is assigned to the evaluation which has implications for future practice and ongoing supervision requirements. Despite being a key component in the curriculum the A-CEX has drawbacks. Its qualitative nature results in variation in feedback provided amongst assessors, which may have ongoing implications for clinical practice. Furthermore, the completion of an A-CEX can be viewed as a 'tick box' exercise and does not guarantee that learning has taken place.  Currently no direct evidence exists as to the benefit of the A-CEX in anaesthetic training, but extrapolated data from other studies may show validity. However, the assessment remains a key part of the 2021 curriculum, Future areas for consideration include education for those assessing trainees via A-CEX, altering the matrix of assessment to a less granular approach and a longitudinal study as to the utility of A-CEX in anaesthetics training.

Assessing electrodiagnostic skills among residents and fellows: Relationships between workplace-based assessments using the Electromyography Direct Observation Tool and other measures of trainee performance

INTRODUCTION/AIMS

Graduate medical education programs must ensure residents and fellows acquire skills needed for independent practice. Workplace-based observational assessments are informative but can be time- and resource-intensive. In this study we sought to gather "relations-to-other-variables" validity evidence for scores generated by the Electromyography Direct Observation Tool (EMG-DOT) to inform its use as a measure of electrodiagnostic skill acquisition.

METHODS

Scores on multiple assessments were compiled by trainees during Clinical Neurophysiology and Electromyography rotations at a large US academic medical center. Relationships between workplace-based EMG-DOT scores (n = 298) and scores on a prerequisite simulated patient exercise, patient experience surveys (n = 199), end-of-rotation evaluations (n = 301), and an American Association of Neuromuscular & Electrodiagnostic Medicine (AANEM) self-assessment examination were assessed using Pearson correlations.

RESULTS

Among 23 trainees, EMG-DOT scores assigned by physician raters correlated positively with end-of-rotation evaluations (r = 0.63, P = .001), but EMG-DOT scores assigned by technician raters did not (r = 0.10, P = .663). When physician and technician ratings were combined, higher EMG-DOT scores correlated with better patient experience survey scores (r = 0.42, P = .047), but not with simulated patient or AANEM self-assessment examination scores.

DISCUSSION

End-of-rotation evaluations can provide valid assessments of trainee performance when completed by individuals with ample opportunities to directly observe trainees. Inclusion of observational assessments by technicians and patients provides a more comprehensive view of trainee performance. Workplace- and classroom-based assessments provide complementary information about trainee performance, reflecting underlying differences in types of skills measured.

[Contribution and limits of "OSCE", "long-case" and "global end-of-placement marking" as end-of-rotation assessment methods. Experience from two internal medicine wards]

INTRODUCTION

During placements, there is an opportunity to learn clinical skills and to assess their application. However, it represents two different goals. The validity of an end-of-placement assessment is questionable, as the medical competency is contextual. We decided to evaluate the contribution and limits of different assessment modalities as an end-of-placement assessment.

MATERIAL AND METHODS

Internal medicine clerks were assessed using the Mini-Cex grid by a structured objective clinical examination (OSCE), a long-case clinical examination (LCE) and a global end-of-placement marking (GEPM). Following these evaluations, students and teachers fulfilled an open questionnaire.

RESULTS

In 2021, 41 students and 16 teachers participated in the study. Physical examination was evaluated in 0%, 97% et 76% of cases during OSCE, LCE and GEPM, respectively; teaching skills were assessed for 100, 42 et 49% of students in OSCE, LCE and GEPM, respectively. As compared to OSCE, there was a perceived superiority of LCE regarding its formative value (P=0.07 and P=0.03) and its summative value (P=0.0007 and P=0.02), for students and teachers, respectively. Qualitative analysis highlights the breadth of clinical skills that could be assessed during OSCE stations. Integration into a team was an additional skill that could specifically be assessed during GEPM. GEPM could also take into account the progress made during placement.

CONCLUSION

Despite its subjectivity, LCE seemed to be the preferred modality for an end-of-rotation assessment.

Developing the Occupational Therapy-Specific Mini-Clinical Evaluation Exercise (Mini-CEX) for Evaluating Interns' Clinical Skills and Attitudes in Pediatric Occupational Therapy

IMPORTANCE

The Mini-Clinical Evaluation Exercise (Mini-CEX) is highly recommended for assessing interns' performance.

OBJECTIVE

To develop a pediatric occupational therapy-specific Mini-CEX and examine its psychometrics.

DESIGN

Stage 1 had a retrospective design; Stage 2 had a prospective design.

SETTING

Pediatric occupational therapy unit in a hospital in Taiwan.

PARTICIPANTS

Thirty-four occupational therapy interns were evaluated with the Mini-CEX (physician version), and 57 were evaluated with the occupational therapy-specific Mini-CEX.

OUTCOMES AND MEASURES

The occupational therapy-specific Mini-CEX was developed with seven items on a 9-point scale categorized into three levels (unsatisfactory, satisfactory, highly satisfactory).

RESULTS

In Stage 1, the frequency of Mini-CEX (physician version) items receiving a rating of not applicable ranged from 1.9% to 88.1%. In Stage 2, the frequency of occupational therapy-specific Mini-CEX items receiving a rating of not applicable ranged from 3.5% to 31.6%. With the theme of evaluation taken into consideration, the frequency of not-applicable ratings was 0% to 8.8%. For the occupational therapy-specific Mini-CEX, content validity (item-level content validity index = 1, scale-level content validity index = 1) and internal consistency (Cronbach's α = .93) were excellent. The interns' scores on the second evaluation were significantly higher than those on their first evaluation, indicating good discriminant validity.

CONCLUSIONS AND RELEVANCE

The occupational therapy-specific Mini-CEX appears to be reliable and valid, and it is appropriate for evaluating interns' skills and attitudes in pediatric occupational therapy practice. What This Article Adds: The results support the development of the occupational therapy-specific Mini-CEX and its application in pediatric internship training.

Design, implementation and evaluation of a virtual clinical training protocol for midwifery internship in a gynecology course during COVID-19 pandemic: A semi-experimental study

BACKGROUND

Over the past year, the occurrence of COVID-19 pandemic has challenged clinical education for health care students, due to the possibility of exposure to the virus and increased spread of the disease. Clinical training of midwifery students in gynecologic problems, based on the Iran midwifery education curriculum, was also disrupted during this pandemic.

OBJECTIVES

This study was aimed at designing, implementing and evaluating a virtual clinical training protocol for midwifery internship in a Gynecology course.

DESIGN

A semi-experimental study.

SETTINGS

Faculty of Nursing and Midwifery, Mazandaran University of Medical Sciences, Sari, Iran.

PARTICIPANTS

Forty-seven midwifery interns in Gynecology course were recruited during two semesters in 2020.

METHODS

Five steps based on the Analysis, Design, Development, Implementation, and Evaluation (ADDIE) model were taken, which included 1) educational and skills needs assessment, 2) design, 3) development via focused group interviews and brainstorming with the presence of the midwifery department members in three sessions, 4) implementation including pretest and posttest, webinar, uploading the information of virtual patients, questions and correct answers, and 5) evaluation including knowledge assessment by a designed questionnaire and skills evaluation by the modified-Mini-CEX checklist. Data were analyzed using mean, standard deviation and paired t-test.

RESULTS

After training, a significant increase (p < 0.001) was observed in scores of knowledge and interview skills, clinical judgment, consultation, efficiency, professionalism, clinical competence and total score of clinical skills.

CONCLUSIONS

Training for gynecological diseases through virtual clinic promoted knowledge and clinical skills of midwifery interns. To enhance education, a virtual clinic may be used in crisis situations and in combination with teaching under normal circumstances by strengthening the infrastructure and removing barriers.

Interrater reliability in the assessment of physiotherapy students

BACKGROUND

Reliable and objective assessment of psychomotor skills in physiotherapy students' education is essential for direct feedback and skill improvement. The aim of this study is to determine the interrater reliability in the assessment process of physiotherapy students and to analyse the assessment behaviour of the examiners.

METHODS

Physiotherapy teachers from two different schools assessed students from two different schools performing proprioceptive neuromuscular facilitation (PNF) patterns. An evaluation sheet with a 6-point rating scale and 20 evaluation criteria including an overall rating was used for assessment. The interrater reliability was determined calculating an intraclass-correlation coefficient (ICC) and Krippendorff's alpha. The assessment behaviour of the examiners was further analysed calculating the location parameters and showing the item response distribution over item in form of a Likert plot.

RESULTS

The ICC estimates were mostly below 0.4, indicating poor interrater reliability. This was confirmed by Krippendorff's alpha. The examiners showed a certain central tendency and intergroup bias.

DISCUSSION AND CONCLUSION

The interrater reliability in this assessment format was rather low. No difference between the two physiotherapy schools concerning the interrater reliability could be identified. Despite certain limitations of this study, there is a definite need for improvement of the assessment process in physiotherapy education to provide the students with reliable and objective feedback and ensure a certain level of professional competence in the students.

TRIAL REGISTRATION

The study was approved by the ethics committee of the Medical Faculty RWTH Aachen University (EK 340/16).

A Pilot Study of Modified Mini-Clinical Evaluation Exercises (Mini-CEX) in Rotation Students in the Department of Endocrinology

BACKGROUND

The mini-clinical evaluation exercise (mini-CEX) is an excellent tool for assessing the clinical abilities of medical students in intense clinical practice. In this study, the Mini-CEX was adapted to professional questionnaires for Diabetes Mellitus (DM), and examined in medical students completing their clerkship rotation in the department of endocrinology.

METHODS

From January 2021 to January 2022, all rotating medical students at Shanghai Pudong Hospital completed two mini-CEX exams before and following their rotation under the supervision and guidance of six tutors. The mini-CEX form was modified in this study primarily for inpatient management based on our clinical experience and updated DM guidelines of the American Diabetes Association (ADA), the European Association for the Study of Diabetes (EASD), and the Chinese Diabetes Society (CDS). Each component of the mini-CEX assessment, including medical interviews, physical examination, clinical judgment, clinical management, and overall clinical competence was evaluated using a nine-item questionnaire.

RESULTS

Our findings revealed that the second-round performance on the assessments significantly improved, as indicated by higher scores on each component. The Pearson association analysis revealed that the feedback time of the first examination was markedly associated with improved overall scores (r= 0.391, p<0.001). However, no correlations were discovered between patient age, gender, disease severity disparity, or the interval between examinations (p>0.05). Additional regression analysis revealed that the feedback time during the initial examination was the most significant contributor to the increased overall scores (β=0.391, p<0.001).

CONCLUSION

This newly designed mini-CEX form based on current ADA and EASD guidelines may assist trainees in more effectively diagnosing and managing DM in inpatients, particularly those with macrovascular, microvascular, or peripheral nerve neuropathy. This study aims to assess the efficacy of administering a modified mini-CEX form to rotating trainees participating in an endocrine clerkship.

Chinese doctors' views on workplace-based assessment: trainee and supervisor perspectives of the mini-CEX

Purpose: This study investigated whether the mini-clinical evaluation exercise (mini-CEX) has been successfully integrated into the Chinese context, following its introduction as part of the national general training programme. Materials and methods: Online questionnaires (N = 91) and interviews (N = 22) were conducted with Year 1 trainee doctors and clinical supervisors at a cancer hospital in China to explore users' experiences, attitudes and opinions of the mini-CEX. Results" Trainees were more likely than supervisors to report understanding the purpose of the mini-CEX and agree that it encouraged reflection and helped improve overall performance. Both trainees and supervisors felt that it provided a framework for learning, that it was useful in identifying underperformance, and that it informed learning progression. Groups were equally positive about the commitment of their counterpart in the process and valued the focus on detailed feedback. It was perceived as cultivating the learner-teacher relationship. Overall, both groups felt they 'bought in' to using the mini-CEX. However, concerns were raised about subjectivity of ratings and lack of benchmarking with expected standards of care. Conclusions: Chinese trainees and supervisors generally perceived the mini-CEX as an acceptable and valuable medical training tool, although both groups suggested enhancements to improve its efficacy.

The use of EPA assessments in decision-making: Do supervision ratings correlate with other measures of clinical performance?

BACKGROUND

Entrustable professional activities (EPAs) have been introduced as a framework for teaching and assessment in competency-based educational programs. With growing use, has come a call to examine the validity of EPA assessments. We sought to explore the correlation of EPA assessments with other clinical performance measures to support use of supervision ratings in decisions about medical students' curricular progression.

METHODS

Spearman rank coefficients were used to determine correlation of supervision ratings from EPA assessments with scores on clerkship evaluations and performance on an end-of-clerkship-year Objective Structured Clinical Examination (CPX).

RESULTS

Both overall clinical evaluation items score (rho 0.40; n = 166) and CPX patient encounter domain score (rho 0.31; n = 149) showed significant correlation with students' overall mean EPA supervision rating during the clerkship year. There was significant correlation between mean supervision rating for EPA assessments of history, exam, note, and oral presentation skills with scores for these skills on clerkship evaluations; less so on the CPX.

CONCLUSIONS

Correlation of EPA supervision ratings with commonly used clinical performance measures offers support for their use in undergraduate medical education. Data supporting the validity of EPA assessments promotes stakeholders' acceptance of their use in summative decisions about students' readiness for increased patient care responsibility.

Longitudinal Outcome of Programmatic Assessment of International Medical Graduates

INTRODUCTION

Australia depends on international medical graduates (IMGs) to meet workforce shortages. The current standard assessment for IMGs is by clinical examination in observed structured clinical encounter (OSCE) format lasting 200 minutes. There are concerns about adequateness of this assessment as it does not test the qualities required to practice in a new country. We introduced a programmatic performance-based assessment for IMGs to prepare them to meet these challenges. The workplace-based assessment (WBA) program involves six-month longitudinal programmatic assessments comprising of 12 mini-clinical evaluation exercises (Mini-CEX), five case-based discussions (CBD), two in-training assessments (ITAs) and two sets of multisource feedback (MSF) assessments. We assessed 254 IMGs since 2010. We conducted a survey to evaluate the satisfaction with the program and the outcomes of these doctors.

METHODS

We surveyed 254 candidates from 2010 to 2020. The survey used "SelectSurvey" tool with 12 questions and free-text comments. All candidates were sent the survey link to their last registered mobile phone using "Telstra Instant Messaging Service". We analysed the data using Microsoft "Excel".

RESULTS

We received 153 (60%) responses. Amongst them, 141 (92%) candidates did not require further supervised practice for general registration and 129 (84%) candidates hold general/specialist registration. The candidates found the program useful and felt well supported. They appreciated real patient encounters. The feedback with positive critiquing was helpful in improving their clinical practice. The negative themes were program costs and frustration with the length of the program.

CONCLUSION

Upon completion of the WBA program and obtaining the AMC certificate, most of the doctors were able to gain general registration. Seventy-eight (50%) candidates chose to continue their careers within the local area with 124 (80%) of them within the state. Our survey shows a comprehensive assessment program with immediate constructive feedback produces competent doctors to fill the medical workforce shortages.

Empirical analysis comparing the tele-objective structured clinical examination (teleOSCE) and the in-person assessment in Australia

PURPOSE

It aimed to compare the use of the tele objective structured clinical examination (teleOSCE) with in-person assessment in high-stakes clinical examination so as to determine the impact of the teleOSCE on the assessment undertaken. Discussion follows regarding what skills and domains can effectively be assessed in a teleOSCE.

METHODS

This study is a retrospective observational analysis. It compares the results achieved by final year medical students in their clinical examination, assessed using the teleOSCE in 2020 (n=285), with those who were examined using the traditional in-person format in 2019 (n=280). The study was undertaken at the University of New South Wales, Australia.

RESULTS

In the domain of physical examination, students in 2020 scored 0.277 points higher than those in 2019 (mean difference -0.277, P<0.001, effect size 0.332). Across all other domains, there was no significant difference in mean scores between 2019 and 2020.

CONCLUSION

The teleOSCE does not negatively impact assessment in clinical examination in all domains except physical examination. If the teleOSCE is the future of clinical skills examination, assessment of physical examination will require concomitant workplace-based assessment.

Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

OBJECTIVES

Direct observation is important for assessing the competency of medical learners. Multiple tools have been described in other fields, although the degree of emergency medicine-specific literature is unclear. This review sought to summarize the current literature on direct observation tools in the emergency department (ED) setting.

METHODS

We searched PubMed, Scopus, CINAHL, the Cochrane Central Register of Clinical Trials, the Cochrane Database of Systematic Reviews, ERIC, PsycINFO, and Google Scholar from 2012 to 2020 for publications on direct observation tools in the ED setting. Data were dual extracted into a predefined worksheet, and quality analysis was performed using the Medical Education Research Study Quality Instrument.

RESULTS

We identified 38 publications, comprising 2,977 learners. Fifteen different tools were described. The most commonly assessed tools included the Milestones (nine studies), Observed Structured Clinical Exercises (seven studies), the McMaster Modular Assessment Program (six studies), Queen's Simulation Assessment Test (five studies), and the mini-Clinical Evaluation Exercise (four studies). Most of the studies were performed in a single institution, and there were limited validity or reliability assessments reported.

CONCLUSIONS

The number of publications on direct observation tools for the ED setting has markedly increased. However, there remains a need for stronger internal and external validity data.

Examining the educational impact of the mini-CEX: a randomised controlled study

BACKGROUND

The purpose of this study is to evaluate the mini-Clinical Evaluation Exercise (mini-CEX) as a formative assessment tool among undergraduate medical students, in terms of student perceptions, effects on direct observation and feedback, and educational impact.

METHODS

Cluster randomised study of 38 fifth-year medical students during a 16-week clinical placement. Hospitals were randomised to provide a minimum of 8 mini-CEXs per student (intervention arm) or continue with ad-hoc feedback (control arm). After finishing their clinical placement, students completed an Objective Structured Clinical Examination (OSCE), a written test and a survey.

RESULTS

All participants in the intervention group completed the pre-planned number of assessments, and 60% found them to be useful during their clinical placement. Overall, there were no statistically significant differences between groups in reported quantity or quality of direct observation and feedback. Observed mean scores were marginally higher on the OSCE and written test in the intervention group, but not statistically significant.

CONCLUSIONS

There is considerable potential in assessing medical students during clinical placements and routine practice, but the educational impact of formative assessments remains mostly unknown. This study contributes with a robust study design, and may serve as a basis for future research.

Motivation and Evaluation in Education from the Sustainability Perspective: A Review of the Scientific Literature

(1) Background: This paper outlines the results of a literature review of meta-analyses published on motivation and evaluation in the last five years. (2) Methods: A systematic review of three educational databases (WoS, SCOPUS and ERIC) was conducted following the PRISMA and PICO approaches. A total of 54 peer-reviewed meta-analysis papers were selected, analysed and compared. (3) Results: A significant number and variety of meta-analyses have been conducted: motivation meta-analyses focus primarily on contextual variables, self-regulation and students’ academic performance, and evaluation meta-analyses examine the effectiveness of the teaching intervention, the use of teaching methodologies and technological resources for learning. (4) Conclusions: There are two important absences: on the one hand, it is necessary to develop meta-analyses that combine motivation and evaluation, also measuring their interaction, from the perspective of sustainability, and not only of educational improvement, and on the other hand, it is necessary to perform meta-analyses on the effectiveness of the formative and shared evaluation of the sustainability of learning processes.

Driving lesson or driving test? : A metaphor to help faculty separate feedback from assessment

Although there is consensus in the medical education world that feedback is an important and effective tool to support experiential workplace-based learning, learners tend to avoid the feedback associated with direct observation because they perceive it as a high-stakes evaluation with significant consequences for their future. The perceived dominance of the summative assessment paradigm throughout medical education reduces learners' willingness to seek feedback, and encourages supervisors to mix up feedback with provision of 'objective' grades or pass/fail marks. This eye-opener article argues that the provision and reception of effective feedback by clinical supervisors and their learners is dependent on both parties' awareness of the important distinction between feedback used in coaching towards growth and development (assessment for learning) and reaching a high-stakes judgement on the learner's competence and fitness for practice (assessment of learning). Using driving lessons and the driving test as a metaphor for feedback and assessment helps supervisors and learners to understand this crucial difference and to act upon it. It is the supervisor's responsibility to ensure that supervisor and learner achieve a clear mutual understanding of the purpose of each interaction (i.e. feedback or assessment). To allow supervisors to use the driving lesson-driving test metaphor for this purpose in their interactions with learners, it should be included in faculty development initiatives, along with a discussion of the key importance of separating feedback from assessment, to promote a feedback culture of growth and support programmatic assessment of competence.

Adaptation, psychometric properties, and implementation of the Mini-CEX in dental clerkship

BACKGROUND

Workplace-based assessment is a key component of dental-student clerkships, allowing students to demonstrate clinical proficiency.

PURPOSE

This study adapts the Mini-Clinical Evaluation Exercise (Mini-CEX) to a dentistry-program clerkship, analyzing the results and examining the psychometric properties of Mini-CEX.

METHODS

First, Delphi panel methodology was used to ensure content validity. Mini-CEX was then piloted in the dental-clerkship program, with each student assessed by at least 2 supervisors and a peer student. Subsequently, psychometric properties, acceptability, and observation time were analyzed.

RESULTS

The study was conducted between July and November 2019. Overall, 140 Mini-CEX evaluation exercises were carried out on 30 students by 84 supervisors and 56 peers. The adapted instrument was found to be unidimensional, obtaining an acceptable internal consistency (α = 0.74). As the assessor type changed, there were differences in observation time; the medians (Q1-Q3) were 10 minutes (5-15) for supervisors and 30 minutes (20-45) for peer students (P < 0.001). This difference was also observed in assessor perceptions (P < 0.001), with supervisors scoring a median of 6 (6-6.75) and peer students scoring a median of 7 (6-7). No differences were found between supervisor and peer scores.

CONCLUSION

The adapted version of Mini-CEX can objectively assess the clinical performance of dental students, achieving validity and reliability values similar to those obtained in the original instrument.

Validity, reliability and feasibility of a new observation rating tool and a post encounter rating tool for the assessment of clinical reasoning skills of medical students during their internal medicine clerkship: a pilot study

BACKGROUND

Systematic assessment of clinical reasoning skills of medical students in clinical practice is very difficult. This is partly caused by the lack of understanding of the fundamental mechanisms underlying the process of clinical reasoning.

METHODS

We previously developed an observation tool to assess the clinical reasoning skills of medical students during clinical practice. This observation tool consists of an 11-item observation rating form (ORT). In the present study we verified the validity, reliability and feasibility of this tool and of an already existing post-encounter rating tool (PERT) in clinical practice among medical students during the internal medicine clerkship.

RESULTS

Six raters each assessed the same 15 student-patient encounters. The internal consistency (Cronbach's alfa) for the (ORT) was 0.87 (0.71-0.84) and the 5-item (PERT) was 0.81 (0.71-0.87). The intraclass-correlation coefficient for single measurements was poor for both the ORT; 0.32 (p < 0.001) as well as the PERT; 0.36 (p < 0.001). The Generalizability study (G-study) and decision study (D-study) showed that 6 raters are required to achieve a G-coefficient of > 0.7 for the ORT and 7 raters for the PERT. The largest sources of variance are the interaction between raters and students. There was a consistent correlation between the ORT and PERT of 0.53 (p = 0.04).

CONCLUSIONS

The ORT and PERT are both feasible, valid and reliable instruments to assess students' clinical reasoning skills in clinical practice.