An objective structured clinical exam to measure intrinsic CanMEDS roles

OSABSS: An authentic examination for assessing basic surgical skills in surgical residents

Objectives

This study aimed to develop and validate the OSABSS (Objective Structured Assessment of Basic Surgical Skills), a modified Objective Structured Clinical Examination (OSCE), to assess basic surgical skills in residents.

Design

A developmental study conducted in two phases. Basic skills were identified through literature review and gap analysis. The OSABSS was then designed as a modified OSCE.

Setting

This study took place at Alborz University of Medical Sciences in Iran.

Interventions

The OSABSS was created using Harden's OSCE (Objective Structured Clinical Examination) methodology. Scenarios, checklists, and station configurations were developed through expert panels. The exam was piloted and implemented with residents as participants and faculty as evaluators.

Participants

32 surgical residents in gynecology, general surgery, orthopedics, and neurosurgery participated. 22 faculty members were evaluators.

Primary and secondary outcome measures

The primary outcome was OSABSS exam scores. Secondary outcomes were written exam scores, and national residency entrance ranks.

Main results

The mean OSABSS score was 16.59 ± 0.19 across all stations. Criterion validity was demonstrated through correlations between OSABSS scores, written scores and entrance ranks. Reliability was high, with a Cronbach's alpha of 0.87. No significant inter-rater score differences were found.

Conclusions

The rigorous OSABSS development process produced an exam demonstrating strong validity and reliability for assessing basic surgical skills. The comprehensive station variety evaluates diverse technical and non-technical competencies. Further research should expand participant samples across surgical disciplines.

Which, how, and what? Using digital tools to train surgical skills; a systematic review and meta-analysis

Background

Digital tools like digital box trainers and VR seem promising in delivering safe and tailored practice opportunities outside of the surgical clinic, yet understanding their efficacy and limitations is essential. This study investigated Which digital tools are available to train surgical skills, How these tools are used, How effective they are, and What skills they are intended to teach.

Methods

Medline, Embase, and Cochrane libraries were systematically reviewed for randomized trials, evaluating digital skill-training tools based on objective outcomes (skills scores and completion time) in surgical residents. Digital tools effectiveness were compared against controls, wet/dry lab training, and other digital tools. Tool and training factors subgroups were analysed, and studies were assessed on their primary outcomes: technical and/or non-technical.

Results

The 33 included studies involved 927 residents and six digital tools; digital box trainers, (immersive) virtual reality (VR) trainers, robot surgery trainers, coaching and feedback, and serious games. Digital tools outperformed controls in skill scores (SMD 1.66 [1.06, 2.25], P < 0.00001, I2 = 83 %) and completion time (SMD -1.05 [-1.72, -0.38], P = 0.0001, I2 = 71 %). There were no significant differences between digital tools and lab training, between tools, or in other subgroups. Only two studies focussed on non-technical skills.

Conclusion

While the efficacy of digital tools in enhancing technical surgical skills is evident - especially for VR-trainers -, there is a lack of evidence regarding non-technical skills, and need to improve methodological robustness of research on new (digital) tools before they are implemented in curricula.

Key message

This study provides critical insight into the increasing presence of digital tools in surgical training, demonstrating their usefulness while identifying current challenges, especially regarding methodological robustness and inattention to non-technical skills.

World Psychiatric Association-Asian Journal of Psychiatry Commission on Psychiatric Education in the 21st century

Psychiatric practice faces many challenges in the first quarter of 21st century. Society has transformed, as have training requirements and patient expectations, underlining an urgent need to look at educational programmes. Meanwhile, awareness has grown around psychiatric disorders and there are evolving workforce trends, with more women going to medical school and specialising in psychiatry. Trainee psychiatrists carry different expectations for work-life balance and are increasingly becoming conscious of their own mental health. A tendency to see health as a commodity and the litigious nature of society has elicited additional pressures for healthcare professionals. Cartesian mind-body dualism has created further complexity and this can often be frustrating for patients and care-partners alike. In many cultures across Asia and beyond, patients can present with physical symptoms to express underlying psychological distress with increasing physical investigations. Simultaneously, in various countries, a shift from asylums to community-based interventions and then home treatments have changed psychiatric care in remarkable ways. These changes have added to pressures faced by mental healthcare professionals. However, trainees and other mental healthcare professionals continue to receive similar training as they did a generation ago. The tensions and differences in ideology/orientation between different branches of psychiatry have made responses to patient needs challenging. Recognising that it is difficult to predict the future, this World Psychiatric Association-Asian Journal of Psychiatry Commission makes recommendations that could help institutions and individuals enhance psychiatric education. This Commission draws from existing resources and recent developments to propose a training framework for future psychiatrists.

Evaluating the Construct Validity of Competencies: A Retrospective Analysis

Background

A competency-based framework focuses on alignment between professional standards and assessment design. This alignment implies improved measurement validity, yet it has not been established that competence in one context predicts performance in another context. High-stakes competence assessments offer insights into the relationship between assessment design and competencies.

Methods/Analyses

The internationally educated nurses competency assessment program (IENCAP) was developed at Touchstone Institute in collaboration with the College of Nurses of Ontario (CNO) and includes a 12-station OSCE. Each station evaluated the same 10 competencies. We submitted competency scores to a multi-trait multi-method matrix analysis to evaluate the convergent and discriminant validity of competencies.

Results/Observations

All correlations were significant and positive; however, we did not find evidence of convergent or discriminant validity. Correlations were higher between different competencies evaluated within the same station (mean correlation = 0.60) compared to identical competencies evaluated across different stations (mean correlation = 0.19).

Discussion

The results do not provide evidence of construct validity for competencies. While competency-based approaches emphasize various generalized knowledge, skills, and attitudes, these findings indicate that the clinical context is a major determinant of performance.

Conclusion

The context-dependent nature of competencies requires multiple assessments in varied contexts. Performance on a single competency cannot be determined in a single occasion.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-023-01794-z.

Competency Based Medical Education-Towards the Development of a Standardized Pediatric Radiology Testing Module

RATIONALE AND OBJECTIVES

To introduce a process that allows for development of standardized competency based testing modules (CBTM) for evaluating resident progress and competence during their radiology training. This work focuses on the development of pediatric imaging CBTMs to be utilized during general radiology residency.

MATERIALS AND METHODS

Multiple in-patient and ER imaging request audits along with surveys of training programs and text recommendations were obtained. A list of 200 total diagnoses accumulated by review was distributed into one of four CBTM folders. Imaging cases which made ≥90% of the indications of the audits were added to Folder 1. Distribution of remaining imaging diagnoses was based on consensus by three subspecialists. A pilot study was performed with residents dictating selected imaging cases in their usual manner mimicking a typical rotation.

RESULTS

The pilot study demonstrated resident grading mean scores significantly associated with both the American College of Radiology (ACR) rank (rho = 0.636, p = 0.035) and the objective structured clinical examinations (OSCE) scores (rho = 0.694, p = 0.018). The mean scores positively associated with the ACR score (rho = 0.466), but fell short of statistical significance (p = 0.149). As expected, the ACR score, ACR rank and OSCE scores all significantly correlated with each other ( < 0.01). PGY also significantly correlated with the ACR score (rho = 0.683, p = 0.021) and the OSCE (rho = 0.767, p = 0.006) but not with the ACR rank (rho = 0.408, p = 0.213).

CONCLUSION

The process utilized to develop a standardized CBTM can be used as a simulation tool to assess radiology resident competence during their training. The format allows for assessment of resident reasoning skills and knowledge base, which provides documentation of progression and throughout residency.

Relationships between objective structured clinical examination, computer-based testing, and clinical clerkship performance in Japanese medical students

Background

It is unclear how comprehensive evaluations conducted prior to clinical clerkships (CC), such as the objective structured clinical examination (OSCE) and computer-based testing (CBT), reflect the performance of medical students in CC. Here we retrospectively analyzed correlations between OSCE and CBT scores and CC performance.

Methods

Ethical approval was obtained from our institutional review board. We analyzed correlations between OSCE and CBT scores and CC performance in 94 medical students who took the OSCE and CBT in 2017 when they were 4th year students, and who participated in the basic CC in 2018 when they were 5th year students.

Results

Total scores for OSCE and CBT were significantly correlated with CC performance (P<0.001, each). More specifically, medical interview and chest examination components of the OSCE were significantly correlated with CC performance (P = 0.001, each), while the remaining five components of the OSCE were not.

Conclusion

Our findings suggest that the OSCE and CBT play important roles in predicting CC performance in Japanese medical education context. Among OSCE components, medical interview and chest examination were suggested to be important for predicting CC performance.

Obstetrics and gynaecology trainees' perceptions of the CanMEDS expertise model: implications for training from a regional questionnaire study in the United Kingdom

The CanMEDS expertise model is a multi-domain competency framework for doctors. The aims of this study were to assess the perceived importance of the CanMEDS roles and achievement among obstetrics and gynaecology trainees of all grades with a view to identifying opportunities to enhance training. This study was exempt from formal ethical or institutional registration. The data collection was completed in 2017. Following a video introduction, the trainees completed a questionnaire. For each of the CanMEDS domains, trainees of different tiers perceived them to be equally important. Indeed, the junior and senior cohorts of trainees perceived all domains to be equally important, as signified by the significant degree of score correlation. Age was a significant variable for achievement of competency in the roles of a Medical Expert (p = .01), a Communicator (p = .04), a Collaborator (p = .002), a Scholar (p = .01) and a Professional (p = .03). Grade was significant for the Medical Expert (p = .001) and Leader (p = .001) role. Better alignment of clinical activities with CanMEDS competencies and faculty development will complement the training in leadership skills. Impact statementWhat is already known on this subject? The CanMEDS medical expertise model is a multi-domain framework of seven components. This framework has been utilised to assess the training efficacy of curricula and unlock opportunities for improvement. The research application of the CanMEDS framework within Obstetrics and Gynaecology is limited.What does this study add? Results indicate that all trainees recognise the importance of CanMEDS roles: age and grade are significant variables in the perceived achievement of CanMEDS roles. The study identifies areas for improvement in the current training strategy.What are the implications for clinical practice/future research? Research should formalise the assessment of competencies in non-technical skills. Efforts should focus on identifying the activities which will develop leadership skills.

How do Lebanese patients perceive the ideal doctor based on the CanMEDS competency framework?

BACKGROUND

During their training, Lebanese medical students develop a high medical expertise but are not focusing on other competencies such as communication, collaboration, erudition, professionalism, leadership and health promotion. There is also insufficient data about patients' preference for these skills. This study describes the different weights patients attribute to these physician's competencies.

METHODS

This is a cross-sectional study based on a questionnaire distributed to 133 Lebanese patients. It included 15 questions assessing how patients prioritize the physician's competencies, with open-ended questions asking them to define "the good doctor". Krippendorff's alpha coefficient was used to analyze the reliability of the competencies' classification.

RESULTS

One hundred twenty five patients completed the questionnaire in this cross-sectional study. Their mean age was 48 ± 16.76 years. When classifying competencies, 73.6% opted for medical expertise as first choice and 48% put communication as second. Based on the Krippendorff's coefficient, we identified a moderate agreement for the seven choices (alpha = 0.44). In open-ended questions, patients defined the good doctor in 325 answers: 64.3% mentioned medical expertise, 34.1% high ethics and 26.2% communication.

CONCLUSIONS

This patient-centered study concurs well with the worldwide practice that puts medical expertise at the center of medical education. However Lebanese patients don't perceive equally other competencies and favor professionalism and communication that should be integrated in priority in students' curricula.

The future of general surgery training: A Canadian resident nationwide Delphi consensus statement

BACKGROUND

Several models have been introduced to improve and restructure surgical training, but continued barriers exist. Residents are uniquely positioned to offer perspective on practical challenges and needs of reformatting surgical education. This study aimed to establish a nationwide, Delphi consensus statement on the perceptions of Canadian residents regarding the future of general surgery training.

METHODS

Canadian general surgery residents participated in a moderated focus group using the Nominal Group Technique to discuss early subspecialization, competency-based medical education, and transition to practice. Qualitative verbal data were transcribed, categorized into themes, and synthesized into recommendation statements. During an iterative Delphi survey, resident leaders ranked each statement on a 5-point Likert scale of agreement. The survey was terminated once consensus was achieved (≥2 survey rounds and Cronbach's α ≥ 0.80).

RESULTS

A total of 66 statements were synthesized by 16 members of the Canadian Association of General Surgeons Resident Committee. A total of 49 residents participated in the Delphi consensus, which was achieved after 2 voting rounds (Cronbach's α = 0.93). Participants agreed that (1) residency should focus on achieving standardized competencies and milestones based on resident ability to meet specific measurable metrics, (2) early streaming should be offered after "core" milestones and competencies have been achieved, and (3) an explicit period should allow transition-to-independent practice with tailored rotations, greater autonomy, and resident-run clinics. We identified 10 barriers to competency-based medical education implementation.

CONCLUSION

A nationwide consensus regarding the future of surgical training was established among current residents. These findings can inform and help implement guidelines and national curricula that meet the needs of the trainee and address the many challenges they face during their training.

The education avenue of the clinical learning environment: A pragmatic approach

Aim: This perspective is part of a series of articles that are outcomes of a consensus conference, that seek to offer a comprehensive examination of the clinical learning environment (CLE), using different academic disciplines and areas of focus termed "avenues." The education dimensions of the CLE are discussed in detail in this perspective, along with critical linkages to the other avenues. Methods: Using iterative presentations, discussion and small group work, in October 2018, the consensus conference participants explored the education, psychological, sociocultural, diversity and inclusion, digital and architectural aspects of the CLE. Results: The education avenue of the CLE includes elements of teaching and learning such as the curriculum, clinical experiences, the assessment system, educational program governance, trainee selection, faculty development, and program evaluation and improvement. Within the educational domain, we focus on organizational and personal/social aspects of the CLE, including (1) curriculum design and deployment, including placement of trainees in clinical settings, organizational culture, practices and policies, and accreditation and regulatory requirements from the organizational domain (2) the education system, including assessment, program evaluation and organization and governance; and (3) elements from the personal and social domains, including peer-to-peer, trainee-faculty, and trainee-patient relationships that influence how and what postgraduate trainees learn, trainee selection, informal and hidden curricula, and trainees' perceptions of their learning environment. Conclusions: We provide suggestions for further research and recommendations for addressing challenges and facilitating improvement in the educational aspects of the CLE, along with actionable practice points.