1
|
de Melo BCP, Falbo AR, Souza ES, Muijtjens AMM, Van Merriënboer JJG, Van der Vleuten CPM. The limited use of instructional design guidelines in healthcare simulation scenarios: an expert appraisal. Adv Simul (Lond) 2022; 7:30. [PMID: 36153603 PMCID: PMC9509554 DOI: 10.1186/s41077-022-00228-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2022] [Accepted: 09/09/2022] [Indexed: 11/10/2022] Open
Abstract
Background Systematic reviews on simulation training effectiveness have pointed to the need to adhere to evidence-based instructional design (ID) guidelines. ID guidelines derive from sound cognitive theories and aim to optimize complex learning (integration of knowledge, skills, and attitudes) and learning transfer (application of acquired knowledge and skills in the workplace). The purpose of this study was to explore adherence to ID guidelines in simulation training programs for dealing with postpartum hemorrhage (PPH), a high-risk situation and the leading cause of maternal mortality worldwide. Methods A total of 40 raters analyzed simulation training programs as described in 32 articles. The articles were divided into four subsets of seven articles and one subset of four articles. Each subset was judged by seven to ten raters on adherence to ID guidelines. The 5-point Likert score rating scale was based on Merrill’s First Principles of Instruction and included items relating to key ID features categorized into five subscales: authenticity, activation of prior knowledge, demonstration, application, and integration/transfer. The authors searched for articles published in English between January 2007 and March 2017 in PubMed, Eric, and Google Scholar and calculated the mean Likert-scale score, per subscale, and interrater reliability (IRR). Results The mean Likert-scale scores calculated for all subscales were < 3.00. For the number of raters used to judge the papers in this study (varying between 7 and 10), the IRR was found to be excellent for the authenticity and integration/transfer subscales, good-to-excellent for the activation of prior knowledge and application subscales, and fair-to-good for the demonstration subscale. Conclusion The results demonstrate a paucity of the description of adherence to evidence-based ID guidelines in current simulation trainings for a high-risk situation such as PPH. Supplementary Information The online version contains supplementary material available at 10.1186/s41077-022-00228-x.
Collapse
|
2
|
Mohtady HA, Könings KD, Al-Eraky MM, Muijtjens AMM, van Merriënboer JJG. High enthusiasm about long lasting mentoring relationships and older mentors. BMC Med Educ 2019; 19:364. [PMID: 31547807 PMCID: PMC6757421 DOI: 10.1186/s12909-019-1791-8] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/28/2018] [Accepted: 09/05/2019] [Indexed: 06/10/2023]
Abstract
BACKGROUND Mentoring plays a pivotal role in workplace-based learning, especially in the medical realm. Organising a formal mentoring programme can be labor and time intensive and generally impractical in resource constrained medical schools with limited numbers of mentors. Hence, informal mentoring offers a valuable alternative, but will be more likely to be effective when mentors and protégés share similar views. It is therefore important to gain more insight into factors influencing perceptions of informal mentoring. This study aims to explore mentors and protégés' perceptions of informal mentoring and how these vary (or not) with gender, age and the duration of the relationship. METHOD We administered an Informal Mentor Role Instrument (IMRI) to medical practitioners and academics from Egypt, Pakistan and Saudi Arabia. The questionnaire was developed for the study from other validated instruments. It contained 39 items grouped into 7 domains: acceptance, counselling, friendship, parenting, psychological support, role modelling and sociability. RESULTS A total of 103 mentors and 91 protégés completed the IMRI. Mentors had a better appreciation for the interpersonal aspects of informal mentoring than protégés, especially regarding acceptance, counselling and friendship. Moreover, being older and engaged in a longer mentoring relationship contributed to more positive perceptions of interpersonal aspects of mentoring, regardless of one's role (mentor or protégé). CONCLUSION It can be concluded that the expectations of mentors and protégés differed regarding the content and aim of the interpersonal characteristics of their mentoring relationship. We recommend mentors and protégés to more explicitly exchange their expectations of the informal mentoring relationship, as typically practiced in formal mentoring. Additionally, in our study, seniority and lasting relationships seem crucial for good informal mentoring. It appears beneficial to foster lasting informal mentoring relationships and to give more guidance to younger mentors.
Collapse
Affiliation(s)
- Heba A. Mohtady
- Medical Education Department, Fakeeh College for Medical Sciences, P.O 2537, Jeddah, 21461 Kingdom of Saudi Arabia
- Maastricht University, Maastricht, the Netherlands
- Microbiology &Immunology Department, Faculty of Medicine, Zagazig University, Zagazig, Egypt
| | | | | | | | | |
Collapse
|
3
|
de Melo BCP, Van der Vleuten CPM, Muijtjens AMM, Rodrigues Falbo A, Katz L, Van Merriënboer JJG. Effects of an in situ instructional design based postpartum hemorrhage simulation training on patient outcomes: an uncontrolled before-and-after study. J Matern Fetal Neonatal Med 2019; 34:245-252. [PMID: 31023119 DOI: 10.1080/14767058.2019.1606195] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/29/2023]
Abstract
Objective: To compare postpartum hemorrhage (PPH) patient outcomes before and after an in situ instructional design based PPH simulation attended by obstetrics and gynecology (OBGYN) residents.Methods: This uncontrolled before-and-after study was conducted in Recife, Brazil including all 1388 women delivering from June to August 2012 and all 1357 delivering from June to August 2013. The 36 OBGYN residents were divided into13 teams of two or three participants and were trained through ID based PPH simulation training with the following eight steps: (1) prior knowledge activation, (2) video demonstration, (3) dual-coding PPH protocol discussion-an image association during the training, (4) training scenario # 1, (5) debriefing, (6) training scenario # 2 with immediate feedback, (7) training scenario # 3, and (8) debriefing with self-assessment. The training scenarios had an increasing level of complexity. The main goal of the training was the adequate management of PPH and situational awareness improvement-the ability to anticipate, recognize, and intercept unfolding error chains. The primary patient outcomes rates used for the before and after comparison were therapeutic uterotonics use within 24 h of birth and blood transfusion. Secondary outcomes were therapeutic oxytocin mean dosage IU within 24 h of birth, postpartum Hb < 6 g/dL, among others. Chi-square test was used for categorical variables comparison and independent t-test for continuous variables.Results: PPH rates were 100 (7.2% of 2012 deliveries) and 80 cases (5.9% of 2013 deliveries), respectively. Comparison of primary post- and pre-simulation outcomes revealed no significant differences. However, in the comparison for therapeutic oxytocin mean dosage IU within 24 h of birth, there was an increase found after the simulation (15.98 ± 7.4 versus 25.1 ± 12.3; p < .001). For all other outcome measures, there were no statistical differences.Conclusions: In situ ID based PPH simulation leads to an increase in the mean dosage of oxytocin after training, in selected cases. This may indicate better situational awareness when managing women with PPH.
Collapse
Affiliation(s)
- Brena Carvalho Pinto de Melo
- Centro de Atenção, à Mulher, Instituto de Medicina Integral Prof Fernando Figueira (IMIP), Recife, Brazil.,Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands.,Medical Course, Faculdade Pernambucana de Saúde (FPS), Recife, Brazil
| | - Cees P M Van der Vleuten
- Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands
| | - Arno M M Muijtjens
- Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands
| | - Ana Rodrigues Falbo
- Medical Course, Faculdade Pernambucana de Saúde (FPS), Recife, Brazil.,Faculty Development Coordination, Faculdade Pernambucana de Saúde (FPS), Recife, Brazil.,Programa de Pós-Graduação, Recife, Brazil
| | - Leila Katz
- Centro de Atenção, à Mulher, Instituto de Medicina Integral Prof Fernando Figueira (IMIP), Recife, Brazil.,Programa de Pós-Graduação, Recife, Brazil
| | - Jeroen J G Van Merriënboer
- Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
4
|
Schreurs S, Cleland J, Muijtjens AMM, oude Egbrink MGA, Cleutjens K. Does selection pay off? A cost-benefit comparison of medical school selection and lottery systems. Med Educ 2018; 52:1240-1248. [PMID: 30324680 PMCID: PMC6282742 DOI: 10.1111/medu.13698] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/02/2018] [Revised: 06/18/2018] [Accepted: 07/23/2018] [Indexed: 05/15/2023]
Abstract
CONTEXT Resources for medical education are becoming more constrained, whereas accountability in medical education is increasing. In this constrictive environment, medical schools need to consider and justify their selection procedures in terms of costs and benefits. To date, there have been no studies focusing on this aspect of selection. OBJECTIVES We aimed to examine and compare the costs and benefits of two different approaches to admission into medical school: a tailored, multimethod selection process versus a lottery procedure. Our goal was to assess the relative effectiveness of each approach and to compare these in terms of benefits and costs from the perspective of the medical school. METHODS The study was conducted at Maastricht University Medical School, at which the selection process and a weighted lottery procedure ran in parallel for 3 years (2011-2013). The costs and benefits of the selection process were compared with those of the lottery procedure over three student cohorts throughout the Bachelor's programme. The extra costs of selection represented the monetary investment of the medical school in conducting the selection procedure; the benefits were derived from the increase in income generated by the prevention of dropout and the reductions in extra costs facilitated by decreases in the repetition of blocks and objective structured clinical examinations. RESULTS The tailor-made selection procedure cost about €139 000 when extrapolated to a full cohort of students (n = 286). The lottery procedure came with negligible costs for the medical school. However, the average benefits of selection compared with the lottery system added up to almost €207 000. CONCLUSIONS This study not only shows that conducting a cost-benefit comparison is feasible in the context of selection for medical school, but also that an 'expensive' selection process can be cost-beneficial in comparison with an 'inexpensive' lottery system. We encourage other medical schools to examine the cost-effectiveness of their own selection processes in relation to student outcomes in order to extend knowledge on this important topic.
Collapse
Affiliation(s)
- Sanne Schreurs
- Department of Educational Development and ResearchInstitute for Education and School of Health Professions EducationFaculty of Health, Medicine and Life SciencesMaastricht UniversityMaastrichtthe Netherlands
| | - Jennifer Cleland
- Centre for Healthcare Education Research and Innovation (CHERI)Institute of Education for Medical and Dental SciencesUniversity of AberdeenAberdeenUK
| | - Arno M M Muijtjens
- Department of Educational Development and ResearchInstitute for Education and School of Health Professions EducationFaculty of Health, Medicine and Life SciencesMaastricht UniversityMaastrichtthe Netherlands
| | - Mirjam G A oude Egbrink
- Department of PhysiologyInstitute for EducationFaculty of Health, Medicine and Life SciencesMaastricht UniversityMaastrichtthe Netherlands
| | - Kitty Cleutjens
- Department of PathologyFaculty of Health, Medicine and Life SciencesMaastricht UniversityMaastrichtthe Netherlands
| |
Collapse
|
5
|
Schreurs S, Cleutjens KB, Muijtjens AMM, Cleland J, Oude Egbrink MGA. Selection into medicine: the predictive validity of an outcome-based procedure. BMC Med Educ 2018; 18:214. [PMID: 30223816 PMCID: PMC6142422 DOI: 10.1186/s12909-018-1316-x] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/26/2018] [Accepted: 08/23/2018] [Indexed: 05/15/2023]
Abstract
BACKGROUND Medical schools must select students from a large pool of well-qualified applicants. A challenging issue set forward in the broader literature is that of which cognitive and (inter)personal qualities should be measured to predict diverse later performance. To address this gap, we designed a 'backward chaining' approach to selection, based on the competences of a 'good doctor'. Our aim was to examine if this outcome-based selection procedure was predictive of study success in a medical bachelor program. METHODS We designed a multi-tool selection procedure, blueprinted to the CanMEDS competency framework. The relationship between performance at selection and later study success across a three-year bachelor program was examined in three cohorts. Study results were compared between selection-positive and selection-negative (i.e. primarily rejected) students. RESULTS Selection-positive students outperformed their selection-negative counterparts throughout the entire bachelor program on assessments measuring cognitive (e.g. written exams), (inter)personal and combined outcomes (i.e. OSCEs). Of the 30 outcome variables, selection-positive students scored significantly higher in 11 cases. Fifteen other, non-significant between-group differences were also in favor of the selection-positives. An overall comparison using a sign test indicated a significant difference between both groups (p < 0.001), despite equal pre-university GPAs. CONCLUSIONS The use of an outcome-based selection approach seems to address some of the predictive validity limitations of commonly-used selection tools. Selection-positive students significantly outperformed their selection-negative counterparts across a range of cognitive, (inter)personal, and mixed outcomes throughout the entire three-year bachelor in medicine.
Collapse
Affiliation(s)
- Sanne Schreurs
- Department of Educational Development and Research, School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Universiteitssingel 60, P.O. Box 616, 6229 ER, Maastricht, the Netherlands.
| | - Kitty B Cleutjens
- Department of Pathology, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| | - Arno M M Muijtjens
- Department of Educational Development and Research, School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Universiteitssingel 60, P.O. Box 616, 6229 ER, Maastricht, the Netherlands
| | - Jennifer Cleland
- Institute of Education for Medical and Dental Sciences, University of Aberdeen, Foresterhill, Aberdeen, UK
| | - Mirjam G A Oude Egbrink
- Department of Physiology, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| |
Collapse
|
6
|
Balslev T, Muijtjens AMM, Maarbjerg SF, de Grave W. Selection and ranking of patient video cases in paediatric neurology in relation to learner levels. Eur J Paediatr Neurol 2018; 22:498-506. [PMID: 29274891 DOI: 10.1016/j.ejpn.2017.11.007] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/07/2017] [Revised: 06/14/2017] [Accepted: 11/22/2017] [Indexed: 10/18/2022]
Abstract
BACKGROUND Teaching and learning with patient video cases may add authenticity, enhance diagnostic accuracy and improve chances of early diagnosis. The aim of this study is firstly to identify selection criteria for key Patient video cases (PVCs), secondly to identify trends in relevance of PVCs for learner levels and thirdly, to rank PVCs for learner levels. METHODS Based on a literature review, we identified criteria for key PVCs for use in paediatric neurology. We then performed a multi-round Delphi analysis to obtain agreement between 28 expert clinician teachers concerning key PVCs for four learner levels. RESULTS We identified two major criteria: key PVCs should demonstrate key movements, and these movements should be subtle and/or difficult to note. The expert clinician teachers subsequently assessed a list of 14 topics for key PVCs. We found a clear, increasing trend in relevance scores, from medical students to young residents to experienced residents and specialists. For medical students and residents, epileptic spasms, Down syndrome, developmental delay, cerebral palsy and absence epilepsy were highly ranked. For specialists, conditions like chorea, focal seizures or eye movement disorders topped the ranking list, although ranking was less clear for this group of advanced learners. DISCUSSION AND CONCLUSION Key PVCs should demonstrate movements that are difficult to note for learners. Ranked lists of key PVCs for teaching and learning at different learner levels are now available and may help institutions build validated local libraries of PVCs.
Collapse
Affiliation(s)
- Thomas Balslev
- Department of paediatrics, Viborg Regional Hospital, Heibergs Alle 4, DK-8200 Viborg, Denmark; Centre for Health Sciences Education, CESU, Palle Juul Jensen Boulevard 82, Health, Aarhus University, DK-8200 Aarhus, Denmark.
| | - Arno M M Muijtjens
- Maastricht University, Department of Educational Development & Research, Faculty of Health, Medicine and Life Sciences (FHML), 6200 MD, Maastricht, The Netherlands
| | | | - Willem de Grave
- Maastricht University, Department of Educational Development & Research, Faculty of Health, Medicine and Life Sciences (FHML), 6200 MD, Maastricht, The Netherlands
| |
Collapse
|
7
|
Pacifico JL, van der Vleuten CPM, Muijtjens AMM, Sana EA, Heeneman S. Cross-validation of a learning climate instrument in a non-western postgraduate clinical environment. BMC Med Educ 2018; 18:22. [PMID: 29370793 PMCID: PMC5785826 DOI: 10.1186/s12909-018-1127-0] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/25/2017] [Accepted: 01/17/2018] [Indexed: 06/07/2023]
Abstract
BACKGROUND In postgraduate training, there is a need to continuously assess the learning and working conditions to optimize learning. Students or trainees respond to the learning climate as they perceive it. The Dutch Residency Educational Climate Test (D-RECT) is a learning climate measurement tool with well-substantiated validity. However, it was originally designed for Dutch postgraduate trainees and it remains to be shown whether extrapolation to non-Western settings is viable. The dual objective of this study was to revalidate D-RECT outside of a Western setting and to evaluate the factor structure of a recently revised version of the D-RECT containing 35 items. METHODS We invited Filipino internal medicine residents from 96 hospitals to complete the revised 35-item D-RECT. Subsequently, we performed a confirmatory factor analysis to check the fit of the 9 scale model of the revised 35-item D-RECT. Inter-rater reliability was assessed using generalizability theory. RESULTS Confirmatory factor analysis unveiled that the factor structure of the revised 35-item D-RECT provided a reasonable fit to the Filipino data, after removal of 7 items. Five to seven evaluations of individual residents were needed per scale to obtain a reliable result. CONCLUSION Even in a non-Western setting, the D-RECT exhibited psychometric validity. This study validated the factor structure of the revised 35-item D-RECT after some modifications. We recommend that its application be extended to other Asian countries and specialties.
Collapse
Affiliation(s)
- Jaime L. Pacifico
- De La Salle University Medical Center, De La Salle Health Sciences Institute, 4114 Dasmarinas, Cavite Philippines
| | | | - Arno M. M. Muijtjens
- Department of Educational Development and Research, Maastricht University, Maastricht, the Netherlands
| | - Erlyn A. Sana
- National Teacher Training Center for the Health Professions, University of the Philippines, Manila, Philippines
| | - Sylvia Heeneman
- Department of Educational Development and Research, Maastricht University, Maastricht, the Netherlands
- Department of Pathology, Maastricht University, Maastricht, the Netherlands
| |
Collapse
|
8
|
de Jonge LPJWM, Timmerman AA, Govaerts MJB, Muris JWM, Muijtjens AMM, Kramer AWM, van der Vleuten CPM. Stakeholder perspectives on workplace-based performance assessment: towards a better understanding of assessor behaviour. Adv Health Sci Educ Theory Pract 2017; 22:1213-1243. [PMID: 28155004 PMCID: PMC5663793 DOI: 10.1007/s10459-017-9760-7] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/07/2016] [Accepted: 01/24/2017] [Indexed: 05/13/2023]
Abstract
Workplace-Based Assessment (WBA) plays a pivotal role in present-day competency-based medical curricula. Validity in WBA mainly depends on how stakeholders (e.g. clinical supervisors and learners) use the assessments-rather than on the intrinsic qualities of instruments and methods. Current research on assessment in clinical contexts seems to imply that variable behaviours during performance assessment of both assessors and learners may well reflect their respective beliefs and perspectives towards WBA. We therefore performed a Q methodological study to explore perspectives underlying stakeholders' behaviours in WBA in a postgraduate medical training program. Five different perspectives on performance assessment were extracted: Agency, Mutuality, Objectivity, Adaptivity and Accountability. These perspectives reflect both differences and similarities in stakeholder perceptions and preferences regarding the utility of WBA. In comparing and contrasting the various perspectives, we identified two key areas of disagreement, specifically 'the locus of regulation of learning' (i.e., self-regulated versus externally regulated learning) and 'the extent to which assessment should be standardised' (i.e., tailored versus standardised assessment). Differing perspectives may variously affect stakeholders' acceptance, use-and, consequently, the effectiveness-of assessment programmes. Continuous interaction between all stakeholders is essential to monitor, adapt and improve assessment practices and to stimulate the development of a shared mental model. Better understanding of underlying stakeholder perspectives could be an important step in bridging the gap between psychometric and socio-constructivist approaches in WBA.
Collapse
Affiliation(s)
- Laury P J W M de Jonge
- Department of Family Medicine, FHML, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands.
| | - Angelique A Timmerman
- Department of Family Medicine, FHML, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands
| | - Marjan J B Govaerts
- Department of Educational Research and Development, FHML, Maastricht University, Maastricht, The Netherlands
| | - Jean W M Muris
- Department of Family Medicine, FHML, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands
| | - Arno M M Muijtjens
- Department of Educational Research and Development, FHML, Maastricht University, Maastricht, The Netherlands
| | - Anneke W M Kramer
- Department of Family Medicine, Leiden University, Leiden, The Netherlands
| | - Cees P M van der Vleuten
- Department of Educational Research and Development, FHML, Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
9
|
Janssen-Brandt XMC, Muijtjens AMM, Sluijsmans DMA. Toward a better judgment of item relevance in progress testing. BMC Med Educ 2017; 17:151. [PMID: 28870176 PMCID: PMC5584338 DOI: 10.1186/s12909-017-0989-x] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/27/2017] [Accepted: 08/27/2017] [Indexed: 06/07/2023]
Abstract
BACKGROUND Items must be relevant to ensure item quality and test validity. Since "item relevance" has not been operationalized yet, we developed a rubric to define it. This study explores the influence of this rubric on the assessment of item relevance and on inter-rater agreement. METHODS Members of the item review committee (RC) and students, teachers, and alumni (STA) reassessed the relevance of 50 previously used progress test (PT) items and decided about their inclusion using a 5-criteria rubric. Data were analyzed at item level using paired samples t-tests, Intraclass Correlation Coefficients (ICC), and linear regression analysis, and at rater level in a generalizability analysis per group. RESULTS The proportion of items that the RC judged relevant enough to be included decreased substantially from 1.00 to 0.72 (p < 0.001). Agreement between the RC and STA was high, with an ICC of >0.7 across items. The relation between inclusion and relevance was strong (correlation = 0.89, p < 0.001), and did not differ between RC and STA. To achieve an acceptable inter-rater reliability for relevance and inclusion, 6 members must serve on the RC. CONCLUSIONS Use of the rubric results in a stricter evaluation of items' appropriateness for inclusion in the PT and facilitates agreement between the RC and other stakeholders. Hence, it may help increase the acceptability and validity of the PT.
Collapse
Affiliation(s)
- Xandra M. C. Janssen-Brandt
- Faculty of Health, Zuyd University of Applied Sciences, Nieuw Eyckholt 300, 6419 DJ Heerlen, The Netherlands
| | - Arno M. M. Muijtjens
- Department of Educational Development and Research, Maastricht University, Universiteitssingel 60, 6229 ER Maastricht, The Netherlands
| | - Dominique M. A. Sluijsmans
- Faculty of Health, Zuyd University of Applied Sciences, Nieuw Eyckholt 300, 6419 DJ Heerlen, The Netherlands
| |
Collapse
|
10
|
Huwendiek S, Reichert F, Duncker C, de Leng BA, van der Vleuten CPM, Muijtjens AMM, Bosse HM, Haag M, Hoffmann GF, Tönshoff B, Dolmans D. Electronic assessment of clinical reasoning in clerkships: A mixed-methods comparison of long-menu key-feature problems with context-rich single best answer questions. Med Teach 2017; 39:476-485. [PMID: 28281369 DOI: 10.1080/0142159x.2017.1297525] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/06/2023]
Abstract
BACKGROUND It remains unclear which item format would best suit the assessment of clinical reasoning: context-rich single best answer questions (crSBAs) or key-feature problems (KFPs). This study compared KFPs and crSBAs with respect to students' acceptance, their educational impact, and psychometric characteristics when used in a summative end-of-clinical-clerkship pediatric exam. METHODS Fifth-year medical students (n = 377) took a computer-based exam that included 6-9 KFPs and 9-20 crSBAs which assessed their clinical reasoning skills, in addition to an objective structured clinical exam (OSCE) that assessed their clinical skills. Each KFP consisted of a case vignette and three key features using a "long-menu" question format. We explored students' perceptions of the KFPs and crSBAs in eight focus groups and analyzed statistical data of 11 exams. RESULTS Compared to crSBAs, KFPs were perceived as more realistic and difficult, providing a greater stimulus for the intense study of clinical reasoning, and were generally well accepted. The statistical analysis revealed no difference in difficulty, but KFPs resulted more reliable and efficient than crSBAs. The correlation between the two formats was high, while KFPs correlated more closely with the OSCE score. CONCLUSIONS KFPs with long-menu exams seem to bring about a positive educational effect without psychometric drawbacks.
Collapse
Affiliation(s)
- Sören Huwendiek
- a Department of Assessment and Evaluation, Institute of Medical Education Bern , University of Bern , Bern, Switzerland
| | - Friedrich Reichert
- b Department of Pediatric Cardiology and Intensive Care Medicine , Klinikum Stuttgart, Stuttgart , Germany
| | - Cecilia Duncker
- c Clinic for Child and Adolescent Psychiatry, University Hospital Kiel , Germany
| | - Bas A de Leng
- d Institute of Medical Education (IfAS), Faculty of Medicine, University of Muenster, Münster , Germany
| | - Cees P M van der Vleuten
- e Department of Educational Development and Research , Maastricht University, Maastricht , the Netherlands
| | - Arno M M Muijtjens
- e Department of Educational Development and Research , Maastricht University, Maastricht , the Netherlands
| | - Hans-Martin Bosse
- f Clinic for General Paediatrics, Neonatology and Paediatric Cardiology, University Children's Hospital Düsseldorf, Düsseldorf , Germany
| | - Martin Haag
- g GECKO Institute of Medicine, Informatics & Economics, Heilbronn University, Heilbronn , Germany
| | - Georg F Hoffmann
- h Clinic I, University Children's Hospital Heidelberg, Heidelberg , Germany
| | - Burkhard Tönshoff
- h Clinic I, University Children's Hospital Heidelberg, Heidelberg , Germany
| | - Diana Dolmans
- e Department of Educational Development and Research , Maastricht University, Maastricht , the Netherlands
| |
Collapse
|
11
|
Asante I, Andoh I, Muijtjens AMM, Donkers J. Stakeholders' perceptions on competency and assessment program of entry-level pharmacists in developing countries. Curr Pharm Teach Learn 2017; 9:360-368. [PMID: 29233272 DOI: 10.1016/j.cptl.2017.02.007] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/10/2016] [Revised: 10/11/2016] [Accepted: 02/04/2017] [Indexed: 06/07/2023]
Abstract
OBJECTIVES To assess the stakeholders' perceptions on the competency of entry-level pharmacists and the use of written licensure examination as the primary assessment for licensure decisions on entry-level pharmacists who have completed the Pharmacy Internship Program1 (PIP) in developing countries. METHOD A cross-sectional survey was conducted among stakeholders in which they completed a web-based 21-item pre-tested questionnaire to determine their views regarding the competency outcomes and assessment program for entry-level pharmacist. RESULTS The stakeholders rated the entry-level pharmacists to possess all competencies except research skills. Stakeholders suggested improvement of the program by defining the competency framework and training preceptors. However, stakeholders disagree on using written examination as the primary assessment for licensure decision and suggested the incorporation of other performance-based assessments like preceptor's assessment reports. CONCLUSION Stakeholders are uncertain on entry-level pharmacists in developing countries possessing adequate research competencies and think their assessment program for licensure need more than written examination to assess all required competencies.
Collapse
Affiliation(s)
- Isaac Asante
- Kintampo Health Research Centre Kintampo, Ghana.
| | | | - Arno M M Muijtjens
- Faculty of Health, Medicine and Life Sciences, University of Maastricht, The Netherlands.
| | - Jeroen Donkers
- Faculty of Health, Medicine and Life Sciences, University of Maastricht, The Netherlands.
| |
Collapse
|
12
|
de Melo BCP, Falbo AR, Muijtjens AMM, van der Vleuten CPM, van Merriënboer JJG. The use of instructional design guidelines to increase effectiveness of postpartum hemorrhage simulation training. Int J Gynaecol Obstet 2017; 137:99-105. [PMID: 28090643 DOI: 10.1002/ijgo.12084] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2016] [Revised: 10/24/2016] [Accepted: 12/12/2016] [Indexed: 11/09/2022]
Abstract
OBJECTIVE To compare learning outcomes of postpartum hemorrhage simulation training based on either instructional design guidelines or best practice. METHODS A pretest-post-test non-equivalent groups study was conducted among obstetrics and gynecology residents in Recife, Brazil, from June 8 to August 30, 2013. The instructional design group included 13 teams, whereas the best practice group included seven teams. A standardized task checklist was used for scenario analysis and the proportion of correctly executed tasks compared (post-test minus pretest). RESULTS The instructional design group scored higher than the best practice group for total number of tasks completed (median difference 0.46 vs 0.17; P<0.001; effect size [r]=0.72). Similar results were observed for communication (median difference 0.56 vs 0.22; P=0.004; r=0.58), laboratory evaluation (median difference 0.83 vs 0.00; P<0.001; r=0.76), and mechanical management (median difference 0.25 vs -0.15; P=0.048; r=0.39). Speed of learning was also increased. The median differences were 0.20 for the instructional design group compared with 0.05 for the best practice group at 60 seconds (P=0.015; r=0.49), and 0.49 versus 0.26 (P=0.001; r=0.65) at 360 seconds. CONCLUSION The use of simulation training for postpartum hemorrhage that was based on instructional design guidelines yielded better learning outcomes than did training based on best practice.
Collapse
Affiliation(s)
- Brena C P de Melo
- Faculdade Pernambucana de Saúde and Centro de Atenção à Mulher, Instituto de Medicina Integral Prof. Fernando Figueira, Centro de Atenção à Mulher, Recife, Brazil.,School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, Netherlands
| | - Ana R Falbo
- Faculdade Pernambucana de Saúde and Centro de Atenção à Mulher, Instituto de Medicina Integral Prof. Fernando Figueira, Centro de Atenção à Mulher, Recife, Brazil
| | - Arno M M Muijtjens
- School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, Netherlands
| | - Cees P M van der Vleuten
- School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, Netherlands
| | - Jeroen J G van Merriënboer
- School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, Netherlands
| |
Collapse
|
13
|
Matsuyama Y, Muijtjens AMM, Kikukawa M, Stalmeijer R, Murakami R, Ishikawa S, Okazaki H. A first report of East Asian students' perception of progress testing: a focus group study. BMC Med Educ 2016; 16:245. [PMID: 27658501 PMCID: PMC5034519 DOI: 10.1186/s12909-016-0766-2] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/10/2016] [Accepted: 09/14/2016] [Indexed: 05/25/2023]
Abstract
BACKGROUND Progress testing (PT) is used in Western countries to evaluate students' level of functional knowledge, and to enhance meaning-oriented and self-directed learning. However, the use of PT has not been investigated in East Asia, where reproduction-oriented and teacher-centered learning styles prevail. Here, we explored the applicability of PT by focusing on student perceptions. METHODS Twenty-four students from Years 2, 3, and 5 at Jichi Medical University in Japan attended a pilot PT session preceded by a brief introduction of its concept and procedures. Variations in obtained test scores were analyzed by year, and student perceptions of PT were explored using focus groups. RESULTS Formula scores (mean ± standard deviation) in Years 2, 3, and 5 were 12.63 ± 3.53, 35.88 ± 14.53, and 71.00 ± 18.31, respectively. Qualitative descriptive analysis of focus group data showed that students disfavored testing of medical knowledge without tangible goals, but instead favored repetitive assessment of knowledge that had been learned and was tested on a unit basis in the past in order to achieve deep learning. Further, students of all school years considered that post-test explanatory lectures by teachers were necessary. CONCLUSIONS East Asian students' perceptions indicated that, in addition to their intensive memorization within narrow test domains compartmentalized by end-of-unit tests, the concept of PT was suitable for repetitive memorization, as it helped them to integrate their knowledge and to increase their understanding. Post-test explanatory lectures might lessen their dislike of the intangible goals of PT, but at the expense of delaying the development of self-directed learning. Key issues for the optimization of PT in East Asia may include administration of PT after completed end-of-unit tests and a gradual change in feedback methodology over school years from test-oriented post-test lectures to the provision of literature references only, as a means of enhancing test self-review and self-directed learning.
Collapse
Affiliation(s)
- Yasushi Matsuyama
- Medical Education Center, Jichi Medical University, 3311-1 Yakushiji, Shimotsuke, 329-0498 Tochigi Japan
| | - Arno M. M. Muijtjens
- Department of Educational Development and Research, Faculty of Health, Medicine, and Life Sciences, Maastricht University, 6200 MD Maastricht, The Netherlands
| | - Makoto Kikukawa
- Department of Medical Education, Kyushu University, 3-1-1 Maidashi, Higashi-ku, Fukuoka, Kyushu Japan
| | - Renee Stalmeijer
- Department of Educational Development and Research, Faculty of Health, Medicine, and Life Sciences, Maastricht University, 6200 MD Maastricht, The Netherlands
| | - Reiko Murakami
- Jichi Medical University of Nursing, Japan, 3311-1 Yakushiji, Shimotsuke, Tochigi Japan
| | - Shizukiyo Ishikawa
- Medical Education Center, Jichi Medical University, 3311-1 Yakushiji, Shimotsuke, 329-0498 Tochigi Japan
| | - Hitoaki Okazaki
- Medical Education Center, Jichi Medical University, 3311-1 Yakushiji, Shimotsuke, 329-0498 Tochigi Japan
| |
Collapse
|
14
|
Michels NRM, Avonts M, Peeraer G, Ulenaers K, Van Gaal LF, Bossaert LL, Driessen EW, Muijtjens AMM, De Winter BY. Content validity of workplace-based portfolios: A multi-centre study. Med Teach 2016; 38:936-45. [PMID: 26829024 DOI: 10.3109/0142159x.2015.1132407] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
BACKGROUND Portfolios are used as tools to coach and assess students in the workplace. This study sought to evaluate the content validity of portfolios as reflected in their capacity to adequately assess achieved competences of medical students during clerkships. METHODS We reviewed 120 workplace portfolios at three medical universities (Belgium and the Netherlands). To validate their content, we developed a Validity Inventory for Portfolio Assessment (VIPA) based on the CanMEDS roles. Two raters evaluated each portfolio and indicated for each VIPA item whether the portfolio provided sufficient information to enable satisfactory assessment of the item. We ran a descriptive analysis on the validation data and computed Cohen's Kappa to investigate interrater agreement. RESULTS The portfolios adequately covered the items pertaining to the communicator (90%) and professional (87%) roles. Coverage of the medical expert, collaborator, scholar and manager roles ranged between 75% and 85%. The health advocate role, covering 59%, was clearly less well represented. This role also exhibited little interrater agreement (Kappa < 0.4). CONCLUSIONS This study lends further credence to the evidence that portfolios can indeed adequately assess the different CanMEDS roles during clerkships, the health advocate role, which was less well represented in the portfolio content, excepted.
Collapse
|
15
|
de Vries AH, Schout BMA, van Merriënboer JJG, Pelger RCM, Koldewijn EL, Muijtjens AMM, Wagner C. High educational impact of a national simulation-based urological curriculum including technical and non-technical skills. Surg Endosc 2016; 31:928-936. [PMID: 27387182 DOI: 10.1007/s00464-016-5060-1] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/14/2015] [Accepted: 06/16/2016] [Indexed: 11/28/2022]
Abstract
BACKGROUND Although simulation training is increasingly used to meet modern technology and patient safety demands, its successful integration within surgical curricula is still rare. The Dutch Urological Practical Skills (D-UPS) curriculum provides modular simulation-based training of technical and non-technical basic urological skills in the local hospital setting. This study aims to assess the educational impact of implementing the D-UPS curriculum in the Netherlands and to provide focus points for improvement of the D-UPS curriculum according to the participants. METHODS Educational impact was assessed by means of qualitative individual module-specific feedback and a quantitative cross-sectional survey among residents and supervisors. Twenty out of 26 Dutch teaching hospitals participated. The survey focussed on practical aspects, the D-UPS curriculum in general, and the impact of the D-UPS curriculum on the development of technical and non-technical skills. RESULTS A considerable survey response of 95 % for residents and 76 % for supervisors was obtained. Modules were attended by junior and senior residents, supervised by a urologist, and peer teaching was used. Ninety percent of supervisors versus 67 % of residents judged the D-UPS curriculum as an important addition to current residency training (p = 0.007). Participants' aggregated general judgement of the modules showed a substantial percentage favorable score (M ± SE: 57 ± 4 %). The impact of training on, e.g., knowledge of materials/equipment and ability to anticipate on complications was high, especially for junior residents (77 ± 5 and 71 ± 7 %, respectively). Focus points for improvement of the D-UPS curriculum according to the participants include adaptation of the training level to residents' level of experience and focus on logistics. CONCLUSION The simulation-based D-UPS curriculum has a high educational impact. Residents and supervisors consider the curriculum to be an important addition to current residency training. Focus points for improvement of the D-UPS curriculum according to the participants include increased attention to logistics and integration of a spiral learning approach.
Collapse
Affiliation(s)
- Anna H de Vries
- Department of Urology, Catharina Hospital, Michelangelolaan 2, 5623 EJ, Eindhoven, The Netherlands.
| | - Barbara M A Schout
- Department of Urology, Alrijne Hospital, Leiden, The Netherlands.,Netherlands Institute for Health Services Research (NIVEL), Utrecht, The Netherlands
| | | | - Rob C M Pelger
- Department of Urology, University Medical Center Leiden, Leiden, The Netherlands
| | - Evert L Koldewijn
- Department of Urology, Catharina Hospital, Michelangelolaan 2, 5623 EJ, Eindhoven, The Netherlands.,School of Health Professions Education, Maastricht University, Maastricht, The Netherlands
| | - Arno M M Muijtjens
- Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
| | - Cordula Wagner
- Netherlands Institute for Health Services Research (NIVEL), Utrecht, The Netherlands.,Department of Public and Occupational Health, EMGO Institute for Health and Care Research, Amsterdam, The Netherlands
| |
Collapse
|
16
|
Ravesloot CJ, Van der Schaaf MF, Muijtjens AMM, Haaring C, Kruitwagen CLJJ, Beek FJA, Bakker J, Van Schaik JPJ, Ten Cate TJ. The don't know option in progress testing. Adv Health Sci Educ Theory Pract 2015; 20:1325-38. [PMID: 25912621 PMCID: PMC4639571 DOI: 10.1007/s10459-015-9604-2] [Citation(s) in RCA: 17] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/30/2014] [Accepted: 03/25/2015] [Indexed: 05/12/2023]
Abstract
Formula scoring (FS) is the use of a don't know option (DKO) with subtraction of points for wrong answers. Its effect on construct validity and reliability of progress test scores, is subject of discussion. Choosing a DKO may not only be affected by knowledge level, but also by risk taking tendency, and may thus introduce construct-irrelevant variance into the knowledge measurement. On the other hand, FS may result in more reliable test scores. To evaluate the impact of FS on construct validity and reliability of progress test scores, a progress test for radiology residents was divided into two tests of 100 parallel items (A and B). Each test had a FS and a number-right (NR) version, A-FS, B-FS, A-NR, and B-NR. Participants (337) were randomly divided into two groups. One group took test A-FS followed by B-NR, and the second group test B-FS followed by A-NR. Evidence for impaired construct validity was sought in a hierarchical regression analysis by investigating how much of the participants' FS-score variance was explained by the DKO-score, compared to the contribution of the knowledge level (NR-score), while controlling for Group, Gender, and Training length. Cronbach's alpha was used to estimate NR and FS-score reliability per year group. NR score was found to explain 27 % of the variance of FS [F(1,332) = 219.2, p < 0.0005], DKO-score, and the interaction of DKO and Gender were found to explain 8 % [F(2,330) = 41.5, p < 0.0005], and the interaction of DKO and NR 1.6 % [F(1,329) = 16.6, p < 0.0005], supporting our hypothesis that FS introduces construct-irrelevant variance into the knowledge measurement. However, NR-scores showed considerably lower reliabilities than FS-scores (mean year-test group Cronbach's alphas were 0.62 and 0.74, respectively). Decisions about FS with progress tests should be a careful trade-off between systematic and random measurement error.
Collapse
Affiliation(s)
- C J Ravesloot
- Radiology Department, University Medical Center Utrecht, Heidelberglaan 100, Room E01.132, 3508 GA, Utrecht, The Netherlands.
| | | | | | - C Haaring
- Radiology Department, University Medical Center Utrecht, Heidelberglaan 100, 3508 GA, Utrecht, The Netherlands
| | - C L J J Kruitwagen
- Julius Center, University Medical Center Utrecht, Heidelberglaan 100, 3508 GA, Utrecht, The Netherlands
| | - F J A Beek
- Radiology Department, University Medical Center Utrecht, Heidelberglaan 100, 3508 GA, Utrecht, The Netherlands
| | - J Bakker
- Albert Schweitzer Hospital, Dordrecht, The Netherlands
| | - J P J Van Schaik
- Radiology Department, University Medical Center Utrecht, Heidelberglaan 100, Room E01.132, 3508 GA, Utrecht, The Netherlands
| | - Th J Ten Cate
- Center for Research and Development of Education, University Medical Center Utrecht, Heidelberglaan 100, 3508 GA, Utrecht, The Netherlands
| |
Collapse
|
17
|
Jacobs JCG, Muijtjens AMM, Van Luijk SJ, Van Der Vleuten CPM, Croiset G, Scheele F. Impact of institute and person variables on teachers' conceptions of learning and teaching. Med Teach 2015; 37:738-746. [PMID: 25373886 DOI: 10.3109/0142159x.2014.970985] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/04/2023]
Abstract
BACKGROUND Teachers' conceptions of learning and teaching are important for faculty development to result in enduring changes in teaching practices. Until now, studies on these conceptions have mostly focused on traditional, lecture-based curricula rather than on small-group student-centred educational formats, which are gaining ground worldwide. AIM To explore which factors predict teachers' conceptions in student-centred curricula. METHODS In two Dutch medical schools with 10 and 40 years of student-centred education, teachers were asked to fill out the Conceptions of Learning and Teaching (COLT) Questionnaire to assess their 'teacher-centredness', 'appreciation of active learning' and 'orientation to professional practice'. Next, we quantitatively assessed the relations of teachers' conceptions with their personal and occupational characteristics and institute. RESULTS Overall response was 49.4% (N = 319/646). Institute was the main predictor for variance in all three scales, and discipline, gender and teaching experience significantly explained variance in two of the scales. More than 80% of the variance was not explained by these factors. CONCLUSION Longer exposure to a student-centred curriculum was associated with fewer teacher-centred conceptions, greater 'appreciation of active learning' and stronger 'orientation towards professional practice'. In line with studies on lecture-based curricula, discipline, gender and teaching experience also appeared important for teachers' conceptions in student-centred curricula. More research is necessary to better understand the influence of institute on the three teachers' conceptions scales.
Collapse
Affiliation(s)
| | | | | | | | | | - Fedde Scheele
- a VU University Amsterdam , the Netherlands
- d St Lucas Andreas Hospital , the Netherlands
| |
Collapse
|
18
|
Huwendiek S, De Leng BA, Kononowicz AA, Kunzmann R, Muijtjens AMM, Van Der Vleuten CPM, Hoffmann GF, Tönshoff B, Dolmans DHJM. Exploring the validity and reliability of a questionnaire for evaluating virtual patient design with a special emphasis on fostering clinical reasoning. Med Teach 2015; 37:775-782. [PMID: 25313931 DOI: 10.3109/0142159x.2014.970622] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/04/2023]
Abstract
BACKGROUND Virtual patients (VPs) are increasingly used to train clinical reasoning. So far, no validated evaluation instruments for VP design are available. AIMS We examined the validity of an instrument for assessing the perception of VP design by learners. METHODS Three sources of validity evidence were examined: (i) Content was examined based on theory of clinical reasoning and an international VP expert team. (ii) The response process was explored in think-aloud pilot studies with medical students and in content analyses of free text questions accompanying each item of the instrument. (iii) Internal structure was assessed by exploratory factor analysis (EFA) and inter-rater reliability by generalizability analysis. RESULTS Content analysis was reasonably supported by the theoretical foundation and the VP expert team. The think-aloud studies and analysis of free text comments supported the validity of the instrument. In the EFA, using 2547 student evaluations of a total of 78 VPs, a three-factor model showed a reasonable fit with the data. At least 200 student responses are needed to obtain a reliable evaluation of a VP on all three factors. CONCLUSION The instrument has the potential to provide valid information about VP design, provided that many responses per VP are available.
Collapse
Affiliation(s)
- Sören Huwendiek
- a University of Bern , Switzerland
- b University Children's Hospital Heidelberg , Germany
| | | | - Andrzej A Kononowicz
- d Jagiellonian University Medical College , Poland
- e Karolinska Institutet , Sweden
| | | | | | | | | | | | | |
Collapse
|
19
|
Houwink EJF, Muijtjens AMM, van Teeffelen SR, Henneman L, Rethans JJ, Jacobi F, van der Jagt L, Stirbu I, van Luijk SJ, Stumpel CTRM, Meijers-Heijboer HE, van der Vleuten C, Cornel MC, Dinant GJ. Effect of comprehensive oncogenetics training interventions for general practitioners, evaluated at multiple performance levels. PLoS One 2015; 10:e0122648. [PMID: 25837634 PMCID: PMC4383330 DOI: 10.1371/journal.pone.0122648] [Citation(s) in RCA: 26] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2014] [Accepted: 02/11/2015] [Indexed: 11/19/2022] Open
Abstract
General practitioners (GPs) are increasingly called upon to identify patients at risk for hereditary cancers, and their genetic competencies need to be enhanced. This article gives an overview of a research project on how to build effective educational modules on genetics, assessed by randomized controlled trials (RCTs), reflecting the prioritized educational needs of primary care physicians. It also reports on an ongoing study to investigate long-term increase in genetic consultation skills (1-year follow-up) and interest in and satisfaction with a supportive website on genetics among GPs. Three oncogenetics modules were developed: an online Continuing Professional Development (G-eCPD) module, a live genetic CPD module, and a "GP and genetics" website (huisartsengenetica.nl) providing further genetics information applicable in daily practice. Three assessments to evaluate the effectiveness (1-year follow-up) of the oncogenetic modules were designed: 1.An online questionnaire on self-reported genetic competencies and changes in referral behaviour, 2.Referral rates from GPs to clinical genetics centres and 3.Satisfaction questionnaire and visitor count analytics of supportive genetics website. The setting was Primary care in the Netherlands and three groups of study participants were included in the reported studies:. Assessment 1. 168 GPs responded to an email invitation and were randomly assigned to an intervention or control group, evaluating the G-eCPD module (n = 80) or the live module (n = 88). Assessment 2. Referral rates by GPs were requested from the clinical genetics centres, in the northern and southern parts of the Netherlands (Amsterdam and Maastricht), for the two years before (2010 [n = 2510] and 2011 [n = 2940]) and the year after (2012 [n = 2875]) launch of the oncogenetics CPD modules and the website. Assessment 3. Participants of the website evaluation were all recruited online. When they visited the website during the month of February 2013, a pop-up invitation came up. Of the 1350 unique visitors that month, only 38 completed the online questionnaire. Main outcomes measure showed long-term (self-reported) genetic consultation skills (i.e. increased genetics awareness and referrals to clinical genetics centres) among GPs who participated in the oncogenetic training course, and interest in and satisfaction with the supportive website. 42 GPs (52%) who previously participated in the G-eCPD evaluation study and 50 GPs (57%) who participated in the live training programme responded to the online questionnaire on long-term effects of educational outcome. Previous RCTs showed that the genetics CPD modules achieved sustained improvement of oncogenetic knowledge and consultation skills (3-months follow-up). Participants of these RCTs reported being more aware of genetic problems long term; this was reported by 29 GPs (69%) and 46 GPs (92%) participating in the G-eCPD and live module evaluation studies, respectively (Chisquare test, p<0.005). One year later, 68% of the respondents attending the live training reported that they more frequently referred patients to the clinical genetics centres, compared to 29% of those who attended the online oncogenetics training (Chisquare test, p<0.0005). However, the clinical genetics centres reported no significant change in referral numbers one year after the training. Website visitor numbers increased, as did satisfaction, reflected in a 7.7 and 8.1 (out of 10) global rating of the website (by G-eCPD and live module participants, respectively). The page most often consulted was "family tree drawing". Self-perceived genetic consultation skills increased long-term and GPs were interested in and satisfied with the supportive website. Further studies are necessary to see whether the oncogenetics CPD modules result in more efficient referral. The results presented suggest we have provided a flexible and effective framework to meet the need for effective educational programmes for non-geneticist healthcare providers, enabling improvement of genetic medical care.
Collapse
Affiliation(s)
- Elisa J. F. Houwink
- Department of Clinical Genetics, Section Community Genetics, EMGO Institute for Health and Care Research, VU University Medical Center, Amsterdam, The Netherlands
- School for Public Health and Primary Care, Department of Family Medicine, Maastricht University, Maastricht, The Netherlands
- * E-mail:
| | - Arno M. M. Muijtjens
- Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
| | - Sarah R. van Teeffelen
- Department of Clinical Genetics, Section Community Genetics, EMGO Institute for Health and Care Research, VU University Medical Center, Amsterdam, The Netherlands
| | - Lidewij Henneman
- Department of Clinical Genetics, Section Community Genetics, EMGO Institute for Health and Care Research, VU University Medical Center, Amsterdam, The Netherlands
| | - Jan Joost Rethans
- Skillslab, Faculty of Health, Medicine & Life Sciences, Maastricht University, Maastricht, The Netherlands
| | - Florijn Jacobi
- Dutch College of General practitioners (NHG), Utrecht, The Netherlands
| | | | - Irina Stirbu
- Centre for Infectious Disease Control, National Institute for Public Health and the Environment, Bilthoven, The Netherlands
| | - Scheltus J. van Luijk
- Department of Resident Training, Maastricht University Medical Centre +, Maastricht, The Netherlands
| | - Connie T. R. M. Stumpel
- Department of Clinical Genetics, School for Oncology and Developmental Biology, Maastricht UMC+, Maastricht, The Netherlands
| | | | - Cees van der Vleuten
- Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
| | - Martina C. Cornel
- Department of Clinical Genetics, Section Community Genetics, EMGO Institute for Health and Care Research, VU University Medical Center, Amsterdam, The Netherlands
| | - Geert Jan Dinant
- School for Public Health and Primary Care, Department of Family Medicine, Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
20
|
Bok HGJ, Teunissen PW, Boerboom TBB, Rhind SM, Baillie S, Tegzes J, Annandale H, Matthew S, Torgersen A, Hecker KG, Härdi-Landerer CM, Gomez-Lucia E, Ahmad B, Muijtjens AMM, Jaarsma DADC, van der Vleuten CPM, van Beukelen P. International survey of veterinarians to assess the importance of competencies in professional practice and education. J Am Vet Med Assoc 2014; 245:906-13. [DOI: 10.2460/javma.245.8.906] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
|
21
|
Berkenbosch L, Muijtjens AMM, Zimmermann LJI, Heyligers IC, Scherpbier AJJA, Busari JO. A pilot study of a practice management training module for medical residents. BMC Med Educ 2014; 14:107. [PMID: 24885442 PMCID: PMC4038828 DOI: 10.1186/1472-6920-14-107] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/12/2013] [Accepted: 05/14/2014] [Indexed: 05/21/2023]
Abstract
BACKGROUND In 2005 a competency based curriculum was introduced in the Dutch postgraduate medical training programs. While the manager's role is one of the seven key competencies, there is still no formal management course in most postgraduate curricula. Based on a needs assessment we conducted, several themes were identified as important for a possible management training program. We present the results of the pilot training we performed to investigate two of these themes. METHODS The topics "knowledge of the healthcare system" and "time management" were developed from the list of suggested management training themes. Fourteen residents participated in the training and twenty-four residents served as control. The training consisted of two sessions of four hours with a homework assignment in between. 50 True/false-questions were given as pre- and post-test to both the test and control groups to assess the level of acquired knowledge among the test group as well as the impact of the intervention. We also performed a qualitative evaluation using evaluation forms and in-depth interviews. RESULTS All fourteen residents completed the training. Six residents in the control group were lost to follow up. The pre- and post-test showed improvement among the participating residents in comparison to the residents from the control group, but this improvement was not significant. The qualitative assessment showed that all residents evaluated the training positively and experienced it as a useful addition to their training in becoming a medical specialist. CONCLUSION Our training was evaluated positively and considered to be valuable. This study supports the need for mandatory medical management training as part of the postgraduate medical curriculum. Our training could be an example of how to teach two important themes in the broad area of medical management education.
Collapse
Affiliation(s)
- Lizanne Berkenbosch
- School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, P,O, Box 616, 6200, MD, Maastricht, the Netherlands.
| | | | | | | | | | | |
Collapse
|
22
|
Naeem N, van der Vleuten C, Muijtjens AMM, Violato C, Ali SM, Al-Faris EA, Hoogenboom R, Naeem N. Correlates of emotional intelligence: results from a multi-institutional study among undergraduate medical students. Med Teach 2014; 36 Suppl 1:S30-S35. [PMID: 24617782 DOI: 10.3109/0142159x.2014.886008] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/03/2023]
Abstract
BACKGROUND Emotional Intelligence (EI) is the ability to deal with your own and others emotions. Medical students are inducted into medical schools on the basis of their academic achievement. Professionally, however, their success rate is variable and may depend on their interpersonal relationships. EI is thought to be significant in achieving good interpersonal relationships and success in life and career. Therefore, it is important to measure EI and understand its correlates in an undergraduate medical student population. AIM The objective of study was to investigate the relationship between the EI of medical students and their academic achievement (based on cumulative grade point average [CGPA]), age, gender and year of study. METHODS A cross-sectional survey design was used. The SSREIS and demographic survey were administered in the three medical schools in Saudi Arabia from April to May 2012. RESULTS The response rate was 30%. For the Optimism subscale, the mean score was M = 3.79, SD ± 0.54 (α = 0.82), for Awareness-of-emotion subscale M = 3.94, SD ± 0.57 (α = 0.72) and for Use-of-emotion subscale M = 3.92, SD ± 0.54 (α = 0.63). Multiple regression showed a significant positive correlation between CGPA and the EI of medical students (r = 0.246, p = 0.000) on the Optimism subscale. No correlation was seen between CGPA and Awareness of Emotions and Use of Emotions subscales. No relationship was seen for the other independent variables. CONCLUSION The current study demonstrates that CGPA is the only significant predictor, indicating that Optimism tends to be higher for students with a higher CPGA. None of the other independent variables (age, year of study, gender) showed a significant relationship.
Collapse
|
23
|
Hejri SM, Jalili M, Muijtjens AMM, Van Der Vleuten CPM. Assessing the reliability of the borderline regression method as a standard setting procedure for objective structured clinical examination. J Res Med Sci 2013; 18:887-91. [PMID: 24497861 PMCID: PMC3897074] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/12/2013] [Revised: 01/28/2013] [Accepted: 04/28/2013] [Indexed: 11/05/2022]
Abstract
BACKGROUND One of the methods used for standard setting is the borderline regression method (BRM). This study aims to assess the reliability of BRM when the pass-fail standard in an objective structured clinical examination (OSCE) was calculated by averaging the BRM standards obtained for each station separately. MATERIALS AND METHODS In nine stations of the OSCE with direct observation the examiners gave each student a checklist score and a global score. Using a linear regression model for each station, we calculated the checklist score cut-off on the regression equation for the global scale cut-off set at 2. The OSCE pass-fail standard was defined as the average of all station's standard. To determine the reliability, the root mean square error (RMSE) was calculated. The R (2) coefficient and the inter-grade discrimination were calculated to assess the quality of OSCE. RESULTS The mean total test score was 60.78. The OSCE pass-fail standard and its RMSE were 47.37 and 0.55, respectively. The R (2) coefficients ranged from 0.44 to 0.79. The inter-grade discrimination score varied greatly among stations. CONCLUSION The RMSE of the standard was very small indicating that BRM is a reliable method of setting standard for OSCE, which has the advantage of providing data for quality assurance.
Collapse
Affiliation(s)
- Sara Mortaz Hejri
- Department of Medical Education, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran
| | - Mohammad Jalili
- Department of Emergency Medicine, Department of Medical Education, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran,Address for correspondence: Dr. Mohammad Jalili, 7th floor, Tehran University of Medical Sciences, Ghods Street, Keshavarz Blvd, Tehran, Iran. E-mail:
| | - Arno M. M. Muijtjens
- Department of Medical Education and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
| | - Cees P. M. Van Der Vleuten
- Department of Educational Development and Research, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
24
|
Heijne-Penninga M, Kuks JBM, Hofman WHA, Muijtjens AMM, Cohen-Schotanus J. Influence of PBL with open-book tests on knowledge retention measured with progress tests. Adv Health Sci Educ Theory Pract 2013; 18:485-95. [PMID: 22736061 PMCID: PMC3728438 DOI: 10.1007/s10459-012-9386-8] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/27/2012] [Accepted: 06/08/2012] [Indexed: 05/05/2023]
Abstract
The influence of problem-based learning (PBL) and open-book tests on long-term knowledge retention is unclear and subject of discussion. Hypotheses were that PBL as well as open-book tests positively affect long-term knowledge retention. Four progress test results of fifth and sixth-year medical students (n = 1,648) of three medical schools were analyzed. Two schools had PBL driven curricula, and the third one had a traditional curriculum (TC). One of the PBL schools (PBLob) used a combination of open-book (assessing backup knowledge) and closed-book tests (assessing core knowledge); the other two schools (TC and PBLcb) only used closed-book tests. The items of the progress tests were divided into core and backup knowledge. T tests (with Bonferroni correction) were used to analyze differences between curricula. PBL students performed significantly better than TC students on core knowledge (average effect size (av ES) = 0.37-0.74) and PBL students tested with open-book tests scored somewhat higher than PBL students tested without such tests (av ES = 0.23-0.30). Concerning backup knowledge, no differences were found between the scores of the three curricula. Students of the two PBL curricula showed a substantially better long-term knowledge retention than TC students. PBLob students performed somewhat better on core knowledge than PBLcb students. These outcomes suggest that a problem-based instructional approach in particular can stimulate long-term knowledge retention. Distinguishing knowledge into core and backup knowledge and using open-book tests alongside closed-book tests could enhance long-term core knowledge retention.
Collapse
Affiliation(s)
- M Heijne-Penninga
- Institute for Medical Education, University of Groningen and University Medical Center Groningen, A. Deusinglaan 1, 9713 AV Groningen, The Netherlands.
| | | | | | | | | |
Collapse
|
25
|
Govaerts MJB, Van de Wiel MWJ, Schuwirth LWT, Van der Vleuten CPM, Muijtjens AMM. Workplace-based assessment: raters' performance theories and constructs. Adv Health Sci Educ Theory Pract 2013; 18:375-96. [PMID: 22592323 PMCID: PMC3728456 DOI: 10.1007/s10459-012-9376-x] [Citation(s) in RCA: 116] [Impact Index Per Article: 10.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/21/2011] [Accepted: 04/25/2012] [Indexed: 05/14/2023]
Abstract
Weaknesses in the nature of rater judgments are generally considered to compromise the utility of workplace-based assessment (WBA). In order to gain insight into the underpinnings of rater behaviours, we investigated how raters form impressions of and make judgments on trainee performance. Using theoretical frameworks of social cognition and person perception, we explored raters' implicit performance theories, use of task-specific performance schemas and the formation of person schemas during WBA. We used think-aloud procedures and verbal protocol analysis to investigate schema-based processing by experienced (N = 18) and inexperienced (N = 16) raters (supervisor-raters in general practice residency training). Qualitative data analysis was used to explore schema content and usage. We quantitatively assessed rater idiosyncrasy in the use of performance schemas and we investigated effects of rater expertise on the use of (task-specific) performance schemas. Raters used different schemas in judging trainee performance. We developed a normative performance theory comprising seventeen inter-related performance dimensions. Levels of rater idiosyncrasy were substantial and unrelated to rater expertise. Experienced raters made significantly more use of task-specific performance schemas compared to inexperienced raters, suggesting more differentiated performance schemas in experienced raters. Most raters started to develop person schemas the moment they began to observe trainee performance. The findings further our understanding of processes underpinning judgment and decision making in WBA. Raters make and justify judgments based on personal theories and performance constructs. Raters' information processing seems to be affected by differences in rater expertise. The results of this study can help to improve rater training, the design of assessment instruments and decision making in WBA.
Collapse
Affiliation(s)
- M J B Govaerts
- Department of Educational Research and Development, FHML, Maastricht University, PO Box 616, 6200 MD Maastricht, The Netherlands.
| | | | | | | | | |
Collapse
|
26
|
Jippes M, Driessen EW, Majoor GD, Gijselaers WH, Muijtjens AMM, van der Vleuten CPM. Impact of national context and culture on curriculum change: a case study. Med Teach 2013; 35:661-70. [PMID: 23627360 DOI: 10.3109/0142159x.2013.785629] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/11/2023]
Abstract
BACKGROUND Earlier studies suggested national culture to be a potential barrier to curriculum reform in medical schools. In particular, Hofstede's cultural dimension 'uncertainty avoidance' had a significant negative relationship with the implementation rate of integrated curricula. AIMS However, some schools succeeded to adopt curriculum changes despite their country's strong uncertainty avoidance. This raised the question: 'How did those schools overcome the barrier of uncertainty avoidance?' METHOD Austria offered the combination of a high uncertainty avoidance score and integrated curricula in all its medical schools. Twenty-seven key change agents in four medical universities were interviewed and transcripts analysed using thematic cross-case analysis. RESULTS Initially, strict national laws and limited autonomy of schools inhibited innovation and fostered an 'excuse culture': 'It's not our fault. It is the ministry's'. A new law increasing university autonomy stimulated reforms. However, just this law would have been insufficient as many faculty still sought to avoid change. A strong need for change, supportive and continuous leadership, and visionary change agents were also deemed essential. CONCLUSIONS In societies with strong uncertainty avoidance strict legislation may enforce resistance to curriculum change. In those countries opposition by faculty can be overcome if national legislation encourages change, provided additional internal factors support the change process.
Collapse
Affiliation(s)
- Mariëlle Jippes
- Department of Educational Development and Research FHML, Maastricht University, POB 616, NL-6200 MD Maastricht, The Netherlands.
| | | | | | | | | | | |
Collapse
|
27
|
Houwink EJF, Muijtjens AMM, van Teeffelen SR, Henneman L, Rethans JJ, van der Jagt LEJ, van Luijk SJ, Dinant GJ, van der Vleuten C, Cornel MC. Effectiveness of oncogenetics training on general practitioners' consultation skills: a randomized controlled trial. Genet Med 2013; 16:45-52. [PMID: 23722870 PMCID: PMC3914027 DOI: 10.1038/gim.2013.69] [Citation(s) in RCA: 28] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2013] [Accepted: 04/16/2013] [Indexed: 01/25/2023] Open
Abstract
PURPOSE General practitioners are increasingly called upon to deliver genetic services and could play a key role in translating potentially life-saving advancements in oncogenetic technologies to patient care. If general practitioners are to make an effective contribution in this area, their genetics competencies need to be upgraded. The aim of this study was to investigate whether oncogenetics training for general practitioners improves their genetic consultation skills. METHODS In this pragmatic, blinded, randomized controlled trial, the intervention consisted of a 4-h training (December 2011 and April 2012), covering oncogenetic consultation skills (family history, familial risk assessment, and efficient referral), attitude (medical ethical issues), and clinical knowledge required in primary-care consultations. Outcomes were measured using observation checklists by unannounced standardized patients and self-reported questionnaires. RESULTS Of 88 randomized general practitioners who initially agreed to participate, 56 completed all measurements. Key consultation skills significantly and substantially improved; regression coefficients after intervention were equivalent to 0.34 and 0.28 at 3-month follow-up, indicating a moderate effect size. Satisfaction and perceived applicability of newly learned skills were highly scored. CONCLUSION The general practitioner-specific training proved to be a feasible, satisfactory, and clinically applicable method to improve oncogenetics consultation skills and could be used as an educational framework to inform future training activities with the ultimate aim of improving medical care.
Collapse
Affiliation(s)
- Elisa J F Houwink
- 1] Department of Clinical Genetics, Section of Community Genetics, EMGO Institute for Health and Care Research, VU University Medical Center, Amsterdam, The Netherlands [2] Department of General Practice, School for Public Health and Primary Care, Maastricht University, Maastricht, The Netherlands
| | - Arno M M Muijtjens
- Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
| | - Sarah R van Teeffelen
- Department of Clinical Genetics, Section of Community Genetics, EMGO Institute for Health and Care Research, VU University Medical Center, Amsterdam, The Netherlands
| | - Lidewij Henneman
- Department of Clinical Genetics, Section of Community Genetics, EMGO Institute for Health and Care Research, VU University Medical Center, Amsterdam, The Netherlands
| | - Jan Joost Rethans
- Skills Lab, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
| | | | - Scheltus J van Luijk
- Department of Education and Resident Training, Maastricht University Medical Centre, Maastricht, The Netherlands
| | - Geert Jan Dinant
- Department of General Practice, School for Public Health and Primary Care, Maastricht University, Maastricht, The Netherlands
| | - Cees van der Vleuten
- Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
| | - Martina C Cornel
- Department of Clinical Genetics, Section of Community Genetics, EMGO Institute for Health and Care Research, VU University Medical Center, Amsterdam, The Netherlands
| |
Collapse
|
28
|
Berkenbosch L, Bax M, Scherpbier A, Heyligers I, Muijtjens AMM, Busari JO. How Dutch medical specialists perceive the competencies and training needs of medical residents in healthcare management. Med Teach 2013; 35:e1090-e1102. [PMID: 23137237 DOI: 10.3109/0142159x.2012.731544] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/01/2023]
Abstract
BACKGROUND The Dutch postgraduate medical training has been revised to focus on seven competencies. The role as manager is one of these competencies. Recent studies show that this competency receives little attention during the residency training. In an earlier study, we discovered that residents perceived their competency as managers to be moderate. AIM In this study, we investigated how medical specialists perceived the managerial competencies of medical residents and their need for management education. METHODS In September 2010, a 46-item questionnaire was designed which examined medical specialists' perceptions of the competency and needs of residents in the field of medical management. Two hundred ninety-eight specialists were invited via email to participate. RESULTS Hundred twenty-nine specialists (43.3%) responded to our survey. They rated the residents' competencies in contract negotiating skills, knowledge of the healthcare system, and specialist department poorly. They felt that residents were competent in updating their medical knowledge. Ninety-four percent reported a need for training in management among residents. Preferred topics were time management and healthcare organization. The preferred training method was a workshop given during residency by an extramural expert. CONCLUSION Dutch medical specialists perceive the management competencies of residents in some areas to be inadequate. They feel that training in medical management during residency is necessary.
Collapse
Affiliation(s)
- L Berkenbosch
- Faculty of Health, Medicine and Life Sciences, Maastricht University, the Netherlands.
| | | | | | | | | | | |
Collapse
|
29
|
Tjiam IM, Persoon MC, Hendrikx AJM, Muijtjens AMM, Witjes JA, Scherpbier AJJA. Program for laparoscopic urologic skills: a newly developed and validated educational program. Urology 2012; 79:815-20. [PMID: 22469576 DOI: 10.1016/j.urology.2012.01.014] [Citation(s) in RCA: 31] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2011] [Revised: 12/09/2011] [Accepted: 01/10/2012] [Indexed: 11/28/2022]
Abstract
OBJECTIVE To develop and evaluate a program for laparoscopic urologic skills (PLUS) to determine the face, content, and construct validation to achieve uniformity and standardization in training residents in urology. METHODS The PLUS consists of 5 basic laparoscopic tasks. Three tasks were abstracted from the Fundamentals of Laparoscopic Surgery program, and 2 additional tasks were developed under continuous evaluation by expert urologists. Fifty participants were recruited from different hospitals and performed the final PLUS training. They all completed a questionnaire after performance. Three outcome parameters were measured: performance quality, time, and dropped objects. The relationship between laparoscopic experience and the outcome parameters was investigated. RESULTS Of the 50 participants, 13 were students, 20 were residents, and 17 were urologists. Double-log linear regression analysis for all 5 tasks showed a significant effect (effect size range 0.53-0.82; P < .0005) for laparoscopic experience on performance time. Substantial correlations were found between experience and quality ratings (log-linear regression effect size 0.37; P = .012) and the number of dropped objects (Spearman correlation effect size 0.49; P < .01). The usefulness of the PLUS model as a training tool for basic laparoscopic skills was rated 4.55 on a scale from 1 (not useful) to 5 (useful) (standard deviation 0.58; range 3-5). CONCLUSION The results of the present study indicated the face, content, and construct validity for the PLUS. The training is considered appropriate for use as a primary training tool for an entry test or as part of a step-wise training program in which basic and procedural laparoscopic skills are integrated.
Collapse
|
30
|
Schmidt HG, Muijtjens AMM, Van der Vleuten CPM, Norman GR. Differential student attrition and differential exposure mask effects of problem-based learning in curriculum comparison studies. Acad Med 2012; 87:463-75. [PMID: 22361797 DOI: 10.1097/acm.0b013e318249591a] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
PURPOSE Studies that compare the effects of problem-based and conventional medical curricula on student performance generally show no significant differences. However, curriculum comparison studies are at best quasi-experimental, so they are prone to forms of selection bias. The purpose of this study was to reanalyze data from such comparisons by controlling for two potential biases: differential student attrition and differential exposure. METHOD The authors reanalyzed 104 previously published comparisons involving a single, problem-based medical school in the Netherlands (Maastricht University's medical school), using student attrition and study duration data from this school and the schools with which it was compared. The authors removed bias by reequalizing the comparison groups in terms of attrition and study duration. RESULTS The uncorrected data showed no differences between problem-based and conventional curricula: Mean effect sizes as expressed by Cohen d were 0.02 for medical knowledge and 0.07 for diagnostic reasoning. However, the reanalysis demonstrated medium-level effect sizes favoring the problem-based curriculum. After corrections for attrition and study duration, the mean effect size for knowledge acquisition was 0.31 and for diagnostic reasoning was 0.51. CONCLUSIONS Effects of the Maastricht problem-based curriculum were masked by differential attrition and differential exposure in the original studies. Because this school has been involved in many studies included in influential literature reviews published in the past 20 years, the authors' findings have implications for the assessment of the value of problem-based learning put forward by these reviews.
Collapse
Affiliation(s)
- Henk G Schmidt
- Department of Psychology, Erasmus University, Rotterdam, The Netherlands.
| | | | | | | |
Collapse
|
31
|
van Mook WNKA, Muijtjens AMM, Gorter SL, Zwaveling JH, Schuwirth LW, van der Vleuten CPM. Web-assisted assessment of professional behaviour in problem-based learning: more feedback, yet no qualitative improvement? Adv Health Sci Educ Theory Pract 2012; 17:81-93. [PMID: 21533755 PMCID: PMC3274685 DOI: 10.1007/s10459-011-9297-0] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/23/2010] [Accepted: 04/14/2011] [Indexed: 05/28/2023]
Abstract
Although other web-based approaches to assessment of professional behaviour have been studied, no publications studying the potential advantages of a web-based instrument versus a classic, paper-based method have been published to date. This study has two research goals: it focuses on the quantity and quality of comments provided by students and their peers (two researchers independently scoring comments as correct and incorrect in relation to five commonly used feedback rules (and resulting in an aggregated score of the five scores) on the one, and on the feasibility, acceptability and perceived usefulness of the two approaches on the other hand (using a survey). The amount of feedback was significantly higher in the web-based group than in the paper based group for all three categories (dealing with work, others and oneself). Regarding the quality of feedback, the aggregated score for each of the three categories was not significantly different between the two groups, neither for the interim, nor for the final assessment. Some, not statistically significant, but nevertheless noteworthy trends were nevertheless noted. Feedback in the web-based group was more often unrelated to observed behaviour for several categories for both the interim and final assessment. Furthermore, most comments relating to the category 'Dealing with oneself' consisted of descriptions of a student's attendance, thereby neglecting other aspects of personal functioning. The survey identified significant differences between the groups for all questionnaire items regarding feasibility, acceptability and perceived usefulness in favour of the paper-based form. The use of a web-based instrument for professional behaviour assessment yielded a significantly higher number of comments compared to the traditional paper-based assessment. Unfortunately, the quality of the feedback obtained by the web-based instrument as measured by several generally accepted feedback criteria did not parallel this increase.
Collapse
Affiliation(s)
- Walther N K A van Mook
- Department of Intensive Care, Maastricht University Medical Centre, P. Debyelaan 25, 6202 AZ, Maastricht, The Netherlands.
| | | | | | | | | | | |
Collapse
|
32
|
Balslev T, Jarodzka H, Holmqvist K, de Grave W, Muijtjens AMM, Eika B, van Merriënboer J, Scherpbier AJJA. Visual expertise in paediatric neurology. Eur J Paediatr Neurol 2012; 16:161-6. [PMID: 21862371 DOI: 10.1016/j.ejpn.2011.07.004] [Citation(s) in RCA: 43] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/23/2011] [Revised: 06/03/2011] [Accepted: 07/17/2011] [Indexed: 11/28/2022]
Abstract
BACKGROUND Visual expertise relies on perceptive as well as cognitive processes. At present, knowledge of these processes when diagnosing clinical cases mainly stems from studies with still pictures. In contrast, patient video cases constitute a dynamic diagnostic challenge that may simulate seeing and diagnosing a patient in person. AIMS This study investigates visual attention and the concomitant cognitive processes of clinicians diagnosing authentic paediatric video cases. METHODS A total of 43 clinicians with varying levels of expertise took part in this cross-sectional study. They diagnosed four brief video recordings of children: two with seizures and two with disorders imitating seizures. We used eye tracking to investigate time looking at relevant areas in the video cases and a concurrent think-aloud procedure to explore the associated clinical reasoning processes. RESULTS More experienced clinicians were more accurate in visual diagnosis and spent more of their time looking at relevant areas. At the same time, they explored data less, yet they built and evaluated more diagnostic hypotheses. CONCLUSIONS Clinicians of varying expertise analyse patient video cases differently. Clinical teachers should take these differences into account when optimising educational formats with patient video cases.
Collapse
Affiliation(s)
- Thomas Balslev
- Aarhus University, Centre of Medical Education, Brendstrupgaardsvej 102, 8200 Aarhus N, Denmark.
| | | | | | | | | | | | | | | |
Collapse
|
33
|
Govaerts MJB, Schuwirth LWT, Van der Vleuten CPM, Muijtjens AMM. Workplace-based assessment: effects of rater expertise. Adv Health Sci Educ Theory Pract 2011; 16:151-65. [PMID: 20882335 PMCID: PMC3068251 DOI: 10.1007/s10459-010-9250-7] [Citation(s) in RCA: 131] [Impact Index Per Article: 10.1] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/01/2010] [Accepted: 09/16/2010] [Indexed: 05/18/2023]
Abstract
Traditional psychometric approaches towards assessment tend to focus exclusively on quantitative properties of assessment outcomes. This may limit more meaningful educational approaches towards workplace-based assessment (WBA). Cognition-based models of WBA argue that assessment outcomes are determined by cognitive processes by raters which are very similar to reasoning, judgment and decision making in professional domains such as medicine. The present study explores cognitive processes that underlie judgment and decision making by raters when observing performance in the clinical workplace. It specifically focuses on how differences in rating experience influence information processing by raters. Verbal protocol analysis was used to investigate how experienced and non-experienced raters select and use observational data to arrive at judgments and decisions about trainees' performance in the clinical workplace. Differences between experienced and non-experienced raters were assessed with respect to time spent on information analysis and representation of trainee performance; performance scores; and information processing--using qualitative-based quantitative analysis of verbal data. Results showed expert-novice differences in time needed for representation of trainee performance, depending on complexity of the rating task. Experts paid more attention to situation-specific cues in the assessment context and they generated (significantly) more interpretations and fewer literal descriptions of observed behaviors. There were no significant differences in rating scores. Overall, our findings seemed to be consistent with other findings on expertise research, supporting theories underlying cognition-based models of assessment in the clinical workplace. Implications for WBA are discussed.
Collapse
Affiliation(s)
- M J B Govaerts
- FHML, Department of Educational Research and Development, Maastricht University, The Netherlands.
| | | | | | | |
Collapse
|
34
|
Boerboom TBB, Dolmans DHJM, Jaarsma ADC, Muijtjens AMM, Van Beukelen P, Scherpbier AJJA. Exploring the validity and reliability of a questionnaire for evaluating veterinary clinical teachers' supervisory skills during clinical rotations. Med Teach 2011; 33:e84-e91. [PMID: 21275538 DOI: 10.3109/0142159x.2011.536277] [Citation(s) in RCA: 10] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/30/2023]
Abstract
BACKGROUND Feedback to aid teachers in improving their teaching requires validated evaluation instruments. When implementing an evaluation instrument in a different context, it is important to collect validity evidence from multiple sources. AIM We examined the validity and reliability of the Maastricht Clinical Teaching Questionnaire (MCTQ) as an instrument to evaluate individual clinical teachers during short clinical rotations in veterinary education. METHODS We examined four sources of validity evidence: (1) Content was examined based on theory of effective learning. (2) Response process was explored in a pilot study. (3) Internal structure was assessed by confirmatory factor analysis using 1086 student evaluations and reliability was examined utilizing generalizability analysis. (4) Relations with other relevant variables were examined by comparing factor scores with other outcomes. RESULTS Content validity was supported by theory underlying the cognitive apprenticeship model on which the instrument is based. The pilot study resulted in an additional question about supervision time. A five-factor model showed a good fit with the data. Acceptable reliability was achievable with 10-12 questionnaires per teacher. Correlations between the factors and overall teacher judgement were strong. CONCLUSIONS The MCTQ appears to be a valid and reliable instrument to evaluate clinical teachers' performance during short rotations.
Collapse
Affiliation(s)
- T B B Boerboom
- Quality Improvement of Veterinary Education, Faculty of Veterinary Medicine, Utrecht University, PO BOX 80163, 3508 TD Utrecht, The Netherlands.
| | | | | | | | | | | |
Collapse
|
35
|
Peeraer G, Donche V, De Winter BY, Muijtjens AMM, Remmen R, Van Petegem P, Bossaert L, Scherpbier AJJA. Teaching conceptions and approaches to teaching of medical school faculty: the difference between how medical school teachers think about teaching and how they say that they do teach. Med Teach 2011; 33:e382-7. [PMID: 21696272 DOI: 10.3109/0142159x.2011.579199] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/20/2023]
Abstract
BACKGROUND It is often assumed that the way teachers approach their teaching is determined by the way they think about learning. This study explores how teachers of an undergraduate medical programme (UMP) think about learning, how they approach teaching and whether their conceptions of learning relate to their teaching approaches. METHODS Quantitative data of academic teachers involved in the undergraduate programme in medicine were collected and analysed. We used a questionnaire designed to measure teachers' conceptions of their own learning (COL) and of student learning as well as teachers' approaches to teaching (AT). RESULTS Teachers of the medical undergraduate programme hold a variety of COL, of how students learn and their AT. No significant correlations were found between teachers' conceptions of learning and their AT. CONCLUSIONS Although UMP teachers' ideas on learning and teaching are very diverse, some of their conceptions are interrelated. Teachers' ideas on their own learning is sometimes - but not always - related to how they think about student learning. But most importantly, the way UMP teachers think about learning is not automatically converted into the way they approach teaching.
Collapse
|
36
|
Stalmeijer RE, Dolmans DHJM, Wolfhagen IHAP, Muijtjens AMM, Scherpbier AJJA. The Maastricht Clinical Teaching Questionnaire (MCTQ) as a valid and reliable instrument for the evaluation of clinical teachers. Acad Med 2010; 85:1732-1738. [PMID: 20881822 DOI: 10.1097/acm.0b013e3181f554d6] [Citation(s) in RCA: 40] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/27/2023]
Abstract
PURPOSE Clinical teaching's importance in the medical curriculum has led to increased interest in its evaluation. Instruments for evaluating clinical teaching must be theory based, reliable, and valid. The Maastricht Clinical Teaching Questionnaire (MCTQ), based on the theoretical constructs of cognitive apprenticeship, elicits evaluations of individual clinical teachers' performance at the workplace. The authors investigated its construct validity and reliability, and they used the underlying factors to test a causal model representing effective clinical teaching. METHOD Between March 2007 and December 2008, the authors asked students who had completed clerkship rotations in different departments of two teaching hospitals to use the MCTQ to evaluate their clinical teachers. To establish construct validity, the authors performed a confirmatory factor analysis of the evaluation data, and they estimated reliability by calculating the generalizability coefficient and standard error measurement. Finally, to test a model of the factors, they fitted a structural linear model to the data. RESULTS Confirmatory factor analysis yielded a five-factor model which fit the data well. Generalizability studies indicated that 7 to 10 student ratings can produce reliable ratings of individual teachers. The hypothesized structural linear model underlined the central roles played by modeling and coaching (mediated by articulation). CONCLUSIONS The MCTQ is a valid and reliable evaluation instrument, thereby demonstrating the usefulness of the cognitive apprenticeship concept for clinical teaching during clerkships. Furthermore, a valuable model of clinical teaching emerged, highlighting modeling, coaching, and stimulating students' articulation and exploration as crucial to effective teaching at the clinical workplace.
Collapse
Affiliation(s)
- Renée E Stalmeijer
- Department of Educational Development and Research, Faculty of Health, Medicine, and Life Sciences, Maastricht University, Maastricht, The Netherlands.
| | | | | | | | | |
Collapse
|
37
|
Abstract
OBJECTIVE The goal was to determine whether the diagnostic accuracy of nonexperts in selected learning environments would improve with the use of patient video cases (PVCs). METHODS We designed a stepwise, team-based, learning approach with a (1) text-based patient presentation, (2) first review of a PVC, (3) small-group discussion, (4) second review of a PVC, and (5) large-group discussion and listening to think-aloud modeling by a content expert. Four pediatric neurology PVCs were analyzed by 44 physicians. After each step, the diagnostic accuracy was assessed with a questionnaire with open-ended questions measuring the frequency of relevant diagnoses and clinical diagnostic reasoning processes. RESULTS The first review of the PVC was followed by a large number of relevant clinical diagnostic reasoning processes. Small-group discussions and listening to a think-aloud procedure with an expert were particularly effective in increasing the diagnostic accuracy of the nonexperts. CONCLUSIONS The diagnostic accuracy of nonexperts was clearly enhanced by interaction in small-group discussions and subsequent listening to a think-aloud procedure with a content expert. Learning through PVCs in clinical settings thus is improved through the interactive participation of junior and senior clinicians. Such an environment (a "virtual examination room") may be introduced at grand rounds, case conferences, or morning rounds, to stimulate the development of diagnostic accuracy in nonexperts.
Collapse
Affiliation(s)
- Thomas Balslev
- MHPE, Viborg Regional Hospital, Department of Pediatrics, Heibergs Allé 4, 8800 Viborg, Denmark.
| | | | | | | |
Collapse
|
38
|
Muijtjens AMM, Timmermans I, Donkers J, Peperkamp R, Medema H, Cohen-Schotanus J, Thoben A, Wenink ACG, van der Vleuten CPM. Flexible electronic feedback using the virtues of progress testing. Med Teach 2010; 32:491-5. [PMID: 20515379 DOI: 10.3109/0142159x.2010.486058] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/11/2023]
Abstract
The potential richness of the feedback for learners and teachers is one of the educational advantages of progress tests (PTs). Every test administration yields information on a student's knowledge level in each sub-domain of the test (cross-sectional information), and it adds a next point to the corresponding knowledge growth curve (longitudinal information). Traditional paper-based feedback has severe limitations and requires considerable effort from the learners to give meaning to the data. We reasoned that the PT data should be flexibly accessible in all pathways and with any available comparison data, according to the personal interest of the learner. For that purpose, a web-based tool (Progress test Feedback, the ProF system) was developed. This article presents the principles and features of the generated feedback and shows how it can be used. In addition to enhancement of the feedback, the ProF database of longitudinal PT-data also provides new opportunities for research on knowledge growth, and these are currently being explored.
Collapse
Affiliation(s)
- Arno M M Muijtjens
- Department of Educational Development and Research, Maastricht University, The Netherlands.
| | | | | | | | | | | | | | | | | |
Collapse
|
39
|
van der Zwet J, Hanssen VGA, Zwietering PJ, Muijtjens AMM, Van der Vleuten CPM, Metsemakers JFM, Scherpbier AJJA. Workplace learning in general practice: supervision, patient mix and independence emerge from the black box once again. Med Teach 2010; 32:e294-9. [PMID: 20653372 DOI: 10.3109/0142159x.2010.489128] [Citation(s) in RCA: 25] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/17/2023]
Abstract
BACKGROUND Medical students increasingly participate in rotations in primary care settings, such as general practices. Although general practices can differ markedly from hospital settings, research on the instructional quality of general practice rotations is relatively scarce. AIM We conducted a study to identify which aspects make a significant contribution to the instructional quality of general practice clerkships, as perceived by students. METHOD After completing their general practice attachment, 155 fifth-year medical students filled out an evaluation questionnaire. Exploratory factor analysis and reliability analysis identified clusters of correlated independent variables. Subsequent regression analysis revealed the relations between the reduced set of independent variables and the dependent variable 'Instructional quality'. RESULTS Both the quality of supervision and the size and diversity of the patient mix substantially affected students' experienced instructional quality. Opportunities and facilities to perform independently were correlated with instructional quality, but did not affect the instructiveness directly. CONCLUSIONS Supervision, patient mix and independence are crucial factors for learning in general practice. This is consistent with findings in hospital settings. The perceived quality of instruction hinges on supervision, which is not only the variable most strongly related to instructional quality, but which also affects both the patient mix and students' independence.
Collapse
Affiliation(s)
- J van der Zwet
- Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, The Netherlands.
| | | | | | | | | | | | | |
Collapse
|
40
|
Michels NRM, Driessen EW, Muijtjens AMM, Van Gaal LF, Bossaert LL, De Winter BY. Portfolio assessment during medical internships: How to obtain a reliable and feasible assessment procedure? Educ Health (Abingdon) 2009; 22:313. [PMID: 20029764] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 05/28/2023]
Abstract
BACKGROUND A portfolio is used to mentor and assess students' clinical performance at the workplace. However, students and raters often perceive the portfolio as a time-consuming instrument. PURPOSE In this study, we investigated whether assessment during medical internship by a portfolio can combine reliability and feasibility. METHODS The domain-oriented reliability of 61 double-rated portfolios was measured, using a generalisability analysis with portfolio tasks and raters as sources of variation in measuring the performance of a student. RESULTS We obtained reliability (Phi coefficient) of 0.87 with this internship portfolio containing 15 double-rated tasks. The generalisability analysis showed that an acceptable level of reliability (Phi = 0.80) was maintained when the amount of portfolio tasks was decreased to 13 or 9 using one and two raters, respectively. CONCLUSIONS Our study shows that a portfolio can be a reliable method for the assessment of workplace learning. The possibility of reducing the amount of tasks or raters while maintaining a sufficient level of reliability suggests an increase in feasibility of portfolio use for both students and raters.
Collapse
|
41
|
Boerboom TBB, Dolmans DHJM, Muijtjens AMM, Jaarsma ADC, Van Beukelen P, Scherpbier AJJA. Does a faculty development programme improve teachers' perceived competence in different teacher roles? Med Teach 2009; 31:1030-1031. [PMID: 19909045 DOI: 10.3109/01421590903183779] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/28/2023]
Abstract
BACKGROUND Changing concepts of education have led many medical schools to adopt student-centred approaches to teaching, requiring different teaching competencies than more traditional approaches. AIMS The aim of this study was to investigate whether participation in a faculty development (FD) programme had a positive effect on veterinary medical teachers' perceptions of their competence in seven different teaching roles. METHOD All teaching staff of a veterinary medical school (251) were invited to complete a questionnaire asking about their perceived competence in seven teaching roles. A stepwise multiple linear regression analysis examined the effect of the completion of a FD programme on perceived teaching competence. RESULTS There was a significant positive effect of the FD programme on teachers' perceived competence in majority of the teaching roles at issue. CONCLUSIONS FD appeared to have a positive effect on the perceived competence of veterinary medical teachers. Further research should investigate whether FD actually results in improved teaching performance.
Collapse
Affiliation(s)
- T B B Boerboom
- Utrecht University, Chair Quality Improvement in Veterinary Education, Faculty of Veterinary Medicine, Utrecht, The Netherlands.
| | | | | | | | | | | |
Collapse
|
42
|
Balslev T, de Grave W, Muijtjens AMM, Eika B, Scherpbier AJJA. The development of shared cognition in paediatric residents analysing a patient video versus a paper patient case. Adv Health Sci Educ Theory Pract 2009; 14:557-565. [PMID: 18846433 DOI: 10.1007/s10459-008-9138-y] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/28/2008] [Accepted: 09/22/2008] [Indexed: 05/26/2023]
Abstract
In a previous study, we established that compared to a written case, a video case enhances observable cognitive processes in the verbal interaction in a postgraduate problem-based learning format. In a new study we examined non-observable cognitive processes using a stimulated recall procedure alongside a reanalysis of the data from the first study. We examined the development of shared cognition as reflected in collaborative concept link formation, an approach to connecting a series of concepts related to a particular topic. Eleven paediatric residents were randomly allocated to two groups. After both analysing the same written case vignette, one group watched a video of the case in the vignette and the other group read a written description of the video. Both groups then reanalysed the vignette. After the group sessions, time-logged transcripts were made of the verbal interaction in both groups and all residents individually took part in a stimulated recall procedure. Causal reasoning concept links were labelled as individual or collaborative depending on whether they originated from individual residents or were directly elicited by verbal utterances from others. The video led to a significantly increased frequency ratio (after intervention: before intervention) of collaborative concept links but did not affect the frequency of individual concept links. This novel process approach to chronological registration of concept link formation offered additional evidence that shared cognition by means of co-elaboration of concept formation is stimulated by the use of patient video recordings in small group learning.
Collapse
Affiliation(s)
- T Balslev
- Department of Paediatrics, Aarhus University Hospital, Skejby, Denmark.
| | | | | | | | | |
Collapse
|
43
|
van Mook WNKA, de Grave WS, Gorter SL, Muijtjens AMM, Zwaveling JH, Schuwirth LW, van der Vleuten CPM. Fellows' in intensive care medicine views on professionalism and how they learn it. Intensive Care Med 2009; 36:296-303. [PMID: 19771410 PMCID: PMC2809310 DOI: 10.1007/s00134-009-1644-8] [Citation(s) in RCA: 22] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2009] [Accepted: 08/04/2009] [Indexed: 11/29/2022]
Abstract
INTRODUCTION The emphasis on the importance of professionalism in a recent CoBaTrICE-IT paper was impressive. However, insight into the elements of professionalism as perceived relevant for intensivists from the fellows' view, and how these are taught and learned, is limited. OBJECTIVES AND METHODS A nationwide study was performed in 2007-2008. All ICM fellows (n = 90) were sent a questionnaire containing the following questions regarding training in professionalism (7-point Likert scale (1 = very inadequate, 7 = very adequate)): which are the elements perceived to be important in intensivists'' daily practice (38 items, cat. I)? Which methods of learning and teaching are recognised (16 items, cat. II)? Which methods of teaching and learning are considered especially useful (16 items, cat. III)? Finally, the perceived quantity and quality of formal and informal learning methods, as well as the responsible organisational body was studied. Data were analysed using SPSS 15.0. RESULTS Response was 75.5 % (n = 68), mean age 34 years. Regarding Elements, scores on virtually all items were high. The factor 'striving for excellence' explained half the variance. Two other aspects, 'Teamwork' and 'Dealing with ethical dilemmas', were identified. Regarding Methods, three dimensions, 'formal curriculum'', 'private and academic experiences' and 'role modelling', proved important. The factor 'formal curriculum' explained most of the variance. Regarding Usefulness the same factors, now mainly explained by the factor Private and academic experiences, emerged with variance. In both categories the items 'observations in daily practice' and 'watching television programmes like ER and House' were the highest- and lowest-scoring items (5.99 and 5.81, and 2.69 and 2.49, respectively). Mean scores regarding the quantity of formal and informal teaching were 4.06 and 4.58 (range 1.841 and 1.519). For the quality of teaching, the figures were 4.22 and 4.52 (range 1.659 and 1.560, respectively). 54 suggestions for improvement of teaching were documented. The need for some form of formal teaching of professionalism aspects as well as for feedback was most frequently mentioned (n = 19 and 16). The local training centres are considered and should remain pivotal for teaching professionalism issues (n = 17 and 28). CONCLUSIONS Almost all elements of professionalism were considered relevant to intensivists' daily practice. Although formal teaching methods regarding professionalism aspects are easily recognised in daily practice, learning by personal experiences and informal ways quantitatively plays a more important, and more valued role. Qualitative comments, nevertheless, stress the need for providing and receiving (solicited and unsolicited) feedback, thereby requesting expansion of formal teaching methods. The local training centres (should continue to) play a major role in teaching professionalism, although an additional role for the (inter)national intensive care organisations remains.
Collapse
Affiliation(s)
- Walther N K A van Mook
- Department of Intensive Care, Maastricht University Medical Centre, P. Debeyelaan 25, 6202 AZ, Maastricht, The Netherlands.
| | | | | | | | | | | | | |
Collapse
|
44
|
Jaarsma DADC, Muijtjens AMM, Dolmans DHJM, Schuurmans EM, Van Beukelen P, Scherpbier AJJA. Undergraduate research internships: veterinary students' experiences and the relation with internship quality. Med Teach 2009; 31:e178-e184. [PMID: 19811123 DOI: 10.1080/01421590902744886] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/28/2023]
Abstract
UNLABELLED The learning environment of undergraduate research internships has received little attention, compared to postgraduate research training. This study investigates students' experiences with research internships, particularly the quality of supervision, development of research skills, the intellectual and social climate, infrastructure support, and the clarity of goals and the relationship between the experiences and the quality of students' research reports and their overall satisfaction with internships. METHOD A questionnaire (23 items, a 5-point Likert scale) was administered to 101 Year five veterinary students after completion of a research internship. Multiple linear regression analyses were conducted with quality of supervision, development of research skills, climate, infrastructure and clarity of goals as independent variables and the quality of students' research reports and students' overall satisfaction as dependent variables. RESULTS The response rate was 79.2%. Students' experiences are generally positive. Students' experiences with the intellectual and social climate are significantly correlated with the quality of research reports whilst the quality of supervision is significantly correlated with both the quality of research reports and students' overall satisfaction with the internship. CONCLUSION Both the quality of supervision and the climate are found to be crucial factors in students' research learning and satisfaction with the internship.
Collapse
|
45
|
Jaarsma ADC, Dolmans DDHJM, Muijtjens AMM, Boerboom TTB, van Beukelen P, Scherpbier AJJA. Students' and teachers' perceived and actual verbal interactions in seminar groups. Med Educ 2009; 43:368-376. [PMID: 19335579 DOI: 10.1111/j.1365-2923.2009.03301.x] [Citation(s) in RCA: 9] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/27/2023]
Abstract
OBJECTIVES This study set out to examine how much time students and teachers devote to different learning-oriented interactions during seminar sessions and students' and teachers' perceptions about the occurrence and desirability of these interactions. METHODS Students and teachers participating in eight seminar group sessions in Year 4 of an undergraduate veterinary curriculum completed an 11-item questionnaire which asked them to rate, on a 5-point Likert scale, the frequency of occurrence and level of desirability of three learning-oriented types of interaction: exploratory questioning; cumulative reasoning, and handling of conflict about knowledge. The questionnaire also invited positive and negative responses to aspects of group interactions and an overall mark (1-10) for the seminars and group interactions. Four group sessions were video-recorded and analysed using a coding scheme. The amount of time devoted to the different interactions was calculated. RESULTS Both students and teachers gave scores of 3.0-3.5 for frequency of occurrence of exploratory questioning and cumulative reasoning and < 3.0 for occurrence of handling of conflict about knowledge. The desired occurrences of all interaction types were significantly higher than the actual occurrences according to students and teachers. Teachers were responsible for the majority of the interactions (93%). The percentages of session time devoted to teacher-centred cumulative reasoning, exploratory questioning and handling of conflict about knowledge were 65.8%, 15.6% and 3.1%, respectively. CONCLUSIONS Group interactions in seminar groups are dominated by the posing of questions by teachers to students. The moderate occurrence of group interactions as perceived by students and teachers may be explained by the inadequate preparation of teachers and students to stimulate group interactions.
Collapse
Affiliation(s)
- A Debbie C Jaarsma
- Department of Education and Student Affairs, Faculty of Veterinary Medicine, Utrecht University, Utrecht, The Netherlands.
| | | | | | | | | | | |
Collapse
|
46
|
Peeraer G, De Winter BY, Muijtjens AMM, Remmen R, Bossaert L, Scherpbier AJJA. Evaluating the effectiveness of curriculum change. Is there a difference between graduating student outcomes from two different curricula? Med Teach 2009; 31:e64-e68. [PMID: 19089721 DOI: 10.1080/01421590802512920] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/27/2023]
Abstract
BACKGROUND Changing a curriculum raises the question whether the results, new curriculum student outcomes, are different from old curriculum student outcomes. AIMS To see whether different curricula produce different outcomes, we compared test and questionnaire results of two cohorts. We wanted to know if there is a difference on knowledge and skills test results, and on the number of times students practiced their skills during their final internships. METHOD We used two validated test instruments: the Dutch Progress Test (PT) and the Objective Structured Clinical Examination (OSCE). For reporting their skills practice, we asked students of both cohorts to fill out a basic skills questionnaire. RESULTS We found no significant difference between both cohorts on the results of their knowledge test and their report on skills practice. On the OSCE, students from the new curriculum cohort scored significantly higher than old curriculum students. CONCLUSION Curriculum change can lead to improvements in graduating students' outcome results.
Collapse
|
47
|
Haarhuis JCM, Muijtjens AMM, Scherpbier AJJA, van Beukelen P. An admissions system to select veterinary medical students with an interest in food animals and veterinary public health. J Vet Med Educ 2009; 36:2-6. [PMID: 19435984 DOI: 10.3138/jvme.36.1.2] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/27/2023]
Abstract
Interest in the areas of food animals (FA) and veterinary public health (VPH) appears to be declining among prospective students of veterinary medicine. To address the expected shortage of veterinarians in these areas, the Utrecht Faculty of Veterinary Medicine has developed an admissions procedure to select undergraduates whose aptitude and interests are suited to these areas. A study using expert meetings, open interviews, and document analysis identified personal characteristics that distinguished veterinarians working in the areas of FA and VPH from their colleagues who specialized in companion animals (CA) and equine medicine (E). The outcomes were used to create a written selection tool. We validated this tool in a study among undergraduate veterinary students in their final (sixth) year before graduation. The applicability of the tool was verified in a study among first-year students who had opted to pursue either FA/VPH or CA/E. The tool revealed statistically significant differences with acceptable effect sizes between the two student groups. Because the written selection tool did not cover all of the differences between the veterinarians who specialized in FA/VPH and those who specialized in CA/E, we developed a prestructured panel interview and added it to the questionnaire. The evaluation of the written component showed that it was suitable for selecting those students who were most likely to succeed in the FA/VPH track.
Collapse
Affiliation(s)
- Jan C M Haarhuis
- Department of Education and Student Affairs of the Faculty of Veterinary Medicine, Utrecht University, The Netherlands.
| | | | | | | |
Collapse
|
48
|
Abstract
CONTEXT Small-group learning is advocated for enhancing higher-order thinking and the development of skills and attitudes. Teacher performance, group interaction and the quality of assignments have been shown to affect small-group learning in hybrid and problem-based curricula. OBJECTIVES This study aimed to examine the perceptions of student groups as to how teacher performance, group interaction and the quality of assignments are related to one another and to learning effects in seminars of 15-30 students in a hybrid curriculum. METHODS We constructed a 28-item questionnaire and administered it to 639 students attending 32 seminars in Years 1-4 of an undergraduate veterinary curriculum. We performed factor analysis and reliability analysis of the questionnaire. We used correlation and regression analyses to explore the interactions of the four-factor model, with teacher performance, group interaction and quality of assignments as independent variables and the perceived learning effect of the seminars as the dependent variable. RESULTS The response rate was 99%. Teacher performance (beta = 0.78) and group interaction (- 0.28) significantly influenced the perceived learning effect. The total effect of the quality of assignments (through effects on teacher performance and group interaction) was 0.47. DISCUSSION The strong relationship between teacher performance and learning effect suggests that students rely strongly on their teachers. The negative effect of group interaction may reflect poor alignment of teaching and assessment and poor organisation of group processes. This should be further examined. Comparative studies on seminars are also recommended.
Collapse
Affiliation(s)
- A Debbie C Jaarsma
- Faculty of Veterinary Medicine, Utrecht University, Utrecht, The Netherlands.
| | | | | | | | | |
Collapse
|
49
|
Muijtjens AMM, Schuwirth LWT, Cohen-Schotanus J, van der Vleuten CPM. Differences in knowledge development exposed by multi-curricular progress test data. Adv Health Sci Educ Theory Pract 2008; 13:593-605. [PMID: 17479352 DOI: 10.1007/s10459-007-9066-2] [Citation(s) in RCA: 9] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/01/2006] [Accepted: 04/13/2007] [Indexed: 05/15/2023]
Abstract
Progress testing provides data on the growth of students' knowledge over the course of the curriculum obtained from the results of all students in the curriculum on periodical similar tests pitched at end-of-curriculum level. Since 2001, three medical schools have jointly constructed and administered four progress tests annually. All students in the 6-year undergraduate curricula of these schools take the same tests resulting in 24 distinct measurements per academic year (four tests for six student year groups), which may be used to compare performance between and within schools. Because single point measurements had proven unreliable, we devised a method to use cumulative information to compare schools' test performance. This cumulative deviation method involves calculation of the deviations of schools' scores from the cross-institutional average score for 24 measurement moments in 1 year. The current study shows that it appears to be feasible to use a combination of the cumulative deviation method and trend analysis for subdomains of medical knowledge to detect strengths and weaknesses in knowledge development in medical curricula. We illustrate the method by applying it to data from 16 consecutive progress tests administered to all students (4,300) of three medical schools in the academic years 2001/2002 through 2004/2005.
Collapse
Affiliation(s)
- Arno M M Muijtjens
- Department of Educational Development and Research, Faculty of Medicine, Maastricht University, P.O. Box 616, Maastricht, 6200 MD, The Netherlands.
| | | | | | | |
Collapse
|
50
|
Peeraer G, Muijtjens AMM, De Winter BY, Remmen R, Hendrickx K, Bossaert L, Scherpbier AJJA. Unintentional failure to assess for experience in senior undergraduate OSCE scoring. Med Educ 2008; 42:669-675. [PMID: 18588647 DOI: 10.1111/j.1365-2923.2008.03043.x] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/26/2023]
Abstract
CONTEXT One goal of undergraduate assessment is to test students' (future) performance. In the area of skills testing, the objective structured clinical examination (OSCE) has been of great value as a tool with which to test a number of skills in a limited time, with bias reduction and improved reliability. But can OSCEs measure undergraduate internship expertise in basic clinical skills? METHODS Undergraduate students (n = 32) were given a questionnaire listing 182 basic clinical skills. We asked them to score the number of times they had performed each skill during their internships (a 12-month period in Year 6). We assessed the students at the end of Year 5 (before the start of their internships) and again at the start of Year 7 (undergraduate training takes 7 years in Belgium, with internships during Year 6), using a 14-station OSCE assessing basic clinical skills. Global ratings were used to score performance. The relationship between internship experience and the OSCE Year 7 score was analysed using a linear regression model, controlling for variation in OSCE scores from Year 5. A multi-level analysis was performed considering students as level-1 units and stations as level-2 units. RESULTS Year 7 OSCE scores (post-internships) were not affected by the number of times that students practised basic medical skills during their internships. DISCUSSION Scores on OSCEs do not seem to reflect clinical expertise acquired during internships. Other more integrated assessment methods may prove to be more valid for testing final undergraduate skills levels.
Collapse
Affiliation(s)
- Griet Peeraer
- Dean's Office, Faculty of Medicine, University of Antwerp, Antwerp, Belgium.
| | | | | | | | | | | | | |
Collapse
|