Development and validation of an evaluation instrument for medical students in tutorials

Measuring group function in problem-based learning: development of a reflection tool

BACKGROUND

Problem-based learning (PBL) is a pedagogy involving self-directed learning in small groups around case problems. Group function is important to PBL outcomes, but there is currently poor scaffolding around key self-reflective practices that necessarily precedes students' and tutors' attempts to improve group function. This study aims to create a structured, literature-based and stakeholder-informed tool to help anchor reflective practices on group function. This article reports on the development process and perceived utility of this tool.

METHODS

Tool development unfolded in four steps: 1) a literature review was conducted to identify existent evaluation tools for group function in PBL, 2) literature findings informed the development of this new tool, 3) a group of PBL experts were consulted for engagement with and feedback of the tool, 4) four focus groups of stakeholders (medical students and tutors with lived PBL experiences) commented on the tool's constructs, language, and perceived utility. The tool underwent two rounds of revisions, informed by the feedback from experts and stakeholders.

RESULTS

Nineteen scales relating to group function assessment were identified in the literature, lending 18 constructs that mapped into four dimensions: Learning Climate, Facilitation and Process, Engagement and Interactivity, and Evaluation and Group Improvement. Feedback from experts informed the addition of missing items. Focus group discussions allowed further fine-tuning of the organization and language of the tool. The final tool contains 17 descriptive items under the four dimensions. Users are asked to rate each dimension holistically on a 7-point Likert scale and provide open comments. Researchers, faculty, and students highlighted three functions the tool could perform: (1) create space, structure, and language for feedback processes, (2) act as a reference, resource, or memory aid, and (3) serve as a written record for longitudinal benchmarking. They commented that the tool may be particularly helpful for inexperienced and poor-functioning groups, and indicated some practical implementation considerations.

CONCLUSION

A four-dimension tool to assist group function reflection in PBL was produced. Its constructs were well supported by literature and experts. Faculty and student stakeholders acknowledged the utility of this tool in addressing an acknowledged gap in group function reflection in PBL.

The collaborative learning development exercise (CLeD-EX): an educational instrument to promote key collaborative learning behaviours in medical students

BACKGROUND

Modern clinical practice increasingly relies on collaborative and team-based approaches to care. Regulatory bodies in medical education emphasise the need to develop collaboration and teamwork competencies and highlight the need to do so from an early stage of medical training. In undergraduate medical education, the focus is usually on collaborative learning, associated with feedback and reflection on this learning This article describes a novel educational instrument, the Collaborative Learning Development Exercise (CLeD-EX), which aims to foster the development of key collaborative learning competencies in medical students. In this article we report on the effectiveness, feasibility and educational impact of the CLeD-EX.

METHODS

In this study, the "educational design research" framework was used to develop, implement and evaluate the CLeD-EX. This involved adopting a systematic approach towards designing a creative and innovative instrument which would help solve a real-world challenge in developing collaborative learning skills. The systematic approach involved a qualitative exploration of key collaborative learning behaviours which are influential in effective collaborative learning contexts. The identified competencies were employed in the design of the CLeD-EX. The design of the CLeD-EX included features to facilitate structured feedback by tutors to students, complemented by self-evaluation and reflection. The CLeD-EX was field-tested with volunteer junior medical students, using a controlled pre-test post-test design. Analysis of the completed CLeD-EX forms, self-perception surveys (i.e. pre-test and post-test surveys) and analyses of reflective reports were used to explore the educational impact of CLeD-EX, as well as its utility and practicality.

RESULTS

After using the CLeD-EX, students showed a significant improvement in critical thinking and group process as measured by a previously validated instrument. Both students and tutors recognised CLeD-EX as an effective instrument, especially as a structured basis for giving and receiving feedback and for completing the feedback loop. CLeD-EX was also found to be feasible, practical and focused, while promoting learning and effective interactions in small group learning.

CONCLUSION

The findings of this study support the introduction of an effective and feasible educational instrument such as the CLeD-EX, to facilitate the development of students' skills in collaborative learning.

Tutor assessment of PBL process: does tutor variability affect objectivity and reliability?

BACKGROUND

Ensuring objectivity and maintaining reliability are necessary in order to consider any form of assessment valid. Evaluation of students in Problem-Based Learning (PBL) tutorials by the tutors has drawn the attention of critiques citing many challenges and limitations. The aim of this study was to determine the extent of tutor variability in assessing the PBL process in the Faculty of Medical Sciences, The University of the West Indies, St Augustine Campus, Trinidad and Tobago.

METHOD

All 181 students of year 3 MBBS were assigned randomly to 14 PBL groups. Out of 18 tutors, 12 had an opportunity to assess three groups: one assessed 2 groups and 4 tutors assessed one group each; at the end each group had been assessed three times by different tutors. The tutors used a PBL assessment rating scale of 12 different criteria on a six-point scale to assess each PBL Group. To test the stated hypotheses, independent t-test, one-way ANOVA followed by post-hoc Bonferroni test, Intra Class Correlation, and Pearson product moment correlations were performed.

RESULT

The analysis revealed significant differences between the highest- and lowest-rated groups (t-ratio = 12.64; p < 0.05) and between the most lenient and most stringent raters (t-ratio = 27.96; p < 0.05). ANOVA and post-hoc analysis for highest and lowest rated groups revealed that lenient- and stringent-raters significantly contribute (p < 0.01) in diluting the score in their respective category. The intra class correlations (ICC) among rating of different tutors for different groups showed low agreement among various ratings except three groups (Groups 6, 8 and 13) (r = 0.40). The correlation between tutors' PBL experiences and their mean ratings was found to be moderately significant (r = 0.52; p > 0.05).

CONCLUSION

Leniency and stringency factors amongst raters affect objectivity and reliability to a great extent as is evident from the present study. Thus, more rigorous training in the areas of principles of assessment for the tutors are recommended. Moreover, putting that knowledge into practice to overcome the leniency and stringency factors is essential.

Validation of a performance assessment instrument in problem-based learning tutorials using two cohorts of medical students

Although problem-based learning (PBL) has been widely used in medical schools, few studies have attended to the assessment of PBL processes using validated instruments. This study examined reliability and validity for an instrument assessing PBL performance in four domains: Problem Solving, Use of Information, Group Process, and Professionalism. Two cohorts of medical students (N = 310) participated in the study, with 2 years of PBL evaluation data extracted from archive rated by a total of 158 faculty raters. Analyses based on generalizability theory were conducted for reliability examination. Validity was examined through following the Standards for Educational and Psychological Testing to evaluate content validity, response processes, construct validity, predictive validity, and the relationship to the variable of training. For construct validity, correlations of PBL scores with six other outcome measures were examined, including Medical College Admission Test, United States Medical Licensing Examination (USMLE) Step 1, National Board of Medical Examiners (NBME) Comprehensive Basic Science Examination, NBME Comprehensive Clinical Science Examination, Clinical Performance Examination, and USMLE Step 2 Clinical Knowledge. Predictive validity was examined by using PBL scores to predict five medical school outcomes. The highest percentage of PBL total score variance was associated with students (60 %), indicating students in the study differed in their PBL performance. The generalizability and dependability coefficients were moderately high (Ep(2) = .68, ϕ = .60), showing the instrument is reliable for ranking students and identifying competent PBL performers. The patterns of correlations between PBL domain scores and the outcome measures partially support construct validity. PBL performance ratings as a whole significantly (p < .01) predicted all the major medical school achievements. The second year PBL scores were significantly higher than those of the first year, indicating a training effect. Psychometric findings provided support for reliability and many aspects of validity of PBL performance assessment using the instrument.

Multiple tutorial-based assessments: a generalizability study

BACKGROUND

Tutorial-based assessment commonly used in problem-based learning (PBL) is thought to provide information about students which is different from that gathered with traditional assessment strategies such as multiple-choice questions or short-answer questions. Although multiple-observations within units in an undergraduate medical education curriculum foster more reliable scores, that evaluation design is not always practically feasible. Thus, this study investigated the overall reliability of a tutorial-based program of assessment, namely the Tutotest-Lite.

METHODS

More specifically, scores from multiple units were used to profile clinical domains for the first two years of a system-based PBL curriculum.

RESULTS

G-Study analysis revealed an acceptable level of generalizability, with g-coefficients of 0.84 and 0.83 for Years 1 and 2, respectively. Interestingly, D-Studies suggested that as few as five observations over one year would yield sufficiently reliable scores.

CONCLUSIONS

Overall, the results from this study support the use of the Tutotest-Lite to judge clinical domains over different PBL units.

Construct Validity and Generalizability of Pediatrics Clerkship Evaluation at a Problem-Based Medical School, Bahrain

The aim of this study is to identify the sources of construct validity evidence and determine generalizability of the scores given to medical students on pediatrics clerkship evaluation at the College of Medicine and Medical Sciences, Arabian Gulf University (CMMS-AGU). The CMMS-AGU is a problem-based medical school in Bahrain. The sample was composed of 140 students. Validity evidence and generalizability theory were sought. Results indicated that pediatrics clerkship evaluation at CMMS-AGU has content validity. Some limitations may have occurred in response process, but the students performance was observed directly by the tutors. There was evidence of range restriction and ``halo'' effect. Interrater reliability was 0.74, and the generalizability coefficient for three raters was 0.61. Students' contribution to the variance component of the global clerkship evaluation scores was 24%. Decision study indicated that seven tutors are required to achieve a generalizability coefficient of about 0.80. In conclusion, this study indicated that the pediatrics clerkship evaluation at the CMMS-AGU has, overall, some acceptable content and internal structure evidence of validity and that the raters are the major source of error. To improve the construct validity, there is a need to increase the number of raters and to train raters about the proper process of clerkship evaluation.

Simulation-based medical education is no better than problem-based discussions and induces misjudgment in self-assessment

Simulation-based teaching (SBT) is increasingly used in medical education. As an alternative to other teaching methods there is a lack of evidence concerning its efficacy. The aim of this study was to evaluate the potency of SBT in anesthesia in comparison to problem-based discussion (PBD) with students in a randomized controlled setting. Thirty-three fourth-year medical students attending a curricular anesthesiology course were randomly allocated to either a session of SBT or a session of PBD on an emergency induction method. Ten days later all students underwent examination in a simulator. The performance of each student was evaluated by weighted tasks, established according to a modified Delphi process. Confidence and a multiple-choice questionnaire were additionally performed pre- and post-intervention. A total of 32 students completed the study. Participants in the SBT group presented with significantly higher self-assessment scores after the intervention than students in the PBD group. However, students in the SBT group achieved only slightly and statistically insignificantly higher scores in the theoretical and simulator examination (p > 0.05) with only a moderate effect size of d = 0.52. The current study demonstrates that both PBD and SBT lead to comparable short-term outcomes in theoretical knowledge and clinical skills. However, undesirably, SBT students overrated their anticipated clinical abilities and knowledge improvement.

Self- and peer assessment may not be an accurate measure of PBL tutorial process

BACKGROUND

Universidade Cidade de São Paulo adopted a problem-based learning (PBL) strategy as the predominant method for teaching and learning medicine. Self-, peer- and tutor marks of the educational process are taken into account as part of the final grade, which also includes assessment of content. This study compared the different perspectives (and grades) of evaluators during tutorials with first year medical students, from 2004 to 2007 (n = 349), from seven semesters.

METHODS

The tutorial evaluation method was comprised of the students' self assessment (SA) (10%), tutor assessment (TA) (80%) and peer assessment (PA) (10%) to calculate a final educational process grade for each tutorial. We compared these three grades from each tutorial for seven semesters using ANOVA and a post hoc test.

RESULTS

A total of 349 students participated with 199 (57%) women and 150 (42%) men. The SA and PA scores were consistently greater than the TA scores. Moreover, the SA and PA groups did not show statistical difference in any semester evaluated, while both differed from tutor assessment in all semesters (Kruskal-Wallis, Dunn's test). The Spearman rank order showed significant (p < 0.0001) and positive correlation for the SA and PA groups (r = 0.806); this was not observed when we compared TA with PA (r = 0.456) or TA with SA (r = 0.376).

CONCLUSION

Peer- and self-assessment marks might be reliable but not valid for PBL tutorial process, especially if these assessments are used for summative assessment, composing the final grade. This article suggests reconsideration of the use of summative assessment for self-evaluation in PBL tutorials.

Validation of the Tutotest in a hybrid problem-based learning curriculum

The Tutotest is one of the few structured instruments developed for the assessment of students' learning skills in a problem-based learning (PBL) curriculum. This study was designed to validate the Tutotest in a hybrid PBL curriculum. Forty-four tutors completed 370 evaluations on second to fourth year medical students at the end of the first semester in 2004 using the Chinese version of the Tutotest. There was significant correlation between global rating and Tutotest-C (r = 0.44, P < 0.001). The Cronbach's alpha coefficient was 0.97. Two-week test-retest correlation coefficient was 0.85. Factor analysis revealed four factors, where three were similar to the factors of "effectiveness in group", "communication and leadership skills", and "respect for others" identified in the original Tutotest. "Hypothesis forming and testing" instead of "scientific curiosity" became the fourth factor in our data. Our study validated the Tutotest-C in a hybrid PBL curriculum and students from the Chinese educational system. The test-retest reliability measure with a 2-week interval at the end of the PBL tutorial confirmed the stability of the Tutotest, which has not been previously reported. Since most Asian medical schools adopted a hybrid PBL curriculum, a valid student evaluation instrument for this type of curriculum is valuable.

Using a sampling strategy to address psychometric challenges in tutorial-based assessments

INTRODUCTION

Tutorial-based assessment, despite providing a good match with the philosophy adopted by educational programmes that emphasize small group learning, remains one of the greatest challenges for educators working in this context. The current study was performed in an attempt to assess the psychometric characteristics of tutorial-based evaluation upon adopting a multiple sampling approach that requires minimal recording of observations.

METHOD

After reviewing the literature, a simple 3-item evaluation form was created. The items were "Professional Behaviour," "Contribution to Group Process," and "Contribution to Group Content." Explicit definition of these items was provided on an evaluation form. Twenty five tutors in five different programmes were asked to use the form to evaluate their students (N=169) after every tutorial over the course of an academic unit. Each item was rated using a 10-point scale.

RESULTS

Cronbach's alpha revealed an appropriate internal consistency in all five programmes. Test-retest reliability of any single rating was low, but the reliability of the average rating was at least 0.75 in all cases. The construct validity of the tool was supported by the observation of increasing ratings over the course of the academic unit and by the finding that more senior students received higher ratings than more junior students.

CONCLUSION

Consistent with the context specificity phenomenon, the adoption of a "minimal observations often" approach to tutorial-based assessment appears to maintain better psychometric characteristics than do attempts to assess tutorial performance using more comprehensive measurement tools.

Development of an instrument to assess individual student performance in small group tutorials

Recognizing the need for a valid and reliable method to assess individual tutorial performance in a problem-based learning curriculum, we developed a 31-item instrument from theoretical frameworks and items used elsewhere. A scale was developed for each of three broad learning domains: self-directed learning (SDL), critical thinking (CT), and group process (GP). The instrument demonstrated high internal consistency (SDL = .88, CT = .90, GP = .83) on a sample of 18 tutors and 167 students. Tutor-student interrater reliability coefficients were estimated to be low (SDL = .16, CT = .18, GP = .14) due to lack of variance on the response scale. The instrument showed high correlation (r = .82) with other forms of summative evaluation. In its current form, this standardized and validated instrument is unreliable in differentiating strong from weak tutorial performance but can have a steering effect on student tutorial behaviors. The process of instrument development has general application to other educational programs.

Measuring inter-rater reliability of the sequenced performance inventory and reflective assessment of learning (SPIRAL)

PURPOSE

To measure the inter-rater reliability of the Sequenced Performance Inventory and Reflective Assessment of Learning (SPIRAL), a twenty-three item scoring rubric designed to assess first and second-year students' competencies such as "acquisition of knowledge," "peer teaching and communication skills," and "professional behavior" in a problem-based learning (PBL) curriculum at the University of North Dakota School of Medicine and Health Sciences.

METHOD

In 2001, the authors constructed a 69-item multiple-choice questionnaire consisting of descriptions of "prototypical" (representing real students) PBL student performances. For each of the 23 SPIRAL items, the authors chose competency descriptions of an "emerging," a "developing," and an "advanced" student (69 prototypical students). These descriptions were obtained from narrative comments by PBL facilitators of real students in their classes in 2000-2001. Seventeen experienced facilitators (PBL tutors) were subsequently asked to rank the 69 prototypical students based on SPIRAL anchor points in the three competency levels. Kendall's coefficient of concordance was used to test inter-rater reliability. Modes were also determined to illustrate the extent to which the ratings of the 17 facilitators aligned with the ratings anticipated by the authors.

RESULTS

Overall, inter-rater scoring for all items combined was considered to be reliable (Kendall's coefficient of concordance W =.75). Inter-rater scoring was also determined for each of the 23 SPIRAL items, with Ws ranging from.97 to.53. Facilitators rated students according to 55 of the 69 SPIRAL descriptors as the researchers anticipated.

CONCLUSIONS

The scoring rubric is reliable. Further criterion-related validation study is warranted.

Self and peer assessment in tutorials: application of a relative-ranking model

PURPOSE

While self assessment continues to be touted as being of paramount importance for continuing professional competence, problem-based learning curricula, and adult learning theory, techniques for ensuring valid judgments have proven elusive. This study tested the applicability of an innovative relative-ranking procedure to problem-based learning tutorials.

METHOD

A total of 36 students in the McMaster University Faculty of Health Sciences' MD program were provided relative-ranking forms listing seven domains of competence along with their definitions. The student, two of the student's peers, and the student's tutor were asked to complete the ranking exercise after their second, fourth, and sixth tutorials.

RESULTS

Combining each level of the time and rater variables generated 66 correlation coefficients, none of which was significantly different from zero. Re-performing the analysis on only the extreme domains did not improve this result.

CONCLUSION

The relative-ranking instrument developed did not prove to be a reliable measure of tutorial performance. Ratings were inconsistent from one week to the next as well as across raters within a week.

Assessment of student performance in problem-based learning tutorial sessions

OBJECTIVES

To assess student performance during tutorial sessions in problem-based learning (PBL).

DESIGN

A 24-item rating scale was developed to assess student performance during tutorial sessions in problem-based learning (PBL) as conducted during the pre-clinical years of Medical School at the National Autonomous University of Mexico. Items were divided into three categories: Independent study, Group interaction and Reasoning skills. Fourteen tutors assessed 152 first and second-year students in 16 tutorial groups. An exploratory factor analysis with an Oblimin rotation was carried out to identify the underlying dimensions of the questionnaire.

SETTING

Medical School at the National Autonomous University of Mexico.

SUBJECTS

Medical students.

RESULTS

Factor analysis yielded four factors (Independent study, Group interaction, Reasoning skills, and Active participation) which together accounted for 76.6% of the variance. Their Cronbach reliability coefficients were 0.95, 0.83, 0.94 and 0. 93, respectively, and 0.96 for the scale as a whole.

CONCLUSIONS

It was concluded that the questionnaire provides a reliable identification of the fundamental components of the PBL method as observable in tutorial groups and could be a useful assessment instrument for tutors wishing to monitor students' progress in each of these components.