Assessment of student performance in problem-based learning tutorial sessions

Measuring group function in problem-based learning: development of a reflection tool

BACKGROUND

Problem-based learning (PBL) is a pedagogy involving self-directed learning in small groups around case problems. Group function is important to PBL outcomes, but there is currently poor scaffolding around key self-reflective practices that necessarily precedes students' and tutors' attempts to improve group function. This study aims to create a structured, literature-based and stakeholder-informed tool to help anchor reflective practices on group function. This article reports on the development process and perceived utility of this tool.

METHODS

Tool development unfolded in four steps: 1) a literature review was conducted to identify existent evaluation tools for group function in PBL, 2) literature findings informed the development of this new tool, 3) a group of PBL experts were consulted for engagement with and feedback of the tool, 4) four focus groups of stakeholders (medical students and tutors with lived PBL experiences) commented on the tool's constructs, language, and perceived utility. The tool underwent two rounds of revisions, informed by the feedback from experts and stakeholders.

RESULTS

Nineteen scales relating to group function assessment were identified in the literature, lending 18 constructs that mapped into four dimensions: Learning Climate, Facilitation and Process, Engagement and Interactivity, and Evaluation and Group Improvement. Feedback from experts informed the addition of missing items. Focus group discussions allowed further fine-tuning of the organization and language of the tool. The final tool contains 17 descriptive items under the four dimensions. Users are asked to rate each dimension holistically on a 7-point Likert scale and provide open comments. Researchers, faculty, and students highlighted three functions the tool could perform: (1) create space, structure, and language for feedback processes, (2) act as a reference, resource, or memory aid, and (3) serve as a written record for longitudinal benchmarking. They commented that the tool may be particularly helpful for inexperienced and poor-functioning groups, and indicated some practical implementation considerations.

CONCLUSION

A four-dimension tool to assist group function reflection in PBL was produced. Its constructs were well supported by literature and experts. Faculty and student stakeholders acknowledged the utility of this tool in addressing an acknowledged gap in group function reflection in PBL.

Medical Students' Perspective on Assessment Mechanism During Problem-Based Learning at Debre Tabor University: An Explanatory Mixed Study

Background

Assessment in problem-based learning should aim to improve students' active learning. In due course, significant student involvement in any assessment process may aid them in meeting the curriculum's objectives.

Purpose

The primary goal of this study is to assess medical students' attitudes towards the assessment method used during PBL tutorials at Debre Tabor College of Health Sciences.

Methods

A mixed explanatory study design was used conducted at Debre Tabor university. For quantitative and qualitative data, cross-sectional survey and phenomenological study designs were used, respectively. A self-administered questionnaire with a 5-point Likert scale was used to collect quantitative data, while Focused Group Discussions (FGDs) were used to collect qualitative data.

Results

The current study included 192 out of 195 medical students. 40%, 57.2%, and 43.2% of study participants felt the tutor did not provide constructive feedback, facilitate self-assessment/self-reflection, or encourage peer assessment, respectively. On the role of tutors in facilitating self and peer assessment, a statistically significant mean difference in agreement is observed. Their PBL assessment did not take into account punctuality or contribution to the discussion. It is perceived as biassed due to tutors' bias towards various factors such as first impressions and student academic rank. They also stated that they did not receive sufficient information about the assessment in PBL.

Conclusion

According to the findings of this study, medical students believed they were not fairly assessed during their PBL tutorial. Due to the uncertainty of the evaluation process, a neutral perspective on comprehension skills was appreciated. The students also perceived that the tutors' ability to assess students, poor feedback experience, and limited information about the assessment mechanism influenced their PBL assessment.

Enhancing students' online collaborative PBL learning performance in the context of coauthoring-based technologies: A case of wiki technologies

Understandability and completeness are essential in modern collaborative digital platforms and their learning systems. These platforms have shaken up the traditional education setting, particularly in leveraging the coauthoring approach in problem-solving and streamlining the learning behavior of cowriting or corevising. Such a learning context has attracted considerable interest from various stakeholders; however, it needs to be explored further as an independent topic. Based on social capital and social identity theories, we explore how online collaborative problem-based learning (PBL) effectiveness, relational quality, and social identity influence students' perceived PBL performance during learning activities. Based on the core elements of online coauthoring processes (i.e., platform, cocreation, and problem-solving), this study employs a holistic view of the coauthor to discuss the effects of understandability and completeness. This study also highlights the mediating impact of trust on students' social identity. Based on the responses of 240 students, the results support the proposed hypotheses using partial least squares analysis. The study's implications suggest guidelines to educators on how to enhance students' perceived PBL performance by using wiki technologies.

The collaborative learning development exercise (CLeD-EX): an educational instrument to promote key collaborative learning behaviours in medical students

BACKGROUND

Modern clinical practice increasingly relies on collaborative and team-based approaches to care. Regulatory bodies in medical education emphasise the need to develop collaboration and teamwork competencies and highlight the need to do so from an early stage of medical training. In undergraduate medical education, the focus is usually on collaborative learning, associated with feedback and reflection on this learning This article describes a novel educational instrument, the Collaborative Learning Development Exercise (CLeD-EX), which aims to foster the development of key collaborative learning competencies in medical students. In this article we report on the effectiveness, feasibility and educational impact of the CLeD-EX.

METHODS

In this study, the "educational design research" framework was used to develop, implement and evaluate the CLeD-EX. This involved adopting a systematic approach towards designing a creative and innovative instrument which would help solve a real-world challenge in developing collaborative learning skills. The systematic approach involved a qualitative exploration of key collaborative learning behaviours which are influential in effective collaborative learning contexts. The identified competencies were employed in the design of the CLeD-EX. The design of the CLeD-EX included features to facilitate structured feedback by tutors to students, complemented by self-evaluation and reflection. The CLeD-EX was field-tested with volunteer junior medical students, using a controlled pre-test post-test design. Analysis of the completed CLeD-EX forms, self-perception surveys (i.e. pre-test and post-test surveys) and analyses of reflective reports were used to explore the educational impact of CLeD-EX, as well as its utility and practicality.

RESULTS

After using the CLeD-EX, students showed a significant improvement in critical thinking and group process as measured by a previously validated instrument. Both students and tutors recognised CLeD-EX as an effective instrument, especially as a structured basis for giving and receiving feedback and for completing the feedback loop. CLeD-EX was also found to be feasible, practical and focused, while promoting learning and effective interactions in small group learning.

CONCLUSION

The findings of this study support the introduction of an effective and feasible educational instrument such as the CLeD-EX, to facilitate the development of students' skills in collaborative learning.

The longitudinal learning outcomes of using different teaching sequences in a nursing administration project

AIM

To examine the effects of using different teaching sequences on knowledge acquisition, knowledge retention, self-directed learning and teamwork in a nursing administration project.

BACKGROUND

Compared to other professional courses, nursing administration is relatively difficult, and it is important for nursing administration students to spend long hours working in hospitals. As such, better teaching strategies utilizing proper sequencing may yield better learning outcomes for students.

METHODS

A longitudinal quasi-experimental design was used. A total of 129 students were assigned to two groups with which different teaching sequences were used. The investigated learning outcomes were knowledge acquisition, knowledge retention, self-directed learning and teamwork. Generalized estimating equations were used to measure the learning outcomes.

RESULTS

There was no significant difference between the two teaching sequences in terms of learning effects. One teaching strategy, problem-based learning (PBL), yielded effective student learning outcomes.

CONCLUSION

PBL increased the students' self-directed learning and teamwork. This strategy can be applied to side-by-side co-teaching and post-graduate year training programmes.

IMPLICATIONS FOR NURSING MANAGEMENT

The results from this study may help hospitals retain nurses and find potential nursing leaders. Instructors and nurse managers should discuss learning goals with students in advance to enhance the students' learning outcomes.

Influence of critical thinking disposition on the learning efficiency of problem-based learning in undergraduate medical students

BACKGROUND

Problem-based learning (PBL), a pedagogical approach, is widely accepted in medical education. Manipulated by many factors, the internal motivation of learner is the most crucial determinant that affects the nature of the outcome, in which the influences of critical thinking (CT) remained elusive.

METHODS

One hundred two third-year undergraduate medical students at Peking University were involved in this study. A Chinese version of the Critical Thinking Disposition Inventory (CTDI-CV) was used to assess the CT disposition, and the performance scores of students in PBL tutorials were compiled. A parametric bivariate correlation analysis was performed between the students' CT scores and their PBL average scores. The PBL scores were compared between the strong and weak CT disposition groups using independent t-test. The analysis of numerical data was conducted using SPSS 16.0.

RESULTS

CT disposition of third-year undergraduate medical students at Peking University was at a positive level, with an average score of 297.72. The total CT scores had a positive correlation with the scores of the PBL performance and its five dimensions significantly. In the majority, students with Strong-CT disposition obtained higher scores in PBL tutorials compared with students with Weak-CT disposition. The performance of these two groups was significantly different in the Late-Half but not in the Early-Half PBL tutorials. Furthermore, a significant improvement was observed in the students with strong CT but not weak CT dispositions.

CONCLUSION

CT disposition positively correlates to a students' PBL performance. Students with stronger CT dispositions perform better in the PBL process and obtain higher scores. Our work suggested that the open-mindedness of the CT disposition is the primary factor that determines the improvement of the preparation dimensions in the PBL process.

The relationship between the monitored performance of tutors and students at PBL tutorials and the marked hypotheses generated by students in a hybrid curriculum

INTRODUCTION

There have been a number of published studies examining the link between the effectiveness of the problem-based learning (PBL) process and students' performance in examinations. In a hybrid PBL/lectures curriculum, the results of such studies are of limited use because of the difficulty in dissociating the knowledge gained at lectures from that gained through PBL-related activities. Hence, the objectives of this study were: (1) to develop an instrument to measure the performance of tutors and students at PBL tutorials, and (2) to explore the contribution of such performances to the marks attained by students from the hypotheses generated at PBL tutorials.

METHODS

A monitoring instrument for assessing the performances of non-expert tutors and students at tutorials was developed and validated using principal component analysis and reliability analysis. Also, a rubric was formulated to enable a content expert to assign marks to the quality of hypotheses generated.

RESULTS

The monitoring instrument was found to be valid and reliable. There was a significant correlation between the performance of tutors at tutorials and hypotheses marks. In contrast, there was no significant correlation between the performance of students and hypotheses marks.

DISCUSSION

The monitoring instrument is a useful tool for improving the PBL process, especially where the medical programme depends on non-expert PBL tutors. In addition to ensuring good PBL processes, it is important that students achieve the desired output at PBL tutorials by producing hypotheses that help them understand the basic sciences underlying the clinical cases. The latter is achieved by the use of an open-ended rubric by a subject expert to assign marks to the hypotheses, a method that also provides additional motivation to students to develop relevant and detailed hypotheses.

Validation of a performance assessment instrument in problem-based learning tutorials using two cohorts of medical students

Although problem-based learning (PBL) has been widely used in medical schools, few studies have attended to the assessment of PBL processes using validated instruments. This study examined reliability and validity for an instrument assessing PBL performance in four domains: Problem Solving, Use of Information, Group Process, and Professionalism. Two cohorts of medical students (N = 310) participated in the study, with 2 years of PBL evaluation data extracted from archive rated by a total of 158 faculty raters. Analyses based on generalizability theory were conducted for reliability examination. Validity was examined through following the Standards for Educational and Psychological Testing to evaluate content validity, response processes, construct validity, predictive validity, and the relationship to the variable of training. For construct validity, correlations of PBL scores with six other outcome measures were examined, including Medical College Admission Test, United States Medical Licensing Examination (USMLE) Step 1, National Board of Medical Examiners (NBME) Comprehensive Basic Science Examination, NBME Comprehensive Clinical Science Examination, Clinical Performance Examination, and USMLE Step 2 Clinical Knowledge. Predictive validity was examined by using PBL scores to predict five medical school outcomes. The highest percentage of PBL total score variance was associated with students (60 %), indicating students in the study differed in their PBL performance. The generalizability and dependability coefficients were moderately high (Ep(2) = .68, ϕ = .60), showing the instrument is reliable for ranking students and identifying competent PBL performers. The patterns of correlations between PBL domain scores and the outcome measures partially support construct validity. PBL performance ratings as a whole significantly (p < .01) predicted all the major medical school achievements. The second year PBL scores were significantly higher than those of the first year, indicating a training effect. Psychometric findings provided support for reliability and many aspects of validity of PBL performance assessment using the instrument.

The impact of an interprofessional problem-based learning curriculum of clinical ethics on medical and nursing students' attitudes and ability of interprofessional collaboration: a pilot study

Clinical ethic situations in modern multiprofessional healthcare systems may involve different healthcare professions who work together for patient care. The undergraduate interprofessional education of clinical ethics would help to incubate healthcare students' ability of interprofessional collaboration in solving ethical problems. However, the impact from an interprofessional educational model on student's attitudes and confidence of interprofessional collaboration should be carefully evaluated during the process of curricular development. This study aimed to conduct a pilot interprofessional PBL curriculum of clinical ethics and evaluate the curricular impact on interprofessional students' attitude and confidence of collaborative teamwork. Thirty-six medical and nursing students volunteered to participate in this study and were divided into three groups (medical group, nursing group, and mixed group). Tutors were recruited from the Medical School and the College of Nursing. The pilot curriculum included one lecture of clinical ethics, one PBL case study with two tutorial sessions, and one session of group discussion and feedback. A narrative story with multiple story lines and a multiperspective problem analysis tool were used in the PBL tutorials. The students' self-evaluation of learning questionnaire was used to evaluate students' learning of clinical ethics and interprofessional collaborative skills and attitude. The internal consistency of the questionnaire was measured by Cronbach α, and the criterion-related validity of the questionnaire was evaluated through associations between the dimension scores with the student group by one-way analysis of variance test (ANOVA) test and Tukey-Kramer honestly significant difference (HSD) comparison. There was significant difference among different groups in students' ability and attitudes about "interprofessional communication and collaboration" (p = 0.0184). The scores in the mixed group (37.58 ± 3.26) were higher than the medical group (32.10 ± 4.98). In conclusion, our model for the interprofessional PBL curriculum of clinical ethics is practicable and will produce positive impacts on students' attitudes and confidence of interprofessional collaboration.

The effectiveness of problem-based learning and concept mapping among Taiwanese registered nursing students

The purpose of this study was to investigate the effectiveness of adopting problem-based learning (PBL) and concept mapping (CM) in the educational programs for Taiwanese registered nursing (RN) students. We used a quasi-experimental design with experimental and control groups to evaluate the effectiveness of PBL-CM in three time schedules: before the course began (pre-test), at the end of the course (post-test), and six months after the end of the course (follow-up test). A convenience sample of 120 RN students participated, 51in the experimental group and 69 in the control group. Finding showed that the experimental group had higher scores than the control group for the Critical-Thinking Scale, Self-Directed Learning Scale, and Students' Performance in PBL Tutorial Sessions Questionnaire at the post-test and follow-up test stages. The PBL-CM increased students' critical-thinking skills and personal accountability for self-directed learning, and it would enhance the skills of independent study, reasoning, group interaction and active participation. This study offers guidelines for new nurse-training programs and continuing nursing education in clinical practice.

Validation of the Tutotest in a hybrid problem-based learning curriculum

The Tutotest is one of the few structured instruments developed for the assessment of students' learning skills in a problem-based learning (PBL) curriculum. This study was designed to validate the Tutotest in a hybrid PBL curriculum. Forty-four tutors completed 370 evaluations on second to fourth year medical students at the end of the first semester in 2004 using the Chinese version of the Tutotest. There was significant correlation between global rating and Tutotest-C (r = 0.44, P < 0.001). The Cronbach's alpha coefficient was 0.97. Two-week test-retest correlation coefficient was 0.85. Factor analysis revealed four factors, where three were similar to the factors of "effectiveness in group", "communication and leadership skills", and "respect for others" identified in the original Tutotest. "Hypothesis forming and testing" instead of "scientific curiosity" became the fourth factor in our data. Our study validated the Tutotest-C in a hybrid PBL curriculum and students from the Chinese educational system. The test-retest reliability measure with a 2-week interval at the end of the PBL tutorial confirmed the stability of the Tutotest, which has not been previously reported. Since most Asian medical schools adopted a hybrid PBL curriculum, a valid student evaluation instrument for this type of curriculum is valuable.

Multiple rubric-based assessments of student case presentations

OBJECTIVES

To evaluate a rubric-based method of assessing pharmacy students' case presentations in the recitation component of a therapeutics course.

METHODS

A rubric was developed to assess knowledge, skills, and professional behavior. The rubric was used for instructor, student peer, and student self-assessment of case presentations. Rubric-based composite scores were compared to the previous dichotomous checklist-based scores.

RESULTS

Rubric-based instructor scores were significantly lower and had a broader score distribution than those resulting from the checklist method. Spring 2007 rubric-based composite scores from instructors and peers were significantly lower than those from the pilot study results, but self-assessment composite scores were not significantly different.

CONCLUSIONS

Successful development and implementation of a grading rubric facilitated evaluation of knowledge, skills, and professional behavior from the viewpoints of instructor, peer, and self in a didactic course.

PBL curriculum improves medical students' participation in small-group tutorials

BACKGROUND

Group learning is the core of problem-based learning (PBL) but has not been extensively studied, especially in Asian students.

METHODS

This study compared students of PBL and non-PBL curricula in students' talking time and participation in small-group tutorials in a medical school in Asia. The proportions of student talking of 46 tutorials in three teaching rotations of the PBL curriculum and those of 43 corresponding tutorials in the non-PBL curriculum were counted. Twelve videotapes of tutorials (six from each curriculum), stratified for tutor, case scenario and students' learning stage, were randomly selected and transcribed. They were rated with the group-interaction (5 items) and active-participation (four items) tutorial assessment scales developed by Valle et al. These outcomes were compared between the students of PBL and non-PBL curricula.

RESULTS

Students from the PBL curriculum talked significantly more. In only two (4.7%) of 43 tutorials in the non-PBL curriculum did the students talk more than the tutors; but students talked more than the tutors in 17 (37.0%) of 46 tutorials in the PBL curriculum. PBL students scored significantly higher than non-PBL students in all items except one item (respect to peers) of the tutorial assessment scales, and in the mean scores of both the group interaction scale (items 1-5) and the active participation scale (items 6-9).

CONCLUSIONS

The results suggested that PBL starting from the early years of a medical curriculum was associated with more active student participation, interaction and collaboration in small-group tutorials.

Increased student self-confidence in clinical reasoning skills associated with case-based learning (CBL)

Second-year veterinary students were surveyed at the beginning and end of a 15-week semester regarding their confidence in performing three clinical reasoning skills: (1) making Problem Lists; (2) making Rule-Out Lists; and (3) selecting appropriate diagnostic tests. Each week during the semester, these skills were practiced in small-group case discussions. Changes in self-confidence were analyzed and studied in light of faculty assessments of student competence in performance of the three skills.

RATIONALE

The purpose of the study was to determine if students' self-confidence in performing three clinical reasoning skills increased with practice.

METHODOLOGY

On the first and last days of class, students rated their confidence in each of the three skills on a scale of 0 to 10. Mean confidence scores for the whole class both for time points and for each of the three skills were analyzed.

RESULTS

There were significant increases in students' self-confidence in all three clinical reasoning skills over the semester each year. A greater percentage of students expressed improved confidence in selecting appropriate diagnostic tests than in the other two skills in three of the four years studied.

CONCLUSIONS

Students' self-confidence in performing three clinical reasoning skills improved over the course of a semester in which they practiced the skills in a CBL format. Subjective faculty assessment of students' competence in these skills generally indicated improvement. However, no meaningful conclusions about the correlation of skill competence and student confidence could be drawn because of inadequacies in the measurement of student performance.

Measuring inter-rater reliability of the sequenced performance inventory and reflective assessment of learning (SPIRAL)

PURPOSE

To measure the inter-rater reliability of the Sequenced Performance Inventory and Reflective Assessment of Learning (SPIRAL), a twenty-three item scoring rubric designed to assess first and second-year students' competencies such as "acquisition of knowledge," "peer teaching and communication skills," and "professional behavior" in a problem-based learning (PBL) curriculum at the University of North Dakota School of Medicine and Health Sciences.

METHOD

In 2001, the authors constructed a 69-item multiple-choice questionnaire consisting of descriptions of "prototypical" (representing real students) PBL student performances. For each of the 23 SPIRAL items, the authors chose competency descriptions of an "emerging," a "developing," and an "advanced" student (69 prototypical students). These descriptions were obtained from narrative comments by PBL facilitators of real students in their classes in 2000-2001. Seventeen experienced facilitators (PBL tutors) were subsequently asked to rank the 69 prototypical students based on SPIRAL anchor points in the three competency levels. Kendall's coefficient of concordance was used to test inter-rater reliability. Modes were also determined to illustrate the extent to which the ratings of the 17 facilitators aligned with the ratings anticipated by the authors.

RESULTS

Overall, inter-rater scoring for all items combined was considered to be reliable (Kendall's coefficient of concordance W =.75). Inter-rater scoring was also determined for each of the 23 SPIRAL items, with Ws ranging from.97 to.53. Facilitators rated students according to 55 of the 69 SPIRAL descriptors as the researchers anticipated.

CONCLUSIONS

The scoring rubric is reliable. Further criterion-related validation study is warranted.