Ten maxims of formative assessment

Collaborative Assessment in the Gross Anatomy Lab

Purpose

In 2018, the Michigan State University College of Human Medicine incorporated two-stage examinations into the gross anatomy curriculum. Multiple studies have investigated two-stage examinations and have largely reported positive findings. Here, we used a mixed-methods approach to further investigate the feasibility and student perceptions of the two-stage examination in the context of a medical school curriculum that emphasizes longitudinal group-based learning and formative assessments.

Methods

Three student cohorts were assessed with a formative two-stage examination at the end of their first-year anatomy experience. Data for the quantitative analysis included examination scores from the individual and group portions of the two-stage examination. For the qualitative stage of this project, we utilized a constructivist grounded theory methodology in which data, including both post-examination survey results and one-on-one semi-structured student interviews, were transcribed (interviews), coded, inductively and iteratively reviewed, and thematically interpreted.

Results

Survey and interview results revealed an overwhelmingly positive perception of the collaborative assessment experience. Student comments demonstrated educational value in the immediate feedback provided by this examination format and suggested that collaboration during the examination transformed the assessment into a learning experience.

Conclusions

While two-stage examinations have the potential to positively transform an assessment into a learning experience, we also identified complex relationships between content knowledge and anxiety that may affect student perceptions. In addition, examination logistics (e.g., curricular timing) have the potential to negatively affect student perceptions, indicating that faculty should consider these factors when implementing collaborative assessments into their curriculum.

Student perceptions of Twitter poll-based formative assessment in a rural medical college in India

Formative assessment is vital for student learning and engagement. Social media platforms like Twitter have gained popularity in medical education, but little research has explored student perceptions of formative assessment through Twitter. This study aimed to observe participation rates in Twitter poll-based formative assessment and survey students' perceptions of it in a rural medical college in India. Fifteen formative assessment sessions were conducted using a Twitter poll after physiology lectures, with five multiple-choice questions per session. The number of participants in each poll was recorded. A questionnaire collected student perceptions after the sessions. Across 15 Twitter poll sessions, we received an average of 12.65 ± 5.53 (median 12) responses. Most students (63%) did not participate, with only 4% participating in over 10 sessions. A total of 107 students participated in the survey, resulting in an 85.6% response rate. Students found social media-based formative assessment interesting and effective for increasing attention in class. However, they preferred alternative platforms like Telegram and Instagram for formative assessment. In conclusion, most students in a rural medical college in India do not participate in Twitter poll-based formative assessment. Nevertheless, students found it interesting and effective, highlighting the importance of considering students' platform preferences for implementing social media-based formative assessment.NEW & NOTEWORTHY The study presents the participation of medical students in Twitter poll-based formative assessment in a rural medical college in India. Despite low participation rates, students found social media-based formative assessment highly effective in increasing their attention during class. The study also reveals student preferences for alternative platforms like Telegram and Instagram. These insights contribute significantly to understanding the impact and relevance of social media-based formative assessment in diverse educational contexts.

Design of online formative assessment of nursing humanities curriculum during the COVID-19 pandemic: A teaching practice research

BACKGROUND

Under the digital transformation trend nursing education, online formative assessment (OFA) provides a new opportunity. However, the OFA of nursing humanities course lacks design and practice, and faces the challenge of enhancing effective communication between teachers and students, student participation and autonomous learning.

OBJECTIVES

To enhance the reliability of OFA in nursing humanities courses and provide practical experience for online teaching in the nursing profession.

DESIGN

A quantitative research approach was used.

SETTING

This study was conducted in a comprehensive university in China.

PARTICIPANTS

We conducted teaching practice on 185 nursing undergraduates, with 89 students in the experimental group, and 96 students in the control group.

METHODS

In the 2020-2021 multicultural nursing course, student learning outcomes and questionnaires were analyzed through the online learning tool Superstar Learning, student feedback and satisfaction questionnaires, and descriptive analysis and independent sample t-tests were conducted using SPSS 25.0 software.

RESULTS

The OFA of students using Superstar Learning differed in learning performance and time to receive feedback from teachers between the experimental and control groups, and both groups had higher satisfaction levels. The experimental group's instructional design contained a synchronous classroom discussion module with better participation.

CONCLUSIONS

During the COVID-19 pandemic, the use of online learning tools can support the implementation of OFA, build an environment where teachers and students participate together, have a positive impact on the continuous updating of teachers' teaching programs and students' learning outcomes. Simultaneous classroom discussions are expected to be an effective way to improve the reliability of OFA. Our instructional design, provides best practice suggestions for future online teaching and learning.

Effective formative assessment for pharmacy students in Thailand: lesson learns from a school of pharmacy in Thailand

INTRODUCTION

Formative assessment (FA) is an assessment concept that is of interest in education. The Doctor of Pharmacy program is one of the programs in which FA is usually implemented. This study aimed to describe the correlation between FA scores and summative assessment (SA) scores and to suggest possible key success factors that affect the effectiveness of FA.

METHODS

This study employed a retrospective design using mixed methods for data collection. Data in the semesters 1/2020 and 2/2020 of the Doctor of Pharmacy curriculum in a Thailand pharmacy school were used. Three sets of data were gathered, including the course information (e.g. FA methods, FA scores, and SA scores) from 38 records, self-reports from 326 students and 27 teachers, and 5 focus group discussions. The quantitative data were statistically analyzed using descriptive statistics and Pearson correlation, while the qualitative data were analyzed using a content analysis framework.

RESULTS

The analysis revealed five main methods that were used for FA, including individual quizzes, individual reports, individual skill assessments, group presentations, and group reports. Of all 38 courses, 29 (76.32%) had significant correlations between FA and SA scores at p-values < 0.05. The individual FA score was related to the correlation coefficient of the courses (p-value = 0.007), but the group FA score was not related (p-value = 0.081). In addition, only the frequency of individual quiz had a significant effect on the correlation coefficient. Moreover, the key success factors which affected the effectiveness of FA were divided into six themes, including the appropriate method, an effective reflection, frequency of assessment, the appropriate score, the adequate support system, and teacher knowledge management.

CONCLUSION

The subjects that used individual FA methods provided a significant correlation between FA and SA, while those who used group FA methods did not show a significant correlation. Additionally, the key success factors in this study were appropriate assessment methods, frequency of assessment, effective feedback, appropriate scoring, and a proper support system.

Formative Objective Structured Clinical Examinations (OSCEs) as an Assessment Tool in UK Undergraduate Medical Education: A Review of Its Utility

The Objective Structured Clinical Examination (OSCE) is a globally established clinical examination; it is often considered the gold standard in evaluating clinical competence within medicine and other healthcare professionals' educations alike. The OSCE consists of a circuit of multiple stations testing a multitude of clinical competencies expected of undergraduate students at certain levels throughout training. Despite its widespread use, the evidence regarding formative renditions of the examination in medical training is highly variable; thus, its suitability as an assessment has been challenged for various reasons. Classically, Van Der Vleuten's formula of utility has been adopted in the appraisal of assessment methods as means of testing, including the OSCE. This review aims to provide a comprehensive overview of the literature surrounding the formative use of OSCEs in undergraduate medical training, whilst specifically focusing on the constituents of the equation and means of mitigating factors that compromise its objectivity.

Does formative assessment help students to acquire prescribing skills?

PURPOSE

Formative assessments can help motivate students and ease students' learning through feedback. There is a pressing need for improvement of clinical pharmacotherapy (CPT) education since junior doctors make many prescribing errors. The aim of this study was to determine whether a formative assessment with personalized narrative feedback helps medical students to increase their prescribing skills.

METHODS

This retrospective cohort study was conducted among masters' medical students at Erasmus Medical Centre, The Netherlands. Students made a formative and a summative skill-based prescription assessment, both during clerkships as part of their regular curriculum. Errors in both assessments were categorized by type and possible consequence and compared with each other.

RESULTS

A total of 388 students made 1964 errors in the formative assessment and 1016 in the summative assessment. Most improvements after the formative assessment were seen for mentioning the weight of a child on the prescription (n = 242, 19%). Most new and repeated errors in the summative assessment were missing usage instructions (n = 82, 16% and n = 121, 41%).

CONCLUSIONS

This formative assessment with personalized and individual narrative feedback has helped students to increase the technical correctness of their prescriptions. However, errors repeated after the feedback were predominantly errors showing that only one formative assessment has not yet enhanced the clinical prescribing enough.

Real-life scenario blended teaching approach for nurturing inquisitive learning of central nervous system in medical students

Among the various systems taught in the preclinical phases, the nervous system is more challenging to learn than other systems. In this report, a novel teaching methodology, "real-life scenario (RLS) blended teaching," is described and its effectiveness in facilitating inquisitive learning in undergraduate medical students is evaluated. This mixed-method study was conducted among three groups (group 1, n = 83; group 2, n = 85; and group 3; n = 79) of undergraduate medical students (18-20 yr) in the neurology and behavioral sciences module. RLS was presented to students in the form of demonstrations, role-plays, videos, and group activities. Group 1 students underwent traditional teaching-learning sessions. Group 2 students underwent RLS blended sessions and were provided with multiple miniassignments in a vignette format. Group 3 students received RLS blended sessions, multiple miniassignments, peer discussions, multiple formative assessments, and facilitator feedback sessions. The student performances on different exams were compared in terms of their group, and their perceptions of RLS were documented. Students exposed to RLS sessions blended with multiple assignments, peer discussions, multiple formative assignments, and facilitator feedback sessions performed well in the final summative assessments (67.87%) compared with those exposed to RLS sessions and assignments (50.21%) or exposed to traditional teaching alone (50.34%). RLS sessions increased students' curiosity and motivated them to learn the subject well. RLS sessions stimulated student interest and facilitated their learning. RLS along with effective use of multiple assignments, formative assessments, and/or feedback sessions significantly improved student learning. This demonstrates the effectiveness of this active method in teaching various subjects with appropriate modifications.NEW & NOTEWORTHY In this report, a novel teaching methodology, "real-life scenario (RLS) blended teaching" is described and its effectiveness in facilitating inquisitive learning in undergraduate medical students is evaluated. Students exposed to RLS sessions blended with multiple miniassignments, peer discussions, multiple formative assignments, and facilitator feedback sessions performed well in the summative assessments compared with those exposed to RLS sessions and assignments or exposed to traditional teaching alone. Students preferred active teaching-learning techniques over the traditional method.

Assessing the Utility of a Quality-of-Care Assessment Tool Used in Assessing Comprehensive Care Services Provided by Community Health Workers in South Africa

Background

Few studies exist on the tools for assessing quality-of-care of community health worker (CHW) who provide comprehensive care, and for available tools, evidence on the utility is scanty. We aimed to assess the utility components of a previously-reported quality-of-care assessment tool developed for summative assessment in South Africa.

Methods

In two provinces, we used ratings by 21 CHWs and three team leaders in two primary health care facilities per province regarding whether the tool covered everything that happens during their household visits and whether they were happy to be assessed using the tool (acceptability and face validity), to derive agreement index (≥85%, otherwise the tool had to be revised). A panel of six experts quantitatively validated 11 items of the tool (content validity). Content validity index (CVI), of individual items (I-CVI) or entire scale (S-CVI), should be >80% (excellent). For the inter-rater reliability (IRR), we determined agreement between paired observers' assigned quality-of-care messages and communication scores during 18 CHW household visits (nine households per site). Bland and Altman plots and multilevel model analysis, for clustered data, were used to assess IRR.

Results

In all four CHW and team leader sites, agreement index was ≥85%, except for whether they were happy to be assessed using the tool, where it was <85% in one facility. The I-CVI of the 11 items in the tool ranged between 0.83 and 1.00. For the S-CVI, all six experts agreed on relevancy (universal agreement) in eight of 11 items (0.72) whereas the average of I-CVIs, was 0.95. The Bland-Altman plot limit of agreements between paired observes were −0.18 to 0.44 and −0.30 to 0.44 (messages score); and −0.22 to 0.45 and −0.28 to 0.40 (communication score). Multilevel modeling revealed an estimated reliability of 0.77 (messages score) and 0.14 (communication score).

Conclusion

The quality-of-care assessment tool has a high face and content validity. IRR was substantial for quality-of-care messages but not for communication score. This suggests that the tool may only be useful in the formative assessment of CHWs. Such assessment can provide the basis for reflection and discussion on CHW performance and lead to change.

How do Trainers and Trainees use Formative Workplace-based Assessments of Operative Competence to Deliver and Receive Structured Feedback?

OBJECTIVE

The aim of this study is to assess the quality of feedback provided to surgical trainees in the operating theatre, and to further investigate how trainees and trainers use workplace-based assessment in practice with regards to frequency and timing of assessments.

DESIGN

A retrospective study of all submitted Supervised Structured Assessments of Operative Performance (SSAOPs) from April 25, 2016 to February 2, 2021 was conducted.

SETTING

Surgical trainees in the Republic of Ireland across all national surgical training sites submitted SSAOPs through an online platform.

PARTICIPANTS

Assessments of operative competence (SSAOPs) from all Core Surgical Trainees (in their first two years of dedicated post-graduate surgical training) were included for analysis, regardless of surgical subspecialty. A total of 2294 assessments were submitted from April 25, 2016 to February 2, 2021 by 330 core surgical trainees and 379 surgeon assessors. Five hundred of these assessments were randomly selected and scored for quality of feedback using a modified "Task, Gap, Action (TGA)" framework.

RESULTS

Of all 2294 submitted assessments, 1905 (83.04%) were submitted in the latter 3 months of each rotation, and 803 (35%) were submitted in the last month. Only 51 of 270 (18.89%) of trainees in their first year and 33 of 236 trainees in their second year (13.98%) submitted more than the minimum required number of assessments (6 per year). Of 500 randomly selected assessments, 362 (72.4%) had documented written feedback. The mean 'Gap' and 'Action' scores were low, at 0.44/3 and 0.53/3 respectively.

CONCLUSIONS

Trainees do not submit more than the required number of operative workplace-based assessments. Assessments are submitted at the end of the trainee's rotation, limiting their formative value. The quality of written feedback is poor and could be improved significantly by encouraging a "Task," "Gap" and "Action" approach.

Web-based formative assessment through clinical cases: role in pathophysiology teaching

BACKGROUND

Active learning strategies such as formative assessment through clinical cases may help to get a deeper learning. We have studied the effect of this kind of online formative assessment in pathophysiology teaching.

METHODS

Seven brief clinical cases were used to give formative assessment in the first semester of a pathophysiology course. To evaluate its effect on learning, we analyzed the proportion of students that passed the end of semester exam with a score above 60 over 100. We also analyzed the effect of the intervention according to the students' previous academic performance.

RESULTS

Ninety-six students participated in the study and sat the exam. Sixty-five of them passed it. Students that passed the exam had a higher previous academic performance and had done a higher number of exercises of formative assessment, both in univariate and multivariate analysis. The participants were divided in three groups, according to their previous academic performance. In the intermediate group, the number of cases done by the students who passed the exam was significantly higher than in those who did not pass it (median: 4 versus 0; P = 0.009).

CONCLUSION

Formative assessment through web-based clinical cases was followed by an improvement of the academic results in pathophysiology, mainly in students with intermediate performance.

Medical students' perception of the reliability, usefulness and feasibility of unproctored online formative assessment tests

Medical education has gone online because of the COVID-19 pandemic. Formative assessment is essential to facilitate the learning process in medical education. However, various challenges arise during online assessment, which include reliability, when done without monitoring and practical concerns like Internet connectivity issues. This study was done to assess the medical students' perceptions of the reliability, usefulness, and practical challenges of online tests. One hundred first-year undergraduate medical students taking up online classes and tests in the subject of physiology were enrolled in this study. A questionnaire with items regarding practical challenges, reliability, and usefulness of the online tests, in general, and about different types of online assessment methods, in particular, were sent to the students online. Each item was rated on a five-point Likert scale, and the responses were analyzed anonymously. A large percentage of students used mobile phones (81.4%) to undertake online tests. Although most students (73.2%; P < 0.001) felt that online tests helped them substantially in learning the subject, network connectivity issues were considered to be a matter of serious concern (85.5%, P < 0.001). Among the assessment methods used, viva voce by video conferencing was thought to be most reliable (83%, P < 0.001). Multiple-choice question-based assessment when done online was felt to be more practically feasible with faster feedback than classroom assessment. The results of the study suggest that medical students find online formative assessments helpful for their learning, despite their concerns about reliability and practical challenges.

Association Between Formative Assessment and Academic Performance for Undergraduate Medical Students in a Chinese Clinical Skills Training Course

BACKGROUND Previous research suggests that formative assessment (FA) enhances learning outcomes, but few studies have evaluated its impact on clinical skills training in China. We conducted this study in a clinical skills integral curriculum to further explore the educational value of FA. MATERIAL AND METHODS Sixty undergraduates from the Second Clinical Medical School of the Southern Medical University in 2016 were selected as the experimental group (consecutive FA), and 50 undergraduates in 2015 were selected as the control term (only final summative assessment, SA). Undergraduates in the FA group completed the after-class questionnaire at each lesson. Teachers, teaching content, assessment objectives, and topics are the same in both groups. RESULTS The results of single-factor covariance (ANCOVA) analysis and Cochran-Mantel-Haenszel (CMH) analysis demonstrated that students of the FA group obtained better performance and higher success rates in summative examination than in the SA group. The students with relatively poor grades benefited more from FA, while the performance of students with higher grades was similar between the FA group and SA group. According to the results of questionnaire for students, the satisfaction of students with the course increased gradually, from 84.4% to 93.0%. CONCLUSIONS Proper use of FA is associated with better learning outcomes for students, especially for those with poorer grades. Our results, together with previous research, indicated that the use of FA may be of great benefit to students' academic performance and satisfaction with the clinical skills training curriculum.

Low performance of internal medicine senior residents in in-service examinations

We observed an unprecedented and consistent low performance of senior residents as compared with juniors in monthly examinations. This compelled us to evaluate systematically and compare the scores of senior residents with their junior colleagues. This retrospective observational study was conducted in April 2020 among internal medicine residents. Residents in first and second year of their training were labelled as junior residents. Residents in third or fourth year of their training were labelled as senior residents. Comparison of mean scores of each resident level was done separately both for monthly formative multiple-choice questions tests, and summative yearly end of term examinations. We discussed the possible reasons as well. There were 67 residents in year 2018 and 69 in 2019. There is no significant difference between scores of monthly examinations of years 2018 and 2019 among residents of each level. Rather, in March and December 2018, junior residents perform better than senior residents with p values of 0.01 and 0.04, respectively. In February and September 2019, senior residents performed better than junior residents with p value of 0.02. Similarly, there is no significant difference in scores of residents of each level in end-of-term examinations of years 2018 and 2019 with p values 0.18 and 0.25, respectively. Performance of senior residents in our residency programme in in-service examinations is relatively low as compared with their junior colleagues. There is a need to evaluate reasons for this relatively low performance of senior residents.

Effect of an active learning methodology combined with formative assessments on performance, test anxiety, and stress of university students

The aim of this study was to evaluate the effect of an active methodology combined with a lecture on undergraduate student learning and levels of stress and anxiety. The active learning methodology consisted of a lecture of 50-min duration, study at home with a textbook, an educational game activity, and three formative assessments on the topic of the cardiac cycle. In a following class, the students provided saliva samples to evaluate their levels of stress, received an anxiety test, and then undertook an exam to assess their understanding of the cardiac cycle. The traditional teaching methodology consisted of two lectures (∼2-h duration) on blood pressure control systems, delivered orally. In the third class, the students provided saliva samples, received an anxiety test, and then undertook an exam to assess their understanding of blood pressure control systems. The level of stress was assessed with the concentrations of the stress biomarkers cortisol and alpha-amylase in saliva. Anxiety was assessed with the State-Trait Anxiety Inventory (STAI) questionnaire. The students achieved significantly higher average scores in exams when the active learning strategy was applied compared with the use of traditional theoretical classes. The active methodology resulted in significantly lower levels of stress and anxiety, as well as improved student performance, compared with the use of traditional lectures.

Ten maxims for creating a safe learning environment

To foster a milieu in which student learning can be optimum, teachers need to be aware of the attributes of a safe learning environment. This is the space created in the students' minds to seamlessly promote learning. The 10 maxims, presented in this paper, are the cornerstones, nay, the capstones, for making this happen.