How are formative assessment methods used in the clinical setting? A qualitative study

A phenomenological study of resident and faculty experiences with learner engagement in the normalization of workplace-based assessment

Background

Workplace-based assessments (WPBA) have become integral to learner-centred medical education. As previous research has linked learner engagement to WPBA implementation, this study explores residents' and faculty members' experiences with learner engagement in the normalisation of WPBA practice.

Methods

Transcendental phenomenology was used as the qualitative approach, focusing on the participants' lived experiences. A semi-structured interview guide was used to interview five faculty members and five residents who had conducted WPBA. The interviews were transcribed and analysed using phenomenological data analysis.

Results

Three themes were identified between learner engagement and WPBA conduct: (a) work environment, (b) roles and relationships, and (c) mutually beneficial teaching and learning. WPBA learner engagement occurred when participants interacted with each other and with the clinical setting to facilitate teaching and learning. Both participant groups reported a desire to participate in WPBA, but time constraints at times hindered participation. The residents indicated that WPBA improved their knowledge and admitted to experiencing negative emotions during the assessment. Overall, participants recognised the reciprocal benefits of WPBA participation for their professional development.

Conclusion

The findings of the study suggest that learner engagement influences the use of WPBA. Consequently, it may be beneficial to consider the role of learner engagement to normalise WPBA application for teaching and learning in the clinical context.

Assessing user perspectives on clinical pharmacogenomics consultation documentation: a user-centered evaluation

The University of Florida Health Precision Medicine Program plays a crucial role in delivering pharmacogenomics (PGx) result notes to providers who request PGx testing. Despite this, there is currently a lack of a formal assessment of provider needs and established best practice design principles to guide the ongoing development of PGx result notes. This study aims to enhance the content and format of the PGx consult note at UF Health by incorporating valuable feedback from healthcare providers. Through in-depth user sessions involving 11 participants, we evaluated the usability of our consult note template. While overall satisfaction with the content was noted, specific sections, including those addressing phenoconversion and the medication list, were identified for revision to enhance clarity based on insightful provider feedback.

Effective Feedback Strategy for Formative Assessment in an Integrated Medical Neuroscience Course

Purpose

Despite the different benefits of formative assessments in an integrated medical curriculum, the effective strategies to provide feedback to medical students to benefit from the different merits of formative assessment are not fully understood. This study aims to determine the effect of different strategies of formative feedback on students' outcomes in a medical neuroscience course.

Method

We compared medical students' performance in summative examinations in the academic year that formative feedback was provided using in-person discussion and compared such performances with the academic year when the feedback was provided by written rationales or a combination of written rationales and in-person discussion. We also surveyed medical students' preferences for whether written or in-person formative feedback is a better strategy to provide feedback at the end of each course.

Results

ANOVA found a significant difference in summative performance scores for those scoring ≥ 70% when formative feedback was provided by providing a rationale, in-person, and a combination of both ([F (2,80) = 247.60, P < 0.001]. Post hoc analysis revealed a significant and highest performance when feedback was provided using the written rationale approach (***P < 0.05), followed by in-person (**P < 0.05). In contrast, the least performance was recorded when formative feedback was provided using a combination of providing a written rationale for the answers to the questions and in-person discussion of the questions (*P < 0.05). Students' preferred approach for receiving formative feedback for their formative assessment was highest for written rationale (***P < 0.05), followed by in-person or a combination of in-person and written rationale (**P < 0.05).

Conclusion

Our results found that medical students preferred a written formative feedback approach, which was associated with better student performance on the summative examination. This study reveals the importance of developing effective strategies to provide formative feedback to medical students for medical students to fully benefit from the merits of formative assessment in an integrated medical school curriculum.

Twelve tips for implementing and teaching anti-racism curriculum in medical education

Racism has implicit and explicit manifestations that perpetuate disparities and negatively influence patient-centered health outcomes. Subsequently, a list of action items was provided to assist medical schools in becoming anti-racist institutions. A deep subject matter knowledge, beliefs, and reflections were a driving force for the management of medical schools or faculty members involved in undergraduate and postgraduate medical education to move forward toward inclusion of anti-racism in traditional medical curriculum or adapting existing training modules on diversity, equity, and inclusion. This paper proposes twelve practical and specific tips for implementing and teaching anti-racism in medical education. These twelve tips elaborate on the proposed actions for leaders in undergraduate and postgraduate medical education, valuable for designing future curricula and educational activities.

Fostering formative assessment: teachers' perception, practice and challenges of implementation in four Sudanese medical schools, a mixed-method study

Formative assessment (assessment for learning) enhances learning (especially deep learning) by using feedback as a central tool. However, implementing it properly faces many challenges. We aimed to describe the perception of medical teachers towards FA, their practice, challenges of implementing FA and present applicable solutions. A mixed-method, explanatory approach study was applied by administering a validated questionnaire to 190 medical teachers in four medical schools in Sudan. The obtained results were further studied using the Delphi method. Quantitative analysis revealed that medical teachers perceived their grasping of the concept of FAs and their ability to differentiate formative from summative assessments as very well (83.7%) and (77.4%), respectively. However, in contradiction to the former results, it was noteworthy that (41%) of them mistakenly perceived FA as an approach conducted for purposes of grading and certification. The qualitative study defined the challenges into two main themes: lack of understanding of formative assessment and lack of resources. Medical teachers' development and resource allocation were the main recommendations. We conclude that there is misunderstanding and malpractice in implementing formative assessment attributed to the lack of understanding of FA as well as the lack of resources. We as well present suggested solutions derived from the perception of the medical teachers in the study and evolved around three approaches: faculty development, managing the curriculum by allocating time and resources for FA, and advocacy among stakeholders.

Optimal training in communication model (OPTiCOM): A programmatic roadmap

OBJECTIVES

Teaching primary care residents patient communication skills is essential, complex, and impeded by barriers. We find no models guiding faculty how to train residents in the workplace that integrate necessary system components, the science of physician-patient communication training and competency-based medical education. The aim of this project is to create such a model.

METHODS

We created OPTiCOM using four steps: (1) communication educator interviews, analysis and theme development; (2) initial model construction; (3) model refinement using expert feedback; (4) structured literature review to validate, refine and finalize the model.

RESULTS

Our model contains ten interdependent building blocks organized into four developmental tiers. The Foundational value tier has one building block Naming relationship as a core value. The Expertize and resources tier includes four building blocks addressing: Curricular expertize, Curricular content, Leadership, and Time. The four building blocks in the Application and development tier are Observation form, Faculty development, Technology, and Formative assessment. The Language and culture tier identifies the final building block, Culture promoting continuous improvement in teaching communication.

CONCLUSIONS

OPTiCOM organizes ten interdependent systems building blocks to maximize and sustain resident learning of communication skills. Practice Implications Residency faculty can use OPTiCOM for self-assessment, program creation and revision.

Opportunities to Increase Workplace Feedback: A Multi-institutional Examination

OBJECTIVE

While feedback is an essential component of resident education, there are few large-scale studies examining when and under what conditions formative feedback is provided. Workplace-based assessment systems offer an opportunity to identify factors influencing when faculty provides feedback to trainees. Influential factors affecting feedback may provide targets for increasing and improving feedback in resident education.

DESIGN

Data on whether dictated feedback was provided were obtained from the Society for Improving Medical Professional Learning (SIMPL) mobile application. We used generalized linear mixed effects models to identify the degree to which faculty members, procedures, surgical case characteristics, and trainee performance were associated with whether narrative feedback was provided using SIMPL.

SETTING

This study was conducted using data from members of the SIMPL collaborative.

PARTICIPANTS

67,434 evaluations from 70 general surgery programs were included from 2015 to 2021. Of these, 25,355 evaluations included dictated feedback.

RESULTS

Approximately 61% of the variation in whether dictated feedback was provided was attributable to the individual faculty member. Compared to residents who achieved autonomy ratings of "Active Help," residents who achieved ratings of "Supervision Only" (odds ratio (OR) = 0.80, 95% confidence interval (CI) = 0.72, 0.88) had a lower likelihood of receiving dictated feedback. Residents who achieved ratings of "Intermediate" (OR = 0.81, CI = 0.74, 0.89), "Practice-Ready" (OR = 0.50, CI = 0.45, 0.57), or "Exceptional (OR = 0.64, CI = 0.54, 0.76) showed a lower likelihood of receiving dictated feedback compared to those rated as "Inexperienced." Cases rated as "High" in terms of complexity were associated with an increased likelihood of having dictation (OR = 1.35, CI = 1.26, 1.44).

CONCLUSIONS

The largest contributing factor for whether dictated feedback is included in a SIMPL evaluation are factors specific to the attending surgeon. Resident performance, resident autonomy, and case complexity had only modest associations with feedback decisions. Efforts to improve the amount of formative feedback for trainees should be directed towards reducing the variation in which attending surgeons elect to provide feedback.

Medical school grades may predict future clinical competence

BACKGROUND

In real-world medical education, there is a lack of reliable predictors of future clinical competencies. Hence, we aim to identify the factors associated with clinical competencies and construct a prediction model to identify "improvement required" trainees.

METHODS

We analyzed data from medical students who graduated from National Yang-Ming University with clerkship training and participated in the postgraduate year (PGY) interview at Taipei Veterans General Hospital. Clinical competencies were evaluated using grades of national objective structured clinical examination (OSCEs). This study used data from medical students who graduated in July 2018 as the derivation cohort (N = 50) and those who graduated in July 2020 (n = 56) for validation.

RESULTS

Medical school grades were associated with the performance of national OSCEs (Pearson r = 0.34, p = 0.017), but the grades of the structured PGY interviews were marginally associated with the national OSCE (Pearson r = 0.268, p = 0.06). A prediction model was constructed to identify "improvement required" trainees, defined: trainees with the lowest 25% of scores in the national OSCEs. According to this model, trainees with the lowest 25% medical school grades predicted a higher risk of the "improvement required" clinical performance (Q1-Q3 vs Q4 = 15% vs 60%, odds ratio = 8.5 [95% confidence interval = 1.8-39.4], p = 0.029). In the validation cohort, our prediction model could accurately classify 76.7% "improvement required" and "nonimprovement required" students.

CONCLUSION

Our study suggests that interventions for students with unsatisfactory medical school grades are warranted to improve their clinical competencies.

"Written Formative Assessments with Peer-Assisted Learning" an Innovative Teaching Program for Postgraduate Students in Community Medicine

Introduction:

There is a paucity of research on conducting written formative assessment with constructive feedback for theory paper writing for postgraduates of Community Medicine in India. The concept of “Written Formative assessments with Peer-Assisted Learning Program” was implemented to improve the first 2 levels of Miller's Pyramid and assess its impact on the summative assessment.

Materials and Methods:

The program was conducted for 2 batches of postgraduate students in the Community Medicine enrolled for the academic session of 2016–2019 and 2017–2020. The written formative assessment was conducted every Saturday for 1 h from August to March month in 2018 and 2019. After each test, answer papers were evaluated by the peer and faculty from the department. Written and oral feedback was given by the peer. After IEC approval, we planned to assess the program's effect on level 1 and level 2 Kirkpatrick's framework. The data were analyzed using SPSS statistical package version 24 (SPSS Inc., Chicago, IL, USA). P < 0.05 was considered statistically significant.

Results:

Total 23 formative written assessments were conducted per year. The proportions of knowledge, comprehension, and analytical type of questions asked were 47%, 32%, and 21%, respectively. The mean attendance rate was 76.28% ±16.4%. There was no statistically significant difference in the average percentage of marks in formative (weekly test) and summative assessment (university final examination). There was a statistically significant positive co-relation of projected mean marks and summative assessment marks with the co-efficient of the determination being 22.6%. There was overall positive feedback of the formative and peer-assisted learning (PAL) from post graduate students.

Conclusions:

Written Formative Assessment with PAL program is one of the effective programs for postgraduate students to gain confidence in writing and presentation skills and to score higher in theory examination.

Association Between Formative Assessment and Academic Performance for Undergraduate Medical Students in a Chinese Clinical Skills Training Course

BACKGROUND Previous research suggests that formative assessment (FA) enhances learning outcomes, but few studies have evaluated its impact on clinical skills training in China. We conducted this study in a clinical skills integral curriculum to further explore the educational value of FA. MATERIAL AND METHODS Sixty undergraduates from the Second Clinical Medical School of the Southern Medical University in 2016 were selected as the experimental group (consecutive FA), and 50 undergraduates in 2015 were selected as the control term (only final summative assessment, SA). Undergraduates in the FA group completed the after-class questionnaire at each lesson. Teachers, teaching content, assessment objectives, and topics are the same in both groups. RESULTS The results of single-factor covariance (ANCOVA) analysis and Cochran-Mantel-Haenszel (CMH) analysis demonstrated that students of the FA group obtained better performance and higher success rates in summative examination than in the SA group. The students with relatively poor grades benefited more from FA, while the performance of students with higher grades was similar between the FA group and SA group. According to the results of questionnaire for students, the satisfaction of students with the course increased gradually, from 84.4% to 93.0%. CONCLUSIONS Proper use of FA is associated with better learning outcomes for students, especially for those with poorer grades. Our results, together with previous research, indicated that the use of FA may be of great benefit to students' academic performance and satisfaction with the clinical skills training curriculum.