Using deliberate practice framework to assess the quality of feedback in undergraduate clinical skills training

Student Pharmacists Provide Similar Quality Clinical Reasoning Feedback as Resident Teaching Assistants

OBJECTIVE

Pharmacists utilize clinical reasoning (CR) to improve patient outcomes via medication optimization. It is critical to develop these skills in student pharmacists, yet optimal pedagogies to teach and assess CR are unknown. Peer feedback may be used to develop CR in student pharmacists, but a certain feedback quality must be reached to be effective. This study sought to evaluate if student pharmacists could provide similar quality peer feedback compared to pharmacy resident teaching assistant (TA) feedback.

METHODS

This was a retrospective, mixed-methods pedagogical analysis comparing the quality of first-year student pharmacist peer feedback to resident TA CR feedback. The CR comments were defined using the intellectual standards of CR. Quality was assessed for task specification, gap identification, actionability, and process orientation by 2 independent investigators. Student performance and perceptions were also assessed. Mann-Whitney U, t tests, and descriptive statistics were used to analyze data where appropriate.

RESULTS

Clinical reasoning feedback from peers (N = 805) and TAs (N = 206) were analyzed. Interrater reliability for feedback quality was moderate to substantial. Overall, peer CR feedback was of higher quality regarding task specification and process orientation while TA CR feedback was of higher quality regarding gap identification and actionability. Students receiving peer feedback performed better on a final patient case than those receiving TA feedback (95.2% vs 92.3%). Overall, the peer feedback process was well received by students.

CONCLUSION

Student pharmacists can provide similar quality feedback as resident TAs. Peer feedback offers an alternative to resident TA feedback and has the potential to contribute to improved CR skills.

An innovative assessment tool for evaluating narrative feedback quality among Medicine and Biomedical Sciences students

Objectives

To develop a reliable instrument to objectively assess feedback quality, to use it for assessment of the quality of students' narrative feedback and to be used as a self-assessment instrument for students in their learning process.

Methods

In a retrospective cohort study, 635 feedback narratives, provided by small groups of Medicine and Biomedical Sciences undergraduate students, have been extracted from available quarterly curriculum evaluation surveys. A rubric was developed based on literature and contents of our feedback education. It consists of seven subitems and has a maximum score of 20 points (sufficient score: >10 points). Rubric reliability was evaluated using intra-class correlation. The rubric was tested by analysing the feedback narratives. To test progression, we compared rubric scores between study years with a Kruskal-Wallis analysis and Dunn's post-hoc testing with Bonferroni correction.

Results

The rubric has an intra-class correlation of 0.894. First year students had a mean rubric score of 11.5 points (SD 3.6), second year students 12.4 (SD 3.4) and third year students 13.1 (SD 3.6). Kruskal-Wallis testing showed significant differences in feedback quality between study years (χ2(2, N=635) = 17.53, p<0.001). Dunn's post-hoc test revealed significant differences between study years one and two (p=0.012) and one and three (p<0.001).

Conclusions

The developed rubric is a reliable instrument to assess narrative feedback quality. Students were able to provide feedback of sufficient quality and quality improved across study years. The instrument will allow students to assess themselves and learn where there is still room for improvement.

Factors That Enhance and Hinder the Retention and Transfer of Online Pre-Clinical Skills Training to Facilitate Blended Learning

Purpose

During the SARS-CoV-2 pandemic, various online instructional strategies in clinical skills training were piloted. The sudden transition to the remote platform circumvented the rigorous planning associated with curriculum reform. This study aimed to explore students' and tutors' perceptions of factors that promoted or hindered successful learning transfer and to propose a blended conceptual model to guide affective, cognitive, and psychomotor clinical skills training in the pre-clinical phase of medical education.

Methods

A mixed-method quasi-experimental study assessed third-year students' transfer of clinical skills and knowledge following online learning in 2021. Students and their tutors completed online surveys that included open and closed-ended questions regarding factors influencing their experience of the adapted teaching methods. Descriptive statistical analysis was used for the quantitative data. Qualitative responses were thematically analyzed.

Results

One hundred fourteen students (48%) and seven tutors (100%) responded to the surveys. The questionnaires' internal consistency and construct validity were determined using Cronbach's α-Coefficient. There was an overall positive response (86%) to the acceptability of the online platform in clinical skills training. Using online simulations with targeted onsite practice was reported as effective in clinical skills training. Tutors perceived students as well-prepared for the skills laboratory. Five emergent themes, qualified by a linear model of asynchronous and synchronous online and onsite teaching with the evaluation of the instructional design and institutional support, informed the proposed blended learning guide for clinical skills training in the pre-clinical phase.

Conclusion

Blended clinical skills learning that included the flipped classroom concept was well-accepted. Virtual patients proved a convenient cognitive preparation tool for skills training and potentially optimized teaching delivery. The study found that the adapted teaching frameworks incorporating an online clinical skills component into a modified onsite curriculum augmented learners' ability to transfer knowledge to the clinical skills laboratory. An integrated five-step blended model is proposed for future interventions.

A systematic review of effective quality feedback measurement tools used in clinical skills assessment

BACKGROUND

Objective Structured Clinical Examination (OSCE) is a valid tool to assess the clinical skills of medical students. Feedback after OSCE is essential for student improvement and safe clinical practice. Many examiners do not provide helpful or insightful feedback in the text space provided after OSCE stations, which may adversely affect learning outcomes. The aim of this systematic review was to identify the best determinants for quality written feedback in the field of medicine.   Methods: PubMed, Medline, Embase, CINHAL, Scopus, and Web of Science were searched for relevant literature up to February 2021. We included studies that described the quality of good/effective feedback in clinical skills assessment in the field of medicine. Four independent reviewers extracted determinants used to assess the quality of written feedback. The percentage agreement and kappa coefficients were calculated for each determinant. The ROBINS-I (Risk Of Bias In Non-randomized Studies of Interventions) tool was used to assess the risk of bias.

RESULTS

14 studies were included in this systematic review. 10 determinants were identified for assessing feedback. The determinants with the highest agreement among reviewers were specific, described gap, balanced, constructive and behavioural; with kappa values of 0.79, 0.45, 0.33, 0.33 and 0.26 respectively. All other determinants had low agreement (kappa values below 0.22) indicating that even though they have been used in the literature, they might not be applicable for good quality feedback. The risk of bias was low or moderate overall.

CONCLUSIONS

This work suggests that good quality written feedback should be specific, balanced, and constructive in nature, and should describe the gap in student learning as well as observed behavioural actions in the exams.  Integrating these determinants in OSCE assessment will help guide and support educators for providing effective feedback for the learner.

Use of digital technology to give and receive feedback in clinical training: a scoping review protocol

BACKGROUND

Feedback is vital to improving trainee competencies in medical education. The challenges of the global COVID-19 pandemic related to social distancing to curb the spread of the virus ignited a rapid transition to online medical education. These changes highlight the need for digital feedback tools that enhance the efficacy and efficiency of feedback practices. This protocol is for a scoping review that aims to identify the different digital tools and applications in medical education as reported in the literature, as well as highlight gaps in the current literature and provide suggestions for future technological developments and research.

METHODS AND ANALYSIS

A review of the relevant literature will be guided using the Joanna Briggs Institute methodological framework for scoping studies. Using the search strategy developed by the authors, an electronic search of the following databases will be conducted: PubMed/MEDLINE, EBSCOhost (academic search complete, CINAHL with full text) Scopus, Google Scholar, Union Catalogue of Theses and Dissertations (UCTD) via SABINET Online and World Cat Dissertations and Theses via OCLC. Studies will be identified by searching literature from January 2010 to date of review. Using a validated data extraction form developed for the scoping review, the review team will screen eligible studies and import them onto an electronic library created specifically for this purpose. Data collection for the review will be documented through a PRISMA-P flowchart, and the scoping review will use a basic descriptive content analysis to analyse and categorise the extracted data. All review steps will involve two or more reviewers.

DISSEMINATION

The review will provide a comprehensive list of digital tools and applications used to enhance feedback in clinical training and inform future technological developments. The findings will be disseminated through medical education conferences and publications.

Narrative Assessments in Higher Education: A Scoping Review to Identify Evidence-Based Quality Indicators

PURPOSE

Narrative comments are increasingly used in assessment to document trainees' performance and to make important decisions about academic progress. However, little is known about how to document the quality of narrative comments, since traditional psychometric analysis cannot be applied. The authors aimed to generate a list of quality indicators for narrative comments, to identify recommendations for writing high-quality narrative comments, and to document factors that influence the quality of narrative comments used in assessments in higher education.

METHOD

The authors conducted a scoping review according to Arksey & O'Malley's framework. The search strategy yielded 690 articles from 6 databases. Team members screened abstracts for inclusion and exclusion, then extracted numerical and qualitative data based on predetermined categories. Numerical data were used for descriptive analysis. The authors completed the thematic analysis of qualitative data with iterative discussions until they achieved consensus for the interpretation of the results.

RESULTS

After the full-text review of 213 selected articles, 47 were included. Through the thematic analysis, the authors identified 7 quality indicators, 12 recommendations for writing quality narratives, and 3 factors that influence the quality of narrative comments used in assessment. The 7 quality indicators are (1) describes performance with a focus on particular elements (attitudes, knowledge, skills); (2) provides a balanced message between positive elements and elements needing improvement; (3) provides recommendations to learners on how to improve their performance; (4) compares the observed performance with an expected standard of performance; (5) provides justification for the mark/score given; (6) uses language that is clear and easily understood; and (7) uses a nonjudgmental style.

CONCLUSIONS

Assessors can use these quality indicators and recommendations to write high-quality narrative comments, thus reinforcing the appropriate documentation of trainees' performance, facilitating solid decision making about trainees' progression, and enhancing the impact of narrative feedback for both learners and programs.

How to enhance students' learning in a patient-centered longitudinal integrated clerkship: factors associated with students' learning experiences

PURPOSE

Longitudinal integrated clerkships (LICs) have been introduced in medical schools, as learning relationships with clinical faculty or peers are important components of medical education. The purpose of this study was to investigate the characteristics of student-faculty and student-student interactions in the LIC and to identify other factors related to whether students understood and acquired the program's main outcomes.

METHODS

The study was conducted among the 149 third-year students who participated in the LIC in 2019. We divided the students into groups of eight. These groups were organized into corresponding discussion classes, during which students had discussions with clinical faculty members and peers and received feedback. Clinical faculty members and students were matched through an e-portfolio, where records were approved and feedback was given. A course evaluation questionnaire was completed and analysed.

RESULTS

A total of 144 valid questionnaires were returned. Logistic regression analysis showed that relevant feedback in discussion classes (adjusted odds ratio [AOR], 5.071; p<0.001), frequency of e-portfolio feedback (AOR, 1.813; p=0.012), and motivation by e-portfolio feedback (AOR, 1.790; p=0.026) predicted a greater likelihood of understanding the continuity of the patient's medical experience. Relevant feedback from faculty members in discussion classes (AOR, 3.455; p<0.001) and frequency of e-portfolio feedback (AOR, 2.232; p<0.001) also predicted a greater likelihood of understanding the concept of patient-centered care.

CONCLUSION

Student-faculty interactions, including relevant feedback in discusstion classes, frequency of e-portfolio feedback, and motivation by e-portfolio feedback were found to be important factors in the LIC program.

A systematic review of effective quality feedback measurement tools used in clinical skills assessment

Background: Objective Structured Clinical Examination (OSCE) is a valid tool to assess the clinical skills of medical students. Feedback after OSCE is essential for student improvement and safe clinical practice. Many examiners do not provide helpful or insightful feedback in the text space provided after OSCE stations, which may adversely affect learning outcomes. The aim of this systematic review was to identify the best determinants for quality written feedback in the field of medicine.   Methods: PubMed, Medline, Embase, CINHAL, Scopus, and Web of Science were searched for relevant literature up to February 2021. We included studies that described the quality of good/effective feedback in clinical skills assessment in the field of medicine. Four independent reviewers extracted determinants used to assess the quality of written feedback. The percentage agreement and kappa coefficients were calculated for each determinant. The ROBINS-I (Risk Of Bias In Non-randomized Studies of Interventions) tool was used to assess the risk of bias. Results: 14 studies were included in this systematic review. 10 determinants were identified for assessing feedback. The determinants with the highest agreement among reviewers were specific, described gap, balanced, constructive and behavioural; with kappa values of 0.79, 0.45, 0.33, 0.33 and 0.26 respectively. All other determinants had low agreement (kappa values below 0.22) indicating that even though they have been used in the literature, they might not be applicable for good quality feedback. The risk of bias was low or moderate overall. Conclusions: This work suggests that good quality written feedback should be specific, balanced, and constructive in nature, and should describe the gap in student learning as well as observed behavioural actions in the exams.  Integrating these determinants in OSCE assessment will help guide and support educators for providing effective feedback for the learner.

Training to proficiency in surgery using simulation: is there a moral obligation?

A deontological approach to surgical ethics advocates that patients have the right to receive the best care that can be provided. The 'learning curve' in surgical skill is an observable and measurable phenomenon. Surgical training may therefore carry risk to patients. This can occur directly, through inadvertent harm, or indirectly through theatre inefficiency and associated costs. Trainee surgeon operating, however, is necessary from a utilitarian perspective, with potential risk balanced by the greater societal need to train future independent surgeons.New technology means that the surgical learning curve could take place, at least in part, outside of the operating theatre. Simulation-based deliberate practice could be used to obtain a predetermined level of proficiency in a safe environment, followed by simulation-based assessment of operative competence. Such an approach would require an overhaul of the current training paradigm and significant investment in simulator technology. This may increasingly be viewed as necessary in light of well-discussed pressures on surgical trainees and trainers.This article discusses the obligations to trainees, trainers and training bodies raised by simulation technology, and outlines the current arguments both against and in favour of a simulation-based training-to-proficiency model in surgery. The significant changes to the current training paradigm that would be required to implement such a model are also discussed.

The Assessment Rationale of Postgraduate Medical Trainees With Incongruent Self and Faculty Assigned Entrustment Scores

Background

Self-assessment is a central skill in competency-based medical education (CBME) and should be fostered in order to promote life-long learning. One measure that will guide the development of self-assessment is the alignment between it and external expert assessment. In this study, we explored the qualitative themes in the self-assessment rationale among trainees with incongruent self and faculty-assigned entrustment scores.

Methods

A total of 40 postgraduate medical trainees completed a four-scenario summative objective structured clinical examination (OSCE) as part of a simulation-based resuscitation curriculum in December 2017. After each scenario, an assessment involving an entrustment score and narrative rationale was completed by both trainee (self) and faculty. The differences between the trainee and faculty scores were calculated for each scenario and summed to give a single “incongruence score”. Trainees who consistently scored themselves higher than the faculty were said to have a “positive-incongruence score” and those scoring below the faculty were said to have a “negative-incongruence” score. Through this method, 10 trainees with the highest and lowest scores were assigned to each group and their narrative rationales were coded and thematically analyzed.

Results

The content of the self-assessment narrative rationale differed between the two groups. Trainees in the positive-incongruence group focused on the concepts of speed and situational management, while trainees in the negative-incongruence group commented on lack of support, and a need to improve communication, diagnosis, and code blue management. The quality of the self-assessment rationale also differed between groups. Trainees in the negative-incongruence group provided higher-quality comments that were more detailed and granular.

Conclusion

We found differences in the content and quality of the self-assessment rationale between trainees whose self and faculty-assigned assessment is incongruent. This provides insight into how these groups differ and has valuable implications for the development of curricula targeting self-assessment skills.

Peer review in team-based learning: influencing feedback literacy

BACKGROUND

Peer review in Team-based learning (TBL) exists for three key reasons: to promote reflection on individual behaviours; provide opportunities to develop professional skills; and prevent 'free riders' who fail to contribute effectively to team discussions. A well-developed process that engages students is needed. However, evidence suggests it remains a difficult task to effectively incorporate into TBL. The purpose of this study was to assess medical students' ability to provide written feedback to their peers in TBL, and to explore students' perception of the process, using the conceptual framework of Biggs '3P model'.

METHODS

Year 2 students (n = 255) participated in peer review twice during 2019. We evaluated the quality of feedback using a theoretically derived rubric, and undertook a qualitative analysis of focus group data to seek explanations for feedback behaviors.

RESULTS

Students demonstrated reasonable ability to provide positive feedback, but were less prepared to identify areas for improvement. Their ability did not improve over time, and was influenced by the perceived task difficulty; social discomfort; and sense of responsibility in providing written feedback.

CONCLUSIONS

To increase student engagement, we require a transparent process that incorporates verbal feedback and team discussion, with monitoring of outcomes by faculty and adequate training.

"What Next?": Toward telebehavioral health sustainability in couple and family therapy

As a response to the COVID-19 global crisis, many in the couple/marital and family therapy (CMFT) professional community quickly met the challenge of providing services to clients via telebehavioral health (TBH) services. As this public health emergency endures, family therapists must continue to engage in TBH practice professionally and ethically. The rapid adoption of TBH with minimal training and experience during this public health emergency can result in crises for both individual therapists as well as for the profession in implementing electronic record-keeping, conducting virtual sessions, and communicating online with various clinical populations. The risk of insufficient training and supervision create a challenge to new and experienced family therapists. This article summarizes the work done by the profession thus far to respond to this public health emergency and presents a roadmap of recommendations for navigating those challenges into the future and offers ideas about how to sustain quality TBH practice.

A Pilot Program Assessing Bronchoscopy Training and Program Initiation in a Low-income Country

BACKGROUND

Flexible bronchoscopy is an essential procedure for the evaluation and management of the pulmonary disease. However, this technology and related training is not available in many low-middle income countries (LMICs). We conducted a pilot training program for flexible bronchoscopy in Uganda.

METHODS

A multimodal curriculum was developed with pulmonologists from Uganda and the United States. The training included an online distance learning management system for video content, simulation, just-in-time training, and deliberate practice via clinical proctoring. Procedural standards and a de novo bronchoscopy suite were concurrently developed. Competency was assessed using the Bronchoscopic Skills and Tasks Assessment Tool written examination and the Ontario Bronchoscopy Assessment Tool.

RESULTS

We trained 3 pulmonary physicians with no prior experience in flexible bronchoscopy. Three bronchoscopies with bronchoalveolar lavage were performed during the training and an additional 11 cases were performed posttraining. All 3 Ugandan physicians had an increase in their written Bronchoscopic Skills and Tasks Assessment Tool and Ontario Bronchoscopy Assessment Tool in the competent range (P<0.05). All bronchoscopies were successfully completed, adequate samples were obtained, and there were no procedure-related complications.

CONCLUSION

Bronchoscopy implementation in LMICs is feasible, but requires competency-based training. Further studies are needed to validate this curriculum in LMICs, including the use of this type of curriculum for more complicated bronchoscopic procedures.

A paediatric logbook: Millstone or milestone?

Logbooks are ubiquitous in undergraduate and postgraduate medical education. Here, two alumni who are in the early phase of their career, reflect on their experience in their undergraduate paediatric term and how this was shaped by their mandatory paediatric logbook.