An international study on the implementation of programmatic assessment: Understanding challenges and exploring solutions

Programmatic assessment for learning: A programmatically designed assessment for the purpose of learning: AMEE Guide No. 174

Programmatic assessment for learning (PAL) involves programmatically structured collection of assessment data for the purpose of learning. In this guide, we examine and provide recommendations on several aspects: First, we review the evolution that has led to the development of programmatic assessment, providing clarification of some of its terminology. Second, we outline the learning processes that guide the design of PAL, including distributed learning, interleaving, overlearning, and test-enhanced learning. Third, we review the evolving nature of validity and provide insights into validity from a program perspective. Finally, we examine opportunities, challenges, and future directions of assessment in the context of artificial intelligence.

Beyond thinking fast and slow: Implications of a transtheoretical model of clinical reasoning and error on teaching, assessment, and research

From dual process to a family of theories known collectively as situativity, both micro and macro theories of cognition inform our current understanding of clinical reasoning (CR) and error. CR is a complex process that occurs in a complex environment, and a nuanced, expansive, integrated model of these theories is necessary to fully understand how CR is performed in the present day and in the future. In this perspective, we present these individual theories along with figures and descriptive cases for purposes of comparison before exploring the implications of a transtheoretical model of these theories for teaching, assessment, and research in CR and error.

Programmatic Assessment of Clinical Reasoning: New Opportunities to Meet an Ongoing Challenge

Issue: Clinical reasoning is essential to physicians' competence, yet assessment of clinical reasoning remains a significant challenge. Clinical reasoning is a complex, evolving, non-linear, context-driven, and content-specific construct which arguably cannot be assessed at one point in time or with a single method. This has posed challenges for educators for many decades, despite significant development of individual assessment methods. Evidence: Programmatic assessment is a systematic assessment approach that is gaining momentum across health professions education. Programmatic assessment, and in particular assessment for learning, is well-suited to address the challenges with clinical reasoning assessment. Several key principles of programmatic assessment are particularly well-aligned with developing a system to assess clinical reasoning: longitudinality, triangulation, use of a mix of assessment methods, proportionality, implementation of intermediate evaluations/reviews with faculty coaches, use of assessment for feedback, and increase in learners' agency. Repeated exposure and measurement are critical to develop a clinical reasoning assessment narrative, thus the assessment approach should optimally be longitudinal, providing multiple opportunities for growth and development. Triangulation provides a lens to assess the multidimensionality and contextuality of clinical reasoning and that of its different, yet related components, using a mix of different assessment methods. Proportionality ensures the richness of information on which to draw conclusions is commensurate with the stakes of the decision. Coaching facilitates the development of a feedback culture and allows to assess growth over time, while enhancing learners' agency. Implications: A programmatic assessment model of clinical reasoning that is developmentally oriented, optimizes learning though feedback and coaching, uses multiple assessment methods, and provides opportunity for meaningful triangulation of data can help address some of the challenges of clinical reasoning assessment.

Trainee-supervisor collaboration, progress-visualisation, and coaching: a survey on challenges in assessment of ICU trainees

BACKGROUND

Assessing trainees is crucial for development of their competence, yet it remains a challenging endeavour. Identifying contributing and influencing factors affecting this process is imperative for improvement.

METHODS

We surveyed residents, fellows, and intensivists working in an intensive care unit (ICU) at a large non-university hospital in Switzerland to investigate the challenges in assessing ICU trainees. Thematic analysis revealed three major themes.

RESULTS

Among 45 physicians, 37(82%) responded. The first theme identified is trainee-intensivist collaboration discontinuity. The limited duration of trainees' ICU rotations, large team size operating in a discordant three-shift system, and busy and unpredictable day-planning hinder sustained collaboration. Potential solutions include a concise pre-collaboration briefing, shared bedside care, and post-collaboration debriefing involving formative assessment and reflection on collaboration. The second theme is the lack of trainees' progress visualisation, which is caused by unsatisfactory familiarisation with the trainees' development. The lack of an overview of a trainee's previous achievements, activities, strengths, weaknesses, and goals may result in inappropriate assessments. Participants suggested implementing digital assessment tools, a competence committee, and dashboards to facilitate progress visualisation. The third theme we identified is insufficient coaching and feedback. Factors like personality traits, hierarchy, and competing interests can impede coaching, while high-quality feedback is essential for correct assessment. Skilled coaches can define short-term goals and may optimise trainee assessment by seeking feedback from multiple supervisors and assisting in both formative and summative assessment. Based on these three themes and the suggested solutions, we developed the acronym "ICU-STAR" representing a potentially powerful framework to enhance short-term trainee-supervisor collaboration in the workplace and to co-scaffold the principles of adequate assessment.

CONCLUSIONS

According to ICU physicians, trainee-supervisor collaboration discontinuity, the lack of visualisation of trainee's development, and insufficient coaching and feedback skills of supervisors are the major factors hampering trainees' assessment in the workplace. Based on suggestions by the survey participants, we propose the acronym "ICU-STAR" as a framework including briefing, shared bedside care, and debriefing of the trainee-supervisor collaboration at the workplace as its core components. With the attending intensivists acting as coaches, progress visualisation can be enhanced by actively collecting more data points.

TRIAL REGISTRATION

N/A.

A Program of Assessment Model for Point-of-Care Ultrasound Training for Pediatric Critical Care Providers: A Comprehensive Approach to Enhance Competency-Based Point-of-Care Ultrasound Training

Point-of-care ultrasound (POCUS) is increasingly accepted in pediatric critical care medicine as a tool for guiding the evaluation and treatment of patients. POCUS is a complex skill that requires user competency to ensure accuracy, reliability, and patient safety. A robust competency-based medical education (CBME) program ensures user competency and mitigates patient safety concerns. A programmatic assessment model provides a longitudinal, holistic, and multimodal approach to teaching, assessing, and evaluating learners. The authors propose a fit-for-purpose and modifiable CBME model that is adaptable for different institutions' resources and needs for any intended competency level. This educational model drives and supports learning, ensures competency attainment, and creates a clear pathway for POCUS education while enhancing patient care and safety.

"It Changed How I Think"-Impact of Programmatic Assessment Upon Practicing Physicians: A Qualitative Study

Programmatic assessment is a systematic approach used to document and assess learner performance. It offers learners frequent formative feedback from a variety of contexts and uses both high- and low-stakes assessments to determine student progress. Existing research has explored learner and faculty perceptions of programmatic assessment, reporting favorable impact on faculty understanding of the importance of assessment stakes and feedback to learners while students report the ability to establish and navigate towards goals and reflect on their performance. The Cleveland Clinic Lerner College of Medicine (CCLCM) of Case Western Reserve University adopted programmatic assessment methods at its inception. With more than 18 years' experience with programmatic assessment and a portfolio-based assessment system, CCLCM is well-positioned to explore its graduates' perceptions of their programmatic assessment experiences during and after medical school. In 2020, the investigators interviewed 26 of the 339 physician graduates. Participants were purposefully sampled to represent multiple class cohorts (2009-2019), clinical specialties, and practice locations. The investigators analyzed interview transcripts using thematic analysis informed by the frameworks of self-determination theory and professional identity formation. The authors identified themes and support each with participant quotes from the interviews. Based on findings, the investigators compiled a series of recommendations for other institutions who have already or plan to incorporate elements of programmatic assessment into their curricula. The authors concluded by discussing future directions for research and additional avenues of inquiry.

Examining the nexus between medical education and complexity: a systematic review to inform practice and research

BACKGROUND

Medical education is a multifarious endeavour integrating a range of pedagogies and philosophies. Complexity as a science or theory ('complexity') signals a move away from a reductionist paradigm to one which appreciates that interactions in multi-component systems, such as healthcare systems, can result in adaptive and emergent outcomes. This examination of the nexus between medical education and complexity theory aims to discover ways that complexity theory can inform medical education and medical education research.

METHODS

A structured literature review was conducted to examine the nexus between medical education and complexity; 5 databases were searched using relevant terms. Papers were included if they engaged fully with complexity as a science or theory and were significantly focused on medical education. All types of papers were included, including conceptual papers (e.g. opinion and theoretical discussions), case studies, program evaluations and empirical research. A narrative and thematic synthesis was undertaken to create a deep understanding of the use of complexity in medical education.

RESULTS

Eighty-three papers were included; the majority were conceptual papers. The context and theoretical underpinnings of complexity as a relevant theory for medical education were identified. Bibliographic and temporal observations were noted regarding the entry of complexity into medical education. Complexity was relied upon as a theoretical framework for empirical studies covering a variety of elements within medical education including: knowledge and learning theories; curricular, program and faculty development; program evaluation and medical education research; assessment and admissions; professionalism and leadership; and learning for systems, about systems and in systems.

DISCUSSION

There is a call for greater use of theory by medical educators. Complexity within medical education is established, although not widespread. Individualistic cultures of medicine and comfort with reductionist epistemologies challenges its introduction. However, complexity was found to be a useful theory across a range of areas by a limited number of authors and is increasingly used by medical educators and medical education researchers. This review has further conceptualized how complexity is being used to support medical education and medical education research.

CONCLUSION

This literature review can assist in understanding how complexity can be useful in medical educationalists' practice.

Teacher, Gatekeeper, or Team Member: supervisor positioning in programmatic assessment

Competency-based assessment is undergoing an evolution with the popularisation of programmatic assessment. Fundamental to programmatic assessment are the attributes and buy-in of the people participating in the system. Our previous research revealed unspoken, yet influential, cultural and relationship dynamics that interact with programmatic assessment to influence success. Pulling at this thread, we conducted secondary analysis of focus groups and interviews (n = 44 supervisors) using the critical lens of Positioning Theory to explore how workplace supervisors experienced and perceived their positioning within programmatic assessment. We found that supervisors positioned themselves in two of three ways. First, supervisors universally positioned themselves as a Teacher, describing an inherent duty to educate students. Enactment of this position was dichotomous, with some supervisors ascribing a passive and disempowered position onto students while others empowered students by cultivating an egalitarian teaching relationship. Second, two mutually exclusive positions were described-either Gatekeeper or Team Member. Supervisors positioning themselves as Gatekeepers had a duty to protect the community and were vigilant to the detection of inadequate student performance. Programmatic assessment challenged this positioning by reorientating supervisor rights and duties which diminished their perceived authority and led to frustration and resistance. In contrast, Team Members enacted a right to make a valuable contribution to programmatic assessment and felt liberated from the burden of assessment, enabling them to assent power shifts towards students and the university. Identifying supervisor positions revealed how programmatic assessment challenged traditional structures and ideologies, impeding success, and provides insights into supporting supervisors in programmatic assessment.