Person-task-context: a model for designing curriculum and in-training assessment in postgraduate education

Development of an electronic learning progression dashboard to monitor student clinical experiences

INTRODUCTION

Clinical experience tracking mechanisms for students at dental schools provide patient assignment, student experience, and learning progression feedback. The purpose of this study was to evaluate dental students' clinical experiences following the implementation of a learning progression dashboard (LPD).

METHODS

After developing and deploying an electronic LPD using PHP, secondary data analysis on dental students' clinical experiences from 2017-2019 was conducted. Student experience differences were compared between the year before continuous use of the LPD and the first year using it. LPD data contained the required clinical procedures dentistry students must perform across all disciplines and the number of planned, in progress, and completed tasks each student has accomplished. Using two time points, the students' experiences were compared. Univariate statistics and independent t-tests were conducted in R for detecting the differences in the number and categories of codes.

RESULTS

The number and category of codes showed significant differences between the academic year 2017-2018 and 2018-2019 for both third- and fourth-year dental students after one and two terms. Overall, students recorded a 26% greater number of treatment codes and experienced a 26% greater number of code categories compared to the previous year.

CONCLUSION

Applying information management methods such as dashboards can better inform educators on student clinical experiences and improve clinical learning outcomes for students.

Judging residents' performance: a qualitative study using grounded theory

BACKGROUND

Although program directors judge residents' performance for summative decisions, little is known about how they do this. This study examined what information program directors use and how they value this information in making a judgment of residents' performance and what residents think of this process.

METHODS

Sixteen semi-structured interviews were held with residents and program directors from different hospitals in the Netherlands in 2015-2016. Participants were recruited from internal medicine, surgery and radiology. Transcripts were analysed using grounded theory methodology. Concepts and themes were identified by iterative constant comparison.

RESULTS

When approaching semi-annual meetings with residents, program directors report primarily gathering information from the following: assessment tools, faculty members and from their own experience with residents. They put more value on faculty's comments during meetings and in the corridors than on feedback provided in the assessment tools. They are influenced by their own beliefs about learning and education in valuing feedback. Residents are aware that faculty members discuss their performance in meetings, but they believe the assessment tools provide the most important proof to demonstrate their clinical competency.

CONCLUSIONS

Residents think that feedback in the assessment tools is the most important proof to demonstrate their performance, whereas program directors scarcely use this feedback to form a judgment about residents' performance. They rely heavily on remarks of faculty in meetings instead. Therefore, residents' performance may be better judged in group meetings that are organised to enhance optimal information sharing and decision making about residents' performance.

Physicians' Practice of Dispensing Medicines: A Qualitative Study

OBJECTIVES

The physical act of giving medication to patients to administer away from a health care setting, dispensing, is normally performed by pharmacists. Dispensing of medication by physicians is a neglected patient safety issue, and having observed considerable variation in practice, the lead author sought to explore this issue further. A literature review yielded zero articles pertaining to this, so an exploratory study was commenced. The qualitative arm, relating to junior physicians' experience of, and training in, dispensing, is reported here.

METHODS

Focus groups were conducted to explore the beliefs, ideas, and experiences of physicians-in-training pertaining to dispensing of medication. These were recorded and transcribed. The transcriptions were thematically analyzed using the grounded theory.

RESULTS

The emergency department was the most common site of dispensing. No formal training in dispensing had been received. Informal training was variable in content and utility. The physicians felt that dispensing was part of their role.

CONCLUSIONS

Despite being expected to dispense, and the patient safety issues involved in giving drugs to patients to use at home, physicians do not feel that they have been trained to undertake this task. These findings from 1 hospital raise questions about the wider quality and safety of this practice.

Programmatic assessment of competency-based workplace learning: when theory meets practice

BACKGROUND

In competency-based medical education emphasis has shifted towards outcomes, capabilities, and learner-centeredness. Together with a focus on sustained evidence of professional competence this calls for new methods of teaching and assessment. Recently, medical educators advocated the use of a holistic, programmatic approach towards assessment. Besides maximum facilitation of learning it should improve the validity and reliability of measurements and documentation of competence development. We explored how, in a competency-based curriculum, current theories on programmatic assessment interacted with educational practice.

METHODS

In a development study including evaluation, we investigated the implementation of a theory-based programme of assessment. Between April 2011 and May 2012 quantitative evaluation data were collected and used to guide group interviews that explored the experiences of students and clinical supervisors with the assessment programme. We coded the transcripts and emerging topics were organised into a list of lessons learned.

RESULTS

The programme mainly focuses on the integration of learning and assessment by motivating and supporting students to seek and accumulate feedback. The assessment instruments were aligned to cover predefined competencies to enable aggregation of information in a structured and meaningful way. Assessments that were designed as formative learning experiences were increasingly perceived as summative by students. Peer feedback was experienced as a valuable method for formative feedback. Social interaction and external guidance seemed to be of crucial importance to scaffold self-directed learning. Aggregating data from individual assessments into a holistic portfolio judgement required expertise and extensive training and supervision of judges.

CONCLUSIONS

A programme of assessment with low-stakes assessments providing simultaneously formative feedback and input for summative decisions proved not easy to implement. Careful preparation and guidance of the implementation process was crucial. Assessment for learning requires meaningful feedback with each assessment. Special attention should be paid to the quality of feedback at individual assessment moments. Comprehensive attention for faculty development and training for students is essential for the successful implementation of an assessment programme.

Self-organisation, integration and curriculum in the complex world of medical education

CONTEXT

The world of medical education is more complex than ever and there seems to be no end in sight. Complexity science is particularly relevant as medical education embraces a movement towards more authentic curricula focusing on integration, interactive small-group learning, and early and sustained clinical and community experiences.

DISCUSSION

A medical school as a whole, and the expression of its curriculum through the interactions, exchanges and learning that take place within and outside of it, is a complex system. Complexity science, a derivative of the natural sciences, is the study of the dynamics, conditions and consequences of interactions. It addresses the nature of the conditions favourable to change and transformation (learning).

CONCLUSIONS

The core process of complexity, self-organisation, requires a system that is open and far from equilibrium, with ill-defined boundaries and a large number of non-linear interactions involving short-loop feedback. In such a system, knowledge does not exist objectively 'out there'; rather, it exists as a result of the exchange between participants, an action that becomes knowing. Understanding is placed between participants rather than being contained in one or the other. Knowledge is not constructed separately in the mind of the knower, but, rather, it emerges; it is co-created during the exchange in an authentic recursive transactive process. Learning and knowing become adaptive responses to continuously evolving circumstances. An approach to curriculum based on self-organisation is characterised as rich, recursive, relational and rigorous and it illuminates how a curriculum can be understood as a complex adaptive system. The perspective of complexity applied to medical education broadens and enriches research questions relevant to health professions education. It focuses our attention onto how we are together as human beings. How we respond to and frame the issues of learning and understanding that challenge contemporary medicine and, by extension, medical education, in a complex and rapidly changing world can have profound effects on the preparedness of tomorrow's health professionals and their impact on society.

Validation and implementation of surgical simulators: a critical review of present, past, and future

BACKGROUND

In the past 20 years the surgical simulator market has seen substantial growth. Simulators are useful for teaching surgical skills effectively and with minimal harm and discomfort to patients. Before a simulator can be integrated into an educational program, it is recommended that its validity be determined. This study aims to provide a critical review of the literature and the main experiences and efforts relating to the validation of simulators during the last two decades.

METHODS

Subjective and objective validity studies between 1980 and 2008 were identified by searches in Pubmed, Cochrane, and Web of Science.

RESULTS

Although several papers have described definitions of various subjective types of validity, the literature does not offer any general guidelines concerning methods, settings, and data interpretation. Objective validation studies on endourological simulators were mainly characterized by a large variety of methods and parameters used to assess validity and in the definition and identification of expert and novice levels of performance.

CONCLUSION

Validity research is hampered by a paucity of widely accepted definitions and measurement methods of validity. It would be helpful to those considering the use of simulators in training programs if there were consensus on guidelines for validating surgical simulators and the development of training programs. Before undertaking a study to validate a simulator, researchers would be well advised to conduct a training needs analysis (TNA) to evaluate the existing need for training and to determine program requirements in a training program design (TPD), methods that are also used by designers of military simulation programs. Development and validation of training models should be based on a multidisciplinary approach involving specialists (teachers), residents (learners), educationalists (teaching the teachers), and industrial designers (providers of teaching facilities). In addition to technical skills, attention should be paid to contextual, interpersonal, and task-related factors.

What training is needed in the fourth year of medical school? Views of residency program directors

PURPOSE

To identify common struggles of interns, determine residency program directors' (PDs') views of the competencies to be gained in the fourth year of medical school, and apply this information to formulate goals of curricular reform and student advising.

METHOD

In 2007, semistructured interviews were conducted with 30 PDs in the 10 most common specialty choices of students at the University of California, San Francisco, School of Medicine to assess the PDs' priorities for knowledge, skills, and attitudes to be acquired in the fourth year. Interviews were coded to identify major themes.

RESULTS

Common struggles of interns were lack of self-reflection and improvement, poor organizational skills, underdeveloped professionalism, and lack of medical knowledge. The Accreditation Council for Graduate Medical Education competencies of patient care, practice-based learning and improvement, interpersonal and communication skills, and professionalism were deemed fundamental to fourth-year students' development. Rotations recommended across specialties were a subinternship in a student's future field and in internal medicine (IM), rotations in an IM subspecialty, critical care, and emergency and ambulatory medicine. PDs encouraged minimizing additional time spent in the student's future field. Suggested coursework included an intensively coached transitional subinternship and courses to improve students' medical knowledge.

CONCLUSIONS

PDs deemed the fourth year to have a critical role in the curriculum. There was consensus about expected fourth-year competencies and the common clinical experiences that best prepare students for residency training. These findings support using the fourth year to transition students to graduate medical training and highlight areas for curricular innovation.

Using participatory design to develop structured training in child and adolescent psychiatry

CONTEXT

Learning during residency in child and adolescent psychiatry (CAP) is primarily work-based and has traditionally been opportunistic. There are increasing demands from both postgraduate trainees and medical organisations for structured programmes with defined learning outcomes.

OBJECTIVES

The aim of this study was to partner with postgraduate trainees and consultants in psychiatry to identify key learning issues that should be considered during CAP residency and to use these in designing a structured programme to meet the learning outcome requirements of a competency framework.

METHODS

Participatory design was used to structure a learning and assessment programme in CAP. First, during working seminars, consultants and postgraduate trainees were interviewed about the characteristics of the learning and working in CAP. These interviews were audio taped, transcribed and analyzed for recurrent themes to identify key issues. Descriptive results were fed back to the participants for validation. In a subsequent iterative process the researchers and practitioners partnered to construct a learning and assessment programme.

RESULTS

The tasks within CAP were poorly described by study participants. Several other types of professionals within the healthcare team perform many of the tasks a CAP postgraduate trainee has to learn. Participants had difficulties describing how learning takes place and what postgraduate trainees need to learn in CAP. The partnership between researchers and practitioners identified three key issues to consider in CAP residencies: (1) Preparation for tasks postgraduate trainees are expected to fulfil, (2) Ensuring acquisition of physician-specific knowledge and skills, and (3) Clarifying roles and professional identity within the team. A structured training programme incorporating the key learning issues identified was created.

CONCLUSION

Participatory design was very helpful to structure a contextually suitable training programme in CAP. The researchers speculate that this approach will result in easier implementation of the new training programme.

Primary prevention of violence against women: training needs of violence practitioners

Practitioners in domestic violence and sexual assault programs have been encouraged by the National Center for Injury Prevention and Control to enhance their activities in violence prevention; however, many practitioners have not been trained in prevention concepts and strategies. Therefore, a needs assessment was undertaken with practitioners in the Domestic Violence Prevention Enhancement and Leadership Through Alliances and the Rape Prevention and Education programs to determine training needs. Results show that practitioners are very interested in primary prevention. They want to learn about working at the community level (rather than the individual level), developing and evaluating prevention activities, and identifying effective primary prevention programs.

Are some of the challenging aspects of the CanMEDS roles valid outside Canada?

CONTEXT

Many countries have adopted the CanMEDS roles. However, there is limited information on how these apply in an international context and in different specialties.

OBJECTIVES

To survey trainee and specialist ratings of the importance of the CanMEDS roles and perceived ability to perform tasks within the roles.

METHODS

We surveyed 8749 doctors within a defined region (eastern Denmark) via a single-issue, mailed questionnaire. Each of the 7 roles was represented by 3 questionnaire items to be rated for perceived importance and confidence in ability to perform the role.

RESULTS

Responses were received from 3476 doctors (42.8%), including 190 interns, 201 doctors in the introductory year of specialist training, 529 residents and 2152 specialists. The overall mean rating of importance (on a scale of 1-5) of the aspects of competence described in the CanMEDS roles was 4.2 (standard deviation 0.6) and did not differ between trainee groups and specialists. Mean ratings of confidence were lower than ratings of importance and increased across the groups from interns to specialists. Differences between specialty groups were evident in both importance and confidence for many of the roles. For laboratory, technical and, to a lesser extent, cognitive specialties, the role of Health Advocate scored the lowest in importance. For general medicine specialties, the roles of Medical Expert, Collaborator, Manager and Scholar all scored lower for importance and confidence.

CONCLUSIONS

This study provides a sketch of the content and construct validity of the CanMEDS roles in a non-Canadian setting. More research is needed in how these aspects of competence can be best taught and applied across specialties in different jurisdictions.