Assessment Practices for Dietetics Trainees: A Systematic Review

Assessment in community settings: Is there any role for individual competency-based assessment?

BACKGROUND

Oral forms of assessment remain a common part of competency-based assessment systems, yet their feasibility is being challenged. The focus on individual competence is problematic given the need to prepare health professionals for teamwork. The present study aimed to investigate how the assessment of collective competence compares to individual assessment, and to explore whether there is a need for individual assessment at all in community or population-based practice.

METHODS

A behavioural-based interview assessment was developed and trialled and correlated to performance as a team on placement. Correlation between student performance on individual behavioural-based interview and teamwork artefacts was assessed using Spearman rho. Differences between performance on individual oral assessment and team performance on artefacts at the cohort level was determined using the Mann-Whitney U-test. Bland-Altman analysis was completed to analyse agreement between performance on the individual oral assessment and team performance at the student level.

RESULTS

Students were final year nutrition and dietetics students from 2020, 2021 and 2022 years (total sample = 216) from one Australian university. There was a difference in performance between assessment types in each year and as a total cohort (p < 0.001) with students performing better in teamwork. There was no correlation between individual oral interview and team performance across all years. The results of the Bland-Altman analysis showed little agreement between the two assessment tasks.

CONCLUSIONS

Assessment of teamwork performance as part of a program of assessment is essential. More valid and reliable tools are needed to assess collective competence.

An exploration of Irish nutrition educators' experiences of competency-based assessment in nutrition science education

BACKGROUND

Competency-based assessment (CBA) supports the development and attainment of skills required for the workforce. Little is known about educators' experience in developing or implementing CBA in nutrition science education or their opinions on how well it captures a student's preparedness for the workforce. The objective of this study was to explore educators' experience of CBA in nutrition education in Ireland.

METHOD

Grounded in interpretivism, in-depth, semi-structured, audio-recorded interviews were conducted with 13 educators from five of the ten undergraduate honours degree nutrition programmes across Ireland. Interviews explored experiences of CBA and perception of students training to prepare for the workforce. A reflexive thematic analysis approach was implemented whereby the data were transcribed, inductively coded, and themes identified.

RESULTS

A clear divide was evident between participants who were confident in their understanding of CBA and those who were unsure or had no knowledge of the term. Those with a clear understanding were more involved in programme development and evaluation. Three themes were identified: 'Assessment process' including intended learning outcomes, assessment design, and grading systems, 'Student-centred approach to assessment' focusing on work-based assessment and preparation for the workforce, and 'Upskilling educators' to equip educators with the skills and knowledge for professional development and to foster student success.

CONCLUSION

The importance of CBA as a student-centred approach, supporting them to meet standards expected to practice as a nutrition professional, was the key experience of educators. Training in CBA and inclusion of more authentic assessment may better prepare students for the workforce.

Oral interview in place of traditional objective structured clinical examinations for assessing placement readiness in nutrition and dietetics education

AIM

Objective structured clinical examinations have long been used in dietetics education. This observational study aims to describe the development, deployment, feasibility and validity of assessment using an oral interview in place of traditional objective structured clinical examinations, and to determine the ability of this assessment to identify students who are either not ready for placement or may require early support and/or remediation.

METHODS

Student assessment data were collected over a two-and-a-half-year period and used to test the predictive ability of an oral interview to determine dietetic placement outcomes and highlight a need for early remediation. Descriptive statistics as well as a between-group one-way ANOVA was used to describe results.

RESULTS

A total of 169 students participated in the oral interview and subsequent medical nutrition therapy placement over the study period. Significant differences in oral interview score were seen between students who passed placement and students who passed with remediation or those who failed. Oral interview performance was able to predict placement outcome, yet required less resources than traditional objective structured clinical examinations.

CONCLUSION

An oral interview may provide the same utility as the objective structured clinical examination in dietetics education .

Validity and Educational Impact of Clinical Entrustable Professional Activities Linked to the Nutrition Care Process for Work-Based Assessment of Entry-Level Dietetics Students: Evaluation of a 3-Year Implementation Study in Australia

BACKGROUND

Entrustable professional activities (EPAs) describe key workplace activities and are increasingly being used for student work-based assessment. An EPA-based work-based assessment tool offers potential to increase understanding of dietetics student skill development and opportunity for standardised work-based student assessment.

OBJECTIVE

To determine construct validity and educational impact of an EPA work-based assessment tool for dietetics placement students in clinical settings.

DESIGN

Using a time series design, supervisor and student self-assessment data collected from an EPA-based assessment tool from three cohorts of dietetics students and supervisors from 2017-2019 was analyzed.

PARTICIPANTS AND SETTING

Dietetics students (n = 145) from an accredited dietetic training program in Australia and affiliated metropolitan and rural hospitals.

MAIN OUTCOME MEASURES

Construct validity was determined through analysis of supervisor evaluation of student performance against EPAs over time. Educational impact was determined through comparing student performance across EPAs to predicted milestones to identify areas of least entrustment.

STATISTICAL ANALYSES PERFORMED

Student performance over time and differences between student self-assessment and supervisor assessment for each EPA were investigated using Wilcoxon signed-rank tests and linear mixed-model analysis. Descriptive statistics summarised student performance against each EPA.

RESULTS

Performance significantly increased over time in 35 out of 37 EPAs. Significant differences between supervisor and student self-assessment were evident in 9 out of 37 EPAs. Dietetics student performance varied across EPAs with 88.2% of students achieving entrustment for nutrition management EPAs compared with 100% for professionalism EPAs.

CONCLUSIONS

The tool's construct validity was established. EPAs identified areas of Nutrition Care Process skills development requiring additional support. Data collected by the tool can inform teaching.

Shifting the narrative and practice of assessing professionalism in dietetics education: An Australasian qualitative study

AIM

We aimed to explore current approaches to assessing professionalism in dietetics education in Australia and New Zealand, and asked the questions what is working well and what needs to improve?

METHOD

We employed a qualitative interpretive approach and conducted interviews with academic and practitioner (workplace-based) educators (total sample n = 78) with a key stake in dietetics education across Australia and New Zealand. Data were analysed using team-based, framework analysis.

RESULTS

Our findings suggest significant shifts in dietetics education in the area of professionalism assessment. Professionalism assessment is embedded in formal curricula of dietetics programs and is occurring in university and placement settings. In particular, advances have been demonstrated in those programs assessing professionalism as part of the programmatic assessment. Progress has been enabled by philosophical and curricula shifts; clearer articulation and shared understandings of professionalism standards; enhanced learner agency and reduced power distance; early identification and intervention of professionalism lapses; and increased confidence and capabilities of educators.

CONCLUSIONS

These findings suggest there have been considerable advances in professionalism assessment in recent years with shifts in practice in approaching professionalism through a more interpretivist lens, holistically and more student-centred. Professionalism assessment in dietetics education is a shared responsibility and requires further development and transformation to more fully embed and strengthen curricula approaches across programs. Further work should investigate strategies to build safer learning cultures and capacity for professionalism conversations and in strengthening approaches to remediation.

Development and evaluation of Entrustable Professional Activities embedded in an e-portfolio for work-based assessment in community and public health dietetics

Entrustable professional activities (EPAs) is a recent concept that helps operationalise competency-based education. In this paper, we report on the development of EPAs and incorporation into an e-portfolio for work-based assessment of final year dietetics students in a community and public health professional placement. We aimed to determine the utility of the EPAs for work-based assessment in this context. We used action research methodology to conduct broad consultation with students and supervisors using anonymous surveys and focus groups to understand their beliefs and attitudes towards work-based assessment. Consequently, 40 EPAs with an accompanying four-point entrustment scale were developed and mapped to the National Competency Standards for Dietitians in Australia. The EPAs and assessment tool were piloted and evaluated via an anonymous online user survey across three cohorts of students (n = 133) and supervisors (n = 67). Following mediocre pilot year evaluation results, EPAs were revised and reduced in number. While students positively evaluated the e-portfolio, their appraisal of the EPAs and entrustment scale was less positive compared to supervisors. Supervisor evaluation of the EPA-based e-portfolio supported the validity, feasibility and acceptability of this novel assessment method in a community and public health setting. Assessment using EPAs, and the resulting educational data collected by the tool, offers potential for individual learners to identify areas needing development during placement, as well as potential to inform curriculum improvements and increase understanding of learning opportunities and outcomes for dietetic students in community and public health settings.

Design, implementation and evaluation of novel work-based clinical assessment tool: An e-portfolio with embedded Entrustable Professional Activities

BACKGROUND

A new concept in work-based assessment, Entrustable Professional Activities (EPAs) describe key activities a work-ready practitioner can perform independently. EPAs are widely used in medicine to support competency-based education and are gaining acceptance in other professions such as nursing, pharmacy and dietetics. There is currently no widely accepted work-based assessment tool for student clinical placement assessment in dietetics and EPAs offer potential to address this gap.

OBJECTIVES

To design, implement and evaluate a work-based assessment tool using EPAs embedded in an e-portfolio for use in clinical dietetics.

SETTING

An accredited dietetic training program in Australian university and affiliated metropolitan and rural hospitals.

PARTICIPANTS

Three consecutive cohorts of final year dietetic students (n = 126) and their professional placement supervisors (n = 101).

METHODS

A working party consisting of dietetic academics with experience in clinical dietetics and education developed clinical dietetic EPAs and milestones that were mapped to the National Competency Standards for Dietitians in Australia. A design model incorporating multiple rounds of user feedback was used to create an e-portfolio with the EPAs embedded. Students and supervisors were invited to complete anonymous on-line surveys with Likert responses (1 = highly dissatisfied, 5 = highly satisfied) to evaluate the face validity, acceptability and feasibility of the tool.

RESULTS

A total of 37 EPAs were developed with an accompanying four-point entrustment scale to measure performance. Evaluation survey response rates were high and ranged from 35 to 45% for students and 57-83% for supervisors across the study period. Both students and supervisors evaluated the tool positively with a mean overall satisfaction of 4.12 ± 0.69 and 4.03 ± 0.68, respectively. Survey items evaluating face validity, feasibility and acceptability scored >3.75 for both groups of users.

CONCLUSIONS

The EPA based e-portfolio demonstrated face validity, feasibility and acceptability and offers potential for a widely accepted tool for work-based assessment of clinical dietetic students.

Competency-based assessment in nutrition education: A systematic literature review

Background

A suitably prepared and qualified nutrition and dietetics workforce is part of the solution to combating the burden of disease. Competency‐based assessment is a key part of the education of future workforces. Although there has been recent attention on competency‐based assessment in dietetics, there is little exploration of competency‐based education for the preparation of nutritionists. The present study aimed to understand how competency‐based assessment is implemented and evaluated in nutrition education.

Methods

A systematic literature review was carried out according to PRISMA guidelines. Four databases were initially searched in February 2020 using key words related to competenc* in combination with nutrition or dietetic and their synonyms. An updated search was completed again in March 2021. Studies that met eligibility criteria where the focus was on nutrition and involved a method of competency‐based assessment were synthesised narratively.

Results

From a total of 6262 titles and abstracts, six studies on competency assessment in nutrition education were identified. The assessments focused on the development of key skills, including motivational interviewing and nutrition assessment, changes to knowledge and attitudes on food and culture, and self‐perceived development of communication, collaboration, management, advocacy, scholarship and professional capabilities. No studies were found that assessed promotion of health and wellbeing or the food chain competencies.

Conclusions

The lack of research in competency‐based assessment must be addressed to ensure we are effectively preparing future nutritionists for work such that they can impact health outcomes.

The Value of Programmatic Assessment in Supporting Educators and Students to Succeed: A Qualitative Evaluation

BACKGROUND

Programmatic assessment has been proposed as the way forward for competency-based assessment, yet there is a dearth of literature describing the implementation and evaluation of programmatic assessment approaches.

OBJECTIVE

To evaluate the implementation of a programmatic assessment and explore its ability to support students and assessors.

DESIGN

A qualitative evaluation of programmatic assessment was employed.

PARTICIPANTS/SETTING

Interviews with graduates (n = 8) and preceptors (n = 12) together with focus groups with faculty assessors (n = 9) from the one Australian university explored experiences of the programmatic approach, role of assessment in learning, and defensibility of assessment decisions in determining competence.

ANALYSIS PERFORMED

Data were analyzed into key themes using framework analysis.

RESULTS

The programmatic assessment increased confidence in defensibility of assessment decisions, reduced emotional burden of assessment, increased value of assessment, and identified and remediated at-risk students earlier when philosophical and practice shifts in approaches to assessment were embraced.

CONCLUSIONS

Programmatic assessment supports a holistic approach to competency development and assessment and has multiple benefits for learners and assessors.

A self-determination theory analysis of reflective debrief themes about dietetic student placement experiences in hospital: implications for education

BACKGROUND

Healthcare placements in dietetics education contribute significantly to student learning. Exploring students' self-conceptualisation of placement experiences may provide insights to better support learning. Self-determination theory (SDT) has been used to seek insight into clinical and educational settings but has not yet been applied to dietetic placement learning. The present study investigated dietetics students' reflections of key influences on placement learning experiences and their alignment with an SDT framework.

METHODS

A post-placement two-stage critical incident debrief was conducted with seven successive cohorts (168 students) of dietetic undergraduate students on final placement. In debriefs, students' anonymous themes were collected and discussed, inductively analysed, and then mapped against an SDT framework of psychological and motivational constructs.

RESULTS

Nine key themes were identified that impacted upon placement experiences. Four themes related to framework constructs: (1) Supervisor (and Peer) Autonomy Support; (2) Perceived Competence; (3) Relatedness; and (4) Autonomy and Intrinsic Motivation. Non-SDT themes were also present, including: (5) Learning Environment and Experience; as well as themes about professional behaviours and identity: (6) Teamwork and Interactions; (7) Managing Emotions and Self-Care; (8) Dietetic Communications and Behaviours; and (9) Developing a Professional Identity.

CONCLUSIONS

Embedding a structured debrief in the curriculum and using a psychological motivational SDT framework to analyse themes arising can provide valuable information about the learning needs of students on placement with potential for wider application in dietetic learning and teaching and workforce employability. The current findings may have application in university curricula before and after professional placement.