1
|
Gupta SK, Srivastava T. Assessment in Undergraduate Competency-Based Medical Education: A Systematic Review. Cureus 2024; 16:e58073. [PMID: 38738047 PMCID: PMC11088485 DOI: 10.7759/cureus.58073] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/11/2024] [Indexed: 05/14/2024] Open
Abstract
BACKGROUND Studies that have methodically compiled the body of research on the competency-based medical education (CBME) assessment procedure and pinpointed knowledge gaps about the structure of the assessment process are few. Thus, the goals of the study were to create a model assessment framework for competency-based medical education that would be applicable in the Indian setting as well as to thoroughly examine the competency-based medical education assessment framework. METHODS PubMed, MEDLINE (Ovid), EMBASE (Ovid), Scopus, Web of Science, and Google Scholar were the databases that were searched. The search parameters were restricted to English language publications about competency-based education and assessment methods, which were published between January 2006 and December 2020. A descriptive overview of the included research (in tabular form) served as the foundation for the data synthesis. RESULTS Databases provided 732 records; out of which 36 fulfilled the inclusion and exclusion criteria. Thirty-six studies comprised a mix of randomized controlled trials, focus group interviews, and questionnaire studies, including cross-sectional studies, qualitative studies (03), mixed-method studies, etc. The papers were published in 10 different journals. The greatest number was published in BMC Medical Education (18). The average quality score for included studies was 62.53% (range: 35.71-83.33%). Most authors are from the UK (07), followed by the USA (05). The included studies were grouped into seven categories based on their dominant focus: moving away from a behavioristic approach to a constructive approach of assessment (01 studies), formative assessment (FA) and feedback (10 studies), the hurdles in the implementation of feedback (04 studies), utilization of computer or online based formative test with automated feedback (05 studies), video feedback (02 studies), e-learning platforms for formative assessment (04 studies), studies related to workplace-based assessment (WBA)/mini-clinical evaluation exercise (mini-CEX)/direct observation of procedural skills (DOPS) (10 studies). CONCLUSIONS Various constructivist techniques, such as concept maps, portfolios, and rubrics, can be used for assessments. Self-regulated learning, peer feedback, online formative assessment, an online computer-based formative test with automated feedback, the use of a computerized web-based objective structured clinical examination (OSCE) evaluation system, and the use of narrative feedback instead of numerical scores in mini-CEX are all ways to increase student involvement in the design and implementation of the formative assessment.
Collapse
Affiliation(s)
- Sandeep K Gupta
- Pharmacology, Heritage Institute of Medical Sciences, Varanasi, IND
| | | |
Collapse
|
2
|
Morbitzer KA, McLaughlin JE, Eckel SF, Shepherd G, Poupart N, Cox WC. An Exploration of the Relationship Between Pharmacy School Admission Variables and PGY1 Pharmacy Residency Program Match. AMERICAN JOURNAL OF PHARMACEUTICAL EDUCATION 2023; 87:100097. [PMID: 37380260 DOI: 10.1016/j.ajpe.2023.100097] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/05/2022] [Revised: 01/22/2023] [Accepted: 03/13/2023] [Indexed: 06/30/2023]
Abstract
OBJECTIVE To explore the relationship between pharmacy school admission variables and matching to a postgraduate year 1 (PGY1) pharmacy residency program. METHODS Demographic data, academic indicators, and application review scores were collected for the 2017-2020 Doctor of Pharmacy (PharmD) graduating classes. Multiple mini-interview (MMI) scores were collected for the 2018-2020 PharmD graduating classes. Postgraduate year 1 matching data were collected for all students. Bivariate analyses were performed comparing students who matched to a PGY1 residency versus those who did not match versus those who did not pursue a residency. Logistic regression modeling was performed to explore predictors of matching to a PGY1 residency program. RESULTS A total of 616 students were included. Bivariate analyses revealed that students who matched to a PGY1 had a higher undergraduate grade point average, higher pharmacy college admissions test composite score, were younger in age, and were more likely to identify as female. Students who matched also scored higher on MMI stations with constructs related to integrity, adaptability, critical thinking, and why pursuing our school. Logistic regression modeling found that an increase in age was associated with lower odds of matching to a PGY1 (0.88 [0.78-0.99]) and an increase in composite MMI station score was associated with higher odds of matching (1.8 [1.31-2.47]). CONCLUSION Several pharmacy school admission variables were found to be associated with successful matching to a PGY1 residency. These findings have the potential for impact at a programmatic level when evaluating the weight of certain criteria for admission decisions and at the individual student level when providing career services support.
Collapse
Affiliation(s)
- Kathryn A Morbitzer
- University of North Carolina at Chapel Hill (UNC) Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA; Center for Innovative Pharmacy Education and Research, UNC Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
| | - Jacqueline E McLaughlin
- University of North Carolina at Chapel Hill (UNC) Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA; Center for Innovative Pharmacy Education and Research, UNC Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Stephen F Eckel
- University of North Carolina at Chapel Hill (UNC) Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Greene Shepherd
- University of North Carolina at Chapel Hill (UNC) Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | | | - Wendy C Cox
- University of North Carolina at Chapel Hill (UNC) Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| |
Collapse
|
3
|
Wolcott MD, Olsen AA, Augustine JM. Item response theory in high-stakes pharmacy assessments. CURRENTS IN PHARMACY TEACHING & LEARNING 2022; 14:1206-1214. [PMID: 36154966 DOI: 10.1016/j.cptl.2022.07.023] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/08/2021] [Revised: 05/31/2022] [Accepted: 07/20/2022] [Indexed: 06/16/2023]
Abstract
OUR SITUATION Classical test theory (CTT) and item response theory (IRT) are two measurement models used to evaluate results from examinations, questionnaires, and instruments. To illustrate the benefits of IRT, we compared how results from multiple-choice tests can be interpreted using CTT and IRT. METHODOLOGICAL LITERATURE REVIEW IRT encompasses a collection of statistical models that estimate the probability of providing a correct response for a test item. The models are non-linear and generate item characteristic curves that illustrate the relationship between the examinee's ability level and whether they answered the item correctly. Several models can be used to estimate parameters such as item difficulty, discrimination, and guessing. In addition, IRT can generate item and test information functions to illustrate the accuracy of ability estimates. OUR RECOMMENDATIONS AND THEIR APPLICATIONS Researchers interested in IRT should gather the necessary resources early in the research process and collaborate with those experienced in quantitative and advanced statistical models. Researchers should confirm IRT is the optimal choice and select the model ideal for their needs. Once data are acquired, confirm model assumptions are met and model fit is appropriate. Lastly, researchers should consider disseminating the findings with accompanying visuals. POTENTIAL IMPACT IRT can be a valuable approach in assessment design and evaluation. Potential opportunities include supporting the design of computer adaptive tests, creating equivalent test forms that evaluate a range of examinee abilities, and evaluating whether items perform differently for examinee sub-groups. Further, IRT can have noteworthy visuals such as test information and functions.
Collapse
Affiliation(s)
- Michael D Wolcott
- HPU Workman School of Dental Medicine, High Point University, One University Parkway, High Point, NC 27268, 336-841-7344, United States; UNC Eshelman School of Pharmacy, The University of North Carolina at Chapel Hill, 301 Pharmacy Lane - Beard Hall 321, Chapel Hill, NC 27599, 919-451-3547, United States.
| | - Amanda A Olsen
- College of Education, University of Texas at Arlington, 701 Planetarium Place, Arlington, TX 76019, 817-272-5641, United States.
| | - Jill M Augustine
- College of Pharmacy, Mercer University, 3001 Mercer University Drive, Atlanta, GA 30341, 678-547-6738, United States.
| |
Collapse
|
4
|
Johnson CE, Keating JL, Leech M, Congdon P, Kent F, Farlie MK, Molloy EK. Development of the Feedback Quality Instrument: a guide for health professional educators in fostering learner-centred discussions. BMC MEDICAL EDUCATION 2021; 21:382. [PMID: 34253221 PMCID: PMC8276464 DOI: 10.1186/s12909-021-02722-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/27/2021] [Accepted: 05/06/2021] [Indexed: 06/13/2023]
Abstract
BACKGROUND Face-to-face feedback plays an important role in health professionals' workplace learning. The literature describes guiding principles regarding effective feedback but it is not clear how to enact these. We aimed to create a Feedback Quality Instrument (FQI), underpinned by a social constructivist perspective, to assist educators in collaborating with learners to support learner-centred feedback interactions. In earlier research, we developed a set of observable educator behaviours designed to promote beneficial learner outcomes, supported by published research and expert consensus. This research focused on analysing and refining this provisional instrument, to create the FQI ready-to-use. METHODS We collected videos of authentic face-to-face feedback discussions, involving educators (senior clinicians) and learners (clinicians or students), during routine clinical practice across a major metropolitan hospital network. Quantitative and qualitative analyses of the video data were used to refine the provisional instrument. Raters administered the provisional instrument to systematically analyse educators' feedback practice seen in the videos. This enabled usability testing and resulted in ratings data for psychometric analysis involving multifaceted Rasch model analysis and exploratory factor analysis. Parallel qualitative research of the video transcripts focused on two under-researched areas, psychological safety and evaluative judgement, to provide practical insights for item refinement. The provisional instrument was revised, using an iterative process, incorporating findings from usability testing, psychometric testing and parallel qualitative research and foundational research. RESULTS Thirty-six videos involved diverse health professionals across medicine, nursing and physiotherapy. Administering the provisional instrument generated 174 data sets. Following refinements, the FQI contained 25 items, clustered into five domains characterising core concepts underpinning quality feedback: set the scene, analyse performance, plan improvements, foster learner agency, and foster psychological safety. CONCLUSIONS The FQI describes practical, empirically-informed ways for educators to foster quality, learner-centred feedback discussions. The explicit descriptions offer guidance for educators and provide a foundation for the systematic analysis of the influence of specific educator behaviours on learner outcomes.
Collapse
Affiliation(s)
- Christina E Johnson
- Department of Medical Education, Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Melbourne, Victoria, Australia.
| | - Jennifer L Keating
- Department of Physiotherapy, School of Primary and Allied Health Care, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
| | - Michelle Leech
- Faculty of Medicine, Nursing and Health Sciences, Monash University, Monash Health, Melbourne, Victoria, Australia
| | - Peter Congdon
- Royal Australian and New Zealand College of Psychiatrists, Melbourne, Victoria, Australia
| | - Fiona Kent
- Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
| | - Melanie K Farlie
- Faculty of Medicine, Nursing and Health Sciences, Monash University, Monash Health, Melbourne, Victoria, Australia
| | - Elizabeth K Molloy
- Department of Medical Education, Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Melbourne, Victoria, Australia
| |
Collapse
|
5
|
The Role of B Companies in Tourism towards Recovery from the Crisis COVID-19 Inculcating Social Values and Responsible Entrepreneurship in Latin America. SUSTAINABILITY 2021. [DOI: 10.3390/su13147763] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
Abstract
One of the particularities of companies with a social purpose is that, through their business model of B companies, they have incorporated into their processes the necessary mechanisms to obtain, simultaneously, the profits to ensure the existence of the organization in the market. At the same time, social value is generated, which is necessary to address the problems of the social crisis caused by COVID-19 and the environmental problems affecting the community. The current global health and economic crisis has opened up the possibility of adopting business model B and focusing more on the individual. Based on the grounded theory method, we have examined 3500 B Corporations in Latin America, of which 57 were examined in 10 countries listed in the Directory of B Corporations for Latin America. The main conclusions are that B Corporations dedicated to tourism through responsible entrepreneurship develop a more inclusive, sustainable and environmentally friendly economy for the benefit of society, go beyond the notion of CSR and move away from traditional business, as B Corporations combine social development and economic growth.
Collapse
|
6
|
Persky AM, Greene JM, Anksorus H, Fuller KA, McLaughlin JE. Developing a Comprehensive First-year Capstone to Assess and Inform Student Learning and Curriculum Effectiveness. AMERICAN JOURNAL OF PHARMACEUTICAL EDUCATION 2019; 83:6730. [PMID: 31333250 PMCID: PMC6630860 DOI: 10.5688/ajpe6730] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/17/2017] [Accepted: 10/25/2017] [Indexed: 05/30/2023]
Abstract
Objective. To describe the development and implementation of an innovative, comprehensive, multi-day module focused on assessing and providing feedback on student cognitive and interpersonal skill development and practice readiness after the first year (PY1) of a Doctor of Pharmacy (PharmD) curriculum. Methods. A multi-day capstone assessment was developed to evaluate first-year students' knowledge of course content, ability to find and apply information, and interpersonal skills, including teamwork and adaptability. The PY1 Capstone consisted of four parts. Knowledge was assessed using 130 multiple-choice items on first-year course content and 50 fill-in-the-blank items on Top 200 brand and generic drug names. The ability to find and apply information was assessed using a 45-question open-book test. Interpersonal skills were assessed using a specially designed multiple mini-interview (MMI). The final part of the assessment was a debriefing session that provided rapid-cycle feedback on capstone performance and a bridge between students' recently completed first-year coursework and an upcoming 2-month experiential immersion. Results. The average score on the closed-book and open-book assessments were 75% and 68%, respectively. Most students displayed satisfactory interpersonal skills based on the MMI. Students viewed the assessment positively based on post-assessment survey responses (>75%). Most students (98%) reported not studying for the assessment, indicating that the results should reflect students' retention of knowledge and skills. Conclusion. The capstone assesses students on knowledge and skills and provides students with feedback on areas to focus on during their early immersion. Continued work is needed to ensure the process is transparent and cost-effective.
Collapse
Affiliation(s)
- Adam M. Persky
- Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina
- Associate Editor, American Journal of Pharmaceutical Education, Arlington, Virginia
| | - Jessica M. Greene
- Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina
| | - Heidi Anksorus
- Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina
| | - Kathryn A. Fuller
- Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina
| | - Jacqueline E. McLaughlin
- Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina
| |
Collapse
|
7
|
Watkins L, DiMeglio M, Laudanski K. Self-Assessment of Preparedness among Critical Care Trainees Transitioning from Fellowship to Practice. Healthcare (Basel) 2019; 7:healthcare7020074. [PMID: 31151167 PMCID: PMC6628175 DOI: 10.3390/healthcare7020074] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2019] [Revised: 05/26/2019] [Accepted: 05/28/2019] [Indexed: 11/16/2022] Open
Abstract
This study evaluated the subjective assessment of preparedness needs of critical care trainees and recent graduates between 2013 and 2014. A questionnaire was developed and validated by the subcommittee of the In-Training Section of Society of Critical Care Medicine (SCCM). The survey was deployed twice between December 2013 and January 2014 via email to any trainee or individual graduated from a critical care fellowship within the previous three years. Six percent (180) of all individuals completed the survey, and 67% of respondents had recently interviewed for a job. Northeast was the preferred location for a job (47%), and academia was favored over private practice (80% vs. 15%). Of the respondents that secured an interview, 55% felt prepared for the interview, 67% felt prepared to build an adequate job portfolio, 33% received formal guidance from their mentor/training program. 89% of total respondents agreed it is important to participate in a formal training course in job search, portfolio development, and interviewing process. The preferred sources of training were equally distributed between their home institution, webinars, and SCCM. There is an ongoing need in education regarding the transition period from fellowship to practice.
Collapse
Affiliation(s)
- Laura Watkins
- Department of Pediatrics/Critical Care, School of Medicine and Dentistry, University of Rochester Medical Center, 601 Elmwood Ave, Box 667, Rochester, NY 14642, USA.
| | - Matthew DiMeglio
- Philadelphia College of Osteopathic Medicine, Philadelphia, PA 19131, USA.
| | - Krzysztof Laudanski
- Department of Anesthesiology and Critical Care, Hospital of the University of Pennsylvania, Philadelphia, PA 19104, USA.
- Leonard Davis Institute of Health Economics, Philadelphia, PA 19104, USA.
| |
Collapse
|